Testing is an essential aspect of software development, especially for big data applications where accuracy and performance are crucial. When working with Scala and Apache Spark, testing can get challenging due to the distributed nature of Spark and the complexity of data pipelines. Fortunately, ScalaTest provides a robust framework to write and manage your tests efficiently.

In this blog, we’ll explore the following topics:

• An Overview of ScalaTest
• Common challenges in testing Spark applications
• Selecting appropriate testing styles
• Defining base test classes
• Writing effective test cases
• Using assertions
• Running tests

Challenges with Testing Scala Spark Applications

Before jumping into how to go about testing using ScalaTest, let’s talk about challenges with testing spark applications.

• Distributed Environment:
  • Since Spark application processes data in a distributed environment testing such applications can be complex. You’ll need to ensure your tests handle distributed execution, memory management, and network issues properly.
  • It is also tedious to replicate the exact execution context in tests.
• Execution Context:
  • Spark actions and transformations can behave differently depending on the data locality and partitioning, making it tricky to test edge cases.
• Large Datasets:
  • Spark applications typically deal with large datasets, which makes it difficult to simulate production-like scenarios in tests. You need to find a balance between test data size and real-world scenarios.
• Lazy Evaluation:
  • Spark’s lazy evaluation model can lead to unexpected behavior in tests if not effectively managed.
• Long Test Runtimes:
  • If tests involve multiple stages, shuffles, or complex transformations, they might take a long time to execute, slowing down development.

Read More: Spark with Pytest : Shaping the Future of Data Testing

ScalaTest

• In ScalaTest, the core concept is the suite, which is a collection of one or more tests.
• A test is anything that has a name, can be started, and can either succeed, fail, be marked as pending, or be cancelled.
• ScalaTest provides style traits that extend Suite and override lifecycle methods, supporting different testing approaches.
• Trait Suite declares run and other “lifecycle” methods that define a default way to write and run tests.
• Mixin traits are available to further override the lifecycle methods of style traits to meet specific testing requirements.
• You define test classes by composing Suite style and mixin traits.

How to Select a Testing Style

ScalaTest offers several testing styles, such as FlatSpec, FunSpec, WordSpec and FunSuite each with its pros and cons. For Spark applications, it’s important to choose a style that balances readability, flexibility, and maintainability.

Two commonly used styles are:

• FunSuite:
  • Simple and straightforward, FunSuite is ideal for developers familiar with traditional unit testing frameworks like JUnit.
  • It provides concise syntax and allows easy organization of tests. It’s a good fit for Spark applications because Spark itself has a `FunSuite`-like style for its tests.

import org.scalatest.funsuite.AnyFunSuite   

class MySparkTest extends AnyFunSuite {

     test(“example test case”) {

       // Your test code here

     }

   }

• FlatSpec:
  • FlatSpec offers more descriptive test names, which can improve readability. It’s a great choice when you want your tests to read like specifications.

import org.scalatest.flatspec.AnyFlatSpec

class MySparkTest extends AnyFlatSpec{

“A Spark job” should “process data correctly” in {

// Your test code here

}

}

Defining base classes

Defining base classes for your tests can help reduce boilerplate code and ensure consistency across your test suite. A common approach is to create a base class or trait that sets up the SparkSession/SparkContext and provides utility methods for every test case.

import org.apache.spark.{SparkConf}
import org.apache.spark.sql.{SparkSession, SQLImplicits}
import org.scalatest.{BeforeAndAfterAll}
import org.scalatest.funsuite.AnyFunSuite

trait YourTesthelpers extends AnyFunSuite with BeforeAndAfterAll {
self =>
@transient var ss: SparkSession = null

override def beforeAll(): Unit = {
super.beforeAll()
val sparkConfig = new SparkConf()
sparkConfig.set(“spark.master”, “local”)

ss = SparkSession.builder().config(sparkConfig).getOrCreate()
}

override def afterAll(): Unit = {
if (ss != null) {
ss.stop()
}
super.afterAll()

}
}

With this trait, all your tests will automatically have access to a configured SparkSession, and you won’t have to repeat the setup logic.

Read More: LambdaTest : A Cloud-Based Testing Platform and Its Integration with the TestNG Automation Framework

Writing a test case

Once the base class is defined, writing a test case becomes straightforward. You define tests within classes that extend a style class, like AnyFlatSpec. Typically, you would extend a base class specific to your project, which in turn extends a ScalaTest-style class.

Let’s write a test for a simple transformation function that filters even numbers from a dataset.

test(“filter even numbers from a dataset”) {
   import ss.implicits._
   import org.apache.spark.sql.Dataset
   val data = Seq(1, 2, 3, 4, 5).toDS()
   val result: Dataset[Int] = data.filter(_ % 2 == 0)

   assert(result.collect().sorted === Array(2, 4))
}

In this example, we test a simple filtering operation on a dataset. We use the `assert` function to ensure the transformation works as expected.

Using Assertions

ScalaTest provides a variety of assertion methods that you can use to verify your results. Which are used to verify that the actual output of your code matches the expected output. There is a wide range of assertion methods available to handle diverse types of comparisons, such as equality, inequality, and exceptions.

Here are some examples:

test(“Simple assert”){
  val left = 2
  val right = 1

  assert(left == right)
}
The detail message in the thrown TestFailedException from this assert will be: “2 did not equal 1”.

test(“Simple assertResult”)
val x = 10
val y = 2

assertResult(200) {
a * b
}

In this case, the expected value is 200, and the code being tested is a * b. This assertion will fail, and the detail message in the TestFailedException will read, “Expected 200, but got 20.”

Here is how you can assert exceptions or errors to ensure your Spark job handles edge cases:

test(“Test exception Assert”) {
def divide(a: Int, b: Int): Double = {
if (b == 0) throw new ArithmeticException("Cannot divide by zero”)
a.toDouble / b
}

val result = assertThrows[ArithmeticException] {
divide(10, 0) // This will throw an ArithmeticException 
}
}
}

Running Tests

Running tests in ScalaTest can be done using various tools, such as sbt, Maven, or IntelliJ IDEA. These tools provide integration with ScalaTest and allow you to run tests from the command line or within your development environment. For example, to run tests using sbt, you can use the following command:

sbt test

The below command will execute all the test cases in your project. You can also run specific test suites by specifying the class name:

sbt “testOnly <TestClassName>”

Conclusion

Testing Scala Spark applications can be challenging due to their distributed nature and the complexity of data processing. However, by choosing the appropriate testing style, and defining reusable base classes, you can ensure your Spark applications are reliable and maintainable.

With the amount of enterprise data today, it is necessary to have a partner that helps in optimizing, organizing and transforming data that helps your business goals by making data readily available for analysis and action. From creating data pipelines to processing, storing and enabling access to processed data, TO THE NEW‘s Data Engineering services help you make better decisions to create robust, scalable & compliant Data Platforms and enterprise-level Data Lakes. Contact us for more details.