Property-based testing with Scalacheck
Property-based testing (PBT) is a way of writing tests for your application by defining properties – the unchanging truths or characteristics of your application’s behaviour.
These properties can be derived by generalising a few examples to a common behaviour – defining invariants of your system and shaping the results of the functionality to be implemented. A common example of a property is that the reverse of a reversed string is the same as the original string.
This post was put together by our Scala team during one of its Community of Practice sessions. We’ll explore the benefits of this approach to testing, and we’ll give you a practical example of PBT in action to get you started.
While we’ll be using the ScalaCheck library here, there are plenty of alternatives available for most programming languages.
Why use property-based testing?
Unlike example-based testing, with property-based testing we don't have to provide concrete examples of what is expected of the function under test. This allows us to test the function using a much wider range of inputs and ensure its output is correct.
Due to the nature of example-based testing, not all possible domain (function input) values will be considered, and it’s possible we may fail to anticipate edge cases that cause errors in our system.
By using PBT instead, the boundaries of the input are stretched to the outer limit and we will usually uncover behaviour that we do not predict before our application fails in production.
What’s different from example-based testing?
Instead of focusing on individual examples, property-based testing forces you to take into consideration the higher-level behavior covered by multiple inputs and edge cases at the same time.
ScalaCheck – a library written in Scala and used for automated property-based testing of Scala or Java programs – comes pre-packed with quite a few generators for simpler data types, which reduces the overall time spent writing test cases.
It simplifies the process of testing edge cases by generating a random test dataset based on the input requirements, so essentially the tests require much less code to be written by the developer and a more thorough coverage.
The difference between a traditional test and a property is that tests traditionally verify behaviour based on specific data points checked by the test. A test might pass three or four specific lists of different sizes to a method under test that takes a list, for example, and check the results are as expected.
A property, by contrast, would describe, at a high level, the preconditions of the method under test, and specify some aspect of the result that should hold no matter what valid list is passed.
Should you use both?
Just because we are using property-based testing it doesn’t mean we can’t also have some example-based testing in the mix – or vice versa. Sometimes a specific example helps clarify an implementation detail better and can therefore improve the development process (test-driven development, or TDD) in such instances.
We could, for example, start by writing a few example-based tests at the start, and extract them out to a system property that will work with any input from the function’s domain.
Alternatively, we could follow a property with a specific example test to help readers of the test to better understand it and ensure that the specific input is always used to test the function.
A practical example: the 2048 game
We have selected the popular 2048 game for the sake of this exercise. We'll begin by identifying the properties, starting with something simple to get us going:
1. The amount of the output cells should match the amount of the input cells.
property("the amount of cells is the same") = forAll { (list: List[Int]) => { val line = list.map(Cell(_)) mergeLine(line).length == line.length } }That was easy – now let’s write some application code!
object Game extends App { def mergeLine(line: List[Cell]): List[Cell] = ??? } case class Cell(tile: Int) { def merge(anotherCell: Cell): (Cell, Cell) = ??? }At this point, just returning the input is enough. Let’s write the next property!
2. The sum of the line should be the same after merging.
This means that after swiping in any direction, each line will be merged and – disregarding the randomly generated tile – will still have the same sum of the tiles’ individual values at the end of merging.
This is a very basic property and just enough to validate our initial implementation steps.
property("the sum of the line should be the same after merging") = forAll { (list: List[Int]) => { val line = list.map(Cell(_)) mergeLine(line).map(_.tile).sum == line.map(_.tile).sum } }Ok, so this one was similar to the last one, and we’re noticing some repetition…let’s remove it.
implicit val arbitraryCell: Arbitrary[Cell] = Arbitrary( for { tile <- Arbitrary.arbitrary[Int] } yield Cell(tile))The above tests can now be written as:
property("the amount of cells is the same") = forAll { (line: List[Cell]) => mergeLine(line).length == line.length }property("the sum of the line should be the same after merging") = forAll { (line: List[Cell]) => mergeLine(line).map(_.tile).sum == line.map(_.tile).sum }Better!
But the same code still passes, so let’s think of something that changes the output.
3. Combining any two same numbers should result in zero and the sum.
We can add this one quite easily. It doesn’t add more functionality, but helps us think more of the properties of the function.
property("combining 2 numbers should result in 0 and the sum") = forAll { (n: Int) => mergeLine(List(Cell(n), Cell(n))) == List(Cell(0), Cell(2 * n)) }Now we need to fix the code. A sprinkle of a pattern match, some recursion, all done.
4. Move spaces to the left.
Shouldn’t be hard, right?
property("moves spaces to the left") = forAll(Gen.choose(0, 100), Arbitrary.arbitrary[List[Int]]) { (n: Int, list: List[Int]) => { val line = (list ++ List.fill(n)(0)).map(Cell(_)) mergeLine(line).startsWith(List.fill(n)(0).map(Cell(_))) } }This is a bit less elegant, but we get the property we wanted. All spaces from the right must appear on the left in the result. Fixing this test was easier than the property itself – but the knowledge of it being verified is worth every character typed!
Now let’s make it a bit more readable before we go:
property("moves spaces to the left") = forAll(Gen.choose(0, 100), Arbitrary.arbitrary[List[Cell]]) { (nSpaces, content) => { val spaces = List.fill(nSpaces)(0).map(Cell(_)) mergeLine(content ++ spaces).startsWith(spaces) } }The result appears to be working well. However, we still need to try and think of more edge cases and more properties. What has to stay invariant that wasn’t already covered?
5. An even amount of same cells results in half as many doubled cells after merging.
property("even amount of same cells results in half as many doubled cells after merging") = forAll(Gen.choose(0, 100)) { (n: Int) => { val line = List.fill(n*2)(n).map(Cell(_)) mergeLine(line).count(_.tile == n * 2) == n } }We’re getting good at this! The code we already had was passing this one without any modifications. With this, we’re confident our merging functionality is working, and we’re happy to proceed to another component.
Different types of properties
As in example-based testing, looking at both negative and positive properties is important. With the negative tests we constrain the shape of the codomain – we ‘cut out’ the impossible outputs of a function – while with the positive properties we aim to ensure the correctness of the values within the codomain, matching them to the expected qualities.
For some good reading material on selecting properties, take a look at this article on choosing properties for property-based testing.
Property-based testing on your project
If you aren’t already using property-based testing, this article will hopefully encourage you to try it out. PBT is a powerful tool that helps ensure the quality of your codebase and reduce the chances of the unexpected happening too late.
Try it. Use it. Love it!
