Test Desiderata

✍︎System under test

Go placidly amid the noise and haste, and remember what peace there may be in silence

Recently, I published this tweet:

Tests should be coupled to the behavior of code and decoupled from the structure of code. Seeing tests that fail on both counts.

I thought this property of tests was obvious and widely understood, but the tweet blew up. I have seen tests which are just a horrible syntax reiterating exactly what is already said in the source code under test. I went and re-read TDD: By Example and these properties are nowhere to be found, so I guess I shouldn’t have been surprised.

The experience got me thinking about the other properties of tests that I take for granted. Here, then, is a list of properties of tests. Not all tests need to exhibit all properties. However, no property should be given up without receiving a property of greater value in return.

Properties

• Isolated — tests should return the same results regardless of the order in which they are run.
• Composable — if tests are isolated, then I can run 1 or 10 or 100 or 1,000,000 and get the same results.
• Fast — tests should run quickly.
• Inspiring — passing the tests should inspire confidence
• Writable — tests should be cheap to write relative to the cost of the code being tested.
• Readable — tests should be comprehensible for reader, invoking the motivation for writing this particular test.
• Behavioral — tests should be sensitive to changes in the behavior of the code under test. If the behavior changes, the test result should change.
• Structure-insensitive — tests should not change their result if the structure of the code changes.
• Automated — tests should run without human intervention.
• Specific — if a test fails, the cause of the failure should be obvious.
• Deterministic — if nothing changes, the test result shouldn’t change.
• Predictive — if the tests all pass, then the code under test should be suitable for production.

Combinations

Here are some attractors in the space of all possible tests:

• Programmer (aka “unit” tests). Give up tests being predictive and inspiring for being writable, fast, and specific.
• Acceptance. Make tests readable for non-programmers even if you give up some speed and specificity.
• Monitoring. Yes, monitoring is a test too, or rather tests and monitoring are both forms of feedback. Monitoring abandons being predictive and to some degree being automated (modulo alerting).

So What?

Look at the last test you wrote. Which properties does it have? Which does it lack? Is that the tradeoff you want to make?