Another blog post on the testing pyramid? RUN!!! But wait, it’s actually about an alternative. Maybe read it first, and then decide to run.
It seems to be an unwritten law that at testing conferences there needs to be at least:
– one Jerry Weinberg quote
– one mention of the 5-letter acronym in a non-positive context
– and there has to be at least one testing pyramid
Whenever test strategies are discussed these days, you will probably also find the testing pyramid been referenced. It is one of the most used models in a testing context that I’m aware of. And yet, my personal opinion is, that too many people don’t understand the actual intention behind it, or are probably unable to properly communicate it. And I blame the model for this!
All models are wrong, but this one isn’t even helpful!
Patrick Prill
The basic testing pyramid (or triangle for some) is mostly mentioned in context with the number of automated test cases to have. A whole lot of unit tests, a bunch of integration tests, and as few as possible end-to-end tests, especially when it comes to this dreadful UI thing. But what does this mean when it comes to actually designing and writing the tests? And this is the point where I feel that the pyramid has some severe short-comings. At least the key aspects that I mean, are often not mentioned together with the pyramid.
Yes, we got it, people misunderstand the pyramid thingy. What’s your solution?
Well, to start with, it’s not my idea. The model that helped me the most for the past four years is Noah Sussman‘s band-pass filter model.
What this model basically states is, that at every testing stage you should find the issues that can be found at that stage. That means, you can probably find the most issues at a unit test level. As a result you will probably have the most test cases here that focus on functional testing.
You don’t want to find basic functionality issues with an end-to-end-UI test!
On a unit test level you won’t be able to catch integration level issues. These stay in the system to be hopefully discovered in that stage.
End-to-end-tests should then find the issues when looking at the system all together, when taking a view through the business process band-pass filter.
And this model is easily extendible, by adding new band-pass filters for security testing, accessibility testing, load & performance testing, and so on. All these kind of issues should be found in their respective test stages, however you implement them.
And, of course, the optimal distribution of tests will – in the end – probably for most systems, look like a pyramid / triangle. But it doesn’t have to! It depends on your system’s architecture, design, testability, influence, and so on. Of course, the root cause that some systems cannot reach a testing pyramid lies in some problems of some kind. But that’s not a reason to not having tests!
The band-pass filter model helps you with that. You have to find the issues at the earliest stage possible. At a recent client of mine I was analyzing the unit test set. And basically most tests weren’t unit tests, but integration tests, as most of the methods under test needed DB access. That’s caused by the fact that the MVC approach is not properly used most of the times. But does this project has to refactor all code before properly starting with testing? NO! They have to start then with integration tests, and find ways to filter out all these functionality bugs on the integration test layer, which runs basically after every commit. That’s much better than waiting for the end-to-tests that only come at the end of each release cycle. As a next step they will find better ways to implement their code, with that comes better unit-testable code and more unit tests. Or not, and they stick with their way of mixing MVC and just drastically increase the amount of integration tests. Fine for me!
I gave the developers and testers of my last client an exercise to experience the power of the band-pass filter:
For the next bug report that lands on their desk, they should find a way to reproduce the issue on the earliest possible testing stage. And if that is not the lowest test level (unit test = low, UI test = high, yes, I know, pyramid lingo, duh!), try to check again, if it’s not reproducible one stage earlier.
If they need data, is there a way to simplify the necessary data to reproduce the issue. On an integration level or higher, how do you select or create proper test data? If you made it to the unit test level, you will probably be able to simply define the data or appropriately mock it.
Now that you have a failing test case on the earliest stage possible to find the issue, fix the bug and see the test case turn green.
While you are at it, you may add a few more tests on that level. Just to be sure.
Use the power of the band-pass filter model!
PS: The idea to this blog post came during TestBash Home 2021, when the testing pyramid appeared for the first time, in the first talk of the day, and the chat went wild. I stated that I prefer the band-pass filter model and earned a lot of ??? So, here is the explanation of what it is, and why I find it more useful than the pyramid!
Test Pappy 