I used the phrase “regression testing” for about 10 years, never even thinking about the definition of the word regression. I just accepted the term as it was used commonly and frequently in our projects. This was before enjoing a webinar of EuroStar with Michael Bolton. One hour talking about the term regression and regression testing. This webinar changed my tester life.
One thing was, that I began to read blogs and articles about testing, the other thing was, I started to think about many of the terms I used in my daily life and if I used them wrong so far. I tried to challenge some of my colleagues with discussions about those terms that are used every day in our project. The outcome was, that they, too, had not spend much time thinking about those terms and accepting it, as it was.
Coming back to regression and regression testing. In both my old company and my new one regression is used as a synonym for regression testing. Because a common problem seems to be, that people don’t know what regression means. Maybe the reason is, that I am German and in German the word “Regression” is not present in the common vocabulary.
Regression: to regress originating from the latin word regressus, means to go back or as a noun a backward movement.
Wikipedia says under “Software Regression”:
A software regression is a software bug which makes a feature stop functioning as intended after a certain event (for example, a system upgrade, system patching or a change to daylight saving time).
So what is our intention, when we speak of regression testing? To check if the (hopefully) unchanged features are still working as intended.
Instead of looking for a definition of regression testing, I want to use the four different but intersecting concepts, that Michael Bolton offered in the aforementioned webinar:
- Any test that we’ve performed before.
- A set of automated checks, run periodically and repeatedly.
- Testing that we perform after some change.
- Testing to probe whether quality has got worse.
When different roles / stakeholders in a project speak about regression testing, which concept or which mixture of concepts do they mean? My tip, ask them. In long grown teams / projects it is interesting to see if all are on the same page or if the definitions vary. If they miss one of the concepts completeley, challenge them by asking about them.
What test cases will be used?
On the one hand we can use any test cases that we’ve performed before. Because their goals should be defined to check some features of the product at test, that we want to re-check. But be careful when reusing test cases that were originally written to test a new requirement / feature / change request. Those test cases might be going to deep into detail and be too time consuming.
If you don’t have a good selection of test cases that you can use for regression testing, what have you done until now? Now is the time to start creating a regression test set.
When automated tests are available, great. Just run them all if possible. In case the features are still working the same as in the last version, all tests have to get the same result as they did on the last run. And yes that means, that we also would expect failed tests to fail the same way. How would you react if a test that failed before is now passing, without knowing of a bug fix for that problem?
Do we have to change the test set every time?
That depends on the reason or event that triggers the execution of the test set. If you change a feature, the related regression test cases have to be reworked. If you prepare for a major release change, what has been changed? Are your test cases general enough to be executed on the new version? If you switch the platform like the web server or the operating system, I would go with the last test set.
In case the software changes, check if those changes affect your test set.
And what is a short regression (test) doing?
I think we all heard from our project managers and steakholders the term, let’s cover this with a short regression (test). And with short, they mean you don’t have much time and budget for this. I will show one of the many choices how to reduce your test set for a short regression test. If that fits to your project or if that fits every time, you have to decide yourself.
I am wondering lately, mostly now that I have to test again, how does a good regression test case look like?
Since it all comes down to money in the end, my main aspects would be these:
- check at least all the box features
- reasonable in execution time
- easy to maintain
For me the answer to this question would be, the test case fits into the big picture of my regression test set strategy. To meet those properties above, I created a strategy to rework the regression test sets of my projects.
My regression test set strategy
Get a good visual of your product at test (use Mindmaps, Visio charts, whatever suits you best). You should find all of your features represented in this picture. Try to make a picture of the specification, as good as you can get with the design techniques at hand.
Create a set of test cases that represent all of your features, also involving roles, use cases and business processes of your customers. Use scenario testing techniques together with claim testing, to cover your complete list of features. Create positive checks of all your features. When performing this list of test cases you get a result if all features are still working. At least for the well-minded users. This test set is in my opinion the smallest test set you should perform, when speaking of regression testing (Short regression test set). If you have to make this even shorter, use risk analysis to skip the not so important functions. But please remember to report, that you skipped it.
Since maintainability and reusability of those test cases is important, keep them at a detail level that is sufficient enough for most testers to understand and high-level enough not to change it every time someone changed a configuration or translation. e.g. if your product or feature is well-documented on the GUI or more or less self-explaining, you can skip most of the details what to do in the test case. You can concentrate on the expectation and goal of the test case rather than on the test steps.
The context-driven part
Now that we have checked that all our features are still working, let’s hunt for some bugs. Nobody said, that you cannot look for new bugs, when performing regression tests.
Use your positive check scenarios and combine them with different assignments (e.g. James Whittaker speaks of tours) to explore and test those features (use function and domain testing techniques or whatever comes to your mind). Not every tester thinks the same, not every tester is capable of the same test techniques. And that’s ok. Because bugs aren’t all the same either. And when it comes down to testing and looking for bugs you should leave your team members a certain degree of freedom what to do to your system. Most testers get new ideas how to break a feature during test execution. And using variation in your techniques is a good way to find new bugs. And in case a tester has no idea, what harm to do next, she can still check old session protocols to get an inspiration.
Take a test case or a subset of test cases and create a variant of them concentrating on different testing techniques to test those features for bugs. When those test cases are performed, you can report, that a search for bugs with certain techniques in this area was performed. Like all test cases, it cannot say, we tested this feature completely and it is bug-free. So this statement is enough for me.
Since you are never finished testing, please don’t discuss if it wouldn’t be better to write down all negative tests performed for a certain feature to repeat them every time again, I disagree. The test case will be too long and unmaintainable in no time. And why should you find more bugs when repeating the same tests over and over again. You can give hints in the test charter what techniques you could use, but this should be the maximum to influence you or other testers. You should try to come up with new ideas every time. Try to learn new techniques and use them in those sessions. If you have a day of no inspiration, use old session protocols, talk with your team members or simply concentrate on positive tests. We all have those days sometimes.
If you have a special technique that you find often bugs with, feel free to use this technique in all of your test executions, mention them in the test charter, but don’t script them in detail. And if this technique really finds bugs always and everywhere, you should have a word with the development lead how to improve the skills of the development team to prevent those bugs in the near future.
Make a list of the last fixed bugs found in production. Retest those bug fixes again. It’s really bad to bring an already fixed bug back into production. Try to retest all of the bugs on your list. Sometimes the reoccurrence of a low priority bug is more noticeable, than a high priority bug in a constellation that happens only now and then, and only a few users would ever see.
So what do we have:
- a big picture of the product
- matching test cases for verifying the features
- a strategy for negative testing (bug hunting)
- a couple of bug fixes to retest
This should leave us with good input to report to stakeholders, a big picture to understand the software, a reasonably maintainable test set and an approach for bug hunting which challenges your team, the capability to find new bugs and a reduced risk of reoccurring bugs that were fixed lately.
I don’t know of this is the best solution for my projects, but I know that after some analysis it should be a better solution than the existing one. But it still has to prove this. I’m sure that this strategy is not reworked for the last time. There is a big part of context-driven testing, which is project-appropriate application of skill and judgment, as it is short and easily described on Cem Kaners blog. This also means, that this concept might only fit partially, or even not at all to your situation. Don’t forget to include the expectations of your stakeholders.
My original intention of this article is to get you think also about your usage of the words regression and regression testing and to give some hints how to improve a strategy every now and then, when the context changes or there might be other or newly learned things that might fit better. Maybe my improvement of my project’s test strategy helps you to come to a better strategy for your project.
Update (11.06.2013): I could have saved a lot of time thinking about regression testing and writing this blog post, if I simply came across Iain McCowatt’s blog earlier. He wrote a five part series about regression testing, that is going even deeper. A must-read if you haven’t already done so: Exploring Uncertainty
But at least I now know that I’m not the only one thinking that way.
Comments are welcome!
Test Pappy 