Test Automation – my definition

This post is an explanation how I see Test Automation. There is nothing new in here, and you probably have read that all somewhere else. I just want to create a reference in case I need to explain myself again.

Over the past few days I wrote some blog post on the test automation pyramid, or rather the band-pass filter. And I wrote about the trust in testing. This lead to some discussion and misunderstanding, hence I decided to write this post.

This post is about my opinion and my take on test automation. Your opinion and experience might differ, and I’m okay with that, because – you know – context matters. You might also prefer different names for the same thing, I still prefer Test Automation.

Disclamier and Differentiation: This article is about test automation, which means tests that have been automated/scripted/programmed to run by themselves without further user/tester interaction until they detect change (see below for more details on this aspect). Using tools, scripts, automation or whatever that assist your testing, go in my understanding under the term that BossBoss Richard Bradshaw coined as “Automation in Testing” or basically as “testing”. This is not part of this article!

In my humble opinion good test automation is providing the following aspects:

Protection of value

Automated tests are implemented to check methods, components, workflows, business processes, etc. for expected outcomes that we see as correct and relevant to our product. This reaches from functionality, design/layout, security, performance or many other quality aspects that we want to protect. We want to protect our product from losing these properties, that’s why we invest in test automation so that we can quickly confirm that the value is still there.

Documentation

Methods or components and especially the code should speak for itself, but sometimes / often it doesn’t. We can’t grasp every aspect that the method should be capable of doing, just by looking at the code. We can help with that, by manifesting abilities of a method or component in automated tests.

Change detection

Test automation is there to detect change. When the code of the product has been touched and an automated test fails, it has detected change. It has detected change of a protected value. This could be a change in functionality, flow, layout, or performance. If that change is okay (desired change) or not (potential issue) has to be decided by a human. Was there a good reason for the change, or do we need to touch the code again to restore the value of our product.

Safety net

The automated tests provide a safety net. You want to be able to change and extend your product, you want to safely refactor code and you want to be sure that you don’t lose existing value. Your automated tests are the safety net to move fast with lower risk. (I don’t say that there is no risk, I just say the risks are reduced with a safety net in place.)
Also maintenance routines like upgrading 3rd party dependencies are protected by the safety net. Because changing a small digit in a dependency file can result in severe changes of code that you rely on.

And this is also where the topic of trust comes back to the table. You want to trust your test automation to protect you, at least to a certain degree!, so that you have a good feeling when touching existing code.

What Test Automation is NOT

Test automation is not there to find new bugs in existing and untouched code. Test automation is not there to uncover risks, evaluate the quality of your product, tell you if you have implemented the right thing, or decide if it can be deployed or delivered.

Test automation cannot decide on usability, or proper design or layout. Automation doesn’t do anything by itself. What it does is enable you to re-execute the implemented tests as often as you want, on whatever environment you want it to run, with as many data as you want. As long as you implemented the tests properly (different story!)

Test automation, when done properly and right, is there to tell you, that all aspects, that you have decided to protect, did not change to the degree of protection you invested in.

In the process of creating automated tests, many more things can happen and can be uncovered. I prefer an exploratory approach to test automation, but that is subject to maybe another blog post. But once the test is in place and ready for regular re-execution, it is there to protect whatever you decided is worth to protect.

Test Automation can never be your only testing strategy. Test Automation is one part of a bigger puzzle.

Advertisement

Oh, this stupid pyramid thingy…

Another blog post on the testing pyramid? RUN!!! But wait, it’s actually about an alternative. Maybe read it first, and then decide to run.

It seems to be an unwritten law that at testing conferences there needs to be at least:
– one Jerry Weinberg quote
– one mention of the 5-letter acronym in a non-positive context
– and there has to be at least one testing pyramid

Whenever test strategies are discussed these days, you will probably also find the testing pyramid been referenced. It is one of the most used models in a testing context that I’m aware of. And yet, my personal opinion is, that too many people don’t understand the actual intention behind it, or are probably unable to properly communicate it. And I blame the model for this!

All models are wrong, but this one isn’t even helpful!

Patrick Prill

The basic testing pyramid (or triangle for some) is mostly mentioned in context with the number of automated test cases to have. A whole lot of unit tests, a bunch of integration tests, and as few as possible end-to-end tests, especially when it comes to this dreadful UI thing. But what does this mean when it comes to actually designing and writing the tests? And this is the point where I feel that the pyramid has some severe short-comings. At least the key aspects that I mean, are often not mentioned together with the pyramid.

Yes, we got it, people misunderstand the pyramid thingy. What’s your solution?

Well, to start with, it’s not my idea. The model that helped me the most for the past four years is Noah Sussman‘s band-pass filter model.

The band-pass filter model by @noahsussman

What this model basically states is, that at every testing stage you should find the issues that can be found at that stage. That means, you can probably find the most issues at a unit test level. As a result you will probably have the most test cases here that focus on functional testing.

You don’t want to find basic functionality issues with an end-to-end-UI test!

On a unit test level you won’t be able to catch integration level issues. These stay in the system to be hopefully discovered in that stage.
End-to-end-tests should then find the issues when looking at the system all together, when taking a view through the business process band-pass filter.

And this model is easily extendible, by adding new band-pass filters for security testing, accessibility testing, load & performance testing, and so on. All these kind of issues should be found in their respective test stages, however you implement them.

And, of course, the optimal distribution of tests will – in the end – probably for most systems, look like a pyramid / triangle. But it doesn’t have to! It depends on your system’s architecture, design, testability, influence, and so on. Of course, the root cause that some systems cannot reach a testing pyramid lies in some problems of some kind. But that’s not a reason to not having tests!

The band-pass filter model helps you with that. You have to find the issues at the earliest stage possible. At a recent client of mine I was analyzing the unit test set. And basically most tests weren’t unit tests, but integration tests, as most of the methods under test needed DB access. That’s caused by the fact that the MVC approach is not properly used most of the times. But does this project has to refactor all code before properly starting with testing? NO! They have to start then with integration tests, and find ways to filter out all these functionality bugs on the integration test layer, which runs basically after every commit. That’s much better than waiting for the end-to-tests that only come at the end of each release cycle. As a next step they will find better ways to implement their code, with that comes better unit-testable code and more unit tests. Or not, and they stick with their way of mixing MVC and just drastically increase the amount of integration tests. Fine for me!

I gave the developers and testers of my last client an exercise to experience the power of the band-pass filter:

For the next bug report that lands on their desk, they should find a way to reproduce the issue on the earliest possible testing stage. And if that is not the lowest test level (unit test = low, UI test = high, yes, I know, pyramid lingo, duh!), try to check again, if it’s not reproducible one stage earlier.

If they need data, is there a way to simplify the necessary data to reproduce the issue. On an integration level or higher, how do you select or create proper test data? If you made it to the unit test level, you will probably be able to simply define the data or appropriately mock it.

Now that you have a failing test case on the earliest stage possible to find the issue, fix the bug and see the test case turn green.
While you are at it, you may add a few more tests on that level. Just to be sure.

Use the power of the band-pass filter model!

PS: The idea to this blog post came during TestBash Home 2021, when the testing pyramid appeared for the first time, in the first talk of the day, and the chat went wild. I stated that I prefer the band-pass filter model and earned a lot of ??? So, here is the explanation of what it is, and why I find it more useful than the pyramid!

In Test We Trust

Have you ever thought about how much trust testing gets in your projects?

Isn’t it strange that in IT projects you have a group of people that you don’t seem to trust? Developers! I mean, why else would you have one or more dedicated test stages to control and double-check the work of the developers? Developers often don’t even trust their own work, so they add tests to it to validate that the code they write is actually doing what it should.

And of course you trust your developers to do a good job, but you don’t trust your system to remain stable. That’s even better! So you create a product that you don’t trust, except when you have tested it thoroughly. And only then you probably have enough trust to send it to a customer.

Testing is all about trust. We don’t trust our production process without evaluating it every now and then. Let’s take some manufacturing industries, they have many decades, even centuries more of experience than IT. They create processes and tools and machines to produce many pieces of the same kind. Expecting it to have the specified properties and reaching the expected quality aspects every time. Depending on the product, they check every piece (like automobiles) or just rare and random spot checks (for example screws and bolts). They trust their machines – usually to a high degree – to reproduce a product thousands or even millions of times.

We are in IT, we don’t reproduce the same thing over and over again.
Can you imagine that for your project you only do some random spot checks, and only check for a handful of criteria each time? If your answer is ‘yes’, then I wonder if you usually test more and why you still test it. If your answer is ‘no’, you belong to what seems to be the standard in IT.

So, what we have established now is, that we don’t overly trust the outcome of our development process. Except when we have some kind of testing in place.

Have you ever realized how much decision makers rely on their trust in test results? If you are a developer, BA, PO, or a tester, who is part of the testing that happens in your delivery process, have you ever felt the trust that is put into your testing? Decision makers rely on your evaluation or the evaluation of the test automation you implemented!

Does your project have automated tests? Do you trust the results of your tests? Always? Do you run them after every check-in, every night, at least before every delivery? Do you double-check the results of your automated tests? When you implement new features, when you refactor existing code, when you change existing functionality, you re-run those tests and let them evaluate if something changed from their expectations. You trust your automated tests to warn you in case something has been messed up. The last code change, a dependency upgrade, a config change, refactoring an old method.

Do you put enough care into your automated tests, that you can really rely on them to do what you want them to do? Why do you have that trust in your tests, but probably not in your production code? And I don’t ask the question “who tests the tests?”

Of course we do some exploratory testing in addition to our test automation. And sure, sometimes this discovers gaps in your test coverage, but most of all exploratory testing is to cover and uncover additional risks, that automation can not supply. So, when you established exploratory testing in some form, alongside your change detection (a.k.a. test automation), you add another layer of trust, or respectively distrust to some parts of your system.

This is not about distrust, we just want to be sure that it works!

In one of my last consulting gigs for QualityMinds, I had an assignment for a small product company, to analyze their unit tests and make suggestions for improvement. The unit test set was barely existent, and many of the tests I checked were rarely doing anything useful. That wasn’t a big problem for the delivery process, as they have a big QA team who is doing lots of (end-to-end) testing before deployment, and even the developers help in the last week before delivery.

Yet they have a big problem. They don’t have enough trust in their tests and test coverage to refactor and modernize their existing code base. So my main message for the developers was to start writing unit tests that they trust. If you have to extend a method, change functionality, refactor it, debug it, fix something in it, you want to have a small test set in place that you can trust! I don’t care about code coverage, path coverage, or whatever metric. The most important metric is, that the developers trust the test set enough to make changes and receive fast feedback for their changes and that they trust that feedback.

I could add more text here about false negatives, false positives, flaky tests, UI tests, and so many more topics that are risks to the trust that we put into our change detectors.
There are also risks in this thing that is often referred to as “manual testing”. When it is based on age-old pre-defined test cases, or outdated acceptance criteria. Even when you do exploratory testing and use your brains, what are the oracles that you trust? You don’t want to discuss every tiny bit of the software with your colleagues all the time, if it makes sense or not.

We can only trust our tests, if we design them with the necessary reliability. The next time you design and implement an automated test, think about the trust you put into it. The trust that you hand over to this piece of code. Is it reliable to detect all the changes you want it to detect? When it detects a change, is it helpful? When you don’t change the underlying code and run the test 1000 times, does it always return the same result? Did you see your test fail, when the underlying code changes?

PS: This blog post was inspired by a rejected conference talk proposal that I submitted for TestBash Philly 2017. All that time since then, I wanted to write it up. Now was the time!

Context eats Process for Breakfast

That was the title of a workshop that I gave at Let’s Tests 2016 in Runö, Sweden. In my opinion I messed up that workshop, as I still think that I was not able to communicate the intended message to the participants. So I want to give it a new try, over 5 years later, this time as a blog post. Because I still think that the message I wanted to teach back then is important and basically necessary to understand certain aspects of software development.

Breakfast?! Why breakfast? Well, that was the theme of this after lunch workshop.

The first task was given to one participant only. They had to describe to me: “How to make good coffee?” Now comes the tricky part. I don’t drink coffee, I don’t like coffee, I often even can’t stand the smell of coffee. But I want to be able to offer visitors a good cup of coffee. So the volunteer described me several steps, how to get some ground coffee, put it in the machine, add some water, bring it to a boil, let it filter through the coffee and et voila, coffee! So they told how to make coffee.
But how do I, as someone who never drank and will probably never drink one, know that I just made good coffee? The volunteer told me again to stick to the process and I will get good coffee. Okay, but how do I know that I don’t produce bad coffee.
You might guess what my problem here was and what I tried to address.

The next task was split into two parts. First, the participants had to individually “draw how to make toast”. We had some excellent process descriptions and most of all some fantastic drawings, of how to make toast. Then as a small group they had to throw their process steps together to get more detailed steps, add steps that were forgotten earlier. This time the task was “to make GOOD toast”! The results were again very detailed and incorporated some excellent improvements. But nobody was able to tell me why THEIR process produces GOOD toast.

You will probably sit there, reading this post, nervously shaking, as you might see since two paragraphs now, where I wanted to point my participants to. They made great task analysis and gave excellent process descriptions. But in these three rounds of exercises nobody gave me any details on how to create something “GOOD”. They all focused on creating “something”. And I had an excellent audience this afternoon, but they refused to “get it”.

A good friend of mine loves coffee, and I like to chat with him about coffee. (“Know your enemy” is one of my mottos!) So basically I know a lot about certain aspects of the coffee making process that can influence the outcome of the taste of the product. Getting the right beans, grinding them to the right coarseness, right water temperature and pressure, preheated cups, and so on. There are so many small adjustment screws in this process that can influence the taste of the outcome.
And the same goes for making toast. What bread do you choose, thickness of the slices, what kind of toaster, temperature, time, degree of roasting. Probably even, what do you serve with the toast? What aspects make it a good toast?

Or to refine it: what quality aspects define “good” in those contexts. The answer is relatively easy: It depends! On the context and the consumer!
In what environment are you preparing your coffee or toast? For yourself, for friends, do you work at a small breakfast place or in a large hotel? Do you know your consumer, do you have influence on certain decisions, and so on.

Everybody was so busy describing the steps, that they totally forgot about the quality criteria that need to go into decisions in several steps to make something “good”.

The part that had to be removed due to not enough time, was to describe one process from their work context. My favorite example would have been processes, like the bug reporting process, because most places have one of these.

What does a bug reporting process describe. Basically a lot of things, like the tool, the workflow, permissions, which priority to select, which information to provide, and so on. The only aspect I have seen so far in the about a dozen different bug reporting processes of my life, that positively influenced the quality of the bug reports are certain rules about what information to provide in a ticket. You can google for that and you will find some great examples.
But what makes a good bug report? When I know the process how to create the ticket and whom to assign it to? When I fill out all the necessary information? Well, that makes it a good ticket for the bug report management. That is only one aspect of the whole process. But is the individual result really a good bug ticket?

How much pre-analysis have been made? Did you add log files, minimum reproduction steps, test data and so on. Did you add some bug advocacy, e.g. business reasons why this bug needs to be fixed with the given priority. Who is your audience and will read your bug report and has to understand it? Of course there are contexts, where you already spoke with the dev and you agreed to enter a bug ticket to document the fix. These tickets will probably hold less information, because they are “good enough” in that context. In other situations you know that the bug won’t be fixed now, but only later. And you don’t know by whom. How do you decide then, what is “good enough”?

That is the message that I wanted to communicate 5 years ago, so here you get it now in all explicitness.
Your context and your consumer influence the “goodness” of your product. You can describe the best step-by-step instructions, but if you are not aware of all the small factors that change the quality of your product, and if you are not aware of who your customer is, and what they value, then your process will only ever describe how to get a product, but never how to get a good product.

As a big fan of the AB Testing podcast and the Modern Testing approach that evolved from the discussions between Alan Page and Brent Jensen, I’m very happy that Modern Testing principle #5 describes exactly that message:

#5 We believe that the customer is the only one capable to judge and evaluate the quality of our product.

You can cook a good coffee for yourself, but if your guest doesn’t like it, then it’s not good coffee for them. And if they like their toast a lot darker than you like it, it’s probably okay toast, but not good.

Your processes can be described as good as you can, but still the result might suck. Please analyze your processes and understand what quality aspects and decisions can be made at every step, and how they influence the outcome of the product. And learn how to understand the actual needs and wants of your consumers to help create a product they like and evaluate as “good”.

Testing, Quality, and my inability to teach

Hi, I’m Patrick and I’m a tester for 18 years now. And I have a problem: I don’t care about testing! I care about Quality! Yet people see and treat me as a tester.

I have to add, that I don’t like testing, as many people see it. And I don’t do testing as most people do it. Most colleagues I have worked with over the past two decades see testing to verify functional correctness and sometimes even conformity with some non-functional requirements, such as load and performance.

My understanding of quality starts where most colleagues understanding ends – when explicit requirements are met. I see quality more like Joseph Juran defined it: “Fitness for use”. And the additions that have been made to Jerry Weinberg’s “Quality is value to someone who matters at some point in time” are very helpful in understanding the flow and continuous urge for adaptation when thinking about quality from my point of view. More on that in a later blog post.

Testing as an activity and my role as tester, especially as test automator, are for me the best means and position to influence a project’s and product’s quality. As long as you don’t put me in the waterfall-ish place after development finished and before delivery/operations and expect me to stay there! I’m usually all over the place, wherever people let me.

I couldn’t care less for approaches like decision matrix, boundary value analysis, path coverage, etc. Probably either because I learned about them in a formal way already in 2004 and have them internalized by now (I never explicitly use them!), or because they are the formal explanations of what I usually call “common sense of a tester”.

Tasks like “Test Analysis”, “Test Design”, “Creating test cases based on requirements” is something against my personal nature. I’m so much driven by the problems in front of me, the context I’m in, the problems that need to be solved, the things I’ve seen, explored and sensed, that coming up with “a full test set” up front, is just not my style of working.

Quality is relative, quality is in a constant flow, quality is highly subjective, quality is everywhere and nowhere, quality can’t be predicted, quality cannot be put in numbers. And that’s why my style of “testing” follows the same behavior. I just cannot reduce my work to writing and executing test cases. I just can’t!

When I have to look into a bugfix, and I see the commit is a one-liner that fixes exactly the problem at hand, probably I have even seen the code in that area before myself, had a short chat with the developer and I came to the conclusion that we are here in what Cynefin describes as the obvious domain, I’m very much fine with closing the issue and not testing it any further. Some colleagues would describe this as “not testing”. For me that is a lot of testing, even if the actual task of creating a test idea/case and executing the code against some predefined scenario never happened.

Don’t get me wrong, I’m not against test documentation and all that, which is probably required and necessary in several contexts. But writing 95% of the documentation upfront, even before any line of code is written, and just adding the fact that I did what is written there, was never for me.

My “testing” is a complex and unpredictable process, that uses a lot of experience, common sense, systems thinking, domain knowledge, and many more aspects. Which is actually causing a big issue for me. I was so far unable to teach other testers, especially the younger generation, to mimic my approaches. Except one. But from a senior role that is sort of expected. Here is an attempt to describe why I am not able to do that.

Being the tester in a team means for me that I support my team to establish and maintain trust in the code we build and deliver, help us optimize the way we work, help to come up with solutions for the problems our clients and we are facing. I try to open up bottlenecks, never be a bottleneck myself, enable the team to act fast, and most of all I help to uncover potential risks, so that we are at least aware of them, talked about them and included mitigations for them, if relevant, in the solution we came up with. I cannot describe what I’m doing any better or more precise than that. Simply because I don’t know where I can help next.

Tomorrow I might:

• write test designs, when I have to,
• automate some test cases,
• improve existing test scripts,
• pair up with a developer
• refactor the test automation framework,
• pimp the Jenkins pipeline,
• explore
• step in for the product owner
• participate actively in refinement meeting, and with actively I mean, I don’t only ask questions for clarification, I also propose actual solutions for the problems at hand,
• I might pick up a story to implement,
• do a code review
• discuss with the business architect
• help my tester colleagues
• suggest architecture improvements
• analyze test failures after the latest pipeline run,
• discuss how we can reorganize the team structure to become better
• update dependencies
• sit in a meeting and just listen
• step in for the scrum master
• take care of the test infrastructure
• or any other task that has to be done to deliver value to the customer, improve the quality of our own working, or just help future me to have a better day in a few weeks/months.

How many of these tasks do you read or expect in actual tester position descriptions. How many of those do you expect to be part of a tester training syllabus. And before the suggestions comes, I don’t see myself as a coach. I’m a hands-on person, I taste my own dog-food, and I want to stand behind things I propose. I lead by example, and hope that others are able to understand what I’m doing, and mimic my behavior to become better. Whenever I see behavior that is worth mimicking, I try to do that.

Quality, value, improvements, reducing waste, and making an impact drive my daily actions. I think, despite the level of Impostor Sydrome I suffer from, that I’m doing a good job, having a big impact on teams. At least that’s the feedback I get sometimes. I don’t even want to teach developers HOW to test. I’m rather good to help developers WANT to test.
But please don’t assign me any rookie and expect me to teach them how to test. In about 17 of 18 cases I will most probably fail miserably.

Something that just came into my mind when reviewing the post a last time:
I try to positively influence systems to heal, maintain, and improve. And as an embedded tester I have the chance to do that from within the system. But what I am doing to achieve that is so much more than just “testing”.