Exploring, Testing, Checking, and the mental model.

The beginning

You might be well aware of “Testing and Checking Refined” by James Bach & Michael Bolton, and maybe you are also aware of “The new model of Testing” by Paul Gerrard. I read both more than once and/or saw the videos and webinars.I find many useful aspects in both pieces of work.

But I want to explain more what happens behind the scenes of testing. What is testing actually when we look behind all the obvious actions. I don’t want to explain what we obviously do in a project when we perform a test. I want to try to explain what goes on in our brains when we test and check. And what value is that information bringing to a project.

Testing and checking

Set-up, assumptions, and expectations

Inside a good tester’s mind there is a huge net of information and interlinked models, many anchor points to add or retrieve information, all based upon her knowledge.

When a tester gets the assignment to test a piece of software, that we will call system under test (SUT), she starts immediately to generate a mental model and collect questions and information, without having any further details yet. This starts in the first seconds and continues until the tester gets access to more information. A matter of maybe seconds or minutes, but sometimes hours or even days.

But who or what is informing the mental model of the tester? It’s her experience paired with a certain expectation depending on the context, and curiosity. This sets the foundation of the mental model about the SUT and where it’s placed upon her existing net of models and generating sources for oracles and heuristics.

And a good tester has usually more questions than answers.

Testing

When there finally is access to more information the testing begins. Testing in the sense of learning about the SUT and collecting new information. Questions are asked by the tester to people, while reading a specification, or when interacting with the system through experiments. Answers get embedded in the model, new questions turn up. Further experiments are thought up.

Learning about a system is, and software is always a system, in my opinion always connected to mental modelling. It’s not learning a poet by heart, it’s trying to figure out how something works, and that is directly connected to system or model thinking. Experience already created a rich amount of existing models in our mind including the interconnectedness. When you now learn about a new SUT, you will start seeing smaller models within the new big model, that look familiar. So you will create a link to an existing model including assumptions and expectations.

Our experience and our current vision of the SUT set certain assumptions of how the model or parts of the model should work. There will be questions and experiments to verify these assumptions.

There are parts of the model that may be blank and need to be explored from scratch. But also these parts will start with the assumption that there is something worth to be explored and certain expectations or desires exist that help to frame the first questions or experiments, usually based on heuristics.

Models are not reality, and never will be. But the more information the tester collects, the more accurate and helpful her mental model will become, to make predictions about the behavior of the SUT. Questions and experiments or tests will be formed, which purpose it is to exclude other possible models and ideas of the SUT.

Important to remember is, that when discussing the SUT with someone, e.g. a stakeholder, you are testing the mental model of the stakeholder, not the actual one. The same is true for documentation, it’s the author’s view on the SUT, and might even bear differences between the author’s mind and what is written.

Critical and creative thinking need to be applied, because a whole army of fallacies await the tester behind every corner. But critical and creative thinking should not be part of this article.

Checking

When a tester is testing, she creates a mental model that reflects all the information, facts and interpretation of the former, and creates concepts and theories how parts of the system behave, up to a particular point in time. When the tester reached a certain confidence that her mental model or parts of it reflect the SUT, checks will be created and executed.

There are also the parts of the model that are connected with existing models which already bring a set of encoded instructions (= checks) with them.

A check is an algorithm that will describe certain steps to be performed during checking that should demonstrate the desires about the SUT based on the tester’s mental model and assumptions. It is both an attempt to make tacit knowledge explicit and an attempt to show that the assumptions are aligned with the real thing.

A positive outcome of a check, where the observation matches the expectation, will show that the SUT could still fit the underlying mental model. The problem with most checks is that they focus on narrow aspects of the model. Especially when automated, checks assert often only the absolute minimum of facts necessary to call it per definition a check.

When a human is performing a check, she is able to evaluate many assertions, that often are not encoded in the explicit check.

One check alone does not reflect a mental model, because a single check can fit like a gazillion different models. The whole set of checks narrows down the possible amount of different models, but will never reflect the whole mental model inside the tester’s mind. The checks will only represent key elements of the mental model, which leaves a lot of interpretation in between.

When a check fails, something is wrong, either with the implementation of the check, the mental model, or the SUT. A failed check does not automatically reflect a defect in the SUT. A failed check is an invitation to explore and investigate where check and SUT differ. Testing is needed here.

Checking – in the context of regression testing – is used to confirm that parts of the mental model and the SUT still fit together and are not an obvious subject to change.

If a new tester is trying to learn about the SUT, the set of checks, often called “the test set”, can help to frame the key elements of the model.

The Role of Acceptance Criteria

Acceptance Criteria describe elements of the common understanding of the SUT within the project team. The danger of using acceptance criteria is that stakeholders and team members can easily reach a shallow agreement about the understanding of the SUT. A list of Acceptance Criteria can never replace a serious discussion and sharing of ideas about the model behind the SUT.

The Role of Bugs

Bugs are deviations between the desired behavior of a SUT with the actual one. Since mental models will differ, the perception if a bug is a bug depends in some cases upon the individual.

“A bug is something that bugs someone who matters.” – James Bach

The tester herself is usually not considered as someone who matters, but she should represent the views of those who matter. Therefore a tester needs to gain the understanding of how the people or systems who matter use the SUT or how the SUT solves a problem for them.

The Tasks of a Tester

The task of the tester is to create, evolve and enhance the model of the SUT, seek discussions with stakeholders, like product owners, users, or business analysts, asking the right questions to hone the understanding of the SUT.

The tester may encode a set of checks or checkpoints based upon the mental model. Those can be used for performing regression checks or automating those scripts.

When testing the tester’s job is to compare the actual SUT with the current common understanding of how the SUT should behave using all sorts of experiments and facilitating tools. This is heavily influenced by the model the tester has in her mind.

Consequences

This understanding of testing and checking implies:

• the people responsible for testing need the skills to utilize, build and enhance a good mental model
• the stakeholders need to share their understanding of the SUT, the problem and the solution with the tester
• the tester needs to reflect the stakeholders’ understanding of the SUT to find the important bugs
• when performing a search for regression, a tester should prefer using checklists that invite to testing over test scripts that invite having a tunnel view on the SUT
• don’t reduce your testing or checking to acceptance criteria

Summary

Testing is all about design, construction, evolving and extension of a mental model. Testing produces the checks to validate the conformity between mental model and SUT.

The mental model should reflect the perception of the stakeholders rather than that of the tester.

As long as all performed check results are positive the SUT is still reflected by the created mental model. It does not prove that this model is the one. Negative check results entail testing to find out if mental model, SUT or the check needs to be adapted.

This does sadly not fit into 140 characters.

And for the end: this is how it all started

On one of my recent lunch break walks I suddenly had an idea how to compress the theory that brewed in my brain for quite a while in 140 characters.

I wanted to explain what happens behind the scenes of testing, in your mind. This was my original try:

Testing = design, construction and extension of a model. Checking = finding out if it’s still valid. Without changing & extending the model.

— Patrick Prill (@TestPappy) 5. Oktober 2015

//platform.twitter.com/widgets.js

After the tweet went unnoticed for a while, nobody less than Michael Bolton and James Bach challenged my idea. In the end I had to realize, that I need more than 140 characters to explain my idea.

I don’t want to mention the discussion here, so if you are interested what happend that day, please investigate via the above tweet.

Invited by Michael Bolton to put my thoughts into a blog, and I did. What you just read  is the third attempt to get it all right.

Thank you Michael Bolton and James Lyndsay for the discussion in the test lab at EuroSTAR.

Advertisement