Weekend Testing America – Testing Deep (WTA-52)

Somehow I managed to join my first session of Weekend Testing, in my case session WTA-52 of Weekend Testing America.

It was a very promising topic, “Testing Deep”. What does it mean, when do we know we are there, and is there a point of “enough”?


The session was facilitated by Justin Rohrman, because Michael Larsen was busy, and there were 13 participants including Justin and Michael. I guess most of the participants, if not all, I know from Twitter. So it was interesting for me, to interact with them kind of live for the first time.

Justin set the mission with some questions to consider during the session:
– how do we know what we are doing is deep testing?
– what do we do different (thought process, approach, techniques, etc.)?
– how to actually do deep testing (hint: staring at a feature longer doesn’t make it “deep”)?
– how do we know when enough is enough?

And he offered some ideas to start with

1 – what is it
2 – how do we know when we are there
3 – how do we know we are not being shallow
4 – how does it feel
5 – what are we actually doing different
6 – how is our mental process different‏

Also I want to mention, that the credit goes to Michael Bolton, who also attended, and James Bach. Justin got the idea for this topic when taking Rapid Testing Intensive (RTI).

Justin chose the online collaboration wordpad titanpad.com for the SUT. And I think this was a good choice for a software to try deep testing. I guess all had immediately an idea what that software was intended to do. So Justin gave three example areas to chose from for “deep testing”. Some participants where building teams to test the collaboration features together, and some, like me, were testing alone. Well, sort of. Being in a Skype chat with 12 other people who explore the same software is not really being alone.

Exploring / Hands on “deep” testing

I set my expectation for testing deep. Instead of wandering around and building up my model of the whole application layer by layer, I chose Export/Import and I wanted to stay at one part of the feature as long as possible. I opened up my XMind and started sketching the feature to test. I started testing the Export feature, file format HTML. I soon realized that my private notebook is not yet set up for supporting testing. But at least I had Firebug installed already and just downloaded Notepad++. Soon I started finding minor problems in the HTML structure, translating empty lines in enumerations, and so on. The ideas, what to do next kept popping into my head. But then I moved to the next export format, taking time, coming up with ideas and exploring further. But as I have to reflect now, the time I took for exploring each export format got shorter and shorter.

And then on Skype problems kept popping in. But I was trying to keep my focus on export. Some interesting bugs in Import were mentioned, so I soon extended my focus a bit. Maybe too much. I wanted to test Import, too. I also wanted to see those errors. I wanted to use the file I exported, and import them back.

At the end I started a bit interacting with the others. Trying to better understand the situation they faced and trying to help. Yes, I know funny. Why should I be needed to help them?


The hands on part was over after nearly an hour. How time flies by. So all were gathering in Skype again for the discussion.

Justin took some interesting notes about his feelings during the session. I find this a good source of information, at least something that informs yourself and those who know you. I read something in a blog lately about capturing the feeling of a tester at the beginning and the end of a session. After that list from Justin, I really have to try that in action.

An interesting comment came from Neil Studd, refining the initial question:
You can’t go deep until you know how deep deep is.

Amy wrote:
How about ‘going beyond the expected’. I felt like I was forcing myself to go back and take another look even when I thought I’d seen it all.

I found that idea rather familiar with what I tried myself. At least in the beginning. I tried to force myself to keep on digging. And I have to say, in the beginning it was rewarding, I found more issues on each return.

Richard and Amy came to the conclusion, they found some sort of wall, where the collating of information by simply using the application stopped. They came up with a model, how to get to more information beyond that wall. I am pretty sure (and hope) that Richard will write something up about that model, so I am stopping here.

I came up with the definition, that it’s “digging so deep that the Information you encouter there are no longer in the responsibility of the owner/creator/dev of the app?”. But is it really helpful to test every feature until you arrive close to the hardware level? I don’t think so now.

I liked Neil’s addition: “if we use depth with an ocean analogy, the seabed is not flat – some investigations are likely to hit bedrock (i.e. non-issues outside our control) sooner than others‏”.

We then discussed a bit around what deep is, without coming to a conclusion that most were comfortable with.

Then Michael brought first this definition from the RST:
Here’s what we say in Rapid Testing: testing is “deep” to the degree that it reliably and COMPREHENSIVELY fulfills its mission AND to the degree that substantial skill, effort, preparation, time, or tooling is required to do so.

I keep falling over the term “fulfills its mission”. At my company the time for testing is sort of fixed and restricted as a ratio of the amount of development effort (in many cases, not all). The time sets for me the main part of my daily mission. So I can only test as deep as possible in the given amount of time. So testing is kind of shallow, in most cases, by definition of the time box. Everything that goes beyond that is deep testing for me. So every feature I decide to spend more time on as planned by someone else, someone who does not know what all is possible, or maybe necessary, is testing deep.

I questioned Michael, if depth is something he would want to “measure” in some way to report on it. Because as with Neil’s example earlier, for some features the sea bed is not as deep as for others. Michael’s response was, that the extent of the mind map one created while exploring might show the depth of investigating. The answer is in most ways okay for me, because it shows that you invested time there and digged up lots of information. If you really hit ground or how deep you came is still hard to tell. But define “ground”, there lies also the solution to the question we were working on.

Michael brought in this list in regards to the SFDIPOT:
For a given feature or function…
– to focus on that feature or function
– to consider a wide variety of risks
– to use and/or develop a very detailed structural diagram
– to break the function down into a detailed set of sub-functions, and to test each one
– to use highly diverse and extensive data sets
– to identify and exercise as many interfaces as are there
– to test on a wide variety of platforms
– to consider and work on a wide variety of operational models
– to consider and test for lots of interactions with time‏

SFIDPOT is hanging on my office wall to remind me every moment I need it. But I didn’t recognize it at first. Maybe too much to read in the Skype chat.

Justin brought then up this idea: If you are testing and discovering / creating the model as you go, you are always at the “bottom” of the model. So, are you always doing deep testing when this is happening?

I was not completely happy with that definition, because it would mean, that you are at this stage from the very beginning. But something of this idea still revolves around in my head.


In the last 5 minutes we were asked to share a definition of what “deep testing” is. Some gave it an immediate try, and some were retrieving to come back with the ultimate answer later. From the definitions that came up, I found none that satisfied my view on the topic, that I just started to think about for the past two hours.

Michael gave a good summary of what is needed to go deeper, and therefore complete the picture he set earlier with his definition and SFDIPOT:
Time: All this takes time to develop, maintain, and perform.
Determination: It’s hard to blunder into deep testing. You have to want it.
Skills: You need to know how to model products, identify testable conditions, and design experiments to evaluate them.
Learning: You need to a rich and detailed model of the product and its risks (may be a mental model, formal model, or both).
Requisite Variety of Test Activities: You need to work out a pattern of test activities that will find the obscure, yet important bugs, based on a good theory of risk.
Tooling: You may need tools to help you cover large areas or to reach otherwise inaccessible areas of the product.
Environments for Testing: You need a requisite variety of test platforms configured and available for tester use.
Data for Testing: You need a requisite variety of test data so you can trigger the important bugs.
Team Support: You may need lots of eyes and minds poring over it. Developers can help immensely by exposing the code.
Testability: You may need special features in the product that help you observe and control it.

After sleeping a night over the session I tried to come up with a perfect definition for me of “Deep Testing”. But the idea is still so vague and tacit, that I am not able to write it down now. If I will be able, you will read it here…

Michael explained the parts how you can structure the width of the hole, SFDIPOT, and also the abilities you need to exploit each of the areas in depth. The initial definiton “testing is “deep” to the degree that it reliably and COMPREHENSIVELY fulfills its mission”, sounds to me, that this is the ultimate depth of testing something. OK, then everything above that level is to a certain degree shallow. But even that is still vague and depends on context.

Now comes the hard part, at the job, you need the budget to get the time to to do all this, so you should be able to estimate and sell this strategy. But how do you know, how deep deep is and how much time it takes? That’s something to sleep over the next night(s).

Thank you Weekend Testing America, that was an interesting and inspiring session. Thanks to all who attended and enriched the discussion.


3 thoughts on “Weekend Testing America – Testing Deep (WTA-52)”

  1. Nice report.

    Credit is due to James Bach for the definition of deep testing, and to the second list (Time, Determination, Skill…). Those things came from Rapid Testing Intensive Online.

    Something that you hint at here, but don’t quite state outright: depth is subject to The Relative Rule. What looks deep to one person might be shallow to another. One way to evaluate your depth is to create a map of what you’ve done and what you’ve learned, perhaps based on SFDIPOT and/or the “Time, Determination, Skill…” lists, and look at it critically. One way to test your self-evaluation is to cultivate a community of people you trust (both inside and outside your workplace) and get review and feedback.

    —Michael B.

    1. Thanks Michael for your comment.
      This component was a bit out my understanding on Saturday, so maybe that’s the reason why it came short in my recap. But with the way you just wrote it, it makes absolutely sense now and adds a missing piece to my puzzle. Now I also understand the 7-8 minutes where the discussion was focusing on objectivity and subjectivity.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: