21 Philosophical Answers

In his blog, Colin Cherry posted 21 philosophical questions about software testing. When reading the questions, I immediately came up with some answers. So at the end of the list I thought, why not answer them in my blog. So here we are.

Software Testing: 21 Philosophical Questions

1) When you get an unexpected outcome, do you assume it’s a Bug?
That depends of course on the outcome. But my first answer would be NO. If the outcome is unexpected, I think about, why did I not expect this. What is wrong, my expectation or the outcome? Are there more possible answers than just one?
I would say, even in case of a check, not all unexpected outcomes might bugs.

2) % Production data (extracts), % manufactured data?
I really like production data. I like to have a big chunk of production data in my systems. Gives good examples on how the customer is using the system.
When it comes to new features or areas like reporting, then I would prefer also good manufactured data. If the system allows it, separated from the production data.
If you really want to know a percentage, I would say about 80/20. And it does not have to be a complete production dump.

3) Controlled analysis or independent thinking?
I like independent thinking. Most important for me is, that my co-workers can explain to me why they are thinking that way. At the end I want to know what went through their mind when testing, how does that fit to my model of the system. What am I missing and what are they missing? If they cannot come up with their own model, I try to assist with explaining my model, to help them build up their own.

4) Software Testing: Science or Art?
I would say both. It’s like an artisan. You need good or even excellent craft skills and you need to be an artist to produce something, that not everybody is able to reproduce.
There is room both for testers like craftsmen and also for artisans, if the environment allows such characters. Artists, like bug magnets without skills, not so much of need.

5) When you find a Bug do you consider it a positive or a negative?
That depends on the where and when. In general bugs are positive. Learning about the system what works and what does not. Often bugs tell me more about the dependencies and relations inside a system than the working processes. In those cases, a-ha effect, awesome.
On the other hand, like on one of my current projects. When the overall quality of a project is deep in the valley, and you think now dev got the turn, and the system gets better build by build, and then you find bugs again or you find bugs that are so easy to spot (like on log-in) and you realize, that they did not even properly test the implementations. When you realize the context of the bug, and it is disappointing, then it’s negative.

6) Is your Testing 50% done or 50% outstanding?
We are 50% done, when we saw big portions of the system and did not find many problems. If we’re halfway through the planned activities and found a lot of problems, then 50% are outstanding, with way more to come.

7) Do you look forward to discussions with Developers regarding their work?
Yes, always. Talking with the developers is always interesting. It gives an insight on the system. You see the different characters of developers. You learn what language they speak and can use their words for a better understanding. You can always ask them what areas of their code, they would test more intense. Ask them for test ideas. So much to talk.

8) How much Testing is enough?
Until some stakeholder is able to make an informed decision, where I’m sure, that he has the biggest part of decision-relevant information at hand. Too vague, of course.

9) Context-Driven or Factory-fed?
I was a context-driven tester in a factory environment for 10 years. Didn’t even know about context-driven testing during all those years, I was always disappointed with the factory-approach to the project. So, definitely context-driven.

10) Automation speeds up or slows down your Testing initiatives?
It helps a lot. But not everywhere. Bring in automation where it makes sense, bring in a good approach to automation, something that is easy to adapt, and it helps.
Try to automate everything, or a bad approach, not good.

11) Vendor, Open-Source or Bespoke tools?

12) Certification or Accreditation?
Proof! Certificates are ok. Accreditation by whom? Most value to me it to see people test, and ask questions along the way.

13) % Prevention, % Cure?
That depends from project to project.

14) Do you get concerned if you don’t find enough Bugs?
Usually, yes. Lately the software tends to break easy. So finding not many bugs makes me skeptical. But what it enough anyway. Looking into the crystal ball, rolling dice, reading tea leaves, asking dev how many bugs they have hidden. All methods are as exact as any other. So it’s only a gut feeling at the end.

15) Should the Testing Team have a say in the release of software?
I like the way my company handles it. Test is providing the information for a Go/NoGo-meeting. There the test department suggests a go or no go based on the open risks. This is not binding the stakeholders. But they have to argue to their bosses when they overrule a no go.

16) What is an acceptable pass/fail ratio for System Test?
Depends on the fail. One epic fail maybe a showstopper, hence 20 small bugs might only be a reason to deliver an update a couple of days after UAT or GoLive.

17) Triage: AM or PM?
Tried different times and approaches. No clear favorite.

18) UAT: Should non-professional Testers (i.e. Users) perform Test execution tasks?
Of course. Non-professional testers will be the users of the system.

19) Testing effort: Onshore or offshore?
Depends on the approach of testing. I tried my first SBTM project and I would not want to do it with offshore.
I like to have my team onsite communicating with each other, learning and improving.
I’m working with offshore now and I have worked both successful and unsuccessful with offshore before. It always depends on the approach, the people involved and the energy needed to keep it rolling.

20) Confidence or scepticism?
Always confident to learn more, always sceptical that PM, BA and Dev really did a good job.

21) Is your Testing completed or finished?
Only on hold for a certain time.

Thanks Colin for your philosophical questions. That brought me the chance to make up my mind a bit in some of those areas.

The Bowl of Communication

I started with my new company this year. And when I came here, the test team had a bad reputation. Managing from trans-Atlantic, bringing in offshore, being a bit offshore at the same time, even if dev is sitting also here, and having someone on the team, most people try to avoid due to bad communication skills. The team members here in Munich were not talking to each other, areas in the system were split between people, even if nearly all parts of the system interact some how.

To break the ice, I tried to communicate as much as possible with development and project management. But nobody came to my desk.

So I brought in the “Bowl of Communication”. Being a hobby woodturner helps. I put that nice thick bowl of beech on my desk and filled it up with sweets. I positioned it near the door, so that everybody coming along would see it. And it soon started to work. People came in, to grab some sweets, have some small talk, used the chance to discuss something technical.

During summer the bowl was empty. Could have brought gummi bears, but chocolates are always good, and our room was tooooo hot. Yes sir, south side, and no air-condition. Visitors came more scarcely.

In September the weather got better again. The project was a bit frustrating, so bring in the sweets again. And it helped again. People were stopping by again, sometimes several times a day. Bringing in news from the project, adding information on where to look more intensively, letting of steam, discussing bug reports, philosophizing about the specification. And all supported by a small bag of gummi bears or chocolates.

What one small bowl of sweets can change…


And yes, that’s “Explore it!” by Elisabeth Hendrickson (@testobsessed)
Always good to have some interesting looking books on the desk. Sometimes also a good way to start communication!!!

Why “why” makes a tester’s life easier…

A good story tells not only what has been done/achieved, but it also tells you WHY. How could you rate the solution (what), if you don’t know the problem (why)?

I have some examples where a short explanation of why saves time and adds so much more.

Writing test cases
Have you ever followed test cases step by step, not thinking what you do, just checking if the system meets the expected result to the letter? Were you happy with that? If yes, please stop reading.

Have you ever tried to learn something from a detailed pre-scripted test set, that explains only what to do in every detail?

“Change the string Test1234_-%$ to Test1234 _-%$”. OK, what? Let’s check the expected result. “An error message should occur.” or better “Saving should not be possible.” Now that explains everything.
A short explanation that spaces are not allowed in that string, but alphanumeric and some special characters are, would have helped to understand the what. And in this special, but real, case you create one set of test data with the same string per every execution of that step. If someone tells you, what you have to do and why you do that, you can easily come up with your own examples of test data. And maybe even find new bugs.

With one good why, you can save so many test steps and make the test case insensible to changes which would lead to a lot of non-testing time going through piles of old test cases and test steps and finding all the little adaptions that need to be made. Or having some trained monkey come to you every other minute telling you that this button is either not there anymore or the label has changed, and if that’s a bug? If you then ask, “Why me?”, because you/someone should have added a “why”.

In my experience it is much easier to write a test case, that can also be executed from someone else, when you explain more detailed why you do the following steps. OK, that someone has to have a brain and be able to use it. Rare, but possible… (sorry, it’s Friday night, I’m in the mood)
Imagine, a year from now, you have long left the project, someone tries to follow your detailed steps, but there have been some minor changes to the system, not much really. A button was added, a string has changed, there are now three options instead of two. What would you hope for? That someone explained why this test case or step has to be done. In case some changes were made the chances are high that the tester knows what to do.

Note Taking
When taking notes, what do you put down? I am a lazy note taker. I tend to write down IDs of test data sets, maybe changes I made and some results. Maybe some hints, more a kind of insider. Same for you?
I don’t know about you, but a short why would help me understand that note in a couple of weeks, without much thinking and trying to reconstruct.
More important when you fill your notes of a test session, that will be reviewed or debriefed. A short “why you did that” shows quick what you were thinking about and gives the debriefer a fast idea if that meets his expectations or if you might have missed something. In case of the test notes it might even help others to learn from your written thoughts, when they read them in a couple of month.
Shmuel Gershon said in his EuroStar webinar about note-taking, always add the why. And I have to agree, really it helps. No, I have to admit, it would help, if I would not be too lazy taking good notes.

Daily Doing
Someone has given you a task, what to do. Do you always ask “why”? Just think a bit about using “why” in your daily doing. You do so many things without asking. A short thinking about “why” might help you better understand the task. Here we come back to, understanding the problem first, then create the solution.
If you are the one giving out tasks, do you add the problem to be solved, the “why” someone has to do “something”. Of course not, you’re the boss, everybody has to do what you say.

If you want someone to learn from your test cases or you want to remember next year what brought you to write it in this or that way, always add the WHY. If you want to know if the time you will spend to fulfill a task is spend right, ask “why” first. If you want a solution to your problem from someone else, add the “why”.

Reading the customer’s expectations

For me it is very important to know my customer, know the problems and challenges he has, and to speak with him on a level, where he realizes that I understand his problems and his point of view.

I’m working in the QA department for a small software company. And so far, QA had no contact to the customers. That might not be the problem, if you have an account manager who transports the information to QA. We have such project managers, but QA was not listening so far. There were written requirement documents against which will be checked and that’s it. Of course, there might be the problem that a project manager is not used to look at the customer through QA glasses. And if he doesn’t know that QA can be flexible, why would he even think of other ways to approach the customer.

I’m used to work and speak with my customers and get an idea of his expectations from QA point of view. So I’m glad to get more and more opportunities to partizipate in client meetings and having the chance to “read” our customers and their expectations. For my team this will result in changes to the test strategy and approaches. Thanks to the participation of the customer we can now facilitate user stories in our test cases and test sessions. On the other hand this improves the customers understanding of what we do in QA (what he gets for his money spend), and has a better feeling that his point of view is used when determining the quality of the product.

I would even go so far, that I say the customer will be willing to spend more money on QA, when he is able to participate in the whole QA process. Nobody is willing to spend a certain amount of money for the paragraph, that the product has been QAed by company standards. You are willing to spend a bit more money for a product if it has a certain certificate that signals a certain amount of trust, even if you have no idea what has to be done to get that certificate on the product. But you’re willing to spend more money, if you know what is done to determine the quality of the product and that this reflects your problems and challenges. Certified or not, this is even better.

If you have a product that is sold only to one or a couple of customers, take your time to analyze your customers expectations. Don’t try to use the same strategy for all customers. Like I read in a comment the other day. If the only tool you have is hammer, all things you see are nails. Try to fill your tool-box with different approaches, strategies and techniques to meet your customers expectations. This will also improve the approaches and strategies you take for the other customers.

Be curious about who your customer is, what problems he wants to solve with your solution and what is important to him.
Try to partizipate in demonstrations and discussions. Observe the situation, observe your customer when he looks at the screen. Where is his main focus? What details are important to him? If not already defined, try to find out what the problem really is, that he needs so get solved and how important that is to him. Try to read his gestures and mimic. You will learn many things about your customers.
And don’t underestimate bugs found in production. Don’t just try to reproduce them to know how to retest them after the fix. Try to understand what your customer has done, that you obviously didn’t do. Learn from that.

But now it is up to you to use this knowledge and adapt your strategy, approaches and techniques. Involve the customer in reviews, improve your reporting and like Michael Bolton and James Bach always say, tell your client a story about your testing. If the client finds himself represented in your story, he will buy it. If not, he will challenge you.

Please don’t hide in your QA offices, go out and experience the customer.

The value of the ISTQB certificates

I want to add something to the ongoing bash to hit on the ISTQB (Keith Klain’s petition and the “discussion” between Rex Black(RBCS), Keith Klein, James Bach and more on Twitter), that does not fit in the 140 characters of Twitter.

To start with, I have an ISEB foundation certificate from 2004, and the ISTQB Advanced Level Test Manager from 2005. As James Bach wrote a couple of month ago, it’s OK to be certified, if you don’t take it too serious. And that’s what is true for me.

Because I realized too late that only taking the course is teaching you nothing as a tester. But! There is a but. But you get an overview of a small portion what is out there in the world of testing. And from there you have to improve yourself and BE a good tester. So the certificate is just a piece of paper stating that you have taken the course and you were able to answer a couple of questions.
I have forgotten most of the stuff, because I was not able to use it all and frequently. And I did not take the initiative to learn on my own, starting with the sources provided in the course material.

Currently RBCS is promoting/congratulating a guy via Facebook and Twitter that he is now certified for the full advanced level. No offense to the guy, I don’t know you. But I know a couple of guys with full advanced level certification, because my former company had some locations were they pushed the people to take the courses, because the company was also certified training provider for ISTQB (cheap in-house training). The exam was the official one, so no bonus there, they earned their full advanced level, whatever it’s worth. But I worked with some of the guys, and I would not let them test notepad without intense monitoring.

To come to the end, I finally took the initiative to learn on my own, improve my skills and add to the community. But my certificates didn’t help with that.
It’s a shame that those certificates are important to find a job.

For those of you, who are still reading, I would assume you are keeping yourself up-to-date. No matter if you are certified or not.
We have to go out there and reach those who don’t participate in self-training, no matter if they are certified or not.


Why I started this blog?

This is an easy question, but the answer is not short.

I recently read a tweet or short notice about what makes a good tester.

1. he smiles when working
2. he adds to the community

I can’t remeber the source. If anyone reads this and remembers, please tell me, I really want to give credit and state this correct.

Smiling is something that is hard to achieve at the moment, but this will come back some day, I’m sure.

So we still have the community. Since I changed companies by the end of last year, I have won a little bit of time every day in the commuter train. And I like to read all the blogs and articles, tweets and comments of the testing community.
My new job as a QA lead makes me think a lot about different aspects of testing, which is absolutley fantastic. I haven’t added much to my tester knowledge since my ISTQB Test Manager course late in 2005 and a two-day seminar from Hans Sch√§fer about risk-based test management in summer of 2006. I was working as a test manager from 2008 until end of last year. So I got mainly project management training. But my new job brings me back into the tester role. So I have to learn testing methods, techniques and all that stuff again. And I see now, that I haven’t learned much in my time as a tester from 2003 until 2007.

Reading / Learning / Thinking about all the testing stuff is great, and I enjoy it every day. And I want to give back to the community and share my testing experience with those who are willing to read it.

Plus, one benefit from writing about it is, that I rethink a topic more often and come to a better conclusion for myself.
For my new company I created a test strategy for the product I’m the responsible QA lead for. And I wrote over a period of two weeks on the set of slides while improving them bit by bit. I discussed it with my test manager and with a test specialist from another product/project and improved it further. But I was still missing the big questions, testing my strategy.
These questions finally came in the presentation. And I had a rather good answer for all of them, because I was prepared in every little bit of my strategy, why I want to do it this way, and why other ways are not as good. I’m really proud of this strategy. I flipped through the pages yesterday, about 2 months after the presentation and said, “yes, that’s still the right way”.

To summarize, why I started this blog. I want to add to the community, I want to have the chance to think more about certain topics by writing about them, and I want to discuss those topics with you. So please, use the comments to challenge my statements. I’m glad for every constructive feedback I can get.