Looking for a mentor

I’m looking for a mentor in testing now for about a decade. I finally found one…

In the beginning of my career I shared the same situation with most coworkers. We were transferred from a development/consulting department to the test department, which was a promising business for my former company back then. We had three big test projects in parallel. With big I mean 50+ people assigned to the testing part of the projects. So every department/team that was not so successful became transferred to test, due to the high need of people. That was my start in testing.
But everybody in my new team was also coming from a development background, so we learned only the testing basics that were taught back then in ISEB test foundation and the rules that were set up by the more experienced colleagues. They were not good testers (at least not in my current view), but good managers for (test) projects of that size. What I learned best in the first five years, was mostly managing. I was managing my 3-man-team, supported my team lead for a 10-15-man-team, and excelled in leading a team in big test scenarios covering the complete system stack for phases of ~2 weeks.

When I finally became a test manager back then, I had a mentor in the first couple of months pushing me to become a good test manager. But my mentor left the project and I was there on my own again. The following 4 years, I was a coach for new test managers, the go-to-guy for nearly the complete project and also some other departments we were working with and even the customer. But no one was there to guide me, no one to challenge me, but myself. Nearly 10 years in testing and managing and no I idea what I was doing, because I was not really that good, better than many others maybe, but not good.

What I realized only over the last year is what my problem was. I was a more or less context-driven tester and test manager, loving to explore, learning, adapting to the current situation, caught in the processes and demands of a 50-70 people test factory. I was not able to understand that problem, and why I was mostly frustrated for the last two to three years in that project. And the ancient processes forbid to act context-driven. And if it wasn’t the process, then it was the team that refused to change. So I finally left.

With my new job I came back to testing, and I caught the fire. I love to test. I’m not an expert or even remarkable at it, but I work on it consciously and consistently.
But again, no one in sight to learn from, at least not testing. I have several good colleagues, all of them across the ocean. The test team in Europe is me and about two to seven team members, rookies or beginners, needing guidance themselves. Again, no mentor left for me.

But I finally found my mentor. Me! I define, where I want to go, what to learn next, where I want to get better. I take my sources for improvement from the community. Anything that is shared in my timeline on Twitter I scan for interesting discussions, blogs, books and videos. There are of course several people to look up to, people of inspiration. I would love to have some budget to visit a conference, to meet some of them in person, or take the RST, or having time to take the BBST.
I found out, in the end you need to be your own mentor. Kick your own butt to learn and get better at what you do. Colleagues, coaches and mentors will help you with that, but when it comes to find and walk your way, you are on your own.

I envy companies like Moolya or the Barclays GTC or the awesome PerScholas project, because they have great people there to look up to, to follow after. But in the end, even they need to find their own way. Even those mentors can only show you the rough direction where the path is or how it might be looking like. You need to walk the path alone. If you have the chance to interact with the community or friends or colleagues, you have (changing) company on the way, helping you, pushing you forward, making it easier. But you still need to do the walking on your own.

The CDT community empowers me to be my own mentor, finding a way to define myself. I’m only at the beginning of that way, but finally I have at least the feeling to be on my way.


My year 2013 in review

The year is finally coming to an end. So it is time for a résumé.

It was a long year for me. On January 2nd, I started my new job and with that a change in my tester’s life began, in several ways.

I’m only following the community now for about 15 months and the CDT-community has changed my life as a tester. I started thinking more about testing and all that belongs to it, than ever before. Following a lot of great people on Twitter, reading lots of blogs and articles, reading great books about testing and not-directly testing related books, testing magazines and watching videos from some of the test conferences brought a new dimension to my tester’s life.
A short Twitter discussion or dialog with great people like Michael Bolton or the always inspiring Leah Stockley can absolutely make my day. Those are only two of many examples.
Thanks everyone for teaching me something new every day.

My new job showed me, how bad I am as a team lead and as a manager. Maybe I am not really that bad, because at least some members of my team think I’m a not so bad as a boss. But I became very critical with myself. Well, my new job showed me, that I can be a good tester again after all those years. Good, nothing more, but I would say also nothing less.
In the beginning I had to deal with my new team members and convincing them, why I am the new lead. And I was not able to convince a “veteran” of 13 years to change his style of testing and documentation. The decision of the upper management was then to let him go instead of me, but that was my first personal defeat of the year. He started reading and learning about testing only after he got his notice, but he never changed his way of doing things.
Another defeat was a situation with the coming back of two contractors to my team, who were trained (badly) on the product I’m responsible for, the year before. Those contractors were not onsite, so harder to control and manage, at least for me. And I was not willing to spend enough time to extend my tracking of their work in every little detail and was trying to compensate it on my own, which was not really possible. I was not willing to teach and train contractors who call themselves senior testers and consultants, who do such a bad testing job, how to do it better. With that I got a lot of problems later that year because of the bad quality we delivered as a company. But that’s what a lead is for, to be blamed, which is OK for me. I am responsible for the testing, and that was done very bad.

But with my improved testing skills and the confidence I gained using them, I became way more efficient to the end of the year. I was even able to serve six or seven projects in parallel, additional to the budget planning for 2014. Not to imagine what would have been possible if I would have had the time to concentrate on one project at a time.

I was becoming so confident with my testing skills and everything else I learned to step forward in the companies’ monthly know how transfer sessions and deliver one or sometimes even two topics per month. Sadly most of the time I was presenting alone. But at least I stood up and tried to spread the word amongst the companies’ test departments. I want to improve the style and quality of testing of my complete company, after I have seen how much I was able to improve in such a short time.

With two contractors, not the same as above, onsite now, and an intern I started my first project with session-based test management, and it went awesome so far. I prepared quite a long time, read lots of blogs and articles about the topic and created a test strategy I am really proud of. With the help of my colleagues who have never even heard of session-based testing before, that became a big success. I still have to do a lot of home work to improve it and focus more on the debriefing, but what a great and fitting approach for the project I planned and used it for. The team has fun, the reporting is inspiring and with things like group debriefing we are all able to learn from each other. And I learned a lot.

The time on the commuter train is awesome to keep up-to-date with Twitter and all my reading. I was driving by car or sometimes by bike to work for the past 13 years. Now I enjoy my time in the morning and evening. I have about half an hour each way for my own reading, which is hard to keep up at home. I nearly tripled the way to work, but I learned to spend the time wisely.

Pocket is an awesome helper to keep track of things to read. Seeing a link on Twitter, add it to Pocket and it read it later. Seeing a link in a blog while using Pocket to read, add the link and read it later. If I want to skim through the articles I have read, it’s easy.
Pocket sent me a mail, that I belong to the top 5% of pocket users. Having read about 14 times the Great Gatsby. I might not have read all of the articles that I saved to Pocket to the full extend, but it did not count the about 5% of articles that pocket was not able to display, the linked PDFs, testing magazines, the books I read, and the videos I watched. So, wow, I think, I read more than the past 5-10 years in total.

One bad thing I also had to learn is, that the tension with all the problems I had to face at the end of the year, can create the most severe neck pain I ever had in my life. I’m glad to have one more week to get rid of that.

And I even have some plans for next year. Not too much, because I like to react to the things as they come along. But the following things are on my list.

  • Improve my test documentation. The documentation suffered a lot, serving too many projects in parallel. So it needs to get better, which should not be that hard.
  • Improve my scripting and automation skills to finally get some automation in place and speed up my testing even more.
  • Read more books. There are a couple of books on my list that sound promising, so I am looking forward to that.
  • Improve my managing skills with the help of the basics I learned from SBTM.
  • Provide more training sessions to my colleagues and inspire them to step forward and deliver some nice topics on their own.
  • Improve my own testing, of course. The poor development team has never seen more bugs found than in the past quarter and I hope to improve that even more with the little team I will have.
  • If I find the time, I want to continue and hopefully finish renovating the guest bath room. My wife has to suffer under that construction site for now nearly a year.
  • I want to spend more time on my lathe and produce some beautiful things of wood. There is so much I want to learn and practice. And I want to make and sell more pens again next year, making people happy who use them or give them away as a gift.
  • And of course I want to spend as much time as possible with my now nearly four year old daughter and my wonderful wife.

I want to thank everyone in the community for their inspiration and I wish you all a happy year 2014.

21 Philosophical Answers

In his blog, Colin Cherry posted 21 philosophical questions about software testing. When reading the questions, I immediately came up with some answers. So at the end of the list I thought, why not answer them in my blog. So here we are.

Software Testing: 21 Philosophical Questions

1) When you get an unexpected outcome, do you assume it’s a Bug?
That depends of course on the outcome. But my first answer would be NO. If the outcome is unexpected, I think about, why did I not expect this. What is wrong, my expectation or the outcome? Are there more possible answers than just one?
I would say, even in case of a check, not all unexpected outcomes might bugs.

2) % Production data (extracts), % manufactured data?
I really like production data. I like to have a big chunk of production data in my systems. Gives good examples on how the customer is using the system.
When it comes to new features or areas like reporting, then I would prefer also good manufactured data. If the system allows it, separated from the production data.
If you really want to know a percentage, I would say about 80/20. And it does not have to be a complete production dump.

3) Controlled analysis or independent thinking?
I like independent thinking. Most important for me is, that my co-workers can explain to me why they are thinking that way. At the end I want to know what went through their mind when testing, how does that fit to my model of the system. What am I missing and what are they missing? If they cannot come up with their own model, I try to assist with explaining my model, to help them build up their own.

4) Software Testing: Science or Art?
I would say both. It’s like an artisan. You need good or even excellent craft skills and you need to be an artist to produce something, that not everybody is able to reproduce.
There is room both for testers like craftsmen and also for artisans, if the environment allows such characters. Artists, like bug magnets without skills, not so much of need.

5) When you find a Bug do you consider it a positive or a negative?
That depends on the where and when. In general bugs are positive. Learning about the system what works and what does not. Often bugs tell me more about the dependencies and relations inside a system than the working processes. In those cases, a-ha effect, awesome.
On the other hand, like on one of my current projects. When the overall quality of a project is deep in the valley, and you think now dev got the turn, and the system gets better build by build, and then you find bugs again or you find bugs that are so easy to spot (like on log-in) and you realize, that they did not even properly test the implementations. When you realize the context of the bug, and it is disappointing, then it’s negative.

6) Is your Testing 50% done or 50% outstanding?
We are 50% done, when we saw big portions of the system and did not find many problems. If we’re halfway through the planned activities and found a lot of problems, then 50% are outstanding, with way more to come.

7) Do you look forward to discussions with Developers regarding their work?
Yes, always. Talking with the developers is always interesting. It gives an insight on the system. You see the different characters of developers. You learn what language they speak and can use their words for a better understanding. You can always ask them what areas of their code, they would test more intense. Ask them for test ideas. So much to talk.

8) How much Testing is enough?
Until some stakeholder is able to make an informed decision, where I’m sure, that he has the biggest part of decision-relevant information at hand. Too vague, of course.

9) Context-Driven or Factory-fed?
I was a context-driven tester in a factory environment for 10 years. Didn’t even know about context-driven testing during all those years, I was always disappointed with the factory-approach to the project. So, definitely context-driven.

10) Automation speeds up or slows down your Testing initiatives?
It helps a lot. But not everywhere. Bring in automation where it makes sense, bring in a good approach to automation, something that is easy to adapt, and it helps.
Try to automate everything, or a bad approach, not good.

11) Vendor, Open-Source or Bespoke tools?

12) Certification or Accreditation?
Proof! Certificates are ok. Accreditation by whom? Most value to me it to see people test, and ask questions along the way.

13) % Prevention, % Cure?
That depends from project to project.

14) Do you get concerned if you don’t find enough Bugs?
Usually, yes. Lately the software tends to break easy. So finding not many bugs makes me skeptical. But what it enough anyway. Looking into the crystal ball, rolling dice, reading tea leaves, asking dev how many bugs they have hidden. All methods are as exact as any other. So it’s only a gut feeling at the end.

15) Should the Testing Team have a say in the release of software?
I like the way my company handles it. Test is providing the information for a Go/NoGo-meeting. There the test department suggests a go or no go based on the open risks. This is not binding the stakeholders. But they have to argue to their bosses when they overrule a no go.

16) What is an acceptable pass/fail ratio for System Test?
Depends on the fail. One epic fail maybe a showstopper, hence 20 small bugs might only be a reason to deliver an update a couple of days after UAT or GoLive.

17) Triage: AM or PM?
Tried different times and approaches. No clear favorite.

18) UAT: Should non-professional Testers (i.e. Users) perform Test execution tasks?
Of course. Non-professional testers will be the users of the system.

19) Testing effort: Onshore or offshore?
Depends on the approach of testing. I tried my first SBTM project and I would not want to do it with offshore.
I like to have my team onsite communicating with each other, learning and improving.
I’m working with offshore now and I have worked both successful and unsuccessful with offshore before. It always depends on the approach, the people involved and the energy needed to keep it rolling.

20) Confidence or scepticism?
Always confident to learn more, always sceptical that PM, BA and Dev really did a good job.

21) Is your Testing completed or finished?
Only on hold for a certain time.

Thanks Colin for your philosophical questions. That brought me the chance to make up my mind a bit in some of those areas.

The Bowl of Communication

I started with my new company this year. And when I came here, the test team had a bad reputation. Managing from trans-Atlantic, bringing in offshore, being a bit offshore at the same time, even if dev is sitting also here, and having someone on the team, most people try to avoid due to bad communication skills. The team members here in Munich were not talking to each other, areas in the system were split between people, even if nearly all parts of the system interact some how.

To break the ice, I tried to communicate as much as possible with development and project management. But nobody came to my desk.

So I brought in the “Bowl of Communication”. Being a hobby woodturner helps. I put that nice thick bowl of beech on my desk and filled it up with sweets. I positioned it near the door, so that everybody coming along would see it. And it soon started to work. People came in, to grab some sweets, have some small talk, used the chance to discuss something technical.

During summer the bowl was empty. Could have brought gummi bears, but chocolates are always good, and our room was tooooo hot. Yes sir, south side, and no air-condition. Visitors came more scarcely.

In September the weather got better again. The project was a bit frustrating, so bring in the sweets again. And it helped again. People were stopping by again, sometimes several times a day. Bringing in news from the project, adding information on where to look more intensively, letting of steam, discussing bug reports, philosophizing about the specification. And all supported by a small bag of gummi bears or chocolates.

What one small bowl of sweets can change…


And yes, that’s “Explore it!” by Elisabeth Hendrickson (@testobsessed)
Always good to have some interesting looking books on the desk. Sometimes also a good way to start communication!!!

I am a man of strong convictions…

“I am a man of strong convictions, but I hold them very lightly”
by James Bach

What does that mean? It means that James will fight tooth and nail for the things he believes, the things he holds dar, the convictions he has, but he reserves the right to be convinced otherwise, and if he can be convinced, he is perfectly willing to drop the previous conviction.

comment by Mike Larsen

Why “why” makes a tester’s life easier…

A good story tells not only what has been done/achieved, but it also tells you WHY. How could you rate the solution (what), if you don’t know the problem (why)?

I have some examples where a short explanation of why saves time and adds so much more.

Writing test cases
Have you ever followed test cases step by step, not thinking what you do, just checking if the system meets the expected result to the letter? Were you happy with that? If yes, please stop reading.

Have you ever tried to learn something from a detailed pre-scripted test set, that explains only what to do in every detail?

“Change the string Test1234_-%$ to Test1234 _-%$”. OK, what? Let’s check the expected result. “An error message should occur.” or better “Saving should not be possible.” Now that explains everything.
A short explanation that spaces are not allowed in that string, but alphanumeric and some special characters are, would have helped to understand the what. And in this special, but real, case you create one set of test data with the same string per every execution of that step. If someone tells you, what you have to do and why you do that, you can easily come up with your own examples of test data. And maybe even find new bugs.

With one good why, you can save so many test steps and make the test case insensible to changes which would lead to a lot of non-testing time going through piles of old test cases and test steps and finding all the little adaptions that need to be made. Or having some trained monkey come to you every other minute telling you that this button is either not there anymore or the label has changed, and if that’s a bug? If you then ask, “Why me?”, because you/someone should have added a “why”.

In my experience it is much easier to write a test case, that can also be executed from someone else, when you explain more detailed why you do the following steps. OK, that someone has to have a brain and be able to use it. Rare, but possible… (sorry, it’s Friday night, I’m in the mood)
Imagine, a year from now, you have long left the project, someone tries to follow your detailed steps, but there have been some minor changes to the system, not much really. A button was added, a string has changed, there are now three options instead of two. What would you hope for? That someone explained why this test case or step has to be done. In case some changes were made the chances are high that the tester knows what to do.

Note Taking
When taking notes, what do you put down? I am a lazy note taker. I tend to write down IDs of test data sets, maybe changes I made and some results. Maybe some hints, more a kind of insider. Same for you?
I don’t know about you, but a short why would help me understand that note in a couple of weeks, without much thinking and trying to reconstruct.
More important when you fill your notes of a test session, that will be reviewed or debriefed. A short “why you did that” shows quick what you were thinking about and gives the debriefer a fast idea if that meets his expectations or if you might have missed something. In case of the test notes it might even help others to learn from your written thoughts, when they read them in a couple of month.
Shmuel Gershon said in his EuroStar webinar about note-taking, always add the why. And I have to agree, really it helps. No, I have to admit, it would help, if I would not be too lazy taking good notes.

Daily Doing
Someone has given you a task, what to do. Do you always ask “why”? Just think a bit about using “why” in your daily doing. You do so many things without asking. A short thinking about “why” might help you better understand the task. Here we come back to, understanding the problem first, then create the solution.
If you are the one giving out tasks, do you add the problem to be solved, the “why” someone has to do “something”. Of course not, you’re the boss, everybody has to do what you say.

If you want someone to learn from your test cases or you want to remember next year what brought you to write it in this or that way, always add the WHY. If you want to know if the time you will spend to fulfill a task is spend right, ask “why” first. If you want a solution to your problem from someone else, add the “why”.

My 2 cents on metrics in software testing… (Part I)

As always, it depends on your situation and your context. This is my personal view on using metrics. And in this article I want to add information and thoughts about the different metrics that are commonly used in software testing projects, when it comes to measuring the success of Testing Service Providers.

In the past I gained a bit of experience with adding off-shore resources to test teams, working together with an off-shore dev team and using a complete near-shore test team for certain parts of a big integration project.
My new team is supported lately, again, by a couple of testers from a Testing Service Provider located abroad. They were assigned to another project team the first half of the year, now they are back to my team. Since I started working at the company only by January 2013, I had no chance yet to evaluate the quality of their work. So I started to look for methods how to measure external testing sources effectively.

When I skimmed through the “Practical Approach to Software Metrics” by Cem Kaner, I came across some points that I want to summarize with my words.
* We use metrics to gain information, but most metrics are invalid (to some degree) for that purpose.
* We have to learn about strength, weakness and risks of our tools / metrics, to improve them and mitigate risks.
* We need to look for the truth behind numbers.
* We need to use detailed, qualitative analysis to evaluate the validity and credibility of the metrics.
So this will be rough guide for me to evaluate the metrics I found.

And I will never forget the statement, if you measure someone by numbers, the measured will become this number.
And there is a saying in Germany from the electrical engineers, that’s translated like: “who measures, measures crap”, in regards to influencing the system being measured by the measuring itself.

One of the first sources of information I found was this webinar by RBCS about “Measuring Testing Service Providers” that I read about on Twitter. Since I know Rex Black only from the Twitter bashing contests about “ISTQB” and “Best Practices”, my expectations were set to a certain level. But I have to tell you, I got not that disappointed I expected to be.

Since RBCS is a testing service provider itself and coaches about measuring his own kind, that’s a bit like asking the wolf to help protect your sheep barn from wolves. But since the companies that use testing service providers are not interested in sharing their knowledge and experience with the world, all that’s out there is coming from TSPs and consulting agencies. But let’s take a look now at those metrics.

“Find defects”
Measure the count and priority of defects and compare them with the defects found in production. The metric is called “defect detection effectiveness” or “defect detection percentage” (DDP) and I first learned about it nearly a decade ago. This is to evaluate the effectiveness of the different test stages. You simply count all bugs found in all test stages available, including production and you compare the number of defects (usually in addition by priority) of your test stage with the next one. From stage to stage you should find fewer defects. The theory expects good test stages to have a DDP of 85% – 90 % and up.
This is only possible to measure once the product/project is live for a certain time frame (usually 90 days). So you get a result only way after the job has been finished. You only get valid results of your TSP if you outsourced the stage completely or have another way to limit the calculation to the work packages the TSP tested.
And you get valid results only for certain kinds of projects. You need a good base for your production bugs. Do you have many customers, a few or only one? How is the discipline of your customers when it comes to using the defect process? Are your customers telling you about every bug they found? And you will need some time to filter through the bugs to prepare the data for comparison.

“Find important defects”
Of course this is a variant of the “Find defect” metric, focusing on e.g. priorities of critical and high or whatever grades you measure your defects with. So all restrictions of the “Find defect” metric count in here, too. Plus there are the risks, that you might not have the same prioritization, once the project is live. And the TSP might try to rate his found defects higher to increase this metric.

“Cover the Test Basis”
I quote this directly from Rex’s slide: “Engagements should include clearly defined test scope (e.g., requirements, risks, etc.), which is the test basis”, then you might measure the test basis coverage.
That is basically a good idea. But how do you define if a requirement, risk, etc. is covered completely, enough, at all, or to your satisfaction. Without a very good understanding on how to test what item on the list you measure, this metric shows completely nothing. And as always with a metric, every item counts the same. Is that the truth in your projects?
You might have a valid point here if you use the “metrics” used in a report dashboard, like for session based testing. (e.g. as described here)
But you still have to trust the TSP about the degree of coverage or you need a very exact description for every item.

“Report in Time”
This “metric” counts if the regular reports are delivered on time. Now that is nice to evaluate the testing skills of your TSP.
Yes, discipline is important for a TSP. But if the report is on time tells you nothing about the quality of the report nor the quality of testing that the report is about. So, nice add-on, maybe, but not useful for the original purpose.

The next metric is only in, because Rex mentioned this.
“Assign skilled, qualified testers”
And the according metric would be, “percentage qualified testers assigned”. I won’t count that in as a reliable field for installing a metric due to many reasons. Qualifications, resumes, certifications can be trimmed in a certain way and in a certain degree. If you really speak with the people themselves that you will hire, there are always some good in self-marketing and some not so good. But that doesn’t say a thing about their abilities as a tester. And of course there is always a chance, that you end up with another “resource” than you initially hired, because that resource was not available and of course you get someone with the same experience and quality. Right!

“Finish within approved budget”
Well, that could be either a metric or a criterion for finishing the tests. Stop testing, when you’re out of budget. But when it comes down to a metric, even Rex states, that you need a good estimation process and change request process in place. OK, but when you don’t hit the budget, was it the estimation or change request process or was it the performance of the TSP?
And Rex mentioned, of course, the positive return on invest. But why should I meet the budget a 100% to keep it positive, Rex? Is your ROI, however you define that for testing, calculated so tight, that you cannot afford to spend even 10% more without additional benefit?

And now to the surprising part of Rex’s webinar. That’s a method I have already seen in action, but completely forgot about.
“Stakeholder Surveys”
“Meaningful, Actionable Results Reporting” and
“Defect Report Satisfaction”
Now that’s something where I see the aspect of quality measured. You ask the stakeholders and project members about an evaluation of different topics, give them grades like in school, and measure that over time.
If you can keep this on an objective level, and use good facts as reason and examples for your evaluation, that has in my opinion the most value.
Negative aspects about that “metric”. First the objectivity; if some project members don’t like each other or have other personal differences, that will influence the report. Second, using the results for project-political reasons (saw that the last time I participated in the evaluation process). That will falsify your context and with that the value of the metric. And last to name here, it is very intense and time-consuming to make this right. So far from this set of slides.

I know of a metric, that is pretty special, when it comes to measuring TSP.
“Number of test cases executed”
If you use a pre-scripted approach and have a certain number of test cases to execute, that is a well known way to measure your progress. You can split it to priorities if you want, but of course it doesn’t take into account a lot of other things, like size of the test cases, time for execution, and so on. It lacks a bit context. And it tells you nothing about the quality of your TSP.
And who is writing the test cases? Do you have them already and you have experience on the execution of the test set. Great, then you will have a benefit for measuring the TSP, if you take the quality of each and every execution into the context. If the TSP writes the test cases himself or gets even paid per test case, that will be a mass production of stupid test cases, guaranteed.
I remember of a special call for tender for a complex project. The customer wanted to pay per test case and wanted a rough number of planned test cases without giving much information about the infrastructure. Now that is one serious base for estimation and offering.

Lately I found a nice white-paper by Infosys: Realizing Efficiency and Effectiveness in Software Testing
What can be used for testing projects might be adapted to measuring outsorced testing as well. Some of the metrics were already covered by Rex’s slides, so I won’t go through all of them, that I find useful.

The “test progress curve” (S curve), well that’s one nice piece of theory. In 11 years of testing I have never seen a S-curve without faking. The theory behind that is quite simple and understood, but reality is something that does not look like that. So even if you want to measure the test progress with this. Keep in mind, the S-curve won’t stand long. So the difficult task is, where to set the expectation.
But you have to measure the test progress somehow, that’s for sure. So keep in mind, you might find the S in the end, but it won’t be there all the time or not at all.

“Test execution productivity trends” is a metric that I would like to try. Short description from the white paper: “The test execution productivity may be defined as the average no. of test cases executed by the team per unit of time.”
It might fit well into the theory of thread based test management, I have to find out more about that. In case of using pre-scripted test cases, where you might have an experienced basis for execution length, this can be covered pretty good. I think the metric needs to be adapted to every project in a way to normalize the measured values. Not every test case takes the same time to execute. You need to take into account number of found bugs, problems with test environment availability, and simple things like meetings, status reporting and so on. So not a simple task, but you might get some good numbers if you can keep it up.

That’s what I’ve found so far, very disappointing overall.

In my opinion you need to monitor at least progress and quality.
What metric you use depends on your project and the possibilities you have.
What you need to do if the metrics don’t hit the expectation depends on your project and the context why it missed the expectations.

If you’re already measuring your projects with some of the metrics described above and never asked yourself what the numbers tell me. Try an experiment/role play and try to explain the numbers from different positions. Don’t forget to subtract tacit knowledge in your experiment. Now re-evaluate the value of your metrics. If you still think those metrics are good, congratulations. I would like to hear about your successfully used metrics. Either via comment, email or Twitter.

This was only part 1 about this topic. I will try to write more about metrics I found and some of the metrics I tried.