Judging the Software Testing World Cup Preliminaries for Europe – Part 2

We are coming closer to announcing the winner. So here is my report of the hard part for the judges. Read here about the first part: the event.

The event itself went rather well. On Saturday around midnight Maik was finished with retrieving the results. So Sunday morning I had my first 28 Teams assigned for rating. I first retrieved all reports and prepared a sheet and spread them on all my devices, so I could do some judging whenever and wherever time allowed.

First I looked shortly through all 28 reports to see what expects me. I had set some expectations from my own experience and there were of course some guidelines for rating already there. Jackie sent a nice summary from her first round through her reports, what she liked and not liked. I fully agreed and had only one detail to add. So I knew I was on the right way for my expectations and that the other judges had a similar view.

The rating of the reports was very interesting. At first I thought I could do a fair rating by simply reading and rating. But details I saw later on reports I found essential, were missing on already rated. So back to start, preparing a spreadsheet with all essential things I want to see and off we went. But the first round of rating was rather good and fair. I had only to make two minor corrections, but with adding the info to a sheet it was easier to rate and better for future ratings. Because 28 won’t be the end.

The quality of the reports was widely spread. From some barely filled documents to 8 page reports I had a good selection of everything. My summary of the test reports from the first 28 teams: there was some good work there. I was not highly impressed nor really blown away, but I would not expect that from such small teams in 3 hours with no preparation of the SUT and so much to do. Except some teams who spent nearly no time on the report, it was a good and solid statement of work by most teams. Well done!

The bug reports were from a different quality. Also teams who did not so well on the test reports were quite good in finding and describing bugs. The main problem I have seen is, that some information were missing, to completely understand the bug and the motiviation behind it, and why development should fix this bug first. But from my list of individual findings, the first 8 teams found over 100 different bugs. So most teams were entering rather more bugs (most of “my” teams ~30), instead of really good described bugs. I think, the amount of bugs “available” had an impact on the quality of the bug reports.
One thing I saw, “there are four severities, so I have to use them all.” A couple of teams seem to have split their bugs and put something under each severity. That reminds me of my past life as a factory tester, when there were preferred / prescribed rates for the usage of test case priorities, bugs and who knows what. Just my feeling.

Regarding the bug descriptions I got the picture that most testers are good at that, which is nice to see, since bug reporting is a central part of a tester’s daily work. For all teams I have rated so far, there were at least several good bug descriptions per team. To excel in that category I was looking for a lot of ingredients. The title should have some prefix, the description should use some template, additional infos about reproduction, helping screen shots, explanations, why this bug deserves that severity and fixing at all. If it was an improvement, it should be clearly marked as such. And what I wanted to see is, that all team members use the same templates for title and description and show the same quality of bug description.

I have not seen many bugs where I would say, I don’t understand what the tester wants to say. So the rating is on a rather high level and looking for perfection. I see it in my daily work, especially when the deadline is coming closer, the bug counts are increasing, you should maintain a high level of bug report quality to help development and stakeholders to triage correctly and fix the right bugs in the available time and budget. So, even with 30+ bugs in 3 hours, you still have to keep a high level. Many teams were above average here, so well done again!

The final round of judging for Europe is just ongoing and I am through with nearly all my tasks and can’t wait for the winners to be announced.

At this point I want to congratulate all teams for participating in the contest and doing a very good job! Thank you all, judging was interesting and fun for me. I really enjoyed all the work.

Advertisement