Paul Coyne’s talk was about the scientific method and what he learned from it about testing.
To start with a preface here. In a talk between Keith Klain and James Bach about what testers can do to improve, James said “I want you to learn how to design experiments”. That means learn about the scientific method. This is the method behind testing. So I put that on my to-do list. I have heard about it before, read the Wikipedia article, but never had the time to dive a bit deeper.
So here was Paul, talking about the scientific method, the room was quite full, and Paul seemed to get comfortable with presenting here.
Paul is a failed zoologist. So he has a scientific background and via some detours came to testing. His knowledge about the scientific method helped him to understand “testing”.
I have heard that before, but it summarizes it greatly. It’s not about “Heureka!”, it’s more like “that’s funny”. And serendipity plays a big role in both science and testing. Rikard Edgren has some good insights into that topic as well.
Science is not a body of knowledge, or a textbook, it’s not a tool, and most of all, it’s not changable. The scientific method is a way of thinking and a way of investigating. It’s there to find new information.
One black swan is all it takes. That’s in reference to Nassim Taleb.
The scientific method is “a systematic and logical approach to discovering how things work”. And that’s exactly what testing is all about.
An observation leads to a hypothesis that can be proven wrong. The hypothesis needs to be testable. And all a scientist tries to do, is prove a hypothesis wrong. Testing is trying to show that the code does not work as intended.
A failure in an experiment is not the failure to get the expected result. Same in testing, and please don’t confuse test case results like “Passed” and “Failed” here. You should always test for failure, because positive testing is not very helpful.
Good testers value the scientific method, as described in this slide:
Interesting for me was as well the statement: “All prior knowledge is provisional until disproven.” That is so true, and often seen in history.
And to come to an end here, a good test result is not “It works!”, but “I did my best and I didn’t find anything that’s wrong.”
This talk was so important in many ways. For rookie-testers who have not much experience or were trained classical in some foundation training, this provides a valuable insight into what testing really is. For more aged testers who may have lost track, that was an important lesson and will bring some of them back on the right road. For me it was both fantastic to see a presentation about the scientific method at a tester’s conference, and to watch the audience and the more and more nodding heads. And it also confirmed my beliefs about what testing really should be in a world of shallow testing.
As a personal note to the end, I had the chance to meet Paul at the conference award dinner and chat some more with him, and even better experience him in a passionate discussion. Paul is a great guy, and I am really happy I went to his talk and met him again at the dinner.
Test Pappy 