MVP or not?! – A misunderstood pic goes viral

This morning a pic showed up in my timeline that I had hoped not to see again. But it seems the pic has gone viral and is used out of context numerous times. It was a picture from the Agile Testing Days Scandinavia keynote “Why we do not need testers on the team” by Bent Myllerup to demonstrate the concept of an agile approach to software development, delivering value more often and developing software iterative and incremental.

Let me describe what I see in this picture, and why I think it’s the wrong picture to describe the concept of MVP (the minimum valuable product). My problem is with the right part of the slide.

This picture is made by Henrik Kniberg and described in this blog. Henrik is also stating that his picture is used often, and not always in the right context. So why do I think that this picture is not showing what it should?

The upper part: For me it looks like that the customer wants a car. Somehow he gets multiple deliveries, while the product is built. I’m not sure, why would you deliver a partial product that’s unusable. But this picture should show the advantage of Agile, so I accept it as a stupid exaggeration for the sake of context.

The lower part: This picture should describe the principle of iterative and incremental development with multiple deliveries bringing (some) value to the client early to gather feedback and deliver the right product in the end. But now comes the problem of using manufacturing comparisons for software development; they usually don’t work. When software is built incrementally, you don’t need to start over after a delivery, and the part that was delivered must not be completely redesigned. In this example the customer gets 5 different products. But the second is not based on the first, the third not on the second and so on. This is an example of poor understanding of the needs of the client. This could either be that the client doesn’t know what he wants, and we slowly find out together what he wants, with an enormous effort to generate five different products. Or we were not able to ask the right questions and get additional information only by delivering something wrong to get more clues on what the client wants.
In the first case at least the client has to pay 5 different products, I suppose. In the second I presume we deliver five products for the price of the last one, which would not bring us enough money for all the effort we had.
I can imagine one scenario where this is absolutely valid. When you are a salesman of a company offering those five products and a customer comes in and doesn’t know what he wants, he can present him five different solutions, until the customer found what he wanted. But this is not Agile! And it does not apply to software development either, at least not if you want to stay in competition tomorrow.

 

So let’s look at different situations what a “minimum valuable product” is, and how you come from there to offer the client what he really wants. The scenarios I want to look at are:

• one customer
• few customers
• many customers

MVP for one customer

A customer comes to us with a problem (or multiple) to solve. Asking the right questions we could find out what the customer most obviously wants to have. We find out what the most basic thing might be to solve a part of the problem. Then start off in that direction and showing him our first throw. The customer then can tell us, if that goes in the right direction or if we need a course change. We adjust and add value to the product as we go on, until the customer really gets what he want.

I like Cassandra Leung’s approach for that to find a real live example:

MVP for a couple of customers

Let’s assume we have a couple of customers who share a similar problem. Our task would be to ask enough questions to all of them to find out about what parts of the problem they have in common, what parts are shared by some of them, and which parts are unique to some customers.

Our MVP in this case would start with providing some basic value of the common part, so that all customers could benefit from it. Getting feedback we sharpen the understanding for the problems of our customers and how to solve parts of the problems in a common way and where we need to start focusing on solving problems differently for some of them. Developing features now after customer directions might also influence some of the customers, showing them different approaches how to solve their problem. This might bring value quicker to them and also influence the desire for the final solution.

An example would be special business software, focusing e.g. on one branch of the industry, where multiple market participants share a common problem to be solved.

MVP for many customers

This is more like looking for a new product that many customers would want to use. So you need to find a common problem for them to be solved. Since there are too many market participants to ask, you better get a product owner who knows his business and has a good understanding and overview of the problem at hand. Now we start much like solving a problem for only one customer. When we bring the MVP to the mass market we start getting feedback from “many”, if our solution is serving the many or not. Based on a vision or on feedback from the customers you now start adding features, continuously collecting feedback from the users. When the product grows you might start developing features that add value to only parts of your clients.

You should not shy away to remove features, if they are not used enough. Features cost money to maintain, and if they bring no or not enough value to your customers, get rid of it again. You might also learn that your MVP is solving problems you hadn’t had in mind, when you were starting to develop it. Then it’s good to have early feedback to change your course and address a different group of customers.

A real live example would be e.g. a text messenger for smartphones. SMS were expensive, but internet data connections were cheap compared to that. So text messengers using the internet were coming up. Then adding feature by feature, like sending pictures, voice bits, videos, etc. Some features are not used by all, some more often, some less, some are removed again, when they flop.

My tip, when explaining the concept of MVP to someone, describe the context. Makes it easier to understand.
If you think I did misunderstand the concept of MVP, please let me know so in the comments.

 

One example to show how MVP development works in non-software industries, car manufacturers. It all started with Ford’s model T. There was one kind in the beginning, then slowly starting to evolve for different problems to be solved. One company that accompanied me all my life is Audi. When I was young there were about three different models. A small medium (80), a big medium (100), and a big model (200 / V8), which then became A4, A6, and A8. Something to find for most people. Then people were looking for smaller cars and Audi wanted to have those customers as well, the A3 came. Then they invented something even smaller with the A2 and something sporty with the TT. Then the palette increased on the small end with the A1, and sporty versions in the medium to large segment with the A5 and A7. Not speaking about adding cabriolets and station wagons in most segments. 2- and 4-door versions. In the meantime SUVs came up, starting with the Q7, later came the Q3 and Q5 to present small and medium-sized SUVs. It took a couple of decades, but now most customers find something to solve their problems (taking into account the size of your purse of course). And it all started with a single model. But Agile is not a solution here, because you don’t want to have half a car.

 

I want to thank the following people for making up my thoughts today, triggering the problem, challenging my statements, agreeing and disagreeing with me. I hope I didn’t forget someone.
Dan Ashby, Jose Diaz, Bent Myllerup, Thomas Ponnet, Aleksis Tulonen, Robert Meaney, Hannes Lindblom, Jokin Aspiazu, Michele Cross, Cassandra Leung, Tim Ottinger

 

Advertisement

Test Automation – Am I the only one?

What would the world of testing be without test automation? Well, I assume a bit slower than it is with it.

In this post I don’t want to speak about:

• There is no such thing as automating “testing” – I know!
• It’s not about using tools to assist testing
• It will not be a rant against vendors who declare 100% test automation is possible – no it’s not!
• Test Automation is for free, besides license costs – no, sir. It’s f**king expensive.
• Test Automation doesn’t find bugs, it simply detects changes. Humans evaluate if it’s a bug.

So what is this post about? It’s about my personal fight with test automation and the risks I identified attached to it, that don’t seem to bother most of the people working with test automation. So, am I worrying too much? You want to know what bothers me? I will explain.

There are lots of people who treat test automation as silver bullet. “We need test automation to be more efficient!”, “We need test automation to deliver better software!”, and “We need test automation because it’s a best practice!” (Writing the b word still makes me shiver.) If you are in a product/project that doesn’t use test automation, you quickly get the impression that you are working outdated.

My personal story with test automation started a while back with entering my current company. Implementing test automation was one of the main goals why I was brought in. After nearly 2.5 years there was still nothing, because me and my team were busy with higher priority stuff. Busy with testing everything with minimal tool support. Sounds a bit like the “lumberjack with the dull ax” problem, when you belong to the test automation as silver bullet fraction. No time to sharpen the ax, because there are so many trees to chop. In May 2015 I got the assignment to finally come up with a test automation strategy and a plan how to implement it. Reading several blogs about the topic, especially Richard Bradshaw’s blog, quickly formed some sort of vision in my head. I know, against a vision, take two aspirins and take a nap. But really, a plan unfolded in my head. And again we had no time at hand to start with it. Some parts of the strategy were started, some proof of concepts were implemented. Since 3 weeks now I got a developer full-time to finally implement it. Things need time to ripe at my place.

Now I am a test lead with no real hands-on experience how to automate tests and I have a developer who can implement what formed in my head. But with all the time between creating the strategy, a strategy I still fully support as useful and right, and implementing it, I also had enough time to use some good critical thinking on the topic.
And finally, last week at the Agile Testers’ Meetup Munich the topic was “BDD in Scrum”, and thanks to QualityMinds who organized the event, we got not only a short introduction to BDD, we also had the opportunity to do some hands-on exercise.

Why am I not a happy TestPappy now that everything comes together? Here are my main pain points. Risks I would like to address and that I need more time to find out about.

Why do people have more trust in test automation than in “manual” testing? It seems that people are skeptic when it comes to let people do their job and test the right things. But once you have written 100 scripts that run on their own, 3-4 per day, every day of the week, producing green and red results. It seems to me that no one is actually questioning an automated test, once it’s implemented.

Automated checks with a “good quality” need well skilled people. Your stomach makes itself ready to turn, when you read about “good quality”? Good, we are on the same page. The most important quality characteristics in my opinion an automated check should have are completeness and accuracy, stability, robustness, and trustworthiness, scalability to some degree, maintainability and testability, and some more. That’s a shitload of things to take care of, when writing some simple checks. To be honest, most of these criteria our application itself doesn’t have to a sufficient degree, at least not, when it has to stand up to my demands. How could a test team take care of all that when generating hundreds of necessary tests? Now I got lucky and was able to hire a developer to implement the automation framework of my dream, so I got some support on that front. But once we start implementing the checks itself, me and the testers need to implement or at least help to implement them. How do you take care of the problem that all checks need have “good quality” to be reliable not only today but also next week or next year.

How do I know that my script checks the right thing? I’m a very explorative tester. I usually don’t prepare too much what I’m looking for. I let my senses guide me. So when I hand over a certain area to be covered by a script, I have to make decisions what to cover. At least in my context I am pretty sure, that I will miss something, when I give that out of my control. How do you handle that?
My first attempt to implement some automated checks 3 years ago was to call every page that I could reach without too much logic and making a screen shot. I would then just quickly walk through the screenshots and check for oddities. But this is more a tool to assist my testing, and not able to run without me or some other human. Simply comparing screenshots and only checking screens with differences to a baseline is not really possible, since displayed data are usually changing often in my context.

What am I missing? My favorite topic of the past 2 years is Ignorance, in this case the unknown unknown. How do I have to handle that question? I’m sure I miss lots of things with a check, but can I be sure that I don’t miss something important? Once an area is covered with automation, how often do you come back to investigate it? Review if you missed something, if you need to extend your checks, or redesign the whole approach?

How to trust a green test? There is always the problem about false positives and false negatives. False negatives are wasting time, but in the end I have double-checked the area and covered more than the script, so I’m okay to handle those. But false positives are mean. They say everything is all right, and – hopefully – they hide in a big list of other positive tests. So for every single check, every single assertion you need to think about if there is a way that the result is “true”, when it’s really “false”.
Now it also depends on what the script is covering. If you forgot to cover essential parts, you will miss something. But the check will not tell you, it simply can’t. It’s not programmed to do so.

Automating the API/service level? Richard Bradshaw presented that nice topic to automate things on the right level on Whiteboard Testing. Many tests would be better to run on the API/service level. I agree to a certain degree. As long as there is no business logic implemented on client (e.g. browser) side that I need to emulate. When I need to mock front-end functionality to effectively interact with an API, I have to re-create the logic, based on the same requirements. Do I implement the logic a second time to also test if the implementation is correct? Do I somehow have the possibility to re-use the original front-end code and miss problems in there? Do I trust the test implementation more than the front-end implementation? If so, why not put the test code into production?

And the list of worries would go on a bit, but I stop here.

Please help me! Do I worry too much about automating stuff? I would be very happy for some comments on my thoughts, if these thoughts are shared or maybe solved? And if they are overdrawn, I want to know as well.