Skip to main content

Don’t answer a question, ask a question - Estimates and Testing

As a tester you may be asked this question…

How long will the feature take to test?

Recently I was asked this and I responded with something like this….

I can test it in 5 minutes, 5 hours, 5 days, 5 weeks or 5 years.

Now I hate giving estimates. People tend to take estimates as gospel and hold you to them if you then have to say that the testing may take a little longer. As testers we can test a feature for 5 minutes but we could also test it for 5 years. Trying to get the balance right between having enough time to test the feature and satisfying someone who needs it to be delivered as quickly as possible can be difficult. However I think that we should not be asked how long something takes to test, I think as testers we should be asking a question instead. 

Going back to answering the question “How Long will the feature take to test?”, lets take an example feature and describe what we could test for each of the estimates I gave above.

Lets say the feature is one where the user can add and remove tasks to a to do list application on a mobile phone (android and IOS) and the items you add and removed are synced to a  database on the cloud so you can access them anywhere.

Below are some of the things you could test for each of the estimates given. 

NOTE: These tasks will be done on both Android and IOS

5 minutes estimate
Add a task and remove it from a to do list

5 hours estimate
Add multiple tasks to multiple to do lists and remove them
Add notes with various special characters

5 days estimate
Add multiple tasks to multiple to do lists and remove them
Add notes with various special characters
Adding and removing tasks in different locations where there may be data coverage issues

5 weeks estimate
Add multiple tasks to multiple to do lists and remove them
Adding and removing tasks in different locations where there may be data coverage issues
Add notes with various special characters
Security testing
performance testing

5 years estimate
Add multiple tasks to multiple to do lists and remove them
Adding and removing tasks in different locations where there may be data coverage issues
Add notes with various special characters
Security testing
performance testing
Testing on new versions of the operating systems

Now the above list is not exhaustive (obviously) but hopefully the get the idea, the greater the estimate the more information we can learn about the product. 

What estimate is best?
There is no best estimate in this example. 

As you can see I could test the application in 5 minutes but the depth of my testing is very shallow and I have just done the bare minimum that would give me a sense that the very basic functionality works. It maybe that when I run this test I will have mobile data coverage and everything will be perfect on my phone and the test passes. Great!!! I can say that all my tests passed and that the feature is ready to go live. What could possibly go wrong?........

I could test the application in 5 years. That would mean that I would cover new operating systems (which I know is never ending but humour me) and I could really test this thing in depth. However, by the time I have finished testing, the competition have release the best To-Do list app the world has ever seen and as a company we are going to have a hard job in convincing people to switch to our app. 

Each estimate will have its benefits (time, coverage, balance between the two) but no particular one is correct. Estimating is such a subjective ‘thing’ that what one person thinks would be enough time would never be enough for another tester. Why is this? Its because of context (and potential the confidence of the tester in themselves and the feature) and also because I think the question “How long will this take to test?” is incorrect. 

What should be asked then?
As testers we are information providers. And to provide that information we are given a limited amount of time to test a feature or application. This is often due to factors such as testing being squeezed and not enough testers on the project. As time is always something that we never have enough of we should be asking as testers “How much do you want to know about the feature?” 

How do we know how long something is going to take to test if we have no way of knowing what the end goal of the testing is? If we are unable to relate the estimate to any kind of 'definition of done' our estimates could be as long or as short as we are willing (or forced) to make them. This will either put us under enormous pressure to finish all of the testing in the allocated time or feel that the time we spent testing was not long enough to give us a level of confidence that the feature is ready to go out to our users. 

So think of it like this....

When someone answers the testers question of "How much do you want to know about the feature?" with something like “I want to know that a user can add a note and the feature should stop users from adding notes if there is no data coverage”, we now have something that has been defined that we can use to base our estimates on. This will potentially remove the pressure of too much testing to do in a short period of time and give us the confidence that when the feature goes out to our users it has had enough testing.

So if someone asked you how long x take to test, then the question around and ask “How much do you want to know about the feature?"

Comments

Post a Comment

Popular posts from this blog

Testing and Mindfulness

How aware are you? Do you live in the here and now or is your mind always somewhere else? This blog post is about Mindfulness. Mindfulness is a simple meditation and is defined as (According to Wikipedia): "The intentional, accepting and non-judgemental focus of one's attention on the emotions, thoughts and sensations occurring in the present moment" Now Mindfulness has become more popular in the west in recent years as it has shown to have benefits for people who are suffering from Depression and Anxiety. It has been around for a while and is often thought to of originated from Buddhism and some people believe it started thousands of years ago. Now modern life is hectic and I’m sure we all have lots of things going on in our lives that keep our Brains busy and trying to focus on one thing at a time can be a challenge. I can't remember the number of times I've been doing something and my mind is somewhere else entirely. Mindfuln...
Post a Comment
Read more

How to deal with pauses and timeouts in specflow

So this blogpost is in response to the weekly Specflow blog posts and challenges that have been written by Gojko Adzic. This weeks challenge was how would you rewrite or rephrase the below scenario: Given a user registers successfully When the account page reloads And the user waits 2 seconds Then the account page displays "Account approved" My initial though was something like this: Given a user registers successfully  When the account page reloads   Then the account page is displayed within a satisfactory time period     And the account page displays "Account Approved" Now the problem with this scenario is what defines a satisfactory time? You could add it as a comment or in a scenario outline but over time the time a user waits could change and if this is updated in the code behind but the scenario outline or comments are not, then what the test does and what is described do not match - this would potentially cause issues in the future. My next ide...
Post a Comment
Read more

Building a test strategy for a new team

Teams, we have all been on them. Some are good and some are bad. Some we never wanted to leave and others we probably couldn't wait to leave. Now most of the time (well in my experience anyway) you tend to get put into a team that already exists. Maybe you are a new hire or maybe you have asked to change to a different product team.  When you do this, more than likely there will already be a testing strategy in place. It may be that you adapt it and change it in any way you see fit to improve the testing. But imagine if everyone on the team was new? How would you decide your testing strategy? This post will go through some useful things you can do to help a new team develop a test strategy. Table of Contents 📈 What is a Test Strategy? 🤔 Where should I start? 🎯 Understand the company and their goals 💪 Play to the teams strengths 👁️‍🗨️ Understand what quality looks like 📏 Understand Scope 🧪 Understand the type of tests you need 📊 Measure your success 🤝 Collaborate 📝 Summar...
Post a Comment
Read more