Skip to main content

Trusting automation

Trust & Automation

The word trust, what springs to mind when you hear that word?

Often we hear this word regarding companies like Facebook and Google. We like to think that we can trust these companies but like with the recent Facebook and Cambridge analytica story, that may not always be the case. Yes, we can complain and say how outrageous it is but at the end of the day if we read the terms and conditions as well as all the other small print we would probably all have a rough idea of what we are setting ourselves up for. After all if the application is free there is probably a catch.....

Trust can be defined as:

noun
1.
reliance on the integrity, strength, ability, surety, etc., of a person or thing; confidence.


Now trust is important when that reliance is required by us in order to perform our roles as testers. In this article I am taking about trust in the automation space…..

Why do we need trust in our automation?


We need to know that the automation that is been developed and run can be relied on to provide the results in which it claims it should (There may not be any explicit claims but there will probably be plenty of implicit assumptions made by people about the automation). If as testers (and a team) we don’t trust the automation then what is the point in doing it? Companies can invest a lot of time and money into automaton and if this is not trusted then all that investment is totally pointless. We need to know that when we come in in the morning, that CI build that has been run, has run a suite of automated tests that we can trust. Now you could say that by trust in the automation what I actually mean is having trust in the developer(s) that write the automation. This, although true, (after all they are the ones that code the automated tests) teams that fail should fail together. So who is responsible for producing automation that everyone trusts? Why of course, the entire team. 

By having automation that we trust can give us testers the freedom to test the areas of the system that would not get touched by the automated tests. We can write charters, delve and explore the system which we otherwise would not be able to do. Imagine if the automated tests were not trusted…..

Inside Testers Head: 

“Mmmmmmmm build is done and latest release is deployed. I need to start testing…… oh hold on a minute!!! Those automated tests don’t fill me with confidence. No one has reviewed them and  person who wrote them is not known for their attention to detail. OK if thats the case I will pull the acceptance criteria from the user story and I will test that these scenarios behave as per the acceptance criteria. Yes that would be the 'safest' thing for me to do.”

Not a great scenario is it? Although the automation has been coded, the tester does not trust it, and they are just running through acceptance criteria. What about that bug which will bring the system to a standstill when a particular edge case occurs? Tough, probably not going to get found. The tester is basically running through the tests that have been automated, which is a total waste of time. Now as you explore a system of course you are going to touch upon areas that have been automated, that is expected, but you shouldn’t have to go through the exact same tests as those that were automated. 

 There are some other negatives in this scenario as well:
  • The tester is not using their thinking skills. They are basically performing scripted testing. (At a slightly higher level)
  • They will have a negative bias towards the developer because they will always think that their automated tests are not that great
  • The tester may get bored
  • The developer will never know that their tests are (in the opinion of the tester) not trusted

How do you build trust?


Like all good things building trust takes time. It is not something that magically appears ones day. It takes work on both sides to build it and it is essential when working in a team.

You can build trust in an automation framework by doing the following:
  • Test the areas that have been automated to see if automated = exploratory
    • You could manually test what has been written in the automated tests. By this I mean test the high level scenario, not test the exact steps that have been automated. Yes this is a duplication of effort (as I have mentioned previously) but over time you will start to have a level of confidence in the automation as well as the people that are writing the automation. If your positive tests reflect what is happening in the automation then you can  start to trust those tests and the people who write them. Then in the future you can stop testing what the automation tests and focus on other areas. It may require some help from the developer to understand what the tests are doing but an advantage of this is that you can view some of the test code and hopefully understand some of it. 
  • Review the tests with the developer
    • Running through these tests with the developer will expose you to the test syntax as well as what the tests are doing. By doing this you can also help the developers by pointing out areas that need changing, which will help the developer grow their testing ‘mindset’ which in turn will give you confidence in the automation. This is where learning to code is useful. In my opinion testers should at least learn to read code, in a perfect world they should be able to code. I don’t see much of a future for testers without coding knowledge and skills.
  • Write some of the tests
    • Now automated frameworks like specflow require a ‘user understandable’ scenario to be written. If you can write some of these it will give you confidence that certain scenarios have been written and are covered. Now yes these should be defined in the acceptance criteria but there are bound to be some that have been missed and this is a great opportunity to cover these off. Now how can you trust that the developer has implemented the underlying code correctly? Well… you can’t. In any workplace there has to be a certain element of trust to start off with. You probably trust that a developer can write the code behind the tests, after all if you didn’t you will probably drive yourself crazy!!!
So there are a few ways you can build trust in automation.

Comments

Post a Comment

Popular posts from this blog

Testing and Mindfulness

How aware are you? Do you live in the here and now or is your mind always somewhere else? This blog post is about Mindfulness. Mindfulness is a simple meditation and is defined as (According to Wikipedia): "The intentional, accepting and non-judgemental focus of one's attention on the emotions, thoughts and sensations occurring in the present moment" Now Mindfulness has become more popular in the west in recent years as it has shown to have benefits for people who are suffering from Depression and Anxiety. It has been around for a while and is often thought to of originated from Buddhism and some people believe it started thousands of years ago. Now modern life is hectic and I’m sure we all have lots of things going on in our lives that keep our Brains busy and trying to focus on one thing at a time can be a challenge. I can't remember the number of times I've been doing something and my mind is somewhere else entirely. Mindfuln...
Post a Comment
Read more

How to deal with pauses and timeouts in specflow

So this blogpost is in response to the weekly Specflow blog posts and challenges that have been written by Gojko Adzic. This weeks challenge was how would you rewrite or rephrase the below scenario: Given a user registers successfully When the account page reloads And the user waits 2 seconds Then the account page displays "Account approved" My initial though was something like this: Given a user registers successfully  When the account page reloads   Then the account page is displayed within a satisfactory time period     And the account page displays "Account Approved" Now the problem with this scenario is what defines a satisfactory time? You could add it as a comment or in a scenario outline but over time the time a user waits could change and if this is updated in the code behind but the scenario outline or comments are not, then what the test does and what is described do not match - this would potentially cause issues in the future. My next ide...
Post a Comment
Read more

Building a test strategy for a new team

Teams, we have all been on them. Some are good and some are bad. Some we never wanted to leave and others we probably couldn't wait to leave. Now most of the time (well in my experience anyway) you tend to get put into a team that already exists. Maybe you are a new hire or maybe you have asked to change to a different product team.  When you do this, more than likely there will already be a testing strategy in place. It may be that you adapt it and change it in any way you see fit to improve the testing. But imagine if everyone on the team was new? How would you decide your testing strategy? This post will go through some useful things you can do to help a new team develop a test strategy. Table of Contents 📈 What is a Test Strategy? 🤔 Where should I start? 🎯 Understand the company and their goals 💪 Play to the teams strengths 👁️‍🗨️ Understand what quality looks like 📏 Understand Scope 🧪 Understand the type of tests you need 📊 Measure your success 🤝 Collaborate 📝 Summar...
Post a Comment
Read more