WHY CONTINUOUS TESTING WILL BOOST YOUR PRODUCTIVITY

Imagine that all of your test cases are running as a nightly regression suite giving you a report every morning. Every second week you get reports on stability, robustness and performance of the system. With this information you can make daily checks that you are on track with the development. You will also have an overall understanding on the changes that are made on the system.

You can now investigate in unforeseen deviations in the results, plan for new test cases and meetings with your team. The information gathered on your screen continuously during the project advancement will be your team’s strongest source of feedback to early make agile changes to face unforeseen issues and risks.

Does this sound like a dream scenario? Well it is not anymore. In this article we’ll show you the gains of CD and a very simple way to start with continuous testing.

Three questions

In my experience there are a lot of projects that are not running their test suites continuously. Usually these projects have a heavy burden of large manual test suites and/or flaky automation tests. Running these test suites for a regression test run before a release are often full of investigations into the test cases and the product, questioning if the test case is correct or if the system under test is actually faulty.

This often results in a dangerous change in mindset of the testers within the project. Instead of finding bugs the goal will become to play the checkbox game of completing the test plan before the release date.

Try to answer three questions regarding your project test status:

  1. Are all the test cases still applicable for the software?
  2. Are all of the automation test cases stable and ready for a regression test run?
  3. Do you really have all the test cases needed for reaching the desired quality confidence of your product before release?

In many cases where no continuous testing is made these questions are really hard to answer, I’ve been there and my response to questions like this typically was “give me three weeks and I should be able to give you an estimate”.

How Continuous Testing changes everything

With continuous testing things can be very different. What if you’d be able to reply to the above questions like this:

  1. Are all the test cases still applicable for the software?
    Yes, all of our test cases are still applicable for the current version of the product. We know this thanks to the test results from last night’s test run.
  2. Are all of the automation test cases stable and ready for a regression test run?
    Yes, at least the ones that matter. We might have made a decision to remove some due to low risk or change in features.
  3. Do you really have all the test cases needed for reaching the desired quality confidence of your product before release?
    Maybe we don´t, we could have found some unforeseen risks regarding the performance and might need some new test cases to explore that risk. Some test cases might have failed showing us a possible clustering of issues and we take the decision to plan for more testing in that area.

The value of turning your testing to a continuous activity is huge in this sense. Without it, the farther we get from the last regression run (perhaps last year’s release) the more unknowingly we will be in creating test cases that are of low value. We risk letting bugs through due to test cases that never were created. The next time we are running the tests for a regression suite a lot of time will be wasted on test case issues instead of finding and fixing bugs in this critical time.

A low effort start

Continuous testing will keep us away from the checkbox game “check all test cases and we are done” and move us towards the bug hunting game!

So how do you get closer to continuous testing? A low effort start could be to run a suite of your automated test cases every day.

Devote time during the week to:

  1. Make sure that the test cases chosen are stable and valid.
  2. When your test suite is under control, add more test cases.
  3. After a while you might want to automate the test execution with some scheduling tool like jenkins and even automate the deployment of software build towards your system under test.

Sooner than you think you are up and running! Now you can leave work with curiosity about the morning reports waiting for you next day.

If you are interested in learning more about how to work with CD, look into this article “CONTINUOUS DELIVERY AS A SMALL SCALE EXPERIMENT