Let’s dive right in.
While manual testing is a slow meticulous process of running tests manually, one step at a time; automated testing allows writing test scripts once and run them automatically every time developers make changes to the application’s code. A test automation tool collects the results and presents them in an easy to read format.
This allows your development team to continuously test software for defects and fix them faster.
Automated testing is crucial for Agile Development with its focus on fast delivery.
According to a 2016 study, the use of automated tests has increased by 85% in a two-year period.
At the same time, many companies have trouble implementing even basic test automation. As many as 41% of IT teams have confessed to lacking proper auto testing methods.
“A lot of automation is happening, especially in the traditional regression testing space, but enterprise-level usage of automation remains a huge challenge” — Mark Buenen, the Lead Author of the World Quality Report.
The challenge comes with writing automation scripts that scale together with your application in a way that maximizes the Return on Investment (ROI).
This requires skilled engineers and careful planning.
Tests can be automated on several different levels. One of the most elegant ways to illustrate their value is by an Automated Testing Pyramid.
- GUI tests
At the top of the pyramid are Graphical User Interface (GUI) tests. Most companies perform these to ensure that their interface works correctly. Automating GUI tests brings the least ROI as they take the most time to create and their maintenance requires more effort.
Automating GUI tests only make sense after you’ve ensured that all the other components of your application work properly.
GUI tests also result in most false positives/negatives of all the automated tests. So you might need to run some additional exploratory tests to confirm their results.
- API tests
APIs represent the middle layer of your application. They take care of business logic and connect your UI to a database.
The best part?
API testing is becoming increasingly popular as a way to save money on GUI tests. API tests are also easier to scale and maintain.
According to Smartbear, a number of QA engineers who automate more than half of their API tests is predicted to increase by 30% (from 59% to 77%) by 2020.
You can test API much earlier in development, even before the UI is ready. Once you’ve completed the unit tests, you should automate as much of API tests as possible.
- Unit tests
Unit tests lay at the foundation of the pyramid. Developers write them to test the smallest components of their applications called units.
These tests are the cheapest and fastest to create and the easiest to maintain. They have the highest ROI and return your investments each time developers compile their code.
Test-driven development (TDD) encourages developers to create unit tests before they start writing the code. Once the code is ready, unit tests automatically check it for defects.
This allows developers to prevent bugs instead of fixing them.
Unit tests should form the basis of your automation strategy as they provide the highest ROI. However, they don’t prevent bugs in logic, so it’s recommended to use them in combination with API tests.
When implemented properly, automated testing offers a tremendous ROI for product development.
There are 7 benefits I’ve noticed myself:
- Decreased costs;
- Increased test coverage;
- Improved test accuracy;
- Improved tester morale;
- Faster delivery;
- More hard-to-find bugs detected;
- Easy reporting.
Let’s take a closer look at each of them.
Automated testing doesn’t eliminate the need for manual testing. It won’t allow you to cut the number of QA engineers as they’d have to write and maintain the test scripts.
So where are these supposed savings coming from?
- Finding bugs early
A recent study suggests that bugs in software have caused $1.7 trillion in lost assets in 2017 (that’s almost $230 per person living today).
While each bug can cost your business paying users, fixing a bug also costs money.
Here’s the thing:
The earlier you discover a defect, the easier (and cheaper) you can fix it.
And this is exactly what automated unit testing can give your development team.
As soon as developers compile code, a test suite automatically checks it for bugs. This creates a short feedback loop allowing them to find and fix defects faster.
Let’s bring some data.
National Institute of Standards and Technology has estimated that it takes almost 4 times more time to fix a bug discovered during Beta Testing than on the Unit Testing stage.
Our own experience supports this state.
Automated testing allows you to catch the majority of bugs long before the application goes into production, at the Unit/API layers.
This can save you as much as 75% on fixing your bugs!
But a cost of fixing a bug during beta testing pales in comparison to the havoc it can wreak post-release.
Not only it can lead to downtime and huge financial losses but the fallout can often destroy a company’s reputation.
And there’s evidence to back this up.
In 2012 a glitch in trading software has cost Knight Capital Group $460 million. Over the next day, the company’s stocks dropped by 75% leading to its eventual acquisition by the competing firm Getco LLC.
- Reducing the average cost of testing hours
A QA engineer can only test your system for 8 hours straight.
(That’s if you don’t employ slaves, of course)
An test automation specialist can spend his/her business hours writing scripts and leave them running for the rest of the day.
What’s more, computers can run as much as 5 TIMES MORE TESTS per hour than humans.
That’s effectively 15x more code tested a day!
Assuming the average rates for QA engineers at $30/hour, 8 hours of manual testing would cost you $240.
An experienced QA automation engineer costing $50/hour could do what amounts to 120 hours of manual testing a day for $400.
This lowers the average cost of a testing hour from $30 to $3.33.
Automated testing allows you to dramatically increase test coverage without spending more money.
Which leads us to benefit #2…
As your product evolves, new features are added on top of the existing functionality. With each new release, you have to make sure there are no bugs in the added features.
You also have to ensure these new changes don’t break the existing functionality (the so-called regression testing).
And as your product grows, so does the amount of code you have to cover with regression tests.
Hiring more testers as time goes by is just too expensive.
So, you’re left with two options:
- You can leave some parts of your app untested and take all the risks associated with releasing a poor product.
- Or you could automate your tests and maintain the high-quality standard.
In my experience, option 2 is MUCH better.
Only automated testing can guarantee that every part of your application is sufficiently tested as it grows with each new release.
Without it, increasing test coverage requires great effort.
You might have to create hundreds of extra test cases covering every possible scenario and workflow as well as make sure the application works correctly in different browsers/devices.
With automated tests, you can launch multiple scripts and have them run during off-work hours. This way, numerous systems and configurations can be tested in parallel.
Finally, QA engineers can use the time freed up by automation for in-depth manual testing and cover more scenarios by generating test data with automated test suites.
Running the same tests over and over can become mind-numbing and lead to errors. Automated scripts can go through the same steps hundreds of times without getting bored or distracted.
This increases the accuracy of your tests.
Freed from repetitive work, QA engineers could tackle more challenging and rewarding tasks. Not only does this lower the number of errors and allows you to test your application more deeply, it also increases work satisfaction.
And satisfied testers are more motivated to stay in your company and achieve great results.
As your project grows, so does the amount of things you need to test.
With a standard two-week sprint, manual regression and debugging can take up to 5 days. With automated tests, you can complete the task in just 2 days significantly decreasing time-to-market.
What’s more, without automation, your test cycles might eventually become longer than development cycles. This goes against the Agile philosophy where you want to finish everything at the same time.
If your QA engineers and developers work on different sprints, the team communication will suffer. This can lead to additional expenses or even delay the release.
And releasing your product later might mean losing precious market opportunities.
This can be deadly for any company, especially startups.
In the time you’ve spent on manual tests, your competitors might release their own versions of the product making it much harder to carve out your niche in the market.
In this case, it often makes sense to outsource automated testing for the previous sprints. This way, your own testers could work on the same sprint as developers and release on schedule.
QA automation allows you to detect bugs that often slip by manual testers.
Some errors like memory leaks accumulate over time. When you execute the same code hundreds of times, an application sometimes fails to release the allocated memory. Such a scenario is unlikely to happen during manual testing and memory leaks could go unnoticed for a long time increasing the costs to fix them.
You can run automated tests hundreds of times in a short time frame to find these issues earlier.
The increased testing speed lets you check more conditions and edge cases. Automating your tests lets you easily repeat these checks in the next development cycles.
This can help you with regression testing.
By automating these tests, you can be sure that regression checks are performed after each build. A continuous integration tool will notify developers as soon as the test fails without having to wait for a report from QA engineers.
As a result, you discover regressions more often and fix them faster.
What’s more, automating regression tests gives developers the confidence to innovate without fearing that changes to the code could break something in your system.
Automated testing tools take reporting to a whole new level.
These tools provide comprehensive reports containing the results from all the tests conducted in a single run. This way, any member of your team can quickly identify the root cause of a bug without contacting the developer of the automated test.
This gives you a real-time picture of application’s stability.
The information and metrics presented in reports can provide valuable insights into the system’s performance.
Now it’s your turn
I hope this article will help to power up your Quality Assurance game.
Automated testing value is measured in hours saved on repetitive manual tasks, in bugs found before they can damage your reputation, in the money you save by fixing defects as early as possible.
Test automation ROI can be high both for the early-stage projects and for the software in production; for startups and enterprises.
The key is knowing how to implement automation so that it brings maximum value and having the right talent for the task.
But that is a matter for another article.