One of the most common mistakes with automation testing is going for the immediate benefits and forgetting about the bigger picture. As the result, many teams abandon their automation efforts after a sprint or two and go back to manual testing.
Here at MindK, we rely heavily on agile automation testing, as this approach leads to maximum testing ROI on our clients’ projects. This is what we do step-by-step:
- Set your goals;
- Choose a test approach;
- Select the tool;
- Create and run the tests; and
- Maintain the scripts.
Now let’s dive deeper.
Your first step to any successful project is careful planning. Same works for automation testing.
The most important thing at this stage is to define the scope of testing.
Repetitive, mundane, and time-consuming activities are the most friendly to automate. At the same time, only the most valuable cases justify the investments.
Look for the test cases that:
- Are critical from the business point of view;
- Carry a high risk of failure;
- Have to be executed repeatedly;
- Involve large data sets;
- Test different browsers/environment/devices/OS;
- Can’t be executed manually or take a lot of time;
- Result in a clear Pass/Fail;
- Check stable features; and
- Involve no creativity (i.e. do exactly what they’re told to do).
With these criteria in mind, let’s find out which tests can provide the most value when automated.
- Unit Tests
They have the highest ROI and should ideally form the backbone of your automation testing strategy. Developers can write unit tests to check the smallest pieces of their code and find bugs on their own.
- Regression tests
Finding out that the feature you’ve used for ages stopped working after an update is one of the things that can make anybody go ballistic. Each time new functionality is introduced into your app, you need to check whether it breaks the existing features.
After unit tests, regressions should be your next priority, as they return investments after each successful build.
- Integration/API testing
As modern applications rely heavily on APIs, testing them allows you to validate the app’s behavior and business logic.
It makes sense to automate as much of your API tests as possible. You should prioritize the cases that require regular testing. Identify where the changes to the code can affect the API functionality and target those areas with your automation tests.
Further reading: The definitive guide to API testing
- Cross-browser tests
It’s hard to guarantee your app works correctly on all devices/OS/browsers/screen resolutions with manual testing alone. To increase the ROI from test automation, you can check only the most common combinations your target audience use and focus on high-risk features.
- Data-driven tests
Entering every possible combination of correct/incorrect username and password manually is extremely tedious. But with automation tools, generating this data is quick and simple.
You can also use automated testing to generate massive amounts of quality data for manual tests.
Although pretty common, automating at the GUI level is highly impractical.
These tests are expensive, require heavy maintenance, and return the highest number of false positives/negatives.
Automation testing pyramid
And then there are some tests that should never be automated.
- Tests that will be executed a single time (except for the data-driven and regression tests),
- UX tests,
- Tests without clear Pass/Fail results,
- Tests based on visual perception,
- Testing raw and unstable functionality,
- Ad hoc and exploratory testing where you check the system based on your knowledge of its internal workings and experience,
- Testing the anti-automation features (e.g. CAPTCHA), and
- Testing native features on Android/iOS devices as their security measures hinder automation.
As business priorities change over time and stable features tend to have fewer bugs, you should periodically reconsider which cases should remain on your automation list.
Testing approach here represents a set of rules and guidelines to follow when writing and running the tests (coding standards, practices, processes, reporting, etc).
There are five main approaches to automation that determine how the tests are carried out:
- Linear. The most basic framework. Testers perform each step manually and record them with an automation tool. They can later replay these steps as many times as needed.
Pros: simple, fast, and requires no planning.
Cons: difficult to reuse and maintain, causes a lot of errors.
- Module-Based. Testers divide a system into a number of independent modules and write scripts to test each one separately.
Pros: modularity (duh), low maintenance, scalability.
Cons: test data is embedded into the scripts meaning you’ll have to rewrite tests to run them with another dataset.
- Test Library Architecture. After recording their actions in the app, testers group the common steps (e.g.logon) into functions. They can later use a library of common functions to build modular tests.
Pros: modularity, low maintenance, high reusability, low costs.
Cons: Require greater skill, time-consuming, test data is embedded into the scripts.
- Data-Driven Testing. In this approach, data is separated from automated test scripts and stored in external files. This allows you to run tests with different data sets without having to rewrite the scripts.
Pros: data and test scripts are independent, easy to reuse, flexible, low-maintenance.
Cons: requires good coding skills, takes more time.
- Hybrid Approach combine some of the approaches above in an attempt to compensate for weaknesses. As the time goes by, most organizations tend to adopt a more hybrid approach to fit their unique needs.
When choosing a test automation framework, you should consider the technologies used in your project, the associated costs, the available talent, and your requirements.
Now it’s time to select a tool that suits the chosen approach and meets your requirements.
First, think about your tech stack.
The tool must support the programming languages used on the project as well as the operating systems your testers are comfortable with (e.g. Windows, Linux, Mac, etc.).
For mobile apps, you’ll also have to consider the supported platforms.
If you target two platforms, you can either choose a tool that supports both iOS and Android (e.g. Selenium, Appium) or use separate testing suites (e.g. Robotium for Android, UI Automation for iOS).
After this, check if the tool has the right balance between the ease of use and advanced features (depending on your requirements and the team skill).
Next, check whether the tool provides comprehensive reports to help QA engineers trace the point of failure and share the results with stakeholders.
Now, consider your budget.
There is a great number of tools you can choose from, both open-source and commercial. The latter often provide better support and teaching resources.
If you’re ready to spend money on a commercial tool, you can start a free trial and experiment on a small project before buying the full package.
There are two types of licenses associated with the commercial tools. Development licenses limit the number of machines on which you can write the scripts. Runtime licenses do the same for executing the scripts.
With a commercial tool, you’ll need to buy both to automate your tests. With an open-source tool, you get both for free.
The last things to consider are customer support, the availability of training resources, and whether the tool integrates with your Continuous Integration (CI) suite and project management software.
From my own experience, I’d recommend building a custom test framework based on a free open-source tool (e.g. Selenium).
For instance, Selenium is only a library for working with DOM elements via browser API; alone, it’s not enough to run automated tests. So, you’ll need a testing framework built on top of it (e.g. TestNG, Selenoid, Protractor, etc.). Your choice here depends mainly on the language used in your project.
Creating a custom testing framework for you project gives you a full control of your testing process. The result is easier maintenance, better scalability, and a large open-source community for support. Oh, and it doesn’t cost you an arm and a leg!
Once you’ve developed your automation strategy and picked the right tools, you can start writing the scripts.
For this purpose, you should have a dedicated automation specialist or a team of QA engineers.
Making test automation a part-time job of your manual testers would lead to inefficiency and lost value.
If you lack the required talent, you can always look for experienced automation specialists to augment your team.
There’s a few industry best practices we follow at MindK when writing and executing TA scripts. I highly recommend you follow them to increase automation testing ROI.
- Adopt Behavior-Driven Development (BDD)
This approach to software development helps engineers and stakeholders be on the same page. While designing an app, the team writes requirements in the form of user stories:
I as a [role] want [feature] so that [reason].
This allows the whole development team to see the product through user eyes and better understand how to build and test it.
BDD makes your application more testable by breaking the system into independent features. A user story itself provides an acceptance test that shows whether the feature fulfills its intended purpose.
If you’re serious about BDD, I’d advise you to use Cucumber as your testing tool.
- Use page object pattern in GUI tests
The primary goal of automation is to save time. Updating the scripts and resolving conflicts shouldn’t take longer than the time saved on automating your tests.
With page object pattern, you separate the code that interacts with UI from the test logic and wrap it in PageObjects (app-specific API). This results in less time needed to adapt tests to the UI changes, simplifying the maintenance.
Page object pattern complements Behavior-driven Development. It decreases code duplication and increases test readability.
- Run your tests in parallel
Automated GUI tests are much slower than Unit/API tests. That’s why it’s often necessary to run your scripts in parallel to increase the testing speed.
Selenium Grid, for example, allows you to distribute tests across several machines with different combinations of browsers and operating systems.
You can integrate Selenium Grid with Jenkins and other CI development tools. The tests can be triggered automatically and their results stored together with the relevant tasks.
- Data-drive your tests
As mentioned before, data-driven approach allows you to reuse automated test scripts. You can generate new cases by changing the data stored in external files.
To get the most out of this approach, you can parametrize both the input data and the expected output data.
By following this advice you can write efficient and maintainable scripts.
Without proper maintenance, automation scripts tend to break when some part of your app changes.
That’s why automation test planning is so important.
You can save a lot of money in the long run if you select the test cases that would stay relevant in the future and insulate your scripts against possible changes.
You can make maintenance easier by creating small test cases.
For example, to check a confirm your order button, manual testers log into the app, navigate to the shopping cart, fill all the necessary data and only then press the button.
Such huge cases won’t work for automated tests. It’s better to break them into smaller cases, each focused on one objective. So pressing the button would be separated from navigation and so on.
This way, tiny changes to the system won’t break the whole chain of actions.
Later, you can combine these small tests into complex workflows.
You should also keep the right balance between Unit/API/GUI tests.
Remember, UI-dependant tests are extremely sensitive to the changes in your app. One of the ways to make them less brittle is to use page object pattern I mentioned earlier. It would make your automation scripts easier to maintain.
Unfortunately, many companies realize they have trouble with maintenance when it’s already too late. They face a choice: continue losing money on inefficient scripts or rethink their approach to automation.
In such cases, it often makes sense to involve experienced automation engineers from outside of your organization. Software development companies like MindK provide QA analysis as a consultancy service. So if you’re facing any problems with test automation, our team can analyze your current approach and suggest the best course of action. When you’re ready to talk, just drop us a line on our contact form.