What’s the first thing that comes to mind when you think about test automation? It’s most likely UI testing, but the truth is there’s so much more that you can — and should — be doing when it comes to test automation.
The Backstory on Test Automation
Since test automation became possible to do at scale, it’s taken hold rapidly. In fact, a recent KPMG survey did not find a single team that wasn’t using test automation. Quite the contrary, it found that nearly 70 percent of teams use test automation for at least a quarter of their tests. We can attribute this widespread adoption to the numerous benefits that test automation offers.
For example, Capgemini’s 2017-18 World Quality Report finds that test automation can help better detect defects, reduce test cycle time, reduce test costs, improve test coverage and improve transparency and control over test activities. Equally as important at a time when DevOps has hit 74 percent adoption, test automation makes key tactics like continuous testing, integration and delivery possible by increasing the efficiency of running certain tests.
Expanding How We Think About Test Automation
Despite all the benefits that test automation can deliver, many testers have yet to tap into the full extent of these benefits simply because they fail to go beyond UI test automation. As Capgemini puts it: “Automation is currently under exploited in QA and testing.”Most testers focus on UI test automation because the UI is what’s immediately visible to both testers and end users. But it’s important to remember that there’s so much more that goes on in any piece of software that you can’t see but that significantly impacts what it is you can see.
And Cognizant finds several limitations to focusing solely on UI test automation, including a high dependency on UI readiness to conduct testing (which often leads to limited reuse of test cases and can defeat the purpose of automation in the first place), a lack of focus on the business layer (as some business logic may require testing separate from the UI), and an inability to meet fast turnarounds (due to constantly needing to build new automated test cases to reflect UI changes).
Expanding how you think about test automation can help resolve these challenges. But looking beyond UI test automation, what exactly are the different types of test automation that you can run and when should you use them? Enter the test pyramid, an easy way to view the different types of test automation and understand how they relate to one another.
You’ll note that the pyramid outlines three types of tests: Low-level unit tests at the bottom, followed by service (aka integration) tests (including component and API tests), and then high-level UI tests to top it off.
Unit tests sit at the bottom of the test automation pyramid because they are the most stable of the three tests since they test the core of the software. Specifically, unit tests review the smallest testable parts of the software (aka units) to make sure they work properly. As a result, there’s nothing that can create a “domino effect” to make these tests fail, as can happen with UI tests.
These are the fastest and least expensive tests to write, which means they also provide faster feedback. Given the stable nature of unit tests and their ability to provide feedback quickly, you should execute these tests early on in order to help prevent bugs. DevOps Zone recommends that unit tests make up 70 percent of your automated tests.
Common tools for unit test automation include NUNit, xUnit.net, JUnit, TestNG, PHPUnit and RSpec.
Next we have integration tests, including component and API tests, which review the interactions between new code and other pieces of the software. Essentially, integration tests put all of the different pieces together to confirm that they work the way they should when they interact. Individual components might work fine on their own, but when put with another piece, something can go wrong. That’s where integration tests come in. DevOps Zone recommends that integration tests make up 20 percent of your automated tests.
Common tools for integration test automation include CA Technologies Application Test, IBM Rational Test Workbench, Parasoft SOAtest, SmartBear Ready! API, Crosscheck Networks SOAPSonar, Apache JMeter, SoapUI, REST Assured and WebInject.
Finally, we have UI tests, which sit at the top of the pyramid because they are the most brittle type of test, meaning that one small change can cause the tests to fail. Quite simply, UI tests review that the software user interface looks and functions properly, but this means that a change to the underlying software or even a misplaced button can lead to a failed test.
UI tests are the slowest and most expensive kind of test automation, especially when testing mobile applications, since mobile testing requires deploying the app to a mobile emulator, simulator or real testing device, all of which are expensive to run and maintain. DevOps Zone recommends that UI tests make up just 10 percent of your automated tests.
Common tools for UI test automation include Katalon Studio, Selenium, HP Unified Functional Testing, Worksoft Certify, TestPlant eggPlant Functional, and Robot Framework.
It’s Time to Think Big Picture About Test Automation
It’s easy to fall into the trap of keeping test automation contained to UI tests, but that would be a mistake. As Cognizant points out, there are several limitations to relying too heavily on UI test automation. Beyond solving these limitations, expanding test automation to unit testing and integration testing also brings several additional benefits around stability and understanding how different components of your software interact with one another. And a time when the software development lifecycle is moving at such a rapid pace and expectations around quality are higher than ever, that confirmation is something every team needs. To maintain control over a variety of automated testing techniques and tools, QASymphony customers rely on qTest Launch, a new tool for managing test automation machines, environments and schedules. qTest Launch offers QA teams a single interface for scheduling, running and reporting on automated tests across frameworks and tools.
We just need a little info from you.