When it comes to quality assurance, there’s no shortage of key performance indicators (KPIs) that you could be monitoring and keeping track of. Some of those KPIs are very specific and tactical in nature, and therefore well suited to helping testing teams manage their overall quality and efficiency. Others, by contrast, are much higher level and therefore of interest to executives making strategic decisions about your company’s bottom line.
While any metric or KPI that can make the quality assurance process more effective is important, the fact is that people tend to have tunnel vision and typically only pay attention to the handful of metrics that they care about most. Practically speaking, that means that regardless of what quality assurance metrics you’re most interested in, you have to be able to track and report on them and the ones that others care about, as quickly and easily as possible.
In this article, we’ll take a look at the range of different quality assurance metrics that you need to be aware of, as well as how you can set up a dashboard that contains the metrics that people in other roles care about, too. Doing so will save you time and energy, because it will allow you to avoid the hassle of manually building custom reports to meet different people’s expectations.
Different Stakeholders Care About Different Metrics
Everyone involved in the software development lifecycle has a different role to play. Software developers, for example, are busy writing code and trying to ensure that it not only does what it’s supposed to, but that it’s also defect free. Meanwhile, teams of automation engineers and manual testers want to ensure that they’re testing that code as efficiently as possible and that the resulting app meets expectations.
As you work your way up the quality assurance ladder, you’ll find people in different roles who are focused on different KPIs. Some of those KPIs include things like whether:
- They’ve written all of the necessary unit tests
- The code is passing or failing all of your company’s automated tests
- There’s traceability between requirements and executed tests
- The appropriate risk factors are being taken into consideration
- The team is executing load tests and, if so, what they cover
- There are enough testers to cover all of your products
Knowing where you stand with these and other KPIs is critical. Not only will that information give you a sense of how you’re doing, having it all in one place will also serve as a useful checklist of all of quality assurance steps you need to take before your next release.
As important as metrics like these are, over in the C-Suite, there’s little time or appetite for this level of detail. Instead, executives are typically focused on three main areas:
1. Speed to market.
Not surprisingly, executives want to know how quickly they can bring new features to market since speed can often correlates with sales. Factors that can affect speed to market include:
- How long your regression suite takes to run
- Whether quality assurance is a bottleneck for the dev team
- The percentage of your tests that are automated
2. The number of bugs in production.
Although there will always be bugs in production, executives want to know how many there are, how severe those bugs are, if a customer reported them, and why there were missed in the first place. This is a key metric because when bugs are found in production it can lower customer satisfaction and ultimately impact usage and even sales.
3. The amount of money being spent on testing.
Last, but certainly not least, executives are concerned with how the cost of testing is impacting your company’s bottom line. Specifically, they’re going to want to know if the team is overstaffed (and therefore costing too much money) or understaffed (and potentially putting the business at risk for a security perspective as a result).
Keeping track of all of these KPIs and being able to report on them quickly and efficiently is a painful process when done manually. Fortunately, it doesn’t have to be that way if you use the right reporting engine or business intelligence tool.
What to Look for in a Reporting Engine
It’s important to find a tool that allows you to measure all of the KPIs your company cares about so that it’s not a manual process. When looking for a tool, you’ll want to find one that’s customizable since no two organizations are the same and how they value metrics will vary. That way you can focus on the metrics that matter most to your organization and quickly create reports that present that information in a visual and easy-to-understand way.
In particular, those reports should give you a clear indication of how far along your team is in everything that needs to be done for your next release or sprint and what percentage of tests you have failed. You’ll also want to make sure that the tool you select automatically pulls in data from your automated tests and that there’s seamless integration with your development and testing tool, so that creating reports from both is a breeze.
Perhaps most important of all, you’re going to want to find a tool that’s intuitive and user friendly so that everyone feels comfortable jumping into it to enter information and access the data and reports that they need. That’s going to be critical for adoption and will help foster a culture within your organization where everyone cares and feels committed to the work you’re doing.
The Way Forward
Make sure that you find a tool that allows you to measure the KPIs that matter most to your organization. Doing so will save you time and energy in the long run because it will give you instant visibility into what the entire team is doing.
If your testers are used to tracking information in Excel and spending hours every month manually creating reports, using a reporting engine like qTest Insights is going to be a huge win. And for the executives demanding those reports, having the ability to quickly go in and find what they need themselves will give them greater insight and autonomy.
We just need a little info from you. We'll only ask you this once!