Test Management Research 2017 – Top Challenges, Recommended Practices & Technology

test management - research masthead

Insight from test management experts on top challenges, recommended practices and technology to bring it all together

In the world of software testing, running effective tests will only get you so far.

While running the right tests and running those tests correctly is arguably the most critical component of the software testing process, that effectiveness means nothing if testers can’t properly manage it all. As a result, test management is of utmost importance to successful software testing.

That said, a lot has changed in recent years when it comes to test management. So what do you need to know to keep up with the latest trends in test management?

Read on for our experts’ advice on:

  1. How Agile has impacted test management
  2. How to overcome common challenges
  3. Why it’s important to be wary of “best practices”
  4. Top tips for selecting test management solutions.

The State of Test Management

While test management has always been important to keeping software testing teams on track and helping them deliver the best possible results, it’s become even more so over the past few years.

The importance of test management has increased due to heightened standards around software quality and the need to get software in the hands of customers faster.

Additionally, the transition of many teams from Waterfall to Agile methodologies has made an impact, since Agile has blurred the lines between development and testing. On the one hand, the fact that quality is now a concern for everyone is a good thing for testers, but on the other hand it adds complexity to an already complex process.

Consider the following stats from Parasoft:

  • Testing modern software requires access to 30 (or so) dependent apps
  • 30% of scheduled testing time is spent on setup and configuration tasks
  • About 50% of testing teams still use spreadsheets to manage setup and configuration
  • 50% of software ends up going without testing

The bottom line? Testers now have to manage more complex environments and complete testing faster than ever before, making proper test management an absolute must.

How Agile Impacts Test Management

Among the many ways that Agile has impacted test management, two of the most notable are people-focused:

  • Where test manager roles fit into Agile teams
  • How testers communicate with developers and product owners in Agile environments

The Role of Test Managers on Agile Teams

First, let’s look at the role of test managers on Agile teams. There’s no test management role defined in any of the Agile books or methodologies, so does that mean that test managers don’t have a place on Agile teams? No.

 Shaun Bradshaw, founder and principal of software delivery solutions firm Zenergy Technologies, shares: “In the Agile world, managers sometimes get a little lost and they’re not sure where they play anymore. So the biggest issue I see in managing Agile from a test perspective is, as a test manager, what do I do if I’m not on the team, if I’m not contributing directly to that work?”

Bradshaw recommends that test managers in an Agile environment play a very large role at the release level planning so that they can set a general testing strategy across multiple sprints. He continues that while individual teams might do retrospectives, test managers should do test team retrospectives on a regular basis to identify what’s working well for one individual or team and to determine how that approach can be replicated across the organization.

“It’s really about looking at the craft of testing across the organization and acting as a facilitator to find standard practices to be used across the different teams and to provide guidance,” Bradshaw shares.

So while Bradshaw does admit that the role of a test manager is significantly different in Agile than it is in Waterfall, he remains firm that there is still very much a need for test managers.

test management - role of test manager cta

How Agile Impacts the Relationship Between Testers, Developers and Product Owners

Second, let’s explore how Agile impacts the relationship between testers, developers and product owners.

According to Paul Gerrard, founder and principal of Gerrard Consulting, a digital assurance consulting firm, to successfully handle test management in an Agile environment, testers need to have a relationship with developers and product owners.

Specifically, he says: “In Agile, the tester is almost an advisor and a facilitator and a bit of a coach. Testers need to help product owners get to the bottom of requirements by clarifying them and flushing out ambiguities, and they need to help developers determine which testing needs to take place at which level of code.”

Top Test Management Challenges Today

Nothing is without its challenges, and test management is no exception, especially as more and more teams make the transition to Agile. So what are the biggest challenges in test management that testing teams face currently? We asked a series of experts who identified four key challenges.

1) Racing Against Time

Bradshaw points out that time is a universal challenge when it comes to test management, although the challenge that time presents differs between Waterfall and Agile environments.

The Challenge for Waterfall Environments:

In a Waterfall environment, testing falls at the end of the development process and testers typically have limited insight into the earlier steps in the process and the design that went into the system. In other words, a “siloed, throw it over the wall mentality” exists. With this setup, when the schedule gets pushed back, testers then have an extremely condensed amount of time in which to execute. And this time constraint is only exacerbated by the limited communication between testers and developers, which creates ambiguity.

The Solution for Waterfall Environments:

Bradshaw says the solution to this challenge is for testers to make a concerted effort to work with project managers, developers and BAs as early as possible, even if it means forcing themselves into meetings about requirements and designs in order to get a clear context and understanding of what is being developed. This can potentially help prevent some bugs from ever coming to fruition, thereby decreasing the number of issues testers will find, which eases the time constraint.

The Challenge for Agile Environments:

On the Agile side, the time constraint still exists, but it’s a self constraint because the team has agreed to iterate through a very short period of time. Ultimately, this time constraint creates the same type of challenge that testers encounter in a Waterfall environment, but how everything gets managed needs to be a bit different.

The Solution for Agile Environments:

With Agile, testers should already be a part of conversations with project managers, developers and BAs, so they already have more of a say in what’s being done and can advise on how features get split up over multiple sprints. As a result, the key in this type of environment is to understand the different testing needs and make recommendations to split those needs up in ways that help ease time constraints. For example, if there are three major features, testers should recommend that those features get split into different sprints and each grouped with smaller features that are less time consuming to test.

Whether you’re doing Waterfall or Agile, Bradshaw warns against falling into the trap of thinking of automation as the cure-all for time constraints. He explains that while automated testing can certainly be faster to execute than manual testing, it doesn’t work in every situation and still takes time to manage.

2) Understanding the Application Under Test

According to John McConda, lead performance engineer at contract research organization Covance, understanding the system well enough to determine what to test and correctly identify issues often proves an initial roadblock for teams to overcome.

The Challenge:

McConda shares:

“I think determining what to test might be the most difficult problem because testing is an infinite activity. Once you have that mapped out and everybody is on board, then I think the real work can happen. But a lot of times that never happens, and people start taking on activities that really don’t contribute to the mission as a result.”

Additionally, even once testing does begin and the team knows where to look for issues, McConda finds that identifying issues can still be a challenge if testers don’t know the application under test well enough.

The Solution:

To overcome this roadblock, McConda recommends the heuristic test strategy model. “I really like the heuristic test strategy model because it helps you think of areas where the product could break and where things could happen. With testing, you’re often doing an activity and you just don’t really know what you’re looking for. It makes sense, because if everyone knew where the bugs were, then you wouldn’t need testing, but that means you’re going in without a map and and it’s kind of like doing investigative work. The heuristic test strategy model helps you make sure you have the proper coverage to find issues that will be problems for the end customer rather than going in blind and doing a CSI-type investigation looking for something that may or may not be there.”

3) Balancing the Budget and Advocating for What You Need

 Jon Hagar, an independent systems software engineer, testing consultant and author, says that working with tight budgets can prove especially challenging for test management leaders.

The Challenge:

“On the budget side, everybody is constantly being pushed to do more for less, so you’re never going to end up with everything you want,” Hagar explains.

The Solution:

Hagar finds that taking an Agile approach can help ease this challenge to an extent. For instance, he cites his own experience working on long term projects where taking an Agile approach made it possible for him to justify budget increases to leadership because he was able to share information that proved the need to conduct certain testing while the project was still underway.

4) Finding the Right Team Mix

Hagar also points to finding the right team dynamic as a critical challenge.

The Challenge:

According to Hagar, building a team with a variety of knowledge, skills and experience can often prove difficult for test management leaders. However, he says that finding the right dynamic between all of these points is of utmost importance.

The Solution:

To overcome this challenge, Hagar recommends paying close attention to not only skills and knowledge, but also experience. He shares: “I like to have a mix of junior people, mid-level people and senior people, and I find that dynamic helps a lot. If you think about it, you get a lot of people that come out of schools or have a degree or maybe a certification and they have knowledge, but there’s a big difference between knowledge and skill and experience, and you need all of those components.”

How to Manage Testing Priorities in an Agile World

Arguably one of the most pressing questions when it comes to test management is: How do you manage testing priorities?

It’s an interesting question since testing is technically an infinite activity. As a result, you need to put boundaries around testing in the form of priorities. These priorities should determine what will be tested and the order in which they will be tested (the second part of which is important in cases where teams bump up against time constraints).

Gerrard explains that he thinks about it like this: “There are an infinite number of things you could try as a tester, so testing is about taking what’s possible to test in the time you have. And so the whole idea of test design and selection is about prioritizing what’s feasible in the time you have and what is practical and useful, and distinguishing that from everything else you could possibly try. So how do you do that? Well, it’s all the methods of test design and modeling, like risk-based testing and so on.”

The Value of Using Risk-Based Testing to Manage Priorities

Across the board, experts recommend taking a risk-based testing approach when it comes to managing testing priorities, as do 26% of respondents in a recent survey conducted by QASymphony. Another 8% of respondents recommend managing testing priorities based on what is most important to the end user experience and 11% recommend managing testing priorities based on business requirements and/or feedback from stakeholders from the business.

As McConda puts it, “risk-based testing is really about figuring out, when this thing gets in the user’s hands, what’s the worst that could happen?” In other words, it’s about prioritizing your requirements and functions based on the risk they pose and then aligning testing around those risks.

Watch the webinar “Using JIRA for Risk Based Testing”

According to Bradshaw, even though the risk-based approach will differ slightly between Waterfall and Agile, it remains an important way to consider prioritization in testing. “Agile throws a slightly different twist to risk-based testing because you’re in smaller time-boxed periods so you have to be more vigilant, but it still works,” he adds.

Specifically, in a Waterfall environment, the risk-based approach requires you to look at the likelihood that something will go wrong, typically based on historical information, and use that to drive your testing. With Agile, you can take a similar approach, but you have to work closely with the product owner to understand how deep they want you to go. Additionally, your priorities might change based on what you actually find since testing is broken up into sprints.

Using Risk-Based Testing as a Starting Point

Meanwhile, McConda says that even though risk-based testing is always where he starts, it’s only a starting point. “I normally approach everything in the context of risk, so everything I test kind of boils down to that. But once you get there, then you start looking at things like coverage and requirements,” he shares.

In general, McConda says that the key is to know when you’re done. “Cem Kaner says testing is an infinite activity, you could do it forever and still not be done. So we have to determine what it looks like to be done with testing, and that really starts with your mission. We try to take the highest risk and the level of coverage we want and get that mapped out early on. It’s also important to communicate that to the stakeholders so that they know what the end goal is and what it means to be done with testing. You want to be clear that once you achieve a certain level of coverage and all of the critical issues that you found are fixed, then the software is ready to be released.”

Hagar echoes the idea that testing is infinite, emphasizing that you simply can’t find every bug. And he says it’s this mentality that led him to risk-based testing.

Like McConda, Hagar uses risk-based testing as a jumping off point. Depending on the situation, Hagar says model-based testing, test-driven development and acceptance-test driven development can all prove useful in managing testing priorities.

He adds: “You have to look at the priorities of both the product line and the team as well as the customer and the users. And those are all different audiences that you have to deal with when you’re doing test plans and test strategies. It takes a lot of work. There isn’t just a simple process (or even a complex process) or one way of doing things. If you have just one way of approaching a problem, it can become very limiting.”

Considering Stakeholder Requirements

While Gerrard agrees that a risk-based testing model can be a good way to manage testing priorities, he approaches the entire process in a different way.

Gerrard says it’s important to start with a stakeholder, and this stakeholder can be the user, the user manager, the product owner, project manager, operations team or developers. Once you identify your stakeholder, you have to understand what they’re looking for, and that’s what becomes most important during testing. “Your stakeholder becomes your customer and it’s surely a case where the customer is always right,” he explains.

However, he says the problem is that these “customers” typically don’t understand what testing is about in the first place, so they don’t know how to make choices about what’s important to them. As a result, the role of the test manager is to explain to that stakeholder what the testing plan will accomplish and what it prioritizes.

When all is said and done, Gerrard says that to him, “managing priorities is about managing the relationship with your stakeholder and finding the appropriate models to have a meaningful conversation about what is useful and what is less useful. Risk is a part of that, but risk is just another mechanism for creating a model.”

The Case Against “Best Practices” for Test Management

Anyone trying to learn more about how to do something usually starts off exploring the state of the industry, top challenges and common approaches. Once you have that under your belt, the next thing people typically search for are best practices to accomplish the task at hand. But our experts warn that in the world of testing, the idea of best practices can be dangerous.

For example, McConda shares: “The term best practice itself is an interesting one that a lot of people in the testing community have been debating.

What does a best practice really mean? What is it best for? It depends on the context. That thinking comes from the context-driven testing community, and one of their biggest points is that there are no best practices — there are only good practices within the given context.

So if you’re in a regulated environment, you’ve got a whole set of practices that are going to be different than they are for an Agile product or for an Android app or for anything else. And what we try to say with that is that we don’t want testers to think there’s only one way to do something, because that one way may not be good for your project.”

Hagar agrees, explaining that best practices must be viewed inside of a larger framework for software testing. “Most places tend to use the term best practice as if it’s an absolute, sort of like a law of physics or nature or something that’s inviolate. And I find that not to be true. I find an awful lot of managers, particularly senior managers, who seem to want one solution and see best practices as something that they can apply all the time, no matter what they’re doing or where they are. And to me, that’s a trap because it tends to oversimplify things,” he says.  

That said, Hagar does believe there is one best practice that actually deserves the label. “As a manager and a tester and now a consultant, I think there is a best practice when it comes to understanding the industry and thinking before you do anything. And what I mean by that is the use of tools or automation or concepts might be good in one place and work with great efficiency, but you have to think and maybe change them a little bit to work in another space. So the best practice for me is thinking and then understanding things like processes and tools and how to change them,” he continues.

According to Hagar, this best practice really comes down to being an expert. “To me, an expert is someone who knows the tool, knows the process, knows the best practice, but then, more importantly, knows how to modify it and change it for the particular situation at hand. And so the best practice is being able to understand the approach at an expert level and look at different situations and go, ‘Gee, we really need to use the tool and approach with this slight change.’”

How to Choose Test Management Software

As testing gets more complex and timelines get crunched, having the right test management software in place can make all the difference when it comes to overcoming challenges, managing testing priorities and adapting best practices to the situation at hand.

Ideally, test management software should accomplish these objectives by helping testing teams:

  • Collaborate closely
  • Work more efficiently
  • Introduce an easy to follow process
  • Reduce documentation requirements
  • Increase transparency
  • Embrace techniques like exploratory testing that help integrate testing into the business

With these goals in mind, you should consider the following points when evaluating test management software:

Vendor Attributes

  • Efficiency: Does the solution make repetitive tasks like test planning and documentation more efficient? Does it integrate with other ALM and development tools? Can it help increase test coverage and insights? The answer to all three should be yes.
  • Flexibility: Does the solution support multiple testing techniques and platforms (e.g. desktop, tablet, mobile)? Does it work for both Waterfall and Agile environments? Given today’s fast pace of change, flexibility to support different needs is a must.
  • Cost: Does the solution provide a true, multi-tenant cloud environment that can grow on demand (ultimately reducing the cost of maintenance and upgrades)? Or does it follow the legacy model of hidden costs around integrations, web hosting access, server maintenance and more (all of which can decrease user adoption and productivity)? If you want to keep costs down and adoption up, make sure your solution offers a true cloud environment.
  • Usability: Does the solution offer the capabilities testers need to get their work done? Can users onboard and feel comfortable working in the new system quickly and easily? Is there enough executive oversight? All of these answers need to be yes in order for the solution to provide everything your testers and your leaders need to do their jobs effectively.
  • Support: What kind of support does the vendor provide — a subscription model (in which the vendor becomes a true partner, ensuring you get value from the system and making resources available to help 24/7/365) or a licensing model (which typically doesn’t provide any incentive for the vendor to provide a high level of support)? If solution support is important to you — and it should be! — make sure it’s the former.
  • Scalability: Will the solution be able to grow alongside your organization? Is the vendor committed to continued improvements when it comes to product development? Make sure you have room to grow, both in terms of size of your organization and needs. Looking at the vendor’s current product roadmap as well as how it’s evolved to date can help you get a good picture of its commitment to innovation.

Solution Capabilities

  • Visual Proof of Testing: Does the solution offer a recorder that can clearly capture everything tested without manual effort? This provides unparalleled visibility into testing coverage, including concrete evidence into what testing occurred and when it occurred. In turn, this visibility makes it possible to identify exactly what went wrong without having to recreate the issue.
  • Customized Integrations: How easy is it to configure the solution to support the latest versions of external, third-party defect management tools and automation frameworks? Are these integrations customizable and extensible via API? Integrating systems not only makes for a better user experience for your testers, but it can also help keep information consistent across the board, making easy integrations a must.
  • Uninhibited Workflows: Does the solution fit with your team’s process or do you have to retrofit your process to fit the solution? Make sure the solution’s features empower testers to gain more insight rather than hindering what they need to do.
  • Visibility and Collaboration: Does the solution offer seamless, drill-down visibility into all testing in a centralized place? Does it make it easy to collaborate on and track testing progress and updates? This kind of visibility and collaboration is extremely important in today’s increasingly complex testing environment.
  • Robust Analytics: Does the solution offer actionable insights into issue areas within the testing process? Do the reporting features help measure and predict software quality? Do they provide team leaders and executives with a high level overview of application health and team effectiveness? Reporting is critical to understanding how your team is doing and identifying areas for improvement, which makes robust analytics a feature that can’t be overlooked.

Be Confident in Your Decision

Once you narrow down your options for a test management solution, engaging in a trial can often help cement your decision. To make the most of your trial, you should:

  • Integrate QA and development in order to:
    • Provide cross-team clarity on what business needs the system will solve
    • Encourage the development and testing teams to collaborate on defining the testing approach moving forward
    • Facilitate conversations about and increase visibility into requirements, plans, priorities and metrics
  • Avoid shortcuts by:
    • Investing as much time as possible in getting real data
    • Taking time up front to “move in” to the system
    • Appropriately connecting the trial to the production instances of your defect trackers to ensure compatibility with the customizations in production
    • Working closely with the software provider to understand the level of service and support you will receive if you become a customer

The Key to Test Management Success

There’s no denying that the world of software testing is becoming more complex and faster paced, and these changes have created a perfect storm.

In the face of this storm, testing teams need to double down on test management in order to keep organized and ensure software gets the testing and coverage it needs in the allotted time.

And while there may be no such thing as a “best practice” for test management, there are several recommended practices that can help teams accomplish this goal. For example, teams need to understand how the move to Agile impacts existing test management practices, be clear about the challenges they face, think carefully about how to prioritize testing needs and ensure they have the right technology in place to help bring all the pieces together.

Only when all of those efforts come together will teams be able to thrive in the midst of today’s rapidly changing testing environment.

Leave a Reply

Your email address will not be published. Required fields are marked *

More Great Content

Get Started with qTest