Strategies for Modernizing Software Testing in Regulated Industries

How agile testing principles align stakeholders to improve software quality and support regulatory compliance

Last year, for the first time ever, the U.S. Food and Drug Administration (FDA) recalled a medical device due to a cybersecurity concern. In August 2017, nearly half a million Abbott pacemakers were recalled to address a vulnerability that could allow hackers to run the batteries down or even alter the device’s pacing commands. Abbott issued a firmware patch, which was applied to patients by medical staff, to address the vulnerabilities. It was the second round of updates required for Abbott heart implants last year.

Just a few months earlier, an Equifax cyber attack put more than 143 million American consumers at risk for identity theft. The attack marked one of the largest risks to personally sensitive information in recent history. According to an investigation by Equifax and its security consultants, hackers exploited a weak point in website software to to retrieve names, birth dates and addresses, as well as credit card numbers for 209,000 consumers.

In addition to putting a large number of people at risk, both of these companies suffered significant blows to their public image and subsequent financial losses.

When the stakes are this high, prioritizing software quality is non-negotiable, and many organizations are modernizing software testing processes to sharpen their focus on quality. Yet a myth persists among development and testing teams in regulated industries that shifting to agile development and testing is risky.

Adam Satterfield, a testing professional at Anthem who is responsible for implementing QA best practices, says that across regulated industries, he has encountered agile bias. “There is a common misconception that agile is a documentation-less process that will not work in companies that need a hefty paper trail or that are going to be audited,” Satterfield says, “but that’s simply not true.”

The key difference is that agile minimizes unnecessary documentation and processes to create a rapid feedback cycle. This enables teams to respond quickly to changing business, customer or regulatory requirements. “If you have a change coming from the government or that needs to be implemented due to a law or regulation change, such as the Affordable Care Act, you want to get that implemented as quickly as possible,” Satterfield says. In addition to promising faster time to market, many core agile practices are specifically designed to improve software quality. And this focus on quality offers distinct advantages to teams developing software in regulated industries.

Most regulated industries do not require developers and testers to adhere to specific methodologies — or even require testing by law. Consider medical software that is classified as a medical device under the FDA’s purview. The FDA does not mandate a specific development or testing type, as long as the organization can produce the required artifacts and can prove the device is safe and effective, meets all design requirements and satisfies user needs.

However, the consequences of a defect in such a heavily regulated environment can be significant. Innovative development organizations in healthcare, banking and other regulated industries are applying agile principles such as behavior-driven development (BDD) and acceptance test-driven development (ATDD) to put quality at the forefront. Here are five ways these methods can help address the unique concerns that testers in regulated industries face.

1. BDD Helps Ensure All Stakeholders Understand Requirements

BDD is an agile process that helps teams more efficiently understand, develop, test and deliver the features that matter most. It involves tight collaboration and communication between business stakeholders, product owners, developers and testers to discover, understand and translate business needs into technical requirements before development begins.

BDD test scripts differ from traditional test scripts in that they are written using a syntax that is similar to the English language. That means it’s easier for business stakeholders to verify that developers and testers understand their needs, and that there is less opportunity for requirements to get lost in translation as they are passed from business stakeholders to product owners to developers to testers. This makes BDD an effective method for ensuring regulatory requirements are clearly understood and accounted for during development and testing.

Satterfield has his teams take BDD one step further and implemented ATDD to streamline new development projects. ATDD builds on BDD by directly tying each feature to a business requirement through testing, so failing tests provide immediate and clear insight into how features are falling short of requirements.

When business needs are understood by the whole team, they can be automated in the form of executable specifications, or tests that will only pass when the software does everything that is expected of it.

2. BDD Crowdsources Domain Expertise

According to Satterfield, shifting to a more collaborative process has enabled testers to come forward as user experts, which has helped bridge the gap between business and technical requirements. There is tremendous value in recognizing testers’ unique insight into user experience. Out of everyone involved in the software development lifecycle, testers arguably spend the most time interacting with the software the way that a user would. And usability becomes extremely important when consumers are interacting with your applications to access health information or perform financial transactions.

“When developers and testers are writing test cases during the sprint, their time is not being spent efficiently,” Satterfield says. “They need to be brought to the table during sprint planning, where they can ask the questions that will inevitably arise. Because it’s the testers day in and day out experiencing the user flows of your product.”

By bringing testers to the table earlier, Satterfield ensures that team gets the test cases right the first time, which not only helps ensure they clearly understand the requirements, but also that they have enough time to write automated scripts and ensure thorough test coverage prior to release.

ScienceSoft suggests that acquiring domain knowledge is one of the most important things testers in regulated industries can do. Testers must know enough about the regulations governing their industries to understand which parts apply to the software they are testing and ensure proper test coverage.

In addition to leveraging testers’ domain expertise, BDD and ATDD crowdsource domain knowledge from stakeholders across the business — including regulatory experts, business leaders, product owners and developers. Because test strategy is discussed in this collaborative environment, BDD makes it easier for regulated businesses to ensure that the right rules are included in the test case design to not only guarantee proper test coverage and regulatory compliance, but also to ensure that the software meets the user’s needs.

3. BDD Increases Traceability

Regulatory clearance often requires demonstrating traceability between requirements, tests and software artifacts. Chris Sears, a Principal Biomedical Engineer at Syncro Medical, says applying the concepts of test-driven development have helped his team maintain traceability throughout the development lifecycle.

“By following TDD, we must write the test early and therefore it must be based on a specific requirement,” Sears wrote in a blog post. “This creates the traceable link between test and requirement. During the implementation step, we only write the code that is needed to pass the test. This creates an additional traceable link between the software artifact and the test. Combining this means that we will have full traceability between our requirements, tests, and software artifacts,” Sears writes.

BDD tools also offer ways to document automated testing, a process which isn’t always the de facto standard in non-regulated industries. Often, automation engineers experiment with writing and running automated scripts, which may make teams in regulated environments shy away from it.

But because BDD ties test cases directly to requirements, the method offers testers in regulated industries an effective way to document test cases. If you are automating tests in BDD, test cases, which are written in plain English, become documentation directly tied to the automated tests that were written for it. This makes it easy for non-technical stakeholders to understand exactly what each script was testing for, and to quickly compare the test cases to both requirements and results.

Tools that integrate with open-source automation frameworks can simplify this process by providing a single view of all testing activity. This offers teams a simple way to report that the software has satisfied each requirement by ensuring test cases are properly documented.

Barriers to Adoption

Modernizing software testing with BDD and ATDD principles can streamline development by aligning stakeholders in advance. These principles also crowdsource domain knowledge to ensure both regulatory compliance and a positive user experience and create a thorough audit trail. So what’s the drawback?

“The only con is that everyone has to be bought in for it to work,” Satterfield says. “You have to re-train testers to write test cases in a different way. And leadership has to understand and make room for a learning curve and provide the appropriate technology to support it. It’s a process change as much as it is a way to write your test cases.”

Continue Reading

We just need a little info from you.

More Great Content