What color is your testing?

Software system testing can be as complex as the problems the software is trying to solve. Testing techniques are often categorized as white-box, black-box, and grey-box, which sounds like a simple way to describe them, until you consider how many shades of grey exist.

The “box” names are analogies for the level of transparency into the code that the tester has. In black-box testing the code is “hidden” in a black box that the tester cannot see into. In white-box testing, the code is “in” a transparent box, given the tester full access to the underlying source code. In grey-box testing, the tester has some knowledge of how the software works. The tester does not have access to the source code, but understands enough about the system to design tests that get at potential vulnerabilities or test specific functions.

Black-box testing

Black-box testing focuses on functionality. Testers care about the software requirements and specifications, but not about the internal code structure. The goal is to figure out how the software will work from the user’s standpoint.

White-box testing

White-box testing focuses on the inner workings of a system. Testers look at the internal structure and test the application’s source code.  The goal is to reduce or eliminate underlying design flaws and optimize the code.

Grey-box testing

Grey-box testing is often used to test functionality from the perspective of the user, but with some of the insights of a developer. The added knowledge of the system allows test cases to be designed specifically for the software.

Which approach to system testing should you use?

The answer, of course, is “it depends.” The three system testing approaches serve different purposes in producing functional software and they all have their role. Each testing method lends itself to different types of tests. For example, black-box testing is often used to test for performance and usability, while white-box testing is often used for security testing. All three approaches can be used for regression testing and integration testing, but at different levels of the system.

The benefits and drawbacks to each approach depend in part on the project and the team. For instance, because they have visibility into the code, agile development teams cannot use a black-box approach unless they bring in testers who are external to the team. Black-box testing can result in redundant testing, though, because developers may have already run similar or the same tests. White-box testing requires an understanding of the source code, so [on non-agile teams] it generally requires at least some of the developers’ time to define and/or perform the tests, which can be expensive.

The level of potential bias in testing is affected by the type of testing employed.

Black-box testing maintains a separation between developers and testers. Since the testers are approaching the application from the perspective of a user, they are not supposed to know anything about the underlying source code. Therefore, they cannot have worked too closely with the developers.

In contrast, white-box testing is often performed by the developers, because they are the ones who best understand the code behind the system. Test cases are designed to specifically test pieces of the code and figure out how it works, sometimes line by line.

Grey-box testing is a sort of hybrid of the two, with all the variations once can imagine in the space between black and white. Experienced developers often use a grey-box approach to system testing without realizing it. As they work with a system, they absorb information about how it works and use that to refine their test cases.

Color is not quality

While the categories of testing are the subject of much discussion in the software development world, when it is time to do the work, things are rarely black or white. Teams choose the testing methods that work best for the software they are building given the skills of their team and the application under test. Teams focus on delivering quality to the end user, not choosing colors.

Leave a Reply

Your email address will not be published. Required fields are marked *

More Great Content

Get Started with QASymphony