For testers in the agile world, not only have time frames for testing shortened, but their whole approach to testing has been upended. Gone are the days of crafting tests from detailed project requirements. In agile development, tests are based on the expectation that the product will match the user’s stated need.
Agile development requires working features to be created in relatively short timeframes. Lengthy documentation doesn’t exist. Regardless of the length of a sprint or a release, adequate QA testing must occur before a feature or product can be declared done.
Testing is recommended as soon as working code exists that can be tested. “Test early and test often” has become the adage for QA testing in the agile world. A “test early and test often” best practice is to run regression tests every day, typically after every piece of code has been loaded into the build.
The results of regression tests inform the team whether new code is compatible with the existing code base. If a system passed a test on Tuesday but fails the same test on Wednesday, its good evidence the new code caused a regression—the new code broke previously working functions, introduced errors, or caused bugs the team thought were fixed to reappear.
Regression testing includes executing a growing set of tests, covering existing functionality, every day until the product is complete. This practice helps ensure that features are not built on top of an unstable foundation of code. Every change to the build carries the risk that new code will conflict with existing code. Sometimes a change causes ripple effects that are not immediately apparent. Instead, a bug might only become obvious after other changes are made. In other instances, a bug fix itself might cause new problems. Regular, continuous regression testing helps teams build software that behaves predictably and remains stable.
Daily regression testing aids a team in identifying the specific window of time in which problematic code was created. The team only has to look at what was changed since the last successful regression test to figure out what went wrong. Identifying and fixing bugs early helps ensure that future code is added to a working base of code. In contrast, if a bug is not caught early, all future code that is written may be dependent on the faulty code. When the problem is discovered, everything that was built after it may have to be unraveled. Even when a bug cannot be fixed quickly, identifying it early still carries the benefits of protecting future code from being dependent on problematic code.
Running daily regression tests is not a new concept. Particularly on agile teams, the benefits are obvious and valuable. It is surprising, then, how many teams choose to skip this step. Like all preventive measures, it can be hard to initiate and develop into a solid habit. One barrier to implementation is that old QA testing practices still influence agile testing beliefs and practices. A second barrier is that QA resources are limited and daily regression testing may not be a top contender for scarce resources. Over time, as knowledge of bad outcomes that could have been prevented by daily regression testing accumulates, one would believe early bug detection will become a higher priority.