QASymphony / Blog / Striking a Balance between Automation and Human Touch: Q&A with Matthew Heusser
Striking a Balance between Automation and Human Touch: Q&A with Matthew Heusser
QASymphony: Does Agile and other modern methods of application development require new approaches to testing?
MH: We are seeing that companies like Facebook, Twitter and Google are developing techniques to reduce risk in other ways, like rolling a feature out to a smaller number of users, monitoring and rolling back quickly if problems develop. Testing itself, I see that changing much less. One mistake that people make is to try new techniques that may not have the right context at their company. I belong to a group called Context Driven Testing. We say that in testing, there are no best practices; practices can be better or worse in a given context. For example, if you have an embedded medical device like a pacemaker, you can’t update it whenever you wish, whereas with a website you can. Take an application or device that is controlling a tank. Wireless updates for a military vehicle could be a security risk, not a feature!
Software development and quality management; those have really changed. In the year 2000, I was writing software that shipped on a physical disk! If there was a bug, it would be on the front page of the newspaper and would remain there until we got a new disk out. That could easily take another week. To avoid that situation, we would have to test until we were exhausted. Now, we can push a rollback button to revert to an earlier version and get rid of that bug. When Gmail came out, for example it was in beta. Sometimes it just wouldn’t work. So you’d go to lunch, come back an hour later and it would probably be working again. From a skills perspective, if you hire someone who used to work on physical disks, he might have a risk adverse mentality. On the other hand, hiring the younger Gmail person could be too aggressive for a certain company. That’s another example of the importance of context.
QASymphony: You recently wrote about the need to more or less completely automate testing. Are there any dangers to using tools for everything?
MH: I was talking about release candidate testing, which is sometimes called regression testing. It means that we have a build that is ready but it needs some final checking. The idea is that we need to move away from release candidate testing, because it encourages big batches of changes to be rolled out at once for “efficiency”, which slows down delivery. You just can’t move to a model of releasing software quickly under those constraints. With automation, I am talking about complementing human touch with continuous monitoring and quick rollbacks, not replacing human intervention altogether. Let the computer handle the boring repetitive tasks, so that I can do other things, including, yes, testing-as-a-human. There are tools now that can take care of tasks in eight hours which used to take weeks. Note: I’m not talking about clicking-and-comparing here as much as build, install, and virtual server setup. That gives time back to investigate issues we couldn’t before: the impact of a new browser on an application, software usage patterns, testing coverage on mobile, new security risks. These are just a few ideas of how people working in quality management can better spend their time.
QASymphony: What are the challenges with continuous integration and continuous delivery as relates to testing?
MH: Some people want to deploy often; twenty times a day, with small changes. Yay. Bully for you. But what happens if there’s a bug? How fast can we roll back and how will we handle rechecking the right things? Even if you can roll back, if there are problems with every rollout, you’ll slowly find yourself spending more and more time managing churn. That’s no good. With continuous delivery, you can deploy with the push of a button. But if you’re not changing the way software is developed you might be deploying lots of bugs in the process. Companies have got to change the way they develop for the goal of less defects.
QASymphony: How exactly do development techniques need to change?
MH: There is behavioral driven development (BDD), specification by example, acceptance test driven development (ATDD), code craftsmanship, design patterns and extreme programming. The first three are all about having a shared understanding of what we build before we start to code. The last three are about skilled development. These aren’t even all new techniques. Extreme programming has been around for about 20 years. But what is challenging is that organizations interpret these new ideas differently, and they might be doing a very shallow version of it. It’s like if you’re right-handed, and somebody asks you to write with your left hand, it would be very hard the first time. The next time around you might go back to using your right hand because it’s much easier — you convince yourself that you’re still reaching the end goal, which is writing. So there is often a disconnect about why it’s important to use a new technique.
Without understanding, we see people who want to automate all the testing and sysadmin work. Yet they aren’t really skilled at the soft skills which are understanding customer goals, figuring out what might go wrong, and building software that is malleable so that it can be released in small chunks. That short of shallow, buzzword-chasing understanding of agile, continuous delivery, DevOps, and so on is an embarrassment to our industry. We’ve got a lot of work to do. And we’re in the fight to do it.
Matthew Heusser is Managing Principal at Excelon Development, an IT staffing training and consulting firm. Matt has deep experience in software testing, project management, development, writing, and systems improvement.