From waterfall to RAD to agile, software development has been moving toward progressively smaller and faster development cycles. Continuous integration and continuous deployment are compressing delivery times even further, which is good for business and good for users—or is it?
In the ’90s, we had automated nightly builds where all code that was checked in would be built and ready for deployment. Then came continuous integration, where a build was done as soon as the code was checked in and automated unit tests could be run on this new build. Now comes continuous deployment, which allows us to painlessly deploy as often as needed.
The ability to deploy to production easily and rapidly is nice to have. We’ve all been in a situation where we need to make an emergency fix due to a major production bug or lived through a painful release process. However, the ability to deploy quickly and frequently, while very appealing to companies, is often not to the users.
I recently heard people from one agile company boast about how they “deploy software to production one hundred to two hundred times a week.” Based on a five-day workweek, this averages twenty to forty deployments per day—or, based on a ten-hour work day, two to four per hour. What could the downside of this be? From a user’s perspective, here is an example.
As a web-based Microsoft Team Foundation Server user, I can recall several times where the look and feel of TFS changed without notice, which threw everyone off for a while. Making matters worse, the “enhancements” were questionable at best, because they usually involved more clicks to perform the same function.
However, the real danger with continuous deployments is that some companies are moving away from having their software tested by professional software testers. If there’s no pain or cost for deploying software, who cares if it’s buggy? "We do continuous deployment, so we’ll just push out more fixes," these software development managers say.
In my experience, testing has never been fully appreciated, and continuous deployment may reduce this appreciation even further. Testers are often the low man on the totem pole, and in tight economic times, they are usually the first ones to be let go—with the assurance that the developers will just have to test more, and that the rest of the testers’ tasks will be replaced with test automation. Automated tests run like magic after each build, and they run quickly, too.
But before considering automated testing as a replacement for professional testers, answer these questions:
- Are your automated tests being checked for false negatives or, worse yet, false positives?
- Are you periodically injecting data that should cause your automated tests to fail and verifying that they are, indeed, failing?
- Is your automation fully exploring the product, seeking to learn and adjust as tests are run?
Remember, not all tests can be automated. Tests based on timing can be difficult to automate, if not impossible.
Some business people and development managers may think, “Okay, our QA person found some bugs that automation did not, but would a ‘real user’ find them? Would they care? Let’s just deploy it and see what happens.” They may see users as unpaid beta testers or crowdsourced testers.
But how many users will take the time to report a bug? How well will their bug reports be written? How easy will the company make it to report bugs, and how responsive will communications between the company and bug reporter be? Also, there are certain kinds of critical bugs that should never happen in production. Are you confident that your test automation is sufficient to find them?
Test automation focuses on testing functionality but has no clue about usability. Who is using your program—another program or a person? If it’s a person, then you will want a professional tester exercising that application with a keen eye on usability. An alternative to having a professional tester do this is having a group of users participate in a formal usability test. This is a good idea, but in all my years of testing, not a single application I’ve tested has ever gone through a formal usability test. I wish more companies would perform usability testing, but it is faster and cheaper to use a knowledgeable tester.
The need for speed in software development and continuous deployment has caused some traditional things, like software versions visible to the user and release notes, to fall by the wayside. For companies that are deploying ten updates a day to an application, how will users know that changes have been made without release notes or versioning? In my experience, testers were the ones who put together release notes. If a user wanted to report a problem, testers are usually the ones who try to reproduce the bug. With no software version being reported, how will users or testers know if the bug has been fixed? Agile is based on feedback, and cutting out versions and release notes hinders feedback. It will also limit your analysis if you’re collecting metrics.
For the company that boasted about deploying to production up to two hundred times a week, is this good, bad, or ugly? I don’t know the number of applications involved, so I can't say. If they have ten apps and we assume half are new or improved features and half are bug fixes, we have five new features and five bug fixes per app per week at the low end and ten each at the high end. If this is a new application, then adding five new features a week may be reasonable, but after six months, this would be excessive. Might this be an indication of feature creep or gold-plating?
The same figures apply to bug fixes. Five bug fixes a week may be very reasonable in the beginning, but if this figure remains steady, then it may be a good indication of inadequate testing.
We’ll have to wait and see if continuous deployment has a positive or negative impact on software quality in the long run. If companies feel they can replace professional software testers completely with automated tests, I think we know the answer. While continuous deployment has the potential to be a very positive trend, never underestimate people’s propensity to misuse a good thing.