A hallmark of truly agile teams is an unquenchable desire to continually find new and better ways of developing software. These teams believe that there really are no "best" practices, only practices which work better or worse for them. This line of thinking is even apparent in the first line of the agile manifesto stating "we are uncovering better ways of developing". Notice the progenitors of the manifesto didn't propose that they had discovered the best way, only better ways.
In this article I will explore two of the most widely accepted agile development practices, Test Driven Development and Continuous Integration, and question how these practices can be made better with continuous testing. I will examine their strengths, shortcomings, and how the ideas behind continuous testing can provide better and faster feedback on the health of a code base. While this article will illustrate with examples from the Infinitest tool, it will will also point out the other common continuous testing tools and categorize their underlying strategies as well. After reading this article you will be grounded in the ideas of continuous testing and ready to select a tool and apply this practice on your own projects.
Test-Driven Development (TDD) and Continuous Integration (CI) are popular and effective software development practices. In TDD developers only write production code in response to failing tests. This helps teams to focus on doing just enough analysis and design to implement the current highest priority feature. One of the benefits of TDD is that production code and tests provide rapid feedback on the quality of one another. The typical Red/Green/Refactor TDD cycle lasts around 5-10 minutes, and the developer usually manually runs the test that is driving the cycle between coding and refactoring iterations. This means that although the feedback loop is very quick, unintended side effects could break other tests and the developer might not receive this feedback until the entire test suite is run.
CI is a technique whereby an automated build for a project is run after check-in, at some pre-defined interval, or a combination of both. Because all the tests for a project are typically run at some interval the test coverage provided by this practice is complete and the development team will all be aware of any unintended breakages. Although CI offers more complete coverage then TDD, the time it takes to get feedback is much greater [AE1] with developers receiving results at best a few minutes after a set of changes is checked in.
Both TDD and CI are based upon the principle that feedback is good, and the quicker the feedback the better. With CI feedback is typically provided to developers no sooner than the moment of check-in. With TDD, feedback is provided for the tests manually run by the developer between every TDD cycle. And as previously mentioned, the quicker feedback of TDD is provided at the expense of more complete test coverage around any given change.
So the question to ask in the spirit of continuous improvement is, "How can we improve on these practices by continuing to reduce the feedback loop while increasing the relevant coverage of our tests?"
The practice of Continuous Testing (CT) aims to reduce the feedback loop of TDD to the moment of compilation while preserving the breadth of test coverage provided by CI.
Since the goal of CT is to provide rapid feedback for every change, and since changes could likely be happening rapidly as well, CT tools need to be smart about which tests to run in what order in response to any