given change. This is because running all tests for a project might be quite lengthy. Although this would provide maximal coverage, it wouldn't provide sufficient rapid feedback to be any improvement over traditional CI. Different CT tools provide different mechanisms for handling this, as will be discussed later in this article. But for the moment, let's consider the approach taken by Infinitest, a Java CT tool.
Infinitest uses a byte-code dependency analysis technique to identify the subset of tests most likely to be affected in response to any given change. For example, consider the related classes illustrated in Figure 1 below:
Figure 1 . FooTest and BarTest each exercise a class under test which in turn depend on other classes in the system under test
For any given change Infinitest will traverse all nodes affected by the change and collect all the nodes which it can identify as being test cases. These tests are re-run by the infinitest test runner and the results displayed within the developer's IDE. In this case, changes to the Fub class cause the FooTest to be re-run.
In this manner Infinitest quickly identifies the smallest sub-set of tests that need to be re-run and displays the results immediately to the developer without interrupting the development workflow. Figure 2 shows how the Infinitest presents the results within the IntelliJ IDE.
Figure 2. Infinitest is continuously running tests affected by code changes in the background, providing developers timely feedback throughout the development process
So if a CT tool is executing affected tests with every change, what happens with slow running tests, particularly if a developer is making frequent changes? First, it is helpful to clarify which kind of tests CT tools are most effective at running. Figure 3 illustrates one classification scheme for automated tests.
Figure 3. Well tested applications will usually have automated tests which run against the system under test at various levels of abstraction
CT tools don't know or care about whether your tests are unit or integration tests. Rather they work most effectively with fast tests regardless of classification as illustrated in Figure 4.
Figure 4. The tests from Figure 3 are reclassified as fast tests and slow tests with lighter shaded tests being slow running tests while darker shaded tests are fast running tests. Notice that tests are classified as fast or slow regardless of test type, i.e. some unit tests are slow while some system tests are fast
Infinitest in particular allows developers to exclude slow tests from the CT monitoring process through. In the future the Infinitest team hopes to automate the identification of slow running tests dynamically through the continuous testing process and exclude them from future test runs.
Because CT is not (yet at least) a complete replacement for CI, developers will still want to execute the project's build before checking in, and still include the full suite of tests in CI runs. While not replacing CI, CT gives developers as much feedback as rapidly as possible. Developers see the results of their changes as they are made rather then only when they run a manual build, or when check in their changes and the CI server runs one or more configurations of tests against the new state of the code base.
CT tools also facilitate rapid feedback by allowing the test preemption and prioritization. After a developer makes a change, affected tests will begin running in the background and feedback is presented in real-time to the developer who is free to continue developing. If the developer completes another change before the previous test run completes,