Open Source Test Automation

The breadth, depth and iterative nature of testing in today's complex application environments does not allow comprehensive testing to be carried out unless the test group has an automation tool for testing. Despite its best intentions, the test group simply does not have the bandwidth to manually execute all the test cases developed.

Even if the test group invests in a commercial test tool, the steep cost of acquiring adequate number of licenses and tool inflexibility prohibits testing at a satisfactory level of efficiency predetermined by the test group. With these identified issues, the most convincing solution seems to lie in adopting open-source test tools and customizing them for in-house testing needs of the organization. Test tool functionality is extended to be time-tested and adaptable to changing technologies, thus assuring a longer shelf life, which extends beyond the lifespan of the development work of a few heroic individuals.

This paper introduces you to Open-Source Test Automation (OSTA) and outlines the procedure for building and/or adopting open-source scripting and customizing for the business needs. It also lists the benefits of OSTA to the industry, its benefits and limitations. In addition, the paper summarizes open source framework and ways of accessing and implementing open-source scripting.

Problem Statement
Despite of huge investment on commercial test tool for automation, their licensing defeats the very purpose of the usage of a test tool. The commercial licensing models keeps the test tools away from the people (like developers, manual testers) who could most benefit from them.

The need for customers to share their test tool licenses to offsite developers has also increased due to tremendous growth in off-shoring practices. This causes the tool user to pay additional overhead to their commercial vendors, based on agreed licensing models. In addition, the customer requires re-licensing of tool for each technology being tested, by which customer pays more than required.

There is a forced need to write test scripts exclusively in the tool's scripting language. This forces the user would need to learn the vendor script in order to use the tool. The tool user should write automation scripts even though tool recorders help generate code for making it run for more reasons than one. The test automation framework written for specific commercial test tool does not allow the user porting to another test tool as neither of them shares a common platform.

Most of the commercial tools are vendor locked with other test support tools like defect tracking, source controlling and test management tools, thus forcing the user to buy those tools for test management and integration. The user may end up in investing more than expected.

Moreover, the commercial test tools have many other limitations such as 'the test scripts are only foreground runners thus consume a machine completely for each test execution instance'. They have been criticized for bloated code, consuming larger memory and more processor cycles with successive tool version releases. There is commercial pressure on the buyers too to purchase the latest tool version due to increase in functionality and bug fixes.

Thereby, commercial test tool(s) causes greater concerns to test automation industry by unkind licensing models, closed source code, inflexible scripting languages and interoperability restrictions which impacts testing and business.

Proposed Solution
The basic idea behind open-source software is very simple: "When the programmer can read, redistribute, and modify the source code for a piece of software, it evolves. Others improve it, adapt it, customize it, and fix bugs. And this can happen at a speed that, if one is used to the slow pace of conventional software development, seems astonishing".

Gartner hypothesized that Open-Source development will be one of solutions in the coming years and they say: "OSS is a catalyst that will restructure the industry, producing higher-quality software at a lower cost. As thought it won’t destroy industry giants, but it will revolutionize software markets by moving revenue streams from license fees to services and support". Hence Open source is expected to drive more revolution in the business world by offering customers better value, flexibility, interoperability, and security.

In few short years, the open source movement has grown from a relatively small niche player to a major force, disrupting the IT status quo and competing with the proprietary platform giants. At the same time, software test automation is growing rapidly by providing quality support to manual testing along with operating cost advantage. The merged benefits of these two significant developments ensure on-time delivery, with notable Return on Investment (ROI) for the business, not to mention customer satisfaction.

Obviously, a big attraction in this proposed solution is COST. Commercial tools require large up-front payouts, rather than ongoing payments for the tools that are actually being used. They are expensively priced per seat, rather than using site-wide licenses that would encourage customers to make maximum use of the test-suites. The cost of open-source test automation as a total cost involving the entire lifecycle of the software (including purchase & maintenance price) is cheaper than any commercial test tool.

This OSTA model can improve the quality of software development by testing early (through developers, manual testers and business analysts using the same test scripts) and by easy integration with other open-source tools like CVS, Bugzilla and TestRunner. This solution can be ported across projects/products, thereby minimizing overall testing and development efforts, improving quality and bringing flexibility to the test automation.

The primary objective of the OSTA is NOT just about saving money on the licensing cost, but also about flexibility, portability and adaptability of the entire test automation exercise. Thereby, Open-source scripting for test automation (both for functional and performance testing) is a highly viable solution. It provides a strong business advantage against the competitors, and also delights the customers.

Implementation Model
This section elaborates the functional web open-source test automation implementation techniques along with the construction mechanism using Java/.Net/JNI library which allows web browsers to be controlled from the chosen language. The primary purpose of these libraries is to allow automated regression testing of web applications using an open-source framework like JUnit.

How does OSTA works (One of the implementation model)?
Microsoft exposes the events, methods and properties which control a remote instance of Microsoft Internet Explorer through automation. The WebBrowser object libraries which add the browsing capabilities of the application under test are also available. They also lists C/C++ object model interfaces, and identifies which scripting objects most closely match these interfaces.

Considering Java as the chosen language, developing JAVA-COM bridge components allows the script to call COM/DOM interfaces of the Internet Explorer API from Java classes. It is recommended to use JACOB project (open-source project) that uses JNI to make native calls into the COM and Win32 libraries instead of home-grown libraries.

Build the automation libraries (methods, scripts and drivers) around the bridge components using JUnit framework and thus making the test automation scripting easier and scalable. There are equivalent interface mechanisms for other technologies like Ruby, .Net, JavaScript and FireFox browsers.

There are well developed open-source GUI drivers for JAVA (like Jemmy, Abbot, and JfcUnit) which use Robot to generate native OS events and then drive control components directly by generating AWT events. In addition, there are a few basic scripting languages like VB, Perl that can act upon the GUI of the Windows using OLE automation.

This approach is superior to commercial tools as standard scripting language is used, facilitating library development. Test scripts can easily be stored in a configuration management tool such as CVS. And these tests can be run either with the browser visible, which aids debugging, or in the background, allowing other work to occur while tests are running.

This model not only benefits the testing and development team but can also impact the methodology and discipline that drive many open-source efforts. This solution also brings the ability to collaborate among companies to share R&D costs and leverage economies of scale.

References

1. Gartner research report, "Positions 2005: Open-Source Solutions Will Restructure the Software Industry", 2005
2. Testing Hotlist Update (blog),  2005
3. Internet Explorer object, MSDN
4. JACOB, "Java COM bridge", 2004