Time to Think about an Open Source Automated Testing Ecosystem

[article]
Summary:
In our modern world of software development, software testing is increasing in importance as well as complexity. Today’s multitude of desktop and mobile operating systems, browsers, and devices, while on one hand, offer a great set of options for the end-user, also make it challenging for testers to ensure application and device compatibility.

In recent years, software development has evolved as a collaborative process not just internally but beyond the bounds of a given organization. It’s become the norm that products are now dependent on a variety of factors that each include numerous variables affecting the development process. For example, a product might be dependent on a specific app in order to function, the app itself might be dependent on a specific device, the specific device might be only available in a certain geographical locale, and so on.

In this new development model, software testing is increasing in importance as well as complexity. Today’s multitude of desktop and mobile operating systems, browsers, and devices, while on one hand, offer a great set of options for the end-user, also make it challenging for testers to ensure application and device compatibility. The industry has progressed well to handle testing’s growing compatibility needs, from the old era of physical machines to the current days of cloud-powered test environments. However, this is just one of several variables to consider. While this is a positive step, there are other unaddressed or partially addressed areas we need to consider, such as incorporating test debugging, productivity, and automation tools in order to meet today’s integrated testing needs.

Problem Statements
Companies still grapple with implementing an automated test framework that can run test scripts from multiple operating system/browser combinations with a single click of a button. An additional requirement to such a framework would be for an unattended (automated) solution to pick a new product build once it has been checked in so it can be deployed and the automation runs can be kicked off. This can occur whether or not the runs are build verification tests, a regression suite, or a set of pre-built sanity tests.

Although the cloud has largely eased test infrastructural pains, a comprehensive solution for testing across the combinations of supported devices and applications is still not very prevalent.

Oftentimes, the tester is switching between machines, running into test labs, or connecting to multiple sources to access resources such as IDEs, knowledge repositories, and test and defect management tools to report results, maintain test scripts, look at developer’s source code, develop new test code, etc. Doing so increases the number of interaction points and associated overhead.

Every tester or test group typically uses test productivity tools that specifically meet their day-to-day testing needs. While the value these tools provide is undeniable, the inconsistencies and dependencies of having them isolated in one-off local machines are also huge because a comprehensive framework that includes all tools in an integrated manner is not available as of today.

As for compatibility testing, we have come a long way since the days of the physical machine set up to the current days of the cloud. It’s become prevalent amongst product teams to leverage infrastructure as a service to pay for machines based on usage and to build machines on the go based on the requirements on hand. While this evolution is welcomed by all product disciplines, especially the testing community, have we maximized our opportunities to create an empowered environment for testers, taking into account all the variables mentioned above? Definitely not. What we really need is a tester ecosystem that supports not just automated compatibility testing but also associated automation tools to aid in the testing, debugging, troubleshooting, and test management efforts. This will create a seamless process that the tester can leverage to improve his productivity and the quality of the product under test. There are some commercial players trying to build such an ecosystem (at least partially) today,

User Comments

2 comments
Gautam Gupta's picture
Gautam Gupta

Very interesting and great vision. Thanks for this great post

March 15, 2013 - 3:51am
Amit Bhatt's picture
Amit Bhatt

I am happy to read this innovative idea being introduced by CEO of our organization, QA Infotech!

This framework will make the task easier for us and increase the productivity on a high scale.

If all or most of the testing groups of the organizations come forward with this endeavor, it may spot an important landmark in coming years down the line.

Being a Web Accessibility Engineer, what I would expect from the related people to make this Automated Testing Ecosystem user friendly to the visually challenged testers/engineers and compliant to the Screen Reading applications such as Jaws and NVDA.

Lets come together and work together!

Thanks

March 29, 2013 - 5:08am

About the author

Mukesh Sharma's picture Mukesh Sharma

Mukesh Sharma created QA InfoTech with a vision to provide unbiased Quality Assurance (QA) solutions for business partners worldwide. As CEO, he is responsible for the company’s global operations, marketing, sales, and development efforts. Under Mukesh's leadership, QA InfoTech has grown to five centers of excellence, 600 employees, and over $15 million in revenue.

Mukesh’s career spans DCM Data Systems, Quark Inc., Gale Group, and Adobe Systems in software quality engineering and test management roles. Mukesh is an active test evangelist.

AgileConnection is one of the growing communities of the TechWell network.

Featuring fresh, insightful stories, TechWell.com is the place to go for what is happening in software development and delivery.  Join the conversation now!

Upcoming Events

Nov 09
Nov 09
Apr 13
May 03