In recent years, software development has evolved as a collaborative process not just internally but beyond the bounds of a given organization. It’s become the norm that products are now dependent on a variety of factors that each include numerous variables affecting the development process. For example, a product might be dependent on a specific app in order to function, the app itself might be dependent on a specific device, the specific device might be only available in a certain geographical locale, and so on.
In this new development model, software testing is increasing in importance as well as complexity. Today’s multitude of desktop and mobile operating systems, browsers, and devices, while on one hand, offer a great set of options for the end-user, also make it challenging for testers to ensure application and device compatibility. The industry has progressed well to handle testing’s growing compatibility needs, from the old era of physical machines to the current days of cloud-powered test environments. However, this is just one of several variables to consider. While this is a positive step, there are other unaddressed or partially addressed areas we need to consider, such as incorporating test debugging, productivity, and automation tools in order to meet today’s integrated testing needs.
Companies still grapple with implementing an automated test framework that can run test scripts from multiple operating system/browser combinations with a single click of a button. An additional requirement to such a framework would be for an unattended (automated) solution to pick a new product build once it has been checked in so it can be deployed and the automation runs can be kicked off. This can occur whether or not the runs are build verification tests, a regression suite, or a set of pre-built sanity tests.
Although the cloud has largely eased test infrastructural pains, a comprehensive solution for testing across the combinations of supported devices and applications is still not very prevalent.
Oftentimes, the tester is switching between machines, running into test labs, or connecting to multiple sources to access resources such as IDEs, knowledge repositories, and test and defect management tools to report results, maintain test scripts, look at developer’s source code, develop new test code, etc. Doing so increases the number of interaction points and associated overhead.
Every tester or test group typically uses test productivity tools that specifically meet their day-to-day testing needs. While the value these tools provide is undeniable, the inconsistencies and dependencies of having them isolated in one-off local machines are also huge because a comprehensive framework that includes all tools in an integrated manner is not available as of today.
As for compatibility testing, we have come a long way since the days of the physical machine set up to the current days of the cloud. It’s become prevalent amongst product teams to leverage infrastructure as a service to pay for machines based on usage and to build machines on the go based on the requirements on hand. While this evolution is welcomed by all product disciplines, especially the testing community, have we maximized our opportunities to create an empowered environment for testers, taking into account all the variables mentioned above? Definitely not. What we really need is a tester ecosystem that supports not just automated compatibility testing but also associated automation tools to aid in the testing, debugging, troubleshooting, and test management efforts. This will create a seamless process that the tester can leverage to improve his productivity and the quality of the product under test. There are some commercial players trying to build such an ecosystem (at least partially) today,