Justin Rohrman writes about the benefits of smoke tests. These tests can be simple and concise, and because of that, low cost. Spending a few minutes of your time watching the smoke test run can be a fantastic way to defocus and notice some things about your product that may normally go unnoticed.
A few days back, a friend and I were chatting on Skype about test automation. I have a really hard time getting excited about automating a graphical interface, but one aspect of automating the browser I am excited about is the ‘really small smoke test’. These tests can be simple and concise, and because of that, low cost. Spending a few minutes of your time watching the smoke test run can be a fantastic way to defocus and notice some things about your product that may normally go unnoticed. Let's explore that idea a little bit.
Try It and See
A smoke test is a short test or set of tests that will run quickly and tell you something about basic functionality in your software. Generally these tests don’t do a whole lot, and that’s just fine.
Selenium IDE is a FireFox plugin that will allow you to record and rerun actions in the web browser. If you are not a programmer, the IDE can be a great tool to create smoke tests. Take a look at this tutorial for selenium IDE to create a smoke test in your browser, but try to do it for your software, not Amazon.
If you are more familiar with programming, you should be able to write something with the Selenium Webdriver framework relatively quickly.
Go ahead and run the test, click the go-button, and then walk away and get some coffee. When you get back, what do you have? A log file and probably nothing else, that’s what. If everything passed, you know very little about what the customer really saw during the test run. If something went wrong, then you have to go back, look at the logs, maybe read some of the test code, and then go to the software to figure out what happened. Sounds time consuming, right? It is.
Now, let’s try that one more time, but this time don’t leave! Stay in front of the monitor and watch what happens as your test moves along. This time when the script is done running, what do you have? You still have your log files, but you also have your observations and all it cost you was a couple of minutes. If the test completed successfully, you know why this time. If the test didn’t complete successfully, you were there to see what happened. Or maybe the test completed without error but you were able to see something else interesting while it was running that the smoke test wasn’t created to catch.
By staying and observing you have expanded this mimeomorphic (done the same way every time) script into a polymorphic-thought exercise that occurred in your brilliant tester brain. You can observe and learn so much more than a simple script ever could.
Focus and Defocus
These tests are so simple that you could potentially execute them yourself without any extra tools. In the world of software testing, we sometimes use focusing and defocusing heuristics to change our perspective. Jesse Alford spoke about these in a lightning talk at the 2013 Conference for the Association for Software Testers in which he compares focusing and defocusing heuristics to photography. Executing a test like this would be a way to focus in on specific detail of your software.
Creating a script can be a way for you to defocus your smoke test. Rather than concentrating on specific details of your test, you have the opportunity to broaden your perception and observe the software in a general way. In the words of Yogi Berra, you can observe a lot just by watching. While the test is running, you can be scanning the software, looking for interesting things that may normally be hidden under layers of focus.
You Can't Go Home Again
I have previously written in my personal blog about a test heuristic I call "You can't go home again." The essence of this heuristic is that running a single test many times is the same as running many slightly different tests. I think this is the case for two reasons: people have extreme difficulty doing a task exactly the same way every single time and there are innumerable variables for a test and knowing how they will affect the outcome is difficult at best. Using the smoke test technique I've described can encourage usage of this heuristic and may give you a more holistic product view. Two ways you could try this out at work tomorrow are watching the smoke test run many times yourself and purposefully defocusing each time, or, by having a few people watch the test run and sharing notes after about what they observed and learned.
The trick goes like this:
1 - Create the script
2 - Run the script
3 - Watch the script run
4 - Learn
5 - Share what you learned
Figure 1. The Share of Actions: What Humans and Machines Can Do by Harry Collins and Martin Kusch