Software Testing has become a self-governing and a very important profession over a period. As the software development process is becoming a complex activity day by day, the demand to continuously evolve the software testing practices and keeping them aligned to the needs of software engineering is becoming important as well.

 

This book does not contain "magic pixie dust" that you can use to create additional time, better requirements, or more enlightened management. It does, however, contain techniques that will make you more efficient and effective in your testing by helping you choose and construct test cases that will find substantially more defects than you have in the past while using fewer resources.

 

 

1. To conceive or fashion in the mind; invent: design a good reason to attend the STAR testing conference. To formulate a plan for; devise: design a marketing strategy for the new product.
2. To plan out in systematic, usually documented form: design a building; design a test case.
3. To create or contrive for a particular purpose or effect: a game designed to appeal to all ages.
4. To have as a goal or purpose; intend.
5. To create or execute in an artistic or highly skilled manner.

 

Each of these definitions applies to good test case design. Regarding test case design, Roger Pressman wrote:

 

"The design of tests for software and other engineering products can be as challenging as the initial design of the product itself. Yet ... software engineers often treat testing as an afterthought, developing test cases that 'feel right' but have little assurance of being complete. Recalling the objectives of testing, we must design tests that have the highest likelihood of finding the most errors with a minimum amount of time and effort."

 

Well designed test cases are composed of three parts:

 

• Inputs
• Outputs
• Order of execution

 

 

 

All of these relevant inputs and outputs are important components of a test case. In test case design, determining the expected outputs is the function of an "oracle."

 

An oracle is any program, process, or data that provides the test designer with the expected result of a test. Beizer lists five types of oracles:

 

• Kiddie Oracles - Just run the program and see what comes out. If it looks about right, it must be right.
• Regression Test Suites - Run the program and compare the output to the results of the same tests run against a previous version of the program.
• Validated Data - Run the program and compare the results against a standard such as a table, formula, or other accepted definition of valid output.
• Purchased Test Suites - Run the program against a standardized test suite that has been previously created and validated. Programs like compilers, Web browsers, and SQL (Structured Query Language) processors are often tested against such suites.
• Existing Program - Run the program and compare the output to another version of the program.

 

• Cascading test cases - Test cases may build on each other. For example, the first test case exercises a particular feature of the software and then leaves the system in a state such that the second test case can be executed. In testing a database consider these test cases:

 

1. Create a record
2. Read the record
3. Update the record
4. Read the record
5. Delete the record
6. Read the deleted record

 

Each of these tests could be built on the previous tests. The advantage is that each test case is typically smaller and simpler. The disadvantage is that if one test fails, the subsequent tests may be invalid.

 

• Independent test cases - Each test case is entirely self contained. Tests do not build on each other or require that other tests have been successfully executed. The advantage is that any number of tests can be executed in any order. The disadvantage is that each test tends to be larger and more complex and thus more difficult to design, create, and maintain.

 

As Lee mentions about this book that, “It does, however, contain techniques that will make you more efficient and effective in your testing by helping you choose and construct test cases that will find substantially more defects than you have in the past while using fewer resources.”

 

The book takes us to understand various test designing techniques that are explained with number of good examples, which makes the reading interesting and an experience. The book is a ‘must_read’ for all test professionals.

 

 

 

In his forthcoming book, Developing Software Test Cases, Ross defines test case as,

 

“A test case exercises one particular situation or a condition of the system being tested.”

 

 

He thinks that the purpose of the test cases is,

 

“A test case describes what to test and how and provides direction to the person (or to automated test tool), which perform testing. The direction includes how to set up and execute the test case, what behaviors to monitor and how to evaluate the results.”

 

 

After a detailed discussion about test case structure- test condition, test procedure, test results and test environment, he says,

 

“there is no standard or correct way to document the test cases. Multiple projects will use multiple formats, or a project will use multiple formats. Such variation is justified when test cases need different information or it might be just due to lack of coordination in test teams, which is often the case.”

 

 

Then Ross brings out a very interesting discussion about the level of details that a test case may have versus the experienced test team and non-experienced test team, who will be executing the test cases. Then he brings out another important aspect of good test design and test cases- collaboration of insiders and outsiders. According to him, the insiders have usually a very good knowledge of the system under test and they can develop better quality tests from this perspective however, they lack detachment from development team and hence they may lose the advantages of this good design by sheer mismatches due to human relationship dynamics and losing the objectivity of the project.

 

Whereas, though the outsiders have less knowledge of the system, due to their detachment, they may become highly successful in bringing out ‘breath of fresh air’. However, the main weakness of the outsiders is the depth into the system knowledge and he thinks that this is one of the areas, where they have to work hard to develop good quality test cases.

 

 

 

 

Harry Robinson is Test Architect for Microsoft's Enterprise Management Division. In addition to his day job, he teaches classes on advanced software test automation and is a driving force behind Microsoft's model-based testing initiative. He has been at Microsoft for five years and has a bachelor's and master's in electrical engineering. You can reach him at [email protected].

 

 

“Intelligent Test Automation”
Let’s look at an example of creating and using a behavioral model to test a software application.

 

Hands-on testing is a good way to start the test automation process. I call this phase “exploratory modeling” because it combines exploratory testing with the discovery of a model that can later be used to generate tests. As you begin to understand the behavior of each action, you can create rules that will help you model and test the application.

 

This is the essence of model-based testing: To describe the behavior you expect in a way that can be used to generate tests. Ask yourself the following two questions for every action you are going to test:

 

1. When is this action possible?
2. What is the outcome when this action is executed?

 

For instance, suppose you have been asked to test the behavior of files in a Windows folder. In particular, you are going to test the Create, Delete, and Invert Selection actions.

 

“It’s Different in Test”
There's a saying that "If you all think alike, some of you are unnecessary." If that is true anywhere, it is certainly true among testers. There may be some justification for having a team of developers think in lockstep, but for testers it would be a catastrophe for everyone to think alike. One of the chief benefits testers bring to any group is a different viewpoint, and they should be encouraged to disagree with each other. I get a kick out of watching serious testers talk; it is usually at elevated volume levels.  
 
Actively seek out a good mix of skills and backgrounds when filling out your test team. Do you have coders and non-coders on your team? Each will see different types of defects. How about both genders? Multilingual? Culturally diverse? Differently-abled? One of my favorite true stories is about the Tablet PC team demonstrating its handwriting recognition function to Bill Gates. The team was very excited and confident, but they had overlooked a big factor. The Tablet PC team was almost all right-handed, but Gates is a lefty. Guess what? The handwriting recognition didn't work well for him. Oops.

 

 

Hung has written a paper- Tests Built-to-Last: Software Tests as an Asset - about good test cases and insists that test cases should be treated as an asset. His thought is analogous to our thought of treating test cases as product.

 

Hung writes,

 

"Software Tests as an Asset" Philosophy

 

Software testing, and the software tests that are created, need to be looked upon as an easily maintained, renewable and reusable resource that inherently captures what an organization knows about how the software under test is supposed to work, and provides visibility into the quality of the software under test. Some of the most fundamental precepts of the "Software Tests as an Asset" philosophy are:

 

• Good test design must be the focus of software testing, with very little time actually spent on automation of the tests.
• Tests should be inherently automation-ready so that a software test case can be automated with very little effort.
• Tests to be automated should follow the "5% rules" developed by LogiGear CTO Hans Buwalda: No more than 5% of tests should be executed manually; No more than 5% of testing efforts should be expended automating the tests.
• Software tests, whether they are manual or automated, should be optimized for visibility, reusability, scalability, and maintainability.

 

Clearly, this is fundamentally different from the more "traditional" approaches to software testing. Only by adopting such a dramatic shift in philosophy will an organization be able to significantly improve the speed and reliability of testing, while reducing costs and improving software quality. By adopting such a philosophy, an organization will have software tests that are built to last. The tests will be reusable as is, or with small amounts of maintenance, to test future generations of the software under test.

 

Exception
There will be tests that will not be subject to the "Software Tests as an Asset" philosophy. For example, tests that are simply focused on defect finding that you know you will never use again once the software is released would not benefit from the "Software Tests as an Asset" philosophy.

 

 

While working as a developer, coding software testing tools, Alan Richardson’s interest switched from programming to software testing. Since 1993, software testing has been Alan’s professional specialism and he has worked at all levels of the testing hierarchy; test execution and design, test management, strategy and methodology. He is currently an independent test consultant and helps his clients with every aspect of software testing.

 

 

Alan wrote a paper about “Practical Experiences in Graph-Based Testing” and this paper has many useful tips about making the test cases effective and reduce efforts on developing the tests. I am going to focus quickly on some of those useful tips in his paper.

 

 

Alan says,

 

 

“One of the main reasons for writing this paper is that, and I generalise wildly here, I don’t think that structural graph models are used enough in testing. I certainly don’t encounter many testers using graphs, which is a shame as graphs are well covered in the testing literature, and they are enormously helpful. I could suppose that this might be due to testing tool support, but I don’t really believe that. Perhaps other testers just haven’t been through the same testing experiences as I have? So one thing I’m going to do in this paper is present an account of my use of graphs in testing.”

 

 

“The use of graphs in testing is not a panacea, but it is useful, it is a technique that is easy to learn and has numerous benefits. Sometimes I don’t apply it, but you have to learn a technique so that you know when to apply it and when to discount it.

 

 

Graphs will be presented as useful models for:

 

• deriving test conditions
• understanding systems
• communicating the tester’s view of the system
• automating the production of test scripts
• assessing a variety of coverage measures
• visualising and reporting coverage measures”

 

 

Alan then takes us to understand how the graphs can be used to negotiate with critical situations and achieve higher satisfactory results with the help of the graphs

 

 

Communication Graphs leading to test development
The project was being conducted in a structured testing and development environment, but it was not going well. The development specifications were ambiguous, being produced late, delivered late to the test team, and they were wrong.

 

Writing test scripts was taking too long and the constant change resulted in too much rework. As a mitigating strategy I produced a number of graphs which I could use to communicate my understanding to the development team. Normally I didn’t use my graphs in this way, but it seemed fairly sensible that if I was using them to understand the system then I could use the graphs to communicate my understanding of the system. The graphs went through quite a few iterations until they were understood by the development team and everyone was satisfied that they represented what the testers could do to the system. The development team also made a few changes to the system as a result of reviewing the graphs as they identified issues with the system or specification. And where the graphs were not complete, we supplemented them with some high level test case descriptions. An unexpected benefit from all this communication was that the development team reported it was the first time they actually had proper visibility into the test process. They had always been too busy to read the test documentation before or plow through dozens of test scripts to see what the testers would be doing. As the project was running to tight timescales, and we still had to plan the testing, the scripts were documented at a high level of abstraction; with an overall aim, the path through the graph to be taken, the test data to be used, and the various test conditions that would be covered.

 

 

 

 

 

Alan summarizes variety of benefits of using graphs for testing-

 

 

For Communication

 

• Rather than have people review hundreds of scripts, which they never seem to manage to get around to doing or doing effectively, they can instead review a handful of diagrams.
• Graphs communicate my understanding and help other people understand the systems better.
• The cliché is that a picture is worth a thousand words. My experience tells me that a graph can summarise several pages of requirements and development specifications.
• Sometimes all that has to be done to prevent defects is draw a graph, show it to the developer and ask a few questions.
• Sometimes the graphs will make it into the development documentation for future readers.
• Show your graphs to other people, don’t keep them to yourself, they are too valuable for that.
• Remember, there is more to a graph than just a picture of a graph. The picture is a visualisation of an underlying representation, and this is what you are communicating, you use the picture as a way of communicating it.

 

 

For Understanding

 

• Generally when logic or interaction is hard to comprehend in textual form, I draw a graph. And when I am drawing graphs for understanding I use my favourite tool; Paper and Pen.

 

 

For Definition

 

• To support script definition, extra step description information is added to the nodes and edges, to tell the tester what to do when traversing a path through the graph.
• Don’t try to model everything in a single graph, use multiple graphs.
• You don’t have to model everything in a graph, you do have other test techniques too.
• You don’t need to get too detailed, keep your graphs manageable.
• If you have a set of test scripts which are similar, consider reverse engineering a model of the scripts as a graph.
• Graphs can evolve, and become more detailed, as the project matures.

 

 

Test Design

 

• Test Design is different from Test Script design. A test is something that you want to check. A script tells you how to check it in order to determine if a particular test can be passed. When viewing the test process through the eyes of the Script Meta Model, tests are justifications for traversing a path.
• In a structured test process graphs can be used to provide a justification for some of the scope for testing, in session-based testing, a graph can provide a justification for some of the test charters.
• The standard way of deriving tests from graphs is by covering the paths through the graph [Beizer95][Binder00][Beizer90]). Despite the utopian ideals of the Script Meta Model, we will not get all our tests from coverage. The graphs we produce are usually not detailed enough.
• We get extra tests from the graph by examining it and identifying the behaviour that is not modelled.
• Graphs which are modelled too deeply too quickly can become unwieldy and hard to maintain. There has to be a trade off between understanding, communication, and derivation and this will depend on why you drew the graph, the position in the lifecycle, and probably the toolset that you are using to model the graphs.

 

 

Test Script Design

 

• A test script and a test design are different. All test scripts could be automatically generated from a very detailed model, all test designs can not.
• In essence we derive paths from the graph. Paths can be described as sequences of edges. Because every edge has a start node and an end node, we only need to use edges.
• Paths are generated from graphs by the application of strategies to the graph: e.g. node coverage, branch coverage, predicate coverage, cover all loops twice
• When we are covering the paths we are not covering the tests. The script meta model shows us that paths are independent from tests. It is possible to build scripts from paths directly but these scripts do not have the contextual aims that the condition model provides, nor does it exercise the data adequately.
• A Path is a script, but it is an uninstantiated script. To become a test script, we need to know the data used and the conditions covered.
• When I extend the detail in graphs to make them more suitable for script generation, I try to avoid confusing the graph. So I add any important linking edges by adding them as dotted lines, and making start and end nodes different colours. These extra visualisation attributes are important for retaining the communication benefits.
• Give each graph, node and each edge a unique ID that you can use in your path edge sequences and in your cross referencing