September 2006 - Using Oracles in Testing and Test Automation (Part 2 of 3)

Douglas Hoffman, Software Quality Methods, LLC

In part 1 of this article series, Doug introduced using oracles in testing and test automation.

Part 2: Modeling Testing With an Oracle

When we first think about a test we picture doing a series of actions and observing results, much like the model shown in Figure 1.

Figure 1

The model is incomplete, however, since it doesn't take into account any of the past history or data not specifically input. Figure 2 presents a much more complete model that includes precondition data, precondition program state, and environmental factors, all of which may significantly change the expected behavior of the SUT and usually need to be known and factored into the picture in order to determine what the SUT is expected to do.

Figure 2

A quick example in each of the six new areas should explain. Assume that the test is supposed to verify whether adding a user works:

• Precondition Data includes databases and memory values already in place when the Inputs are provided. This would include the set of existing users. The SUT should behave differently for a new user name vs. an existing name.
• Precondition Program State is the obvious and not so obvious state of the program. If an error dialog is currently displayed it will have to be dismissed before the new user can be added.
• Environmental Inputs includes physical and computational elements. The temperature of equipment, operating system, other running programs, memory size, system administration activities, and user permissions are a few impacting environmental factors. A busy network connection to a database server could cause timeouts and error dialogs to appear.
• Post-condition Data is the state of databases and memory values after the exercise. There is potential for the added user to be in the database but other users' data could be changed or removed. (For buggy code, any accessible data sets could have been modified.)
• Post-condition Program State is the obvious and not so obvious state of the program. Error dialogs are easy to see, but internal states may not be accessible outside of the SUT.
• Environmental Results include possible interference with other programs, tying up the network, leaving files around, etc. Memory leaks are a prime example of errors detected in the environment.

Some of the biggest difficulties in software test automation are in knowing what results are expected from the SUT. There are many issues with the huge number of potentially relevant results and how to record them. (This is especially true when we consider that there may be errors in the SUT, so the actual outcomes may be outside of the expected realm.) Often, it is extremely difficult to predict what the SUT should do and what outcomes are expected, even for the set of expected results. Figure 3 illustrates a model incorporating an oracle to predict expected results from actual inputs.

Figure 3

Figure 3 shows the input factors for the SUT being applied to the oracle. The oracle then generates the expected outcomes based on the inputs. A complete oracle would be capable of accepting the same set of inputs and generating the expected behaviors and outputs. No test oracle is complete, so what the diagram really shows is some subset of the input factors being used to generate some subset of the outcomes. Indeed, we frequently apply multiple oracles to a test case - the functional result may be judged based on business rules while the screen navigation is based on program specifications

Often the post-condition information and environment are overlooked when testing, but oracles can often provide inexpensive and powerful ways to monitor and verify them. Many new opportunities open up when oracles and automation tools are brought together. For example, most applications are not supposed to change any environmental factors when they are done. It is usually not difficult to capture environmental variables, file characteristics, and other information before the test and compare with the environment afterwards.

In part 3 of this article series, Doug will discuss characteristics of all oracles.

About the Author

Douglas Hoffman has over 30 years experience as a consultant, manager, and engineer in the computer and software industries. He has held numerous quality-related positions including the position of Vice President of Quality for a 500+ person company, as well as Manager of Software Quality Assurance for several other companies. He has been a registered ISO Lead Auditor, holds Certificates in Software Quality Engineering and Quality Management from ASQ and is an ASQ Fellow. He has also taught Computer Science at the college level at the University of San Francisco, UC Santa Cruz, and Howard University in Washington, DC. Douglas holds a BA in Computer Science, an MSEE, and an MBA.

Copyright © 2006, Software Quality Methods, LLC. All rights reserved. Reprinted here with permission.