WANT TO:

September 2006 - Using Oracles in Testing and Test Automation (Part 3 of 3)

Douglas Hoffman, Software Quality Methods, LLC

In part 1 of this article series, Doug introduced using oracles in testing and test automation. In part 2 of this series, Doug discussed modeling testing with an oracle.Part 3: Characteristics of All Oracles

Deterministic and Heuristic Oracles

When we think of software testing and the test results, it is usually in a binary sense: the result is right or wrong; the test passed or failed. We generally don't allow for "maybe it's OK" or "possibly right" as outcomes. We consider the test outcomes to be deterministic; there is one right answer [and we know what it is]. When we have a means of knowing the outcome, we have a deterministic reference function. Nearly all test automation I've seen is based on deterministic reference functions. However, software and systems today are so complex that more and more frequently we really can't know all the results. In many cases we can use a fallible reference function - a heuristic. So, the use of oracles can be generally categorized as deterministic or heuristic approaches.

Some examples of such deterministic and heuristic verification strategies are outlined below. Each example provides some oracle for determining whether or not a test result is correct. It is useful to note that although the strategy allows us to pass or fail a particular result, in many cases there are ways that the SUT can give us a wrong result, and yet the test can pass even with a deterministic oracle (e.g., an undetected error in an expected results file or an error in an item we don't check). It is also useful to realize that all test results are heuristic because we check a subset of everything the SUT could conceivably do in the case of errors.

Examples of deterministic reference functions

  • Saved result from a previous test
  • Parallel function
    • previous version
    • competitor
    • standard function
    • custom model
  • Inverse function
    • mathematical inverse
    • operational inverse (e.g. split a merged table)
  • Useful mathematical rules (e.g. sin2(x) + cos2(x) = 1)
  • Expected result encoded into data

Examples of heuristic reference functions

  • Similar statistical distributions
    • test for outliers, means, predicted distribution
  • Incidental but informative attributes
    • durations
    • order
  • Insufficiently complete attributes
    • ZIP Code entries are 5 or 9 digits
  • Probabilistic attributes
    • X is usually greater than Y
    • Children are usually younger than their parents

Independent of whether the reference function is deterministic or heuristic, there are seven key characteristics regarding oracles in relationship to the SUT. The results predicted by an oracle can range from having almost no relationship to exact duplication of the SUT behaviors. Completeness, for example, can range from no predictions (which may not be very useful) to exact duplication in all results categories (an expensive reimplementation of the SUT).

The key characteristic areas include:

  • Completeness of information
  • Accuracy of information
  • Usability of the oracle or its results
  • Maintainability of the oracle
  • Complexity
  • Temporal relationships
  • Costs

The most significant components for each are outlined below.

Completeness of information:

  • Input Coverage
  • Result Coverage
  • Function Coverage
  • Sufficiency
  • Types of errors possible
  • SUT Environments

Accuracy of information:

  • How similar to SUT
    • Arithmetic accuracy
    • Statistically similar
  • How independent from SUT
    • Algorithms
    • Sub-programs & libraries
    • System platform
    • Operating environment
    • Close correspondence makes common mode faults more likely and reduces maintainability
  • How extensive
    • The more ways in which the oracle matches the SUT, i.e. the more complex the oracle, the more errors
  • Types of possible errors in the oracle generated results
    • Misses actually wrong value (silent miss)
    • Flags correct data as an error (false alarm)

Usability of the oracle or of its results:

  • Form of information
    • Bits and bytes
    • Electronic signals
    • Hardcopy and display
  • Location of information
  • Data set size
  • Fitness for intended use
  • Availability of comparators
  • Support in SUT environments

Maintainability of the oracle:

  • COTS or custom
    • Custom oracle can become more complex than the SUT
    • More complex oracles make more errors
  • Cost to keep correspondence through SUT changes
    • Test exercises
    • Test data
    • Tools
  • Ancillary support activities required

Complexity:

  • Correspondence with SUT
  • Coverage of SUT domains and functions
  • Accuracy of generated results
  • Maintenance cost to keep correspondence through SUT changes
    • Test exercises
    • Test data
    • Tools
  • Ancillary support activities required

Temporal relationships:

  • How fast to generate results
  • How fast to compare
  • When is the oracle run
  • When are results compared

Costs:

  • Creation or acquisition costs
  • Maintenance of oracle and comparators
  • Execution cost
  • Cost of comparisons
  • Additional analysis of errors
  • Cost of silent misses
  • Cost of false alarms

About the Author

Douglas Hoffman has over 30 years experience as a consultant, manager, and engineer in the computer and software industries. He has held numerous quality-related positions including the position of Vice President of Quality for a 500+ person company, as well as Manager of Software Quality Assurance for several other companies. He has been a registered ISO Lead Auditor, holds Certificates in Software Quality Engineering and Quality Management from ASQ and is an ASQ Fellow. He has also taught Computer Science at the college level at the University of San Francisco, UC Santa Cruz, and Howard University in Washington, DC. Douglas holds a BA in Computer Science, an MSEE, and an MBA.

Copyright © 2006, Software Quality Methods, LLC. All rights reserved. Reprinted here with permission.
 
Back Top