| WANT TO: |
By Bob Galen, Founder RGalen Consulting Group, LLCIntroduction The purpose of this series of articles is to share the breadth of factors that make for a compelling business case for test automation. The breadth of factors that make up a good business case for test automation include:
The first article in this series covered the most common assessment areas to examine when reviewing the automation business climate. This, the second of three articles, will explore determining the "hard" return on investment (ROI) in some detail. Determining "Hard" ROI Hard ROI refers to savings related to testing time or results quality and subsequently dollars. Typically, an organization will make some general assumptions related to the effect that automation will have on their test execution and extrapolate savings versus the overall costs associated with the automation. There are 4 primary areas where test automation will impact the current testing expense structure:
Improved Test Coverage Simply put, improved test coverage is the ability to execute more test cases in less time - leading to improving the overall coverage per the same unit of time. Given that testing is very much a cyclical activity, then increasing coverage in standard cycle times can significantly improve the ability to influence quality. In order to extrapolate the savings, an organization will need historical insights into the time required to execute manual test cases and average numbers of tests executed in their typical cycle times. Coverage increases vs. time saved can be based upon the organization's estimated automation rates. One of the initial challenges facing an organization when establishing the ROI is having sufficient data surrounding their current execution cycle time. Many teams have not established baselines from which to measure. For any organization where this is true, then one of the first steps that should be taken is to establish current performance dynamics. Reduced Testing Execution Time Reduced testing execution time, which is related to the above coverage improvement, is simply the reduced overall time to execute tests. As above, there should be a baseline estimate associated with an organization's previous execution speed - usually measured in average test cases, per person, per day. With this, depending on the number of testers assigned to the project, an organization can calculate release testing phase speed. Commonly, testing cycles usually fall within a 1-2 week interval. Implementing automation can have the effect of driving coverage up, driving time down, or doing both in predetermined moderation. An organization's focus should be determined by project requirements and their current coverage levels. The initial focus should be on driving coverage up, because it usually has the greatest exposure. Later the organization should work to drive time down and then establish a more effective balance between the two. Decrease of Defect Escapes into Production Improving coverage and execution time are relatively easy to characterize. However, showing improved quality is a bit harder. One good measure is to review defects escaping into the customer base. An organization will need to mine their defect data for production level problems and identify root causes that relate to their verification testing. In some cases, they will establish this metric as part of their automation development and then measure improvement over time. The organization will need to extrapolate the cost of production or customer found defects (on average) as well. Typically these costs are unique to a given organization and can be derived with the help of the QA or Operations team. Once the organization has a baseline for escapes and understands the cost, they can track improvement and potential cost savings as a result of expected and/or actual trending after their automation is running. Improved Test Repeatability Often stakeholders look to offshore testing as an exercise in moving less interesting, repeated test activity to lower cost, manufacturing oriented testers. Their intent is to improve the human repeatability. However, boring is boring, and virtually anyone will get tired of running things repeatedly and begin to make more and more mistakes. Particularly if working overtime is part of the equation. While this is the softest of the hard ROI measures, it can be quite significant on larger scale systems. Automation entirely resolves this problem. Once a test case is effectively reviewed and automated, it will execute the same steps time after time. This is why one of the factors for selecting automation candidates should be the repetition requirements. Conclusion This article intentionally leaves the baseline capture, performance planning and savings calculations as an exercise to the reader. Shaun Bradshaw authored an article, Automation Application Tests to Achieve Maximum Benefit (ST&P. November 2006) that can serve as a great example for these details. As part of the Hard ROI calculations, the organization will obviously have to offset any savings against the cost of initiating the automation program. That is why a complete budget analysis is recommended as part of the climate assessment (see Part 1 of this article set). The organization can factor those budget analysis findings clearly against the savings to determine ROI potential. The "soft" ROI goals for the automation effort will be the subject of the third and final article in this series. About the Author Bob Galen is the author of Software Endgames (Dorset House, 2005). He is also the founder of The RGalen Consulting Group, a technical consulting company that focuses on agility and pragmatism. Bob has over 25 years of experience as a software developer, tester, project manager and leader. |
| Back Top |
Subscribe in a reader 
Follow us on Twitter 
Find us on Facebook 
Read our Blog 
Read our Whitepapers 
Get monthly Newsletter