By Bob Galen, Founder RGalen Consulting Group, LLC Introduction In the first article in this series we explored the anti-patterns that impede good test case selection. The second article examined general selection patterns that should normally be used to govern thinking around selecting good test cases. In this the last article in the series, we examine prioritization and criteria adjustment factors to allow you to be nimble over time. Prioritization and Criteria Adjustment Factors One of the challenges associated with selection criteria is that they don't remain constant, or at least they shouldn't. All sorts of conditions change within your automation scheme and projects that should cause you to pause, reflect, and adjust your criteria. Here I'll simply list a few of the primary conditions that usually cause me to reflect and potentially change selection criteria. While I'm certain there are others, these will certainly illustrate the point for criteria adjustment.
- Changes in Skill Set - Either positive or negative can certainly change your criteria. If your capabilities increase, you can and should take on more with each project and automation iteration. If they decrease, then you must take on less, which places more importance on what you select.
There are many factors that can influence this, attrition, outsourcing, changing project priorities, and changing tool sets come to mind as frequent drivers. The real point is to closely monitor your performance capabilities and adjust as they do. - Changes in Application Technologies - Quite often developers adopt new technologies that clearly disrupt ongoing automation efforts. If you're lucky, you get an early warning so you can experiment to ascertain the impact prior to product release. However, much of the time you'll become aware of automation implications at the point of implementation.
If I anticipate technology changes that will impact automation tools or strategies, I often plan for a small, impact evaluation iteration. Within it I look to fully understand the impact of the changes, evaluate the impact to my tools and infrastructure, and carefully estimate the change impact to my existing set of automation. Once I have a good feel for the full impact, I'll adjust my plans and criteria as required. - Changes in Team Balance - I often like to think in terms of "producers" (developers) and "consumers" (testers) when planning my testing and automation efforts. That's why developer to tester ratios are so important for capacity and workflow planning. If you get a drastic change in your ratio, then it will certainly have an effect on your automation capacity. Again, as in the case for Changing Skill Set, I want to reevaluate my prioritization to ensure that I'm selecting the more relevant candidates.
If you don't have a dedicated automation team this becomes an ongoing challenge as you multi-task and reallocate automation resources towards other project-driven testing activity. - Ongoing Setup Time - The time to setup and teardown your automation run-time environment can be a significant factor in how you select automation candidates. This becomes particularly important in some database applications that require vast amounts of time sensitive data in order to physically setup and run the automation. Or any automation candidates that simply requires exorbitant time to establish an initial state.
I usually prefer to attack time sensitive test candidates quite early in my automation efforts as a risk mitigation strategy. Often, I don't understand the full implications of the setup nor the impact it will have on my scheduling other, much shorter, automation. - Evolution of Maintenance Costs - I've found that automation, depending upon how it's architected, can be particularly sensitive to maintenance costs. Part of this burden is naturally driven when you try and implement too early - before interfaces or functionally within the application have stabilized. Another aspect is internally driven, for example when you make infrastructural changes or upgrade versions of your automation tool-set.
One reality of maintenance costs is that you need to pay attention to detect them and address them immediately. However, there is good news in that they are frequently cyclical in nature so that once they are addressed you can fallback to your normal selection scheme. - Evolution of Development Methodology - Clearly if your organizations SDLC or methods change it will probably impact your automation efforts for an overview of the factors. The most drastic case of this resides in the Agile Methodologies. In these cases, you'll want to shift your automation focus towards each of the development iterations, while helping to automate the unit and acceptance testing activity within the context of each.
In addition, you'll want to target automating candidates as they "exit" the iteration. However, there can be tremendous volatility across the feature and interface set, so you'll want to carefully consider the stability of each feature or candidate before implementation. - Changing Business Conditions - This can take two primary perspectives, budget and investment or changing business value and requirements. Often it has the same effect as changing skill sets or technology in that it causes you to change your view to automation selection either positively or negatively.
- Retirement Consideration - Just as software incurs technical debt over time, so do your automation efforts as test cases can lose their relevancy and connection to the current application. Or what was a high priority candidate is now something that needs to be executed infrequently or not at all. As part of my changing selection criteria planning, I usually drive reevaluation of automation from a retirement perspective.
Retirement can imply, just that, a removal of an automated test case from the execution environment or a reduction in the frequency of execution. In either case, you should formally note the change and, if removing it, move it to a retired automation archive. My regular interval for reevaluating my automation environment and selection criteria is either on a quarterly basis or before starting a significant new project. That way I cyclically evaluate the landscape and consider any important changes that might influence an adjustment of my overall selection criteria. Wrap-up The overall focus of this article was to influence your planning and strategies surrounding the selection of automation candidates. A few of my early reviewers reacted badly to all of the factors - concerned that considering all of them would bog down the automation process with overbearing planning and simply take too long. However, my experience is just the opposite. I've found that when I develop a selection criteria that maps to my overall automation strategy that I actually develop more automation. It also has greater impact on the business and projects, while increasing the visibility of my testing team in the overall product development lifecycle. One final point, don't ever think that your automation efforts are done. It's an ongoing challenge and I think you need to stay on top of new tools, techniques, maintenance, products, etc. Viewing it as a short term exercise with a succinct end point is certainly not the right mental model. About the Author Bob Galen is the author of Software Endgames (Dorset House, 2005). He is also the founder of The RGalen Consulting Group, a technical consulting company that focuses on agility and pragmatism. Bob has over 25 years of experience as a software developer, tester, project manager and leader. Other Articles by this Author |
Subscribe in a reader 
Follow us on Twitter 
Find us on Facebook 
Read our Blog 
Read our Whitepapers 
Get monthly Newsletter