Skip to main content
© Utopia Solutions
Building a Strategy for Test Automation Success
Lee Barnes, CTO
Utopia Solutions
© Utopia Solutions
Agenda
•What is the end goal?
•Why do we need a strategy?
•What do we need to consider?
•What should a strategy look like?
Test Automation Strategy Overview
Interactive Case Study
© Utopia Solutions
SECTION 1
What is Our End Goal?
© Utopia Solutions4
Test Automation Success Criteria
• Issue detection and recovery
• Accurate verification
• Unattended execution
Reliable
• Minimum sensitivity to application and test
case changes
• Test case definition separate from
automation code
Maintainable
• Test coverage expanded efficiently
• Automated test cases created by non-
automation resources
Scalable
© Utopia Solutions5
Test Automation Approach
Test Automation Framework
Automated Test Cases Test Results
Test Definition
Test Execution
Non-technical / Test Analysts & Subject Matter Experts
Technical / Automation Engineers
© Utopia Solutions
SECTION 2
Why Do We Need a Strategy?
© Utopia Solutions8
Why Test Automation Fails
Organizational
• Lack of support
• Unrealistic expectations
• Disjointed org structure
• Missing skill sets
Process
• Immature testing process
• Segregated processes
• Lack of test environment / data
control
Technical
• Poor tool support for AUT
• Under or over design automation
approach
© Utopia Solutions9
Are We Ready?
© Utopia Solutions
SECTION 3
What Do We Need to Know First?
© Utopia Solutions11
Test Automation Readiness Assessment
Org
• Structure
• Support
• Goals and
Expectations
• Resources and Skills
Process
• Test Maturity
• Test Assets
• Process Integration
• Test Environment /
Data
Tech
• AUT Technology
• AUT / Tool
Integration
Assessment
Topics
© Utopia Solutions
Where Do We Get the Information?
Assessment
Input
Stakeholders
Existing
Assets
AUTs
© Utopia Solutions
Ideal Organizational Attributes
Structure
• Centralized
testing
function
• Testing is a
peer of
development
• Automation
roles are
defined
Goals and
Expectations
• Automation is
an investment
• 100%
automation is
not likely
• Automation
requires
specialized
skills
• Automation
will free up
resources to
work on high-
value tasks
Support
• Executive
management
has visibility to
automation
• Automation
has a
dedicated
budget
Resources and
Skills
• Resources are
dedicated to
automation
• Resources are
skilled in
automation as
a process AND
technology
© Utopia Solutions
Ideal Process Attributes
Test Process
• Test objectives
are clear and
measurable
• Test cases are
defined and
documented
• Expected
results are
predictable
SDLC Integration
• AUT in a
frequent
release state
with visibility to
future roadmap
• Visibility to
system changes
as early as
possible in
release cycle
Test Environment
• Test
environment
dedicated to
automation
• Test data
controlled
© Utopia Solutions
Ideal Technology Attributes
AUT Technology
• Multiple AUTs built on
common technology
and/or development
framework
• Consistent use of UI
objects
• Consistent navigation
• Consistent presentation
of user feedback
(warnings, errors, etc.)
Tool / AUT Integration
• All UI objects supported
by tool
• Any alternative interfaces
(web services, API, DB,
etc.) supported by tool
• All target end user
environments supported
by tool
© Utopia Solutions
SECTION 4
Common Assessment Findings
© Utopia Solutions17
Organizational
Distributed
Testing /
Automation
Efforts
App
App
App
App
• Uneven automation
resource demand
• Inconsistent approach
• Duplicated efforts
• Restricted skill growth
and knowledge sharing
Leads to…
© Utopia Solutions18
Organizational
AUT A
AUT B AUT C
AUT D
Centralized
Test Automation
Team
Common Resources,
Processes &
Technology
• Flatten demand curve
• Accelerate skills
development
• Leverage reusable
knowledge and technology
Benefits
Centralize Automation
Activities
© Utopia Solutions19
Process
Analyze Design Build
Test Deploy
No ability to identify impact of app changes on
automated test suite
Unexpected /
Unpredictable
Maintenance
© Utopia Solutions20
Process
Analyze Design Build Test Deploy
Capture potential impacts as early as possible to enable
proactive maintenance
Capture App Changes
Analyze
Impact
Maintain Execute
Automation Activities
Application SDLC
© Utopia Solutions21
Environment
Shared Test Environment
Uncontrolled Test Data
Integration
Functional
Performance
Integration
Test
• Schedule conflicts
• Unreliable execution
• Unnecessary “hardening”
of the test suite
Leads to…
Automation
© Utopia Solutions22
Environment
Shared Test Environment
Controlled Test Data
Integration
Functional
Performance
Integration
Test
Dedicated Environment for
Automation
Test Data Controls
Dedicated
Environment
Controlled
Test Data
Automation
• Restore to known state
• Partition data related
to specific test
activities
• Create required data
each execution
© Utopia Solutions
SECTION 5
What Does a Strategy Look Like?
© Utopia Solutions24
Mobile Test Strategy Topics
Test
Automation
Strategy
Process &
Organization
Environment
TechnicalResources
Scope &
Roadmap
© Utopia Solutions25
Automation Strategy – Process and Organization
• Political and financial support
• Realistic expectations
• Centralized automation function
Key Organizational Components
• Test objectives and coverage goals
• Integration with SDLC → proactive
maintenance
Key Process Components
© Utopia Solutions26
Automation Strategy – Environment
• Dedicated environment for execution
• Controlled test data
• Controlled code migration
Key Environment Components
© Utopia Solutions27
Automation Strategy – Resources
• Roles
• Skills
• Time
Key People Components
• AUT test environment(s)
• Test suite development
• Test suite execution
Key Hardware Components
• Test tool
• Supporting utilities
Key Software Components
© Utopia Solutions28
Automation Strategy – Scope
• Test data setup
• AUT / environment preparation
• Complex / tedious results verification
Test Support Tasks
• Applications built on common technology
• Applications with frequent releases & friendly UI’s
Application Selection
• Frequently tested
• High test case to UI ratio
• Predictable results
• Easy to automate
Test Case Selection
© Utopia Solutions29
Automation Strategy – Roadmap
• Begin with required organizational and process changes
• Overlap with base framework and minimal coverage
• Enhance and add test coverage over small iterations
Create Implementation Timeline
Phase nPhase 1
Continuous Process & Organizational Improvements
Base Framework
Initial Coverage
AUT 1
Framework Enhancements
Expand Coverage
Initial Coverage
Phase 2
AUT 2
Expand Coverage
Expand Coverage
Initial Coverage
Phase 3
AUT 3
© Utopia Solutions
SECTION 6
Case Study
© Utopia Solutions31
Key Assessment Findings
Organization
• All enterprise testing activities
currently performed centrally
• Management believes automation
is a tool implementation
Process
• Manual tests are well structured,
but not maintained consistently
• Rely on SME’s to execute tests
Environment
• Manual regression test performed
in a isolated environment
• Test data refreshed in all
environments once per quarter
Technology
• All AUTs have browser based UI
• Developed on disparate
technologies
• Test tool supports all AUT
components
Resources
• Test organization consists of non-
technical SME’s
• System performance (mobile)
• Device performance
© Utopia Solutions32
Key Summary Points
• DO NOT automate without a clear vision and plan
• Understand your readiness from these perspectives
• Organization
• Process & Environment
• Technical
• Resources
• Implement in small phases
© Utopia Solutions33
Questions…
…. and Answers!
Direct future questions to:
Lee Barnes
Founder and CTO
Utopia Solutions, Inc.
Email: lee.barnes@utopiasolutions.com
Twitter: twitter.com/USI_LeeBarnes
LinkedIn: linkedin.com/in/leebarnes
Blog: www.RaiseYourMQ.com