Testingexperience14 06 11_beersma

ISSN 1866-5705 www.testingexperience.com free digital version print version 8,00 € printed in Germany
14

© iStockphoto.com/ jgroup
The Magazine for Professional Testers

Improving the Test Process
June 2011

© Thomas Aumann - Fotolia.com

Improving automated
regression testing
by Bernd Beersma

Better quality of results and completed products
With new versions of software rapidly following each other and
business demanding shorter time to market and higher quality Because the manual execution of regression tests can be experi-
to market, the way is open for automated regression testing. Why enced as routine and monotonous by testers, it is possible that
is it, that even despite of all the advantages, test automation is errors occur in the execution and therefore in the results. This can
still not a common approach for regression testing? lead to inferior quality of the produced software. This is eliminat-
ed by automated regression testing.
In this article I will focus on automated regression testing and
an approach for successful implementing an automated test tool
The pitfalls in automated regression testing
with a scenario driven framework. This can be implemented for Many organizations see in automated testing the solution for re-
almost all kinds of automated test execution tools (commercial or ducing test time and test capacity, but when implemented in an
open source) and all types of interfaces, depending on your choice incorrect way, the opposite can be the case. Organizations who
of tool. are not familiar with test automation and had a poor, or no tool
selection project at all, often don’t realize the impact of imple-
Why automate your test execution? menting test automation in a good way. The most common ap-
There are a number of reasons to automate test execution. Here proaches are to select a tool based on some experience a tester
is a short overview of the most common reasons. has, a tool review on the web, a good sales pitch or hear saying.
One or more ‘technical’ members of the test team are selected to
Consistent test execution
implement this tool and set up test automation, besides all their
With automated test execution the human factor is taken out of other testing related work.
the equation. Despite of the tester’s good effort, it’s hard to re-
peatedly execute tests in exactly the same way, under the exact Despite of the many advantages test automation provides, many
same conditions. Chance is that not each test step is executed in organizations are not capable of implementing test automation
the way it was executed during an earlier test run. These differ- in a structured, effective way. The problems during implementa-
ences in execution can lead to errors and influence the outcome tion are often a result of poor knowledge of test automation and
of a test run. Small errors in the input can have large consequenc- the wrong use of this knowledge. Bearing this in mind, test auto-
es in a complex test scenario. mation leads to a maintenance intensive, automated ‘bug’ find-
ing solution, that costs a lot and brings little or nothing.
Better utilization of time

For testers, improving the utilization of time is probably the best Where did it go wrong? There are a few pitfalls leading to the fail-
reason for automated test execution. In an ideal world, testers ure of implementing test automation in a successful way.
can execute their newly made test script automatically straight
High maintenance
away. So they don’t have to execute it manually and start with
other (test) activities. After the automated test run, they can ana- One of the most common mistakes is taking the manual test
lyze the results and interpret the outcome, and they can run the cases and translating them directly into an automated test script
test again if needed. using record and playback. The first impression is that a lot is
achieved because now all test cases can be executed automati-
Reduce testing time is also a good motivation for automated test cally, hooray!!! However, what if objects in the System Under Test
execution. A ‘Test Robot’ can execute scheduled tests 24/7. This (SUT) change or new technology is used for a new version of the
can reduce overall testing time dramatically, but it also can in- SUT. You need to update all your test automation scripts (record
crease your test coverage. Since you can execute more extensive them again). Your test automation project becomes a software
tests in the same time, you will shorten Time to Market and/or development project on its own that may well be stopped and l
increase Quality to Market. end up as shelfware. This is the case in a lot of automation pro-

128 The Magazine for Professional Testers www.testingexperience.com

jects I have seen. So it is important to think about a structured process for test au-
tomation (keyword-driven, data-driven or scenario-driven), and
This makes this test automation approach an expensive, ineffec- to also consider how to set up a good test automation framework.
tive and time consuming one, with little or no benefits.
How to start with a structured and generic approach
Garbage in, garbage out for test automation
A common misunderstanding of test automation is the fact that
automated test scripts are of better quality than manual test In this part we will focus on creating a structured, generic frame-
scripts. Ok, by eliminating the human factor in execution of the work for test automation. In a few steps I will try and explain how
test scripts you will reduce the chance of errors through repeated you can set up an automation framework that will help you to
execution of the script during retest or regression test. But still, improve your automated testing. The first important step is:
the automated test script is as good as the tester who created
Selecting the right person for the job
it. For example, if you set up an automated test with poor cover-
age or one of poor quality, automating this test will only lead to As mentioned in one of the pitfalls listed above, you need to select
speeding up the execution and will give a poor result in less time. the right person for the job. Test automation needs a specialist
So test automation starts with setting up a proper test case de- with the right skills and knowledge. He/she needs testing experi-
sign which leads to good test cases to automate. ence, development skills, planning skills etc. You need someone
dedicated for building your test automation framework and not
Lack of knowledge and expertise
someone who is only available part-time.
A lot of organizations set up test automation as part of a test
Selecting the right tool for the job
project because they want to improve testing. They already have
a test policy and a good testing methodology and want to start There are hundreds of test tools for automating test execution.
with the next step, test automation. Often this starts with some- From simple open source web automating tools to very complex,
one who has heard of test automation as the best solution for multi-platform and multi-technology tools (tools that use record
improving quality with less effort and at lower costs. and playback, tools for scripting etc.). The most important rule to
keep in mind is to select the right tool for the job. Start with defin-
So without any knowledge of test automation, the green light ing requirements for your tool and start a tool selection project.
is given and some tester starts with test automation. Tool selec- Don’t underestimate this part of test automation; it is a crucial
tion is done by this tester and in most cases the outcome is a tool part for success.
which is already in use by development, or a tool part of a broader
Creating a different mindset
suite of tools which is already part of the organization’s software
landscape. The tester has to implement this tool alongside other We have selected one or more tools for doing the job. The next
test activities. The biggest mistake to make is to think that you can thing I want you to do is to create a different mindset about inter-
do test automation in parallel to your other activities. You need to acting with any application. Most of the people I advised on test
think of it as a project of its own. The tester has to be dedicated automation had their eyes opened after I told them the following:
to set up test automation. He /she has to have an affinity to tools,
and a technical background (former developer) can be a big plus. We want to test a chain of applications, from a web front-end to
The tester needs to know the language of testers, but also that of a mainframe back-end. We have a variety of screens to test. In the
developers. The tester doing test automation is a specialist. web front-end we create a new account for an online insurance
company. To do so we have a few screens to interact with:
If the tester is not able to focus entirely on the task of setting up
• Press the link Create new account on the Home screen
test automation, there is a good chance that test automation will
fail. What if the tester is only partially available for test automa- • Fill and submit the data in screen My account information
tion and spends the other part doing regular testing activities? • Verify New account has been created
When a change in the SUT occurs, the tester has to divide the time
between two tasks, where the priority is on test automation. If Now we need to modify some information in the back-end via ter-
this is not the case due to a lack of time, the quality of testing minal emulation. To do so, we again have a few screens to interact
can degrade, and also testing time can take longer because auto- with:
mated test execution is not possible.
• Search account on the Search screen
The lack of a structured process and a test automation framework
• Modify account information and save account information
As mentioned earlier, organizations start with test automation by • Verify account information has been changed
selecting a tool and then trying to automate as many test scripts
as possible. Unfortunately, not much thought is given on how to We just created an account in a web front-end and changed in-
set up test automation in a way that it is easy, maintainable and formation in the back-end. We identified several screens, also
scalable. After a while these organizations conclude that hun- two different technologies, but is there a difference for us? No,
dreds or even thousands of test scripts have been made, without because it makes no difference, a screen is a screen independent
thinking about the use of external test data, re-usable tests or from technology or environment. This is true for most of the ob-
setting up a generic framework for test automation. Missing this jects that are in a SUT. In almost every application we expect the
is missing out on successful test automation. Maintaining the same, we fill in the fields on a screen and do some kind of action
automation software and creating new scripts will be time con- (Enter or F4), and we arrive in a new state. This new state can be a
suming. More effort is needed from the test automation team, new screen or the existing screen that has been modified.
with higher costs but little result.

www.testingexperience.com The Magazine for Professional Testers 129

Now that we have reached this mindset that interacting with an test case which can make it complex to automate.
application, any application, is the same for us, no matter what
kind of application or technology, we can go to the next step. A possible better way is to translate the business process that
needs to be tested into separate steps. Let’s get back to the ac-
Format of a test case
count we just created. Several steps were needed to create the
If you want to start setting up test automation in a structured account:
and generic way, you have to focus on two sides, namely test case
• Press link Create new account
design on the one side and automation of these test cases on the
other. There has to be a uniform way to create new test cases, so • Fill and submit the data in screen My account information
they themselves can be a uniform input for the test automation • Verify New account has been created
scripts.
Once the steps have been identified, we translate them into a
One such way is the use of key words or action words for test case logical business flow (in most cases the Happy Flow) and create
design. You create your test step with an action word, e.g. “Press this for example in Microsoft Excel™ or Open Office Calc™. I prefer
Submit”, and in the automation software you program all steps a spreadsheet program for the overview in rows and columns and
necessary to press the submit button in the SUT. The problem for its power to manipulate data like variable dates etc.
here is that you still have a lot of different action words in your

The flow consists of different steps ,and each step represents a or environment, we can start with the automation process itself.
screen or part of a screen with input, verifications and actions.
For example: For the automation we use the selected tool for its unique ca-
pability to steer objects in the SUT. So this can be a commercial
• Fill the Name field with First Name
multi-platform tool, or an open source web automation tool, or
• Verify the default value in the Country combo box any other tool. The overall approach stays the same.
• Press Continue Creating generic interaction functions

Once we set up this vertical flow, you will see that it’s a highly So, let’s start automating test cases. As stated before, a screen
readable and understandable way of setting up your test cases. is a screen and a button is a button. Keeping this in mind you
This test case can be read by the tester, the test automation spe- can start defining your first generic interaction functions to start
cialist, but also by business users. This is a great advantage, be- with a test automation framework. Simple functions that can
cause it can be used as communication with all parties involved push a button or fill in an edit field. You can do so by recording or
in the testing process. Because of the power of the spreadsheet programming (depends on the selected tool) a single steps “Push
program, new alternative flows can be developed very quickly and Button Next” and try and make a generic function out of it like
a high coverage can be reached. “Push Button(x)”. Where Button(x) is of course any button avail-
able in the SUT.
Now that we have this uniform way of creating test cases and the
mindset that every screen is the same regardless of technology


Interacting with different types of objects
If you do this for a variety of objects that are available in the SUT,
you build a library with all kinds of these generic functions, e.g. First of all you have to have an idea of how these test tools work.
When trying to interact with a SUT, a test tool needs technical
• Push Button(x)
information on the object it wants to interact with. This steering
• Fill Edit box(x) information is crucial to the test tool. Many test tools use a repos-
• Select combo box value(x), etc. itory to store this information in, but I choose to keep it outside
of the tool. If we store this information within an XML file (ob-
Of course you need to do all kinds of verifications on objects, for ject map), we can maintain it outside the tool when something
example check the default country selected in a combo box when changes. For example, maintenance can be done by the develop-
you first access a webpage. For this you create a set of functions ment team, they know best what technical information is needed
like above, but only now for verification purposes. to steer an object.

Now you are able to automate the testing of an entire screen and So now we have all the information needed for objects to be
verify values etc., but where is the next catch? steered in this XML file, and the generic interaction functions that
need this information are also available. Within the framework
you need to build in some functionality to read the technical in-
formation from the XML file and use it in the functions that need

this information. The XML functionality will be used for other Preferably, you will want to know what happened, in what step,
parts of the framework as well (this will be explained below). on which screen and on what object on this screen. I believe the
best way is to create reporting functionality which stores the
Error handling and reporting results in an XML-file for each test step executed. Why XML? Be-
cause then you can import it to a big variety of tools available and
What if unexpected behavior occurs, or objects on your screen create custom reports.
have changed? You need to built proper error handling. Building
Final steps
functions for error handling is an important part of your test au-
tomation framework. So now almost everything is in place for a generic test automa-
tion framework which can help improve your automated regres-
At this moment let’s look back at the primary goal of automated sion testing. There are just a few things left to do:
testing: “Automation of test cases”. When we manually execute
• Get the information stored within your test case in the
the test cases we just created, we will verify them, and if some- spreadsheet program into your test automation framework;
thing is incorrect we log a defect. In the case of automated test
execution, however, what if the test executes on a system on an- • Run the test case within your automation framework.
other floor or even in another country. How do we know if an error
occurs and what defect to create. We need some kind of reporting The next big step is to read your test cases from the spreadsheet
to these things, so the next step is to build in reporting function- program, because all the information you need for the flow you
ality in your test automation framework. want to test is in the spreadsheet. There are many ways to do so,


for instance create a comma separated file, open a text stream
from the test tool and read line by line from the file. However, > biography
since we use XML for other parts of the framework, why not ex-
port the spreadsheet to an XML file. Because the functionality for Bernd Beersma
reading from an XML file was already created, we can now com- is competence leader test au-
tomation and senior test con-
bine test step information and steering information for screens/
sultant with Squerist. He has
objects etc. at runtime. over 9 years experience with
different forms of test auto-
At this moment a generic test automation framework has been mation and performance test-
created that is ready to use. ing for different companies.
Bernd holds a bachelor de-
Conclusion
gree in software engineering
Can we avoid the pitfalls mentioned earlier? First of all the main- and became acquainted with
tenance issue. By using a framework as described in this article software testing during this
period.
you can reduce maintenance, mostly because you only need to
During his numerous customer assignments Bernd creat-
identify new or modified screens/objects and add or change the ed different frameworks and solutions for test automation
technical information in the object map. with a broad variety of tools. These different approaches
led to creating a generic approach for test automation.
The second pitfall, garbage in is garbage out. Because of the way As a result of his knowledge about test automation, he also
test cases are created in the spreadsheet, you always will get the gives advice on how to implement test automation and
same input for your test automation. This reduces the amount of does workshops and trainings.
garbage in, because there is less room for errors or own interpre- Bernd is a member of the TestNet Test Automation work-
tation of test cases. Ok, spelling errors or faulty test data are still group and co-initiator of the Test Automation Day.
His goal is to keep on learning and thus improving his skills
a risk, because the test cases are as good as the tester creating
on test automation.
them.

Third, Lack of knowledge and expertise. This one is very important,
you need to have the right person and the right tool for the job.
Skilled test automation specialists are a must, but also make sure
you have a good tool selection process. Your framework depends
heavily on both.

The last pitfall is of course the one about the lack of a structured
process and a test automation framework. If you take all of the
above into account and you have the knowledge, time and the
right tools, you are able to create a structured process and a test
automation framework.

You and your test organization are now ready to experience all
the advantages that good test automation can provide, and in do-
ing so your regression testing will improve.


Testingexperience14 06 11_beersma

More Related Content

What's hot

Viewers also liked

Similar to Testingexperience14 06 11_beersma

Testingexperience14 06 11_beersma