Skip to main content
Università degli Studi dell’Aquila




L22. Model-based Testing (Principles)
                                    Henry Muccini
                  DISIM, University of L’Aquila
 www.henrymuccini.com, henry.muccini@univaq.it
Copyright Notice
The material in these slides may be freely reproduced
and distributed, partially or totally, as far as an explicit
reference or acknowledge to the material author is
preserved.



                           Henry Muccini
Agenda
Modeling


MBT:
 →Why?

 →Modelsfor MBT
 →UML-based Testing
    ─ Testing activities
AGENDA
Modeling
                               Model-based testing

           Why?
                                   Models for MBT

           UML-based Testing
Modeling: there is not just programming

What is a model:
 →A model is a simplification of the reality
 →The reality is abstracted, since it is too complex

Why modeling
 →Models  are built for a better understanding of the
  system under development
Everything is a model ?!?
Non-software Models …
Specification: What
Software Models
Other Software Models (Web models)
… other software models…
… and many others…




                         Pilot              Pilot         CoPilot           CoPilot
                     Multifunction      Multifunction   Multifunction     Multifunction
                      Display1           Display2        Display1          Display2




                      Display             Display           Display          Display
                     Processor           Processor         Processor        Processor


                                           High speed network
                                            High speed network


                            Mission             Mission          Mission
                           Processor           Processor        Processor


                                                1553 bus
                                                 1553 bus

                           Auto-Pilot          GPS            Nav Radio
Model Driven Development (MDD)
Models are becoming first core assets in
Software Engineering/Architecting:
 →models   are primary artifacts retained as first class
 entities that can be analyzed and manipulated by
 means of automated tools
 →the focus moves from writing and sharing
 documents to writing and sharing models.
From Document Driven to Model Driven
Models in MDD
Models in MDD can be used for:

 →Documenting    design decisions
 →Guiding the coding phase

 →For analyzing and validating design choices

    ─ by complementing traditional code-level
      analysis techniques
Why modeling? For Documenting and Analysis
What is MBT?
It consists in:
 →ExtractingTest Cases from the Model
 →Run the test case over the system implementation



Purpose:
 →Tovalidate the implementation conformance to the
  model
       ─ The model itself is the oracle, i.e., it represents the
        expected behavior
Model-based Testing
A model based testing approach

accepts two main inputs
 →a model of the software under test,
 →a set of test generation directives which guide the test cases
 selection
and outputs a test specification
 →which   includes a set of stimuli the tester should introduce in
  the system together with expected responses
Modeling for testing
Essentially, we need (at least) a behavioral model of
the software system:
 →Labelled  Transition System
 →StateCharts

 →UML State of Sequence Diagrams

 →IOLTS

 →Finite State Automata

 →…
The base-line technique for designing test cases
  →Timely
         ─ Often useful in refining specifications and assessing testability
            before code is written
  →Effective
         ─ finds some classes of fault (e.g., missing logic) that can elude
            other approaches
  →Widely      applicable
         ─ to any description of program behavior serving as spec
         ─ at any level of granularity from module to system testing.
  →Economical
         ─ typically less expensive to design and execute than structural
            (code-based) test cases


(C) 2007 MAURO PEZZÈ & MICHAL YOUNG
Program code is not necessary
  →Only a description of intended behavior is needed
  →Even incomplete and informal specifications can be used
         ─ Although precise, complete specifications lead to better test suites
Early functional test design has side benefits
  →Often reveals ambiguities and inconsistency in spec
  →Useful for assessing testability
         ─ And improving test schedule and budget by improving spec
  →Useful     explanation of specification
         ─ or in the extreme case (as in XP), test cases are the spec




(C) 2007 MAURO PEZZÈ & MICHAL YOUNG
Different testing strategies (functional, structural,
fault-based, model-based) are most effective for
different classes of faults
Functional testing is best for missing logic faults
  →A common problem: Some program logic was simply
  forgotten
  →Structural (code-based) testing will never focus on code that
  isn’t there!



(C) 2007 MAURO PEZZÈ & MICHAL YOUNG
Modeling for testing
Such a model can be produced:
 1. Through a formal specification language
    ─ Typically referred as formal testing


 2. Through a diagrammatic (visual) notation
    ─ Typically referred as UML-based Testing
1. MBT based on Formal Specifications
Since the 80’s, many specification-based testing
approaches have been proposed, based on formal
languages such as
 →Z,   VDM, CSP, CCS, LOTOS, SDL, and Petri Nets.
More recently:
 →Based    on dynamic behavior:
        ─ FSM-based (Bochmann&Petrenko, Lee&Yannakis, …)
        ─ LTS-based (Brinksma&Tretmans, Jard&Jeron …)

 →Focussing    on static aspects,
        ─ ADT theory (Bernot&Gaudel…)
        ─ Z-based (Hierons…)
Formal Specification-based Testing
TorX (Côte de Resyste)
   → on-the-fly test generation and execution
  → random

  → LOTOS and Promela

TGV (IRISA - Rennes)
   →derives tests in TTCN from LOTOS or SDL
  → uses test purposes

TVEDA (CNET - France Telecom)
   →   derives TTCN tests from SDL specification
2. MBT based on the UML
UML State-machine based testing:
  →which might require the translation of the UML
  diagrams into an intermediate formal description
Scenario-based testing:
  →based   on the use of opportunely annotated UML Sequence Diagram
State-based and Scenario-based:
combine the use of interaction diagrams and state diagram,
possibly annotated with further (formal) information (e.g. OCL)
Model-based Testing: the overall idea




                        From Leila Naslavsky Advancement report
AsmL
                                       L
                                   TESTO
                                       OR
                                   UMLAU
                                       UT
Model-based Testing Approaches




                                   UMLTe
                                       est
                                   AGEDI
                                       IS
                                   TOTEM
                                       M
                                   SOOT
                                      TF
                                 COW-SU
                                      UITE
                                  SCENTO
                                       OR
                                   SeDiTe
                                        eC
                                 UCSC-Sy
                                       ystem
Main Testing Activities
Test Selection/Generation:
 →it consists in selecting a suitable and finite set of test cases
  from the possibly infinite set;


Test Execution and Evaluation:
 →it consists in executing the code accordingly to the selected
  test cases and comparing real and expected results;
Test Selection
A test case, in MBT, is typically a possible
“scenario”
 →In   scenario-based testing:
       ─ A scenario itself is an abstract test case


 →In   state-based testing
       ─ Execution flows are extraced from the state machine
         and represent a test case
       ─ MB coverage criteria are applied on this model
Test Case Selection
Abstract Test case = path over the LTS/FSM,
corresponding to sequences of events
     ─ They are not “concrete” test cases, but abstract
      (depending on the specification)


 →Concrete Test cases = mapping asbtract test cases
 into execution statements to be launched
Test Case Selection
                                                               Model of the
                                                 Test            system
                                              directives
            inSD                                           1
    1
                 C1        C2

                      m9

                      m1



                                         Test
                                     specification
        outSD1

2          C1         C2        C3
                 m9
                           m3

                 m8
                           m1
Test Case Selection
Test Execution
It consists in running the test cases on the system
implementation
 →So   to compare the real execution with the expected behavior
MBT practical concerns
Traceability, i.e. relating the abstract values of the
  specification to the concrete values of the
  implementation.


Execution, i.e. forcing the Implementation Under Test
  (IUT) to execute the specific sequence of events that
  has been selected.
Testing and Industry
In order to be suitable for industrial needs a testing
approach has to emphasize the following qualities:
 →Usability
      ─ Not ad-hoc models
 →Timeliness
      ─ Soon and even incomplete
                                      Opposed to
 →Tool   support                      accuracy


Goal of a suitable testing approach should be
“to improve the test results accuracy without sensibly
raising the testing effort”
MBT Tools
There are some automated tools for MBT:
 →Telelogic Tau and Tau Tester
 →Leirios Test Designer

 →Conformiq Qtronic

 →Rhapsody Test Conductor and Automatic Test Generation

 →AGEDIS
MBT in industry: challenges and issues
As remarked by Alan Hartman in his 2006 speech at ISSTA:
 →“Only   1% of industry uses it”


Why?
Why?
The main reasons I foresee:
 →Model2Code    (and viceversa) traceability problems
 →modeling is not yet so widely recognized as part of
 the development process.
     ─ Legacy systems


 →because  of tool/service problems
 →cost/effectiveness is still under exploration
Conformiq: Two models are required in
Input


•SutModel.Java (in QML)
•SutModel.xmi (Graphical Notation)
QML
Just like Java, QML consists of variables and
expressions, all strictly typed and known before the
compilation.


The types are divided into the same three groups that
Java has: primitives, references and values
QML
QML extends Java with:
      Records
      Ports
      System block
      State machines
SutModel Graphical
SutModel Graphical
Coverage




           Figure 5 – The Coverage Editor window