Post on 06-Apr-2018
8/3/2019 FSM Testing
1/52
Model Based testing:
FSM-based Testing
Instructor: Rachida DssouliEmail: dssouli@ciise.concordia.ca
Office: EV 007.648
URL: http://www.ciise.concordia.ca/~dssouli
October, 2007
8/3/2019 FSM Testing
2/52
8/3/2019 FSM Testing
3/52
FSM
S1 S2
S4 S3
t1: 1/1
t2: 2/2t4: 2/2
t3:1/1
t6: 2/2
t7: 1/2
t8: 2/2
t5: 1/2
S1 is an initial state
Is a transition
it has a starting state
S1,
and an ending state S2
Its label is t1
The input is 1 and an
output 1
/ separates the input
from the output
T1: 1/1
8/3/2019 FSM Testing
4/52
: Ds --> S: Ds --> Y
Mealy Machine
stateset
initialstate
M = < S, S1, X, Y, Ds, , >
inputset
outputset
spec.domain
transferfunction
outputfunction
Ds S x X
partially defined (specified), deterministic, initialized
S = {S1, S2, S3, S4}
X = {1, 2}
Y = {1, 2}Ds = S x X - {}
S1 S2
S4 S3
t1: 1/1
t2: 2/2
t4: 2/2t3:1/1
t6: 2/2
t7: 1/2
t8: 2/2
t5: 1/?
?
An FSM ExampleAn FSM Example
8/3/2019 FSM Testing
5/52
1) Output fault: point a in FSM fault model.
2) Transfer fault: point b in FSM fault model.
3) Transfer fault with additional states: point c in FSM fault
model.4) Additional or missing transitions: point d in FSM fault model.
5) additional or missing states
Fault Model for Finite State Machine (FSM)Fault Model for Finite State Machine (FSM)
8/3/2019 FSM Testing
6/52
Specification
S1 S2
S4 S3
t1: 1/1
t2: 2/2t4: 2/2
t3:1/1
t6: 2/2
t7: 1/2
t8: 2/2
t5: 1/2
S1 S2
S4 S3
t1: 1/2
t2: 2/2t4: 2/2
t3:1/1
t6: 2/2
t7: 1/2
t8: 2/2
t5: 1/2
Output Fault on transition t1
Implementation under test
IUT
8/3/2019 FSM Testing
7/52
S1 S2
S4 S3
t1: 1/1
t2: 2/2 t4: 2/2t3:1/1
t6: 2/2
t7: 1/2
t8: 2/2
t5: 1/2
S1 S2
S4 S3
t1: 1/1
t2: 2/2t4: 2/2
t3
:1/1
t6: 2/2
t7: 1/2
t8: 2/2
t5: 1/2
Transfer fault on t2The ending state is now S3
Specification IUT
8/3/2019 FSM Testing
8/52
S1 S2
S4 S3
t1: 1/1
t2: 2/2
t4: 2/2t3:1/1
t6: 2/2
t7
: 1/2
t8: 2/2
t5: 1/?
?
S1 S2
S4 S3
t1: 1/1
t2: 2/2 t4: 2/2t3:1/1
t6: 2/2
t7: 1/2
t8: 2/2
t5: 1/2
Transfer fault on t5 with
Additional state
Specification IUT
8/3/2019 FSM Testing
9/52
Example of implementation with additional
state
Example of implementation with additional
state
S1
S2
S0
b/f
a/e
a/f
c/f
b/fc/ec/e
b/e
a/f
I0
b/f
a/e
a/f
c/e
b/e
I1
I2
b/fc/e
a/f
c/
c/f
a/f
I 1
I 2
I o
b/f
a/e
a/f
b/f
c/ec/e
b/e
I 3
a/e
b/f
c/e
Specification Impl. 1
Impl. 2
8/3/2019 FSM Testing
10/52
Example of a test suiteExample of a test suite
S1 S2
S4 S3
t1: 1/1
t2: 2/2t4: 2/2
t3:1/1
t6: 2/2
t7: 1/2
t8: 2/2
TS = { r.1.1.2.1, r.2.2.1.2.2}
A test suite is a set of inputsequences starting from the
initial state of the machine
r.1.1.2.1
r.2.2.1.2.2
Test Case MS
1.1.2.2
2.2.1.2.2
MIMI
1.1.2.2
2.2.1.2.2
1.1.2.2
2.2.2.2.2
Conforming Non-conforming
Pass TS Fail to pass TS
8/3/2019 FSM Testing
11/52
Possible changes made by a developerPossible changes made by a developer
Type 1: change the tail state of a transition
Type 2: change the output of a transition
Type 3: add a transition; and Type 4: add an extra state.
S1 S2
S4 S3
t1: 1/1
t2: 2/2t4: 2/2
t3:1/1
t6: 2/2
t7: 1/2
t8: 2/2
t5: 1/?
?
No limitation on the numberof such changes allows for
an infinite set of possible
implementations !!!
8/3/2019 FSM Testing
12/52
Fault model for FSM specificationsFault model for FSM specifications
For the given transition: change the output (output fault) change the next state (transfer fault)if a new state can be added, then
assume an upper bound on thenumber of states in implementations.
For the example above, there are (SxO)SxI = 4x74x5=2820 mutantswith up to 4 states. Among them, 36 mutants represent single(output or transfer) faults, as only 9 transitions are specified.
An example of a very specific fault domain: Only the transitions
related to data transfer may be faulty. These are 4 transitions.This results in only 284 mutants (faulty implementations in
mplf).s3 s4
DT1/IDATind,AK1
DT0/IDATind,AK0
DT0/AK0 DT1/AK1
mutations
s1
IDISreq/DR
CR/ICONinds3s2 s4
ICONresp/CCDT1/IDATind,AK1
DT0/IDATind,AK0
DT0/AK0 DT1/AK1
IDISreq/DRIDISreq/DR
8/3/2019 FSM Testing
13/52
S1 S2
S4 S3
t1: 1/1
t2: 2/2t4: 2/2
t3:1/1
t6: 2/2
t7: 1/2
t8: 2/2
S1 S2
S4 S3
t1: 1/1
t2: 2/2t4: 2/2
t3:1/1
t6: 2/1
t7: 1/2
t8: 2/2
t5: 1/2
S1 S2
S4 S3
t1: 1/1
t2: 2/2t4: 2/2
t3:1/1
t6: 2/2
t7: 1/2
t8: 2/2
t5: 1/2
8/3/2019 FSM Testing
14/52
Test Derivation Methods
8/3/2019 FSM Testing
15/52
Transition tour [Nait 81]
For a given FSM S, a transition tour is a sequence whichtakes the FSM S from the initial state, traverses everytransition at least once, and returns to the initial state .
Detects all output errors, There is no guarantee that all transfer errors can bedetected.
Fault detection power
8/3/2019 FSM Testing
16/52
a/x
b/x
b/y
a/x
a/y
b/y
1
2
3
a/x
b/x
b/y
a/x
a/y
b/x
1
2
3
a/x
b/x
b/y
a/x
a/y
b/y
1
2
3
The specification S
A transition tour is :a.a.a.b.b.b
The implementation I1 contains anoutput error. Our transition tour will
detect it.
The implementation I2 contains a transfererror. Our transition tour will not detect it.
SI1
I2
Transition Tour ExampleTransition Tour Example
8/3/2019 FSM Testing
17/52
An input sequence is a distinguishing sequence (DS )
for an FSM S, if the output produced by the FSM S isdifferent when the input sequence is applied to eachdifferent state. A DS is used as a state identificationsequence.
Detects all output errors, Detects all transfer errors, ADS may not be found for a given FSM.
DS-method [Gonenc 70]
8/3/2019 FSM Testing
18/52
DS method Example
a/x
b/x
b/y
a/x
a/y
b/y
1
2
3
The specification S
S
A distinguishing sequence is :b.b
If we apply it from : state 1 we obtain y.y state 2 we obtain y.x state 3 we obtain x.y
a/x
b/x
b/ya/x
a/y
b/y
1
2
3
I2
A test case which allow thedetection of the transfer error is :
a.b.b.b
If we apply it from the initial state of : the specification we obtain x.x.y.y the implementation we obtain x.x.x.x
Impl.
8/3/2019 FSM Testing
19/52
DS method
a/x
b/x
b/y
a/x
a/y
b/y
1
2
3
Phase 1: Identification of all states/ State cover
From state 1, we can reach state 2 with b/y
and state 3 with a/x
We assume that the reset exist,
Q = { , a, b}
DS = b.b
Test suite = {r.b.b, r.a.b.b, r.b.b.b}
Phase 2, to cover all transitions for output faults
and transfer faults
P = { , a, b, a.b, a.a, b.b, b.a}
Test suite:{r.b.b, r.a.b.b, r.b.b.b, r.a.b.b.b, r.a.a.b.b,
r.b.b.b.b, r.b.a.b.b}
8/3/2019 FSM Testing
20/52
General methodology for state identification based methods
A) Test generation based on Specification
A-1) Find the Q set or the State cover: minimal inputs that
reach a state from the initial one
A-2) Find the P set or Transition cover: that will cover all remaining transitions
Generate Test Suites using Q and P sets
B) Fault detection
B-1) Apply the generated test suites to the specification to obtain Expected Outputs
B-2) Apply the generated test suites to the implementation to obtain Observed Outputs
Compare the expected and observed outputs (test results)
If they are different then the verdict is fail otherwise it is a pass for the applied test suites.
8/3/2019 FSM Testing
21/52
The test cases are :
state 1:
state 3 :
state 2 :
Test case structure:
preamble.tested transition.state identification
a.b.bb.b.ba.a.b.b
a.b.b.bb.a.b.bb.b.b.b
DS method Example
8/3/2019 FSM Testing
22/52
The UIO-method can be applied if for each state of thespecification, there is an input sequence such that theoutput produced by the machine, when it is initially in thegiven state, is different than that of all other states.
The UIOv-method is a variant of the UIO-method. it checkthe uniqueness of the applied identification sequences onthe implementation, meaning that each identification
sequence must be applied on each state of theimplementation and the outputs are compared with thoseexpected from the specification.
UIO-Method [Sabnani 88]
and UIOv-Method [Vuong 89]
8/3/2019 FSM Testing
23/52
a/x
b/x
b/ya/x
a/y
b/y
1
2
3
The specification S
SUIO sequences are :
state 1 : a.b state 2 : a.a state 3 : a
We assume the existence of areset transition with no output(r/-) leading to the initial statefor every state of S
A transition cover set is :P={e, a, a.b, a.a, b, b.a, b.b}
The test sequences generatedby the UIO-method are :
r.a.b, r.a.a, r.a.b.a.b, r.a.a.a.a,r.b.a.a, r.b.a.a.b, r.b.b.a
UIO ExampleUIO Example
8/3/2019 FSM Testing
24/52
The W-method involves two sets of input sequences : W-set is a characteristic set of the minimal FSM, andconsists of input sequences that can distinguish
between the behaviors of every pair of states P-set is a set of input sequences such that for eachtransition from state A to state B on input x, there areinput sequences p and p.x in P such that p takes theFSM from the initial state into state A.
Method W[Chow 78]
8/3/2019 FSM Testing
25/52
a/e
a/f
b/f
b/e
c/eb/f
c/e
c/f
a/f
1
3
2
The specification S
We assume the existence of areset transition with no output(r/-) leading to the initial statefor every state of S
A characterization set is W={a, b}W1 state 1 : a/e,W2 state 2 : a/f, b/f W3 state 3 : b/e
W = Union of all Wi
A transition cover set for the specificationS is :
P={e, a, b, c, b.a, b.b, b.c, c.a, c.b, c.c}
The W-method generates the
following test sequences: (P.W) =r.a, r.b, r.a.a, r.a.b, r.b.a, r.b.b, r.c.a,r.c.b, r.b.a.a, r.b.a.b, r.b.b.a, r.b.b.b,r.b.c.a, r.b.c.b, r.c.a.a, r.c.a.b, r.c.b.a,r.c.b.b, r.c.c.a, r.c.c.b
W method Example
8/3/2019 FSM Testing
26/52
This method is a generalization of the UIOv method which isalways applicable. It is as the same time an optimization ofthe W-method. The main advantage of the Wp-method, overthe W-method, is to reduce the length of the test suite.
Instead of using the set W to check each reached state si,only a subset of W is used in certain cases. This subset Widepends on the reached state si, and is called an identificationset for the state si.
Wp method [Fujiwara 90]
8/3/2019 FSM Testing
27/52
The specification S
A characterization set is W={a, b} for W method for state 1 : a/e for state 2 : a/f, b/f for state 3 : b/e
We assume the existence of a
reset transition with no output(r/-) leading to the initial statefor every state of S
The identification sets are : W1={a}, distinguishes the state 1 fromall other states W2={a, b}, distinguishes the state 2
from all other states W3={b}, distinguishes the state 3from all other states
a/e
a/f
b/f
b/e
c/eb/f
c/e
c/f
a/f
1
3
2
Example of Wp method (1/3 )Example of Wp method (1/3 )
effb
ffea321state
Derivation of W
8/3/2019 FSM Testing
28/52
A state cover set for the specification S is : Q={, b, c}
A transition cover set for the specification S is :P={, a, b, b.c, b.a, b.b, c, c.a, c.c, c.b}
P-Q={a, b.c, b.a, b.b, c.a, c.c, c.b}
Based on these sets, the Wp-method yields the following testsequences :
Phase 1: Q.Wi = {r.a1, r.b.a2, r.b.b2, r.c.b3}The ending state Wi is given in subscript
Phase 2 : (P-Q).Wi ={r.a.a2, r.a.b2, r.b.c.a2, r.b.c.b2, r.b.a.a1,r.b.b.b3, r.c.a.b3, r.c.c.a2, r.c.c.b2, r.c.b.a1}
Example of Wp method (2/3)Example of Wp method (2/3)
W 1 : { a/e } , W 2 : { a/f, b/f } , W 3 : { b/e }
8/3/2019 FSM Testing
29/52
a/e
a/f
b/f
b/e
c/eb/f
c/e
c/f
a/f
1 2
3
A faulty implementationI
I contains a transfer error 2-
a/f->1 (fat arrow) instead of2-a/f->2 as defined in thespecification S
The application of the test sequencesobtained in Phase 2 leads to thefollowing sequences of outputs :
e.f, e.f, f.f.f, f.f.f, f.f.e, f.f.e, e.f.f, e.e.f,e.e.f, e.e.e
The output printed in bigger size isdifferent from the one expectedaccording to the specification. Therefore,the transfer error in the implementation
is detected by this test sequence.
Example of Wp method (3/3)Example of Wp method (3/3)
8/3/2019 FSM Testing
30/52
Test derivation based on FSM (Resum)Test derivation based on FSM (Resum)
Transition tour
guaranteed coverage only for output faults
Methods using state identification
with coverage guarantee for output and transfer faults. Three cases:
number of states same for implementation I and specification S
number of states for I possibly larger than for S, but bounded
coverage only for a selected set of transitions (fault function)
Methods without coverage guarantee
Hand made test suite without test derivation procedure
Single long test sequence vs. set of shorter test cases (e.g. test case forspecific transition, test purpose)
Usually, each test case requires reset to initial state; correct resetassumption
8/3/2019 FSM Testing
31/52
Transition Tour example
Transition tourTT: t1, t4, t3, t9, t2, t3, t6, t7, t8
TT (input/expected output): a/1.b/2.a/1.a/2.b/2.a/1.b/2.a/2.b/2
S1 S2
S4 S3
t1: a/1
t2: b/2t4: b/2
t3:a/1
t6: b/2
t7: a/2
t8: b/2t9
: a/2
Test hypothesis: Initially connected machine
8/3/2019 FSM Testing
32/52
All state identification Methods
Distinguishing Sequence, UIO, W
Test hypothesis
H1) Strongly connected machine
H2) Contain no equivalent states
H3) deterministic
H4) Completely specified machine
H5) the failure which increases the number of states doesnt occur
The method is applied in two phases from the initial state
phase 1) -sequence to check that each state defined by the specification also
exist in the implementation.
phase 2) -sequence to check all the individual transitions in the specification for
correct output and transfer in the implementation.
8/3/2019 FSM Testing
33/52
8/3/2019 FSM Testing
34/52
W method
Assume that the reset exist and it brings the machine from any state to the initial state.
a) Find characterization set W and generate the set of test cases for the specification S
using the W method.
b) Does S have a DS sequence? If not explain why?
a/0
b/0
b/1a/1
a/0
S0
S2
S1
b/0
8/3/2019 FSM Testing
35/52
8/3/2019 FSM Testing
36/52
S1
S0
S2
a/0
b/1
a/0b/0
a/1
b/0
S0State
Input
Output
a
0
S1 S2
a a
1 0
S0 S1 S2
b b b
1 0 0
S0 S1 S2
a.b a.b a.b
0.1 1.0 0.0
Derive a DS of length up to 2 for S
a.b is a DS for S
Specification S
Comment: a as input at each state will loop on the state, sequence of a.a. cannot be a DS, the output will
be 0.0.. or 1.1
Transition tour:
Input
Output
a.b.a.b.a.b
0.1.0.0.1.0
Examples Suite
Q set: permits to reach each state
8/3/2019 FSM Testing
37/52
Q set: permits to reach each state
from the initial state
Q = { , b,b.b}
The first b to reach the state S2
b.b to reach the state S1.
P set is transition cover, permits to execute
each transition at least one starting from the
initial stateS1
S0
S2
a/0
b/1
a/0b/0
a/1
b/0
S0
a
bb
a
b
b
b
b
b
b
b
a
How to derive P set: find allPath starting from the size1 and up and each transition
should be traversed at least once
P = {, a, b, b.a, b.b, b.b.a, b.b.b}
more than one p set may exist, this depends on the alternative
paths that the automata may have.
8/3/2019 FSM Testing
38/52
The goal of the Phase 1 is identification of
the states in the implementationDS = a.b, Q = { , b,b.b}, P = {, a, b, b.a, b.b, b.b.a, b.b.b}
Phase 1
Q.DS = {r.a.b, r.b.a.b, r.b.b.a.b} Expected output of phase 1is:
{-.0.1, -.1.0.0, -.1.0.1.0}
Phase 2 ( DS in bold)
P.DS= {r.a.b, r.a.a.b,r.b.a.b, r.b.a.a.b, r.b.b.a.b, r.b.b.a.a.b, r.b.b.b.a.b}
{-,0.1, -.0.0.1, -.1.0.0.0, -.1.0.1.0, -.1.0.1.1.0, -.1.0.0.0.1}
S1
S0
S2
a/0
b/1
a/0b/0
a/1
b/0
Note that, the test suites for phase 1 and 2 should be
Derived from the specification and applied to the
implementation to check it for output and
transfer faults.
8/3/2019 FSM Testing
39/52
S1
S0
S2
a/0
b/1
a/0b/0
a/1
b/0
S1
S0
S2
a/0
b/1
a/0
b/0
a/1
b/0
Specification SImplementation I
Apply the transition tour to the implementation I and comment
Transition tour applied to S
Input
Output of S
Output of I
a.b.a.b.a.b
0.1.0.0.1.0
0.1.0.0.1.0
The implementation I has a transfer fault,
the fault is not detected byTransition tour.
Transition tour detects all output faults but
Doesnt guarantee the detection of transfer faults
8/3/2019 FSM Testing
40/52
8/3/2019 FSM Testing
41/52
S2
S0
S1
a/0
b/0
a/0b/0
c/0
a/1
C/0
b/0
c/1
Specification S
State
Input
Output
S0 S1 S2 S0 S1 S2 S0 S1 S2 S0 S1 S2
a a a b b b c c c a.c a.c a.c
0 0 1 0 0 0 1 0 0 0.1 0.0 1.1
Derive a UIO sequence for S
UIO state S0 = c/1
UIO state S2 = a/1
UIO state S1 = a/0.c/0
Transition tour for S
a.b.a.b.c.a.c.b.c
0.0.0.0.0.1.1.0.0
8/3/2019 FSM Testing
42/52
8/3/2019 FSM Testing
43/52
completeness: completely specified or partially specified
connectedness: strongly connected or initialy connected
reducibility: reduced or non-reduced
determinism: deterministic or non-deterministic
13
Assumptions about specificationsAssumptions about specifications
8/3/2019 FSM Testing
44/52
Assumptions about implementationsAssumptions about implementations
t7: 1/2
S1 S2
S4 S3
t1: 1/1
t2: 2/2t4: 2/2t3:1/1
t6: 2/2
t8: 2/2
r/-
r/-
r/-
r/-
Deterministic
Completely defined
react to any input
Limited extra states
Reliable reset
not necessary
15
R l it t ti ti
8/3/2019 FSM Testing
45/52
Regularity, a testing assumptionRegularity, a testing assumption
This type of assumption allows to limit testing to a finite set of behaviors in thecase of systems that exhibit an infinite behaviors. Examples are
programs (or specifications) with loops and integer input and outputparameters
finite state machines reactive systems, en general
Principle: assume that the implementation has a regular behavior, whichmeans that the number of control states of the implementation is limited.
If the number of states is not bigger than the corresponding numberof states of the specification, then all loops (of the specification) haveto be tested only once.
This is the idea behind the FSM fault model where the number of
implementation states is limited to n, or to some number m > n. This is also the idea behind certain approaches for testing
program loops and for testing in respect to specifications in theform of abstract data types.
Independency, a testing assumption
Independency, a testing assumption
8/3/2019 FSM Testing
46/52
Independency, a testing assumptionp y g p
Principle:
The different submodules of the system under test are
independent, and faults in one module do not affect the possibilityof detecting the faults in the other modules.
This is a controversial assumption:
In most complex systems, modules or components are dependent.The reasons are:
they share resources (e.g. memory)
they have explicit interactions
Example:
several connections supported by a protocol entity
test only one connection in detail (it is independent of the
others) the others need not be tested, since they are all equal
(uniformity assumption, see below)
8/3/2019 FSM Testing
47/52
Independency (suite)Independency (suite)
The independency relation is a reasonable assumption in certain cases.
Example:
Equipment to test
Entity N Entity NEntity N
Entity N+1
SAPSAP SAP
Uniformity, a testing assumption
Uniformity, a testing assumption
8/3/2019 FSM Testing
48/52
y, g p
Uniformity assumption / Congruence
Origin: Partition Testing [Weyuker 91]
Principle There exist similar behaviors. If they are grouped under an
equivalence relation, then it is sufficient to test one behavior ofeach equivalence class for conformance testing.
Special cases:
Principle of partition testing: Apply test for at least onerepresentative for each partition of the input domain (software
testing, EFSM testing)
Equivalent actions for EFSM Equivalent states for FSM
F i i t t d t i i
8/3/2019 FSM Testing
49/52
Fairness in respect to non-determinismFairness in respect to non-determinism
Many systems have a non-deterministic nature. In particular, theparallelism of distributed systems introduces many possible interleaving
of individual actions within the different system components.
The assumption is that all the execution paths effectively realizedduring testing cover all paths that are pertinent for detecting thepossible implementation faults.
a/1
a/2
a/4
s1
s2
s3s4
non-determinism
Partially defined FSMs
Partially defined FSMs
8/3/2019 FSM Testing
50/52
Partially defined FSM sPartially defined FSM s
Non-specified transitions need not be tested. However different
interpretations of undefinedness have an impact on testing:
completeness assumption
non-specified transition is implicitly defined, e.g. stay in same
state (as in SDL), or go to an error state
methods for completely defined FSMs may be applied, however,test will rely on implied transitions
dont care
no specific behavior is specified non-specified transitions must be avoided by test cases
robustness tests may be applied to check the reaction of theimplementation for non-specified situations
forbidden not possible to invoke non-specified transitions
8/3/2019 FSM Testing
51/52
Fault Coverage Evaluation
M th d f F lt C E l ti
Methods for Fault Coverage Evaluation
8/3/2019 FSM Testing
52/52
Methods for Fault Coverage EvaluationMethods for Fault Coverage Evaluation
The definition of fault coverage always depends on fault model!
Exhaustive mutation analysis
Monte-Carlo simulation method
Deciding completeness
minimize an FSM which is given in the form of the TS, if its minimalform is equivalent to the given FSM then TS is complete (the max #states is assumed), otherwise it is not complete [see Yao]
Structural Analysis
it evaluates the fault coverage of a given test suite by directly
analyzing the test suite against the given FSM. Count the number ofstates distinguished and transitions checked by the test suite. Anumeric measure easy to evaluate (linear complexity) [see Yao]
Different possible measures
compare number of implementations (common approach)
compare the log of number of implementations (corresponds tocounting transitions covered) [called order coverage by Yao]