Mutation Testing for Industrial Robotic Systems (FMAS 2025)

M. dos Santos, S. Hallé, F. Petrillo
Marcela Gonçalves dos Santos
Sylvain Hallé
Université du Québec à Chicoutimi
CANADA
Mutation Testing for Industrial
Robotic Systems
CRSNG
NSERC
Fábio Petrillo
École de technologie supérieure
CANADA
FMAS, September 17th 2025

Industrial robotic system
A robot is "a programmed actuated mechanism
with a degree of autonomy to perform locomotion,
manipulation or positioning" (ISO 9373).
This excludes:
Software
Voice assistants
Autonomous cars
ATMs
Smart appliances
Industrial robots are
robots for use in
automation applications in
an industrial environment.

When IRSs fail
369 robot-related accident cases reported in Korea
betwen 2010 and 2019*
*Lee, K., Shin, J. & Lim, J.-Y. (2021). Critical
Hazard Factors in the Risk Assessments of
Industrial Robots: Causal Analysis and Case
Studies. Safety and Health at Work.
IRSs are non-operational
12% of the time. This
downtime is partly due to
program faults and
sensor faults**
**Tsarouhas, P
. H. & Fourlas, G. K. (2015).
Reliability and Maintainability Analysis ofa
Robotic System for Industrial Applications: A
Case Study. International Journal
ofPerformability Engineering, 11(5), 453

Testing practices in IRSs*
There are multiple ways to assess the robustness of
an IRS...
*A. Afzal, C. L. Goues, M. Hilton, and C. S. Timperley. 2020. A Study on Challenges of
Testing Robotic Systems. ICST 2020
Logging and playback
Compliance testing
Robustness testing
Regression testing
Performance testing
Continuous integration
...and multiple challenges:
Cost and resources
Lack of oracle
Environmental complexity
Software/hardware
integration
Distrust of simulation

Testing practices in IRSs*
There are multiple ways to assess the robustness of
an IRS...
*A. Afzal, C. L. Goues, M. Hilton, and C. S. Timperley. 2020. A Study on Challenges of
Testing Robotic Systems. ICST 2020
Logging and playback
Compliance testing
Robustness testing
Regression testing
Performance testing
Continuous integration
...and multiple challenges:
Cost and resources
Lack of oracle
Environmental complexity
Software/hardware
integration
Distrust of simulation
"We mostly do field testing.
That’s what really affects what
happens. [...] Simulation just doesn’t
reflect the real world."

Simulation and testing
Advantages of simulation:
eliminates the necessity for physical
prototypes1
oﬀer a cost-eﬀective means to implement
changes2
many real-world robotics bugs can be
replicated and addressed in simulation
environments3
mitigates the risk of damaging equipment4
Roth, H., Ruehl, M. & Dueber, F. (2003). A robot simulation system for small and medium-
sized companies.
Robert, C., Sotiropoulos, T., Waeselynck, H., Guiochet, J. & Vernhes, S. (2020). The virtual
lands of Oz: testing an agribot in simulation.
Timperley, C., Afzal, A., Katz, D. S., Hernandez, J. & Le Goues, C. (2018). Crashing
Simulated Planes is Cheap: Can Simulation Detect Robotics Bugs Early?
Bossecker, E., Sousa Calepso, A., Kaiser, B., Verl, A. & Sedlmair, M. (2023). A Virtual Reality
Simulator for Timber Fabrication Tasks Using Industrial Robotic Arms.
1.
2.
3.
4.

Advantages of simulation:
eliminates the necessity for physical
prototypes1
oﬀer a cost-eﬀective means to implement
changes2
many real-world robotics bugs can be
replicated and addressed in simulation
environments3
mitigates the risk of damaging equipment4
+ ability to run a large number of scenarios(us)

Formal veriﬁcation in the physical world is
infeasible...
...but how can we assess the strength of a test
suite? ("lightweight formal method")
capability to detect faults

Formal veriﬁcation in the physical world is
infeasible...
...but how can we assess the strength of a test
suite? ("lightweight formal method")
capability to detect faults
Mutation testing
Technique consisting of applying small
modiﬁcations to a program to simulate
potential defects

Principle

Principle
original
program

Principle
mutant #1
mutant #n
.
.
.
original
program

Principle
mutant #1
mutant #n
.
.
.
TEST
SUITE
original
program

Principle
mutant #1
mutant #n
.
.
.
mutant is
killed
fail
TEST
SUITE
original
program

Principle
mutant #1
mutant #n
.
.
.
mutant is
killed
fail
mutant has
survived
pass
TEST
SUITE
original
program

Surviving mutants
A surviving mutant is indicative of a
gap in the testing coverage:
the mutant program is (likely)
incorrect, yet no test case
catches the error*
*Unless it is an equivalent mutant
Typically, new cases are added to the test suite
until the mutant is killed

Example

Example
Program
function sum(a)
local t = 0
for i = 0, |a| - 1 do
t = t + a[i]
end
return t
end
P

Example
Program
function sum(a)
local t = 0
for i = 0, |a| - 1 do
t = t + a[i]
end
return t
end
P
assert sum([0]) == 0
1
assert sum([2,0]) == 2
2
assert sum([]) == 0
3
Test cases

Example
Program
function sum(a)
local t = 0
for i = 0, |a| - 1 do
t = t + a[i]
end
return t
end
P
1
2
assert sum([]) == 0
3
Test cases
Mutant A
function sum(a)
local t = 1
for i = 0, |a| - 1 do
t = t + a[i]
end
return t
end
M

Example
Program
function sum(a)
local t = 0
for i = 0, |a| - 1 do
t = t + a[i]
end
return t
end
P
1
2
assert sum([]) == 0
3
Test cases
Mutant A
function sum(a)
local t = 1
for i = 0, |a| - 1 do
t = t + a[i]
end
return t
end
M
Mutant B
function sum(a)
local t = 0
for i = 0, |a| - 2 do
t = t + a[i]
end
return t
end
M

Creating mutants
Mutants are created by applying mutation
operators:
They are intended to represent typical errors that
could be made by a programmer.

A robot program
A typical robot program has most of its logic
embedded in string parameters:
It contains relatively few arithmetic operations and
control structures
send(“pick”)
send(“lift/5”)
if read(“color”) == “red” then
send(“turn/90”)
else
send(“turn/270”)
end
send(“drop/5”)
send(“release”)

Issue(s)
Issue #1: applying "traditional" mutation operators
often generate invalid robot commands...
send(“pick”)
send(“lift/5”)
send(“turn/90”)
else
end
send(“drop/5”)
send(“release”)

Issue(s)
Issue #1: applying "traditional" mutation operators
often generate invalid robot commands...
send(“pickl”)
send(“lift/5”)
send(“turn/90”)
else
end
send(“drop/5”)
send(“release”)
...resulting in mutants that fail every test and
provide no informative value

send(“pick”)
send(“lift/5”)
send(“turn/90”)
else
end
send(“drop/6”)
send(“release”)
Issue(s)
Issue #2a: even when mutation operators follow
correct syntax...
...small parameter changes, like oﬀ-by-one errors,
can create robot actions that cannot be executed in
the real world

send(“pick”)
send(“lift/5”)
send(“turn/90”)
else
end
send(“drop/6”)
send(“release”)
5
Issue(s)
correct syntax...
the real world

send(“pick”)
send(“lift/5”)
send(“turn/90”)
else
end
send(“drop/6”)
send(“release”)
5 6
!
Issue(s)
correct syntax...
the real world

send(“pick”)
send(“lift/5”)
send(“turn/90”)
else
end
send(“drop/5”)
send(“release”)
Issue(s)
Issue #2b: even when mutation operators follow
correct syntax...
...on the contrary, a mutation may be valid but fail
to introduce any meaningful behavioral diﬀerence

Issue(s)
Issue #2b: even when mutation operators follow
correct syntax...
...on the contrary, a mutation may be valid but fail
to introduce any meaningful behavioral diﬀerence
send(“pick”)
send(“lift/5”)
send(“turn/89”)
else
end
send(“drop/5”)
send(“release”)

The need for new operators
Either way, these mutants provide no added value
as they are unable to reveal any new gap in the
coverage achieved by the test suite.
Hence the need for...

mutation operators taking into account the fact
that a robot program is not just any piece of
procedural code

mutation operators taking into account the fact
that a robot program is not just any piece of
procedural code
but a speciﬁc type of program that manipulates
robotic components interacting with the physical
world

Mutation operators are representative of typical
errors made by programmers.
So what are the typical errors made by robot
programmers?

Confuse forward and backward
Invert left and right
Forget an instruction (e.g. pick before lift)
Apply a rotation to the wrong joint
Fail to identify non-idempotent instructions
Don't account for sensor imprecision
Exceed the technical limitations of the robot
(not exhaustive!)
Mutation operators are representative of typical
errors made by programmers.
So what are the typical errors made by robot
programmers?

turn left by X turn right by X
move forward
by X
move backward
by X
OP
operation nothing
x-axis y-axis
success failure
angular unit X angular unit Y
rad
o
angle = X angle = -X
angle = X angle = X+k
length unit X length unit Y
cm
in
distance = X distance = -X
distance = X distance = X+k

Preliminary evaluation
We used the Gazebo simulator to simulate a Gen3
robot with an end-point gripper from Robotiq
(2F-85)

The task is a (classical) pick-and-place scenario
https://www.youtube.com/watch?v=9S0R0mGKA-I
Yu Sun et al. (2018). Robotic Grasping and
Manipulation Competition: Task Pool. In Yu
Sun & JoeFalco, editors: Robotic Grasping
and Manipulation, Springer.
10.1007/978-3-319-94568-2_1

Sample of 26 generated mutants:

5 rounds of testing (to account for variation in
generated noise)
Mutation score
Round
1
2
3
4
5
%
92
85
92
88
88
+
% =
Surviving mutants were
#5, #10, #14, #20 and
#24.

Examining surviving mutants
Mutant #5
5
Inserts a redundant grip instruction
Does not prevent task
completion...
...but the extra motion
could impact the overall
mission timing

Mutant #5
5
Inserts a redundant grip instruction
Does not prevent task
completion...
...but the extra motion
could impact the overall
mission timing
Test suite is missing a non-
functional requirement

Mutant #10
1
0
Reverses the sign on position sensor on
the y-axis
The deviation remained
within the 0.02 cm
threshold
...but only because the
expected position was
close to 0

Test suite vulnerable to a translation
of the coordinate system
Mutant #10
1
0
Reverses the sign on position sensor on
the y-axis
The deviation remained
within the 0.02 cm
threshold
...but only because the
expected position was
close to 0

Mutants #14, #20, #24
2
4
1
4
Add Gaussian noise to position readings
Perhaps the position is
within the tolrance limit...
...but then these mutants
are equivalent to the
original program
2
0

Test suite partially accounts for
imprecision in sensor readings
Mutants #14, #20, #24
2
4
1
4
Add Gaussian noise to position readings
Perhaps the position is
within the tolrance limit...
...but then these mutants
are equivalent to the
original program
2
0

Take-home points
Classical mutation operators treat a robot
program just like any other program, and result
in a low mutation score*
Using domain-speciﬁc operators increases the
likelihood of generating mutants that reveal
gaps in the test coverage
Simulation is essential to run the mutants due to
their large number
*Uğur Yayan (2023): ROSMutation: Mutation Based Automated
Testing for ROS Compatible Robotic Software. Advances in
Electrical and Computer Engineering 23(3), pp. 47–56, doi:
10.4316/AECE.2023.03006.

Threats to validity
Internal validity
Results depend on the ﬁdelity of the Gazebo
simulator
External validity
Findings based on a single task (pick-and-place)
Set of mutation operators is not exhaustive
Construct validity
Uses mutation score as the measure of test
eﬀectiveness

Future work
Develop framework for ROS to automatically
generate robot-mutants
Apply methodology on other scenarios
Extend the set of mutation operators to account
for diﬀerent sensors/actuators

Thank you!
Future work
Develop framework for ROS to automatically
generate robot-mutants
Apply methodology on other scenarios
Extend the set of mutation operators to account
for diﬀerent sensors/actuators

Mutation Testing for Industrial Robotic Systems (FMAS 2025)

More Related Content

Similar to Mutation Testing for Industrial Robotic Systems (FMAS 2025)

More from Sylvain Hallé

Recently uploaded

Mutation Testing for Industrial Robotic Systems (FMAS 2025)