Università
della
Svizzera
italiana
A Declarative Approach for
Performance Tests Execution
in Continuous Software
Development Environments
Vincenzo Ferme
Software Institute
Faculty of Informatics
USI Lugano, Switzerland
Cesare Pautasso
2
Repo
Continuous Software Development Environments
Developers,
Testers,
Architects
2
CI ServerRepo
Continuous Software Development Environments
Developers,
Testers,
Architects
2
CI ServerRepo
Continuous Software Development Environments
Developers,
Testers,
Architects
2
CI ServerRepo
Continuous Software Development Environments
Developers,
Testers,
Architects
Continuous Changes
2
CI ServerRepo
Continuous Software Development Environments
Developers,
Testers,
Architects
Continuous Changes
Continuous Test Execution
3
CI ServerRepo
Continuous Software Development Environments
Developers,
Testers,
Architects
Continuous Changes
Continuous Test Execution
Load Test, Spike Test
3
CI ServerRepo
Continuous Software Development Environments
Developers,
Testers,
Architects
Continuous Changes
Continuous Test Execution
Capacity Test
Configuration Test
Load Test, Spike Test
State of Practice
4
Script/UI Based
End-to-end Tests against REST APIs
State of Practice
4
Script/UI Based Integrated in CI Pipelines
End-to-end Tests against REST APIs
State of Practice
4
Script/UI Based Integrated in CI Pipelines
Performance Tests
Execution Automation
Google Vizier
AutoPerf
DataMill
CloudPerf
End-to-end Tests against REST APIs
Define a Configuration Test
5
Test Definition
Define a Configuration Test
5
Test Definition Configuration Parameters Change
Simulated Users
Define a Configuration Test
5
Test Definition Configuration Parameters Change
Simulated Users
System Under Test 3rd-party tools
Define a Configuration Test
5
Test Definition Configuration Parameters Change
Simulated Users
System Under Test 3rd-party tools
Test Execution, Data Collection, Metrics Computation
Define a Configuration Test
5
Test Definition Configuration Parameters Change
Simulated Users
System Under Test 3rd-party tools
Test Execution, Data Collection, Metrics Computation
Test Goals are not
Explicit in the Definition
Define a Configuration Test
5
Test Definition Configuration Parameters Change
Simulated Users
System Under Test 3rd-party tools
Test Execution, Data Collection, Metrics Computation
N
o
Sem
antic
V
alidation
(Runtim
e
Errors)
Test Goals are not
Explicit in the Definition
Define a Configuration Test
5
Test Definition Configuration Parameters Change
Simulated Users
System Under Test 3rd-party tools
Test Execution, Data Collection, Metrics Computation
N
o
Sem
antic
V
alidation
(Runtim
e
Errors)
Scarce Visibility and
Control on the Test
Execution Lifecycle
Test Goals are not
Explicit in the Definition
Declarative Performance Engineering (DPE)
6
“Enabling the performance analyst to declaratively specify what
performance-relevant questions need to be answered without
being concerned about how they should be answered.
”
[Walter et al.]
Jürgen Walter, André van Hoorn, Heiko Koziolek, Dusan Okanovic, and Samuel Kounev. Asking ”What”?,
Automating the ”How”? -TheVision of Declarative Performance Engineering. In Proc. of ICPE 2016. 91–94.
[Walter et al.]
Declarative Performance Engineering (DPE)
6
“Enabling the performance analyst to declaratively specify what
performance-relevant questions need to be answered without
being concerned about how they should be answered.
”
[Walter et al.]
Jürgen Walter, André van Hoorn, Heiko Koziolek, Dusan Okanovic, and Samuel Kounev. Asking ”What”?,
Automating the ”How”? -TheVision of Declarative Performance Engineering. In Proc. of ICPE 2016. 91–94.
[Walter et al.]
Developers,
Testers,
Architects,
Performance Analyst …
DPE: Approaches for Performance Testing
7
[Omar et al.]
Towards an automated approach to use expert systems in the performance testing of distributed systems.
[Westermann] [Scheuner et al.][Omar et al.]
“Enabling the performance analyst to declaratively specify what
performance-relevant questions need to be answered without
being concerned about how they should be answered.
”[Walter et al.]
[Westermann]
Deriving Goal-oriented Performance Models by Systematic Experimentation
[Scheuner et al.]
Cloud Work Bench - Infrastructure-as-Code Based Cloud Benchmarking.
Our Proposal: BenchFlow
8
Declarative 

Performance Testing DSL
Our Proposal: BenchFlow
8
Declarative 

Performance Testing DSL
Model-driven
Framework
Our Proposal: BenchFlow
8
Declarative 

Performance Testing DSL
Model-driven
Framework
https://github.com/benchflow
BenchFlow: Elements of a Test
9
Test Definition
BenchFlow: Elements of a Test
9
Test Definition SUT Configuration
and Deployment
Declarative Test Definition
10
configuration:
goal:
load_function:
quality_gates:
termination_criteria:
data_collection:
workload:
sut:
11
configuration:
goal:
type: configuration
observe:
exploration:
…
Goal Definition
(Configuration Test)
Observed Metrics
12
service_A
service_B
API_1
…
goal:
type: configuration
observe:
workload:
API_1:
- 90thp_response_time
services:
service_A:
- cpu_avg
service_B:
- ram_avg
…
Exploration Space
13
service_A
service_B
API_1
…
goal:
type: configuration
exploration:
exploration_space:
service_A:
resources:
- memory:
range: 1GB... 5GB
step: +1GB
- cpus:
range: 0.1…0.5
- replicas: 1…4
environment:
- SIZE_OF_THREADPOOL:
range: 5...100
step: +5
…
14
Load Function
…
goal:
load_function:
users: 1000
ramp_up: 30s
steady_state: 10m
ramp_down: 30s
… Faban
15
Quality Gates
service_A
service_B
API_1
…
goal:
quality_gates:
workload:
API_1:
90thp_response_time: <= 250ms
services:
service_A:
cpu_avg: <= 70%
…
16
Termination Criteria
…
goal:
termination_criteria:
test:
max_time: 3h
failed_experiments: 30%
…
17
Data Collection
…
data_collection:
client_side:
faban:
interval: 1s
server_side:
service_A:
resource
mysql:
environment:
SETTING_A : example_setting
service_B:
- logs
- resource
…
service_A
service_B
API_1
18
Workload
service_A
service_B
API_1
…
workload:
<name_of_the_workload>:
driver_type: "http"
popularity: 100%

inter_operation_timings: "uniform"

operations:
- …
mix:
- …
…
Faban
19
System Under Test Deployment
service_A
service_B
API_1
…
sut:
configuration:
target_service:
name: service_A
endpoint: /
sut_ready_log_check: "Started in (.*)"
deployment:
service_A: server_1
service_B: server_2
…
19
System Under Test Deployment
service_A
service_B
API_1
…
sut:
configuration:
target_service:
name: service_A
endpoint: /
sut_ready_log_check: "Started in (.*)"
deployment:
service_A: server_1
service_B: server_2
…
https://bit.ly/2EBNrDu
BenchFlow
Model-driven Framework
20
CI ServerRepo
BenchFlow
BenchFlow
BenchFlow
BenchFlow
BenchFlow
BenchFlow
BenchFlow
BenchFlow
Model-driven Framework
20
CI ServerRepo
BenchFlow
BenchFlow
BenchFlow
BenchFlow
BenchFlow
BenchFlow
BenchFlow
BenchFlow
Model-driven Framework
20
CI ServerRepo
BenchFlow
BenchFlow
BenchFlow
BenchFlow
BenchFlow
BenchFlow
BenchFlow
Model-driven Framework
Experiment Execution
ExplorationExecutionAnalysis
Test YAML + SUT Deployment
Metrics
Failures
Result Analysis
Goal
Exploration
Experiment Generation
Experiment YAML + SUT Deployment
Success
Execution
21
Model-driven Framework
Experiment Execution
ExplorationExecutionAnalysis
Test YAML + SUT Deployment
Metrics
Failures
Result Analysis
Goal
Exploration
Experiment Generation
Experiment YAML + SUT Deployment
Success
Execution
21
Test Execution Lifecycle
22
Terminated
Running
Handle Experiment Result
Determine Exploration Strategy
Determine and Execute
Experiments: Experiment
Life Cycle
[can reach goal]
[cannot reach goal]
user terminated
OR
[max_time]
Goal Reached
Completed with Failure
experiment
results
available
[experiments
remaining]
[all experiments executed]
Partially Complete
[number_of_experiments]
Validate
Termination
Criteria
Terminating
Check Quality Gates
[quality
gates pass]
[failed quality gates]
23
Running
Handle Experiment Result
Determine Exploration Strategy
Determine and Execute
Experiments: Experiment
Life Cycle
experiment
results
available
[experiments
remaining]
[all experiment
Terminating
[quality
gates pass]
[failed qu
Test Execution Lifecycle
23
Running
Handle Experiment Result
Determine Exploration Strategy
Determine and Execute
Experiments: Experiment
Life Cycle
experiment
results
available
[experiments
remaining]
[all experiment
Validate
Termination
Criteria
Terminating
Check Quality Gates
[quality
gates pass]
[failed qu
Test Execution Lifecycle
23
Handle Experiment Result
Determine and Execute
Experiments: Experiment
Life Cycle
[can reach goal]
[cannot reach goal
experiment
results
available
[all experiment
[number_of_experimen
Validate
Termination
Criteria Check Quality Gates
[quality
gates pass]
[failed qu
Test Execution Lifecycle
23
Termi
iment Result
d Execute
Experiment
ycle
[can reach goal]
[cannot reach goal]
Goal Re
Completed w
experiment
results
available
remaining]
[all experiments executed]
Partially
[number_of_experiments]
ate
ation
ria Check Quality Gates
[quality
gates pass]
[failed quality gates]
Test Execution Lifecycle
23
Terminated
[cannot reach goal]
Goal Reached
Completed with Failure
[all experiments executed]
Partially Complete
umber_of_experiments]
ality Gates
[failed quality gates]
Test Execution Lifecycle
23
Handle Experiment Result
Determine Exploration Strategy
Determine and Execute
Experiments: Experiment
Life Cycle
[can reach goal]
OR
[max_
experiment
results
available
[experiments
remaining]
[all experiments execut
Validate
Termination
Criteria
Terminating
Check Quality Gates
[quality
gates pass]
[failed quality ga
Test Execution Lifecycle
23
Terminated
]
Goal Reached
Completed with Failure
ents
g]
[all experiments executed]
Partially Complete
[number_of_experiments]
Terminating
Quality Gates
s]
[failed quality gates]
Test Execution Lifecycle
configuration:
goal:
type: configuration
observe:
exploration:
…
Extending the Proposed Approach
24
configuration:
goal:
type: stability_boundary
observe:
exploration:
…
Extending the Proposed Approach
24
LOAD

CONFIGURATION

STABILITY_BOUNDARY
Available Goals:
Extending the Proposed Approach
25
Declarative 

Performance Testing DSL
Terminated
Running
Handle Experiment Result
Determine Exploration Strategy
Determine and Execute
Experiments: Experiment
Life Cycle
[can reach goal]
[cannot reach goal]
user terminated
OR
[max_time]
Goal Reached
Completed with Failure
experiment
results
available
[experiments
remaining]
[all experiments executed]
Partially Complete
[number_of_experiments]
Validate
Termination
Criteria
Terminating
Check Quality Gates
[quality
gates pass]
[failed quality gates]
Extending the Proposed Approach
25
Declarative 

Performance Testing DSL
Approach: Summary
26
Explicit Test Goals in
the Definition
Approach: Summary
26
Explicit Test Goals in
the Definition
Control on the Test
Execution Lifecycle
Approach: Summary
26
Explicit Test Goals in
the Definition
Control on the Test
Execution Lifecycle
Terminated
Running
Handle Experiment Result
Determine Exploration Strategy
Determine and Execute
Experiments: Experiment
Life Cycle
[can reach goal]
[cannot reach goal]
user terminated
OR
[max_time]
Goal Reached
Completed with Failure
experiment
results
available
[experiments
remaining]
[all experiments executed]
Partially Complete
[number_of_experiments]
Validate
Termination
Criteria
Terminating
Check Quality Gates
[quality
gates pass]
[failed quality gates]
Visibility on the Test
Execution Lifecycle
Approach: Summary
26
Static Syntactic and
Semantic Validation
Explicit Test Goals in
the Definition
Control on the Test
Execution Lifecycle
Terminated
Running
Handle Experiment Result
Determine Exploration Strategy
Determine and Execute
Experiments: Experiment
Life Cycle
[can reach goal]
[cannot reach goal]
user terminated
OR
[max_time]
Goal Reached
Completed with Failure
experiment
results
available
[experiments
remaining]
[all experiments executed]
Partially Complete
[number_of_experiments]
Validate
Termination
Criteria
Terminating
Check Quality Gates
[quality
gates pass]
[failed quality gates]
Visibility on the Test
Execution Lifecycle
Approach: Summary
26
Extensible for different
Test Goals
Static Syntactic and
Semantic Validation
Explicit Test Goals in
the Definition
Control on the Test
Execution Lifecycle
Terminated
Running
Handle Experiment Result
Determine Exploration Strategy
Determine and Execute
Experiments: Experiment
Life Cycle
[can reach goal]
[cannot reach goal]
user terminated
OR
[max_time]
Goal Reached
Completed with Failure
experiment
results
available
[experiments
remaining]
[all experiments executed]
Partially Complete
[number_of_experiments]
Validate
Termination
Criteria
Terminating
Check Quality Gates
[quality
gates pass]
[failed quality gates]
Visibility on the Test
Execution Lifecycle
Future Work
27
Apply to real-world use cases:

- Apply the approach in real-world contexts

Future Work
27
Apply to real-world use cases:

- Apply the approach in real-world contexts

Add more goals:

- Regression

- Assess different micro services deployment alternatives

Future Work
27
Apply to real-world use cases:

- Apply the approach in real-world contexts

Add more goals:

- Regression

- Assess different micro services deployment alternatives

Leverage performance test execution data:

- Prioritise performance test execution

- Skip execution of tests

Future Work
27
Apply to real-world use cases:

- Apply the approach in real-world contexts

Add more goals:

- Regression

- Assess different micro services deployment alternatives

Leverage performance test execution data:

- Prioritise performance test execution

- Skip execution of tests

Integrate with other frameworks:

- ContinuITy framework, to leverage production data
State of Practice
4
Script/UI Based Integrated in CI Pipelines
Performance Tests
Execution Automation
Google Vizier
AutoPerf
DataMill
CloudPerf
End-to-end Tests against REST APIs
State of Practice
Highlights
28
Declarative DSL
Declarative Test Definition
10
configuration:
goal:
load_function:
quality_gates:
termination_criteria:
data_collection:
workload:
sut:
Model-driven Framework
BenchFlow
Model-driven Framework
20
CI ServerRepo
BenchFlow
BenchFlow
BenchFlow
BenchFlow
BenchFlow
BenchFlow
BenchFlow
Extending the Approach
configuration:
goal:
type: configuration
observe:
exploration:
…
configuration:
goal:
type: stability_boundary
observe:
exploration:
…
Extending the Proposed Approach
24
LOAD

CONFIGURATION

STABILITY_BOUNDARY
Available Goals:

A Declarative Approach for Performance Tests Execution in Continuous Software Development Environments