Performance Testing Transformation - LTB 2024

PERFORMANCE TESTING TRANSFORMATION
© 2024, Amazon Web Services, Inc. or its affiliates.
Alexander Podelko
Performance Testing
Transformation
Sr. Performance Engineer
Amazon Web Services
Workshop on Load Testing & Benchmarking
LTB 2024
London | May 7, 2024

Alex Podelko
• Has specialized in performance since
1997
• Senior Performance Engineer at AWS –
Amazon Aurora
 Before worked for MongoDB,
Oracle/Hyperion, Intel, and Aetna
• SPEC RG Steering Committee Member
Disclaimer: The views expressed here are my personal views only and do not necessarily represent those of
my current or previous employers. All brands and trademarks mentioned are the property of their owners. All
products are mentioned as examples only, not as recommendations.

© 2024, Amazon Web Services, Inc. or its affiliates. 3
Adjusting Performance Testing
to Industry Trends:
Adding early and continuous performance
testing

Industry Trends
• Web
 Centralization, open / unlimited workload
• Cloud
 Further centralization, price tag (FinOps)
 Dynamic configurations / Self-Management
• Agile / iterative development
 Continuous Integration / Delivery / Deployment
 DevOps / SRE
The Past, Present, and Future of Performance Engineering
4

Industry Trends
Centralization
=> Control over deployments
=> Ability to deploy small changes
=> Agile development
=> Fuzzier line between Dev and Ops (DevOps, SRE)
=> Need for continuous performance engineering
The Past, Present, and Future of Performance Engineering

Continuous Performance Testing
• Continuous performance testing
 To catch regressions early
• Collecting all info needed to investigate regressions
 In the form convenient for further analysis
• Foundation to build further automation on the top of it
 For further performance optimization
• All context-dependent
 Don’t wait for an exact recipe, figure it out depending on your needs

Performance Testing
Traditional vs Continuous
• Before releases
• Realistic Mix
• As close to production as possible
• Checking Service Level Objectives
(SLOs)
• Using a load testing tool or
harness
• The approach is relatively
consistent and well described
• Often (maybe even each build)
• Different tests
• To maximize coverage
• Checking the difference between
builds
• Using an additional layer of
automation on the top of load testing
tool
• All context-dependent

Integrating Performance Engineering into DevOps
Developmen
t
Operations
PE
Performance
Testing
Capacity Planning
Tuning
Shift Right
Shift Left
Monitoring
Capacity Planning
SRE
FinOps
Efficiency
Performance
Tester / Engineer /
Architect
Developer
SDET
8

Challenges of Continuous Performance Testing
• Integration
• Decomposition
• Coverage Optimization
• Variability / Noise Reduction
• Change Detection
• Advanced Analysis
• Operations / Maintenance
9

The Challenge of
Integration

Continuous Integration: Load Testing Tools
• CI support in load testing tools
 Integration with CI Servers (Jenkins, Hudson, etc.)
 Automation support
• CI tools support for performance testing
 Jenkins Performance Plugin
• Performance Testing Frameworks
 Combining multiple tools

A Performance Testing Framework
An example:
https://github.com/serputk
o/performance-testing-fram
ework

Distributed Load Testing on AWS
From AWS Solutions Library
https://aws.amazon.com/solutions/implementations/distributed-load-tes
ting-on-aws/
13

Closely Integrated Systems
• Sophisticated, but proprietary closely integrated systems
 Creating a Virtuous Cycle in Performance Testing at MongoDB
 Fallout: Distributed Systems Testing as a Service (DataStax)
 Tracking Performance of the Graal Compiler on Public Benchmarks (Charles
University / Oracle Labs)
 Introducing Ballast: An Adaptive Load Test Framework (Uber)
14

The Challenge of
Decomposition

Decomposition
• For most complex systems, continuous performance testing
should be done on component level / limited scale
 To align with development
 System-level requirements -> Component-level requirements
 Record/playback approach -> Programming
– Custom Load generation
– Stubbing/Mocking/Service Virtualization

Result Interpretation [Modeling]
• If the results are for component / small-scale environment,
changes should be modeled into end-to-end performance
 Performance Testing and Modeling for New Analytic Applications
 Or/and confirmed by full-scale end-to-end performance test

The Challenge of
Coverage
Optimization

Time / Resource Considerations
• Performance tests take time and resources
 The larger tests, the more
• May be not an option on each commit
• Need of a tiered solution
 Some performance measurements each commit
 Daily mid-size performance tests
 Periodic large-scale / uptime tests outside CI

Coverage Optimization
• A multi-dimensional problem
 Configuration
 Workloads / Tests
 Frequency of runs
• A trade off between coverage and costs
 Costs of running, analyzing, maintenance, etc.

The Challenge
• If addressed seriously, the number of workloads / tests /
configurations is growing
• No good way to optimize
• One approach is to see if some results are correlated
 If we find same problems on the same set of tests, we can use just one
or few tests from this group
 Tracking Performance of the Graal Compiler on Public Benchmarks (Charles
University / Oracle Labs)
• Combinatorial testing approaches (PairWise / Covering Arrays)
 From functional testing

The Challenge of
Variability

Variability - System
• Inherent to the test setup

Addressing Variability
• Methodological principles for reproducible performance eva
luation in cloud computing. 2019
(SPEC RG – Cloud)
• Reducing variability in performance tests on EC2: Setup and
Key Results
(MongoDB)
• Tracking Performance of the Graal
Compiler on Public Benchmarks

Addressing Variability
• Same environment / starting config
 For example, AWS cluster placement groups
• No other load
• Multiple iterations
• Reproducible multi-user tests
 Restarts between tests
 Clearing caches / Warming up caches
 Staggering / Sync points

The Challenge of
Change Detection

Complex Results
• No easy pass/fail
 Individual responses, monitoring results, errors, etc.
• No easy comparison
 Against SLA
 Between builds
• Variability

Simple Comparison
Jenkins Performance Plugin

keptn.sh
Quality Gates
SLIs / SLOs as code

Change Point Detection
• Statistical methods taking noise in consideration
• E-Divisive means algorithm
 ICPE Paper: Change Point Detection in Software Performance Testing
 Fixing Performance Regressions Before they Happen, Netflix
Technology Blog
 https://github.com/mongodb/signal-processing-algorithms
– Open sourced, generic
 Need several data points. May miss a gradual degradation.

The Challenge of
Advanced
Analysis

Keep All Artifacts for Further Analysis
• Get all metrics
 Throughputs, latencies, resource utilizations, etc.
• Save all related artifacts
 Exact code / configuration
 Logs, etc.
• Ability to re-run the test in the exactly same configuration is
helpful

Root Cause Analysis
• Collecting artifacts to do root cause analysis
• Insights snapshots
 Flamegraphs (perf, eBPF)
• Continuous Profiling
 Java Flight Recorder
 APM
 Tracing
 Observability
 eBPF-based tools

Visualization
• Visualizing systems and software performance - Report on the
GI-
Dagstuhl
• Sometimes helps to catch an issue

Looking Beyond Aggregate Info

The Challenge of
Operations and
Maintenance

Operations
• Scheduling / execution tests
• Results analysis
• Triaging and escalating issues
• Maintenance

Coverage / Maintenance Trade-Off
Coverage
Maintenance

Catching / Troubleshooting Errors
• Catching errors is not trivial
 Building in checks
 Depends on interfaces used
– Protocol-level [recording]
– GUI
– API/Programming
– Production Workloads
• Keeping logs / all info needed to investigate issues

Changing Interfaces
• If using protocol-level or GUI scripts, minor changes may break
them
 It may be not evident
 If recording used, a change in interfaces may require to recreate the
whole script
• API / Programming is usually more stable / easier to fix
• AI to catch the changes / self-healing scripts

Who Is Doing Maintenance?
• Who is responsible for what?
• Infrastructure Code
 Tools, plumbing code, integration
• Specific tests
• Integrated workloads
 Covered multiple functional areas

SUMMARY
• Adjusting performance testing to industry trends
• Specific challenges should be addressed:
 Integration
 Coverage Optimization
 Variability / Noise Reduction
 Change Detection
 Advanced Analysis
 Operations / Maintenance
• Performance engineering gets more integrated, context-dependent
 Integrated into both Development and Operations

Thank you!
Alex Podelko
podealex@amazon.com

Performance Testing Transformation - LTB 2024

More Related Content

Similar to Performance Testing Transformation - LTB 2024

More from Alexander Podelko

Recently uploaded

Performance Testing Transformation - LTB 2024

Editor's Notes