AWS Fault Injection Service for Serverless applications - SLA Berlin 2025

AWS Fault Injection Service
for Serverless applications
Vadym Kazulkin, ip.labs , Serverless Architecture Conference Berlin, 22 October 2025

Vadym Kazulkin
ip.labs GmbH Bonn, Germany
Co-Organizer of the Java User Group Bonn
v.kazulkin@gmail.com
@VKazulkin
https://dev.to/vkazulkin
https://github.com/Vadym79/
https://de.slideshare.net/VadymKazulkin/
https://www.linkedin.com/in/vadymkazulkin
https://www.iplabs.de/
Contact

Image: burst.shopify.com/photos/a-look-across-the-landscape-with-view-of-the-sea
About ip.labs

AWS Fault Injection Service for Serverless applications
Agenda
▪ Introduction to AWS Fault Injection Service (FIS)
▪ Configure sample application
▪ Create AWS FIS Experiment Templates
▪ Start AWS FIS Experiment Templates
▪ CloudWatch logs and metrics for AWS FIS Experiment
▪ Wrap up

Introduction to AWS Fault Injection
Service (FIS)
https://aws.amazon.com/de/fis/

What Is AWS Fault Injection Service (FIS)?
▪ As a part of AWS Resilience Hub, AWS Fault Injection Service is a fully
managed service for running fault injection experiments to improve an
application’s performance, observability, and resilience.
▪ These experiments stress an application by creating disruptive events so
that you can observe how your application responds.
▪ To use AWS FIS, you run experiments on your AWS resources to test your
theory of how an application or system will perform under fault conditions.

What Is FIS?
▪ You can then use this information to improve the performance
and resiliency of your applications so that they behave as
expected.
▪ Fault injection is based on the principles of chaos engineering.

https://docs.aws.amazon.com/resilience-hub/latest/userguide/test-assessment-report.html
Initiating, creating, and running AWS FIS experiments
▪ To run experiments, you first create an experiment template. An
experiment template is the blueprint of your experiment. It contains
the actions, targets, and stop conditions for the experiment.
▪ After you created an experiment template, you can use it to run an
experiment.
▪ While your experiment is running, you can track its progress and view
its status.
▪ An experiment is complete when all of the actions in the experiment
have run.

Chaos Engineering for Serverless
applications on AWS

Chaos Engineering for Serverless applications on AWS
Below is the list of the 3rd party resources and tools already
available for AWS Lambda:
▪ aws-lambda-chaos-injection
▪ failure-lambda
▪ chaos-lambda-extension
▪ Steadybit
▪ Gremlin

Introducing AWS Fault Injection Service
Actions to Inject Chaos in Lambda
functions
https://aws.amazon.com/de/blogs/mt/introducing-aws-fault-injection-service-actions-to-inject-chaos-in-lambda-functions/

https://docs.aws.amazon.com/fis/latest/userguide/use-lambda-actions.html
AWS FIS Actions to Inject Chaos in Lambda functions
AWS FIS now injects faults into AWS Lambda functions using a
“chaos” Lambda extension that runs as a separate process within
the Lambda execution environment, intercepting invocations
before they reach the runtime.

AWS FIS Actions to Inject Chaos in Lambda functions
With the launch of FIS Lambda actions, the following 3 actions are
now available to us in the FIS experiment templates:
▪ Add start delay: aws:lambda:invocation-add-delay
▪ Modify integration response: aws:lambda:invocation-http-
integration-response
▪ Enforce invocation errors: aws:lambda:invocation-error

https://github.com/Vadym79/AWSFaultInjectionSimulatorForLambda/tree/main/java21-fis-lambda/

AWS Fault Injection Service for AWS
Lambda:
FIS Lambda Action invocation-add-delay
https://dev.to/aws-heroes/aws-fault-injection-service-for-aws-lambda-part-2-fis-lambda-action-invocation-add-delay-4ffn

AWS Fault Injection Service for AWS:
Configure the sample application

Adding AWS Lambda FIS Extension
Add recent version of AWS Lambda
FIS extension

• Go to the "AWS Systems
Manager" in your region, for
example for eu-central-1 to this
page
• Find "Parameter Store" in the
navigation bar. You need to store
at least 1 parameter to search for
Public parameters.
• Select fis in the list of services

• Select the Lambda architecture of
your choice (default one is
x86_64) and copy the extension
ARN

AWS Lambda FIS Extension Global Parameters
• AWS_LAMBDA_EXEC_WRAPPER ‐
Required. Location of the AWS
Lambda wrapper script used to
configure the AWS FIS Lambda
extension. This should be set to
the /opt/aws-fis/bootstrap script
that is included with the
extension

AWS Lambda FIS Extension Global Parameters
• AWS_FIS_EXTENSION_METRICS ‐
Optional. Possible values are all
and none. If set to all the extension
will emit EMF metrics under the
aws-fis-extension namespace
• Set the
AWS_FIS_CONFIGURATION_LOCATION
variable to the ARN of the Amazon
S3 FIS configuration folder

Adding Lambda Permissions to access AWS FIS
Configuration Location on S3
• Give all Lambda functions
participating in the FIS
experiment permission to access
the created S3 Bucket

Create AWS FIS Experiment Template for
aws:lambda:invocation-add-delay Lambda
action

aws:lambda:invocation-add-delay
• In eu-central one please navigate
to Experiment Template
• "Create Experiment template“.
• Specify description and name and
what AWS account to target:
single or multiple AWS accounts

• For the "Action type" select
Lambda and
"aws:lambda:invocation-add-
delay"
In the "Action parameters" specify:
• 10 minutes "duration" of the
experiment
• "invocation percentage" of 100%
(means all invocations)
• "startup delay in milliseconds" of
10.000, which means we'll inject
10 seconds latency to the target
Lambda function

• Click on "Function-Target-1„ to
specify the concrete Lambda
target

• Specify the concrete Lambda
target: you can use resource tags,
filters or parameters or give the
concrete ARN of the Lambda
function

• Create an IAM role for and give it
enough permissions

• In the "Logs" section (the
destination that receives the
experiment log data) specify S3
bucket. It's the same S3 bucket
specified in the IAM permissions
policies
• Create "Log Group“ and specify its
ARN to store the logs

• The experiment template is
created
• We can start it by hitting "Start
experiment"

Start AWS FIS Experiment Template for
aws:lambda:invocation-add-delay Lambda
action

https://aws.amazon.com/de/blogs/mt/introducing-aws-fault-injection-service-actions-to-inject-chaos-in-lambda-functions/
Start AWS FIS Experiment Template Workflow

{"faults":[{"actionId":"aws:lambda:invocation-add-delay","parameters":
{"invocationPercentage":"100","startupDelayMilliseconds":"10000","qualifier":"$ANY",
"minimumRequiredExtensionVersion":"1.0.0"},
"expiration":1751113980295,"startTime":1751113858706}]}

• We see the response time
approx. 14 seconds : big Lambda
cold start (the application uses
managed Java 21 runtime on
Lambda) and 10 additional
seconds of the injected latency

CloudWatch logs and metrics Experiment
Template for aws:lambda:invocation-add-
delay Lambda action

CloudWatch logs for AWS FIS Experiment Template for
• We see in CloudWatch Logs that
Lambda FIS extension was
recognized
• It’s polling the configured S3
bucket for the active faults and
found them

FIS experiments details and effect
(adding latency to function
invocation) have been logged as
well

custom CloudWatch metrics has
been sent

CloudWatch metrics for AWS FIS Experiment Template
for aws:lambda:invocation-add-delay
• CloudWatch aws-fis-extension
Metrics

• FIS Fault Active/Injected
CloudWatch metrics

• FIS experiments instructions, like
LatencyInjected CloudWatch
metrics

• FIS Lambda extension
configuration CloudWatch metrics

aws:lambda:invocation-http-integration-
response Lambda action
https://dev.to/aws-heroes/aws-fault-injection-service-for-aws-lambda-part-3-fis-lambda-action-2hce

AWS FIS Experiment Template for aws:lambda:invocation-
http-integration-response
Supported integration responses for:
▪ API Gateway (REST and HTTP API)
▪ Application Loadbalancer (ALB)

aws:lambda:invocation-http-integration-response
Lambda and
"aws:lambda:invocation-http-
integration-response"
experiment
• "status code" of 500
• enable "prevent execution"

function

aws:lambda:invocation-http-integration-
response Lambda action

{"faults":[{"actionId":"aws:lambda:invocation-http-integration-response",
"parameters"{"invocationPercentage":"100","preventExecution":"true",
"replaceOutputWithString":"{"body":"","headers":{"Content-Type":"text/plain"},
"statusCode""500"}","injectAsError":"false","qualifier":"$ANY","minimumRequiredExtension
Version":"1.0.0"},"expiration":1751296234169,"startTime":1751296009540}]}

https://dev.to/aws-heroes/aws-fault-injection-service-for-aws-lambda-part-3-fis-lambda-action-2hce https://github.com/rakyll/hey
hey -q 3 -z 1m -c 4 -m PUT -d '{"id": 29, "name": "Print 10x13", "price": 0.15}' -H "X-API-Key:
a6ZbcDefQW12BN56WEAW7" -H "Content-Type: application/json;charset=utf-8"
https://{$API_GATEWAY_URL}/prod/products/

Template for aws:lambda:invocation-http-
integration-response Lambda action

recognized
found them

(modifying the function response)
have been logged as well

CloudWatch metrics for AWS FIS Experiment Template for
CloudWatch metrics

aws:lambda:invocation-error Lambda action
https://dev.to/aws-heroes/aws-fault-injection-service-for-aws-lambda-part-4-fis-lambda-action-invocation-error-493a

aws:lambda:invocation-error
Lambda and
"aws:lambda:invocation-error”
experiment
• enable "prevent execution"

function

aws:lambda:invocation-error Lambda action

{"faults":[{"actionId":"aws:lambda:invocation-error", "parameters„
{"invocationPercentage":"100","preventExecution":"true","replaceOutputWithString":"{"error
Message":"FIS Injected Fault", "errorType„ :"FISInjectedFault„ ,"stackTrace":[]}„
,"injectAsError":"true","qualifier":"$ANY","minimumRequiredExtensionVersion":"1.0.0"},
"expiration":1751303485580,"startTime":1751303322914}]}

https://dev.to/aws-heroes/aws-fault-injection-service-for-aws-lambda-part-4-fis-lambda-action-invocation-error-493a https://github.com/rakyll/hey
hey -q 5 -z 1m -c 10 -H "X-API-Key: a6ZbcDefQW12BN56WEAW7"
{$API_GATEWAY_URL}/prod/products/1

Template for aws:lambda:invocation-error
Lambda action

recognized
found them

(modifying the function response)
have been logged as well

CloudWatch metrics for AWS FIS Experiment Template for
CloudWatch metrics

Other Serverless FIS supported Actions

Other Serverless FIS supported Actions
https://docs.aws.amazon.com/fis/latest/userguide/fis-actions-reference.html
▪ aws:dsql:cluster-connection-failure
▪ Creates controlled connection failures in an Aurora DSQL cluster for a
specified duration to test application resilience.
▪ aws:dynamodb:global-table-pause-replication
▪ Pauses Amazon DynamoDB multi-Region eventually consistent
(MREC) global table replication to any replica table. Tables may
continue to be replicated for up to 5 minutes after action begins.
▪ This action does not support multi-Region strongly consistent
(MRSC) global tables.

Introduction to AWS Fault Injection
Service (FIS)
Wrap up and suggestions

AWS Fault Injection Service for AWS Lambda:
Things to improve
https://docs.aws.amazon.com/fis/latest/userguide/fis-actions-reference.html
▪ Add throttling DynamoDB action
▪ Add different actions for different Lambda errors
▪ Currently only generic invocation-error Lambda action is available
▪ Possible Lambda errors : throttling, timeout, initialization, (application) error
▪ You can simulate API Gateway throttling by sending HTTP 429
status in “invocation http integration response” action

AWS services such as S3, SNS, EventBridge
and others invoke Lambda functions
asynchronously.
Lambda uses an internal queue to store
events. A separate process reads events from
the queue and sends them to the function.
https://aws.amazon.com/blogs/compute/introducing-new-asynchronous-invocation-metrics-for-aws-lambda/
Asynchronous invocations of Lambda functions

https://aws.amazon.com/blogs/compute/introducing-new-asynchronous-invocation-metrics-for-aws-lambda/
https://docs.aws.amazon.com/lambda/latest/dg/invocation-async.html
Asynchronous invocations of Lambda functions and retry
behavior
Lambda discards events from its event queue if the retry policy has
exceeded the number of configured retries or the event reached its
maximum age.
The event once discarded from the event queue goes to the destination or
DLQ, if configured.

AWS FIS + Amazon DevOps Guru
https://aws.amazon.com/de/fis/ https://aws.amazon.com/devops-guru

Amazon DevOps Guru offers a fully managed AIOps platform powered
by machine learning (ML) that is designed to make it easy to improve an
application’s operational performance and availability
DevOps Guru helps detect behaviors that deviate from normal operating
patterns so you can identify operational issues long before they impact
your customers
• increased latency
• error rates (timeouts, throttles, CPU, memory and, disk utilization)
• resource constraints (exceeding AWS account limits)
What is AWS DevOps Guru
https://aws.amazon.com/devops-guru

Benefits of DevOps Guru

75 AWS Fault Injection Service for Serverless applications
How DevOps Guru work

https://dev.to/vkazulkin/series/24936
Amazon DevOps Guru for Serverless application article series

AWS Fault Injection Service for Serverless applications - SLA Berlin 2025

More Related Content

More from Vadym Kazulkin

Recently uploaded

AWS Fault Injection Service for Serverless applications - SLA Berlin 2025