REAL WORLD SERVERLESS
Going Lambda without being burned too much
@Koenighotze
DAVID SCHMITZ
SENACOR TECHNOLOGIES
@KOENIGHOTZE
@Koenighotze
4 QUESTIONS
@Koenighotze
MICROSERVICES
CONTAINERS
CLOUD
SERVERLESS
@Koenighotze
VENDOR?
CLOUD GURU?
@Koenighotze
@Koenighotze
THINGS ARE MOVING
FAST
@Koenighotze
EXPECTATION
MANAGEMENT
@Koenighotze
–Me
“Some of the things we learned,
while moving an application
(partially) to AWS Lambda.”
@Koenighotze
ASP.NET
MSSQL
EMPLOYEE
ADMINISTRATION
PLATFORM
@Koenighotze
TARGET ARCHITECTURE
@Koenighotze
Users Projects Budget
ASP.NET
MSSQL
@Koenighotze
WORK IN PROGRESS!
@Koenighotze
Some basics
Tools and stuff
Architecture
Security
Housekeeping
Operations
@Koenighotze
@Koenighotze
AWS LAMBDA 1-0-1
@Koenighotze
Ye olde days
@Koenighotze
@Koenighotze
INFRASTRUCTURE AS A
SERVICE
@Koenighotze
@Koenighotze
PLATFORM AS A SERVICE
@Koenighotze
@Koenighotze
CONTAINERS
@Koenighotze
@Koenighotze
Provisioning
Scaling
Monitoring
State
Security
Deploying
…
@Koenighotze
DESIGNING DISTRIBUTED
SYSTEMS IS DIFFICULT
@Koenighotze
HOW TO DRAW AN OWL?
@Koenighotze
1. Draw some circles 2. Draw the rest of the damn owl
@Koenighotze
@Koenighotze
@Koenighotze
@Koenighotze
@Koenighotze
ENTER SERVERLESS
@Koenighotze
SO WHAT IS
SERVERLESS?
@Koenighotze
Serverless computing allows you to build and
run applications and services without thinking
about servers. Serverless applications don't
require you to provision, scale, and manage any
servers…
Building serverless applications means that
your developers can focus on their core product
instead of worrying about managing and
operating servers or runtimes…
@Koenighotze
Serverless computing allows you to build and
run applications and services without thinking
about servers. Serverless applications don't
require you to provision, scale, and manage any
servers…
Building serverless applications means that
your developers can focus on their core product
instead of worrying about managing and
operating servers or runtimes…
@Koenighotze
FOCUS ON WHAT MATTERS
@Koenighotze
@Koenighotze
@Koenighotze
BASIC VOCABULARY
@Koenighotze
LAMBDA
-
SERVERLESS COMPUTE
@Koenighotze
S3
-
DROPBOX
@Koenighotze
IAM
-
SECURITY
@Koenighotze
CLOUDFORMATION
-
PROVISIONING
@Koenighotze
WHAT IS A
“LAMBDA FUNCTION”?
@Koenighotze
Event driven
Stateless
Asynchronous
Single purpose
@Koenighotze
@Koenighotze
@Koenighotze
EXECUTION MODEL
@Koenighotze
@Koenighotze
@Koenighotze
S3
@Koenighotze
@Koenighotze
@Koenighotze
@Koenighotze
@Koenighotze
Some time later…
@Koenighotze
@Koenighotze
SCALE BY REQUEST
@Koenighotze
@Koenighotze
@Koenighotze
NO OVERPROVISION
NO UNDERPROVISION
@Koenighotze
CHARGE BY
EXECUTION
@Koenighotze
FRAMEWORKS, RUNTIMES
AND TOOLS
@Koenighotze
@Koenighotze
AWS LAMBDA MAY NEED
NO SPECIAL FRAMEWORK
@Koenighotze
VENDOR LOCK-IN
@Koenighotze
LOG4J
VS.
COMMONS-LOGGING
@Koenighotze
SEPARATE HANDLER
AND BUSINESS CODE
@Koenighotze
Handler
create Order
forward Order
cancel Order
@Koenighotze
Handler
create Order
forward Order
cancel Order
Vendor
agnostic
Vendor
specific
Anticorruptionlayer
Pure domain logic
@Koenighotze
YOU ABSOLUTELY NEED
TOOLING FOR
DEPLOYMENT
@Koenighotze
AVOID UI DEPLOYMENTS
@Koenighotze
@Koenighotze
@Koenighotze
AUTOMATION IS KEY
@Koenighotze
$ zip index.zip index.js node_modules
$ aws lambda create-function 
--function-name HelloWorld 
--runtime nodejs6.10 
--role arn:aws:iam::…:… 
--handler index.handler 
--zip-file fileb://index.zip
@Koenighotze
TAGGING, UPDATE,
RESOURCES, ROLES…
@Koenighotze
$ claudia create 
--region eu-west-1 
--deploy-proxy-api 
--handler index.handler 
--name HelloWorld
@Koenighotze
STATE, ROLLBACK,
ERROR HANDLING…
@Koenighotze
DECLARATIVE
DEPLOYMENTS
@Koenighotze
@Koenighotze
AWS SERVERLESS
APPLICATION
MODEL
@Koenighotze
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Hello World
Resources:
HelloWorld:
Type: AWS::Serverless::Function
Properties:
Timeout: 5
Runtime: nodejs6.10
Handler: index.handler
CodeUri: .
@Koenighotze
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Hello World
Resources:
HelloWorld:
Type: AWS::Serverless::Function
Properties:
Timeout: 5
Runtime: nodejs6.10
Handler: index.handler
CodeUri: .
@Koenighotze
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Hello World
Resources:
HelloWorld:
Type: AWS::Serverless::Function
Properties:
Timeout: 5
Runtime: nodejs6.10
Handler: index.handler
CodeUri: .
@Koenighotze
$ aws cloudformation package 
--template-file sam.yaml 
--s3-bucket somebucket 
> build/packaged.yaml
$ aws cloudformation deploy 
--template-file build/packaged.yaml 
--stack-name voxxed-demo 
--capabilities CAPABILITY_NAMED_IAM
@Koenighotze
$ aws cloudformation package 
--template-file sam.yaml 
--s3-bucket somebucket 
> build/packaged.yaml
$ aws cloudformation deploy 
--template-file build/packaged.yaml 
--stack-name voxxed-demo 
--capabilities CAPABILITY_NAMED_IAM
@Koenighotze
$ aws cloudformation package 
--template-file sam.yaml 
--s3-bucket somebucket 
> build/packaged.yaml
$ aws cloudformation deploy 
--template-file build/packaged.yaml 
--stack-name voxxed-demo 
--capabilities CAPABILITY_NAMED_IAM
@Koenighotze
$ aws cloudformation package 
--template-file sam.yaml 
--s3-bucket somebucket 
> build/packaged.yaml
$ aws cloudformation deploy 
--template-file build/packaged.yaml 
--stack-name voxxed-demo 
--capabilities CAPABILITY_NAMED_IAM
@Koenighotze
$ aws cloudformation package 
--template-file sam.yaml 
--s3-bucket somebucket 
> build/packaged.yaml
$ aws cloudformation deploy 
--template-file build/packaged.yaml 
--stack-name voxxed-demo 
--capabilities CAPABILITY_NAMED_IAM
@Koenighotze
ALL TOOLS HAVE
DIFFERENT DRAWBACKS
@Koenighotze
DEBUGGING
ROLLBACK
STATE
FEATURE COMPLETENESS
@Koenighotze
CLEANLINESS IS NEXT
TO GODLINESS
@Koenighotze
@Koenighotze
TEST, TEST, TEST
@Koenighotze
INTEGRATION TEST
RIGHT FROM THE START
@Koenighotze
SAM LOCAL
@Koenighotze
@Koenighotze
@Koenighotze
YOUR TESTING TOOLS
SHOULD BE AS
MATURE AS YOUR
PRODUCTION TOOLS
@Koenighotze
TESTING LAMBDA FOR
REAL IS JUST TOO EASY
@Koenighotze
aws lambda invoke 
--function-name ReceiptUpload 
--region eu-central-1 
--payload file://expensive_receipt.json 
out.json
@Koenighotze
aws lambda invoke 
--function-name ReceiptUpload 
--region eu-central-1 
--payload file://expensive_receipt.json 
out.json
@Koenighotze
PROTECT PRODUCTION
@Koenighotze
STAGING IS A THING
@Koenighotze
@Koenighotze
Parameters:
…
Stage:
Type: String
Default: DEV
ForwardBilling:
Type: AWS::Serverless::Function
Properties:
FunctionName: !Sub ForwardBilling-${Stage}
@Koenighotze
Parameters:
…
Stage:
Type: String
Default: DEV
ForwardBilling:
Type: AWS::Serverless::Function
Properties:
FunctionName: !Sub ForwardBilling-${Stage}
@Koenighotze
aws cloudformation deploy
--template-file …
--stack-name …
--parameter-overrides Stage=PROD
@Koenighotze
aws cloudformation deploy
--template-file …
--stack-name …
--parameter-overrides Stage=PROD
@Koenighotze
PROTECT STAGES
WITH ACCOUNTS
@Koenighotze
@Koenighotze
@Koenighotze
aws cloudformation deploy
…
--parameter-overrides Stage=PROD
--profile ACCOUNT_A
@Koenighotze
aws cloudformation deploy
…
--parameter-overrides Stage=PROD
--profile ACCOUNT_A
@Koenighotze
CLEANUP UNUSED
LAMBDAS
@Koenighotze
@Koenighotze
HOW DO YOU
REASON ABOUT
CLOUD
ARCHITECTURES?
@Koenighotze
HOW TO KNOW WHAT
TO CLEANUP?
@Koenighotze
PYTHON LIST_LAMBDAS.PY 
—INACTIVE-DAYS-FILTER 10
https:!//github.com/epsagon/list-lambdas
@Koenighotze
@Koenighotze
@Koenighotze
@Koenighotze
REDUCE ATTACK SURFACE
REDUCE COGNITIVE
CLUTTER
STAY IN CONTROL
@Koenighotze
HOT AND COLD
@Koenighotze
COLD START HAPPENS
ONCE, RIGHT?
@Koenighotze
Users
@Koenighotze
UsersCold
COLD
START
@Koenighotze
UsersCold
@Koenighotze
UsersCold
HOT
START
@Koenighotze
SCALE BY REQUEST
@Koenighotze
UsersCold
@Koenighotze
@Koenighotze
COLD START WILL HAPPEN
ONCE FOR EACH
CONCURRENT EXECUTION
OF YOUR FUNCTION
@Koenighotze
WHAT CAN YOU DO?
@Koenighotze
@Koenighotze
NO SLA
@Koenighotze
NO HARD DETAILS
JUST GUIDELINES
@Koenighotze
PACKAGE SIZE?
@Koenighotze
@Koenighotze
NO OBVIOUS
CORRELATION!?
@Koenighotze
MEMORY SETTINGS?
@Koenighotze
@Koenighotze
/AWS/LAMBDA/…REPORT
REQUESTID:
DURATION: 3208.99 MS
BILLED DURATION: 3300 MS
MEMORY SIZE: 128 MB
MAX MEMORY USED: 59 MB
@Koenighotze
/AWS/LAMBDA/…REPORT
REQUESTID:
DURATION: 3208.99 MS
BILLED DURATION: 3300 MS
MEMORY SIZE: 128 MB
MAX MEMORY USED: 59 MB
@Koenighotze
SOME CORRELATION
@Koenighotze
WARMUP-PING?
@Koenighotze
@Koenighotze
AWS WILL KILL EVEN
WARM LAMBDAS AFTER A
CERTAIN TIME FRAME
@Koenighotze
MEASURE
CHANGE
MEASURE
@Koenighotze
MEASURE PERCEIVED
EXTERNAL PERFORMANCE
@Koenighotze
MEASURE USER
EXPERIENCE
@Koenighotze
DESIGN WITH
LATENCY IN MIND
@Koenighotze
TIGHT SECURITY
@Koenighotze
SERVERLESS IS GOOD
FOR SECURITY
@Koenighotze
NO LONG LIVED SERVERS
OR MANUAL PATCH
SCHEDULES
@Koenighotze
…THAT YOU NEED TO
HANDLE
@Koenighotze
…BUT AMAZON DOES
@Koenighotze
HOW DO YOU DEPLOY A WEB
APPLICATION FIREWALL…
…WITHOUT A WEB
…WITHOUT AN APPLICATION
…WITHOUT A SERVER
@Koenighotze
NEW OR ADAPTED
SECURE CODING
PATTERNS NEEDED
@Koenighotze
INCREASED ATTACK
SURFACE
@Koenighotze
@Koenighotze
IMAGINE EVERYBODY
WANTS TO HARM YOU
@Koenighotze
@Koenighotze
EVENT INJECTION
@Koenighotze
@Koenighotze
Validation
Magic
@Koenighotze
CLASSIC LAYERED CAKE
ARCHITECTURE DOES NOT
APPLY TO SERVERLESS
@Koenighotze
NEVER TRUST YOUR
CALLER
@Koenighotze
const Joi = require('joi')
const eventSchema = {
'lambdaName': Joi.string().required(),
'invokationCount': Joi.number().required(),
'dryRun': Joi.boolean().default(true)
}
exports.handler = (event, context, callback) => {
const { error, value } = Joi.validate(event, eventSchema)
if (error) {
return callback(new Error('Cannot parse event'))
}
...doSomethingWithValue()
}
@Koenighotze
const Joi = require('joi')
const eventSchema = {
'lambdaName': Joi.string().required(),
'invokationCount': Joi.number().required(),
'dryRun': Joi.boolean().default(true)
}
exports.handler = (event, context, callback) => {
const { error, value } = Joi.validate(event, eventSchema)
if (error) {
return callback(new Error('Cannot parse event'))
}
...doSomethingWithValue()
}
@Koenighotze
const Joi = require('joi')
const eventSchema = {
'lambdaName': Joi.string().required(),
'invokationCount': Joi.number().required(),
'dryRun': Joi.boolean().default(true)
}
exports.handler = (event, context, callback) => {
const { error, value } = Joi.validate(event, eventSchema)
if (error) {
return callback(new Error('Cannot parse event'))
}
...doSomethingWithValue()
}
@Koenighotze
BILLING ATTACK
@Koenighotze
@Koenighotze
SECURITY IS LIKE CSS
NO ONE DELETES RULES
YOU ONLY ADD RULES
@Koenighotze
PRINCIPLE OF
LEAST PRIVILEGES
@Koenighotze
BEWARE CRAPPY
CODE EXAMPLES
@Koenighotze
"Statement": [{
"Effect": "Allow",
"Action": [
“s3:*",
],
"Resource": [
“arn:aws:s3:::*”
]
}, …
@Koenighotze
"Statement": [{
"Effect": "Allow",
"Action": [
"s3:*",
],
"Resource": [
"arn:aws:s3:::*"
]
}, …
@Koenighotze
"Statement": [{
"Effect": "Allow",
"Action": [
“s3:*",
],
"Resource": [
“arn:aws:s3:::senacor/fis/orders/*”
]
}, …
@Koenighotze
"Statement": [{
"Effect": "Allow",
"Action": [
"s3:GetObject"
],
"Resource": [
“arn:aws:s3:::senacor/fis/orders/*”
]
}, …
@Koenighotze
"Statement": [{
"Effect": "Allow",
"Action": [
"s3:GetObject"
],
"Resource": [
“arn:aws:s3:::senacor/fis/orders/*”
]
}, …
@Koenighotze
@Koenighotze
DO NOT SHARE
POLICIES
@Koenighotze
@Koenighotze
NO EXCUSES WITH
PROPER TOOLING
@Koenighotze
Resources:
ForwardBillingRole:
Type: "AWS::IAM::Role"
Properties:
RoleName: !Sub "ForwardBillingRole-${Stage}"
AssumeRolePolicyDocument: ...
ManagedPolicyArns: ...
Policies:
-
PolicyName: !Sub "StoreBillingInS3Policy-${Stage}"
PolicyDocument: ...
ForwardBilling:
Type: AWS::Serverless::Function
Properties:
Role: !GetAtt ForwardBillingRole.Arn
@Koenighotze
Resources:
ForwardBillingRole:
Type: "AWS::IAM::Role"
Properties:
RoleName: !Sub "ForwardBillingRole-${Stage}"
AssumeRolePolicyDocument: ...
ManagedPolicyArns: ...
Policies:
-
PolicyName: !Sub "StoreBillingInS3Policy-${Stage}"
PolicyDocument: ...
ForwardBilling:
Type: AWS::Serverless::Function
Properties:
Role: !GetAtt ForwardBillingRole.Arn
@Koenighotze
Resources:
ForwardBillingRole:
Type: "AWS::IAM::Role"
Properties:
RoleName: !Sub "ForwardBillingRole-${Stage}"
AssumeRolePolicyDocument: ...
ManagedPolicyArns: ...
Policies:
-
PolicyName: !Sub "StoreBillingInS3Policy-${Stage}"
PolicyDocument: ...
ForwardBilling:
Type: AWS::Serverless::Function
Properties:
Role: !GetAtt ForwardBillingRole.Arn
@Koenighotze
Resources:
ForwardBillingRole:
Type: "AWS::IAM::Role"
Properties:
RoleName: !Sub "ForwardBillingRole-${Stage}"
AssumeRolePolicyDocument: ...
ManagedPolicyArns: ...
Policies:
-
PolicyName: !Sub "StoreBillingInS3Policy-${Stage}"
PolicyDocument: ...
ForwardBilling:
Type: AWS::Serverless::Function
Properties:
Role: !GetAtt ForwardBillingRole.Arn
@Koenighotze
USE A SECURITY
WATCHDOG
@Koenighotze
YOUR FUNCTION MAY BE
SMALL, BUT YOUR
DEPENDENCIES MAY NOT
@Koenighotze
$ du -s src/
8.0K src/
$ ls -lh lambda.zip
7.6M Mar 1 07:13 lambda.zip
@Koenighotze
$ du -s src/
8.0K src/
$ ls -lh lambda.zip
7.6M Mar 1 07:13 lambda.zip
@Koenighotze
@Koenighotze
@Koenighotze
@Koenighotze
@Koenighotze
SECURITY MUST BE PART
OF DEVELOPMENT RIGHT
FROM THE START
@Koenighotze
YOU BUILD IT
YOU RUN IT
@Koenighotze
YOU BUILD IT
YOU SECURE IT
AND THEN RUN IT
@Koenighotze
NEW ARCHITECTURES
NEW TRAPS
@Koenighotze
PROTECT YOUR LEGACY
@Koenighotze
BEWARE OF KILLING
YOUR VINTAGE BACKENDS
@Koenighotze
LAMBDA SCALES BY
REQUEST
@Koenighotze
…YOUR BACKEND
PROBABLY DOES NOT
@Koenighotze
“Billings are
submitted using
our legacy API”
@Koenighotze
@Koenighotze
…AND AT THE END OF A
MONTH…
@Koenighotze
@KoenighotzeGrigoriy Myasoyedov.1897
Please stop
this lambda
non-sense
But the HYPE!
@Koenighotze
ALWAYS USE
CONCURRENCY LIMITS
@Koenighotze
aws lambda put-function-concurrency
--function-name <NAME>
--reserved-concurrent-executions <N>
@Koenighotze
aws lambda put-function-concurrency
--function-name <NAME>
--reserved-concurrent-executions <N>
@Koenighotze
KEEP AN EYE ON METRIC
FOR CONCURRENCY FAILURES
@Koenighotze
BE AWARE BOTTLENECKS
TEND TO MOVE UPWARDS
@Koenighotze
PREFER ASYNCHRONOUS
INTEGRATION
@Koenighotze
ATTACK OF SELF DENIAL
@Koenighotze
CONCURRENCY IS
LIMITED
@Koenighotze
BE AWARE OF
PER ACCOUNT LIMITS
@Koenighotze
@Koenighotze
@Koenighotze
@Koenighotze
Projects BudgetUsers
@Koenighotze
Projects BudgetUsers
@Koenighotze
Projects BudgetUsers
@Koenighotze
Projects BudgetUsers
@Koenighotze
USE MULTIPLE ACCOUNTS
@Koenighotze
SPLIT PER BOUNDED
CONTEXT
@Koenighotze
Projects BudgetUsers
@Koenighotze
Projects
Project Account
Users
Users Account
Budget
Budget Account
@Koenighotze
PATTERNS, SHARING,
DEPENDENCIES
@Koenighotze
ONE LAMBDA WITH DISPATCH
MULTI LAMBDA - SHARED ZIP
MULTI LAMBDA - SHARED NOTHING
@Koenighotze
ONE LAMBDA WITH
DISPATCH
@Koenighotze
create Order
forward Order
cancel Order
Order ZIP
@Koenighotze
create Order
forward Order
cancel Order
Order ZIP
@Koenighotze
EASE OF TRANSITION
FLEXIBILITY
SECURITY
SIZING
DEPLOYMENT
COMPLEXITY
@Koenighotze
MULTI LAMBDA
-
SHARED ZIP
@Koenighotze
create Order
forward Order
cancel Order
Order ZIP
@Koenighotze
@Koenighotze
EASE OF TRANSITION
FLEXIBILITY
SECURITY
SIZING
DEPLOYMENT
COMPLEXITY
@Koenighotze
MULTI LAMBDA
-
SHARED NOTHING
@Koenighotze
create Order
forward Order
cancel Order
Order ZIP
@Koenighotze
Create
Order ZIP
Forward
Order ZIP
Cancel
Order ZIP
cancel Orderforward Ordercreate Order
@Koenighotze
cancel Orderforward Ordercreate Order
@Koenighotze
EASE OF TRANSITION
FLEXIBILITY
SECURITY
SIZING
DEPLOYMENT
COMPLEXITY
@Koenighotze
@Koenighotze
@Koenighotze
@Koenighotze
SUMMARY
@Koenighotze
SERVERLESS HYPE TRAIN
@Koenighotze
DESIGNING DISTRIBUTED
SYSTEMS IS STILL HARD
@Koenighotze
MOSTLY DIFFERENT
CHALLENGES
@Koenighotze
YOU NEED A CLOUD-SAVVY
OPERATIONS TEAM MEMBER
@Koenighotze
DEVOPSSEC DONE RIGHT
@Koenighotze
@Koenighotze
@Koenighotze
@Koenighotze
TOOLING IS STILL
EVOLVING
@Koenighotze
NOT A SILVER BULLET
@Koenighotze
SOMETIMES A SIMPLE
VM IS ENOUGH
@Koenighotze
SERVERLESS
@Koenighotze
LESS SERVERS THAT WE
NEED TO TAKE CARE OF
THANK YOU!
FEEDBACK? COMMENTS?
QUESTIONS?
@KOENIGHOTZE

Real world serverless - architecture, patterns and lessons learned