Deploying a python package to AWS Lambda via Cloudformation, CodeBuild and CodePipeline

Question

I want to setup a CI/CD pipeline for my AWS infrastructure and a AWS Lambda function. The idea is to have everything in code, version-controlled and automated. I just want to git push to a repository and have CodePipeline take over from there, updating my infrastructure, running tests and, if successful, updating my Lambda function with the latest code.

I'm basing my CloudFormation template on this excellent example. It looks like this:

AWSTemplateFormatVersion: 2010-09-09
Description: playground pipeline 1
Parameters:
  SourceRepositoryName:
    Type: String
    Default: lambda-playground
  SourceBranchName:
    Type: String
    Default: master

Resources:
  ArtifactsBucket:
    Type: AWS::S3::Bucket
    DependsOn: CloudFormationRole
    DeletionPolicy: Delete
    Properties:
      BucketName: lambda-playground-artifacts

  CodeBuildRole:
    Type: AWS::IAM::Role
    DependsOn: CloudFormationRole
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action:
              - sts:AssumeRole
            Principal:
              Service:
                - codebuild.amazonaws.com
      Policies:
        - PolicyName: ServiceRole
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Sid: CloudWatchWriteLogsPolicy
                Effect: Allow
                Action:
                  - logs:CreateLogGroup
                  - logs:CreateLogStream
                  - logs:PutLogEvents
                Resource: '*'
              - Sid: CodeCommitPullPolicy
                Effect: Allow
                Action:
                  - codecommit:GitPull
                Resource: '*'
              - Sid: S3GetObjectPolicy
                Effect: Allow
                Action:
                  - s3:GetObject
                  - s3:GetObjectVersion
                Resource: '*'
              - Sid: S3PutObjectPolicy
                Effect: Allow
                Action:
                  - s3:PutObject
                Resource: '*'

  CodePipelineRole:
    Type: AWS::IAM::Role
    DependsOn: CloudFormationRole
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action:
              - sts:AssumeRole
            Principal:
              Service:
                - codepipeline.amazonaws.com
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/AdministratorAccess

  CloudFormationRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action:
              - sts:AssumeRole
            Principal:
              Service:
                - cloudformation.amazonaws.com
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/AdministratorAccess

  CodeCommitRepository:
    Type: AWS::CodeCommit::Repository
    Properties:
      RepositoryName: !Ref SourceRepositoryName

  CodeBuildProject:
    Type: AWS::CodeBuild::Project
    DependsOn: CloudFormationRole
    Properties:
      Description: A playground of Lambda
      Artifacts:
        Type: CODEPIPELINE
      Environment:
        ComputeType: BUILD_GENERAL1_SMALL
        Image: aws/codebuild/python:2.7.12
        Type: LINUX_CONTAINER
      Name: lambda-playground
      ServiceRole: !GetAtt CodeBuildRole.Arn
      Source:
        Type: CODEPIPELINE
      TimeoutInMinutes: 5

  CodePipeline:
    Type: AWS::CodePipeline::Pipeline
    Properties:
      ArtifactStore:
        Type: S3
        Location: !Ref ArtifactsBucket
      Name: !Ref AWS::StackName
      RestartExecutionOnUpdate: true
      RoleArn: !GetAtt CodePipelineRole.Arn
      Stages:
        - Name: Source
          Actions:
            - Name: Source
              ActionTypeId:
                Category: Source
                Owner: AWS
                Provider: CodeCommit
                Version: 1
              Configuration:
                RepositoryName: !Ref SourceRepositoryName
                BranchName: !Ref SourceBranchName
              OutputArtifacts:
                - Name: SourceOutput
        - Name: PipelineDeploy
          Actions:
            - Name: UpdatePipeline
              ActionTypeId:
                Category: Deploy
                Owner: AWS
                Provider: CloudFormation
                Version: 1
              Configuration:
                ActionMode: CREATE_UPDATE
                Capabilities: CAPABILITY_IAM
                RoleArn: !GetAtt CloudFormationRole.Arn
                StackName: !Ref AWS::StackName
                TemplatePath: SourceOutput::infra.yml
              InputArtifacts:
                - Name: SourceOutput
        - Name: Build
          Actions:
            - Name: BuildAndTest
              ActionTypeId:
                Category: Build
                Owner: AWS
                Provider: CodeBuild
                Version: 1
              Configuration:
                ProjectName: !Ref CodeBuildProject
              InputArtifacts:
                - Name: SourceOutput
              OutputArtifacts:
                - Name: BuildOutput

  LambdaFunction:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        S3Bucket: !Ref ArtifactsBucket
        S3Key: !Ref BuildOutput # DOES NOT WORK
      FunctionName: playground-fc
      Handler: src.main.handler
      # TODO: Role: foo
      Runtime: python2.7

Outputs:
  ArtifactsBucketURL:
    Description: Artifacts bucket URL
    Value: !GetAtt ArtifactsBucket.WebsiteURL
  RepositoryURL:
    Description: SSH URL of the repository
    Value: !GetAtt CodeCommitRepository.CloneUrlSsh

So I have a CodePipeline with 3 stages - Source, which grabs code from a CodeCommit repo, PipelineDeploy, which updates my CloudFormation stack if necessary and Build, which runs the configured CodeBuild project.

My buildspec.yml is here:

version: 0.1

phases:
  install:
    commands:
      - pip install -r requirements.txt -t lib
  pre_build:
    commands:
      - python lib/pytest.py src
artifacts:
  type: zip
  files:
    - src/**/*
    - lib/**/*

It just installs the necessary libraries, runs the tests via pytest and creates a deployment zip. This zip file is then the OutputArtifact of the Build stage and gets stored in the ArtifactsBucket. However, each time, it gets a unique name (e.g. dfVV6Uh), which makes sense, but I don't know how to reference it in the LambdaFunction -> Properties -> Code -> S3Key field.

So my question is, how can I create a stack/pipeline, that after doing all the steps, will deploy the latest version to my AWS Lambda function? Is there a way to maybe use CodeDeploy to do this? What is the best practice here?

wjordan · Accepted Answer · 2017-02-24 04:29:24Z

4

You can use a Parameter Override with Fn::GetArtifactAtt and the ObjectKey attribute to dynamically provide the the name of the artifact .zip generated by AWS CodePipeline to your CloudFormation deploy action.

Using your example, the configuration for your UpdatePipeline CloudFormation deploy action would look something like this:

Configuration:
  ActionMode: CREATE_UPDATE
  Capabilities: CAPABILITY_IAM
  RoleArn: !GetAtt CloudFormationRole.Arn
  StackName: !Ref AWS::StackName
  TemplatePath: SourceOutput::infra.yml
  ParameterOverrides:
    {
      "LambdaKey" : { "Fn::GetArtifactAtt" : ["LambdaFunctionSource", "ObjectKey"]}
    }
InputArtifacts:
- Name: SourceOutput
- Name: BuildOutput

Then, declare and then reference the LambdaKey Parameter within your CloudFormation stack template:

Parameters:
  LambdaKey:
    Type: String
  # ...
Resources:
  LambdaFunction:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        S3Bucket: !Ref ArtifactsBucket
        S3Key: !Ref LambdaKey
      # ...

answered Feb 24, 2017 at 4:29

wjordan

20.5k3 gold badges92 silver badges101 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Milan Cermak Over a year ago

So the idea is to have teh AWS::Lambda::Function declared in a separate CloudFormation template? Originally, I thought of having everything in one template, that's why the UpdatePipeline (a misleading name, now I realize) step is second in my pipeline. My Lambda function code is the output of the CodeBuild step, third one in the pipeline, so I guess I can't reference it in the second step (I haven't tried yet).

wjordan Over a year ago

I based my answer on the template you provided in your question, which referenced SourceOutput::infra.yml. It's possible to set the template in your source repo as the same template you use to define your Code Pipeline resource, to make it self-referencing. See my repo at github.com/wjordan/aws-starter-kit for an fully self-contained example using GitHub.

Milan Cermak Over a year ago

I studied your aws-starter-kit and also attempted to modify my template based on your advice, but wasn't successful. The problem is I want to have everything in one template so I can bootstrap my infrastructure and update it continuously. To do that, when creating the Lambda function resource, I don't have the S3Key yet, that comes only as an output of my thirds pipeline step - the Build action. I guess I'll just create a dummy deployment package for the bootstrap step.

Milan Cermak Over a year ago

Took me a while to realize, but you were right. I'm quite new to CFN, I wasn't sure what the best practice is, but I've now split my infrastructure to two templates - on for the pipeline and one for the lambda function. I've added another deploy step to the pipeline using the way you suggested and it worked. Thanks a lot!

Milan Cermak Over a year ago

I now created a repo with the whole solution.

awsnitin · Accepted Answer · 2017-02-24 00:42:50Z

1

There is an example of how to do achieve something similar (deploying lambda functions via CodePipeline/CodeBuild). http://docs.aws.amazon.com/lambda/latest/dg/automating-deployment.html

This example is for lambda functions written in NodeJS, but the basic idea is the same. You use CloudFormation to deploy/update your lambda functions after you've build your artifact via CodeBuild and let CodePipeline manage artifact propagation within stages.

Let me know if this helps.

answered Feb 24, 2017 at 0:42

awsnitin

6363 silver badges6 bronze badges

Collectives™ on Stack Overflow

Deploying a python package to AWS Lambda via Cloudformation, CodeBuild and CodePipeline

2 Answers 2

5 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related