AWS Lambda function does not recognize my Python dependency layer

Question

My goal is to upload a small layer for an aws lambda function using the python 3.7 runtime that stores python dependencies, but I cannot seem to get it to recognize the modules.

I created a Makefile to generate the zip file

create-lambda-layer:
    docker run -dit --name AL amazonlinux:latest
    docker cp requirements.in AL:/root/requirements.in
    docker exec -it AL bash -c "yum update -y \
        && yum install -y python3.7 zip  \ 
        && python3 -m pip install --upgrade pip \
        && pip install wheel \
        && mkdir -p /root/lambda/ \
        && pip install -r /root/requirements.in -t /root/lambda/ \
        && cd /root/lambda/ \
        && zip -r python.zip ."
    docker cp AL:/root/lambda/python.zip .
    docker stop AL
    docker rm AL

and my required modules are in requirements.in in the same folder

holidays==0.11.2
python-gitlab==2.10.1

I put the layer in a service and deploy it via the serverless framework, but I also tested to upload the same .zip directly using the AWS console and it failed. My lambda function source file (linked correctly to the layer) is solely:

import gitlab
import holidays

print("holidays: ", holidays.__version__)
print("gitlab", gitlab.__version__)

def lambda_handler(event, context): 
    return()

I get the following errors (analogously for holidays):

{
  "errorMessage": "Unable to import module 'lambda_function': No module named 'gitlab'",
  "errorType": "Runtime.ImportModuleError",
  "stackTrace": []
}

The content of my python.zip is not remarkable to me, it just contains the dependencies it should:

PyMeeus-0.5.11.dist-info            hijri_converter                        pytz-2021.1.dist-info
__pycache__                         hijri_converter-2.2.1.dist-info        requests
bin                                 holidays                               requests-2.26.0.dist-info
certifi                             holidays-0.11.2.dist-info              requests_toolbelt
certifi-2021.5.30.dist-info         idna                                   requests_toolbelt-0.9.1.dist-info
charset_normalizer                  idna-3.2.dist-info                     six-1.16.0.dist-info
charset_normalizer-2.0.4.dist-info  korean_lunar_calendar                  six.py
convertdate                         korean_lunar_calendar-0.2.1.dist-info  tests
convertdate-2.3.2.dist-info         pymeeus                                urllib3
dateutil                            python_dateutil-2.8.2.dist-info        urllib3-1.26.6.dist-info
docs                                python_gitlab-2.10.1.dist-info
gitlab                              pytz

I wonder why this is. I have previously tried to create the same layer zip using ubuntu and it worked, but to implement it officially, I cannot use an ubuntu image.

Does anyone have an idea what I can do to track down the problem or how to solve this? Could it have any impact that I use pip as root while installing the dependencies?

Thanks!

djvg · Accepted Answer · 2024-06-07 10:08:22Z

7

I had this SAME problem and it took me a week to solve. I think this answer can help you: Import libraries in lambda layers

Here is the key stuff:

You want to make sure your .zip follows this folder structure when unzipped

python/lib/python3.6/site-packages/{LibrariesGoHere}.

Upload that zip, make sure the layer is added to the Lambda function and you should be good to go.

Also, I think your question would get better attention if you update your tag to "aws-lambda-layers"

edited Jun 7, 2024 at 10:08

djvg

14.8k7 gold badges84 silver badges120 bronze badges

answered Sep 9, 2021 at 21:59

Mickle-The-Pickle

2613 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Carmella Over a year ago

Thank you very much! I'm happy, it worked! Although I'm a bit puzzled, because I have the same directory structure (just python/{modules}) inside the ubuntu-made layer and when using this layer, the lambda does not have any problems finding the modules and I wrongly excluded that it would be because of the zip structure. Magic? Thanks also for your tip regarding the tags, I'm a first time poster, just lurked here :) But it seems someone nice already tagged it correctly for me.

Collectives™ on Stack Overflow

AWS Lambda function does not recognize my Python dependency layer

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related