AWS python lambda function:No module named requests

Question

I am fairly new to AWS and I am having some issues. Here is my code:

from __future__ import print_function
from urllib2 import Request, urlopen, URLError
import requests
import boto3
import json

def lambda_handler(event, context):
    url = "https://globalcurrencies.xignite.com/xGlobalCurrencies.json/GetHistoricalRatesRange?Symbol=BTCUSD&PriceType=Mid&StartDate=01/01/2017&EndDate=10/27/2017&PeriodType=Daily&FixingTime=22:00&_token=some_token_xyz"
    response = requests.get(url).json()
    # print json.dumps(response, indent=4) # gives a syntax error
    return response

Name of the file is lambda_function.py; I have checked similar problems on stackoverflow and some mentioned that I have to change the file naming. But it didn't help. Here is how python method was named:
Here is the error I am getting: START RequestId: cf24e9be-bbef-11e7-97b4-d9b586307f3e Version: $LATEST Unable to import module 'lambda_function': No module named requests And when try to print it gives me a syntax error. Sorry for the formatting. Any suggestions?

requests isn't in the standard library. You'll need to look into how to provide dependencies on that platform; it's common to use a requirements.txt file to specify them, for example. Have a look at e.g. docs.aws.amazon.com/lambda/latest/dg/… — jonrsharpe
– jonrsharpe, Commented Oct 28, 2017 at 15:05
@jonrsharpe I have virtual environment setup on my machine. Should I just upload everything on do pip install -r requirements.txt ? — Zafar
– Zafar, Commented Oct 28, 2017 at 15:11

Nilo_DS · Accepted Answer · 2019-11-19 22:10:32Z

51

requests is not a standard library in AWS lambda.

So two ways to solve this:

1- Import it from the Botocore libraries stack as: (Option to be deprecated after the answer was given)

from botocore.vendored import requests

Here there is a list of all the available libraries to import in lambda

2- Create a deployment package with virtualenv.

edited Nov 19, 2019 at 22:10

answered May 21, 2018 at 19:20

Nilo_DS

9497 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

ilvidel Over a year ago

Option number 1 shouts this message: /var/runtime/botocore/vendored/requests/api.py:67: DeprecationWarning: You are using the get() function from 'botocore.vendored.requests'. This is not a public API in botocore and will be removed in the future. Additionally, this version of requests is out of date. We recommend you install the requests package, 'import requests' directly, and use the requests.get() function instead.

Nilo_DS Over a year ago

I just added the heads up. Thanks @ilvidel for the heads up!

McGlothlin Over a year ago

According to an updated version of the warning: "This dependency was removed from Botocore and will be removed from Lambda after 2020/03/31." Stick with option 2.

Paolo · Accepted Answer · 2023-05-19 13:53:40Z

35

N.B requests is already included in lambda functions with runtime Python 3.7:

In fact:

from pkg_resources import working_set

def lambda_handler(event, context):
    for p in list(working_set):
        print(p.project_name, p.version)

urllib3 1.26.6 six 1.16.0 s3transfer 0.6.0 requests 2.26.0 python-dateutil 2.8.2 jmespath 1.0.1 idna 2.10 charset-normalizer 2.0.12 chardet 4.0.0 certifi 2020.11.8 botocore 1.29.90 boto3 1.26.90 setuptools 47.1.0 pip 22.0.4

For all other runtimes, see below.

If you want to include libraries which are not part of Python's standard library, such as requests, you can use lambda's layers.

1. Create the zip

This is a zip which contains all the libraries you want the lambda function to use. First, create a folder called python:

$ mkdir python
$ cd python

then, install the Python libraries you need in there. You can do this either with a single library:

$ pip install --target . requests

or with a list of libraries (requirements.txt)

$ pip install --target . -r requirements.txt

finally, zip up everything.

... in Bash:

$ zip -r dependencies.zip ../python

... in Powershell:

$ Compress-Archive -Path . -DestinationPath dependencies.zip

2. Create a layer

You can do this either in the AWS console or in the CLI. Follow these instructions.

3. Add the layer to the lambda function

You can do this either with the Add a layer option in the lambda page, or in the CLI. Follow these instructions.

edited May 19, 2023 at 13:53

answered Oct 21, 2020 at 11:16

Paolo

26.6k8 gold badges51 silver badges88 bronze badges

3 Comments

rlucatoor Over a year ago

Great answer! In my case, step number 2 threw the error ERROR: Can not combine '--user' and '--target'. As it turns out, this happens if Python is installed as an app from the store and in some other cases. The issue is easily solved by adding a --no-user at the end of the command: pip install --target . requests --no-user. Source

Philosopher-Programmer Over a year ago

If you're using the AWS SAM Workflow, create the directory with dependencies described in part 1 of answer, then build the layer by including the Metadata property in the Layer resource in the SAM Template: docs.aws.amazon.com/serverless-application-model/latest/… . Then attach the layer to the function's Layers property: docs.aws.amazon.com/serverless-application-model/latest/… . There's no need to zip and upload using CLI.

kchuying Over a year ago

Tested for lambda runtime python 3.9, pip install --target . requests==2.29.0 would work

Mark Kell · Accepted Answer · 2017-11-15 10:55:10Z

14

This is because it is missing the requests library when running in the lambda - its likely that its installed globally on your local machine. If you run:
pip install requests -t .
in the same directory as your source code it will install the requests package in that directory then you can upload it to lambda along with your lambda_function.py. You may need to do the same for boto3 and json:
pip install boto3 -t .
pip install json -t .

answered Nov 15, 2017 at 10:55

Mark Kell

1653 bronze badges

2 Comments

Zafar Over a year ago

I am using virtual environment in my machine. How can I upload and retrieve the dependancy from requirements.txt?

nnsense Over a year ago

boto3 and json are both already included in AWS lambda, see gist.github.com/gene1wood/4a052f39490fae00e0c3

OmegaOdie · Accepted Answer · 2019-11-18 10:57:19Z

8

You want to use Layers. You download a python module e.g pip install requests -t . then simply put all the module folders into a folder called python and zip it and upload it in the lambda layers section of the aws console. Add it to your lambda function and it will work. The dir structure is very important so for requests when you unzip the folder it should be RequestLayer/python/requests/"requests and all the other folders that are part of that package". This guy gives a good step by step.

answered Nov 18, 2019 at 10:57

OmegaOdie

3772 silver badges8 bronze badges

2 Comments

LoMaPh Over a year ago

What should be the folder structure? What do you mean by "RequestLayer"? I tries your version and a bunch of others (with or without "RequesLayer", with or without "requests") but no luck.

ninsonTan Over a year ago

@LoMaPh I tried without RequestLayer and it works. So after installing the package using pip, place all content within a folder called python, zip that and add that as a Layer.

dwp7kp · Accepted Answer · 2023-03-30 20:55:15Z

7

Use urllib3 instead

While yes, requests is not a standard library in AWS lambda, there are alternatives within the standard library that will allow you to avoid Lambda layers.

If you check the list of libraries that Nilo_DS linked (https://gist.github.com/gene1wood/4a052f39490fae00e0c3), you'll find that urllib3 is available.

import requests

requests.post(<url>, data=json.dumps(<message>))

can be rewritten as

import urllib3

http = urllib3.PoolManager()
http.request('POST', '<url>',
             headers={'Content-Type': 'application/json'},
             body=json.dumps(<message>))

and similar changes for a get request.

answered Mar 30, 2023 at 20:55

dwp7kp

791 silver badge1 bronze badge

Comments

kgrvamsi · Accepted Answer · 2024-02-20 19:50:55Z

3

For Python 3.12 lambda runtime use the following import for requests module

from pip._vendor import requests

answered Feb 20, 2024 at 19:50

kgrvamsi

516 bronze badges

Comments

artifex_knowledge · Accepted Answer · 2019-11-25 14:40:51Z

2

For future readers using Python: If you created your Lambda app directly in Cloud9, you would notice that there is a virtual environment for your app. To install requests (or any other package for that matter), do the following:

Right-click on the folder for your app
Select "Open terminal here"
In the terminal type source venv/bin/activate to activate your virtual environment
Type pip3 install requests to install requests

That is it. You can now use requests with import requests as per normal.

answered Nov 25, 2019 at 14:40

artifex_knowledge

5726 silver badges16 bronze badges

1 Comment

Anum Sheraz Over a year ago

I don't see any such option "Open terminal here"

vedat · Accepted Answer · 2018-02-26 16:30:49Z

1

'requests' module is not in your 'zip' file that your are trying to install. you have to put all modules into the zip file by 'pip install module_name(such as requests) -t .'

answered Feb 26, 2018 at 16:30

vedat

1,29910 silver badges10 bronze badges

Comments

Navjot Kashi · Accepted Answer · 2024-02-20 09:00:05Z

0

POST: 20Feb24

You can use the http.client like below.

This solves the purpose and you don't have to install anything else for http.client.

For scheduler cronjob use https://docs.aws.amazon.com/lambda/latest/dg/services-cloudwatchevents-expressions.html

and https://stackoverflow.com/a/68146073/11207658

PS: cronjob uses the UTC timezone. so apply it accordingly

edited Feb 20, 2024 at 9:00

answered Feb 20, 2024 at 8:03

Navjot Kashi

1511 silver badge7 bronze badges

Comments

Carlos Cortez Bazan · Accepted Answer · 2017-10-28 15:22:54Z

-9

you have to name your lambda this way in python code:

def lambda_function(event, context):

and in lambda console handler:

main.lambda_function

for your request error, must have the folder of that module on your .zip before you upload to lambda.

answered Oct 28, 2017 at 15:22

Carlos Cortez Bazan

11 bronze badge

2 Comments

Zafar Over a year ago

No, there's another error: Unable to import module 'main': No module named requests

Carlos Cortez Bazan Over a year ago

do you have the .zip folder with the request library inside? before upload to lambda? thats the problem

Collectives™ on Stack Overflow

AWS python lambda function:No module named requests

10 Answers 10

3 Comments

3 Comments

2 Comments

2 Comments

Comments

Comments

1 Comment

Comments

Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

10 Answers 10

3 Comments

3 Comments

2 Comments

2 Comments

Comments

Comments

1 Comment

Comments

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related