AWS lambda: Invoking locally a python lambda function with access to s3

Question

I just begun learning to use amazon's serverless framework to develop python lambda functions locally on my linux PC, before deploying. The process involves initializing a file structure by sam init, then building the app by sam build and finally invoking the function with something ike sam local invoke.

I have trouble accessing my s3 buckets when invoking the function like this, as I demonstrate below. I would be grateful if somebody could pinpoint why the erroneous example does not work. Apologies for the lengthy text.

Working example

Suppose that my app is like this:

# app.py

import json

def lambda_handler(event, context):
    return {
        "statusCode": 200,
        "body": json.dumps({
            "message": "hello world",
        }),
    }

Sure enough, if I run sam build; sudo sam local invoke HelloWorldFunction --no-event, this will run succesfully and return:

>sam build
Building resource 'HelloWorldFunction'
Running PythonPipBuilder:ResolveDependencies
Running PythonPipBuilder:CopySource

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Deploy: sam deploy --guided

>sudo sam local invoke HelloWorldFunction --no-event
Invoking app.lambda_handler (python3.8)

Fetching lambci/lambda:python3.8 Docker container image......
Mounting /home/username/Tmp/myapp/myapp/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated inside runtime container
START RequestId: af24271f-50f0-15b7-2e09-f11a513c04d4 Version: $LATEST
END RequestId: af24271f-50f0-15b7-2e09-f11a513c04d4
REPORT RequestId: af24271f-50f0-15b7-2e09-f11a513c04d4  Init Duration: 265.92 ms    Duration: 1.94 ms   Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 25 MB  

{"statusCode":200,"body":"{\"message\": \"hello world\"}"}

Example variation that causes an error

However, when boto3 is involved like below, invoking does not work.

  1 # app.py                                                                       
  2                                                                                
  3 import json                                                                    
  4 import boto3                                                                                        
  5                                                                                
  6 s3 = boto3.resource('s3')                                                      
  7 buckets = [bucket.name for bucket in s3.buckets.all()]                         
  8 print('a bucket is', buckets[0])                                               
  9                                                                                
 10 def lambda_handler(event, context):                                            
 11     return {                                                                   
 12         "statusCode": 200,                                                     
 13         "body": json.dumps({                                                   
 14             "message": "hello world",                                          
 15         }),                                                                    
 16     }

sam local build runs fine, however sudo sam local invoke HelloWorldFunction --no-event gives this error:

>sudo sam local invoke HelloWorldFunction --no-event
Invoking app.lambda_handler (python3.8)

Fetching lambci/lambda:python3.8 Docker container image......
Mounting /home/username/Tmp/myapp/myapp/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated inside runtime container
[ERROR] ClientError: An error occurred (InvalidAccessKeyId) when calling the ListBuckets operation: The AWS Access Key Id you provided does not exist in our records.
Traceback (most recent call last):
  File "/var/lang/lib/python3.8/imp.py", line 234, in load_module
    return load_source(name, filename, file)
  File "/var/lang/lib/python3.8/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 702, in _load
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/var/task/app.py", line 7, in <module>
    buckets = [bucket.name for bucket in s3.buckets.all()]
  File "/var/task/app.py", line 7, in <listcomp>
    buckets = [bucket.name for bucket in s3.buckets.all()]
  File "/var/runtime/boto3/resources/collection.py", line 83, in __iter__
    for page in self.pages():
  File "/var/runtime/boto3/resources/collection.py", line 161, in pages
    pages = [getattr(client, self._py_operation_name)(**params)]
  File "/var/runtime/botocore/client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/runtime/botocore/client.py", line 626, in _make_api_call
    raise error_class(parsed_response, operation_name)

{"errorType":"ClientError","errorMessage":"An error occurred (InvalidAccessKeyId) when calling the ListBuckets operation: The AWS Access Key Id you provided does not exist in our records.","stackTrace":["  File \"/var/lang/lib/python3.8/imp.py\", line 234, in load_module\n    return load_source(name, filename, file)\n","  File \"/var/lang/lib/python3.8/imp.py\", line 171, in load_source\n    module = _load(spec)\n","  File \"\u003cfrozen importlib._bootstrap\u003e\", line 702, in _load\n","  File \"\u003cfrozen importlib._bootstrap\u003e\", line 671, in _load_unlocked\n","  File \"\u003cfrozen importlib._bootstrap_external\u003e\", line 783, in exec_module\n","  File \"\u003cfrozen importlib._bootstrap\u003e\", line 219, in _call_with_frames_removed\n","  File \"/var/task/app.py\", line 7, in \u003cmodule\u003e\n    buckets = [bucket.name for bucket in s3.buckets.all()]\n","  File \"/var/task/app.py\", line 7, in \u003clistcomp\u003e\n    buckets = [bucket.name for bucket in s3.buckets.all()]\n","  File \"/var/runtime/boto3/resources/collection.py\", line 83, in __iter__\n    for page in self.pages():\n","  File \"/var/runtime/boto3/resources/collection.py\", line 161, in pages\n    pages = [getattr(client, self._py_operation_name)(**params)]\n","  File \"/var/runtime/botocore/client.py\", line 316, in _api_call\n    return self._make_api_call(operation_name, kwargs)\n","  File \"/var/runtime/botocore/client.py\", line 626, in _make_api_call\n    raise error_class(parsed_response, operation_name)\n"]}
START RequestId: e8a504ed-3f84-13d2-5f8a-846c6aa34da8 Version: $LATEST
END RequestId: e8a504ed-3f84-13d2-5f8a-846c6aa34da8
REPORT RequestId: e8a504ed-3f84-13d2-5f8a-846c6aa34da8  Init Duration: 1035.42 ms   Duration: 5.52 ms   Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 41 MB

The strange thing is that if I run this with python everything is fine:

>python app.py 
a bucket is bucket-fjlkajsdr

or

> aws s3 ls
2020-04-29 13:31:49 bucket-fjlkajsdr
2020-04-29 13:59:37 bucket2-lkjlasdfasdf
2020-04-28 16:07:37 bucket3-lkadsfasd

So I do have the right credentials installed and have access to my s3 buckets from my PC.

My question is: Could somebody please explain why invoking the function locally does not work?

template.yaml

Here also you can find the template.yaml which is placed in the app folder:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  myapp

  Sample SAM Template for myapp

# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
  Function:
    Timeout: 30

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      CodeUri: hello_world/
      Handler: app.lambda_handler
      Runtime: python3.8
      Policies: AmazonS3FullAccess
      Events:
        HelloWorld:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            Path: /hello
            Method: get

Outputs:
  # ServerlessRestApi is an implicit API created out of Events key under Serverless::Function
  # Find out more about other implicit resources you can reference within SAM
  # https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api
  HelloWorldApi:
    Description: "API Gateway endpoint URL for Prod stage for Hello World function"
    Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/"
  HelloWorldFunction:
    Description: "Hello World Lambda Function ARN"
    Value: !GetAtt HelloWorldFunction.Arn
  HelloWorldFunctionIamRole:
    Description: "Implicit IAM Role created for Hello World function"
    Value: !GetAtt HelloWorldFunctionRole.Arn

This is a very well-written question. Should be a model for Stack Overflow newbies. Well done. — jarmod
– jarmod, Commented Apr 29, 2020 at 15:37

jarmod · Accepted Answer · 2020-04-29 15:33:57Z

3

I think it's because you are using sudo to invoke sam local and hence the effective user is root, so the AWS SDK is picking up root's ~/.aws/credentials file rather than your ~/.aws/credentials file.

PS I wouldn't expect you to have to sudo to run SAM.

answered Apr 29, 2020 at 15:33

jarmod

79.8k18 gold badges132 silver badges137 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

geo909 Over a year ago

That was precisely it. Thank you, that was killing me for hours. I am using sudo because the invoke parameter involves docker which seem to require sudo. In fact, running without sudo simply gives me: "Error: Running AWS SAM projects locally requires Docker. Have you got it installed?"

jarmod Over a year ago

I suspect that can be solved too if Docker's installed to run as a service and your user is added to the docker group.

geo909 Over a year ago

Thank you again, Indeed, I also just found this information: docs.docker.com/engine/install/linux-postinstall . I am a little worried with that warning, as I don't fully understand the implications they explain in docs.docker.com/engine/security/security/… But I will go with that for now

Collectives™ on Stack Overflow

AWS lambda: Invoking locally a python lambda function with access to s3

Working example

Example variation that causes an error

template.yaml

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

Working example

Example variation that causes an error

template.yaml

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related