Lambda + API Gateway: Long executing function return before task finished

Question

I have a function in Lambda that executes for up to 30s depending on the inputs. That function is linked to an API gateway so I am able to call it through POST, unfortunatly the API Gateway is limited to 30s exaclty, so if my function runs for longer I get an "internal server error" back from my POST.

I see 2 solutions to this:

Create a "socket" rather then a simple POST which connects to the lambda function, not sure if this is even possible or how would this work.
Return a notification to api gateway before the lambda function is actually finished. This is acceptable for my use case, but I am not sure again how would this work.

The function is coded in Python3.6 I therefore have access to the event and context object.

statut · Accepted Answer · 2018-03-23 19:38:30Z

6

Return a notification to api gateway before the lambda function is actually finished. This is acceptable for my use case, but I am not sure again how would this work.

Unfortunately, you will not be able to return a result until the Lambda is finished. Otherwise, AWS will interrupt execution of Lambda if you try to do it, for example, via multithreading.

I suggest you create the following system:

API Gateway puts a request to Kinesis stream and returns a response of successfully putting. The response will contain SequenceNumber of the Kinesis Record. (How to do it)
Lambda is invoked by Kinesis Stream Event, processes the request and saves that your job is done (or completed result) to DynamoDB with SequenceNumber as id.
You invoke another API Gateway method and SequenceNumber to check the status of your job in DynamoDB (if necessary).

If you need just to run the Lambda without knowing of job result you can avoid using DynamoDB.

edited Mar 23, 2018 at 19:38

answered Mar 23, 2018 at 14:52

statut

9095 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

wrschneider Over a year ago

You could also simplify this and use two Lambdas, as in @cementblock's solution does below Also if you do this, the common RESTful pattern for long-running processes is something like: POST returns something a "201 accepted" with a Location header that you can use to poll for status with GET.

Samantha Atkins Over a year ago

Well you could have a non-REST reqular labmba that does the real work. Call it asynchronously and return a task_id to the caller. Have caller expect a PUT of task id and result from the lambda later. Several variations such as putting result in SQS or as SNS message that fires whatever sends the result. Much simpler that adding Kinesis and probably cheaper. But really this sort of thing shows that everything should not be a lambda much less behind API Gateway in the first place.

Samantha Atkins Over a year ago

In particular putting full web servers into a lambda via ECR under API Gateway is insanity.

balsick · Accepted Answer · 2018-03-25 14:49:35Z

5

statut approach is nice, but I'd like to give you an alternative: AWS step functions. Basically it does what statut's solution does but it might be more easy to deploy.

All you have to do is to create an AWS Step Functions state machine that has your lambda function as its only task. AWS Step functions intrinsically works asynchronously, and has methods to know the status and, eventually, the result. Check the docs for more info.

answered Mar 25, 2018 at 14:49

balsick

1,2111 gold badge11 silver badges29 bronze badges

1 Comment

Tobi Over a year ago

worked great for me with step functions, even if I didn't even knew them 2 hours ago... github.com/mavi888/stepfunction-triggers was pretty useful as a starting point

cementblocks · Accepted Answer · 2018-03-23 15:14:26Z

2

A very simple solution would be for your lambda function to just asynchronously invoke another lambda function and then return a result to API Gateway.

answered Mar 23, 2018 at 15:14

cementblocks

4,64520 silver badges26 bronze badges

2 Comments

0xtuytuy Over a year ago

sure that sounds good but I am not sure how to connect 2 lambda functions. SNS ? Any link to another topic on this ?

statut Over a year ago

use invoke method with Event as Invocation Type docs.aws.amazon.com/lambda/latest/dg/API_Invoke.html

grantr · Accepted Answer · 2022-09-07 21:49:22Z

Here's a real example of executing one lambda function asynchronously from another lambda function:

import json
import boto3


def lambda_handler(event, context):
    
    payload = {}
    
    ##take the body of the request and convert to string    
    body = json.loads(event['body'])
    
    ##build new payload with body and anything else your function downstream needs
    payload['body'] = json.dumps(body)
    payload['httpMethod'] = 'POST'
    payloadStr = json.dumps(payload)
    
    client = boto3.client('lambda')
    client.invoke(
        FunctionName='container name',
        InvocationType="Event",
        Payload=payloadStr
    )


    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

In this example, the function invokes the other one and returns right away.

invoke documentation: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/lambda.html#Lambda.Client.invoke

Collectives™ on Stack Overflow

Lambda + API Gateway: Long executing function return before task finished

4 Answers 4

3 Comments

1 Comment

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

3 Comments

1 Comment

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related