AWS lambda: Execute function on timeout

Question

I am developing a lambda function that migrates logs from an SFTP server to an S3 bucket.

Due to the size of the logs, the function sometimes is timing out - even though I have set the maximum timeout of 15 minutes.

try:
   logger.info(f'Migrating {log_name}... ')
   transfer_to_s3(log_name, sftp)
   logger.info(f'{log_name} was migrated succesfully ')

If transfer_to_s3() fails due to timeoutlogger.info(f'{log_name} was migrated succesfully') line won't be executed.

I want to ensure that in this scenario, I will somehow know that a log was not migrated due to timeout.

Is there a way to force lambda to perform an action, before exiting, in the case of a timeout?

"force lambda do something, before timing out?" what does it mean? You can program any logic you want. — Marcin
– Marcin, Commented Oct 25, 2021 at 9:54
@Dimitris you can increase the lambda function timeout in Configuration --> General Setting — Jebasuthan
– Jebasuthan, Commented Oct 25, 2021 at 9:59
@jebasuthan My question was not phrased correctly. Will edit it. I have already set lambda timeout to 15 minutes which is the maximum allowed. — Dimi
– Dimi, Commented Oct 25, 2021 at 10:32

Marcin · Accepted Answer · 2021-10-25 10:47:11Z

1

Probably a better way would be to use SQS for that:

Logo info ---> SQS queue ---> Lambda function

If lambda successful moves the files, it removes the log info from SQS queue. If it fails, the log info persists in the SQS queue (or goes to DLQ for special handling), so the next lambda invocation can handle it.

answered Oct 25, 2021 at 10:47

Marcin

241k16 gold badges315 silver badges368 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Dimi Over a year ago

Just to understand, do you recommend something like: SFTP Server ---> Lambda Function ---> SQS queue ---> Lambda function ? I do not quite get it to be honest

Marcin Over a year ago

@Dimitris it's not specified in your question how your server works. But if cannot interact with sqs directly, then you can add extra lambda as you described.

Collectives™ on Stack Overflow

AWS lambda: Execute function on timeout

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related