When can a Node.js AWS Lambda Function Exit without Invoking its Callback?

Question

The short version of my question: What, other than a timeout, can trigger a Node.js based AWS Lambda function to exit prior to the handler's callback function or context.success|done|fail being called?

The long (convoluted) version is: We've inherited an AWS Lambda function that we didn't write. We're seeing behavior on cold starts (the first invocation after a deploy) where the Lambda function will return a null response to API Gateway before the program has had a chance to invoke the handler's callback.

Also, during the second invocation of the function, we see the following error in our log.

[ERROR] [1626907621359] LAMBDA_RUNTIME Failed to post handler success response. Http response code: 400.

But this second response is processed normally and correctly.

A simplified version of the function looks like this

const otherCodeThatWrapsAbunchOfStuff = require('./other-code')

exports.handler = otherCodeThatWrapsAbunchOfStuff(
    function handler (event, context, callback) {
        callback(null, `Hello World`)
    })
}

This otherCodeThatWrapsAbunchOfStuff returns a wrapped handler function, and also wraps the callback.

Using some local debugging our best reasoning about things right now is

During a cold start, this code in otherCodeThatWrapsAbunchOfStuff we've inherited schedules the callback to be invoked via async means
[something happens] and the Lambda returns a null
The Lambda freezes
The next request comes in and the Lambda unfreezes
The original callback fires, resulting in the LAMBDA_RUNTIME error
The callback for the second invocation fires, resulting in a normal response

I realize you can't debug this program for me -- what I'm interesting in knowing is what sort of scenarios (other than a timeout -- we've bumped the Lambda timeout value and this "return null on cold start" behavior still happens and happens well before the timeout is reached) can cause an AWS Lambda function to "exit/respond early" without the callback being called.

this is not a good way of doing it. This is a hard stop. It will exit with an error - error : Error: Runtime exited without providing a reason. Runtime.ExitError — Akhil Ghatiki
– Akhil Ghatiki, Commented Sep 6, 2022 at 7:24

Alana Storm · Accepted Answer · 2021-07-23 18:18:49Z

I think we've got this one solved -- here's the magic doc that helped us understand the execution model of the callback based handlers.

For non-async handlers, function execution continues until the event loop is empty or the function times out. The response isn't sent to the invoker until all event loop tasks are finished

First, when your function is invoked, Lambda will

Run the code in your handler
Wait until there's nothing left to process in the event loop
Send a response

That is, our my main misunderstanding was assuming it was calling the callback that triggered a response. That's now how lambda works.

Second, the-code-we-didn't-write dropped the handler-callback into a request-callback and called unref on the request's socket. Something like this (but more complicated in reality)

    exports.handler = function handler(event, context, callback) {
      const response = {
          statusCode: 200,
          body: "default"
      };  
      const options = {
        hostname: 'www.google.com',
        port: '443',
        path: '/',
        method: 'GET'
      }

      const transport = require('https')

      const req = transport.request(options, function(res) {
        const finalData = []

        res.on('data', function(data) {
          finalData.push(data)
        })

        res.on('end', function(data) {
          console.log('all the data')
          response.body = finalData.join('')
          callback(null, response)
        })
      })

      req.on('socket', function (socket) {
          socket.unref()
      })

      req.end()
    }

It appears that AWS Lambda honors the intent of un-refing the socket -- that is these callbacks are no longer considered as necessary to process before returning a result.

Collectives™ on Stack Overflow

When can a Node.js AWS Lambda Function Exit without Invoking its Callback?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related