2

I have a Lambda function running a .NET Core application from which I try to interact with an ElastiCache Redis cluster using StackExchange.Redis. The cluster is a single node. The Lambda is configured in the same VPC and security group as the ElastiCache Redis cluster. The security group has inbound rules set for the all subnets in the VPC for port 6379. All these subnets are attached to the Lambda.

I keep getting the following error as soon as my code tries to connect to the Redis cluster:

{
  "errorType": "RedisConnectionException",
  "errorMessage": "No connection is active/available to service this operation: GET <REDACTED-KEY>; UnableToConnect on <REDACTED-PRIMARY-ENDPOINT-NAME>:6379/Interactive, Initializing/NotStarted, last: NONE, origin: BeginConnectAsync, outstanding: 0, last-read: 5s ago, last-write: 5s ago, keep-alive: 60s, state: Connecting, mgr: 10 of 10 available, last-heartbeat: never, global: 5s ago, v: 2.2.50.36290, mc: 1/1/0, mgr: 10 of 10 available, clientName: 169, IOCP: (Busy=0,Free=1000,Min=2,Max=1000), WORKER: (Busy=0,Free=32767,Min=2,Max=32767), v: 2.2.50.36290",
  "stackTrace": [
    "at StackExchange.Redis.ConnectionMultiplexer.ThrowFailed[T](TaskCompletionSource`1 source, Exception unthrownException) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2799",
    "--- End of stack trace from previous location where exception was thrown ---",
    "at <REDACTED-STACK-TRACE>"
    "at lambda_method(Closure , Stream , Stream , LambdaContextInternal )"
  ],
  "cause": {
    "errorType": "RedisConnectionException",
    "errorMessage": "UnableToConnect on <REDACTED-PRIMARY-ENDPOINT-NAME>:6379/Interactive, Initializing/NotStarted, last: NONE, origin: BeginConnectAsync, outstanding: 0, last-read: 5s ago, last-write: 5s ago, keep-alive: 60s, state: Connecting, mgr: 10 of 10 available, last-heartbeat: never, global: 5s ago, v: 2.2.50.36290"
  }
}

Note: abortConnect=false is set on the Redis configuration

Things I've already tried:

  • Add the elasticache:* action to my Terraform IAM policy for the Lambda permissions. Full policy:
data "aws_iam_policy_document" "iam_policy_document_mylambda" {
  statement {
    actions = [
      "ec2:DescribeNetworkInterfaces",
      "ec2:CreateNetworkInterface",
      "ec2:DeleteNetworkInterface",
      "ec2:DescribeInstances",
      "ec2:AttachNetworkInterface",
      "elasticache:*",
      "execute-api:ManageConnections"
    ]
    resources = ["*"]
    effect    = "Allow"
  }
}
  • Add resolveDns=true to the Redis configuration. The IP address of the Redis cluster shows up in the error above instead of the primary endpoint name and I can confirm it is within the IP range of one of the subnets that is configured for the Lambda (the subnet on eu-west-1a).
  • Add ssl=false explicitly, since encryption in transit is disabled (for now) on the Redis cluster. Also tried with ssl=true,sslProtocols=tls12 which was mentioned in a GitHub issue, just to make sure.
  • Remove abortConnect=false, then it fails with the following error:
{
  "errorType": "RedisConnectionException",
  "errorMessage": "It was not possible to connect to the redis server(s). Error connecting right now. To allow this multiplexer to continue retrying until it's able to connect, use abortConnect=false in your connection string or AbortOnConnectFail=false; in your code.",
  "stackTrace": [
    "at StackExchange.Redis.ConnectionMultiplexer.ConnectImpl(ConfigurationOptions configuration, TextWriter log) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 1164",
    "at StackExchange.Redis.ConnectionMultiplexer.Connect(ConfigurationOptions configuration, TextWriter log) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 1032",
    "at ...",
    "at lambda_method(Closure , Stream , Stream , LambdaContextInternal )"
  ]
}

I'm at my wits end with this and every search online seems to point in the same direction; make sure your Lambda and ElastiCache Redis cluster run in the same VPC and Security Group and set abortConnect=false on the Redis configuration, which doesn't seem to be sufficient for my case. Does anyone know what else I could try?

1 Answer 1

2

I figured it out. The problem was with my infrastructure and not an issue in how StackExchange.Redis is connecting to the cluster. I had to create a security group for my Lambda function and add that security group to the allow_connections_from_security_groups on my Redis Terraform module, which relies on the Gruntwork module, instead of assigning the Redis security group to my Lambda.

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.