stackexchange.redis with .net core 1.1 timeout

Question

I have .net core 1.1 based web application and using Amazon ElastiCache of Redis for cache management. I am using "StackExchange.Redis": "1.1.608" version for cache related work. Following is the code used to establish connection public class CacheManager { private static Lazy connection; private static object lockObject = new Object(); private readonly ILogger logger;

    public CacheManager(IConfiguration config, ILoggerFactory loggerFactory)
    {
        this.logger = loggerFactory.CreateLogger<CacheManager>();
        if (connection == null)
        {
            lock (lockObject)
            {
                if (connection == null)
                {
                   connection = new Lazy<ConnectionMultiplexer>(() =>
                    {
                        return ConnectionMultiplexer.Connect(config["Cache:ConnectionString"]);
                    });
                }
            }
        }
    }

    public static ConnectionMultiplexer Connection
    {
        get
        {
            try
            {
                return connection.Value;
            }
            catch (Exception e)
            {
                return null;
            }
        }
    }}

The cache get method try to fetch using the key and if data not found it just fetches it from database and the set it.

We are getting frequent timeout error on redis on production, and not only that, it's causing client to become unresponsive, i.e., the website gives an IIS 500 error. There is nothing in log and redis logs are very simple:

Failed to get cahced value: Timeout performing GET , inst: 61, queue: 2, qu: 0, qs: 2, qc: 0, wr: 0, wq: 0, in: 207, ar: 0, clientName: (Please take a look at this article for some common client-side issues that can cause timeouts: https://github.com/StackExchange/StackExchange.Redis/tree/master/Docs/Timeouts.md)

Redis is hosted on a separate server instance. How can I get a better idea what is the root cause of this ?

What size of cache.node are you running redis on? Have you tried running it on a larger instance to see if the problem goes away? Have you tried running slowlog while you're seeing issues? Note that redis is single threaded, so it can only use 1 core. You can be CPU bound before you see 100% CPU utilization. Basically have you run through all the troubleshooting steps in Timeouts.md? Here's a better link: github.com/StackExchange/StackExchange.Redis/blob/master/docs/… — RaGe
– RaGe, Commented Apr 10, 2018 at 12:35

MichaelBolton · Accepted Answer · 2018-04-18 18:58:21Z

Mukesh, If the redis server's resources (CPU/RAM/hard drive space) aren't constraining things, I would look at the possibility of the threads allocated to your software locking up (where the errors on the logs are actually just a secondary symptom).

I would make sure your redis factory is a singleton instance registered with the services collection on startup. The "connection" field should be set up like so:

private static readonly Lazy<ConnectionMultiplexer> connection = new Lazy<ConnectionMultiplexer>(() => ConnectionMultiplexer.Connect(config["Cache:ConnectionString"]));

This way you never would have to assign it, and it's shared across your application (like intended) with no risk of overrunning your servers with open connections. Your "ConfigureServices" method would have something like the following:

services.AddSingleton<ICacheManager, CacheManager>();

If for some reason you have to lock, instead of locking on an object, await locking on a semaphore slim (figure 10 in this link): https://msdn.microsoft.com/en-us/magazine/jj991977.aspx

If those code changes don't make things better, I would double-check for server resource fluctuations (I don't know what kind of user-base you have, but I've seen website bandwidth become an issue at times). If it turns out it's bandwidth then I would try pub/sub and/or client-side caching.

You could also double check the RAM settings of your redis install. If you override the defaults and then later change the available resources you can run into some problems.

Collectives™ on Stack Overflow

stackexchange.redis with .net core 1.1 timeout

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related