0

We're running a large web application across several servers. We also have a dedicated linux server that hosts Redis v6.2.1. Our web application uses StackExchange.Redis to connect. Most of the time, everything works fine. But occasionally, we get a burst of timeouts. I've searched SO and other places (also followed the link in the timeout exception) for similar issues, but most of the questions deal with not enough min worker threads. But what i've noticed is that our timeout errors seem to be coming from the IN buffer. This question StackExchange.Redis timeouts GET shows a timeout error with an in buffer like ours, but it doesn't seem to be addressed in the answer.

I've posted a sample of the error we get below. From my own findings, it seems we have enough worker threads available, but the issue seems to be the in buffer. I'm just not sure what could cause this issue.

in: 65536

Timeout performing GET (5000ms), next: LRANGE my_key, inst: 1, qu: 0, qs: 48, aw: False, rs: ReadAsync, ws: Idle, in: 65536, in-pipe: 0, out-pipe: 61, serverEndpoint: [server], mc: 1/1/0, mgr: 10 of 10 available, clientName: My_App, IOCP: (Busy=0,Free=400,Min=200,Max=400), WORKER: (Busy=44,Free=356,Min=200,Max=400), v: 2.2.4.27433 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts) at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server) at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor`1 processor, ServerEndPoint server) at StackExchange.Redis.RedisDatabase.StringGet(RedisKey key, CommandFlags flags) at GetCachedObject[T](String key, T& obj)

I've checked the slowlog in our server, but there doesn't seem to be any correlation to the timeouts we're getting, and from my understanding, the in buffer is at the client side of things and isn't processing the data like it should be.

Has anyone else come across this issue and has some insight as to what I should look into? Am I even in the ballpark of what is actually happening? If you need any specific details about our configuration, I can add that as well. We do host our ASP.NET web app with at most 4 different worker threads and we are using the default 5 second timeout in StackExchange.Redis

Improve this question
3
  • We had similar problem before when there are too many small, quick operations. Slowlog is empty because individual operation is completed quick enough. However the server cannot response fast enough for other reasons. Maybe you can check if CPU usage of redis server or your app is high. Commented Oct 26, 2021 at 8:13
  • Thanks. We did notice that CPU usage on our web server was high during the time. I wonder if that is causing the buffer to back up? Commented Oct 26, 2021 at 12:03
  • It's possible if CPU usage of web server is near 100%. As far as I know, redis does not multi-thread very well. Beware if you notice redis server use more than 1 core. Commented Oct 27, 2021 at 1:18
Related questions
4
how to deal with read() timeout in Redis client?
6
Redis connection/buffer-size limit exceeded
0
stackexchange redis, reading large file = timeouts
Load 7 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.