We are including Redis cache in our Saas application in Azure.
But we are having some problems, we see these types of errors from time to time
Timeout performing EVAL, inst: 2, queue: 23, qu: 0, qs: 23, qc: 0, wr: 0, wq: 0, in: 65536, ar: 0,
IOCP: (Busy=1,Free=999,Min=1,Max=1000), WORKER: (Busy=2,Free=8189,Min=1,Max=8191)
Timeout performing EVAL, inst: 2, queue: 3, qu: 0, qs: 3, qc: 0, wr: 0, wq: 0, in: 65536, ar: 0,
IOCP: (Busy=1,Free=999,Min=1,Max=1000), WORKER: (Busy=3,Free=8188,Min=1,Max=8191)
When trying to interpret these logs, I have a hard time knowing where to start, where to look and draw conclusions.
Please, can someone tell me something about these particular values? The Redis instance is in the same zone as the server, I use the c1 plan with 1GB of memory, however these errors come out by doing simple tests.
The settings are:
ConnectionTimeoutInMilliseconds = "2000" operationTimeoutInMilliseconds = "1500" retryTimeoutInMilliseconds = "4500"
In the redis dashboards in Azure I do not see any memory spikes, connections or anything else, should I look at any particular value?
The “in: 65536” value in the timeout is very high. This value indicates how much data is sitting in the client’s socket kernel buffer. This indicates that the data has arrived at the local machine but has not been read by the application layer. This typically happens when 1) thread pool settings need to be adjusted or 2) when client CPU is running high.
IOCP: (Busy=1,Free=999,Min=1,Max=1000), WORKER: (Busy=2,Free=8189,Min=1,Max=8191) and IOCP: (Busy=1,Free=999,Min=1,Max=1000), WORKER: (Busy=3,Free=8188,Min=1,Max=8191) indicate that you are hitting threadpool growth throttling issues. You might want to look into the explanation and recommendations here: https://gist.github.com/JonCole/e65411214030f0d823cb#file-threadpool-md.