Connecting to Redis from Azure throwing Intermittent Exceptions

Question

I have a Windows Azure Server running my C# application. It is distributed on 4 medium instances and I am using Redis for my L2 caching. The application is handling a pretty decent amount of traffic (around 300,000 pageviews a day). I am using BookSleeve for the redis connection and after the application gets up and going it will start throwing SocketExceptions from BookSleeve about four times a minute. The exact exception is:

Exception type: System.Net.Sockets.SocketException
Exception message: A connection attempt failed because the connected party did not     properly respond after a period of time, or established connection failed because connected host has failed to respond

It seems to only be happening when I am reading something from the server:

using (var connection = ConnectionGateway.GetReadConnection())
{
    var task = connection.Hashes.GetString(App.RedisDatabase, CacheKeys.My_KEY);
    var result = connection.Wait(task);
}

My GetReadConnection is setup like this:

    public static RedisConnection GetReadConnection()
    {
        RedisConnection conn = getNewConnection();

        try
        {
            var openAsync = conn.Open();
            conn.Wait(openAsync);

            return conn;
        }
        catch (SocketException ex)
        {
            throw new Exception(RedisConnectionFailed, ex);
        }
    }

Now all my writes share a single connection like the author describes so this is only happening on the reads which require the use of connection.Wait(). That all seems to work fine. The writes use code similar to Maintaining an open Redis connection using BookSleeve

I have tried changing the timeouts of the Redis server to account for the Azure load balancer with no success. I have tried both set timeout 30 and set timeout 0 as described here: Redis connection errors when using Booksleeve Redis client in Azure VM

Any help would really be appreciated.

Answer 1

Glancing at the code, the first thing that occurs is that you seem to be creating a connection per read operation. This is not required, and could be adversely impacting both .net and redis: BookSleeve is a multiplexer - it is designed to process lots of requests concurrently over a single connection. When you call Wait, it is only the waiting thread that blocks - the internals of BookSleeve will carry on handling other threads etc.

Additionally, creating a connection has quite a lot of overhead: in addition to the costs of establishing a new TCP connection, BookSleeve needs to talk to the redis server to find some key configuration information that it needs for correct operation, so it is always recommended to re-use a single shared connection or a small number of shared connections.