How to reconnect redis client after redis server reboot/scale

Question

I have an azure app service (based on Docker) that uses Redis as a cache memory. When I reboot/scale redis server, redis client inside azure app service lose connection with server and throws the following exception:

Timeout awaiting response (outbound=0KiB, inbound=0KiB, 2852ms elapsed, timeout is 2000ms), command=SETEX, next: GET test, inst: 0, qu: 0, qs: 45, aw: False, rs: ReadAsync, ws: Idle, in: 0, serverEndpoint: Unspecified/redis-server-com:6380, mgr: 10 of 10 available, clientName: wallet-api, IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=1,Free=32766,Min=4,Max=32767), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

It takes up to 15min to reconnect with redis server from azure app service, however if I restart app service as soon as app is up, redis client connection is established successfully. From documentation, ConnectionMultiplexor object should manage reconnection, but it does not look like he is doing his job.

Here the redis client code:

public class RedisStore : IRedisStore, IDisposable
{

    private readonly ConfigurationOptions _options;
    private static IConnectionMultiplexer _connection;

    public RedisStore(RedisConfiguration redisConfiguration)
    {
        _options = ConfigurationOptions.Parse(redisConfiguration.ConnectionString);
        _options.ReconnectRetryPolicy = new ExponentialRetry(redisConfiguration.RetryFromMilliSeconds);
    }

    async Task IRedisStore.InitializeConnection()
    {
        if (_connection == null)
        {
            _connection = await ConnectionMultiplexer.ConnectAsync(_options);
        }
    }

    async Task<T> IRedisStore.SetGet<T>(string key)
    {
        var value = await _connection.GetDatabase().StringGetAsync(key);

        if (value.IsNull)
            return default(T);

        return JsonConvert.DeserializeObject<T>(value);
    }

    async Task IRedisStore.SetStore<T>(string key, T value)
    {
        var serialized = JsonConvert.SerializeObject(value);
        await _connection.GetDatabase().StringSetAsync(key, serialized);
    }

    void IDisposable.Dispose()
    {
        _connection.Dispose();
    }
}

The redis connection is initialized from bootstrap code:

private async Task InitializeRedis()
    {
        var redis = Container.GetInstance<IRedisStore>();
        await redis.InitializeConnection();
    }

Also, while app service is throwing redis timeout exceptions, netstat displayed that redis connection is established:

Just before to establish connection again, I got the following 2 exceptions, I guess one for each connection:

SocketFailure on redis-server.com:6380/Interactive, Idle/Faulted, last: GET, origin: ReadFromPipe, outstanding: 52, last-read: 982s ago, last-write: 6s ago, unanswered-write: 938s ago, keep-alive: 60s, state: ConnectedEstablished, mgr: 9 of 10 available, in: 0, last-heartbeat: 0s ago, last-mbeat: 0s ago, global: 0s ago, v: 2.0.601.3402 <--- Unable to read data from the transport connection: Connection timed out. <--- Connection timed out

SocketFailure on redis-server.com:6380/Subscription, Idle/Faulted, last: PING, origin: ReadFromPipe, outstanding: 16, last-read: 998s ago, last-write: 36s ago, keep-alive: 60s, state: ConnectedEstablished, mgr: 9 of 10 available, in: 0, last-heartbeat: 0s ago, last-mbeat: 0s ago, global: 0s ago, v: 2.0.601.3402 <--- Unable to read data from the transport connection: Connection timed out. <--- Connection timed out

Why connection is not refreshed? Is there any way to improve reconnection? 15min is too much for a production environment.

UPDATE 03/09/2020. I did a quick test rebooting redis server with same client but using a secured connection via SSL (port 6380) and a plain connection (port 6379). Checking netstat (netstat -ptona) with a plain connection, redis client reconnect successfully. However checking again with SSL enabled, connection keeps established but there is no response from redis server.

Possible workaround: It looks like something related to framework. As @Json Pan suggested in his reply, I will try upgrading to netcore 3.1 and force app to refresh connection periodically.

Answer 1

UPDATE

After read this blog, I modify the source code, upgrade the project from .net core 1.0 to 3.1.

I suggest you can try it or modify it in your project, to test reconnect time.

PRIVIOUS

I recommand you use Reconnecting with Lazy pattern.

And the answer in How does ConnectionMultiplexer deal with disconnects?, will useful to you.