ServiceStack Redis problems with simultaneous read requests

Question

I'm using the ServiceStack.Redis implementation for caching events delivered over a Web API interface. Those events should be inserted into the cache and automatically removed after a while (e.g. 3 days):

private readonly IRedisTypedClient<CachedMonitoringEvent> _eventsCache;

public EventMonitorCache([NotNull]IRedisTypedClient<CachedMonitoringEvent> eventsCache)
{
    _eventsCache = eventsCache;
}

public void Dispose()
{
    //Release connections again
    _eventsCache.Dispose();
}

public void AddOrUpdate(MonitoringEvent monitoringEvent)
{
    if (monitoringEvent == null)
        return;

    try
    {
        var cacheExpiresAt = DateTime.Now.Add(CacheExpirationDuration);

        CachedMonitoringEvent cachedEvent;

        string eventKey = CachedMonitoringEvent.CreateUrnId(monitoringEvent);
        if (_eventsCache.ContainsKey(eventKey))
        {
            cachedEvent = _eventsCache[eventKey];
            cachedEvent.SetExpiresAt(cacheExpiresAt);
            cachedEvent.MonitoringEvent = monitoringEvent;
        }
        else
            cachedEvent = new CachedMonitoringEvent(monitoringEvent, cacheExpiresAt);

        _eventsCache.SetEntry(eventKey, cachedEvent, CacheExpirationDuration);
    }
    catch (Exception ex)
    {
        Log.Error("Error while caching MonitoringEvent", ex);
    }
}

public List<MonitoringEvent> GetAll()
{
    IList<CachedMonitoringEvent> allEvents = _eventsCache.GetAll();

    return allEvents
        .Where(e => e.MonitoringEvent != null)
        .Select(e => e.MonitoringEvent)
        .ToList();
}

The StructureMap 3 registry looks like this:

public class RedisRegistry : Registry
{
    private readonly static RedisConfiguration RedisConfiguration = Config.Feeder.Redis;

    public RedisRegistry()
    {
        For<IRedisClientsManager>().Singleton().Use(BuildRedisClientsManager());

        For<IRedisTypedClient<CachedMonitoringEvent>>()
            .AddInstances(i => i.ConstructedBy(c => c.GetInstance<IRedisClientsManager>()
                .GetClient().GetTypedClient<CachedMonitoringEvent>()));
    }

    private static IRedisClientsManager BuildRedisClientsManager()
    {
        return new PooledRedisClientManager(RedisConfiguration.Host + ":" + RedisConfiguration.Port);
    }
}

The first scenario is to retrieve all cached events (several hundred) and deliver this over ODataV3 and ODataV4 to Excel PowerTools for visualization. This works as expected:

public class MonitoringEventsODataV3Controller : EntitySetController<MonitoringEvent, string>
{
    private readonly IEventMonitorCache _eventMonitorCache;

    public MonitoringEventsODataV3Controller([NotNull]IEventMonitorCache eventMonitorCache)
    {
        _eventMonitorCache = eventMonitorCache;
    }

    [ODataRoute("MonitoringEvents")]
    [EnableQuery(AllowedQueryOptions = AllowedQueryOptions.All)]
    public override IQueryable<MonitoringEvent> Get()
    {
        var allEvents = _eventMonitorCache.GetAll();
        return allEvents.AsQueryable();
    }
}

But what I'm struggling with is the OData filtering which Excel PowerQuery does. I'm aware of the fact that I'm not doing any serverside filtering yet but that doesn't matter currently. When I filter for any property and click refresh, PowerQuery is sending multiple requests (I saw up to three) simultaneously. I believe it's fetching the whole dataset first and then executing the following requests with filters. This results in various exceptions for ServiceStack.Redis:

An exception of type 'ServiceStack.Redis.RedisResponseException' occurred in ServiceStack.Redis.dll but was not handled in user code

With additional informations like:

Additional information: Unknown reply on multi-request: 117246333|company|osdmonitoringpreinst|2014-12-22|113917, sPort: 54980, LastCommand:

Or

Additional information: Invalid termination, sPort: 54980, LastCommand:

Or

Additional information: Unknown reply on multi-request: 57, sPort: 54980, LastCommand:

Or

Additional information: Type definitions should start with a '{', expecting serialized type 'CachedMonitoringEvent', got string starting with: u259447|company|osdmonitoringpreinst|2014-12-18|1

All of those exceptions happen on _eventsCache.GetAll().

There must be something I'm missing. I'm sure Redis is capable of handling a LOT of requests "simultaneously" on the same set but apparently I'm doing it wrong. :)

Btw: Redis 2.8.12 is running on a Windows Server 2008 machine (soon 2012).

Thanks for any advice!

Answer 1

The error messages are indicative of using a non-thread-safe instance of the RedisClient across multiple threads since it's getting responses to requests it didn't expect/send.

To ensure your using correctly I only would pass in the Thread-Safe IRedisClientsManager singleton, e.g:

public EventMonitorCache([NotNull]IRedisClientsManager redisManager)
{
    this.redisManager = redisManager;
}

Then explicitly resolve and dispose of the redis client in your methods, e.g:

public void AddOrUpdate(MonitoringEvent monitoringEvent)
{
    if (monitoringEvent == null)
        return;

    try
    {
        using (var redis = this.redisManager.GetClient())
        {
            var _eventsCache = redis.As<CachedMonitoringEvent>();
            var cacheExpiresAt = DateTime.Now.Add(CacheExpirationDuration);

            CachedMonitoringEvent cachedEvent;

            string eventKey = CachedMonitoringEvent.CreateUrnId(monitoringEvent);
            if (_eventsCache.ContainsKey(eventKey))
            {
                cachedEvent = _eventsCache[eventKey];
                cachedEvent.SetExpiresAt(cacheExpiresAt);
                cachedEvent.MonitoringEvent = monitoringEvent;
            }
            else
                cachedEvent = new CachedMonitoringEvent(monitoringEvent, cacheExpiresAt);

            _eventsCache.SetEntry(eventKey, cachedEvent, CacheExpirationDuration);
        }
    }
    catch (Exception ex)
    {
        Log.Error("Error while caching MonitoringEvent", ex);
    }
}

And in GetAll():

public List<MonitoringEvent> GetAll()
{
    using (var redis = this.redisManager.GetClient())
    {
        var _eventsCache = redis.As<CachedMonitoringEvent>();
        IList<CachedMonitoringEvent> allEvents = _eventsCache.GetAll();

        return allEvents
            .Where(e => e.MonitoringEvent != null)
            .Select(e => e.MonitoringEvent)
            .ToList();
    }
}

This will work irrespective of what lifetime of what your EventMonitorCache dependency is registered as, e.g. it's safe to hold as a singleton since EventMonitorCache is no longer holding onto a redis server connection.