Search code examples
c#wcfredisbooksleeve

How to save and retrieve channel data from Redis (pub/sub) with date ranges?

Initially I asked about querying .rdb files which is not what I want to do - I will be querying an active server.

I am about to start a project where we will use Redis in a pub/sub scenario, using RDB snapshotting.

A .Net WCF client will configure the Redis connection through Marc Gravell's BookSleeve and accept incoming requests.

With RDB snapshotting enabled, incremental snapshots will be created.

I want a client to be able to retrieve a dataset based on a given channel and date range. I was thinking that I would use the same WCF client for both read and writes (unless there are any notable objections). I am not entirely clear on what method to use to query server to retrieve date-range, channel specific data, and deliver that through a WCF client.

Can I achieve this data querying functionality through BookSleeve (if so, how) or are there other, better-suited access wrappers to use, such as ServiceStack.Redis.

Would it be better to use BookSleeve for writes, and ServiceStack.Redis as the client?

-- UPDATE --

I have snapshotting set to the default:

save 900 1
save 300 10
save 60

Browsing the data with Redis Admin UI, I would expect that after 15 minutes (900 seconds) to see some data stored in a key, however, searching keys shows the channel name (and other keys), but no values within.

Since pub/sub events are not persisted, what is a good way of capturing channel data and storing it?

Solution

  • Edit to address the OP's edit:

    As noted: pub/sub is not persisted. I would suggest use a queue; the "list" type in redis is ideal for that, with lpush, rpush, lpop, rpop, rpoplpush and ltrim all being key commands for that scenario. You might choose to use pub/sub as well, to notify the client of the availability of new data - or you can use the blocking pop commands, blpop, brpop and brpoplpush. All these operations are defined here.

    Can I clarify: it sounds like you have a bunch of rdb files, and want to read from arbitrary files (the "incremental snapshots"). Is that about right?

    rdb files on their own aren't very useful unless you use dedicated tools to parse them. Redis clients (like BookSleeve and ServiceStack) expect to communicate with a running Redis instance. Your question sounds a bit like "I have sql-server .bak files for a range of dates; how can I get data from SomeTable?" - to which the answer is "first load the bak into SQL Server".

    Starting a redis-server is pretty easy, but it doesn't expect to read from multiple rdb files.

    Options:

    1. parse the rdb manually; see this thread - expect pain
    2. start a redis-server instance on the fly per-file when needed... well, it'll work, but process management could get "fun"
    3. start a redis-server instance per-file in advance... then you need to worry about port management / mapping, and monitoring for new files
    4. hot-load using DEBUG RELOAD; however, this is not really very supported, and in testing I couldn't get it to behave properly (in particular, it seems to do a SAVE before the reload, making it unsuitable for this scenario)

    But ultimately, having a range of different rdb files that you want to pick from on-the-fly is not the normal use case of Redis.

    If I've understood your requirements, then this isn't a client issue, let alone a platform-specific issue (.NET, C#, WCF, etc): first you need to figure out how you're going to run the server(s) or otherwise read the files. That would be best asked in Redis DB