Search code examples
mysqlredisbigdatareal-timelambda-architecture

How to upgrade our short/long memory term for real time processing

Our mobile app track user events (Events can have many types)

Each mobile reporting the user event and later on can retrieve it.

I thought of writing to Redis and Mysql.

When user request:

1. Find on Redis
2. If not on Redis find on Mysql
3. Return the value
4. Keep Redis modified in case value wasnt existed.
5. set expiry policy to each key on redis to avoid out of mem.

Problem:

1. Reads: If many users at once requesting information which not existed at Redis  mysql going to be overloaded with Reads (latency).

2. Writes: I am going to have lots of writes into Mysql since every event going to be written to both datasources.

Facts:

1. Expecting 10m concurrect users which writes and reads.
2. Need to serv each request with max latency of one second.
3. expecting to have couple of thousands requests per sec.

Any solutions for that kind of mechanism to have good qos?

3. Is that in any way Lambda architecture solution ?

Thank you.

Solution

  • Sorry, but such issues (complex) rarely have a ready answer here. Too many unknowns. What is your budget and how much hardware you have. Since 10 million clients are concurrent use your service your question is about hardware, not the software.

    Here is no any words about several important requirements:

    1. What is more important - consistency vs availability?
    2. What is the read/write ratio?

    Read/write ratio requirement

    If you have 10,000,000 concurrent users this is problem in itself. But if you have much of reads it's not so terrible as it may seem. In this case you should take care about right indexes in mysql. Also buy servers with lot of RAM to keep at least index data in RAM. So one server can hold 3000-5000 concurrent select queries without any problems with latency requirement in 1 second (one of our statistic project hold up to 7,000 select rps per server on 4 years old ordinary harware).

    If you have much of writes - all becomes more complicated. And consistency becomes main question.

    Consistency vs availability

    If consistency is important - go to the store for new servers with SSD drives and moder CPU. Do not forget to buy much RAM as possible. Why? If you have much of write requests your sql server would rebuild index with every write. And you can't do not use indexes because of your read requests do not to keep in latency requirement. Under consistency i mean - if you write something, you should do this in 1 second and if you read this data right after write - you get actual written information in 1 second.

    Your problem 1:

    Reads: If many users at once requesting information which not existed at Redis mysql going to be overloaded with Reads (latency).

    Or well known "cache miss" problem. And it has just some solutions - horizontal scaling (buy more hardware) or precaching. Precaching in this case may be done in at least 3 scenarios:

    1. Using non blocking read and wait up to one second while data wont be queried from SQL server. If it not, return data from Redis. Update in Redis immediately or throw queue - as you want.
    2. Using blocking/non blocking read and return data from Redis as fast as possible, but with every ready query push jub to queue about update cache data in Redis (also may inform app it should requery data after some time).
    3. Always read/write from Redis, but register job in queue every write request to update data in SQL.

    Every of them is compromise:

    1. High availability but consistency suffers, Redis is LRU cache.
    2. High availability but consistency suffers, Redis is LRU cache.
    3. High availability and consistency but requires lot of RAM for Redis.

    Writes: I am going to have lots of writes into Mysql since every event going to be written to both datasources.

    The filed of compromise again. Lot's of writes rests to hardware. So buy more or use queues for pending writes. So availability vs consistency again.

    Event tracking means (usualy) you can return data close to real time but not in real time. For example have 1-10 seconds latency to update data on disk (mysql) keeping 1 second latency for write/read serving requests. So, it's combination of 1/2/3 (or some other) techniques for data provessing:

    • Use LRU in Redis and do not use expire. Lot's of expire keys - problem as is. So we can't use to be sure we save RAM.
    • Use queue to warm up missing keys in Redis.
    • Use queue to write data into mysql server from Redis server.
    • Use additional requests to update data from client size of cache missing situation accures.