Search code examples
javascriptphpredisreal-time

Javascript realtime visitor information

Need some feedback for an implementation that I am currently using...

I created a javascript file that is embedded into my website that sends a request every 3 seconds to an endpoint. This request has a unique identifier that is created for every new visitor. It then gets added to redis using some key like site_123_unique_identifier with a json encoded value {"load_time":189225}. The keys have a 5 second expire time and since I am using a unique identifier per each visitor, it doesn't duplicate visitors, just increases the expire time.

For our realtime dashboard, we are sending requests to our backend every few seconds and it does a get on all the keys matching site_123*. I then loop through all the redis keys it returns and run a json_decode on it and add up the values.

Would this work with sites that get over millions of views a day? If not what do you recommend as a solution, must work with ie7+[xmlhttp and not sockets]

Solution

  • Schema

    • HASH site:[siteId]:[visitorId] a hash for each visitors with fields like "loadt", "onlinet", "loc", "avgSpeed" ...

    • ZSET site:visitors a global zset with siteId:visitorsId as member and lastSeen_timestamp as value, used for manual key expiration.

    • SET site:[siteId] a set of all visitorId presents in siteId

    • SET site (optional) a set of all siteId

    Pros

    • It works and everything live inside Redis
    • Thanks to HASH site:[siteId]:[visitorId] you can store/increment any number of attributes for each visitors

    Cons

    • You will have to manage expiring yourself. A cron (each ~5 seconds for instance) will have to retrieve the expiring visitorId with: ZRANGE site:visitors (NOW()-5 seconds) -1

    Then loop over all members and execute:

    DEL site:(member_value)

    And then extract from the first member the siteId and execute (visitorId is extract from the member each time)

    SREM site:[siteId] visitorId1 visitorId2 visitorId3

    (See my note on LUA scripts below)

    • Each time you update a value in HASH site:[siteId]:[visitorId] you will have

    to execute:

    ZADD site:visitors [siteId:visitorId] now()
SADD site:[siteId] [visitorId]

    ... along with the HSET or HINCRBY site:[siteId]:[visitorId] field value

    • You don't need json_decode. For instance if you want to retrieve the number of current visitors on a specific website just execute: SCARD site:[siteId]

    • If you need to need to retrieve the average load time of all visitors currently connected to siteId the best way is to use a LUA script to first retrieve the visitorId with SMEMBER site:[siteId] then loop over the visitorId and sum the data with HGET site:[siteId]:[visitorId] loadt, divide by len(visitorIds) and you're done :)

    Note

    • You should implement the above algorithms in lua in order to do all this stuff inside Redis to only receive what you need, see EVAL.