Search code examples
restcachingwebsockettimescaledbccxt

How to overcome API/Websocket limitations with OHCL data for trading platform for lots of users?


I'm using CCXT for some API REST calls for information and websockets. It's OK for 1 user, if I wanted to have many users using the platform, How would I go about an inhouse solution?

Currently each chart is either using websockets or rest calls, if I have 20 charts then thats 20 calls, if I increase users, then thats 20x whatever users. If I get a complete coin list with realtime prices from 1 exchange, then that just slows everything down.

Some ideas I have thought about so far are:

  1. Use proxies with REST/Websockets
  2. Use timescale DB to store the data and serve that OR
  3. Use caching on the server, and serve that to the users

Would this be a solution? There's got to be a way to over come rate limiting & reducing the amount of calls to the exchanges.


Solution

  • Probably, it's good to think about having separated layers to:

    1. receive market data (a single connection that broadcast data to OHLC processors)
    2. process OHLC histograms (subscribe to internal market data)
    3. serve histogram data (subscribe to processed data)

    The market data stream is huge, and if you think about these layers independently, it will make it easy to scale and even decouple the components later if necessary.

    With timescale, you can build materialized views that will easily access and retrieve the information. Every materialized view can set a continuous aggregate policy based on the interval of the histograms.

    Fetching all data all the time for all the users is not a good idea. Pagination can help bring the visible histograms first and limit the query results to avoid heavy IO in the server with big chunks of memory.