Search code examples
pythonjsonwebsockethttprequest

Calling the same API over 2000 times every minute (JSON/PYTHON), use (HTTP sync, HTTP async, Websocket, or Other)?

I am trying to gather 1 minute data across 2000+ stocks from a financial institution API, every minute as soon as the data is available. I want to gather data during the trading hours. I want to gather this data using python.

Example API URL [Not a valid URL]: https://api.finance.com/v1/marketdata/[STOCK]/1minute

Conditions:

  1. We know that the all 2000+ stocks 1 minute data is available for retrieval once the minute hits. For example, if the current time is 10:02:00AM and I wanted to get 10:01:00AM data from GOOG, I would call the URL: https://api.finance.com/v1/marketdata/GOOG/1minute , and I would see the 10:01:00AM data.
  2. We know the data is stored in JSON format.
  3. There exists a throttling limit. Suppose 500millisecond wait between requests.
  4. I need the one minute data tick data (i.e. Open, Low, High, Close).

Question: How can I gather all 2000+ stocks data within 30 seconds?

Solutions I came up with, although I don't know if it is the most optimal in this situation or if my understanding of HTTP Request, HTTP Asynchronous, WebSocket is lacking in some way.

Possible Solutions?:

  1. HTTP Request with For Loop: Currently I am using a simple for loop and a time.sleep() function. It is the simplest to implement. But the problem with it is at best it takes 16 minutes because of the throttling limit.
  2. HTTP Asynchronous: From what I understand, I could create a separate thread for each stock and gather the one minute data that way. But based on what I have read, at most I can probably have about 100 threads running simultaneously. Is that a correct assumption? Also wouldn't most servers not allow that many requests be made simultaneously from one client machine?
  3. Websocket: From what I understand, I could simply create one connection with the server and get the data without having to worry about that throttle limit. Ideally, I would build the application with websocket. Is it a correct assumption that this is the best method for this sort of problem? The issue I have currently with this method, however, is that their 1 tick minute data is only available via this API URL call method. As far as I know, I cannot retrieve that data through a websocket connection(i.e. If I connect to their websocket url: wss://stream-finance.com/ws, the 1 minute data is not one of the available data on the other end) The question I have here is: is it possible to create a websocket connection with the https url? Also, is it possible to retrieve that 1 minute data through their websocket url wss://stream-finance.com/ws, if that 1minute data isnt one of the available options to get?
  4. Other: Is their another method that would work better for this instance?

Best Solution?: The best solution I see is simply to create one single connection to their server, then call each stock to update every 1 minute in "realtime". But I don't know how to implement that through that HTTPS URL they provide.

enter image description here

Solution

  • Your question is a bit confusing. Lets separate some concerns in here.

    The server

    The selection of the communication protocol is entirely dependent on what the server side implements, you just develop your client accordingly.

    Bellow are a set of options and techniques I've seen been used:

    • HTTP Pooling (what I like to call the hit and run technique) Simply do an HTTP request to the interested endpoint on an interval basis; every minute seems to be your use case. This will be supported by pretty much every HTTP API out there.
    • HTTP Long Pooling Similar pooling, a server supporting long pooling will hang onto an HTTP connection until it has appropriate data provide or a timeout is reached. From a client perspective, you're still continuously requesting data on an interval basis, its just the server that postpones replying until data is available.
    • Web sockets (This is ideal for your case) This is essentially combining the two techniques (oversimplifying here), the server will respond while holding onto the connection, and continuously send new data over it.

    The client

    Regardless of the protocol you decide to use, your client software will have to handle the network latency in someway or another.

    For the sake of simplicity I will suppose you have a function, called get_data() that fetches data from the server using one the techniques above.

    • Synchronous Your client will call get_data(), process the response, and repeat. You're used to this, that's how python normally works.

    • Asynchronous Your client will dispatch calls to get_data() to some worker, and some other function will be executed when data comes in. Essentially allowing your program to do multiple API calls, and process them in the response order, rather than request order.

    • Threads Analogous to the synchronous behavior, you would process each request/response a separate CPU thread, handling data as it comes in. (with all the caveats of the Python's GIL)

    Note

    As obvious as it may seem, 2000+ threads is not a good solution.