Python Interactive brokers IB API very very slow

I am trying out the new Python Interactive Broker API, but I am experiencing some serious speed issues at the very first step...

The following code (see below) times

0:00:08.832813 until the data is done being received

0:00:36.000785 until the app is fully disconnected...

Why is it so slow? What would be the best way to speed it up?

from ibapi import wrapper
from ibapi.client import EClient
from ibapi.utils import iswrapper #just for decorator
from ibapi.common import *
from ibapi.contract import *
import datetime
from datetime import timedelta


class DataApp(wrapper.EWrapper, EClient):
    def __init__(self):
        wrapper.EWrapper.__init__(self)
        EClient.__init__(self, wrapper=self)

    @iswrapper
    def historicalData(self, reqId: TickerId, date: str, open: float, high: float,
                            low: float, close: float, volume: int, barCount: int,
                            WAP: float, hasGaps: int):
        super().historicalData(reqId, date, open, high, low, close, volume,
                                barCount, WAP, hasGaps)
        print("HistoricalData. ", reqId, " Date:", date, "Open:", open,
               "High:", high, "Low:", low, "Close:", close, "Volume:", volume)

    @iswrapper
    def historicalDataEnd(self, reqId: int, start: str, end: str):
        super().historicalDataEnd(reqId, start, end)
        print("HistoricalDataEnd ", reqId, "from", start, "to", end)
        print(datetime.datetime.now()-startime)
        self.done = True # This ends the messages loop - this was not in the example code...

    def get_data(self):        
        self.connect("127.0.0.1", 4002, clientId=10)
        print("serverVersion:%s connectionTime:%s" % (self.serverVersion(),
                                                self.twsConnectionTime()))

        cont = Contract()
        cont.symbol = "ES"
        cont.secType = "FUT"
        cont.currency = "USD"
        cont.exchange = "GLOBEX"
        cont.lastTradeDateOrContractMonth = "201706"
        self.reqHistoricalData(1, cont, datetime.datetime.now().strftime("%Y%m%d %H:%M:%S"),
                               "1800 S", "30 mins", "TRADES", 0, 1, [])
        self.run()        
        self.disconnect()
        print(datetime.datetime.now()-startime)

global starttime
startime = datetime.datetime.now()
DA = DataApp()
DA.get_data()

I also tried to continually run it is the background, in order to only submit requests on the fly with

def runMe():
    app.run() # where run() has be removed from the class definition

import threading
thread = threading.Thread(target = runMe)
thread.start()

But it was also incredibly slow. Any suggestions appreciated

Solution

I would recommend you modify the connection socket lock within the connection class in the ibapi module. The recommendation came from heshiming on github; if you have access to the private interactive brokers repo you can access the discussion here https://github.com/InteractiveBrokers/tws-api/issues/464

I did this and it improved performance significantly.

Heshiming recommends you reduce the timeout on the socket lock object, which is called every time you send or receive a message. To modify the socket lock, Go to the site-packages folder for ibapi and modify the connect function within connection.py, changing "self.socket.settimeout(1)" to "self.socket.settimeout(0.01)". This is line 48 in connection.py for the version I have.

In case you can't see heshiming's post, I've included it at the bottom of this post.

Alternative Option: another interesting solution would be to leverage asyncio for an asynchronous event loop. I have not done this but it looks promising. See the example Ewald put together https://github.com/erdewit/tws_async

Heshiming Comment:

The implementation of Connection /ibapi/connection.py has a Lock object shared in both sendMsg and recvMsg. Since connect, self.socket.settimeout(1) is called, therefore the underlying self.socket.recv(4096) only times out once per second.

Such implementation creates a performance problem. Since the lock is shared, the socket cannot send data while receiving. In the scenario where the message received is less than 4k bytes long, the recvMsg function will wait for 1 second before releasing the lock, making subsequent sendMsg wait. In my experiment, most messages appear to be shorter than 4k bytes. In other words, this imposes a cap of one recvMsg per second.

There are couple strategies to mitigate this. One can reduce the receive buffer to a number much less than 4k, or reduce the socket timeout to something like 0.001 second to make it block less.

Or according to http://stackoverflow.com/questions/1981372/are-parallel-calls-to-send-recv-on-the-same-socket-valid , the socket itself is actually thread-safe. Thus no locks are necessary.

I tried all three strategies. Removing the lock works the best. And reducing the timeout to 0.001 works in similar ways.

I can only vouch for linux/unix platforms, and I haven't tried it on Windows. Would you consider to change the implementation to improve this?