Search code examples
zeromqdistributed-systemmql4low-latencymetatrader4

Why does ZeroMQ not receive a string when it becomes too large on a PUSH/PULL MT4 - Python setup?

I have an EA set in place that loops history trades and builds one large string with trade information. I then send this string every second from MT4 to the python backend using a plain PUSH/PULL pattern.

For whatever reason, the data isn't received on the pull side when the string transferred becomes too long. The backend PULL-socket slices each string and further processes it.

Any chance that the PULL-side is too slow to grab and process all the data which then causes an overflow (so that a delay arises due to the processing part)?

Talking about file sizes we are well below 5kb per second.

This is the PULL-socket, which manipulates the data after receiving it:

while True:
    # check 24/7 for available data in the pull socket
    try:
        msg = zmq_socket.recv_string()
        data = msg.split("|")
        print(data)

        # if data is available and msg is account info, handle as follows
        if data[0] == "account_info":
[...]
    except zmq.error.Again:
        print("\nResource timeout.. please try again.")
        sleep(0.000001)

I am a bit curious now since the pull socket seems to not even be able to process a string containing 40 trades with their according information on a single MT4 client - Python connection. I actually planned to set it up to handle more than 5.000 MT4 clients - python backend connections at once.

Solution

  • Q : Any chance that the pull side is too slow to grab and process all the data which then causes an overflow (so that a delay arises due to the processing part)?

    Zero chance.

    Sending 640 B each second is definitely no showstopper ( 5kb per second - is nowhere near a performance ceiling... )

    The posted problem formulation is otherwise undecidable.

    Step 1) POSACK/NACK prove whether a PUSH side accepts the payload for sending error-free.

    Step 2) prove the PULL side is not to be blamed - [PUSH.send(640*chr(64+i)) for i in range( 10 )] via a python-2-python tcp://-transport-class solo-channel crossing host-to-host hop, over at least your local physical network ( no VMCI/emulated vLAN, no other localhost colocation )

    Step 3) if either steps above got POSACK-ed, your next chances are the ZeroMQ configuration space and/or the MT4-based PUSH-side incompatibility, most probably "hidden" inside a (not mentioned) third party ZeroMQ wrapper used / first-party issues with string handling / processing ( which you must have already read about, as it has been so many times observed and mentioned in the past posts about this trouble with well "hidden" MQL4 internal eco-system changes ).

    Anyway, stay tuned. ZeroMQ is a sure bet and a truly horsepower for professional and designs in 's domain.