Search code examples
pythonwebsocketwiresharklatencytcpdump

Latency debug for websocket communication

I'm working on a debug latency problem of websocket.

I am trying to receive some price information from crypto-currency exchange with its websocket interface. The data packets we receive include the timestamp generatee on the exchange server. I log the time when we receive the tick information on our computer (the "client box") and compare the latency between the arrival time and the server generation time. Most of the ticks show a few tens of milliseconds which is more or less fine. But everyday we see a few times the latency becomes several seconds or even more then ten seconds and I would like to figure out where are these large latency come from.

The system is using Python programming language and the websocket module I'm using is websocket-client (https://pypi.org/project/websocket_client/, https://github.com/websocket-client/websocket-client), I tried to add logs inside the module and see if it is due to module processing time but still no luck.

One idea currently in my mind is to use tcpdump to capture the net traffic and record the time the tcp packet arrives my network card. If this time still presents the latency, I will have no way other than move the program to a co-located server. However, I encounters difficult here as the websocket connection is SSL-encrypted. Thus I cannot see the tick generation time packed inside the message.

Does anyone have some solution here ? In particular:

  • Is there any way to retrieve the private key of SSL from the websocket-client python package from client-end? (I assume the key should be available somewhere local side, otherwise the websocket-client cannot decrypt the data itself. And WireShark should be able to decrypt the message for TSL1.2 protocol)
    • if it is not easy to do this with websocket-client package, I'm happy to try other websocket lib written by python, C/C++.
  • Can tcpdump get the timestamp when the TCP data packet sent from server (even in server time)?

Any other advices are highly appreciated as well.

Thanks a lot!

Thanks @Eugène Adell My tcpdump opened in WireShark is mostly like below enter image description here and I can see the TSval in TCP Option - Timestamps enter image description here Can these indicate something? Sorry for probably basic questions, really lack of experience in this area & Thanks again.

Solution

  • EDIT

    Can tcpdump get the timestamp when the TCP data packet sent from server (even in server time)?

    Open your capture and see if the packets have the TCP timestamps option (as defined in RFC 1323 but better explained in RFC 7323). If so, the very first SYN packet should already mention it.

    Unluckily the meaning of the TSval (Timestamp value in milliseconds) given in these packets is not the real clock and is not always changing like a real clock (it depends on the implementation used by your computers). If the conversation with your server lasts for 60s for example, check if this TSval also moves from 60s, if so maybe can you use this field to track when the packets were sent.