Search code examples
pythontwistedbenchmarkingspyne

Spyne rpc server benchmark (Failed to establish a new connection)

I have python rpc server written above http://spyne.io (and twisted). I've done some Multi-Mechanize benchmarks on it and as you can see in the image bellow - after a minute of testing it starts to have problems establishing connections.

111274, 254.989, 1516806285, user_group-1, 0.017, HTTPConnectionPool(host='0.0.0.0' port=4321): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f2c78bf2810>: Failed to establish a new connection: [Errno 99] Cannot assign requested address')), {'increment': 0.0179598331451416}

Since this happens as a clock (after 60s) I'm suspecting, I've run into some implicit/default rate limits from twisted (but my search for them wasn't successful).

Is this possible - if so can someone point me to those limits?

Or is this just overload of the server?

Thanks

multi mechanize benchmark image

EDIT:

Thanks to Jean-Paul Calderone answer I've looked at the number of TCP connections netstat -at | wc -l.

Every time it gets above 28K I get the Cannot assign requested address.

I'm happy this isn't a server issue. :)

Solution

  • The error

    Cannot assign requested address

    is probably telling you that you've run out of IP/port combinations. A particular IP address can be combined with about 65535 different port numbers to form an address. A TCP connection involves two addresses, one for each end of the connection.

    Thus, a particular IP address can not establish more than 65535 TCP connections. And if both ends of the TCP connection fall on the same IP address, that drops the limit to half of this.

    Furthermore, TCP connection cleanup involves the TIME_WAIT state - a time-bounded interval during which the connection is still exists though it has already been closed. During this interval, the address of its endpoint is not available for re-use. Therefore, in addition to a hard maximum on the number of TCP connections that you can have open at a given time on a given IP address, there is also a hard maximum to the number of TCP connections that you can open within a given window of time.

    If you are benchmarking on your local system, it seems likely that you've run in to these limits. You can expand your benchmark beyond them by using more IP addresses.

    Of course, if your server can deal with enough connections that your benchmark actually encounters these limits, perhaps you've answered the question of whether the server is fast enough, already. There's not much to be gained by going from (for example) 1000 connections/second to 10000 connections/second unless your application logic can process requests faster by order of magnitude or so.

    Consider an application which requires 1ms processing time. Paired with an RPC server which can service 1000 connections/sec, you'll be able to service 500 requests/second (1ms app + 1ms server -> 2ms -> 1/500th sec -> 500/sec). Now replace the server with one with one tenth the overhead: 1ms app + .1ms server -> 1.1ms -> 909/sec. So your server is 10x faster but you haven't quite achieved 2x the throughput. Now, almost 2x is not too bad. But it's rapidly diminishing returns from here - a server ten times faster again only gets you to 990 requests/second. And you'll never beat 1000 requests/second because that's your application logic's limit.