Python3 ThreadingHTTPServer fails to send chunked encoded response

Question

I'm implementing a simple reverse proxy in Python3 and I need to send a response with transfer-encoding chunked mode.

I've taken my cues from this post but I have some problems when sending the chunks in the format described here

If I send chunks of length <= 9 bytes the message is received correctly by the client, otherwise when sending chunks of length >= 10 bytes, it seems that some of them are not received and the message remains stuck in the client waiting indefinitely

Here is an example of non working code:

from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer


class ProxyHTTPRequestHandler(BaseHTTPRequestHandler):
    protocol_version = 'HTTP/1.1'
 
    def do_GET(self, body=True):

        # HTTP 200 + minimal HTTP headers in response
        self.send_response(200)
        self.send_header('transfer-encoding', 'chunked')
        self.send_header('Content-Type', 'text/plain')
        self.end_headers()

        # writing 5 chunks of 10 characters
        for i in range(5):
            text = str(i+1) * 10  # concatenate 10 chars
            chunk = '{0:d}\r\n'.format(len(text)) + text + '\r\n'
            self.wfile.write(chunk.encode(encoding='utf-8'))

        # writing close sequence
        close_chunk = '0\r\n\r\n'
        self.wfile.write(close_chunk.encode(encoding='utf-8'))


def main():
    try:
        server_address = ('127.0.0.1', 8099)

        # I use ThreadingHTTPServer but the problem persists also with HTTPServer
        httpd = ThreadingHTTPServer(server_address, ProxyHTTPRequestHandler)
        print('http server is running')
        httpd.serve_forever()
    except KeyboardInterrupt:
        print(" ^C entered, stopping web server...")
        httpd.socket.close()


if __name__ == '__main__':
    main()

In this case, after several seconds, and only if I manually stop python execution the result in Postman is the following. Please note the missing "2222222222" chunk

But if I use this length instead:

    # writing the same 5 chunks of 9 characters
    for i in range(5):
        text = str(i+1) * 9  # concatenate 9 chars
        chunk = '{0:d}\r\n'.format(len(text)) + text + '\r\n'
        self.wfile.write(chunk.encode(encoding='utf-8'))

    # writing close sequence
    close_chunk = '0\r\n\r\n'
    self.wfile.write(close_chunk.encode(encoding='utf-8'))

The communication ends correctly (after 6ms all the 5 chunks are interpreted correctly)

Some version informations:

HTTP Client: Postman 8.10 

(venv) manuel@MBP ReverseProxy % python -V
Python 3.9.2

(venv) manuel@MBP ReverseProxy % pip freeze
certifi==2021.10.8
charset-normalizer==2.0.6
idna==3.2
requests==2.26.0
urllib3==1.26.7

Thanks in advance for any hints!

Answer 1

I post the solution (thanks to Martin Panter from bugs.python.org) in case anyone else will have the same problem in the future.

The behaviour was caused by the chunk size part, that must be in hex format, not decimal.

Unfortunately from the Mozilla docs the format was not specified and the example used only length < 10. A formal definition is found here

In conclusion, the working version is the following (using {0:x} instead of {0:d})

# writing the same 5 chunks of 9 characters
    for i in range(5):
        text = str(i+1) * 9  # concatenate 9 chars
        chunk = '{0:x}\r\n'.format(len(text)) + text + '\r\n'
        self.wfile.write(chunk.encode(encoding='utf-8'))

    # writing close sequence
    close_chunk = '0\r\n\r\n'
    self.wfile.write(close_chunk.encode(encoding='utf-8'))