Mock S3 server stalls if using HTTP/1.1

Question

I'm writing a test where a simple mock S3 is loaded up in the test environment using http.server.HTTPServer/http.server.BaseHTTPRequestHandler, to test multipart download behaviour involving Boto's S3Transfer.

It works fine, unless I specify that the server uses HTTP/1.1. In this case, it would download 2 8mb parts of a 100mb file, and then just hang. I would like for the mock server to use HTTP/1.1 since that's what the real S3 uses (I believe).

A simplified version of the test is below, that can be run by...

pip3 install boto3
python3 test.py    

---

# test.py

import http.server
import re
import threading

import boto3
from botocore import (
    UNSIGNED,
)
from botocore.client import (
    Config,
)

length = 100 * 2**20

class MockS3(http.server.BaseHTTPRequestHandler):
    # If the below line is commented, the download completes
    protocol_version = 'HTTP/1.1'

    def do_GET(self):
        range_header = self.headers['Range']
        match = re.search(r'^bytes=(\d+)-(\d*)', range_header)
        start_inclusive_str, end_inclusive_str = match.group(1), match.group(2)
        start = int(start_inclusive_str)
        end = int(end_inclusive_str) + 1 if end_inclusive_str else length
        bytes_to_send = end - start

        self.send_response(206)
        self.send_header('Content-Length', str(bytes_to_send))
        self.end_headers()
        self.wfile.write(bytearray(bytes_to_send))

    def do_HEAD(self):
        self.send_response(200)
        self.send_header('Content-Length', length)
        self.end_headers()

server_address = ('localhost', 5678)
server = http.server.HTTPServer(server_address, MockS3)
thread = threading.Thread(target=server.serve_forever)
thread.daemon = True
thread.start()

class Writable():
    def write(self, data):
        pass

s3_client = boto3.client('s3',
  endpoint_url='http://localhost:5678',
  config=Config(signature_version=UNSIGNED),
)

s3_client.download_fileobj(
  Bucket='some',
  Key='key',
  Fileobj=Writable(),
)

Note that Writable is deliberately not seekable: in my real code, I'm using a non-seekable file-like object.

Yes, moto can be used to to make a mock S3, and I do so for other tests, but for this particular test I would like "real" server. There are custom file objects involved, and want to ensure that S3Transfer, and other code that isn't relevant to this question, behaves together as I expect.

How can I setup a mock S3 server that uses HTTP/1.1 and that S3Transfer can download from?

Answer 1

There is a bug in your threading logic. What you're currently doing is serving on a separate thread, but what you really want to do is concurrently handle requests on multiple threads.

This can be achieved by creating a very dumb HTTP server which just mixes in a threading capabilities:

class ThreadingServer(ThreadingMixIn, HTTPServer):
    pass

and serving from this server instead of the base HTTPServer.

As for why this works with HTTP/1.0, the connection was closed after a single request was serviced.