Search code examples
mitmproxy

How to stream response in mitmproxy

Since I use a crappy internet connection I have to download large packages using a download manager then stream them to chocolatey (and it still lacks resume capability). To do the MITM job I use mitmproxy and a simple script.

Script

from mitmproxy import http

dlFileLoc = "C:\\Users\\Pouya\\Downloads\\flutter_windows_2.0.4-stable.zip"

def responseheaders(flow: http.HTTPFlow):
    flow.response.stream = True

def request(flow: http.HTTPFlow):
    if flow.request.pretty_host.find("storage.googleapis.com") >= 0:
        # flow.intercept()
        with open(dlFileLoc, mode="rb") as fb:
            flow.response = http.HTTPResponse.make(content=fb.read())
            # flow.resume()

Loading script to mitmdump

mitmdump.exe -s D:\Code\cli\rend\main.py

setting proxy for chocolatey

choco upgrade flutter -y --proxy http://127.0.0.1:8080

and mitmproxy crashes

127.0.0.1:50395: Traceback (most recent call last):
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\server.py", line 121, in handle
    root_layer()
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\modes\http_proxy.py", line 9, in __call__
    layer()
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\tls.py", line 285, in __call__
    layer()
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\http1.py", line 100, in __call__
    layer()
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\http.py", line 204, in __call__
    if not self._process_flow(flow):
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\http.py", line 277, in _process_flow
    return self.handle_regular_connect(f)
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\http.py", line 222, in handle_regular_connect
    layer()
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\tls.py", line 285, in __call__
    layer()
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\http1.py", line 100, in __call__
    layer()
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\http.py", line 204, in __call__
    if not self._process_flow(flow):
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\http.py", line 457, in _process_flow
    self.send_response_body(f.response, chunks)
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\http1.py", line 71, in send_response_body
    for chunk in http1.assemble_body(response.headers, chunks, response.trailers):
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\net\http\http1\assemble.py", line 44, in assemble_body
    for chunk in body_chunks:
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\net\http\http1\read.py", line 137, in read_body
    content = rfile.read(chunk_size)
AttributeError: 'NoneType' object has no attribute 'read'
Traceback (most recent call last):
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\server.py", line 121, in handle
    root_layer()
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\modes\http_proxy.py", line 9, in __call__
    layer()
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\tls.py", line 285, in __call__
    layer()
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\http1.py", line 100, in __call__
    layer()
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\http.py", line 204, in __call__
    if not self._process_flow(flow):
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\http.py", line 277, in _process_flow
    return self.handle_regular_connect(f)
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\http.py", line 222, in handle_regular_connect
    layer()
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\tls.py", line 285, in __call__
    layer()
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\http1.py", line 100, in __call__
    layer()
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\http.py", line 204, in __call__
    if not self._process_flow(flow):
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\http.py", line 457, in _process_flow
    self.send_response_body(f.response, chunks)
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\http1.py", line 71, in send_response_body
    for chunk in http1.assemble_body(response.headers, chunks, response.trailers):
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\net\http\http1\assemble.py", line 44, in assemble_body
    for chunk in body_chunks:
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\net\http\http1\read.py", line 137, in read_body
    content = rfile.read(chunk_size)
AttributeError: 'NoneType' object has no attribute 'read'

Environment

> mitmdump.exe --version
Mitmproxy: 6.0.2
Python:    3.9.4
OpenSSL:   OpenSSL 1.1.1i  8 Dec 2020
Platform:  Windows-10-10.0.19041-SP0
Solution

  • Although my prior script should work but somehow it didn't I eventually get it working by changing the script as follows:

    from mitmproxy import http

dlFileLoc = "C:\\Users\\Pouya\\Downloads\\flutter_windows_2.0.4-stable.zip"


def request(flow: http.HTTPFlow):
    if flow.request.pretty_host.find("storage.googleapis.com") >= 0:
        with open(dlFileLoc, mode="rb") as fb:
            body = fb.read()
            size = str(len(body))
            flow.response = http.HTTPResponse.make(
                status_code=200,
                content=body,
                headers={
                    "Content-Type": "application/octet-stream",
                    "Content-Length": size,
                },
            )

    Then pass --set stream_large_bodies to a reasonable value something like 10Mb

    mitmdump.exe -s D:\Code\cli\rend\main.py --set stream_large_bodies=10m

    And chocolatey has to request via mitmproxy

    choco upgrade flutter -y --proxy http://127.0.0.1:8080