Search code examples
pythonhttppython-requestssimplehttpserver

Why is difference between form sent from browser and Requests when catched by SimpleHttpServer


I have SimleHttpServer with `do_POST' function

def do_POST(self):

    print self.path

    form = cgi.FieldStorage(
        fp=self.rfile, 
        headers=self.headers,
        environ={'REQUEST_METHOD':'POST',
                 'CONTENT_TYPE':self.headers['Content-Type'],
        })
    for key in form.keys():
        print key

And now the problem, If I send the request with browser (HTML form like this)

<form method="post" action="/">
  <input type="text" name="a" value="1"><br>
  <input type="submit" value="Send">
</form>

then values are:

self.path = /
form.keys = [a]

If I send the same request with python Requests

parameters = {
    "a": 2,
}

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64)",
    "Content-Type": "application/x-www-form-urlencoded",
    "Referer": "http://127.0.0.1:8001/",
    "Host": "127.0.0.1:8001",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Encoding": "gzip, deflate",
    "Connection": "keep-alive",
    "Upgrade-Insecure-Requests": "1",
}

r = requests.post("http://127.0.0.1:8001", params=parameters, headers=headers)

the values on server are

self.path = /?a=2
form.keys = []

Where is the problem? What is different? I copied the headers from browser, so everything should be the same.

Thanks in advance for any help or clue.

Working workaround:

If you run into this issue, you can use following workaround in do_POST (you need to import cgi, urlparse):

    if "?" in self.path:
        qs = self.path.split("?")[1]
        query_dict = dict(urlparse.parse_qsl(qs))
    else:       
        form = cgi.FieldStorage(
            fp=self.rfile, 
            headers=self.headers,
            environ={'REQUEST_METHOD':'POST',
                     'CONTENT_TYPE':self.headers['Content-Type'],
            })
        query_dict = {key: form.getvalue(key) for key in form.keys()}

But this is not answer for the question.


Solution

  • If you want to send data as POST (i.e. not as GET query-string prameters) you should pass it as data parameter, not params:

    Typically, you want to send some form-encoded data — much like an HTML form. To do this, simply pass a dictionary to the data argument. Your dictionary of data will automatically be form-encoded when the request is made

    (Source: More complicated POST requests)

    params -- (optional) Dictionary or bytes to be sent in the query string for the Request.

    data -- (optional) Dictionary or list of tuples [(key, value)] (will be form-encoded), bytes, or file-like object to send in the body of the Request.

    (Source: Main Interface, emphasis mine)