How to translate url requests using pycurl to python-requests?

Question

I would like to modify following code to run use 'requests' module. I have the following code which is working on a website:

def post(url, message, key, sign):

    curl = pycurl.Curl()
    curl.setopt(pycurl.URL, url)
    curl.setopt(pycurl.SSL_VERIFYPEER, 0)
    curl.setopt(pycurl.SSL_VERIFYHOST, 0)
    buf = cStringIO.StringIO()
    curl.setopt(pycurl.WRITEFUNCTION, buf.write)
    curl.setopt(pycurl.POSTFIELDS, message)
    curl.setopt(pycurl.HTTPHEADER, ['Key:' + key,
                                'Sign:' + (sign)])
    curl.perform()
    response = buf.getvalue()
    buf.close()
    return response        

I tried accessing the website with requests and got rejected on invalid request values using following code:

def post(url, message, key, sign):
    import requests
    session = requests.session()
    session.headers = {'Key': key, 'Sign': sign}
    response = session.post(url, message)
    return response

What am I doing wrong that these methods don't behave the same?

Thank you.

Answer 1

Using Pycurl:

POST /post HTTP/1.1
User-Agent: PycURL/7.32.0
Host: 127.0.0.1:4000
Accept: */*
Key:key
Sign:sign
Content-Length: 3
Content-Type: application/x-www-form-urlencoded

foo

With requests:

POST /post HTTP/1.1
Host: 127.0.0.1:4000
Accept-Encoding: identity
Content-Length: 3
Key: key
Sign: sign

foo

There are several differences which could lead to your error:

• Missing User-Agent and Accept headers. This is because you overwrite the session.headers attribute which contains those default headers. Try this instead:

  session.headers.update({'Key': key, 'Sign': sign})

• Missing Content-Type header. I think you passed a string as the message parameter. Requests doesn't know that this is application/x-www-form-urlencoded and therefore doesn't set the relevant header. Either:

  • Set the header yourself
  • Better: pass requests a dictionary of your parameters. They will be encoded and declared correctly