Search code examples
pythoncookiespython-requests

Cannot get cookies with python requests while postman, curl, and wget work

I'm trying to authenticate on a French water provider website to get my water consumption data. The website does not provide any api and I'm trying to make a python script that authenticates on the website and crawls the data. My work is based on a working Domoticz python script and a shell script. The workflow is the following:

  1. Get a token from the website
  2. Authenticate with login, password, and token get at step 1
  3. Get 1 or more cookies from step 2
  4. Get data using the cookie(s) from 3

I'm stuck at step 2 where I can't get the cookies with my python script. I tried with postman, curl, and wget and it is working. I even used the python code generated by postman and I still get no cookies.

Heres is a screenshot of my postman post request enter image description here which gives two cookies in the response.

And here is my python code:

import requests

url = "https://www.toutsurmoneau.fr/mon-compte-en-ligne/je-me-connecte"

querystring = {"_username":"mymail@gmail.com","_password":"mypass","_csrf_token":"knfOIFZNhiCVxHS0U84GW5CrfMt36eLvqPPYGDSsOww","signin[username]":"mymail@gmail.com","signin[password]":"mypass","tsme_user_login[_username]":"mymail@gmail.com","tsme_user_login[_password]":"mypass"}

payload = ""
headers = {
    'Accept': "application/json, text/javascript, */*; q=0.01",
    'Content-Type': "application/x-www-form-urlencoded",
    'Accept-Language': "fr,fr-FR;q=0.8,en;q=0.6",
    'User-Agent': "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Mobile Safari/537.36",
    'Connection': "keep-alive",
    'cache-control': "no-cache",
    'Postman-Token': "c7e5f7ca-abea-4161-999a-3c28ec979628"
    }

response = requests.request("POST", url, data=payload, headers=headers, params=querystring)

print(response.cookies.get_dict())

The output is {}.

I cannot figure out what I'm doing wrong. If you have any help to provide, I'll be happy to get it.

Thanks for reading.

Edit:

Some of my assumptions were wrong. The shell script was indeed working but not Postman. I was confused because of response 200 I receive.

Solution

  • So I answer my own question.

    First, when getting the token at step 1, I receive a cookie. I'm supposed to use this cookie when logging in which I did not do before.

    Then, when using this cookie and the token to log in step 2, I was not able to see any cookie in the response I receive while I was well connected (I find in the content a "disconnect" string which is here only if well logged in). That's a normal behavior since cookies are not sent in the response of a post request.

    I had to create a requests.session to post my log in form, and the session stores the cookie.

    Now, I'm able to use this information to grab the data from the server. Hope that will help others.