Understanding the difference between these two python requests POST calls (data vs. json args)

This is a toy, non-reproducible example as I can't share the original. I think it's answerable, and might help others. From other SO posts like this and this, my understanding is that given some dictionary d of params, these are equivalent:

requests.post(url, data=json.dumps(d))
requests.post(url, json=d)

The parameters for a token endpoint were defined in documentation like so:

url: {{base_url}}/token
parameters
- grant_type={{password}}
- username={{username}}
- password={{password}}
- scope={"account":"{{account}}", "tenant":"{{tenant}}"}

I started with this, with variables loaded from a .env file:

resp = requests.post(f'{base_url}/token',
                     json={'grant_type': 'password', 'username': uname, 'password': pwd,
                           'scope': {'account': account, 'tenant': tenant}})
resp.text
# '{"error":"unsupported_grant_type"}'

I tried changing to the data argument, and got a more sane error:

resp = requests.post(f'{base_url}/token',
                     data={'grant_type': 'password', 'username': uname, 'password': pwd,
                           'scope': {'account': account, 'tenant': tenant}})
resp.text
# '{"error":"invalid_grant","error_description":"{\\"ErrorMessage\\":\\"Error trying to Login  - User [username] Account [] Unexpected character encountered while parsing value: a.

I tried a few other things like forcing quotes around args (e.g. {'account': f"{account}"}) without success, and ultimately succeeded with this "hybrid" method:

resp = requests.post(f'{base_url}/token',
                 data={'grant_type': 'password', 'username': uname, 'password': pwd,
                       'scope': json.dumps({'account': account, 'tenant': tenant})})

My questions:

is this nuance "real" vs. the straightforward reading of the linked questions? Namely, it seemed like one either uses data=json.dumps(d) or json=d, but I have not found an answer mixing the two (and wrapping the entire data arg in json.dumps() breaks my working final version)
as a relative noob in APIs/network things, would this be discernible to me from the documentation arguments listed above, or was trial and error the only way to discover this?
given my final solution was there a better/more correct way to pass these params?

Solution

There is a big difference in providing json= or data= argument.

Providing json= will send Content-Type: application/json and will format the data you sent as json-string. E.g. the following request:

resp = requests.post(url, json={"a": 42, "b": 55})

Will result in the server receiving this:

POST / HTTP/1.1
Host: localhost:12001
User-Agent: python-requests/2.28.1
Accept-Encoding: gzip, deflate, br
Accept: */*
Connection: keep-alive
Content-Length: 18
Content-Type: application/json


{"a": 42, "b": 55}

whereas sending the same as data will send it as form data with content-type application/x-www-form-urlencoded:

resp = requests.post(url, data={"a": 42, "b": 55})

Will result in the server receiving this:

POST / HTTP/1.1
Host: localhost:12001
User-Agent: python-requests/2.28.1
Accept-Encoding: gzip, deflate, br
Accept: */*
Connection: keep-alive
Content-Length: 9
Content-Type: application/x-www-form-urlencoded


a=42&b=55

If you are using the data=json.dumps(...) variant, you essentially pass a single string as data. The string is json-format but requests does not know this (it could be anything). In this case, it seems, requests does not send any Content-Type, you would have to explicitly set it in the code.

Now, which variant is correct, depends on the API you are using, but as the data seems to be structured, it cannot be encoded well into form-data, so probably json is what you want to send.

With respect to the mixed solution, I would argue that it shows that the API is very ill-designed, but here is what the server receives:

POST / HTTP/1.1
Host: localhost:12001
User-Agent: python-requests/2.28.1
Accept-Encoding: gzip, deflate, br
Accept: */*
Connection: keep-alive
Content-Length: 145
Content-Type: application/x-www-form-urlencoded


grant_type=password&username=treuss&password=s3cret&scope=%7B%22account%22%3A+%22treuss%40stackoverflow.com%22%2C+%22tenant%22%3A+%22tenant%22%7D

Everything after scope= is the encoded json-string generated from the json.dump in your example, i.e. an url-encoded version of the string {"account": "[email protected]", "tenant": "tenant"}