Getting form results using python request

Admittedly, this is my first time using requests. The form I'm attempting to use is here: https://trade.cbp.dhs.gov/ace/liquidation/LBNotice/

Here is my code:

import requests

headers = {
    'Accept': 'application/json, text/javascript, */*; q=0.01',
    'Content-Type': 'application/json; charset=UTF-8',
    'User-Agent': 'Mozilla/5.0',
    'X-Requested-With': 'XMLHttpRequest'
}


payload = {
        "dtPageVars":
        {
            "draw":1,
            "columns":
            [
                {"data":"postedDate","name":"","searchable":"true","orderable":"false","search":{"value":"","regex":"false"}},
                {"data":"eventDate","name":"","searchable":"true","orderable":"false","search":{"value":"","regex":"false"}},
                {"data":"voidedDate","name":"","searchable":"true","orderable":"false","search":{"value":"","regex":"false"}},
                {"data":"event","name":"","searchable":"true","orderable":"false","search":{"value":"","regex":"false"}},
                {"data":"basis","name":"","searchable":"true","orderable":"false","search":{"value":"","regex":"false"}},
                {"data":"action","name":"","searchable":"true","orderable":"false","search":{"value":"","regex":"false"}},
                {"data":"entryNumber","name":"","searchable":"true","orderable":"false","search":{"value":"","regex":"false"}},
                {"data":"portOfEntry","name":"","searchable":"true","orderable":"false","search":{"value":"","regex":"false"}},
                {"data":"entryDate","name":"","searchable":"true","orderable":"false","search":{"value":"","regex":"false"}},
                {"data":"entryType","name":"","searchable":"true","orderable":"false","search":{"value":"","regex":"false"}},
                {"data":"teamNumber","name":"","searchable":"true","orderable":"false","search":{"value":"","regex":"false"}}
            ],
            "order":[],
            "start":0,
            "length":100,
            "search":
            {
                "value":"",
                "regex":"false"
            }
        },
        "searchFields":
        {
            "portOfEntry":"0101",
            "entryType":"01"
        }
}

url = 'https://trade.cbp.dhs.gov/ace/liquidation/LBNotice/'
searchUrl = 'https://trade.cbp.dhs.gov/ace/liquidation/LBNotice/search'

response = requests.post(searchUrl, headers=headers, data=payload)

print(response.status_code)

I keep getting 500 status code but I am not sure why. Could it be the headers I'm using? I've tried many different combinations of headers without luck. Is it possible to use requests for this form or am I better off finding a different approach?

Any help would truly be appreciated. Thank you.

Solution

Cause

In headers you are telling the server you are sending it a JSON data but in request body, you are sending Form data. There is inconsistency.

Solution: send json data

response = requests.post(url, json=payload)

Remove headers too.

Explanation

In requests library, if you pass a dictionary to data parameter, your dictionary of data will automatically be form-encoded when the request is made. But based on your header, it seems you want to send a JSON data: 'Content-Type': 'application/json; charset=UTF-8',

headers = {
    'Accept': 'application/json, text/javascript, */*; q=0.01',
    'Content-Type': 'application/json; charset=UTF-8',
    'User-Agent': 'Mozilla/5.0',
    'X-Requested-With': 'XMLHttpRequest'
}

So, basically In headers you are telling the server you are sending it a JSON data but in request body, you are sending Form data. If you don't want to send a form data (if you want to send JSON or just string data), you can send a string data directly.

If you pass in a string instead of a dict, that data will be posted directly.

requests.post(url, data=json.dumps(payload))

json.dumps(payload) essentially turns your dictionary into (JSON) string. But it doesn't change/set headers. You need to set headers manually.

There is an easier way:

If you need a header set to application/json and you don’t want to encode the dict yourself, you can pass the dcit to json parameter of requests.post method instead of data parameter.

response = requests.post(url, json=payload)

It will both update/set your headers to application/json and stringifies your dictionary.

Tip

If you do not need to specify custom headers, you don't need to add headers to the request, i.e., you can go with default options.