Python Requests form automation - combine POST request responses (XMLHttpRequests) and text inputs for a single form

Question

Question:

I’m trying to automate form filling for this site: https://secure.ami.co.nz/css/car/step1#noBack

How can I combine:

1. The responses from Form elements that require “onclick” POST requests (XMLHttpRequests) , and,
2. Other Form input fields data (text/radio inputs) that just need to be passed values ...

... In the same session, to fill this form completely, and then submit it?

Background

Im working with the car insurance quote web-form to get quotes (Fields are car rego; address; name; email etc). Calling it example_site - which I am loading as so:

import requests
url = "https://example_site.com"
payload = {}
headers = {}
response_site = requests.request("GET", url, headers=headers, data = payload)

The form has text inputs such as name, but also some onclick events (XMLHttprequests) for address and rego (to determine vehicle make and address).

I have determined how to submit a POST request for these onclick events in isolation to generate a response, by using dev tools and the network tab, to generate the below code. Example below for rego = abc321:

url = "https://example_site.com/searchVehicleByRegNo"
payload = "regNo=abc321"
headers = {
     'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
     'Host': 'secure.ami.co.nz',
     'X-Requested-With': 'XMLHttpRequest',
}
# Post response contains all information on vehicle
response_rego = requests.request("POST", url, headers=headers, data = payload)

Note: avoiding Selenium as is overkill for this purpose

Answer 1

It doesn't Seem like you need to do those request in the same session, infact if you look at the final request that was made at secure.ami.co.nz/url/css/car/step2/:

vehicle.redbookVehicleKey=NZVDAEW2001AEAF
garagingAddress.manualSuburbTown=ARO VALLEY, WELLINGTON 6011
policy.coverageTypeCd=FULL
vehicle.isManualEntry=false
vehicle.searchRegNo=abc321
garagingAddress.fullAddress
garagingAddress.manualUnitNumber=a
garagingAddress.manualStreetNumber=a
garagingAddress.manualStreetName=a
garagingAddress.isManual=true
garaging.garagingCd=OTH
policy.effective=2020-09-18
vehicle.hasImmobilizer=false
vehicle.isBusinessUse=false
transferKey=xxxx
ccssSessionId=xxxx
...

You'll find the "Vehicle Key" was sent as the response for the searchVehicleByRegNo

{
  "registrationNo_vehicleList": [
    {
      "id": null,
      "vehicleCode": "NZVDAEW2001AEAF",
      "registrationNo": "ABC321",
...

Which you can use in the payload of your final request.

Though Mind you, the website uses recaptcha, and its sole job is to stop what you are doing right now, so you'll be facing a lot more trouble.

P.S. I must say though searchVehicleByRegNo has some interesting information that is not shown on the webpage, I don't know what you are looking for but I recommend looking at the response.