Search code examples
pythonajaxpython-3.xpython-requestsxmlhttprequest

Incomplete response for XHR request in python with requests.get()


I am trying to scrape German zip codes (PLZ) for a given street in a given city using Python's requests on this server. I am trying to apply what I learned here.

I want to return the PLZ of Schanzäckerstr. in Nürnberg.

import requests

url = 'https://www.11880.com/ajax/getsuggestedcities/schanz%C3%A4ckerstra%C3%9Fe%20n%C3%BCrnberg?searchString=schanz%25C3%25A4ckerstra%25C3%259Fe%2520n%25C3%25BCrnberg'

data = 'searchString=schanz%25C3%25A4ckerstra%25C3%259Fe%2520n%25C3%25BCrnberg'


headers = {"Authority": "wwww.11880.com",
                "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko/20100101 Firefox/70.0",
                "Accept": "application/json, text/javascript, */*; q=0.01",
                "Accept-Language": "de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7",
                "Accept-Encoding": "gzip, deflate, br",
                "X-Requested-With": "XMLHttpRequest",
                "Content-Type": "application/x-www-form-urlencoded; charset=UTF-8",
                "Content-Length": "400",
                "Origin": "https://www.postleitzahlen.de",
                "Sec-Fetch-Site": "cross-site",
                "Fetch-Mode": "cors",
                "DNT": "1",
                "Connection": "keep-alive",
                "Referer": "https://www.postleitzahlen.de",
               }
multipart_data = {(None, data,)}

session = requests.Session()
response = session.get(url, files=multipart_data, headers=headers)

print(response.text)

The above code yields an empty response of the type 200. I want to return:

'90443'

Solution

  • I was able to solve this problem using nominatim openstreetmap API. One can also add street numbers

    import requests
    city = 'Nürnberg'
    street = 'Schanzäckerstr. 2'
    
    response = requests.get( 'https://nominatim.openstreetmap.org/search', headers={'User-Agent': 'PLZ_scrape'}, params={'city': city, 'street': street[1], 'format': 'json', 'addressdetails': '1'}, )
    print(street, ',', [i.get('address').get('postcode') for i in response.json()][0])
    

    Make sure to only send one request per second.