Search code examples
pythonopenstreetmapreverse-lookup

How batch reverse lat/long on OSM

I have a list of lat/long coordinates and need to obtain the state for each. That can be done with the code:

df = pd.read_csv('SOL_A.dsv', delimiter = '|', low_memory=False)
for index, row in df.iterrows(): 
    lat = row['LAT']
    lon = row['LONG']
    g = geocoder.osm([lat,lon], method='reverse')
    st = '_UN'
    if g.state != None:
        st = g.state
    geom_states.append(st)
df['STATE'] = geom_states

But for my ~5k records it eventually starts yielding Status code 429 from https://nominatim.openstreetmap.org/search: ERROR - 429 Client Error: Too Many Requests for URL: tps://nominatim.openstreetmap.org/search?q=0.0%2C+0.0&format=jsonv2&addressdetails=1&limit=1 which is expected.

I only have to process this once and don't mind if it takes a whole day. I read through the OSM Acceptable Use Policy and it goes:

  • No heavy uses (an absolute maximum of 1 request per second).
  • Provide a valid HTTP Referer or User-Agent identifying the application (stock User-Agents as set by HTTP libraries will not do).
  • Clearly display attribution as suitable for your medium.
  • Data is provided under the ODbL license which requires to share alike (although small extractions are likely to be covered by fair usage / fair dealing).

So.. It should be possible (?)

I tried adding my API Key (geocoder.osm([lat,lon], method='reverse', key=API_KEY)) and also added a time.sleep(1.1) before each call to be sure, but didn't really help.

Ideas?

Solution

  • Nominatim Usage Policy clearly states:

    • No heavy uses (an absolute maximum of 1 request per second).
    • Provide a valid HTTP Referer or User-Agent identifying the application (stock User-Agents as set by http libraries will not do).
    • Clearly display attribution as suitable for your medium.
    • Data is provided under the ODbL license which requires to share alike (although small extractions are likely to be covered by fair usage / fair dealing).

    Looks like you don't limit your requests to a maximum of 1 per second. Also I'm not sure if you pass a valid HTTP referer (aka user-agent).

    Note that this usage policy only applies to OSM's public Nominatim instance. You can always install your own Nominatim service or switch to an alternative/commercial Nominatim instance.