Python: Foursquare API and Requests requires cookies and javascript

Question

Issue

I am trying to contact the Foursquare API, specifically the checkin/resolve endpoint. In the past this has worked, but lately I am getting blocked with an error message saying I am a bot, and that cookies and javascript cannot be read.

Code

response = "Swarmapp URL" # from previous functions, this isn't the problem
checkin_id = response.split("c/")[1] # To get shortID
url = "https://api.foursquare.com/v2/checkins/resolve"
params = dict(
 client_id = "client_id",
 client_secret = "client_secret",
 shortId = checkin_id,
 v = "20180323")
headers = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'}
time.sleep(8.5) # Limit of 500 requests an hour
resp = requests.get(url = url, params=params, headers = headers)

data = json.loads(resp.text)

This code will work for about 30-40 requests, then error and return an HTML file including: "Please verify you are human", "Access to this page has been denied because we believe you are using automation tools to browse the website.", "Your browser does not support cookies" and so on.

I've tried Googling and searching this site for similar errors, but I can't find anything that has helped. Foursquare API does not say anything about this either.

Any suggestions?

Answer 1

Answer According to the Foursquare API documentation, this code should work:

import json, requests
url = 'https://api.foursquare.com/v2/checkins/resolve'

params = dict(
  client_id='CLIENT_ID',
  client_secret='CLIENT_SECRET',
  v='20180323',
  shortId = 'swarmPostID'
)
resp = requests.get(url=url, params=params)
data = json.loads(resp.text)

However, the bot detection Foursquare uses evidently contradicts the functionality of the API. I found that implementing a try except catch with a wait timer fixed the issue.

import json, requests

url = 'https://api.foursquare.com/v2/checkins/resolve'

params = dict(
  client_id='CLIENT_ID',
  client_secret='CLIENT_SECRET',
  v='20180323',
  shortId = 'swarmPostID'
)
try:
    resp = requests.get(url=url, params=params)
except:
    time.sleep(60) # Avoids bot detection
    resp = requests.get(url=url, params=params)
    try:
        resp = requests.get(url=url, params=params)
    except:
        print("Post is private or deleted.")
        continue
data = json.loads(resp.text)

This seems like a very weird fix. Either Foursquare has implemented a DDoS prevention system that contradicts its own functionality, or their checkin/resolve endpoint is broken. Either way, the code works.