Search code examples
pythonhtml-parsinghttp-status-code-429grequests

How to avoid 429 error when using grequests?

I want to make a parser using "grequests", but as expected, due to many requests, i get an error 429.

import grequests


sites = [f"https://fantlab.ru/work{i}" for i in range(1, 10)]

response = (grequests.get(url, timeout=5)  for url in sites)
resp = grequests.map(response)

print(resp)

[<Response [200]>, <Response [200]>, <Response [200]>, <Response [404]>, <Response [429]>...]
Solution

  • A 429 (Too many requests) status code is returned when you are sending too many requests. You could try showing down the requests (using time.sleep(n) from the time module, or see if the owner of the server can increase your quota.