I am working on a project using Spotipy that makes a lot of requests. ~18 hours ago, I presume that I hit the rate limit because I was sending a lot of requests and the code started to just stop and hang forever once it got to a line that made an API reference. After that I gave up and went to bed. ~12 hours ago I decided to rewrite the whole project since it was a mess, and so I could implement spacing out of the API requests. I also learned that I could set retries = 0 so it would throw an error rather than indefinitely hang the program. I was rewriting the project, I was only able to make 1 API request before I started getting http 429 errors every time. I left it alone for another couple of hours, until now when i tried troubleshooting some more, and the same thing happened where I was able to make one success and then it would throw errors.
restart.py:
import spotipy # type: ignore
from spotipy.oauth2 import SpotifyOAuth # type: ignore
import time
def create_client() -> spotipy.Spotify:
sp: spotipy.Spotify = spotipy.Spotify(
retries=0,
auth_manager=SpotifyOAuth(
client_id='my_id',
client_secret='_my_secret',
scope='playlist-read-private playlist-modify-public playlist-modify-private',
redirect_uri='http://localhost:8888/callback'
)
)
return sp
class APIRequests:
def __init__(self, user_id: str) -> None:
self.recent_request: float = time.time()
self.sp: spotipy.Spotify = create_client()
self.user_id: str = user_id
def get_playlist_tracks(self, playlist: str, offset: int, retries: int=0):
delta_time = time.time() - self.recent_request
if delta_time > 1:
try:
return self.sp.user_playlist_tracks(self.user_id, playlist, offset=offset, limit=100)
except spotipy.exceptions.SpotifyException as e:
if e.http_status == 429:
print(e)
print(e.headers.get('retry-after'))
print(f'retrying: waiting {(2 ** retries) * 4} seconds...')
time.sleep((2 ** retries) * 4)
retries += 1
return self.get_playlist_tracks(playlist, offset, retries)
else:
raise e
else:
time.sleep(1 - delta_time)
self.get_playlist_tracks(playlist, offset)
api_client: APIRequests = APIRequests('my_user_id')
def get_playlist_tracks(playlist: str) -> list:
offset: int = 0
tracks: list = []
while True:
print('sending request ...')
request_batch = api_client.get_playlist_tracks(playlist, offset)
print('request recieved')
for track in request_batch['items']:
tracks.append(track['track'])
if not request_batch["next"]:
break
return tracks
if __name__ == '__main__':
tracks = get_playlist_tracks('my_playlist')
print(len(tracks))
I my rewritten code, all requests are separated by 1 second at a minimum (though it doesn't even matter because I cant get past 1 request anyways). Additionally, upon recieving an http 429 error, it will wait an exponentially increasing amount of time before retrying. It has no effect. I have run it up to until there was 2 minutes in between attempts, and it still threw the error.
What doesn't make sense to me is that the rate limit only applies to the number of requests made in any 30 second window. If I am waiting up to 2 minutes in between there should be absolutely no issues. If I only get error, I would have assumed that I was on some kind of 24-hour API ban or something along those lines, but being able to make 1 successfull request doesn't make sense.
I tried with the cliend ID/secret for both of the 2 apps I have on my Spotify developer account and I had the same issue.
Am I approaching this correctly? Do I have an issue in my code? Any help is appreciated.
Additionally:
From the Spotify API Documentation:
When your app has been rate limited it will receive a 429 error response from Spotify. Your app can use this information as a cue to slow down the number of API requests that it makes to the Web API. The header of the 429 response will normally include a Retry-After header with a value in seconds. Consider waiting for the number of seconds specified in Retry-After before your app calls the Web API again.
I don't see anything like this printed in the error message. How can I access the retry-after and would that solve my issue?
After reading the error message more closely, I found that I was timed out for hitting the rate limit too many times. That time, my time out lasted a little less than 24 hours. The only solution is to wait.
I would presume that the less you attempt to send requests during your timeout, the shorter it is, since the two more times I have been timed out since I asked the question were much shorter than the first one and I didn’t check to see if I was un-timed out as much as I did the first time.
As to avoiding the time out, I am not sure. Even with limiting my requests to 1 per second, I still got timed out twice more, but I was also sending requests nonstop for over an hour when I got timed out.
To solve the time out: just wait and occasionally check to see if you can send requests again. For me it lasted around 12 hours the second times.
To avoid the time out: don’t send too many requests at once.