"HTTPError: 429 Client Error: Too Many Requests for url" in Python [w/ Azure developer account]

Getting continual exception when trying to run Python code on a small sample directory of 5-20 images.

I have a developer account with Microsoft (as of this morning) and have inquired with Azure Support, but this issue wasn't solvable through chat alone. They've instructed me to post this here, so I'm sorry if it comes off as an eye-roll to everyone else!

There's very little documentation on this web-wide. Maybe this'll help someone else, because this is potentially a game-changer for people in graphics and media who have a huge amount of disorganized images like I do.

Note that this code DID work once last night! But only once. No idea what went wrong!

# API reference :

# https://westus.dev.cognitive.microsoft.com/docs/services/5adf991815e1060e6355ad44/operations/56f91f2e778daf14a499e1fa

# 参考 : https://ledge.ai/microsoft-computer-vision-api/

# 機能概要 : img フォルダ中の画像をAI解析し、ファイルのリネームを行います。

# 使い方 : python3 cv_demo.py

# 注意 : サブスクリプションキーは変更してください


import requests

import glob

import os

import time

import urllib



subscription_key = "i do have a real subscription key don't worry"

assert subscription_key



vision_base_url = "https://westcentralus.api.cognitive.microsoft.com/vision/v2.0/"

analyze_url = vision_base_url + "analyze"



# ファイル名を変更

def file_rename(list_1, list_2):

    for i in range(len(list_1)):

        os.rename(list_1[i], './img/' + list_2[i] + '.jpg')



def ms_computer_vision_api(filepath):

    headers = {'Ocp-Apim-Subscription-Key': subscription_key,'Content-Type': 'application/octet-stream'}

    params = urllib.parse.urlencode({
    # Request parameters
   'visualFeatures': 'Categories,Tags,Description,Faces'
   })





    img = open(filepath, 'rb')

    img_byte = img.read()



    response = requests.post(analyze_url, data=img_byte, headers=headers, params=params)

    response.raise_for_status()



    return response.json()



if __name__ == "__main__":

    # 画像ファイルを配列に格納

    image_file = glob.glob('./img/*')



    vision_file_name = []



    start = time.time()



    # Computer Vision APIにリクエストして結果を取得

    for i in range(len(image_file)):

        json_data = ms_computer_vision_api(image_file[i])



        # 生成された文章を取得

        file_name = json_data['description']['captions'][0]['text']

        vision_file_name.append(file_name)



    # 文章の空白をファイル名用にアンダーバーに修正

    for i in range(len(vision_file_name)):

        vision_file_name[i] = vision_file_name[i].replace(' ', '_')



    file_rename(image_file,vision_file_name)



    # 経過時間を出力

    print("elapsed_time:{0}".format(time.time() - start) + "[sec]")

 File "C:\Users\MyName\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\requests\models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://renameimages.cognitiveservices.azure.com/vision/v2.0/analyze?visualFeatures=Description

Solution

Based on your code this error comes from a loop which requesting a cognitive API.

Actually each Cognitive Service has the limit of TPS (Transaction Per Second), and will report 429 error when exceed the TPS. Even if specific Cognitive Service has a higer TPS such as 50 TPS, maybe you still have 429 error. You should always use the following policy to avoid 429 in the future.

The following is the explanation 429 and how to handle 429.

HTTP 429 would indicate RateLimitExceeded, meaning you are making too many API calls per second or minute.
When HTTP 429 happens, you must wait for some time to call API again, or else the next call of API will be refused. Usually we retry the operation using something like an exponential back off retry policy to handle the 429 error:

2.1) You need check the HTTP response code in your code.

2.2) When HTTP response code is 429, then retry this operation after N seconds which you can define by yourself such as 10 seconds…

For example, the following is a response of 429. You can set your wait time as (26 + n) seconds. (PS: you can define n by yourself here, such as n = 5…)

{
    "error":{
       "statusCode": 429,
        "message": "Rate limit is exceeded. Try again in 26 seconds." 
    }
}

2.3) If step 2 succeed, continue the next operation.

2.4) If step 2 fail with 429 too, retry this operation after N*N seconds (you can define by yourself too) which is an exponential back off retry policy..

2.5) If step 4 fail with 429 too, retry this operation after NNN seconds…

2.6) You should always wait for current operation to succeed, and the Waiting time will be exponential growth.