I've had a Cloud Function that makes Vision API requests, specifically Document Text Detection requests. My peak requests rate is usually around ~120-150 requests per minute on an average day.
I've suddenly been getting resource quota exceeded errors for Vision API requests with a request rate at 2500 requests per minute. Some things to note:
I've had no code changes in 3 months
I deleted and redeployed the Cloud Function making these requests to stop any problematic image that was causing a runaway loop
My code calling the API nor the cloud functions themselves were getting retried so there really wasn't a way that I could exponentially increase my request rate overnight with no changes introduced.
The service account making the Vision calls is making the normal amount of requests and is only in use by the cloud function i.e. not being used by someone's local script
I've since turned on retries to mitigate this issue since it'll "work" with exponential back off but this is expensive to do, especially with the vision API. Is there anything I can do to find out the root cause of this issue?
To identify the specific quota being exceeded, Stackdriver API helps by using Monitoring quota metrics as explained here .
GCP lets you specify quota being exceeded in greater depth using the Stackdriver API and UI, with quota metrics appearing in the Metrics Explorer.