Currently integrating a project with Google Drive using the Drive API Client Library for Java, when using a Service Account to impersonate a user to retrieve its Drive contents, userRateLimitExceeded is triggered when the number of reported requests is way below the lowest defined quota that can be seen in the Console.
The domain being used to test Google Drive integration currently has requests per user per 100 seconds quota set to 1000. During a run of the program, where a Service Account is used to impersonate a user and retrieve its files, the Java client casts a GoogleJsonResponseException due to usageLimits, namely userRateLimitExceeded. However, the maximum ever console reported spike is of 198 requests/minute, way below said limit.
Have attempted setting a random quotaUser parameter per request as detailed in the error resolution page but this wielded the same result.
The exponential backoff policy described in the documentation of starting waits at 1 second to then increment really does not help much, with requests basically trickling through after waits of 20, 30 seconds as the quota keeps getting triggered.
To diagnose this, have created a small unit test for different run scenarios, where we run a 1000 callables that simply list the first 100 files in a well known Google Drive area in said domain using an instance of Drive object.
public class GoogleDriveRequestTest {
private Drive googleDrive;
//other class attributes
@Before
public void setup() throws Exception {
//sensitive data
final Credential credential = new GoogleCredential.Builder()
.setTransport(httpTransport)
.setJsonFactory(JacksonFactory.getDefaultInstance())
.setServiceAccountId(serviceAccountId)
.setServiceAccountPrivateKey(gdrivePrivateKey)
.setServiceAccountScopes(ImmutableSet.of(DriveScopes.DRIVE,
DirectoryScopes.ADMIN_DIRECTORY_USER,
DirectoryScopes.ADMIN_DIRECTORY_GROUP))
.setServiceAccountUser(serviceAccountUser)
.build();
this.googleDrive = new Drive.Builder(httpTransport, JacksonFactory.getDefaultInstance(), credential)
.setApplicationName("Google Drive API Load Test")
.build();
//other initialization code
}
@Test
public void shouldRequestListOfFilesOverAndOverAgain() {
Stream<Integer> infiniteStream = Stream.iterate(0, i -> i + 1);
AtomicInteger requestCounter = new AtomicInteger(1);
infiniteStream
.limit(1000)
.map(i -> new GoogleDriveCallable(requestCounter))
.parallel()
.map(executorService::submit)
.map(execution -> {
try {
return execution.get();
} catch (Exception e) {
throw new RuntimeException(e);
}
})
.forEach(triple -> System.out.println("Completed request n " + triple.getMiddle() + " in " + triple.getRight() + " millis on thread " + triple.getLeft()));
}
private class GoogleDriveCallable implements Callable<Triple<String, Integer, Long>> {
private final AtomicInteger requestNumber;
public GoogleDriveCallable(AtomicInteger requestNumber) {
this.requestNumber = requestNumber;
}
@Override
public Triple<String, Integer, Long> call() throws Exception {
try {
try {
StopWatch timeIt = StopWatch.createStarted();
googleDrive
.files()
.list()
.setSpaces("drive")
.setQuotaUser(UUID.randomUUID().toString())
.setFields("nextPageToken, files(id, name, mimeType)")
.setPageSize(100)
.execute();
timeIt.stop();
return new ImmutableTriple<>(Thread.currentThread().getName(), requestNumber.getAndIncrement(), timeIt.getTime());
} catch (GoogleJsonResponseException gjre) {
GoogleJsonError.ErrorInfo firstReportedError = gjre.getDetails().getErrors().get(0);
if (USER_LIMIT_QUOTA_EXCEEDED_ERROR_REASON.equals(firstReportedError.getReason())) {
fail("Google user rate limit triggered during request n " + requestNumber);
} else {
throw gjre;
}
}
} catch (Exception e) {
throw new RuntimeException("BOOM during request n " + requestNumber, e);
}
return null;
}
}
}
Running this unit test with different number of threads (minimum 5 minute difference between runs to guarantee no interference) wields the following:
It has been confirmed that no one else is using the domain so nothing should interfering with these tests.
Why would those last two scenarios fail with user quota being triggered if we haven't hit the 100 second timepoint and, even if extrapolating the rates to 100 seconds and they do get close, they are still short of the 1000 request per user per 100 second quota?
Communication with Google support has highlighted that, aside from the known quota, the backend services have burst protection.
As such, if the rate of request is constant, quotas will apply but, in case the request rate suffers bursts and said bursts, if extrapolated as normal traffic, cause the quotas to be overrun, the API will reply with usageLimit errors.