Search code examples
javagoogle-apigoogle-cloud-spanner

Spanner retries doesn't work after migration from 6.38.2 to 6.45

After migrating java-spanner lib from version 6.38.2 to 6.45.0 (gax migrated from 2.24 to 2.32) I have observed that retries on DEADLINE_EXCEEDED exception doesn't work.

I have custom retry configuration:

SpannerStubSettings.Builder settingsBuilder = builder.getSpannerStubSettingsBuilder();
settingsBuilder.readSettings()
            .setRetryableCodes({StatusCode.Code.DEADLINE_EXCEEDED, StatusCode.Code.UNAVAILABLE})
            .setRetrySettings(
                RetrySettings.newBuilder()
                    .setInitialRetryDelay(Duration.ofMillis(10))
                    .setMaxRetryDelay(Duration.ofMillis(200))
                    .setRetryDelayMultiplier(1.5)
                    .setInitialRpcTimeout(Duration.ofMillis(100))
                    .setMaxRpcTimeout(Duration.ofMillis(1000))
                    .setRpcTimeoutMultiplier(1.5)
                    .setTotalTimeout(Duration.ofMillis(2000))
                    .build());

Having that and version 6.45.0 of the java-spanner lib many requests are ending up with exception DEADLINE_EXCEEDED after 0.0999s, so after initial RPC timeout, no retries after it. Before retries were applied and requests ended up with success.

How can it be explained? I've checked changes in the lib repository (both java-spanner and gax) but couldn't match anything for this specific behavior.

Solution

  • We solved the issue by upgrading spring-cloud-gcp-dependencies to version 3.7.1, it contains:

    com.google.cloud:google-cloud-spanner:jar:6.49.0:compile
com.google.api:gax:jar:2.34.1:compile

    Version 3.7.0 of spring-cloud-gcp-dependencies was still affected by the issue, it contains:

    com.google.cloud:google-cloud-spanner:jar:6.47.0:compile
com.google.api:gax:jar:2.33.0:compile