I am using Objectify(v5) for accessing Datastore, in App Engine Standard Environment, with Java. Entities are cached by Objectify automatically, and I am also using Memcache separately.
This issue of Datastore APIs timing out started happening exactly post REDIRECT_STRONGLY_CONSISTENT_READS migration step. Strangely, it happens roughly after every hour or 2 hours, lasts for 3-4 mins and then gets back to normal. Since requests latency is going up from ~200ms to more than 60s, lots of new instances are getting created and am getting charged heavily.
here are some of the errors:
com.google.api.server.spi.SystemService invokeServiceMethod: exception occurred while calling backend method
java.util.concurrent.CancellationException: Task was cancelled.
at com.google.common.util.concurrent.AbstractFuture.cancellationExceptionWithCause(AbstractFuture.java:1550)
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:590)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:467)
at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:122)
at com.google.appengine.tools.development.TimedFuture.get(TimedFuture.java:55)
at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:68)
at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:89)
at com.google.appengine.api.datastore.Batcher$ReorderingMultiFuture.get(Batcher.java:114)
at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:89)
at com.googlecode.objectify.cache.TriggerFuture.get(TriggerFuture.java:100)
at com.googlecode.objectify.impl.ResultAdapter.now(ResultAdapter.java:34)
at com.googlecode.objectify.util.ResultWrapper.translate(ResultWrapper.java:22)
at com.googlecode.objectify.util.ResultWrapper.translate(ResultWrapper.java:10)
at com.googlecode.objectify.util.ResultTranslator.nowUncached(ResultTranslator.java:21)
at com.googlecode.objectify.util.ResultCache.now(ResultCache.java:30)
at com.googlecode.objectify.util.ResultWrapper.translate(ResultWrapper.java:22)
at com.googlecode.objectify.util.ResultWrapper.translate(ResultWrapper.java:10)
at com.googlecode.objectify.util.ResultTranslator.nowUncached(ResultTranslator.java:21)
at com.googlecode.objectify.util.ResultCache.now(ResultCache.java:30)
and
com.googlecode.objectify.cache.EntityMemcache getAll: Error obtaining cache for [<dummy-entity-name>]
java.util.concurrent.CancellationException: Task was cancelled.
and
java.lang.InterruptedException
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:460)
I have tried upgrading to latest versions:
implementation 'com.google.appengine:appengine-api-1.0-sdk:2.0.5'
implementation 'javax.servlet:javax.servlet-api:3.1.0'
implementation 'com.googlecode.objectify:objectify:5.1.25'
and also flushed memcache. nothing has worked.
Has anyone faced this issue?
It turned out to be an issue with migration itself. Reached out to Google Cloud support(paid), Datastore/Firestore Product team paused the migration, which I could not do it myself by the way, wasn't allowed for my project. They did some fixing and completed the migration. Now Datastore queries as running as expected.