My app engine service is written in Go. I have code that connects to Cloud Datastore before even the server listens on the port. There is a single Projection query that takes about 500ms reading just 4 entities. Does the first interaction with datastore have higher latency potentially as a connection needs to be established? Any way this datastore connection latency be reduced? Also, is there any difference in doing this db call before listening to the port vs doing it within the warmup request (this is an autoscaled instance).
Similar to high initial latency for Cloud Datastore, I see a similar pattern for Cloud Tasks. Initial task creation could be as high as 500ms but even subsequent ones are any where from 200 to 400ms. This is in us-central. I was actually considering moving a db update to a background task but in general I am seeing the latency of task creation to be more or less same as doing a transaction to read and update the data giving no net benefit.
Finally, instance startup time is typically 2.5 to 3 seconds with the main getting called after about 2 seconds. My app startup time is the above mentioned project query cost of 500ms and nothing else. So, no matter how much I optimize my app startup, should I assume an additional latency of about 2 seconds?
Note that the load on the system is very light so these issues can't be because of high volume.
Update: deployment files as requested by Miguel (this is for a test environment investigating performance characteristics. Prod deployment will be more generous for instances)
default app:
service: default
runtime: go112
instance_class: F1
automatic_scaling:
min_instances: 0
max_instances: 1
min_idle_instances: 1
max_idle_instances: 1
min_pending_latency: 200ms
max_pending_latency: 500ms
max_concurrent_requests: 10
target_cpu_utilization: 0.9
target_throughput_utilization: 0.9
inbound_services:
- warmup
backend app:
service: backend-services
runtime: go112
instance_class: B1
basic_scaling:
idle_timeout: 1m
max_instances: 1
200-500ms to initialize a client seems reasonable because there is a remote connection being established. Also, a 1-2 seconds cold start for App Engine also seems normal.
As you mentioned, you can experiment with a warmup request to reduce cold starts and initialize clients.
I would also recommend looking into the mode you are running your Datastore in (native vs datastore). There is increase latency when using datastore mode, for more info see Cloud Datastore Best Practices.