We are trying to find out why there is a high latency from Google CDN.
Our site is behind Google'a http_load_balancer with CDN turned on.
For example, by inspecting sampe GET request for a jpg file (43Kb), we can see from http_load_balancer logs that around 30% of such requests have httpRequest.latency > 1 second, and a lot are taking much longer like several or hundreds of seconds....
This is just by looking at 24h log sample (around 6K the same requests).
The httpRequest.cacheLookup and httpRequest.cacheHit for all of those requests are true.
Also jsonpayload_type_loadbalancerlogentry.statusdetails is response_from_cache and jsonpayload_type_loadbalancerlogentry.cacheid values shows correct region.
When doing the same GET request manually in the browser we are getting expected results with TTFB around 15-20ms.
Any idea where to look for a clue?
The httpRequest.latency field measures the entire download duration, and is directly impacted by slow clients - e.g. a mobile device on a congested network or throttled data plan.
You can check this by looking at the frontend_tcp_rtt metric (which is the RTT between the client and Cloud CDN) in Cloud Monitoring, as well as the average, median and 90th percentile total_latencies, where the slow clients will show up as outliers: https://cloud.google.com/load-balancing/docs/https/https-logging-monitoring#monitoring_metrics_fors
You may find that slow clients are from a specific group of client_country values.