I have been serving static files from my NodeJS application deployed on Google App Engine, I have now setup a Google Cloud load balancer (GCLB) with CDN enabled and have moved my static files into a cloud bucket used at the backend.
The whole process works fine, however the issue is that the performance has not improved when testing using Apache Benchmark.
Testing 1000 requests, with 100 concurrency level:
Document Length: 748366 bytes
Concurrency Level: 100
Time taken for tests: 37.069 seconds
Complete requests: 1000
Failed requests: 1
(Connect: 0, Receive: 0, Length: 1, Exceptions: 0)
Total transferred: 748686041 bytes
HTML transferred: 747940252 bytes
Requests per second: 26.98 [#/sec] (mean)
Time per request: 3706.873 [ms] (mean)
Time per request: 37.069 [ms] (mean, across all concurrent requests)
Transfer rate: 19723.87 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 446 748.1 261 9033
Processing: 397 2920 2407.1 2252 17769
Waiting: 31 85 109.7 56 1595
Total: 444 3366 2578.2 2665 18986
Percentage of the requests served within a certain time (ms)
50% 2665
66% 3702
75% 4410
80% 4937
90% 6504
95% 8125
98% 10853
99% 13447
100% 18986 (longest request)
Testing 1000 requests, with 10 concurrency level:
Document Length: 748366 bytes
Concurrency Level: 10
Time taken for tests: 35.386 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 749112083 bytes
HTML transferred: 748366000 bytes
Requests per second: 28.26 [#/sec] (mean)
Time per request: 353.863 [ms] (mean)
Time per request: 35.386 [ms] (mean, across all concurrent requests)
Transfer rate: 20673.37 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 20 74 96.3 41 757
Processing: 98 278 159.5 240 2165
Waiting: 30 59 67.0 49 1915
Total: 135 352 196.6 290 2201
Percentage of the requests served within a certain time (ms)
50% 290
66% 334
75% 381
80% 442
90% 577
95% 742
98% 935
99% 1152
100% 2201 (longest request)
I would not expect the concurrency level to dramatically increase the time taken to process requests. Maybe I am somehow misreading these results?
The other (related) issue to note is that CDN cache hits are shown as 0 in the Google LB UI.
It sounds like your files aren't being cached. Cloud CDN caches only responses that satisfy the criteria listed at cloud.google.com/cdn/docs/caching#cacheability. You can use the troubleshooting steps at cloud.google.com/cdn/docs/troubleshooting#responses-not-cached to identify the specific issue. If that doesn't help, I'd be happy to take a look if you share a specific URL.
As you probably know, ab isn't the most realistic way to assess CDN performance, and you'll run into bottlenecks on your local machine long before you hit any bottlenecks in Cloud CDN itself. If you do use ab, make sure to specify the -k option. If you don't, you can run out of ephemeral ports.