I have an Angular website with static assets of around 1.5 mb and gzipped it is around 400 kb, I have nginx as my webserver & reverse proxy to the API server, when I test nginx with Apache benchmark tool, I find huge drop in performance if I test the https site compared to the http (https is 10 times slower) & the cpu utilization & memory is not high at all (cpu 30% memory is only 1 mb!!)
I have been searching for hours & tried all possible enhancements but none worked, as far as I have read https shall not be that much slower on modern web servers (http around 1500 req/sec & https is 46 req/sec for nginx), this is mostly from the Nginx https very high connect time but I have no clue how to solve this.
Can someone advise how to improve this?
(Also to my surprise, Apache performs much better in both cases but doesn't respond if I set concurrent connections to more than 200) & this is not nginx vs apache I am just stating my situation.
Important note:
I am not comparing the 2 web servers that is not the point of this site, but generally they have comparable performance so if https in nginx is 10 times slower than Apache I feel that something is wrong in my Nginx configuration & I want to fix it.
All test are on my windows machine i7 & 16 gb ram.
Nginx http only:
C:\Apache24\bin>ab -n 5000 -c 200 http://localhost:8100/abc/index.html?param=abc
This is ApacheBench, Version 2.3 <$Revision: 1826891 $>
Server Software: nginx/1.15.4
Server Hostname: localhost
Server Port: 8100
Document Path: /abc/index.html?param=abc
Document Length: 1099 bytes
Concurrency Level: 200
Time taken for tests: 3.246 seconds
Complete requests: 5000
Failed requests: 0
Total transferred: 6665000 bytes
HTML transferred: 5495000 bytes
Requests per second: 1540.32 [#/sec] (mean)
Time per request: 129.843 [ms] (mean)
Time per request: 0.649 [ms] (mean, across all concurrent requests)
Transfer rate: 2005.12 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 1.3 0 16
Processing: 31 87 12.8 94 124
Waiting: 0 87 13.7 94 124
Total: 31 87 12.8 94 124
Percentage of the requests served within a certain time (ms)
50% 94
66% 94
75% 94
80% 94
90% 99
95% 109
98% 109
99% 113
100% 124 (longest request)
Nginx https (with http2 enabled)
C:\Apache24\bin>abs -n 5000 -c 200 https://localhost:8200/abc/index.html?param=abc
This is ApacheBench, Version 2.3 <$Revision: 1826891 $>
Server Software: nginx/1.15.4
Server Hostname: localhost
Server Port: 8200
SSL/TLS Protocol: TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256
TLS Server Name: localhost
Document Path: /abc/index.html?param=abc
Document Length: 1099 bytes
Concurrency Level: 200
Time taken for tests: 108.985 seconds
Complete requests: 5000
Failed requests: 0
Total transferred: 6780000 bytes
HTML transferred: 5495000 bytes
Requests per second: 45.88 [#/sec] (mean)
Time per request: 4359.386 [ms] (mean)
Time per request: 21.797 [ms] (mean, across all concurrent requests)
Transfer rate: 60.75 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 16 4201 506.8 4251 4755
Processing: 0 32 12.6 31 88
Waiting: 0 32 12.6 31 88
Total: 62 4232 506.9 4283 4800
Percentage of the requests served within a certain time (ms)
50% 4283
66% 4342
75% 4413
80% 4439
90% 4484
95% 4547
98% 4694
99% 4727
100% 4800 (longest request)
Compared to Apache http (here CPU is around 90 to 100% utilized)
C:\Apache24\bin>ab -n 5000 -c 200 http://localhost:6200/abc/index.html?param=abc
Server Software: Apache/2.4.33
Server Hostname: localhost
Server Port: 6200
Document Path: /abc/index.html?param=abc
Document Length: 1099 bytes
Concurrency Level: 200
Time taken for tests: 1.781 seconds
Complete requests: 5000
Failed requests: 0
Total transferred: 6810000 bytes
HTML transferred: 5495000 bytes
Requests per second: 2806.99 [#/sec] (mean)
Time per request: 71.251 [ms] (mean)
Time per request: 0.356 [ms] (mean, across all concurrent requests)
Transfer rate: 3733.51 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 1.6 0 16
Processing: 16 69 16.0 63 125
Waiting: 0 57 16.0 63 125
Total: 16 69 16.0 63 125
Percentage of the requests served within a certain time (ms)
50% 63
66% 78
75% 78
80% 78
90% 94
95% 94
98% 94
99% 109
100% 125 (longest request)
And Apache https is as follows (http 1.1) & note that http 1.1 in nginx didn't improve its performance:
C:\Apache24\bin>abs -n 5000 -c 200 https://localhost:7200/abc/index.html?param=abc
This is ApacheBench, Version 2.3 <$Revision: 1826891 $>
Server Software: Apache/2.4.33
Server Hostname: localhost
Server Port: 7200
SSL/TLS Protocol: TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256
TLS Server Name: localhost
Document Path: /abc/index.html?param=abc
Document Length: 1099 bytes
Concurrency Level: 200
Time taken for tests: 8.747 seconds
Complete requests: 5000
Failed requests: 0
Total transferred: 6810000 bytes
HTML transferred: 5495000 bytes
Requests per second: 571.60 [#/sec] (mean)
Time per request: 349.894 [ms] (mean)
Time per request: 1.749 [ms] (mean, across all concurrent requests)
Transfer rate: 760.27 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 198 42.7 188 391
Processing: 62 145 39.1 140 385
Waiting: 0 76 28.3 78 250
Total: 62 343 63.0 331 615
Percentage of the requests served within a certain time (ms)
50% 331
66% 369
75% 380
80% 389
90% 422
95% 465
98% 500
99% 536
100% 615 (longest request)
My nginx configuration:
worker_processes auto;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
server {
listen 8100;
server_name localhost;
location / {
root html;
index index.html index.htm;
}
#error_page 404 /404.html;
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
server {
listen 8200 ssl http2;
server_name localhost;
ssl_certificate C:/nginx-1.13.12/conf/server.crt;
ssl_certificate_key C:/nginx-1.13.12/conf/server.key;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
ssl_ciphers HIGH:!aNULL:!MD5;
ssl_prefer_server_ciphers on;
gzip on;
gzip_comp_level 1;
gzip_vary on;
gzip_types
text/css
text/javascript
text/xml
text/plain
text/x-component
application/javascript
application/json
application/xml
application/rss+xml
font/truetype
font/opentype
application/vnd.ms-fontobject
image/svg+xml;
gzip_static on;
location /ipo_reits/ {
root html;
index index.html index.htm;
## here we redirect to the homepage in case of nginx 404
try_files $uri $uri/ /ipo_reits/index.html;
# error_page 404 =301 /;
}
location /api/ {
proxy_pass https://localhost:7001/;
}
}
}
I hope that this will help someone else, It seems that is related to nginx on windows issue, I wrongly assumed that the performance of nginx on windows & linux is similar but clearly it is not.
I have tried the benchmark again with nginx on Linux on the same machine & got excellent performance as shown below
ab -n 5000 -c 200 https://localhost:8200/abc/index?param=abc
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Finished 5000 requests
Server Software: nginx/1.10.3
Server Hostname: localhost
Server Port: 8200
SSL/TLS Protocol: TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256
Document Path: /abc/index?param=abc
Document Length: 1099 bytes
Concurrency Level: 200
Time taken for tests: 4.179 seconds
Complete requests: 5000
Failed requests: 0
Total transferred: 6825000 bytes
HTML transferred: 5495000 bytes
Requests per second: 1196.37 [#/sec] (mean)
Time per request: 167.173 [ms] (mean)
Time per request: 0.836 [ms] (mean, across all concurrent requests)
Transfer rate: 1594.77 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 15 141 185.3 106 1322
Processing: 1 22 13.1 20 82
Waiting: 1 14 9.5 13 81
Total: 24 163 185.7 128 1351
Percentage of the requests served within a certain time (ms)
50% 128
66% 142
75% 148
80% 155
90% 208
95% 260
98% 1100
99% 1164
100% 1351 (longest request)
Also for sustained higher load & concurrency, performance was still the same:
ab -n 25000 -c 1000 https://localhost:8200/abc/index?param=abc
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Benchmarking localhost (be patient)
Completed 2500 requests
....
Completed 25000 requests
Finished 25000 requests
Server Software: nginx/1.10.3
Server Hostname: localhost
Server Port: 8200
SSL/TLS Protocol: TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256
Document Path: /abc/index?param=abc
Document Length: 1099 bytes
Concurrency Level: 1000
Time taken for tests: 20.149 seconds
Complete requests: 25000
Failed requests: 0
Total transferred: 34125000 bytes
HTML transferred: 27475000 bytes
Requests per second: 1240.76 [#/sec] (mean)
Time per request: 805.960 [ms] (mean)
Time per request: 0.806 [ms] (mean, across all concurrent requests)
Transfer rate: 1653.94 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 7 687 711.8 492 7694
Processing: 2 89 50.1 81 516
Waiting: 0 57 48.9 41 509
Total: 15 776 723.4 600 7756
Percentage of the requests served within a certain time (ms)
50% 600
66% 812
75% 1095
80% 1186
90% 1397
95% 1631
98% 3183
99% 3442
100% 7756 (longest request)