nginxhaproxyhttp2proxy-protocol
Using http2_push with nginX in conjunction with HAProxy does not work
I can not get HTTP/2 push to work when nginX is configured behind HaProxy. It does work when nginX is hit directly by the web browser however.
Did a lot of research already but did not find any hints. Hope anybody knows what I am doing wrong. See configuration and further observations below.
Configuration
The relevant HaProxy (version 1.8.7) configuration is given by:
frontend appname
bind *:443 ssl crt certificate.pem alpn h2,http/1.1
mode tcp
use_backend app-http2 if { ssl_fc_alpn -i h2 }
default_backend app
backend app-http2
mode tcp
server lamp2 127.0.0.1:8002 check send-proxy
And the relevant nginX (version 1.14.0) configuration is as follows:
http {
# This is the one I would like to use
server {
listen 8002 http2 proxy_protocol;
server_name _;
root /usr/share/nginx/html;
location / {
http2_push /image.jpg;
}
}
# This one can be accessed directly; and *does* work
server {
listen 8004 http2 ssl;
ssl_certificate certificate.pem;
ssl_certificate_key private.key;
server_name _;
root /usr/share/nginx/html;
location / {
http2_push /image.jpg;
}
}
}
Observations
- In the nginx logs I can verify that both ways of accessing the content do use HTTP2.
- When I use Chrome to access the page I can see that push is used only when directly visiting nginX
Update 9 may 2018 Still not solved. But people seem to agree that it is a bug. I opened an issue at their issue tracker: https://trac.nginx.org/nginx/ticket/1549#ticket
Update 26 april 2018
It appears the problem is bigger than just the http2 push. If I log the $scheme nginX variable it is always set to http. Both when accessing from http as from http2.
So that obviously seems like the problem. However I am not sure how I can fix this. Haproxy is working tcp mode; thus will likely not be doing anything wrong.
A related (but possibly outdated) Stack Overflow topic is nginx $scheme variable behind load balancer. But that answer does not help solving this problem!
Update 25 april 2018
Still not working. But a step closer. Ran nghttp2 on both, and results are found below.
Both seem to have the /image.jpg resource embedded. But the one that goes via haproxy has it's scheme set to http; and not to https. As one can see in this diff:
I assume due to this; Chrome will not use this pushed resource. I am however not sure what causes this!
Does anyone have a clue?
The complete output of both commands:
nghttp -nv https://127.0.0.1:8004/
[ 0.001] Connected
The negotiated protocol: h2
[ 0.003] send SETTINGS frame
(niv=2)
[SETTINGS_MAX_CONCURRENT_STREAMS(0x03):100]
[SETTINGS_INITIAL_WINDOW_SIZE(0x04):65535]
[ 0.003] send PRIORITY frame
(dep_stream_id=0, weight=201, exclusive=0)
[ 0.003] send PRIORITY frame
(dep_stream_id=0, weight=101, exclusive=0)
[ 0.003] send PRIORITY frame
(dep_stream_id=0, weight=1, exclusive=0)
[ 0.003] send PRIORITY frame
(dep_stream_id=7, weight=1, exclusive=0)
[ 0.003] send PRIORITY frame
(dep_stream_id=3, weight=1, exclusive=0)
[ 0.003] send HEADERS frame
; END_STREAM | END_HEADERS | PRIORITY
(padlen=0, dep_stream_id=11, weight=16, exclusive=0)
; Open new stream
:method: GET
:path: /
:scheme: https
:authority: 127.0.0.1:8004
accept: */*
accept-encoding: gzip, deflate
user-agent: nghttp2/1.25.0
[ 0.003] recv SETTINGS frame
(niv=3)
[SETTINGS_MAX_CONCURRENT_STREAMS(0x03):128]
[SETTINGS_INITIAL_WINDOW_SIZE(0x04):65536]
[SETTINGS_MAX_FRAME_SIZE(0x05):16777215]
[ 0.003] recv WINDOW_UPDATE frame
(window_size_increment=2147418112)
[ 0.003] send SETTINGS frame
; ACK
(niv=0)
[ 0.003] recv SETTINGS frame
; ACK
(niv=0)
[ 0.003] recv (stream_id=13) :method: GET
[ 0.003] recv (stream_id=13) :path: /image.jpg
[ 0.003] recv (stream_id=13) :scheme: https
[ 0.003] recv (stream_id=13) :authority: 127.0.0.1:8004
[ 0.003] recv (stream_id=13) accept-encoding: gzip, deflate
[ 0.003] recv (stream_id=13) user-agent: nghttp2/1.25.0
[ 0.003] recv PUSH_PROMISE frame
; END_HEADERS
(padlen=0, promised_stream_id=2)
[ 0.003] recv (stream_id=13) :status: 200
[ 0.003] recv (stream_id=13) server: nginx/1.14.0
[ 0.003] recv (stream_id=13) date: Wed, 25 Apr 2018 15:08:26 GMT
[ 0.003] recv (stream_id=13) content-type: text/html
[ 0.003] recv (stream_id=13) content-length: 638
[ 0.003] recv (stream_id=13) last-modified: Wed, 25 Apr 2018 11:42:58 GMT
[ 0.003] recv (stream_id=13) etag: "5ae069c2-27e"
[ 0.003] recv (stream_id=13) accept-ranges: bytes
[ 0.003] recv HEADERS frame
; END_HEADERS
(padlen=0)
; First response header
[ 0.004] recv DATA frame
; END_STREAM
[ 0.004] recv (stream_id=2) :status: 200
[ 0.004] recv (stream_id=2) server: nginx/1.14.0
[ 0.004] recv (stream_id=2) date: Wed, 25 Apr 2018 15:08:26 GMT
[ 0.004] recv (stream_id=2) content-type: image/jpeg
[ 0.004] recv (stream_id=2) content-length: 182884
[ 0.004] recv (stream_id=2) last-modified: Sat, 18 Jun 2016 15:42:26 GMT
[ 0.004] recv (stream_id=2) etag: "57656be2-2ca64"
[ 0.004] recv (stream_id=2) accept-ranges: bytes
[ 0.004] recv HEADERS frame
; END_HEADERS
(padlen=0)
; First push response header
[ 0.004] recv DATA frame
[ 0.004] recv DATA frame
[ 0.004] recv DATA frame
[ 0.004] recv DATA frame
[ 0.004] recv DATA frame
[ 0.004] send WINDOW_UPDATE frame
(window_size_increment=33248)
[ 0.004] send WINDOW_UPDATE frame
(window_size_increment=32768)
[ 0.004] recv DATA frame
[ 0.004] recv DATA frame
[ 0.004] recv DATA frame
[ 0.046] recv DATA frame
[ 0.046] recv DATA frame
[ 0.046] recv DATA frame
[ 0.046] recv DATA frame
[ 0.046] recv DATA frame
[ 0.046] send WINDOW_UPDATE frame
(window_size_increment=32925)
[ 0.046] send WINDOW_UPDATE frame
(window_size_increment=32767)
[ 0.046] recv DATA frame
[ 0.046] send WINDOW_UPDATE frame
(window_size_increment=32768)
[ 0.046] send WINDOW_UPDATE frame
(window_size_increment=32768)
[ 0.046] recv DATA frame
[ 0.090] recv DATA frame
[ 0.090] recv DATA frame
[ 0.090] recv DATA frame
[ 0.090] recv DATA frame
[ 0.090] recv DATA frame
[ 0.090] recv DATA frame
[ 0.090] recv DATA frame
[ 0.090] send WINDOW_UPDATE frame
(window_size_increment=32767)
[ 0.090] send WINDOW_UPDATE frame
(window_size_increment=32767)
[ 0.090] send WINDOW_UPDATE frame
(window_size_increment=32768)
[ 0.090] send WINDOW_UPDATE frame
(window_size_increment=32768)
[ 0.134] recv DATA frame
[ 0.134] recv DATA frame
[ 0.134] recv DATA frame
; END_STREAM
[ 0.134] send GOAWAY frame
(last_stream_id=2, error_code=NO_ERROR(0x00), opaque_data(0)=[])
and
nghttp -nv https://127.0.0.1:8002/
[ 0.001] Connected
The negotiated protocol: h2
[ 0.003] send SETTINGS frame
(niv=2)
[SETTINGS_MAX_CONCURRENT_STREAMS(0x03):100]
[SETTINGS_INITIAL_WINDOW_SIZE(0x04):65535]
[ 0.003] send PRIORITY frame
(dep_stream_id=0, weight=201, exclusive=0)
[ 0.003] send PRIORITY frame
(dep_stream_id=0, weight=101, exclusive=0)
[ 0.003] send PRIORITY frame
(dep_stream_id=0, weight=1, exclusive=0)
[ 0.003] send PRIORITY frame
(dep_stream_id=7, weight=1, exclusive=0)
[ 0.003] send PRIORITY frame
(dep_stream_id=3, weight=1, exclusive=0)
[ 0.003] send HEADERS frame
; END_STREAM | END_HEADERS | PRIORITY
(padlen=0, dep_stream_id=11, weight=16, exclusive=0)
; Open new stream
:method: GET
:path: /
:scheme: https
:authority: 127.0.0.1:8002
accept: */*
accept-encoding: gzip, deflate
user-agent: nghttp2/1.25.0
[ 0.003] recv SETTINGS frame
(niv=3)
[SETTINGS_MAX_CONCURRENT_STREAMS(0x03):128]
[SETTINGS_INITIAL_WINDOW_SIZE(0x04):65536]
[SETTINGS_MAX_FRAME_SIZE(0x05):16777215]
[ 0.003] recv WINDOW_UPDATE frame
(window_size_increment=2147418112)
[ 0.003] send SETTINGS frame
; ACK
(niv=0)
[ 0.004] recv SETTINGS frame
; ACK
(niv=0)
[ 0.004] recv (stream_id=13) :method: GET
[ 0.004] recv (stream_id=13) :path: /image.jpg
[ 0.004] recv (stream_id=13) :scheme: http
[ 0.004] recv (stream_id=13) :authority: 127.0.0.1:8002
[ 0.004] recv (stream_id=13) accept-encoding: gzip, deflate
[ 0.004] recv (stream_id=13) user-agent: nghttp2/1.25.0
[ 0.004] recv PUSH_PROMISE frame
; END_HEADERS
(padlen=0, promised_stream_id=2)
[ 0.004] recv (stream_id=13) :status: 200
[ 0.004] recv (stream_id=13) server: nginx/1.14.0
[ 0.004] recv (stream_id=13) date: Wed, 25 Apr 2018 15:08:45 GMT
[ 0.004] recv (stream_id=13) content-type: text/html
[ 0.004] recv (stream_id=13) content-length: 638
[ 0.004] recv (stream_id=13) last-modified: Wed, 25 Apr 2018 11:42:58 GMT
[ 0.004] recv (stream_id=13) etag: "5ae069c2-27e"
[ 0.004] recv (stream_id=13) accept-ranges: bytes
[ 0.004] recv HEADERS frame
; END_HEADERS
(padlen=0)
; First response header
[ 0.004] recv DATA frame
; END_STREAM
[ 0.004] recv (stream_id=2) :status: 200
[ 0.004] recv (stream_id=2) server: nginx/1.14.0
[ 0.004] recv (stream_id=2) date: Wed, 25 Apr 2018 15:08:45 GMT
[ 0.004] recv (stream_id=2) content-type: image/jpeg
[ 0.004] recv (stream_id=2) content-length: 182884
[ 0.004] recv (stream_id=2) last-modified: Sat, 18 Jun 2016 15:42:26 GMT
[ 0.004] recv (stream_id=2) etag: "57656be2-2ca64"
[ 0.004] recv (stream_id=2) accept-ranges: bytes
[ 0.004] recv HEADERS frame
; END_HEADERS
(padlen=0)
; First push response header
[ 0.004] recv DATA frame
[ 0.004] recv DATA frame
[ 0.004] recv DATA frame
[ 0.004] recv DATA frame
[ 0.004] recv DATA frame
[ 0.004] send WINDOW_UPDATE frame
(window_size_increment=33406)
[ 0.004] send WINDOW_UPDATE frame
(window_size_increment=32768)
[ 0.004] recv DATA frame
[ 0.004] recv DATA frame
[ 0.004] recv DATA frame
[ 0.044] recv DATA frame
[ 0.044] send WINDOW_UPDATE frame
(window_size_increment=32767)
[ 0.044] send WINDOW_UPDATE frame
(window_size_increment=32767)
[ 0.044] recv DATA frame
[ 0.044] recv DATA frame
[ 0.045] recv DATA frame
[ 0.045] recv DATA frame
[ 0.045] send WINDOW_UPDATE frame
(window_size_increment=32768)
[ 0.045] send WINDOW_UPDATE frame
(window_size_increment=32768)
[ 0.045] recv DATA frame
[ 0.045] recv DATA frame
[ 0.045] recv DATA frame
[ 0.045] recv DATA frame
[ 0.045] send WINDOW_UPDATE frame
(window_size_increment=32767)
[ 0.045] send WINDOW_UPDATE frame
(window_size_increment=32767)
[ 0.045] recv DATA frame
[ 0.046] recv DATA frame
[ 0.046] recv DATA frame
[ 0.046] recv DATA frame
[ 0.046] send WINDOW_UPDATE frame
(window_size_increment=32768)
[ 0.046] send WINDOW_UPDATE frame
(window_size_increment=32768)
[ 0.046] recv DATA frame
[ 0.046] recv DATA frame
[ 0.046] recv DATA frame
; END_STREAM
[ 0.046] send GOAWAY frame
(last_stream_id=2, error_code=NO_ERROR(0x00), opaque_data(0)=[])
Solution
I ended up opening a ticket on the NginX tracker. And it got fixed. The fix is available in the recently released 1.15.1 version.
Bugfix: HTTP/2 server push did not work if SSL was terminated by a proxy server in front of nginx.
Thanks for the help!
- How to enable .htaccess support with PHP-FPM, Nginx, and Apache2 on Ubuntu 24.04 with FastPanel?
- How to simulate DDOS/Slashdotting?
- How can I allow access to a single IP address via Nginx.conf?
- Is there a way to generate a helm install script from a ConfigMap?
- FastAPI with nginx does not serve static files in HTTPS
- NGINX reverse proxy returns HTTP 400 with a invalid header line for my flutter app.Works fine in Postman
- File copied in dockerfile: Not found if in specific location
- CentOS 7 - NGINX - DNS Load Balance
- What is the right way to declare memory resource in AWS ECS task definition for Docker Nginx?
- nginx "server_tokens off" does not remove the server header
- How to correctly run streamlit app with nginx on https?
- nginx in docker 502
- How to parse several log nginx files using goaccess
- Conditional set up of a proxy server on Server A that allows Server B to perform Docker operations through a http proxy
- HTTP to HTTPS Nginx too many redirects
- Permission Denied on Socket File - NGINX + Gunicorn + Flask
- Are location specific logfiles treated different?
- nginx the "ssl" directive is deprecated, use the "listen ... ssl"
- Run Nginx as Windows service
- Unicode special character U+FFFD returning from my Nest.JS API, but only sometimes
- SSL: error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch
- fatal: repository not found, 404
- Undefined variable $form on remote server
- nginx how to check the value of optional header
- Pm2 process stops running automatically in ubuntu server
- NGINX Ingress Controller Proxying Websocket Connections
- Why is the initial connection time for a HTTP request so long?
- Nginx Error 413 Entity too large on AWS ELB
- Connection Refused Between SSR Next.js and Laravel API
- Kubernetes nginx Ingress rewrite-target results in double slash