nginxhaproxyhttp2proxy-protocol
Using http2_push with nginX in conjunction with HAProxy does not work
I can not get HTTP/2 push to work when nginX is configured behind HaProxy. It does work when nginX is hit directly by the web browser however.
Did a lot of research already but did not find any hints. Hope anybody knows what I am doing wrong. See configuration and further observations below.
Configuration
The relevant HaProxy (version 1.8.7) configuration is given by:
frontend appname
bind *:443 ssl crt certificate.pem alpn h2,http/1.1
mode tcp
use_backend app-http2 if { ssl_fc_alpn -i h2 }
default_backend app
backend app-http2
mode tcp
server lamp2 127.0.0.1:8002 check send-proxy
And the relevant nginX (version 1.14.0) configuration is as follows:
http {
# This is the one I would like to use
server {
listen 8002 http2 proxy_protocol;
server_name _;
root /usr/share/nginx/html;
location / {
http2_push /image.jpg;
}
}
# This one can be accessed directly; and *does* work
server {
listen 8004 http2 ssl;
ssl_certificate certificate.pem;
ssl_certificate_key private.key;
server_name _;
root /usr/share/nginx/html;
location / {
http2_push /image.jpg;
}
}
}
Observations
- In the nginx logs I can verify that both ways of accessing the content do use HTTP2.
- When I use Chrome to access the page I can see that push is used only when directly visiting nginX
Update 9 may 2018 Still not solved. But people seem to agree that it is a bug. I opened an issue at their issue tracker: https://trac.nginx.org/nginx/ticket/1549#ticket
Update 26 april 2018
It appears the problem is bigger than just the http2 push. If I log the $scheme nginX variable it is always set to http. Both when accessing from http as from http2.
So that obviously seems like the problem. However I am not sure how I can fix this. Haproxy is working tcp mode; thus will likely not be doing anything wrong.
A related (but possibly outdated) Stack Overflow topic is nginx $scheme variable behind load balancer. But that answer does not help solving this problem!
Update 25 april 2018
Still not working. But a step closer. Ran nghttp2 on both, and results are found below.
Both seem to have the /image.jpg resource embedded. But the one that goes via haproxy has it's scheme set to http; and not to https. As one can see in this diff:
I assume due to this; Chrome will not use this pushed resource. I am however not sure what causes this!
Does anyone have a clue?
The complete output of both commands:
nghttp -nv https://127.0.0.1:8004/
[ 0.001] Connected
The negotiated protocol: h2
[ 0.003] send SETTINGS frame
(niv=2)
[SETTINGS_MAX_CONCURRENT_STREAMS(0x03):100]
[SETTINGS_INITIAL_WINDOW_SIZE(0x04):65535]
[ 0.003] send PRIORITY frame
(dep_stream_id=0, weight=201, exclusive=0)
[ 0.003] send PRIORITY frame
(dep_stream_id=0, weight=101, exclusive=0)
[ 0.003] send PRIORITY frame
(dep_stream_id=0, weight=1, exclusive=0)
[ 0.003] send PRIORITY frame
(dep_stream_id=7, weight=1, exclusive=0)
[ 0.003] send PRIORITY frame
(dep_stream_id=3, weight=1, exclusive=0)
[ 0.003] send HEADERS frame
; END_STREAM | END_HEADERS | PRIORITY
(padlen=0, dep_stream_id=11, weight=16, exclusive=0)
; Open new stream
:method: GET
:path: /
:scheme: https
:authority: 127.0.0.1:8004
accept: */*
accept-encoding: gzip, deflate
user-agent: nghttp2/1.25.0
[ 0.003] recv SETTINGS frame
(niv=3)
[SETTINGS_MAX_CONCURRENT_STREAMS(0x03):128]
[SETTINGS_INITIAL_WINDOW_SIZE(0x04):65536]
[SETTINGS_MAX_FRAME_SIZE(0x05):16777215]
[ 0.003] recv WINDOW_UPDATE frame
(window_size_increment=2147418112)
[ 0.003] send SETTINGS frame
; ACK
(niv=0)
[ 0.003] recv SETTINGS frame
; ACK
(niv=0)
[ 0.003] recv (stream_id=13) :method: GET
[ 0.003] recv (stream_id=13) :path: /image.jpg
[ 0.003] recv (stream_id=13) :scheme: https
[ 0.003] recv (stream_id=13) :authority: 127.0.0.1:8004
[ 0.003] recv (stream_id=13) accept-encoding: gzip, deflate
[ 0.003] recv (stream_id=13) user-agent: nghttp2/1.25.0
[ 0.003] recv PUSH_PROMISE frame
; END_HEADERS
(padlen=0, promised_stream_id=2)
[ 0.003] recv (stream_id=13) :status: 200
[ 0.003] recv (stream_id=13) server: nginx/1.14.0
[ 0.003] recv (stream_id=13) date: Wed, 25 Apr 2018 15:08:26 GMT
[ 0.003] recv (stream_id=13) content-type: text/html
[ 0.003] recv (stream_id=13) content-length: 638
[ 0.003] recv (stream_id=13) last-modified: Wed, 25 Apr 2018 11:42:58 GMT
[ 0.003] recv (stream_id=13) etag: "5ae069c2-27e"
[ 0.003] recv (stream_id=13) accept-ranges: bytes
[ 0.003] recv HEADERS frame
; END_HEADERS
(padlen=0)
; First response header
[ 0.004] recv DATA frame
; END_STREAM
[ 0.004] recv (stream_id=2) :status: 200
[ 0.004] recv (stream_id=2) server: nginx/1.14.0
[ 0.004] recv (stream_id=2) date: Wed, 25 Apr 2018 15:08:26 GMT
[ 0.004] recv (stream_id=2) content-type: image/jpeg
[ 0.004] recv (stream_id=2) content-length: 182884
[ 0.004] recv (stream_id=2) last-modified: Sat, 18 Jun 2016 15:42:26 GMT
[ 0.004] recv (stream_id=2) etag: "57656be2-2ca64"
[ 0.004] recv (stream_id=2) accept-ranges: bytes
[ 0.004] recv HEADERS frame
; END_HEADERS
(padlen=0)
; First push response header
[ 0.004] recv DATA frame
[ 0.004] recv DATA frame
[ 0.004] recv DATA frame
[ 0.004] recv DATA frame
[ 0.004] recv DATA frame
[ 0.004] send WINDOW_UPDATE frame
(window_size_increment=33248)
[ 0.004] send WINDOW_UPDATE frame
(window_size_increment=32768)
[ 0.004] recv DATA frame
[ 0.004] recv DATA frame
[ 0.004] recv DATA frame
[ 0.046] recv DATA frame
[ 0.046] recv DATA frame
[ 0.046] recv DATA frame
[ 0.046] recv DATA frame
[ 0.046] recv DATA frame
[ 0.046] send WINDOW_UPDATE frame
(window_size_increment=32925)
[ 0.046] send WINDOW_UPDATE frame
(window_size_increment=32767)
[ 0.046] recv DATA frame
[ 0.046] send WINDOW_UPDATE frame
(window_size_increment=32768)
[ 0.046] send WINDOW_UPDATE frame
(window_size_increment=32768)
[ 0.046] recv DATA frame
[ 0.090] recv DATA frame
[ 0.090] recv DATA frame
[ 0.090] recv DATA frame
[ 0.090] recv DATA frame
[ 0.090] recv DATA frame
[ 0.090] recv DATA frame
[ 0.090] recv DATA frame
[ 0.090] send WINDOW_UPDATE frame
(window_size_increment=32767)
[ 0.090] send WINDOW_UPDATE frame
(window_size_increment=32767)
[ 0.090] send WINDOW_UPDATE frame
(window_size_increment=32768)
[ 0.090] send WINDOW_UPDATE frame
(window_size_increment=32768)
[ 0.134] recv DATA frame
[ 0.134] recv DATA frame
[ 0.134] recv DATA frame
; END_STREAM
[ 0.134] send GOAWAY frame
(last_stream_id=2, error_code=NO_ERROR(0x00), opaque_data(0)=[])
and
nghttp -nv https://127.0.0.1:8002/
[ 0.001] Connected
The negotiated protocol: h2
[ 0.003] send SETTINGS frame
(niv=2)
[SETTINGS_MAX_CONCURRENT_STREAMS(0x03):100]
[SETTINGS_INITIAL_WINDOW_SIZE(0x04):65535]
[ 0.003] send PRIORITY frame
(dep_stream_id=0, weight=201, exclusive=0)
[ 0.003] send PRIORITY frame
(dep_stream_id=0, weight=101, exclusive=0)
[ 0.003] send PRIORITY frame
(dep_stream_id=0, weight=1, exclusive=0)
[ 0.003] send PRIORITY frame
(dep_stream_id=7, weight=1, exclusive=0)
[ 0.003] send PRIORITY frame
(dep_stream_id=3, weight=1, exclusive=0)
[ 0.003] send HEADERS frame
; END_STREAM | END_HEADERS | PRIORITY
(padlen=0, dep_stream_id=11, weight=16, exclusive=0)
; Open new stream
:method: GET
:path: /
:scheme: https
:authority: 127.0.0.1:8002
accept: */*
accept-encoding: gzip, deflate
user-agent: nghttp2/1.25.0
[ 0.003] recv SETTINGS frame
(niv=3)
[SETTINGS_MAX_CONCURRENT_STREAMS(0x03):128]
[SETTINGS_INITIAL_WINDOW_SIZE(0x04):65536]
[SETTINGS_MAX_FRAME_SIZE(0x05):16777215]
[ 0.003] recv WINDOW_UPDATE frame
(window_size_increment=2147418112)
[ 0.003] send SETTINGS frame
; ACK
(niv=0)
[ 0.004] recv SETTINGS frame
; ACK
(niv=0)
[ 0.004] recv (stream_id=13) :method: GET
[ 0.004] recv (stream_id=13) :path: /image.jpg
[ 0.004] recv (stream_id=13) :scheme: http
[ 0.004] recv (stream_id=13) :authority: 127.0.0.1:8002
[ 0.004] recv (stream_id=13) accept-encoding: gzip, deflate
[ 0.004] recv (stream_id=13) user-agent: nghttp2/1.25.0
[ 0.004] recv PUSH_PROMISE frame
; END_HEADERS
(padlen=0, promised_stream_id=2)
[ 0.004] recv (stream_id=13) :status: 200
[ 0.004] recv (stream_id=13) server: nginx/1.14.0
[ 0.004] recv (stream_id=13) date: Wed, 25 Apr 2018 15:08:45 GMT
[ 0.004] recv (stream_id=13) content-type: text/html
[ 0.004] recv (stream_id=13) content-length: 638
[ 0.004] recv (stream_id=13) last-modified: Wed, 25 Apr 2018 11:42:58 GMT
[ 0.004] recv (stream_id=13) etag: "5ae069c2-27e"
[ 0.004] recv (stream_id=13) accept-ranges: bytes
[ 0.004] recv HEADERS frame
; END_HEADERS
(padlen=0)
; First response header
[ 0.004] recv DATA frame
; END_STREAM
[ 0.004] recv (stream_id=2) :status: 200
[ 0.004] recv (stream_id=2) server: nginx/1.14.0
[ 0.004] recv (stream_id=2) date: Wed, 25 Apr 2018 15:08:45 GMT
[ 0.004] recv (stream_id=2) content-type: image/jpeg
[ 0.004] recv (stream_id=2) content-length: 182884
[ 0.004] recv (stream_id=2) last-modified: Sat, 18 Jun 2016 15:42:26 GMT
[ 0.004] recv (stream_id=2) etag: "57656be2-2ca64"
[ 0.004] recv (stream_id=2) accept-ranges: bytes
[ 0.004] recv HEADERS frame
; END_HEADERS
(padlen=0)
; First push response header
[ 0.004] recv DATA frame
[ 0.004] recv DATA frame
[ 0.004] recv DATA frame
[ 0.004] recv DATA frame
[ 0.004] recv DATA frame
[ 0.004] send WINDOW_UPDATE frame
(window_size_increment=33406)
[ 0.004] send WINDOW_UPDATE frame
(window_size_increment=32768)
[ 0.004] recv DATA frame
[ 0.004] recv DATA frame
[ 0.004] recv DATA frame
[ 0.044] recv DATA frame
[ 0.044] send WINDOW_UPDATE frame
(window_size_increment=32767)
[ 0.044] send WINDOW_UPDATE frame
(window_size_increment=32767)
[ 0.044] recv DATA frame
[ 0.044] recv DATA frame
[ 0.045] recv DATA frame
[ 0.045] recv DATA frame
[ 0.045] send WINDOW_UPDATE frame
(window_size_increment=32768)
[ 0.045] send WINDOW_UPDATE frame
(window_size_increment=32768)
[ 0.045] recv DATA frame
[ 0.045] recv DATA frame
[ 0.045] recv DATA frame
[ 0.045] recv DATA frame
[ 0.045] send WINDOW_UPDATE frame
(window_size_increment=32767)
[ 0.045] send WINDOW_UPDATE frame
(window_size_increment=32767)
[ 0.045] recv DATA frame
[ 0.046] recv DATA frame
[ 0.046] recv DATA frame
[ 0.046] recv DATA frame
[ 0.046] send WINDOW_UPDATE frame
(window_size_increment=32768)
[ 0.046] send WINDOW_UPDATE frame
(window_size_increment=32768)
[ 0.046] recv DATA frame
[ 0.046] recv DATA frame
[ 0.046] recv DATA frame
; END_STREAM
[ 0.046] send GOAWAY frame
(last_stream_id=2, error_code=NO_ERROR(0x00), opaque_data(0)=[])
Solution
I ended up opening a ticket on the NginX tracker. And it got fixed. The fix is available in the recently released 1.15.1 version.
Bugfix: HTTP/2 server push did not work if SSL was terminated by a proxy server in front of nginx.
Thanks for the help!
- I want to install the "n" package and I get an error
- n <version> command does not activate specified version
- Change n install location
- How to install a specific version of Node on Ubuntu/Debian?
- Different node version for different projects, is there a way of telling node which version to use?
- Install Node.js to install n to install Node.js?
- How to select the latest node.js v6 version using n?
- n-install: ERROR: GNU Make not found, which is required for operation
- How to downgrade Node version with n
- how switch to previous version in n (Node version manager)?
- Automatically use the right version of Node for a package
- internal/modules/cjs/loader.js:905 -> throw err;
- Why doesn't "n" downgrade my node version on a Mac?
- Node version manager
- n failed to install/switch node in Linux?
- vue command not found on Mac
- How to uninstall n and all node versions installed by n
- Angular CLI on HTTPS - can't install CI as root
- n (node version manager): cannot create directory
- npm module n emits errors
- How to update npm permanently?
- Cannot change nodejs version using n
- upgrade nodejs to stable version
- How should I install and use multiple versions of Node on the same production machine?