I am using Haproxy 1.5.14 in a VirtualBox VM (Boot2docker) where assets it loads intermittently show up 503 without any real rhyme or reason, this is especially true on startup of the cluster.
The cluster looks like this, 1 front end with port 80 and 443 to 2 backends servicing static resources and websocket stuff respectively.
Haproxy
For example the static asset served by the front end might be
https://local.dev.myproject.com/assets/images/back.png
Despite the front end server being up, and nothing has changed, hitting refresh and looking at chrome debugger I will see numerous status 503 or OK 200 304, but it's not determinstic. It can go from 503 to OK to back to 503, on any asset. When connected to the webserver directly the assets return fine so it seems something with haproxy.
The best I can figure is that the health check isn't working properly and the FE/BE server are being removed temporarily from haproxy's internal roster, but that doesn't makes sense it's checking every half second and I can see the calls haproxy is sending being returned by the FE/BE terminal output window ok, every half second as expected.
If I look at the haproxy statistics report, I can see servers periodically come and go, flickering, despite in the terminal window haproxy is still calling the health checks without gaps and the servers are returning them as expected.
Attached is the current haproxy configuration I'm using, any help is appreciated.
#---------------------------------------------------------------------
# Example configuration for a possible web application. See the
# full configuration options online.
#
# http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
#
#---------------------------------------------------------------------
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
#
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
#log 127.0.0.1 local2
# log /lnl_zoom_shared/log local0
# log /lnl_zoom_shared/log local1 notice
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
# SSL
#ca-base /etc/ssl
#crt-base /etc/ssl
ca-base /myproject_shared/SECURITY/local.dev.myproject.com/
crt-base /myproject_shared/SECURITY/local.dev.myproject.com/
tune.ssl.default-dh-param 1024
# turn on stats unix socket
#stats socket /var/lib/haproxy/stats
# Exposes the stat socket so we can manage the proxy through node.js
stats socket /tmp/haproxy.sock level admin
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option http-server-close
option http-pretend-keepalive
option dontlognull
option redispatch
option contstats
option forwardfor except 127.0.0.0/8
retries 3
backlog 10000
timeout client 25s
timeout connect 10s
timeout server 25s
#long timeoutfor websocket connections
timeout tunnel 3600s
timeout http-keep-alive 1s
timeout http-request 15s
timeout queue 30s
timeout tarpit 60s
default-server inter 3s rise 2 fall 3
#timeout check 10s
maxconn 256
#---------------------------------------------------------------------
# Haproxy's internal stats on the servers below: password protected
#---------------------------------------------------------------------
stats enable
stats auth admin:myadminpassword
stats uri /haproxy
stats refresh 5s
#---------------------------------------------------------------------
#
#---------------------------------------------------------------------
frontend public
# HTTP
bind *:80
# Redirect all HTTP traffic to HTTPS
redirect scheme https if !{ ssl_fc }
# HTTPS
# Example with CA certificate bundle
# bind :443 ssl crt cert.pem ca-file bundle.crt
# Example without CA certification bunch
bind *:443 ssl crt /myproject_shared/SECURITY/local.dev.myproject.com/local.dev.myproject.com.pem
acl url_static_BE path_beg -i /BE /primus
use_backend BE if url_static_BE
# FRONT END (aka FE) STATIC ASSETS SERVER
# if path is a static asset, assume the front end server to handle it
acl url_static path_beg -i /static /images /javascript /stylesheets
acl url_static path_end -i .jpg .gif .png .css .js .html .ico
use_backend FE if url_static
# GIT HOOKS for UPDATE on the git repo changes
acl url_githook path_beg -i /gitupdate
use_backend HACNTL if url_githook
#BACK END (aka BE)
default_backend BE
#---------------------------------------------------------------------
# controller for haproxy
#---------------------------------------------------------------------
backend HACNTL
# Tell the backend that this is a secure connection,
# even though it's getting plain HTTP.
option forwardfor
http-request add-header X-Forwarded-Proto https if { ssl_fc }
server SELF 127.0.0.1:3300
#---------------------------------------------------------------------
# static backend for serving up images, stylesheets and such
#---------------------------------------------------------------------
backend FE
# Tell the backend that this is a secure connection,
# even though it's getting plain HTTP.
option forwardfor
http-request add-header X-Forwarded-Proto https if { ssl_fc }
option httpchk GET /haproxy/getstatus
option httpchk HEAD /
balance roundrobin
#server FE1 11.22.33.44:8000 maxconn 256
server FE_172.17.0.2 172.17.0.2:8000 maxconn 256 check inter 500ms
#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend BE
# Tell the backend that this is a secure connection,
# even though it's getting plain HTTP.
option forwardfor
http-request add-header X-Forwarded-Proto https if { ssl_fc }
#http-request set-header X-Custom-Header %[url]
#http-request set-header Connection upgrade
#http-request set-header Upgrade websocket
option httpchk GET /haproxy/getstatus
cookie SRVNAME insert nocache
balance roundrobin
server BE_172.17.0.3 172.17.0.3:8888 maxconn 256 cookie BE_172.17.0.3 check inter 500ms
while not a absolute fix, allowing each server to start one at a time has fixed the issue for now. basically adding a sleep between docker run command