Search code examples
dockerhaproxyhaproxy-ingress

HAProxy Server going into maintenance mode at startup

I am relatively new to HAProxy, so if you see anything I'm doing weird, please let me know.

I'm setting up HAProxy in Docker to load balance my micro services (also in docker containers) across 2 hosts. The two hosts are setup with docker swarm and I'm configuring HAProxy to use the local container for the primary server and the other nodes container for a backup server.

I'm setting up the health checks and want to get the passive health check setup correctly. I want it to pull the server out if there are 10 consecutive errors. The problem is I have a basic health check endpoint that gets called externally every 5 seconds that just does some basic checks in the app. So if actual endpoint has an issue and it's called once every second and there is an issue, the counter will reset every 5 seconds because of the call to this health endpoint. (provided I'm understanding how it works)

My solution to this was to create another server pointing to the same container, set the weight to 0 with no checks and pass all the /health endpoint calls to that. Here is my config:

resolvers docker
  nameserver dns 127.0.0.11:53

frontend daapi
    bind :443 ssl crt /run/secrets/ssl_cert
    acl micro path -i -m beg /micro
    use_backend back_micro if micro

backend back_micro
    option httpchk GET /micro/health
    http-check expect status 200
    default-server inter 3s fall 3 rise 2
    use-server health if { path_end /health }
    server s1 micro-01:8560 check observe layer7 error-limit 10 on-error mark-down resolvers docker init-addr none
    server s2 micro-02:8560 check resolvers docker init-addr none backup
    server health micro-01:8560 weight 0 resolvers docker init-addr none

The problem is with this config, the "health" server will only come up in "maintenance mode".

maintenance mode

If I remove the resolvers docker init-addr none from the health server, it starts up fine. But the problem with that is if I start haproxy while that container is down, haproxy fails to start:

[ALERT]    (1) : config : [/usr/local/etc/haproxy/haproxy.cfg:148] : 'server back_micro/health' : no method found to resolve address 'micro-01'.
[ALERT]    (1) : config : Failed to initialize server(s) addr.

My questions are...

  1. Does this even matter? Is the passive check smart enough to increment the error counter based off of endpoint? /micro/health has 0 errors, /micro/app1 has 10 errors, etc?
  2. How can I have the /micro/health endpoint NOT count in the passive error checks and also allow haproxy to start even if one of the servers are down?

I'm running HAProxy version 2.6.3

Solution

  • Change:

    resolvers docker
  nameserver dns 127.0.0.11:53

    To:

    resolvers docker
  parse-resolv-conf

    That will fix your issue.

    Why? I'm not really sure, but this is what my resolv.conf file contains (inside the docker container):

    nameserver 127.0.0.11
options ndots:0

    Could it be that ndots option has something to do here?

    Sorry I don't know the answers to you other questions. Unless it is for an specific reason, I suggest you to use init-addr last,libc,none instead of init-addr none