I have a server that runs HAProxy to load balance our mysql servers. Some of the server may go down when we have low average load for a extensive period of time, but, in the future, if the load becomes high again, those servers goes up automatically. The problem is when an instance goes down, HAProxy never looks to it again, so when the instance is up again, it is ignored. To fix this we reboot when needed.
Here is our configuration file:
global
log 127.0.0.1 local0 notice
user haproxy
group haproxy
defaults
log global
retries 2
timeout connect 3000
timeout server 5000
timeout client 5000
listen mysql-cluster
bind 0.0.0.0:3306
mode tcp
option mysql-check user haproxy_check
balance leastconn
server mysql-1 ********:3306 check
server mysql-2 ********:3306 check
Maybe if I change the retries from 2 to a big number it could solve our problem?
EDIT As requested, here is my HAProxy version:
$ haproxy -v
HA-Proxy version 1.4.24 2013/06/17
Copyright 2000-2013 Willy Tarreau <w@1wt.eu>
Thanks
We had similar situation in our AWS infrastructure: HAProxy on every instance for RDS replicas access (we have 3 replicas, but application able to work only with one hostname). We solve this issue globally by replacing HAProxy with Route53 internal using with multiple records with same name ( for example db.example.internal) and same weight (Weighted Route53 policy). Also we create Route53 health check for every replica (TCP 3306 port check). For us this solution works great and if we need to add/delete RDS replica - the only place where we need to change - Route53 record and health check (we don't need to maintain every instance HAProxy status/conf)