I'm currently playing with consul. For that I have a vagrant test setup with 4 VMs:
1: consul server, 2 through 4: nodes.
Each node is running a consul agent, registrator and some micro-services (all with Docker).
After starting the cluster, all services and nodes are marked as "passing" in consul.
So far so good.
Now when I shut down one of the nodes, consul marks the "Serf Health Status" as failed, but the HTTP Health check is still marked as "passing" although the whole VM is shut down.
According to the consul documentation the health check timeout should be 10 seconds so I assumed the health checks to be marked as failed 10 seconds after shutdown of the VM. Any idea why it doesn't?
Okay, got this. It seems to be consul logic. As soon as the SERF fails, the last state of the service is maintained. Once I use the correct health-url (http://localhost:8500/v1/health/service/my-cool-service-name?passing), consul returns only the two remaining services as expected, unless of the "passing" state when looking directly at the service.