Could not connect to consul agent, error was HTTPConnectionPool

Question

I am using the Consul Ansible role playbook to install configure a new server to use consul. When I run my playbook it gives the following error (I have manullay turned no_log to true so I get more output):

TASK [idealista.consul-role : Consul | Create ACL] ***************************************************************************************
task path: /Users/albertski/.ansible/roles/idealista.consul-role/tasks/consul_acl.yml:41
The full traceback is:
WARNING: The below traceback may *not* be related to the actual failure.
  File "/tmp/ansible_consul_acl_payload_m6FF6v/__main__.py", line 646, in main
    output = set_acl(consul_client, configuration)
  File "/tmp/ansible_consul_acl_payload_m6FF6v/__main__.py", line 241, in set_acl
    acls_as_json = decode_acls_as_json(consul_client.acl.list())
  File "/usr/local/lib/python2.7/dist-packages/consul/base.py", line 1931, in list
    CB.json(), '/v1/acl/list', params=params)
  File "/usr/local/lib/python2.7/dist-packages/consul/std.py", line 22, in get
    self.session.get(uri, verify=self.verify, cert=self.cert)))
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 543, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)

failed: [php7d] (item={'key': 'Agent', 'value': {'token_type': 'client', 'token': 'xxxxxxxxxxxx', 'rules': [{'node': '', 'policy': 'write'}, {'service': '', 'policy': 'read'}]}}) => {
    "ansible_loop_var": "item",
    "changed": false,
    "invocation": {
        "module_args": {
            "host": "xxx.xx.xx.xx",
            "mgmt_token": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "name": "Agent",
            "port": 8500,
            "rules": [
                {
                    "node": "",
                    "policy": "write"
                },
                {
                    "policy": "read",
                    "service": ""
                }
            ],
            "scheme": "http",
            "state": "present",
            "token": "xxxxxxxxxxxx",
            "token_type": "client",
            "validate_certs": true
        }
    },
    "item": {
        "key": "Agent",
        "value": {
            "rules": [
                {
                    "node": "",
                    "policy": "write"
                },
                {
                    "policy": "read",
                    "service": ""
                }
            ],
            "token": "xxxxxxxxxxxx",
            "token_type": "client"
        }
    },
    "msg": "Could not connect to consul agent at xxx.xx.x.xx:xxxx, error was HTTPConnectionPool(host='xxx.xx.x.x', port=8500): Max retries exceeded with url: /v1/acl/list?token=******** (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb1f1a7e6d0>: Failed to establish a new connection: [Errno 110] Connection timed out',))"

I am using ansible 2.8.4.

Answer 1

I started to compare my new droplet vs. my old droplet (I have this up on Digital Ocean), and I noticed that there were some extra tags on my old droplet: web php74. Once I added those tags in, the issues went away. I’m guessing the ansible script to setup the servers somehow needs those tags.