Search code examples
pythonelasticsearchelastic-stackurllib3elasticsearch-curator

SSL: CERTIFICATE_VERIFY_FAILED error when curator access elasticsearch


Im trying to setup elasticsearch-curator (version 5.6.0) to delete indices in elasticsearch (version 7.3.1).

Theirs versions should be compatible (https://www.elastic.co/guide/en/elasticsearch/client/curator/current/version-compatibility.html).

Elasticseach is protected by SSL using self-signed certificate, so I need to turn off certificate verification.

This is my curator.yml conf:

client:
  hosts:
    - 127.0.0.1
  port: 9201
  url_prefix:
  use_ssl: True
  certificate: /opt/elastic-stack/curator/security/ca.crt
  client_cert:
  client_key:
  ssl_no_validate: True
  http_auth: curator:************
  timeout: 30
  master_only: False

logging:
  loglevel: INFO
  logfile: /var/log/elastic-stack/curator/curator.log
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

When I run

curator --config /opt/elastic-stack/curator/curator.yml  /opt/elastic-stack/curator/actions.yml

Even though ssl_no_validate is set to True, I am getting:

/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/curator/utils.py:53: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  return yaml.load(read_file(path))
/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py:175: UserWarning: Connecting to 127.0.0.1 using SSL with verify_certs=False is insecure.
  % host
Traceback (most recent call last):
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 376, in _make_request
    self._validate_conn(conn)
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
    conn.connect()
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connection.py", line 394, in connect
    ssl_context=context,
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 383, in ssl_wrap_socket
    return context.wrap_socket(sock)
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib64/python3.6/ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib64/python3.6/ssl.py", line 814, in __init__
    self.do_handshake()
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib64/python3.6/ssl.py", line 1068, in do_handshake
    self._sslobj.do_handshake()
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib64/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py", line 217, in perform_request
    method, url, body, retries=Retry(False), headers=request_headers, **kw
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 376, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 376, in _make_request
    self._validate_conn(conn)
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
    conn.connect()
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connection.py", line 394, in connect
    ssl_context=context,
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 383, in ssl_wrap_socket
    return context.wrap_socket(sock)
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib64/python3.6/ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib64/python3.6/ssl.py", line 814, in __init__
    self.do_handshake()
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib64/python3.6/ssl.py", line 1068, in do_handshake
    self._sslobj.do_handshake()
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib64/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/curator/utils.py", line 899, in get_client
    check_version(client)
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/curator/utils.py", line 685, in check_version
    version_number = get_version(client)
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/curator/utils.py", line 658, in get_version
    version = client.info()['version']['number']
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/elasticsearch/client/utils.py", line 84, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/elasticsearch/client/__init__.py", line 245, in info
    return self.transport.perform_request("GET", "/", params=params)
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/elasticsearch/transport.py", line 353, in perform_request
    timeout=timeout,
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py", line 226, in perform_request
    raise SSLError("N/A", str(e), e)
elasticsearch.exceptions.SSLError: ConnectionError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)) caused by: SSLError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/python36/python/opt/rh/rh-python36/root/usr/bin//curator", line 11, in <module>
    sys.exit(cli())
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/curator/cli.py", line 213, in cli
    run(config, action_file, dry_run)
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/curator/cli.py", line 160, in run
    client = get_client(**client_args)
  File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/curator/utils.py", line 906, in get_client
    'Error: {0}'.format(e)
elasticsearch.exceptions.ElasticsearchException: Unable to create client connection to Elasticsearch.  Error: ConnectionError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)) caused by: SSLError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777))

I got successful response, when I run

curl -k -u curator:******** https://127.0.0.1:9201

Also kibana and logstash communicates with elasticsearch properly.

Does anyone knows, what might be the problem or how to get more info?

Edit 1:

unfortunately I dont have permissions to use yum, when I installed elasticsearch-curator-5.8.1-1.x86_64.rpm to my home dir using

cd {{ python_installation_dest }} && rpm2cpio ../elasticsearch-curator-5.8.1-1.x86_64.rpm| cpio -idmB 

and then run curator, I got:

Fatal Python error: initfsencoding: Unable to get the locale encoding Traceback (most recent call last): File "/opt/python/3.7.4/lib/python3.7/encodings/__init__.py", line 31, in <module> zipimport.ZipImportError: can't decompress data; zlib not available 

I dont use pythin 3.7.4, how can I change path to python?


Solution

  • The error appears to be tied to Python issues within RedHat and associated variants. As the official RPM version of Curator 5.8.x now bundles both its own version of Python 3.7.4 and its own up-to-date OpenSSL shared library (1.1.1c in Curator 5.8.1), you will get better results using the official RPM build.