Search code examples
metricbeatlogz.io

Metricbeat (To logz.io) throws "Failed to connect to backoff: x509: certificate signed by unknown authority"

When running metricbeat against logz.io, metricbeat throws the following error:

ERROR   [publisher_pipeline_output]     pipeline/output.go:154  Failed to connect to backoff(async(tcp://listener-eu.logz.io:5015)): x509: certificate signed by unknown authority

The setup is the following:

  • CentOs 8 behind a corporate firewall
  • SELinux is enabled
  • Metricbeat is installed according to the descriptions under Elastic Docs and logz.io Docs
  • Configuration (Shortened for readability, but the rest is uncommented):
# Standard config, I changed nothing here
metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

# Standard config, I changed nothing here
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression

# "MyToken" is the correct token for metrics in logz.io, of course
fields:
  logzio_codec: json
  token: "MyToken"
fields_under_root: true

# Standard config, I changed nothing here
setup.kibana:
  host: "localhost:5601"

# listener is correct, we use EU access, certificate is in the folder (Download as in the description)
output.logstash:
  # The Logstash hosts
  hosts: ["listener-eu.logz.io:5015"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  ssl.certificate_authorities: ['/etc/pki/tls/certs/COMODORSADomainValidationSecureServerCA.crt']

# Standard config, I changed nothing here
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

What I tried so far / what I have checked:

  • We have one server where it works (I have no idea why or what is different there, the configuration and the firewall settings are the same)
  • Netcat tells that the listener from logz.io is reachable (listener-eu.logz.io:5015)
  • The metric token for logz.io is the correct one
  • The certificate COMODORSADomainValidationSecureServerCA.crt is the one from https://github.com/logzio/public-certificates and located under /etc/pki/tls/certs
  • No special folder access rights are given on the certificate folder and the /etc/metricbeat/metricbeat.yml folder / file (The working reference server also has set chmod 700 on both only)
  • I have tried to set ssl.enabled: true within the configuration section for output.logstash:, but this didn't change anything
  • The metricbeat service itself is running properly and logs to the syslog (Well, it logs error messages), so I would claim that the installation was okay and according to the standard

Something strange I noticed is that sometimes not the X509 error occurrs, but a timeout error (I will post it here once I get it again in the logs).

Is there anything obvious I'm missing with the configuration or something more that I can check on the servers?

Solution

  • The logz.io documentation uses an updated certificate. The updated certificate is available here.

    Note the notice in the docs: "Metrics accounts created after March 2021 use Prometheus instead of ElasticSearch."

    Disclaimer: I work at logz.io