Search code examples
spring-bootopensslprometheus

Prometheus cannot scrape from spring-boot application over HTTPS

I'm deploying a spring-boot application and prometheus container through docker, and have exposed the spring-boot /actuator/prometheus endpoint successfully. However, when I enable prometheus debug logs, I can see it fails to scrape the metrics:

ts=2022-02-02T03:54:46.210Z
caller=scrape.go:1292
level=debug
component="scrape manager"
scrape_pool=spring-actuator
target=https://127.0.0.1:8443/actuator/prometheus/
msg="Scrape failed"
err="Get \"https://127.0.0.1:8443/actuator/prometheus/\": dial tcp 127.0.0.1:8443: connect: connection refused"

I'm thinking it's something to do with how I've set up my spring-boot HTTPS. I am generating a self-signed certificate during the building of my spring-boot application, using the command:

keytool
  -genkey
  -alias <alias>
  -dname <dname>
  -keyalg RSA
  -keysize 4096
  -storetype PKCS12
  -keystore <path_to_keystore>
  -validity 3650
  -storepass <keystore_pass>

I then export the cert to a .pem file, and extract the .crt and .key:

openssl pkcs12 -in cert.p12 -out cert.pem -nodes -passin pass:<pass>

This is mounted through a shared volume to my prometheus container, which has a --web.config.file containing:

tls_server_config:
  cert_file: /path/to/cert.crt
  key_file: /path/to/cert.key

And for good measure I added insecure_skip_verify: true to the prometheus.yml config:

- job_name: 'spring-actuator'
metrics_path: '/actuator/prometheus/'
scrape_interval: 60s
scheme: https
static_configs:
  - targets: [ '127.0.0.1:8443' ]
tls_config:
  insecure_skip_verify: true
Solution

  • Ok, I think I found my problem. I made two changes:

    First, I moved the contents of the web.config.file into the prometheus.yml file under the 'spring-actuator'. Then I changed the target to use the hostname for my backend container, rather than 127.0.0.1.

    The end result was a single prometheus.yml file:

    - job_name: 'spring-actuator'
metrics_path: '/actuator/prometheus/'
scrape_interval: 60s
scheme: https
static_configs:
  - targets: [ 'backend:8443' ]
tls_config:
  cert_file: /path/to/cert.crt
  key_file: /path/to/cert.key
  insecure_skip_verify: true

    So just some silly mistakes, not caused by the certs from what I can see. :)