Promethues check external API json response

Question

We are using the kube-prom-stack installed in our cluster and we need now to check an endpoint (outside the cluster) . the endpoint return the response in JSON format and we need to take some property from this JSON response and decide to fire an alert or not, I found the black-box exporter however, I didn't find a way to get the data from the json response, any idea if it possible?

https://github.com/prometheus/blackbox_exporter

Or if there any other tool/exporter which can help us to check JSON response, please let me know :-)

lets assume that if the response in the json foo=1 I need to fire alert and if foo=2 not

Answer 1

Blackbox exporter

The blackbox exporter does not have a feature to validate JSON (in a sane way) but it can validate HTTP response using regex. In other words, you may try something like this:

# blackbox exporter configuration file
# reference: https://github.com/prometheus/blackbox_exporter/blob/master/CONFIGURATION.md
  my_custom_module:
    prober: http
    http:
      valid_status_codes: []  # Defaults to 2xx
      method: GET
      # it's not necessary to use both of these
      fail_if_body_matches_regexp:
        - '.*"status":"failure".*'
      fail_if_body_not_matches_regexp:
        - '.*"status":"success".*'

To use this, you also have to patch your scrape job in Prometheus so that it uses my_custom_module, e.g.:

scrape_configs:
  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [my_custom_module]

This will work well in the majority of situations but it may be not good enough to validate the full schema. In other words, it will do if your response JSON is short and simple.

Other options

I know no alternative exporter that can do JSON validation (and I didn't bother myself looking for it) but it's not a big deal to write a custom exporter. There are client libraries for Prometheus (https://prometheus.io/docs/instrumenting/clientlibs/) and using one of them you can easily do what you need. Here's a short example in python for you to get started:

from prometheus_client import start_http_server, Gauge
from time import sleep
import requests

my_server_status = Gauge('here_goes_metric_name', 'here goes the description')

def update_status():
    response = requests.get('https://api.example.com').json()
    # check whatever you like
    if response['status'] == 'success':
        # after you scrape this, you can write an alert like "here_goes_metric_name != 1"
        my_server_status.set(1)
    else:
        my_server_status.set(0)

if __name__ == '__main__':
    # Start up the server to expose the metrics.
    start_http_server(8000)
    # And update the status once in a while.
    while True:
        update_status()
        sleep(30)
    # A better way would be to write a collector class instead, 
    # but it's rather complicated for this example.