Search code examples
javaprometheusprometheus-java

Flaky prometheus java client

I'm using the Prometheus 0.6.0 Java client in a relatively simple app. All I have are two counters and two gauges, and they are exposed via an HTTP REST endpoint, using the built-in Prometheus HTTPServer And I'm seeing pretty flaky behavior.

Sometimes, the counters won't be registered at all (e.g., never show up when hitting the HTTP endpoint), other times the # HELP and # TYPE lines show up, but not the actual metric. I'm able to see the flakiness both in my local dev box, and deployed out on kubernetes pods.

To narrow this down further, I have this bit of code to initialize the counter. And I'm consistently getting an NPE in the call to counter.inc().

Counter counter = Counter.build().name("counter").help(helpText).labelNames(tags).register();
counter.inc();

I tried the above with gauges, and I get the same error.

For context, this is being called from a class ApplicationMetrics with the following code:

public static Counter counter1;
public static Counter counter2;

static void initializeMetrics() {
    counter1 = Counter.build().name("counter1").help(helpText1).labelNames(tags1).register();
    counter1.inc();

    counter2 = Counter.build().name("counter2").help(helpText2).labelNames(tags2).register();
    counter2.inc();
}

And I'm issuing a static method call from a start method in my main application launch class.

void start() {
    // Initialize stuff for the rest of the app
    ApplicationMetrics.initializeMetrics();
}

Am I doing something wrong with how the counters are being configured/set up? I wasn't able to find anything on the prometheus issues list regarding this.

Solution

  • You're creating a counter with labels (the .labelNames(tags) bit), which actually means you're creating multiple metrics. As in:

    request_count{api="/users"} 15
request_count{api="/groups"} 3

    So when you want to increment one of those metrics you need to say which one.

    counter1.labels(tagValues1).inc();

    Else, if you actually only want a single counter, just drop the .labelNames(tags1) call to the builder.

    Regardless, this question (and answer) would have been more obvious had you used a real-life example instead of "counter1" and tags1:

    Counter requestCount = Counter.build()
    .name("request_count")
    .help("Number of requests, per API")
    .labelNames("api")
    .register();

    Then it would have been obvious that incrementing requestCount directly doesn't make much sense.