I have an unscheduled batch job that may or may not be triggered multiple times a day. For every invocation, I'm generating a count metric incremented by 1.
I expected to apply increase() function over 1 day for the metric and see a spike for every invocation of that day.
However, I noticed increase function doesn't show or extrapolate data for extended missing period (like the period before adding these metrics) and hence the first invocation always makes zero increments.
I'm trying something like increase(job_invocation_total[1d]).
I also tried to set the non-existent values as zero with OR on() vector(0) but increase() function didn't allow it.
Similarly for clamp_min()
I'm pretty sure there must be a standard way of achieving it but unfortunately, I can't think of it.
If you have access to the code where the metric job_invocation_total is created and the cardinality of the metric labels is bound, you can initialize the counter with the value 0.
If this is not the case, you can create a recording rule that is either 0 or equal to the metric job_invocation_total.