Splunk: search for “a first log that got printed, but the second was not printed”

Question

Small question for a Splunk query please.

May I ask if there is a way to search for “a first log that got printed, but the second was not printed” statement please? Background, I have a very simple piece of Java logic as follow:

LOGGER.info("START/END compute something that might result in a bad exception for id START " + id);
invote_method_which_can_fail(id);
LOGGER.info("START/END compute something that might result in a bad exception for id END " + id);

Which results in something like (snippet from a million):

        START/END compute something that might result in a bad exception for id START 12345
        START/END compute something that might result in a bad exception for id END 12345

        START/END compute something that might result in a bad exception for id START 88888

        START/END compute something that might result in a bad exception for id START 98765
        START/END compute something that might result in a bad exception for id END 98765

As you can see, the id 88888 in my example got the start statement printed, but not the end statement, because something bad happened in the java code. (the question is not about how to make the java code reliable)

May I ask if there is a Splunk query which can find me those id please?

What I tried: So far, I am downloading the search result containing all the starts. Then, downloading the search results with all the ends. Once having both, I am running another offline script in order to find all the id from the first search result that are not there from the second...

I do not think this is "the smart thing to do" and was wondering if there is a smarter query which can give me the expected result directly in Splunk.

Thank you

Answer 1

You can try something along these lines (with rex and stats):

index=... "START/END compute something that might result in a bad exception for id"
| rex "(?<operation>(START|END))\s+(?<id>\d+)"
| stats count(eval(operation="START")) as start count(eval(operation="END")) as end by id
| where NOT start=end 

I have not tested this SPL code