Search code examples
splunksplunk-query

How to Cluster and create a timechart in splunk

I have a field with LogMsg error messages That I am grouping based on similarities using cluster.

What I am trying to achieve is a display that will show a timeseries with the grouped error

index="my_index_here" LogLevel=ERROR
  | cluster showcount=t t=0.2 field=Message | eval "Error Count" = cluster_count
  | head 10 | timechart count("Error Count") By LogMsg span=60m

The Idea is this

  1. Get all the error Messages LogLevel=ERROR
  2. Group the items based on Message field | cluster showcount=t t=0.2 field=Message | eval "Error Count" = cluster_count
  3. Get top 10 results | head 10
  4. Draw a timechart timechart count("Error Count") By LogMsg span=60m. The time chart should have a plot of number different error messages generated from the cluster against time, something like
Message 8.00 9:00 10.00 11:00
Unable to authenticate 90 40 30 60
Another Error 80 40 30 60
Yet another error 70 40 30 60
--- --- --- --- ---
The 10th most frequent error 50 40 30 60

My approach above is not working returning a blank plot,

Solution

  • The way to debug SPL is to execute one pipe at time and verify the results before adding the next pipe.

    One thing I believe you'll discover is the head command ruins the timechart. It's possible all of the top 10 results will be in the same hour so the results may be less than useful.

    A common cause of a "blank plot" is a stats or timechart command that references a non-existent or null field. You should discover which field is null during the debug.

    FWIW, here's a run-anywhere query similar to yours that produces a plot.

    index=_internal log_level=INFO 
| cluster showcount=t t=0.2 field=event_message 
| eval "Error Count" = cluster_count 
| head 10 
| timechart count("Error Count") By group span=60m