Search code examples
splunksplunk-querysplunk-dashboard

How to use tokens from 2 time range inputs in single Splunk dashboard query?

I'm using Splunk classic dashboards where I have 2 time range inputs. I want to compare data for 2 time frames in a single table. Essentially, I want to perform query which counts errors by type for period A and B, then join the searches by error type so that I can see how many errors of each type there were in period A as opposed to period B.

I added a panel as follows:

enter image description here

because I want to use tokens from both time inputs for the query:

(index=myindex) earliest="$runATimeInput.earliest$" latest="$runATimeInput.latest$" environment="$runAEnvironment$"  level=ERROR 
| spath input=message 
| stats count by  logIdentifier 
| sort count desc 
| join left=L right=R where L.logIdentifier = R.logIdentifier 
    [| search (index=myindex) earliest="$runBTimeInput.earliest$" latest="$runBTimeInput.latest$"  environment="$runBEnvironment$"  level=ERROR 
    | spath input=message 
    | stats count by logIdentifier ]

The problem is that the query doesn't return any results although it should. The main query returns results:

(index=myindex) earliest="$runATimeInput.earliest$" latest="$runATimeInput.latest$" environment="$runAEnvironment$"  level=ERROR 
| spath input=message 
| stats count by logIdentifier 
| sort count desc

However the subsearch query doesn't return any results (although a separate search for the same period in a new tab returns results):

[| search (index=myindex) earliest="$runBTimeInput.earliest$" latest="$runBTimeInput.latest$"  environment="$runBEnvironment$"  level=ERROR 
| spath input=message 
| stats count by  logIdentifier ]

When I click on Run Search in Splunk panel in order to open the search in a new tab I see strange values for earliest/latest tokens. For the main query the values are: earliest="1669500000" latest="1669506493.677" where 1669500000 is the Tue Jan 20 1970 09:45:00 and 1669506493.677 is Sun Nov 27 2022 01:48:13 whereas the timeframe for period 1 was Sun Nov 27 2022 00:00:00 - Sun Nov 27 2022 01:48:13. That being said the main query works and it respects the original time frame.

The values for the second query are earliest="1669813200" latest="1669816444.909" where 1669813200 is Tue Jan 20 1970 09:45:00 and 1669816444.909 is Wed Nov 30 2022 15:54:04 whereas the period 2 timeframe was Wed Nov 30 2022 15:00:04 - Wed Nov 30 2022 15:54:04`.

Am I doing something wrong in the panel settings or the query? Or maybe there's another way to do this in Splunk?

Below is the dashboard XML:

<form>
  <label>My Dashboard</label>
  <description>My Dashboard</description>
  <fieldset submitButton="false" autoRun="true">
    <input type="time" token="runATimeInput" searchWhenChanged="true">
      <label>Run A</label>
      <default>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </default>
    </input>
    <input type="dropdown" token="runAEnvironment" searchWhenChanged="true">
      <label>Run A Environment</label>
      <choice value="prod">prod</choice>
      <default>prod</default>
    </input>
    <input type="time" token="runBTimeInput" searchWhenChanged="true">
      <label>Run B</label>
      <default>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </default>
    </input>
    <input type="dropdown" token="runBEnvironment" searchWhenChanged="true">
      <label>Run B Environment</label>
      <choice value="prod">prod</choice>
      <default>prod</default>
    </input>
  </fieldset>
  <row>
    <panel>
      <title>Top Exceptions</title>
      <table>
        <title>Top Exceptions</title>
        <search>
          <query>(index=distapps) earliest="$runATimeInput.earliest$" latest="$runATimeInput.latest$" environment="$runAEnvironment$"  level=ERROR  | spath input=message 
| stats count by  logIdentifier 
| sort count desc 
| join left=L right=R where L.logIdentifier = R.logIdentifier 
    [| search (index=myindex) earliest="$runBTimeInput.earliest$" latest="$runBTimeInput.latest$"  environment="$runBEnvironment$"  level=ERROR 
    | spath input=message 
    | stats count by logIdentifier ]</query>
          <earliest>$runATimeInput.earliest$</earliest>
          <latest>$runBTimeInput.latest$</latest>
        </search>
        <option name="drilldown">none</option>
        <option name="refresh.display">progressbar</option>
      </table>
    </panel>
  </row>
</form>
Solution

  • Here's a test dashboard I created that uses two timepickers. It produces results for both time periods. How is yours different? Could it be the count field is used in both the main and subsearches?

    <form version="1.1">
  <label>test</label>
  <fieldset submitButton="false">
    <input type="time" token="runATimeInput">
      <label>A</label>
      <default>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </default>
    </input>
    <input type="time" token="runBTimeInput">
      <label>B</label>
      <default>
        <earliest>-48h@h</earliest>
        <latest>-24h@h</latest>
      </default>
    </input>
  </fieldset>
  <row>
    <panel>
      <table>
        <search>
          <query>(index=_internal) earliest="$runATimeInput.earliest$" latest="$runATimeInput.latest$"
| stats count as countA by component 
| join component [| search (index=_internal) earliest="$runBTimeInput.earliest$" latest="$runBTimeInput.latest$"  
    | stats count as countB by component ]</query>
          <earliest>$runATimeInput.earliest$</earliest>
          <latest>$runATimeInput.latest$</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="drilldown">none</option>
        <option name="refresh.display">progressbar</option>
      </table>
    </panel>
  </row>
</form>