Search code examples
splunksplunk-querysplunk-dashboard

How to use a token for a rex in Splunk?


I have a token $token_rex$ set up as follows in the dashboard:

<set>mvjoin(mvmap('token_keywords_mv',"(?&lt;".'token_keywords_mv'."&gt;".'token_keywords_mv'."+?)"), "|")</set>

token_keyrowrds_mv is basically the following:

lorem
ipsum
situs

The resulting token looks like this:

(?<lorem>lorem+?)|(?<ipsum>ipsum+?)|(?<situs>situs+?)

The query I am trying to run is as follows:

index=* | rex field _raw "(?i)".$token_rex$

Ideally the above should be rendered like so:

index=* | rex field _raw "(?i)(?<lorem>lorem+?)|(?<ipsum>ipsum+?)|(?<situs>situs+?)"

I have tried using the token filters $token_rex|s$ and $token_rex|n$ but neither work. I have even tried to return the value from a subsearch:

index=* | rex field _raw [| makeresults | eval string_rex=$token_rex$ | return $token_rex]

Update: Found out that the token is being given as verbose to the rex command. My token is set as follows:

<set token="token_rex">mvjoin(mvmap('token_keywords_mv',"(?&gt;".'token_keywords_mv'."&lt;".'token_keywords_mv'."+?)"), "|")</set>

When I used it as ... | rex field=_raw '(?i)$token_rex$' it gives me the following error:

Error in 'rex' command: Encountered the following error while compiling the regex ''(?i)mvjoin(mvmap('token_keywords_mv'': Regex: missing closing parenthesis.

When I set it as the value directly, however, it works:

<set token="token_rex">(?&lt;lorem&gt;lorem+?)|(?&lt;ipsum&gt;ipsum+?)|(?&lt;situs&gt;situs+?)</set>

Update: Here's a sample dashboard.

<dashboard theme="dark">
  <init>
    <set token="token_keywords_mv">split("lorem,ipsum,situs", ",")</set>
    <set token="token_keywords_starred">"*".mvjoin($token_keywords_mv$, "* OR *")."*"</set>
    <set token="token_rex">"(?i)".mvjoin(mvmap('token_keywords_mv', "(?&lt;".'token_keywords_mv'."&gt;".'token_keywords_mv'."+?)"), "|")</set>
    <set token="token_raw">(?&lt;lorem&gt;lorem+?)|(?&lt;ipsum&gt;ipsum+?)|(?&lt;situs&gt;situs+?)</set>
  </init>
  <label>Test Search</label>
  <description>Multivalue</description>
  <row>
    <panel>
      <table>
        <search>
          <query>index=* 
    [| makeresults 
    | eval string_search=$token_keywords_starred$ 
    | return $string_search] 
| rex field=_raw $token_raw|s$ 
| stats count(lorem) AS Lorem, count(ipsum) AS Ipsum, count(situs) AS Situs
| eval header="Count" 
| transpose column_name="String" header_field=header</query>
          <earliest>-24h@h</earliest>
          <latest>now</latest>
        </search>
        <option name="drilldown">none</option>
        <option name="refresh.display">progressbar</option>
      </table>
    </panel>
  </row>
  <row>
    <panel>
      <table>
        <title>ECHO</title>
        <search>
          <query>| localop 
| makeresults 
| eval token_keywords_starred=$token_keywords_starred$, token_keywords_mv=$token_keywords_mv$, token_rex=$token_rex$, token_raw=$token_raw|s$</query>
          <earliest>-24h@h</earliest>
          <latest>now</latest>
        </search>
        <option name="drilldown">none</option>
        <option name="refresh.display">progressbar</option>
      </table>
    </panel>
  </row>
</dashboard>

If any other token replaces $token_raw|s$, there will be an error. Using Splunk 8.0.3.


Solution

  • When I try to reproduce your results with this dashboard code:

    <form>
      <label>test</label>
      <fieldset submitButton="false">
        <input type="text" token="token_rex">
          <label>field1</label>
        </input>
      </fieldset>
      <row>
        <panel>
          <event>
            <title>test</title>
            <search>
              <query>| makeresults | eval _raw="lorem ipsum dolor"
    | rex "(?i)$token_rex$"
    |  table lorem ipsum situs</query>
              <earliest>$earliest$</earliest>
              <latest>$latest$</latest>
            </search>
            <option name="refresh.display">progressbar</option>
          </event>
        </panel>
      </row>
    </form>
    

    I get this optimized search (as reported by Job Inspector):

    | makeresults | rex field=_raw "(?i)(?<lorem>lorem+?)|(?<ipsum>ipsum+?)|(?<situs>situs+?)"
    

    It looks like the concatenation is not needed.


    Update: Change all the set tags to eval and use mvzip instead of mvmap. The set tag does not compute, while the eval tag does.

        <eval token="token_keywords_mv">split("lorem,ipsum,situs", ",")</eval>
        <eval token="token_keywords_starred">"*".mvjoin($token_keywords_mv$, "* OR *")."*"</eval>
        <eval token="token_rex">"(&lt;".mvjoin(mvzip('token_keywords_mv','token_keywords_mv',"&gt;"),"+?)|(?&lt;")."+?)"</eval>
        <set token="token_raw">(?&lt;lorem&gt;lorem+?)|(?&lt;ipsum&gt;ipsum+?)|(?&lt;situs&gt;situs+?)</set>