Search code examples
elasticsearchlucenegrafanaopensearchgrafana-variable

Properly plotting on Grafana from Elasticsearch data


I wan to plot a time series on Grafana from some documents stored in ElasticSearch. The documents are structured as follows:

{
    "timestamp": date,
    "name1": str,
    "name2": str,
    "number": float
 }

Since for a given a value of name1 only certain values of name2 are possible, I set up two variables on Grafana:

Name1: {"find": "terms", "field":"name1.keyword"}
Name2: {"find":"terms", "field":"name2.keyword", "query":"name1:$Name1"}

These two variables correctly generate two drop-down menus that I can use to select the proper value of name2 given a value of name1. Now, after selecting a given value of name1 and name2 in the menus I would like Grafana to plot the time series of number corresponding to that (unique) combination of name1 and name2, and I'm trying to do it through the following query:

name1:$Name1 AND name2:$Name2

selecting Group By: Date Histgram and timestamp as the time axis. Problem is, this does not work properly unless I have only one single value of name2 for each name1, and let me explain why via an example. Suppose I have these two documents:

{
    "timestamp": 20220110,
    "name1": "foo",
    "name2": "bar",
    "number": 2.0
 }

{
    "timestamp": 20220110,
    "name1": "foo",
    "name2": "tee",
    "number": 3.0
 }

and that I choose max as metric. In that case, no matter what I select in the name2 drop down menu, Grafana will always plot a point (20220110, 3), where 3 corresponds to the maximum between 2 and 3. I understand the reason of this behavior: what I do not understand is how to obtain what I want. So: How can I change my query in such a way that, selecting bar in the name2 drop down menu I get a point (20220110, 2), and if instead I select tee from the same menu I obtain (20220110, 3)?


Solution

  • For sake of completeness, I figured out the solution. So, the behavior was not supposed to be the one I described above, and the reason is that name2 was usually using - characters in the string, which were poorly interpreted by the Lucene query. The only safe way to proceed in this case is by writing the variables in this way:

    Name1: {"find": "terms", "field":"name1.keyword"}
    Name2: {"find":"terms", "field":"name2.keyword", "query":"name1.keyword:$Name1"}
    

    and the corresponding Lucene query as

    name1.keyword:$Name1 AND name2.keyword:$Name2
    

    This works as expected.