Elasticsearch: Accessing multiple indices created by filebeat

Question

I'm using Elasticsearch 6.8 and i'm not able to access multiple indices in a single query. I've read the documentation and also previous questions, but for some reason i can't figure this out.

The setup is fairly standard, i think. I have a filebeat pushing logs into elasticsearch. Didn't really adapt the standard config, so filebeat is creating a new index everyday. I can nicely query every single index to get all results from a day:

# works, returns data from filebeat-6.1.2-2019.12.20
curl -X GET "https://whatever:9200/filebeat-6.1.2-2019.12.20/_search?pretty" -H 'Content-Type: application/json' -d'
{
    "query": {
        "wildcard" : {
            "nginx.access.url" : "/something/*"
        }
    }
}
'

# works, returns data from filebeat-6.1.2-2019.12.19
curl -X GET "https://whatever:9200/filebeat-6.1.2-2019.12.19/_search?pretty" -H 'Content-Type: application/json' -d'
{
    "query": {
        "wildcard" : {
            "nginx.access.url" : "/something/*"
        }
    }
}
'

I'm now trying to query more than one index in a query, to get data from more than one day at once, but i always only get data from the the index filebeat-6.1.2.-2019.12.19, no matter what i do.

# comma-separated list of indeces, nope.
curl -X GET "https://whatever:9200/filebeat-6.1.2-2019.12.20,filebeat-6.1.2-2019.12.19/_search?pretty" -H 'Content-Type: application/json' -d'
{
    "query": {
        "wildcard" : {
            "nginx.access.url" : "/something/*"
        }
    }
}
'

# _all, nope.
curl -X GET "https://whatever:9200/_all/_search?pretty" -H 'Content-Type: application/json' -d'
{
    "query": {
        "wildcard" : {
            "nginx.access.url" : "/something/*"
        }
    }
}
'

# just not specifying an index, nope nope nope.
curl -X GET "https://whatever:9200/_search?pretty" -H 'Content-Type: application/json' -d'
{
    "query": {
        "wildcard" : {
            "nginx.access.url" : "/something/*"
        }
    }
}
'

since this should just work according to all the resources i find, i have the feeling i'm missing something super-obvious.

FWIW, i'm coming from the python library elasticsearch-dsl where i had the exact same problems, but i've used curl in this examples, because it's most generic thing to do. so if anybody has an answer in this direction, this would be very welcome as well.

Answer 1

Maybe this is just the size that you need to specify. by default, the search query 10 result. Have you tried to increase this value ?

you can try to add "size":"10000" before "query" in your request.

curl -X GET "https://whatever:9200/filebeat-6.1.2-2019.12.20,filebeat-6.1.2-2019.12.19/_search?pretty" -H 'Content-Type: application/json' -d'
{
    "size":"10000",
    "query": {
        "wildcard" : {
            "nginx.access.url" : "/something/*"
        }
    }
}