Fetch all the rows using elasticsearch_dsl

Question

Currently i am using the following program to extract the id and its severity information from elastic search .

from elasticsearch import Elasticsearch
from elasticsearch_dsl import Search, Q

client = Elasticsearch(
    [
        #'http://user:secret@10.x.x.11:9200/',
        'http://10.x.x.11:9200/',
    ],
    verify_certs=True
)

s = Search(using=client, index="test")

response = s.execute()

for hit in response:
  print hit.message_id, hit.severity,  "\n\n"

i believe by default the query returns 10 rows. I am having more than 10000 rows in elastic search. I need to fetch all the information.

Can some one guide me how to run the same query to fetch all records ?

Answer 1

You can use the scan() helper function in order to retrieve all docs from your test index:

from elasticsearch import Elasticsearch, helpers

client = Elasticsearch(
    [
        #'http://user:secret@10.x.x.11:9200/',
        'http://10.x.x.11:9200/',
    ],
    verify_certs=True
)

docs = list(helpers.scan(client, index="test", query={"query": {"match_all": {}}}))

for hit in docs:
  print hit.message_id, hit.severity,  "\n\n"