Search code examples
fiwarefiware-orion

FIWARE - Orion LD offset parameter does not provide any results after threshold

I am using FIWARE Orion-LD in order to store Social Media posts, whose number exceeds 300,000.

I would like to access all the entities of type SMPost which belong to a specific SMCollection.

GET <orion_host>:1026/ngsi-ld/v1/entities?options=keyValues&q=belongsToCollection==%22urn:ngsi-ld:SMCollection:XX%22&type=SMPost&limit=1000&offset=XXX&options=count

For this purpose, I am employing pagination, as instructed in the official documentation, using the parameters limit and offset. Using the parameter options=count, I confirm that indeed all the entities are correctly inserted in the MongoDB.

Everything goes well until the offset is set to 34000, then, when exceeded, an empty list is returned, as a response.

Is this a known issue? Could you please provide an alternative way that I could get this data?

Thank you very much in advance for your response.

Solution

  • In the end, the issue was the lack of indexing in the mongodb for the attribute creDate. After adding the appropriate index, the problem was solved. Thank you for the support!