Search code examples
rubysolrsunspot

Select all records with Sunspot

Looks like Solr always paginates the results. What's the best approach to select all records at once with sunspot? I see two options

  • set the page size to very large number
  • use cursor-based pagination and a few requests

The cursor-based pagination has its obvious drawbacks.

Are there any potential issues with setting the page size to a large number?

Thanks!

Solution

  • If you really need to get all index entries at once, you can use a very large page size. The time to select all documents will not be that large, but it will cost a bit of time and space to transmit that complete dataset, obviously.

    There is one trap, though. When you hardwire your very large page size it's always possible for the index to grow larger than the page size you estimated in the first place. You can avoid this by querying the index for all documents while setting the page size to 0, parsing the numFound and then asking for a page sized numFound + 1.