I'm currently using the Bing Web Search API v7 to query Bing for search results. As per the API docs, the parameters count and offset are used to paginate through the results, the total number of which are defined in the results themselves by the value of totalEstimatedMatches.
As below from the documentation:
totalEstimatedMatches: The estimated number of webpages that are relevant to the query. Use this number along with the count and offset query parameters to page the results.
This seems to work up to a point, after which the API just continues to return the exact same results over and over, regardless of the values of count and offset.
In my specific case, the totalEstimatedMatches was set at 330,000. With a count of 50 (i.e. 50 results per request) the results begin repeating at around offset 700 i.e. 3,500 results into the estimated 330,000.
In playing with the bing front end, I have noticed a similar behaviour once the page count get sufficiently high e.g.
51,000 resultsAm I using the API incorrectly or is this just some sort of limitation or bug in which the totalEstimatedMatches is just way off?
totalEstimatedMatches provides total number of matches for that query around the web - that includes duplicate results and near similar content as well.
In order to optimize indexing all search engines restrict results to top N webpages. This is what you are seeing. This behavior is consistent across all the search engines as typically near all the users change a query/select a webpage/abandon within 2-3 search pages.
In short, this is not a bug/incorrect implementation but optimization of index that's restricting you from getting more results. If you really need to get more results, you can use the related searches and append the unique webpages.