Search code examples
bing-apiazure-cognitive-services

Bing custom search apis returning only limited results from one location and full result from different location


I am trying to use Bing Custom Search's API for documents from Cognitive Services. The strange thing is that when I run it from India, it gives me more than a thousand results, but when I run it from a US server, it returns only 25 (sometimes 50 results). Here is the sample code for that:

var totalCount = 0;
var filetypes = new List<string> { "pdf", "docx", "doc" };
foreach (var filetype in filetypes)
{
    var searchTerm = "microsoft%20.net%20resume+filetype%3a" + filetype;
    Console.WriteLine("Searching for : " + filetype);

    for (var i = 0; i < 40; i++)
    {
        var nextCount = 0;
        var url = "https://api.cognitive.microsoft.com/bingcustomsearch/v7.0/search?" +
                  "q=" + searchTerm +
                  "&customconfig=" + customConfigId +
                  "&count=25" + "&offset=" + ((i * 25) + nextCount);

        using (var client = new HttpClient())
        {
            client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", subscriptionKey);
            var httpResponseMessage = client.GetAsync(url).Result;
            var responseContent = httpResponseMessage.Content.ReadAsStringAsync().Result;
            BingCustomSearchResponse response =
                JsonConvert.DeserializeObject<BingCustomSearchResponse>(responseContent);

            if (response.webPages == null || response.webPages.value.Length <= 0)
            {
                Console.WriteLine("response.webPages is null ");
                break;
            }

            foreach (var webPage in response.webPages.value)
            {
                Console.WriteLine("name: " + webPage.name);
                Console.WriteLine("url: " + webPage.url);
                Console.WriteLine("displayUrl: " + webPage.displayUrl);
                Console.WriteLine("snippet: " + webPage.snippet);
                Console.WriteLine("dateLastCrawled: " + webPage.dateLastCrawled);
                Console.WriteLine();
            }
            totalCount = totalCount + response.webPages.value.Length;
        }
    }
}

The subscription key I am using is a trial key.


Solution

  • I got the reason of this behavior. Actually it had nothing to do with region/country/market. After looking into the response i got this message. "Rate limit is exceeded. Try again in 1 seconds" It means for after each call in the loop i have to wait for 1 second to give next call. Now need to know is this limit for trial subscription or this is kept for all calls to prevent DDOS attack or something. May be from India it was working because may one iteraction is already taking one or more second.