Search code examples
azureazure-cosmosdbazure-cognitive-search

Create Indexer for Azure AI Search in HTTP for private CosmosDB for MongoDB?

Currently, I am able to create a datasource and an index through HTTP requests through Postman. Both my Azure AI Search and my CosmosDB for Mongo have public access disabled.

POST request for my datasource which works:

POST https://example-mongo.search.windows.net/datasources?api-version=2024-07-01
    Content-Type: application/json
    api-key: [my key for azure search]

{   
    "name" : "collection-datasource",  
    "description" : "Anything you want, or nothing at all",  
    "type" : "cosmosdb",
    "credentials" : { 
        "connectionString" : "AccountEndpoint=https://example-mongo.documents.azure.com:443/;AccountKey=abcd==;Database=DB;ApiKind=MongoDB" 
        },
    "container": {
        "name": "collection"
    }
}

POST request for indexer which does not work and gives me a 403 error:

POST https://example-mongo.search.windows.net/indexers?api-version=2024-07-01
    Content-Type: application/json
    api-key: [my key for azure search]

{
  "name": "collection-indexer",
  "description": null,
  "dataSourceName": "collection-datasource",
  "targetIndexName": "collection-index", #Created previously through postman as well
  "parameters": {
    "batchSize": null,
    "maxFailedItems": null,
    "maxFailedItemsPerBatch": null
  },
  "fieldMappings": [
    {
      "sourceFieldName": "productDescription",
      "targetFieldName": "productDescriptionCustom",
      "mappingFunction": null
    }
  ]
}

The 403 error:

Error with data source: Response status code does not indicate success: Forbidden (403);
Substatus: 0; ActivityId: 1ffa970a-5bb7-42fa-8ace-968fc329bae1;
Reason: (Request originated from IP --- through public internet.
This is blocked by your Cosmos DB account firewall settings. 
More info: https://aka.ms/cosmosdb-tsg-forbidden\r\nActivityId: --,
Microsoft.Azure.Documents.Common/2.14.0, Windows/10.0.20348 cosmos-netstandard-sdk/3.30.20); 
Please adjust your data source definition in order to proceed

The same error happens when I try to create an indexer through the portal with my existing index and datasource.

I'm unsure what "Please adjust your data source definition" means. Am I able to somehow add the connection string for my CosmosDB into my request to create an indexer? Or is there another issue that I have to fix?

Solution
    • Due to the public access restrictions on the Cosmos DB account. Azure Cognitive Search prevents accessing to the Cosmos DB data source because it's coming from an IP address that's not whitelisted in your Cosmos DB firewall settings.

    First, configure cosmos DB account firewall settings to allow access from Azure Cognitive Search. By this we can add the IP range or service tag of Azure Cognitive Search to the Cosmos DB firewall.

    enter image description here

    • Data Source Configuration which is given in the question is correct check all the necessary permissions needed.

    enter image description here

    Update:

    • The RU version of Cosmos DB, especially when using the MongoDB API, allows you to restrict access using firewalls and virtual networks. However, it doesn't support outbound connections via private endpoints for Azure Cognitive Search, meaning that private network setups won't work directly in this context.

    • Configure the firewall settings on the Cosmos DB account to allow the IP ranges. It is crucial because, by default, public access to Cosmos DB is restricted, and any attempt by Azure Cognitive Search to access the data source will be blocked unless the IP address is whitelisted.