Search code examples
azureazure-cognitive-search

Azure Search: 16 MB limit not applicable on Edm.String field

For the indexing REST API call (/indexes/test-index/docs/index), the documentation states that: "For a batch with a single document, the maximum document size is 16 MB of JSON."

However, for the field 'body2' of type Edm.String in my index, I passed a 50 MB long string (5,24,28,800 ASCII characters) which was successfully added to the index.

Search index definition When does the 16 MB limit apply? Do the same limits apply to the .NET SDK as well?

The powershell script which I ran:

$headers = @{
'api-key' = '<API Key>'
'Content-Type' = 'application/json' 
'Accept' = 'application/json' 
}

$body = @"
{
    "value": [
    {
    "@search.action": "upload",
    "id":"ddf56362-378d-481f-9313-7b8f9cd99999",
    "body2":"<50 MB long string>"
    }
]}"@

$url = "https://<index-name>.search.windows.net/indexes/test-index/docs/index?api-version=2019-05-06"

Invoke-RestMethod -Uri $url -Headers $headers -Method Post -Body $body | ConvertTo-Json

Response:

{
  "key": "ddf56362-378d-481f-9313-7b8f9cd99999",
  "status": true,
  "errorMessage": null,
  "statusCode": 200
}
Solution

  • You're probably seeing the some late-stage flighting of a potential limits increase. We're exploring increasing the overall request size in order to give people room to index larger batches, which can have some nice improvements on overall request rates.

    We're looking for data on what to do around the edges. The driving scenario is larger batches of reasonably sized documents, but we're considering what do about really big documents and really big fields. It's seems likely that we'll end up relaxing the overall request size and impose a limit on field size, but that remains to be seen.

    We take care to plan our flighting in ways that optimize for application compatibility, but if you want to be 100% sure of forward compatibility stay close to the documented limits until we update the documentation.