Using DataType.Complex in Azure Search Indexer using FieldMapping()

Question

I am trying to map a nested object from MongoDB (CosmosDB) to Azure Search Indexer.

First, here is what I have stored in MongoDB.

{
    "_id" : {
        "$binary" : "eHemgNj2FkWlqECKkGKnJA==",
        "$type" : "03"
    },
    "UpdatedBy" : {
        "_id" : {
            "$binary" : "0wtu6BgDm0GrTbffr1EmhQ==",
            "$type" : "03"
        },
        "Email" : "[email protected]"
    },
    "Status" : "New",
    "Name" : "123",
    "CustomerName" : ""
}

Then, I have c# program using Microsoft.Azure.Search.Models nuget package to create an index programatically.

private async Task StartIndexAsync(bool resetIndexer = true)
{
    await CreateIndexAsync(new[]{
        new Field(nameof(ProjectSearchModel.Id),              DataType.String)     { IsKey = true,  IsSearchable = false, IsFilterable = false, IsSortable = false, IsFacetable = false, IsRetrievable = true},
        new Field(nameof(ProjectSearchModel.Name),            DataType.String)     { IsKey = false,  IsSearchable = true, IsFilterable = true, IsSortable = true, IsFacetable = false, IsRetrievable = true},
        new Field(nameof(ProjectSearchModel.CustomerName),    DataType.String)     { IsKey = false,  IsSearchable = true, IsFilterable = true, IsSortable = true, IsFacetable = false, IsRetrievable = true},
        Field.NewComplex(nameof(ProjectSearchModel.UpdatedBy), false, new [] {
            new Field(nameof(ProjectSearchModel.UpdatedBy.Id),     DataType.String)     { IsKey = false,  IsSearchable = false, IsFilterable = false, IsSortable = false, IsFacetable = false, IsRetrievable = true},
            new Field(nameof(ProjectSearchModel.UpdatedBy.Email),  DataType.String)     { IsKey = false,  IsSearchable = true, IsFilterable = true, IsSortable = true, IsFacetable = false, IsRetrievable = true}
        })
        },
    new[] {
        nameof(ProjectSearchModel.Name),
        nameof(ProjectSearchModel.Number),
        nameof(ProjectSearchModel.CustomerName),
        $"{nameof(ProjectSearchModel.UpdatedBy)}/{nameof(ProjectSearchModel.UpdatedBy.Email)}"
    });

    await CreateDatasourceAsync();
    await StartIndexerAsync(resetIndexer);
}

Then, for indexer, I have defined some FieldMappings because I wanted to map _id in MongoDB to Id field in Indexer.

public async Task CreateIndexerAsync(string indexerName, string datasourceName, string indexName)
{
    _logger.LogInformation("{0}", "Creating Indexer and syncing data...\n");

    var indexer =
        new Indexer()
        {
            Name = indexerName,
            Description = "Data indexer",
            DataSourceName = datasourceName,
            TargetIndexName = indexName,
            FieldMappings = new List<FieldMapping> { new FieldMapping() { SourceFieldName = "doc_id", TargetFieldName = "Id" } }
        };

    try
    {
        await _searchClient.Indexers.CreateOrUpdateAsync(indexer);
    }
    catch (Exception ex)
    {
        _logger.LogError("Error creating and running indexer: {0}", ex.Message);
        return;
    }

    await StartCreation(indexerName);
}             

Now, _id from MongoDB is correctly mapped to Id field in Indexer from the code above.

{
    "@odata.context": "myprojectendpoint/indexes('myproject-index-dev')/$metadata#docs(*)",
    "value": [
        {
            "@search.score": 1,
            "Id": "30dbf04d-cbc7-4597-8d48-209f3a320cf8",
            "Name": "friday soon",
            "CustomerName": "Kyle Ahn",
            "UpdatedBy": {
                "Id": null,
                "Email": "[email protected]"
            }
        }
    ]
}

I would like to do the same for Id sub-field inside UpdatedBy field. So, I would like to map UpdatedBy._id in MongoDB to UpdatedBy/Id in index.

Is there a way to achieve this?

Huge thanks to everyone in advance!

Answer 1

This is not supported by Azure search indexers. When adding field mappings, your target should be a top-level index field. This means, you can possibly map one complex object from your data source as a whole to a complex field in the index - but you cannot map one sub-field from a complex object in the source to a "child" of a complex field in your index.

Point #2 in the field mappings document states this, but I'll update it to be clearer.

As a work around, you could either try to modify the "UpdatedBy" property to have sub-fields that align with the index definition; or you could try to modify the sub-field directly using the SDK, using something like the following (I am assuming a name for your index data model)

IndexAction.MergeOrUpload(
    new Customer()
    {
        Id = "....",
        UpdatedBy = new 
        {
            Id = "..."
        }
    }
)

This will "merge" (add) the missing Id property from your MongoDb into your search index - note that this only works because your "UpdatedBy" complex field is not a collection.