How to index multiple blobs under a main record in Azure Search?

Question

I followed the steps described on this tutorial. My case is a little bit different:

• Instead of indexing Hotels and Rooms, I am indexing Candidates and Resumes.
• Instead of using CosmosDB I am using an Azure SQL Database.

Following the tutorial, I am able to create the Index, the 2 Indexers (one for the SQL DB and one for the Blobs storage), and the 2 data sources.

The SQL DB contains all my candidates, and the storage contains all their resumes (files with PDF/DOC/DOCX formats). Each blob has a metadata "ResumeCandidateId" that contains the same value as the "CandidateId" for the Candidate.

I have the following fields for my Index:

    [SerializePropertyNamesAsCamelCase]
    public partial class Candidate
    {
        [Key]
        [IsFilterable, IsRetrievable(true), IsSearchable]
        public string CandidateId { get; set; }

        [IsFilterable, IsRetrievable(true), IsSearchable, IsSortable]
        public string LastName { get; set; }

        [IsFilterable, IsRetrievable(true), IsSearchable, IsSortable]
        public string FirstName { get; set; }

        [IsFilterable, IsRetrievable(true), IsSearchable, IsSortable]
        public string Notes { get; set; }

        public ResumeBlob[] ResumeBlobs { get; set; }
    }

    [SerializePropertyNamesAsCamelCase]
    public class ResumeBlob
    {
        [IsRetrievable(true), IsSearchable]
        [Analyzer(AnalyzerName.AsString.StandardLucene)]
        public string content { get; set; }

        [IsRetrievable(true)]
        public string metadata_storage_content_type { get; set; }

        public long metadata_storage_size { get; set; }

        public DateTime metadata_storage_last_modified { get; set; }

        public string metadata_storage_name { get; set; }

        [Key]
        [IsRetrievable(true)]
        public string metadata_storage_path { get; set; }

        [IsRetrievable(true)]
        public string metadata_content_type { get; set; }

        public string metadata_author { get; set; }

        public DateTime metadata_creation_date { get; set; }

        public DateTime metadata_last_modified { get; set; }

        public string ResumeCandidateId { get; set; }
    }

As you can see, one Candidate can have multiple Resumes. The challenge is to populate the ResumeBlobs property...

The data from the SQL DB is indexed and mapped correctly by the Indexer. When I run the Blobs Indexer, it loads documents, however it does not map them and they never show up in the search (ResumeBlobs is always empty). Here is the code used to create the Blobs Indexer:

var blobDataSource = DataSource.AzureBlobStorage(
                name: "azure-blob-test02",
                storageConnectionString: "DefaultEndpointsProtocol=https;AccountName=yyy;AccountKey=xxx;EndpointSuffix=core.windows.net",
                containerName: "2019");

            await searchService.DataSources.CreateOrUpdateAsync(blobDataSource);

            List<FieldMapping> map = new List<FieldMapping> {
                new FieldMapping("ResumeCandidateId", "CandidateId")
            };

            Indexer blobIndexer = new Indexer(
                name: "hotel-rooms-blobs-indexer",
                dataSourceName: blobDataSource.Name,
                targetIndexName: indexName,
                fieldMappings: map,
                //parameters: new IndexingParameters().SetBlobExtractionMode(BlobExtractionMode.ContentAndMetadata).IndexFileNameExtensions(".DOC", ".DOCX", ".PDF", ".HTML", ".HTM"),
                schedule: new IndexingSchedule(TimeSpan.FromDays(1)));

            bool exists = await searchService.Indexers.ExistsAsync(blobIndexer.Name);
            if (exists)
            {
                await searchService.Indexers.ResetAsync(blobIndexer.Name);
            }
            await searchService.Indexers.CreateOrUpdateAsync(blobIndexer);

            try
            {
                await searchService.Indexers.RunAsync(blobIndexer.Name);
            }
            catch (CloudException e) when (e.Response.StatusCode == (HttpStatusCode)429)
            {
                Console.WriteLine("Failed to run indexer: {0}", e.Response.Content);
            }

I commented the parameters for the blobIndexer but I get the same results even if it's not commented.

When I run a search, here is an example of what I get:

{
    "@odata.context": "https://yyy.search.windows.net/indexes('index-test01')/$metadata#docs(*)",
    "value": [
        {
            "@search.score": 1.2127206,
            "candidateId": "363933d1-7e81-4ed2-b82e-d7496d98db50",
            "lastName": "LAMLAST",
            "firstName": "ZFIRST",
            "notes": "MGA ; SQL ; T-SQL",
            "resumeBlobs": []
        }
    ]
}

"resumeBlobs" is empty. Any idea how to do such a mapping?

Answer 1

AFAIK, Azure Search doesn't support a collection merge feature that seems to be necessary to implement your scenario.

An alternative approach to this is to create a separate index for resumes and point the resume indexer to that index. That means that some of your search scenarios will have to hit two indexes, but it's a path forward.