Search code examples
c#elasticsearchnest

Index Dynamic Settings when performing bulk operation


I am trying to find a way to specify the number of replicas for my indices when performing a BulkAll operation, I know how to do this when creating an individual index with CreateIndex:

var createIndexResponse = client.CreateIndex("index-name", c => c
    // settings for the index
    .Settings(s => s
        .NumberOfReplicas(1)
    )

Second argument contains CreateIndexDescriptior which contains:

public CreateIndexDescriptor Settings(Func<IndexSettingsDescriptor, IPromise<IIndexSettings>> selector);

I looked at BulkIndexDescriptor but could not find the settings option, so how can I specify the number of replicas when performing a bulk operation?

Here is my current code:

var bulkAllObservable = elasticClient.BulkAll(accessLogs, b => b
                  .Index(null)
                  .MaxDegreeOfParallelism(Environment.ProcessorCount)
                  .BackOffRetries(2)
                  .ContinueAfterDroppedDocuments(true)
                  .BufferToBulk((descriptor, buffer) =>
                  {
                      foreach (var al in buffer)
                      {
                          descriptor.Index<object>(bi => bi
                              .Document(al)
                              .Index($"my-log-{al.Properties.UserId}")
                              .Id($"{al.Properties.Id}")
                          );
                      }
                  })

I am using Nest 7.10.0


Solution

  • Assuming that you're relying on an index being created when it does not exist and a document is indexed into it, you can use index templates to template the settings to use when the index is created

    var client = new ElasticClient();
    
    var putTemplateResponse = client.Indices.PutTemplate("my-log", t => t
        .IndexPatterns("my-log-*")
        .Settings(s => s
            .NumberOfReplicas(1)
        )
    );
    

    Now, when an index is created whose name matches the index pattern "my-log-*", it'll be created with one replica.