Wait for ElasticSearch to be up and running

Question

I have a C# WebAPI project and I'm using NEST to connect to a local ElasticSearch database. The WebAPI and ElasticSearch are running as two separate Windows services. One problem I noticed is that the ES service has to boot up first, before the WebAPI one, otherwise the WebAPI may fail trying to initialize a non available database.

The problem has been partially solved by enforcing a dependency in the Windows service option, so Windows will start ES first, then the WebAPI. The problem doesn't go away completely as ES may take a long time before actually being ready, and the WebAPI service may fail anyway.

Is there any option/timeout/retry I can set on the NEST library to just wait before issuing the very first request to the database? Do I need to go DIY and code my own retry loop?

Answer 1

There's nothing built-in to solve this, but you can likely use something like Polly, with its retry policies, to overcome your issue.

Policy
    .Handle<YourNESTExceptionHere>()
    .WaitAndRetryForever(retryAttempt => TimeSpan.FromSeconds(1));
    .Execute(() =>
    {
        // code that raises the exception
    });

This will continuously try to run the code, waiting 1 second between each attempt, until it completes without raising an exception. You can change the time between retries, obviously, to whatever makes sense in your scenario. This will essentially skirt around the issue of your Elasticsearch cluster not being immediately available by swallowing the exception and just retrying until it's up.

There's other policies Polly provides that you could use as well. For example, you might want to use a circuit breaker, so it will eventually quit and stop further attempts for a period of time if there's something wrong with the service or such, preventing it from ever working. You should go through the Polly readme and understand your different options. The long and short, is that you simply need some way to catch the exception and retry.