How to handle unavailable services using service remoting

Question

I'm using service remoting to connect service A with service B. While deploying, I've noticed that it is possible for service A to call a method in service B while service B is still starting up. In which case AddStringToDictionary throws a NullReferenceException because RunAsync has not been called yet.

public interface IServiceB : IService
{
    Task AddStringToDictionary(string key, string value);
}

internal sealed class ServiceB : StatefulService, IServiceB
{
    private IReliableDictionary<string, string> myDictionary;

    public ServiceB(StatefulServiceContext context)
        : base(context)
    { 
        // StateMananger is null here.
    }

    public async Task AddStringToDictionary(string key, string value) 
    {
        using (var tx = StateManager.CreateTransaction())
        {
            // myDictionary is null if this is called before RunAsync.
            await this.myDictionary.SetAsync(tx, key, value);
            await tx.CommitAsync();
        }
    }

    protected override async Task RunAsync(CancellationToken cancellationToken) 
    {
        this.myDictionary = await this.StateManager.GetOrAddAsync<IReliableDictionary<string, string>>("myDictionary");
    }
}

I know that I can use the StateManager.GetOrAddAsync method in AddStringToDictionary, but I might want to initialize some more fields. The constructor is not useful either because the StateManager doesn't exist there yet. My current solution is to add a field:

private bool serviceAvailable = false;

I'm setting this field to true at the end of RunAsync. Every public method then calls ThrowIfServiceUnavailable().

private void ThrowIfServiceUnavailable()
{
    if (!serviceAvailable)
    {
        throw new ServiceUnavailableException();
    }
}

public async Task AddStringToDictionary(string key, string value) 
{
    ThrowIfServiceUnavailable();
    ...
}

Currently I have to handle the ServiceUnavailableException every time I call a method with remoting. My question is: are there any better ways to handle this situation or is this a good solution? And is it possible to add my custom exception to the ServiceProxy's transient exceptions to retry?

Answer 1

Here's an implementation that blocks calls while the service is being initialized. It uses ManualResetEventSlim to block. And it uses SemaphoreSlim to enforce single threaded access to an async operation.

All Service operations call await WaitForInitializeAsync(CancellationToken.None); And the implementation of this method is:

private async Task WaitForInitializeAsync(CancellationToken cancellationToken)
{
    if (_initializer.IsSet) return;
    await Task.Run(() => InitializeAsync(cancellationToken), cancellationToken);
   _initializer.Wait(cancellationToken);
}

The implementation of InitializeAsync:

private async Task InitializeAsync(CancellationToken cancellationToken)
{
   if (_initializer.IsSet) return;
   try
   {
      _semaphore.Wait(cancellationToken);
      if (_initializer.IsSet) return;
      [initializer logic here]