Second level retries in Rebus with Rabbitmq

Question

I have a scenario where I call an api in one of my handlers and that Api can go down for like 6 hours per month. Therefore, I designed a retry logic with 1sec retry, 1 minute retry and a 6 hour retry. This all works fine but then I found that long delay retries are not a good option.Could you please give me your experience about this?

Thank you!

Answer 1

If I were you, I would use Rebus' ability to defer messages to the future to implement this functionality.

You will need to track the number of failed delivery attempts manually though, by attaching and updating headers on the deferred message.

Something like this should do the trick:

public class YourHandler : IHandleMessages<MakeExternalApiCall>
{
    const string DeliveryAttemptHeaderKey = "delivery-attempt";

    public YourHandler(IMessageContext context, IBus bus)
    {
        _context = context;
        _bus = bus;
    }   

    public async Task Handle(MakeExternalApiCall message)
    {
        try
        {
            await MakeCallToExternalWebApi();
        }
        catch(Exception exception)
        {
            var deliveryAttempt = GetDeliveryAttempt();

            if (deliveryAttempt > 5)
            {
                await _bus.Advanced.TransportMessage.Forward("error");
            }
            else
            {
                var delay = GetNextDelay(deliveryAttempt);
                var headers = new Dictionary<string, string> {
                    {DeliveryAttemptHeaderKey, (deliveryAttempt+1).ToString()}
                };

                await bus.Defer(delay.Value, message, headers);
            }
        }    
    }

    int GetDeliveryAttempt() => _context.Headers.TryGetValue(DeliveryAttemptHeaderKey, out var deliveryAttempt)
        ? deliveryAttempt
        : 0;

    TimeSpan GetNextDelay() => ...
}

When running in production, please remember to configure some kind of persistent subscription storage – e.g. SQL Server – otherwise, your deferred messages will be lost in the event of a restart.

You can configure it like this (after having installed the Rebus.SqlServer package):

Configure.With(...)
    .(...)
    .Timeouts(t => t.StoreInSqlServer(...))
    .Start();