The lock supplied is invalid. Either the lock expired, or the message has already been removed from the queue

Question

I'm using a Microsoft Azure Service Bus queue to process calculations and my program runs fine for a few hours but then I start to get this exception for every message that I process from then on. I have no clue where to start since everything runs fine for the first few hours. My code seems to be accurate as well. This the method where I handle the Azure Service Bus message.

public static async Task processCalculations(BrokeredMessage message)
{
    try
    {
        if (message != null)
        {
            if (connection == null || !connection.IsConnected)
            {
                connection = await ConnectionMultiplexer.ConnectAsync("connection,SyncTimeout=10000,ConnectTimeout=10000");
                //connection = ConnectionMultiplexer.Connect("connection,SyncTimeout=10000,ConnectTimeout=10000");
            }
            cache = connection.GetDatabase();
            string sandpKey = message.Properties["sandp"].ToString();
            string dateKey = message.Properties["date"].ToString();
            string symbolclassKey = message.Properties["symbolclass"].ToString();
            string stockdataKey = message.Properties["stockdata"].ToString();
            string stockcomparedataKey = message.Properties["stockcomparedata"].ToString();
            var sandpTask = cache.GetAsync<List<StockData>>(sandpKey);
            var dateTask = cache.GetAsync<DateTime>(dateKey);
            var symbolinfoTask = cache.GetAsync<SymbolInfo>(symbolclassKey);
            var stockdataTask = cache.GetAsync<List<StockData>>(stockdataKey);
            var stockcomparedataTask = cache.GetAsync<List<StockMarketCompare>>(stockcomparedataKey);
            await Task.WhenAll(sandpTask, dateTask, symbolinfoTask,
                stockdataTask, stockcomparedataTask);
            List<StockData> sandp = sandpTask.Result;
            DateTime date = dateTask.Result;
            SymbolInfo symbolinfo = symbolinfoTask.Result;
            List<StockData> stockdata = stockdataTask.Result;
            List<StockMarketCompare> stockcomparedata = stockcomparedataTask.Result;
            StockRating rating = performCalculations(symbolinfo, date, sandp, stockdata, stockcomparedata);
            if (rating != null)
            {
                saveToTable(rating);
                if (message.LockedUntilUtc.Minute <= 1)
                {
                    await message.RenewLockAsync();
                }
                await message.CompleteAsync(); // getting exception here
            }
            else
            {
                Console.WriteLine("Message " + message.MessageId + " Completed!");
                await message.CompleteAsync();
            }
        }
    }
    catch (TimeoutException time)
    {
        Console.WriteLine(time.Message);
    }
    catch (MessageLockLostException locks)
    {
        Console.WriteLine(locks.Message);
    }
    catch (RedisConnectionException redis)
    {
        Console.WriteLine("Start the redis server service!");
    }
    catch (MessagingCommunicationException communication)
    {
        Console.WriteLine(communication.Message);
    }
    catch (Exception ex)
    {
        Console.WriteLine(ex.Message);
        Console.WriteLine(ex.StackTrace);
    }
}

I check the time until the lock expiration and I call lock renew if it needs it - it renews the lock with no errors, but I'm still getting this exception.

timeLeft = message.LockedUntilUtc - DateTime.UtcNow;
if (timeLeft.TotalMinutes <= 2)
{
    //Console.WriteLine("Renewed lock! " + ((TimeSpan)(message.LockedUntilUtc - DateTime.UtcNow)).TotalMinutes);
    message.RenewLock();
}

catch (MessageLockLostException locks)
{
    Console.WriteLine("Delivery Count: " + message.DeliveryCount);
    Console.WriteLine("Enqueued Time: " + message.EnqueuedTimeUtc);
    Console.WriteLine("Expires Time: " + message.ExpiresAtUtc);
    Console.WriteLine("Locked Until Time: " + message.LockedUntilUtc);
    Console.WriteLine("Scheduled Enqueue Time: " + message.ScheduledEnqueueTimeUtc);
    Console.WriteLine("Current Time: " + DateTime.UtcNow);
    Console.WriteLine("Time Left: " + timeLeft);
}

All I know so far is that my code runs fine for awhile, and the renew lock gets called and works but I'm still getting the lock exception and inside that exception.

I output the time left and it keeps increasing the time difference as the code runs, which makes me believe that the time until lock expiration is not being changed somehow?

Answer 1

I spent hours trying to understand why I was getting a MessageLockLostException. The reason for me was due to AutoComplete defaulting to true.

If you're going to call message.Complete() (or CompleteAsync()) then you should instantiate an OnMessageOptions object, set AutoComplete to false, and pass it into your OnMessage call.

var options = new OnMessageOptions();
options.AutoComplete = false;

client.OnMessage(processCalculations, options);