Can we prevent deadlocks and timeouts on ReliableQueue's in Service Fabric?

Question

We have a stateful service in Service Fabric with both a RunAsync method and a couple of service calls.

One service calls allows to enqueue something in a ReliableQueue

using(ITransaction tx = StateManager.CreateTransaction())
{
  await queue.EnqueueAsync(tx, message);
  queueLength = await queue.GetCountAsync(tx);
  await tx.CommitAsync();
}

The RunAsync on the other hand tries to dequeue things:

using(ITransaction tx = StateManager.CreateTransaction())
{
  await queue.TryDequeueAsync(tx);
  queueLength = await queue.GetCountAsync(tx);
  await tx.CommitAsync();
}

The GetCountAsync seems to cause deadlocks, because the two transactions block each other. Would it help if we would switch the order: so first counting and then the dequeue/enqueue?

Answer 1

This is likely due to the fact that the ReliableQueue today is strict FIFO and allows only one reader or writer at a time. You're probably not seeing deadlocks, you're seeing timeouts (please correct me if that is not the case). There's no real way to prevent the timeouts other than to:

• Ensure that the transactions are not long lived - any longer than you need and you're blocking other work on the queue.
• Increase the default transaction timeout (the default is 4 seconds, you can pass in a different value)

Reordering things shouldn't cause any change.