Search code examples
concurrencynservicebusazure-table-storagenservicebus4

Particular NServiceBus Sagas: Concurrent Access to Saga Data Persisted in Azure Table Storage

This question is in regards to concurrent access to saga data, when saga data is persisted in Azure Table Storage. It is also references information found in Particular's documentation: http://docs.particular.net/nservicebus/nservicebus-sagas-and-concurrency

We've noticed that, within a single saga executing handlers concurrently, modifications to saga data appear to be operating in a "last one to post changes to azure table storage wins" scenario. Is this intended behavior when using NSB in conjunction with Azure Table Storage as the Saga data persistence layer?

Example:

  1. Integer property in Saga Data, assume it currently = 5
  2. 5 commands are handled by 5 instances of the same handler in this saga
  3. Each command handler decrements the integer property in saga data
  4. Final value of the integer property in saga data could actually be 4 after handling these 5 messages - if each message was handled by a new instance of the saga, potentially on different servers, each having a copy of saga data indicating the integer property is 5, decrement it to 4, and post back up. What I just described was the extremely concurrent example, however it is likely the integer will be greater than 0 if any of the 5 messages were handled concurrently, the only time the saga data integer property reaches 0 is when the 5 commands happen to have executing serially.

Also, as Azure Table Storage supports Optimistic Concurrency, is it possible to enable the use of of this feature for Table Storage just as it is enabled for RavenDB when Raven is used as the persistence tech?

If this is not possible, what is the recommended approach for handling this? Currently we are subscribing to the paradigm that any handler in a saga that could ever potentially be handling multiple messages concurrently is not allowed to modify saga data, meaning our coordination of saga message is being accomplished via means external to the saga rather than using Saga Data as we'd initially intended.

Solution

  • After working with Particular support - the symptoms described above ended up being an defect in NServiceBus.Azure. This issue has been patched by Particular in NServiceBus.Azure 5.3.11 and 6.2+. I can personally confirm that updating to 5.3.11 resolved our issues.

    For reference, a tell-tale sign of this issue manifesting itself is the following exception getting thrown and not getting handled.

    Failed to process message Microsoft.WindowsAzure.Storage.StorageException: Unexpected response code for operation : 0

    The details of the exception will indicate that "UpdateConditionNotSatisfied" - referring to the optimistic concurrency check.

    Thanks to Yves Goeleven and Sean Feldman from Particular for diagnosing and resolving this issue.