Search code examples
.netwcfmsmqmessage-queuetransactionscope

NetMsmqBinding: messages are being processed multiple times

My team has a windows service which host a wcf service using http binding. When the wcf service receives a message, it immediately sends it to another wcf service, using NetMsmqBinding, also hosted in a windows service. The msmq service process the message and save it do our DB.

In the development and staging environments, everything worked fine. When we deployed to production though, we saw that messages sent to our service are being multiplied by a factor of n+1 (for n messages).

For example: 1 message received, 2 saved to DB. 4 messages received, 20 saved to DB etc.

We suspect that it has something to do with the msmq service settings of Transactions,

Concurrency, and InstanceContextMode, but we can't figure it out.

The settings of the service are:

  • InstanceContextMode: Default.
  • ConcurrenyMode: Default
  • Transaction: TransactionScoreRequired = True and TransactionAutoComplete = True.
  • RetryCount = 1, RetryCycles = 1

We use .Net 4, MSMQ 3.0 and Windows server 2003.

Any ideas to what might cause this issue?

Edit:

After a lot of research, we have discovered the following things:

  • When the wcf runtime tries to commit the transaction for a single message, it throws an exception of "Transaction was aborted asynchronously". This exception is thrown each time a message is duplicated.
  • In the wcf trace, we don't see any exception other then the one stated above.
  • In the System.Transaction trace, we see a transaction is being opened (for the 1st message), and then after the wcf runtime creates a clone of that transaction (with the property RollbackIfNotCompleted=true), a few seconds pass without no message in the trace and then the same thing happens again.

We really are clueless...

Thanks

Solution

  • Well, we finally got it: we use Entlib for logging, and we added a DB listener. Entlib DB listener uses the same transaction of the service. When it tries to pass the transaction to the DB server, an error is thrown because MSDTC wasn't configured correctly on the DB server, which causes the transaction to rollback. When the WCF runtime tries to commit the transaction, it fails because it was already aborted, and then it tries to process the same message again (because of the built-in retry mechanism).

    When we fixed the MSDTC settings, the duplication stopped.
    Now, all we need to figure out is: why rollback to the service transaction causes messages to be multiplied so many times?