How to achieve message deduplication when using MassTransit with postgres transport and EF Core for persistence

Question

How do I make sure that there are no duplicate queue messages when they have a common correlation ID? How do I ignore the publishing of a queue message when there is already one on the queue with the same correlation ID?

This is needed to keep the queues clean. For example: Amazon SQS has a Message deduplication ID

Does MassTransit also have this functionality when using a postgres database?

I tried creating messages with the same correlation ID but duplicate ones are still added to the queue.

Answer 1

There is no deduplication ID in the SQL transport.

A message store should not be used as a database. Relying on the transport to deduplicate data is a bad idea in general. You should have an idempotent domain that can deal with duplicates.

Create a saga with a unique property (your de-duplication value) and ensure that any duplicate messages are properly handled. The saga repository (EF Core? it's a database table) then becomes your deduplicated data store – not the queue.