Search code examples
cqrsevent-driven-design

Handling DB Failure during projection in cqrs


We are creating system using CQRS. Our projections are in mongodb. We are facing some cases. We have an event say OrderCreated. We need to produce a sequential order_no for example #3, #4 etc. We could use a projection and keep a sequence in table then called upsert method. and get a new number. Post a new command : GenerateOrderNumber. now before this post accepted hardware failure occur. If we retry we will have another number. It is not good. How to solve such use case in cqrs.


Solution

  • Our projections are in mongodb <...>
    now before this post accepted hardware failure occur

    Most likely that described issue is not about CQRS or EventSoucring itself, but related to projection storage, which is MongoDB in question above.

    You are trying to perform potential atomic operation without transaction guarantees. Since hardware failure can be caused within random time, database should provide ability for rollbacking failed atomic operations in current transaction.

    Best choice is native MongoDB transactions, which are available since 4.0 version - https://docs.mongodb.com/manual/core/transactions/ - and your code will look like this:

    session.startTransaction( … );
    try {
      const lastNo = await eventsCollection.findOne( ... )
      await eventsCollection.insertOne( …, lastNo +1 )
      session.commitTransaction()
    } catch (error) {
      session.abortTransaction()
    }
    

    If you have to use older MongoDB versions, transactions still can be used. But instead of using intrinsic operator, you should manually write transaction log, and after reconnect to database perform monitoring for broken transactions and revert them manually via log.