Event sourcing and microservices, messaging system
We are designing our new system which will be written most likely from scratch, as the old one is really really old. It's really important for our system to keep audit trail logs of everything that happened in the system.
Due to the audit trail importance we decided to follow the event sourcing architecture for all of its benefits. Another key factor is that we have multiple teams who work on different "domains". Saying that, we would like to split each domain into it's own service (microservices architecture) so every team can work independently.
The biggest question that we have is who's going to be responsible for the event sharing between the microservices. For example, an event that happened in one of the services, should trigger another event in 2 other services as well. We can see 2 options:
- A central streaming service (AWS Kinesis most likely) with all services registering a publisher and a subscriber. Every service will publish all its events there and the consumers will decide if they "care" about the specific event and then do something or not.
- Multiple categorised streams (AWS Kinesis most likely) with specific service publishers and subscribers that really need to do something with the events that are published.
We've spent a lot of hours already trying to figure out what is the best option for us or what questions we should ask ourselves to take the right decision. I'm leaning more to the first option although i do have some concerns regarding performance and bottleneck for the consumers, since every service will listen to all events and then they will decide if they have to act on it or not.
Do you see any no-goes in the first option or the second one? Are we looking in the right direction or should we take a step back?
An image that describes a bit what we need to achieve (except CQRS) in a short scale is this one 
which is taken from here
I would tend to have a stream of events per publishing service. Then you have some level of categorization and don't have to deal (or at least not deal as much) with synchronizing consumption of multiple streams when you want to process events in order of emission.
The events will often have a natural key and you also often only need strict ordering after grouping by that key; accordingly, you can partition/shard the events.
There's nothing intrinsically preventing you from having both a firehose stream of all the events emitted by a publishing service and categorized streams of events so consumers can decide to not subscribe to streams they're not interested in.
- anti-if campaign
- In SOLID, what is the distinction between SRP and ISP? (Single Responsibility Principle and Interface Segregation Principle)
- Python Design Pattern Logger Class
- What is the Facade design pattern?
- Cross module communication in modular monolith
- REST API design: one endpoint with if/else logic or two separate role based endpoints
- AWS Step Function - Wait until a group of other Step Functions have finished then fire a different Step Function
- How to implement SOLID principles into an existing project
- Is this a bad use of a static property?
- Groovy loop through Pattern matches
- Example of Builder Pattern in Java API?
- Factory Method: Prevent a class from Direct Instancing
- Name of design pattern creating the same object for equal arguments
- If View Model property has value with certain length, replace it with query from database.Do it in Controller, Validation Attribute or somewhere else?
- Singleton Properties
- How to "mark" a function-like trait to be optionally executed in parallel?
- Angular Service Injection in reusable components
- What are MVP and MVC and what is the difference?
- Design Pattern HTTP API
- Deriving from a concrete class?
- Does javascript singleton require double parentheses?
- Strategy Pattern with Different parameters in interface (C#)
- Using abstract classes from another assembly in .NET
- DDD Read-Only Repositories returning "value objects"
- What is the difference between Builder Design pattern and Factory Design pattern?
- How to have a model Manager in Symfony2?
- design pattern with Python -> abstract factory
- Architecture design for divide the subsystems
- Returning a "NULL reference" in C++?
- Decorator functions in Go