Given Event-A, Event-B, Event-C that arrive (potentially out of order) within days of each other, I want to trigger processing to generate derivative Event-ABC once I know I have all events in the set.
The events are grouped by userId/sessionId
Currently I read all events from a single queue, write to database, and update metadata saying which events have been written. Once the metadata contains all events based on the rule, I trigger aggregation processing. This approach has some performance issues due to queue workers potentially hammering the same key when processing events that belong to the same group, so I am looking for alternatives.
What I would like is a more fine grained software defined routing and queueing events based on their userId/sessionId for processing. I think what I am trying to do is somewhat similar to event sourcing.
I was looking at whether Akka could help with this type of problem. With an actor per userId/sessionId it would reduce unneeded concurrency and contain trigger logic within the actor. My concern is the potentially large memory requirements when using so many Actors.
What you're describing is more akin to a Saga or Process Manager than Event Sourcing. You need something that handles multiple messages and then reacts once a specification has been satisfied.
Akka can certainly cope with this. With Akka, you could create an actor per key and then route messages to individual actors when you receive them. I wouldn't be too concerned about memory issues as Actor systems are supposed to cope with thousands and thousands of Actors. I think you need to measure the performance of any solution you arrive at though.
You also need to consider how you deal with the servers crashing - if you keep everything in memory then you are susceptible to losing your sagas when the servers crash. This may or may not be a problem depending on your requirements (i.e. if you can recover from this). You could look into Akka Persistence if it is important to account for this.