wait for several different events and then only do a processing once (no periodic pooling or jobs or etc)

Question

For mobile app backend there are 2 API GWs:

1. Rest API GW for file upload to S3 bucket
2. Http API GW for JSON message communication (these messages are going to lambdas -> DynamoDB / SQS)

Clients are mobile apps which perform business processes. Each business process could have few steps and there could be a significant pause between steps (hours), especially in worst case scenario when users can't accomplish their duties because of environment/assets at hand etc.

Each step of process on mobile produce 1 or several messages to API GWs, for instance

1. http file uploads to server, there could be multiple uploads simultaneously. This is very IMPORTANT
2. JSON messages (step A complete, B step started etc).

According to logic, in some cases backend should accumulate several events from both API GWs and only then start processing of uploaded data (for example, wait until 3 files are uploaded and JSON with user entry is received).

If server didn't get all data (all files + all JSONs) it should not start any processing. Currently it is a very primitive code which is writing into persisitence layer (Dynamo) something like:

1. JSON captured
2. file 1 stored in S3
3. file 2 stored in S3 ...
4. file 5 stored in S3

Upon each write into DynamoDB it checks if all pre-conditions are satisfied and they are - starts processing of an accumulated data. What's bad is that this business logic state management is inside Dynamo, I want to actually aggregate events somehow in the queue(s) and only fire execution once all events are accumulated in the queues. This would allow to get rid of Dynamo persistence for this part of app logic.

I'm considering either EventBridge or StepsFunctions, but not sure if it is worth and if it won't bring more problems like more expensive than DynamoDB stack. What would you say, is there any elegant and inexpensive solution to such event aggregation or better to leave it as it is?

Answer 1

"What's bad is that this business logic state management is inside Dynamo, I want to actually aggregate events somehow in the queue(s) and only fire execution once all events are accumulated in the queues."

DynamoDB is the correct AWS service to use for this process. DynamoDB's features are exactly what you need to track multiple events and know when to trigger your process. I don't understand why you think this is "bad".

I want to actually aggregate events somehow in the queue(s) and only fire execution once all events are accumulated in the queues. This would allow to get rid of Dynamo persistence for this part of app logic.

There is no queue system that I'm aware of that would work this way. You need a single database record with multiple attributes in order to track the multiple states of multiple events in one place.

I'm considering either EventBridge or StepsFunctions

I don't see how either of those would replace DynamoDB here. EventBridge certainly won't work for your use case, because it has no state tracking across multiple events at all.

It sounds like you are thinking about calling StepFunctions StartExecution for each event that comes in, and somehow have StepFucntions track when all the different events have come in before it actually starts, but StepFunctions doesn't work that way.