Combine several events from event-bridge

Question

I have some work that needs two s3 objects. Object A is uploaded by another system; I have to generate the Object B.

In fact, there is not one Object A, but several (A1, A2, A3). Each one is uploaded by an external system at any time. For each object A, another instance of the work has to be launched.

On the other hand, Object B remains the same for a specified period time, after which I have to regenerate it. Generating the object takes time.

I can use EventBridge Scheduler to generate the object B, and I can also use event bridge to fire events for each Object Ax that gets uploaded.

My question is how do combine these two events, so that I can launch a job only after both Object B is generated, and Object Ax is uploaded, ensuring that for every object Ax that gets uploaded, exactly one job is launched.

(Something similar to Promise.all in javascript)

Answer 1

While the step functions approach above seems like the right solution generally for these kind of orchestration scenarios, I wonder if in this simple case it would be sufficient to have a single lambda listen to both S3 notifications and simply check if the other file already exists?

Something like:

exports.handler = async function (event, context) {
  const s3Object = extractS3Object(event);
  if (isSecondFile(s3Object) && firstFileExists() 
    || isFirstFile(s3Object) && secondFileExists()) {
    // Do stuff
  } else {
    // Don't do stuff
  }
};

Update: This doesn't work as great if you need a strong guarantee that job is only executed once, but it's still possible with a bit of a hack. You can:

• Set your lambda concurrency limit to 1
• Store IDs of successfully processed jobs in a DynamoDB table
• Check ID in the table before processing a job, and if found, skip.

However, I would first think if it's possible to make your job idempotent, so that you don't have to enforce "only-once" policy