Is there a way for AWS s3 source in pipeline to detect actual file changes?

Question

I have configured an s3 bucket as a source for my pipeline.

But whenever I upload the files, the pipeline triggers, regardless if the newly updated file is exactly the same/no change.

I was wondering if there's a configuration that will only trigger the pipeline to detect actual change of the file.

Answer 1

The put actually occurs, so from S3s point of view, the object has changed. So the pipeline will fire, even if the new file happens to be identical to the old file. After all, you will also see a previous object version added to the version history, etc.

If you control the uploading process, I would suggest first fetching the object metadata, and only putting an updated version if it actually mismatches (eg. a different MD5 hash). This makes sense since it also saves you the upload itself.