aws s3 sync command intermittently fails midway without error

Question

Trying to sync buckets (one s3 bucket to other s3 bucket) in AWS S3 through Jenkins job command:

aws s3 sync s3://bucketname1 s3://bucketname2 --exclude *snaps/**** --exclude '*.html' --no-progress

Problem: The aws s3 sync command intermittently fails midway without error. Meaning, all the files are not getting synced to the destination bucket.

Using below version:

    aws-cli version:
        aws-cli/2.11.23 Python/3.11.3 Linux/5.4.0-1039-aws exe/x86_64.ubuntu.20 prompt/off
    Jenkins version: 
        Jenkins 2.361.4

Observations:

• After printing around 4k lines to the stdout, sync is exiting with non-zero status code
• We enabled --debug flag with the above AWS CLI command and did not find any errors or hints about this abrupt stopping of the command
• The same commands works without any issues when we run this from local laptop.

Version using in local laptop:

aws-cli/2.7.16 Python/3.9.11 Darwin/21.5.0 exe/x86_64 prompt/off

Answer 1

Object size might be the reason for faliure in sync operation. As per AWS documentation, the limit is 5 GB per object. If that's the case, you may have to look for multi-part upload.