Downloading from S3 with aws-cli using filter on specific File name

Question

I have data structure something like

s3://bucket/folder1/folder11/mylog.log
s3://bucket/folder1/folder12/mylog.log
s3://bucket/folder2/folder21/folder211/mylog.log
s3://bucket/folder3/folder31/mylog.log
s3://bucket/folder4/mylog.log
....

something like that

How can I download every file named mylog.log

Answer 1

You can filter by folders. You check whether the path ends with "/mylog.log", if the condition is met, then you can download the file

const bucketName = 'bucket';
const prefix = 'folder';

const s3 = new AWS.S3();

const listObjectsParams = {
  Bucket: bucketName,
  Prefix: prefix
};

s3.listObjectsV2(listObjectsParams, (err, data) => {
    if (err) {
      console.error('Error listing objects:', err);
      return;
    }

const objects = data.Contents;

for (const object of objects) {
     // Filtering your files
     const key = object.Key;
     if (key.endsWith('/mylog.log')) {
       const downloadParams = {
         Bucket: bucketName,
         Key: key
       };


      // AWS doc for Downloading an object*
      try {
       const response = await client.send(downloadParams);
       // The Body object also has 'transformToByteArray' and 'transformToWebStream' methods.
       const str = await response.Body.transformToString();
       console.log(str);
      } catch (err) {
        console.error(err);
      }
}

}

*AWS doc for Downloading an object