Search code examples
amazon-web-servicesamazon-s3amazon-data-pipeline

AWS CLI moving file with wildcard (asterisk) in path

I am attempting to move a file, from on s3 location to another, using an activity in a AWS data pipeline.

The command I am using is:

(aws s3 mv s3://foobar/Tagger/out//*/lastImage.txt s3://foobar/Tagger/testInput/lastImage.txt)

But I receive the following error:

A client error (404) occurred when calling the HeadObject operation: Key "Tagger/out//*/lastImage.txt" does not exist

But, if I replace the "*" with the specific directory name, it will work. The problem is I won't always know the name of the directory, so I was hoping I could use the "*" as a wild card.

Solution

  • Wildcards in the AWS S3 CLI only work when using the --recursive flag.

    So this should work for you:

    aws s3 mv s3://foobar/Tagger/out/ s3://foobar/Tagger/testInput/ --recursive --exclude "*" --include "*/lastImage.txt"

    Unfortunately, this will recreate the entire directory structure in your target location, and I'm not immediately sure that can be solved by just using the AWS CLI.