In one of my Argo workflow steps a Docker container splits up a large file into a number of smaller files.
The tutorials show how one can save a small and pre-determined number of outputs (e.g., 2 or 3) as artifacts in an S3 bucket by going through each output one at a time.
In my use case, I do not know in advance how many smaller files will be created; it can be upwards of hundreds. The large number of output files makes it hard, if not impossible to follow the tutorials and specify each one by one even if I know how many smaller files are create in advance.
Is there a way to save all the outputs to an S3 bucket?
This sounds like standard output artifacts. You can put all your files in a single directory, and then have the directory be the output artifacts.
Here are some examples to help you:
https://argoproj.github.io/argo-workflows/examples/#artifacts