wildcard and dynamic filename on datafactory read file

Question

ok I have to copy from a remote storage account a daily file. this file is generated with format name: File20230515063915.TXT meaning: "the word File"+year+month+date+hour, etc. everything on my time (+5)

The thing is, the storage account have several dates, so I need to copy just today date file. (today as local time, not utc) and the part of hour, minutes, seconds, there is no way for me to calculate it as string, since it vary always.

My solution was to create a variable that can store up to the day part, and use wildcard (*) for the rest of the file name.

but in this point I am confused about the Pipeline expression builder to calculate and where/how to set the wildcard, if on the filepath for the dataset or on the pipeline itself, and how.

I really appreciate if somebody could help me on this.

an alternative solution could be to list the files on the storage account, detect new files comparing with previus run, and use the new file names only. but I think that will require more work.

dynamic content pipeline expression example.. If that fix the issue. or a way to workaround the issue.

Answer 1

You can give the date in the wild card file name.

These are my files in the source folder:

First generate the yyyyMMdd format filename using a set variable activity with the below expression.

@formatDateTime(utcnow(),'yyyyMMdd')

Then, use this in the copy activity source wild card file name. Give the below expression in the file name.

File@{variables('date')}*.TXT

Give your target file in sink and this is my result.