Copy data every 1 minute from DataLake by DataFactory

Question

I have a Data Lake storage with the following folder structure:

{YEAR}
 - {MONTH}
  - {DAY}
   - {HOUR}
     - {sometext}_{YEAR}_{MONTH}_{DAY}_{HOUR}_{Minute}_{someuuid}.json

example

Could you please help me to configure Data Factory Copy data action? I need to run Trigger every 1 minute - to copy data from Data Lake by previous minute to Cosmos DB I've tried this where the first expresion is

@formatDateTime(utcnow(),'yyyy/MM/dd/HH')

and the second one

@{formatDateTime(utcnow(),'yyyy')}_@{formatDateTime(utcnow(),'MM')}_@{formatDateTime(utcnow(),'dd')}_@{formatDateTime(utcnow(),'HH')}_@{formatDateTime(addMinutes(utcnow(), -1),'mm')}*.json

But it can skip some data, especially when Hour changes. I'm a new in Data Factory and don't know what is the more efficient way how to do that. Please help

Answer 1

The Pipeline Expression Language has a number of Date functions built in. You can use the addMinutes function to add 1 minute.

To avoid clock skew, I would capture the utcnow() value and store it without any formatting:

In another variable, add a minute to the captured value rather than executing utcnow() again:

Once you have those variables, just use them to format the date string(s).

Result:

NOTE: use concat with the formatDateString to get the wildcard value you want:

Result: