Search code examples
u-sqlazure-data-factory

Input file path with dates ADFV2

Currently we are moving out data analytics pipeline to ADFV2. In V1 we defined the input filepath in the parameter file as - /Inputpath/{filedate:yyyy}.{filedate:MM}.{filedate:dd}/RealtimeUsage.{filedate:yyyy}.{filedate:MM}.{filedate:dd}.{filedate:HH}.{*}csv

How is this accomplished in ADFV2. The activity reads in the parameter from the parameter file as such - 

"ActivityName": {
"path": "$.properties.activities[?(@.name=='ActivityName')].typeProperties.ParameterName",
          "value": "/Input/{filedate:yyyy}.{filedate:MM}.{filedate:dd}/RealtimeUsage.{filedate:yyyy}.{filedate:MM}.{filedate:dd}.{filedate:HH}.{*}csv"
Solution

  • If you want to read a file in the following pattern: Inputpath/2018.07.26/RealtimeUsage.2018.07.26.01.{*}csv

    You could write your file path expression in the following:

    "fileName": {
type: "Expression",
value: "RealtimeUsage.@{formatDateTime(pipeline().parameters.windowStart,'yyyy')}.@{formatDateTime(pipeline().parameters.windowStart,'MM')}.@{formatDateTime(pipeline().parameters.windowStart,'dd')}.@{formatDateTime(pipeline().parameters.windowStart,'HH')}.{*}csv",
}

    You could also use the copy data tool to help you generate this expression. enter image description here