Search code examples
azureazure-data-factoryazure-data-lakedataflow

Passing timestamp to azure mapping data flow

I'm trying to pass last modified date and time to my data flow as parameter. Can anyone tell me what is a correct way to pass it as parameter. I've tried multiple things like. passing utcnow() from activity throws error saying file not found whereas passing directly from dataflow works fine. I figured out using dataflow expression works fine for ucnow() whereas pipeline expressions fails.

Solution

  • The Pipeline expression language is a different and a little more limited than the Data Flow expression language. While Data Flow supports a richer variable type system, Pipelines only support String, Boolean, and Array types. Since there is no Date or Timestamp types, the date functions in the Pipeline expression language return strings:

    enter image description here

    If you want to use the UTC value from the pipeline instead of the data flow, you will need define a string parameter on the Data Flow: enter image description here

    Then pass the string of utcnow() to the Data Flow as a Pipeline Expression: enter image description here

    In the expression, use the utcnow() function to get the string value: enter image description here

    In the Data Flow, use Derived Column to convert it to the desired type: enter image description here