Search code examples
csvdelimiterazure-data-factoryparameterization

How to parameterise the delimiter for the Source in Azure Data Factory (V2) Data Flows?

I have a situation where I am using Data Flows to ingest data from Blob Storage. The file format is CSV, however some files are comma delimited and some are pipe delimited.

It is possible to parameterise the delimiter of the dataset itself in ADF.

enter image description here

However in the Dataflow activity the Source Settings do not provide the option to either dataset parameters or change the delimiter.

enter image description here

It's not even possible to parameterise the choice of dataset. This means that I have to create a new Dataflow for each delimiter which is not practical.

Solution

  • I tried this and it works well.

    Source dataset parameter setting, add parameter ColDelimiter:

    enter image description here

    Create a pipeline parameter ColDelimiter:

    enter image description here

    Then click the Data Flow, we can set the source parameter:

    enter image description here

    When run the pipeline, we could set the column delimiter from paramter: enter image description here

    Hope this helps.