Search code examples
azure-data-factoryexpressionbuilder

An easy way of running code snippets written in Azure Data Flow script / Expression Builder?

Some context : When you use Azure Data Factory, you can create "Data Flows" which in turn contain graphical logic bricks, which themselves can transform data with an embedded code editor (the "Dataflow expression builder") that lets you execute scripts written in a native language called Azure Data Flows script (see picture).

enter image description here

Questions:

  1. Is that the real name of this language? Does it have a more common name? Is this language the twin brother of another, more widely adopted language?

  2. Is there an easy way of executing code snippets of it? Nowadays you can execute C# or many other languages directly in the web browser, with sites such as "Fiddle" and whatnot. Is there a place where I can test simple scripts of this scripting language without all the hassle of setting up Azure Data Flows, which are unbearably unreadable and unbearably slow?

Note: My specific use case is that I quickly want to test some regExp and text replacement, but even the Debug mode is excruciatingly slow, it takes several minutes to run the flow after each change.

Solution

  • The language is the Data Flow Expression language. Data Flows are executed as Spark Notebooks at run time, so the code is ultimately converted to Scala.

    Since the conversion is handled internally, there is no way for you to interject custom code snippets. Synapse can more readily support custom notebooks than Data Factory.