How can i pass the file name / path after "Copy data" step to Azure Functions in ADF

Question

In the current pipeline there is a "Copy data" step, which copies the files from a sftp server to a data lake. The second step would be to process the newly copied data with azure funtions. Thereby, it would be nice to pass the name of the file or file path to the azure function. So that it can read / open the file from the data lake and eventually store the processed data in Postgresql (inside a vnet).

For example, how could I embed the file name or file path inside the body?

There is another option using trigger on blob storage / data lake. I would slightly prefer to pass the path or the file name. Suggestions, ideas are appreciated.

Answer 1

Since you copy the files to data lake, you can utilize the storage event trigger to get the created file names and paths. You can then pass the parameters or variables storing them to the Azure functions in POST body JSON.

example:

1. create a trigger, optionally set blob path begin or end with fields to filter out specific files.

2. For the previously created parameters, get and store the file_name and file_path using @triggerBody().fileName and @triggerBody().folderPath respectively.

3. Pass the parameters in Azure Functions POST body