Search code examples
pythonazureazure-databricksazure-data-factory

Get runOutput from python activity (not notebook activity) in ADF V2

I need to receive the runOutput from python activity in ADF. In Notebook activity we can receive it using @ {activity ('databricks notebook activity name').output.runOutput}. But same thing when I am trying to do using python activuty, it is not working even though I am exiting notebook with dbutils.notebook.exit('my message).

Any help would be appreciated. Thanks.

Solution

  • I have reproduced the above and not able to get the runOutput of Python script like we get in Notebook activity.

    dbutils.notebook.exit('my message')

    I have used the above script in my .py file.

    enter image description here

    As per my repro, runOutput might not be there for python activity. You can try this workaround where I am able to get the Output from REST API job runs.

    First extract the particular run id of the python activity from the above output.

    @string(last(split(activity('Python1').output.runPageUrl,'/')))

    enter image description here

    Then use web activity to get the details of that particular run with Bearer token as authorization.

    enter image description here

    URL: https://< databricks-instance >/api/2.0/jobs/runs/get-output?run_id=@{variables('run_id')}

    For Header: give name as Authorization and for value give Bearer < your databricks bearer token>

    In the logs of the above web activity output, you can find the python activity output.

    @string(activity('Web1').output.logs)

    enter image description here

    Output:

    enter image description here