Search code examples
pythonpipazure-synapseazure-synapse-pipelinemagic-command

Azure Synapse: How to get notebook magic command %pip to work in a pipeline

When running %pip install in my Synapse notebook manually, it works as intended, without any problems.

However, when I try to run that same notebook in a pipeline, I get:

{
    "errorCode": "6002",
    "message": "MagicUsageError: %pip magic command is disabled.",
    "failureType": "UserError",
    "target": <my_notebook_name>,
    "details": []
}

I have looked at these two posts:

Why magic command doesn't work when running pipeline?

Synapse : Execute the Magic Command (%run) Notebook from Pipeline

But these are both only concerned with the %run command, not %pip

The solution to both of those posts is to add an absolute path, which makes sense in the context of a %run command, but I'm not sure how to translate adding an absolute path to %pip? Or if there is something else that can make this work?

Solution

  • I have received same ERROR when using the %pip command and running pipeline. enter image description here

    I have tried the below approach:

    I have created a requirements.txt file for the dependencies that need to be installed.

    For example I have created a dependency for splink==1.0.3

    I have uploaded the requirements.txt

    Go to Manage tab > Spark Pool enter image description here

    enter image description here

    enter image description here

    Now I ran the notebook to verify the if the version of the dependency is correct?

    enter image description here

    Next, I ran the pipeline.

    Results:

    enter image description here