Search code examples
pysparkazure-data-factorymicrosoft-fabric

Passing SecureString Parameter Values in Fabric Notebooks

I'm having a lot of trouble using securestring parameters on my pipelines, as I need to pass some credencials to multiple notebooks in a pipeline as parameters and, as we all know, using plain text for passwords/keys is not secure at all, so I tried using the SecureString parameter type for pipeline parameters as the name sounded just like what I needed.

Now, when runing the pipeline I got no errors on the notebook, but when looking at the notebook run the autentication was failing, and further experimentation revealed that the notebook was geting literally '********' as the string value, not whatever I wrote in the parameter value box. Is there a dynamic content method I should be using the pass the secured value to a notebook, a variable type I should initialize the parameter as in the notebook (using PySpark, btw)?

Solution

  • I have tried the same scenario, and these are my observations.

    I have given a value Rakesh to the a secureString type parameter in the pipeline parameters section and ran the pipeline by passing it to two notebook parameters.

    It gave the same result.

    enter image description here

    The reason for this behavior is when we run the pipeline, it will again ask the parameter value even though if you have given a value in the parameter section. Here, it is considering the default secureString value **********.

    You can see the same when running the pipeline.

    enter image description here

    In the above picture, you can see the length difference between both values. This is the reason for the above behavior.

    When I have changed the value during this time to original value Rakesh and clicked on ok, it took the original value instead of default value.

    enter image description here

    This might be a bug because the same scenario is working fine in synapse notebooks.

    enter image description here

    If you are running the pipeline only one time, you can manually change the parameter to original and continue your pipeline design. But, if you want to automate this, then storing the secureString value in a key vault secret and accessing it in notebook code might be a possible workaround for this.