Search code examples
azureapache-sparkazure-devopsazure-synapse

Azure Synapse CI/CD Pipeline - Override Apache Spark Pool Name in Notebooks

I'm working on setting up a CI/CD pipeline for an Azure Synapse workspace, and I need to create custom parameters in the deployment process to handle environment-specific values. Specifically, I need to override the Apache Spark pool name used in the development environment (synDEVSPark) to the production Apache Spark pool name (synPROSPark1).

I've followed the instructions from the Microsoft documentation, but I'm encountering an error during the synapse deployment task.

Here are the steps I've taken:

  1. Created a template-parameters-definition.json file to define the parameters (shown in the image below).enter image description here
  2. Set up the override parameter in the Synapse deployment task (shown in the image below).enter image description here
  3. Created a pipeline variable to hold the production Apache Spark pool name (shown in the below image).enter image description here

Despite these efforts, the deployment fails with the following error (shown in the image below):

Failed to fetch the deployment status {"code":"400","message":"CreateOrUpdateNotebook failed: [statusCode from ADF:BadRequest, ErrorMessage:{\"code\":\"BadRequest\",\"message\":\"The document creation or update failed because of invalid reference 'synDEVSPark'...

error message screem

I also read a blog post, used in a comment to a somewhat similar question, which mentioned using information within a JSON file that defines the linked service to create the parameter override. However, synDEVSPark is not a linked service but an Apache Spark pool under Analytics pools, so it seems that the blog post cannot help me in this case.

How can I correctly override the Apache Spark pool name in the Synapse deployment task to ensure it uses the production Spark pool (synPROSPark1) instead of the development Spark pool (synDEVSPark)? Is there a specific way to reference the Apache Spark pool in the parameters, considering it is not a linked service?

Any help or guidance on what I might be doing wrong or how to properly configure the override parameters would be greatly appreciated.

Thank you!

Solution

  • To override Apache Spark Pool Name in Notebooks, can follow the blog to update the template definition file (template-parameters-definition.json):

    https://techcommunity.microsoft.com/t5/azure-synapse-analytics-blog/cicd-automation-in-synapse-analytics-taking-advantage-of-custom/ba-p/3513546