Search code examples
apache-sparkpysparkazure-synapse

Azure Synapse Notebook not running in Pipeline

I cant seem to run my Synapse Notebook in a pipeline, however if i run it by itself it works fine. The error i get is simply : "Operation on target LoadFactSalesDoc_without_DimResource failed: MessageQueueFullException: The message queue is full or is completed and cannot accept more items."

Anyone had this error before?

I tried running the notebook in a different pipeline, I also tried allocating more ressources to the notebook. It always has the same error.

Solution

  • https://learn.microsoft.com/en-us/azure/synapse-analytics/troubleshoot/troubleshoot-synapse-studio#message-queue-full-or-is-completed-and-cannot-accept-more-items

    Root cause: There is a limitation of 128 of cells when executing a Synapse notebook activity from a pipeline.

    Action: You could merge cells in order to reduce the number of cells below 128.