Search code examples
azure-data-factoryazure-synapse

How to build OR-behavior into Azure Synapse pipelines?


I have a pipeline in Synapse that reloads data from a number of REST-endpoints. It looks as follows: enter image description here

What I want to achieve is that the 'CheckAccessTokenValidity' activity checks whether my API-token is still valid. If yes, all is fine, proceed to ForEachDivision activity. If no, refresh tokens, Proceed to ForEachDivision activity.

I thought I had implemented this logic in the screenshot provided. However, this is not the case. My pipeline is now deadlocked because the ForEachDivision apparently expects all three preceding activities to be succesful before running (Which is impossible by design).

How do I implement the logic described above in Azure Synapse?


Solution

  • You can modify your pipeline and accommodate following changes:

    1. Create a variable 'var1' , by default set it to 'False'
    2. After 'CheckAccessTokenValidity' , add set variable activity 'set variable1' pointing to 'var1' and set its value as 'True'
    3. After SetRefreshToken , add another set variable activity 'set variable2' pointing to same 'var1' and set its value as 'True' , so that either of the step 2 or 3 can turn the value of variable to 'True'.
    4. Remove all activities starting ForEach block from this pipeline and cut paste those activities and keep in another new pipeline 'pipeline2'
    5. After 'set variable2' activity, attach an If activity with 'success' and 'Skipped' conditional path . If step 2 gets successful, 'set variable2' would be skipped and it will call the next activity that is 'Step6' . If step 2 fails, and step 3 is called, it will call 'set variable2' and after its success, Step6 will be completed.
    6. In this if activity, write the condition to check the value of var1 . If var1=='True' , then execute the 'pipeline2'