How to tell Octopus Deploy to wait until another deployment finishes on the same machine?

Question

Sometimes it is preferred and/or required to host dozens of applications on a single server. Not saying this is "right" or "wrong," I'm only saying that it happens.

A downside to this configuration is the error message Waiting for the script in task [TASK ID] to finish as this script requires that no other Octopus scripts are executing on this target at the same time appears whenever more than one deployment to the same machine is running. It seems like Octopus Deploy is fighting itself.

How can I configure Octopus Deploy to wait for one deployment to completely finish before the next one is started?

Answer 1

Before diving into the answer, it is important to understand why that message is appearing in the first place. Each time a step is run on a deployment target, the tentacle will create a "Mutex" to prevent others projects from interfering with it. An early use case for this was updating the IIS metabase during a deployment. In certain cases, concurrent updates would cause random errors.

Option 1: Disable the Mutex

We've seen cases where the mutex is the cause of the delay. The mutex is applied per step, not per deployment. It is common to see a situation where it looks like Octopus is "jumping" between deployments. Depending on the number of concurrent deployments, that can slow down the deployment. The natural thought is to disable the mutex altogether.

It is possible to disable the mutex by adding the variable OctopusBypassDeploymentMutex and setting it to True. That variable can exist in either a specific project or in a variable set.

More details on what that variable does can be found in this document. If you do disable the mutex please test it and monitor for any failures. For the most part, we don't see issues disabling the mutex, but it has happened from time to time. It depends on a host of other factors such as application type and Windows version.

Option 2: Leverage Deploy a Release Step

Another option is to coordinate the projects using the deploy a release step. Typically this works best when the projects being deployed are part of the same application suite. In the example screenshot below I have five "deployment" projects:

• Azure Worker IaC
• Database Worker IaC
• Kubernetes Worker IaC
• Script Worker IaC
• OctoStudy

The project Unleash the Kraken coordinates deployments for those projects.

It does this by using the Deploy a Release step. First it spins up all the infrastructure, then it deploys the application.

This won't work as well if the server is hosting 50 disparate applications.

Option 3: Leverage the API to check for running deployments

The final option is to include a step at the start of each project which hits the API to check for active releases to the deployment targets for the deployment target. If an active deployment is found then wait until it is done.

You can do this by hitting the endpoint https://[YOUR URL]/api/[SPACE ID]/machines/[Machine Id]/tasks?skip=0&name=Deploy&states=Executing%2CCancelling&spaces=[SPACE ID]&includeSystem=false. That will tell you all the active tasks being run for a specific machine.

You can get Machine Id by pulling the value from Octopus.Deployment.Machines. You can get Space Id by pulling the value from Octopus.Space.Id.

The pseudo code for this approach could look like this (I'm not including the actual code as your requirements could be very different).

activeDeployments = true

while (activeDeployments)
{
      activeDeployments = false
      foreach(machineId in Octopus.Deployment.Machines)
      {
             activeTasks = https://[YOUR URL]/api/[Octopus.Space.Id]/machines/[Machine Id]/tasks?skip=0&name=Deploy&states=Executing%2CCancelling&spaces=[Octopus.Space.Id]&includeSystem=false

             if (activeTasks.Count > 0)
             {
                      activeDeployments = true
             }
      }

      if (activeDeployments = true)
      {
            Sleep for 5 seconds
      }
}