Error: Blob LeaseIdMissing when trying to update remote state in Azure Blob Storage

Question

I'm encountering an error when trying to update my Terraform remote state stored in Azure Blob Storage:

Error: blobs.Client#PutBlockBlob: Failure responding to request: StatusCode=412 -- Original Error: autorest/azure: Service returned an error. Status=412 Code="LeaseIdMissing" Message="There is currently a lease on the blob and no lease ID was specified in the request.\nRequestId:6a464060-f01e-009f-0634-4a77e1000000\nTime:2024-12-09T12:18:32.3005371Z"

The error occurs when I try to run terraform apply. The state file is stored in Azure Blob Storage using the azurerm backend configuration.

I've tried:

• Running terraform force-unlock but it fails with "failed to retrieve lock info: blob metadata 'terraformlockid' was empty"
• Manually breaking the lease through the Azure portal (Break Lease disabled)
• Deleting and recreating the .terraform directory (Not sure how to do this in Azure Devops)
• Updating Terraform and the azurerm provider to the latest versions (version 1.10)

azure-terraform-plan.yml

# https://learn.hashicorp.com/terraform/development/running-terraform-in-automation
# https://www.terraform.io/docs/providers/azurerm/guides/service_principal_client_secret.html

steps:
  - task: ms-devlabs.custom-terraform-tasks.custom-terraform-installer-task.TerraformInstaller@0
    displayName: "Install Terraform latest"

  - bash: |
      set -e -x
      mkdir -p ${WORKSPACE}/artifacts
    env:
      WORKSPACE: $(Pipeline.Workspace)
    displayName: "Create artifacts folder"

  - task: AzureCLI@2
    inputs:
      azureSubscription: $(TF_VAR_SUBSCRIPTION_NAME)
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: |
        echo "Working directory: $(pwd)"
        export ARM_CLIENT_ID=${TF_VAR_CLIENT_ID}
        export ARM_CLIENT_SECRET=${TF_VAR_CLIENT_SECRET}
        export ARM_SUBSCRIPTION_ID=${TF_VAR_SUBSCRIPTION_ID}
        export ARM_TENANT_ID=${TF_VAR_TENANT_ID}
        export TF_IN_AUTOMATION=1

        chmod +x az.sh
        chmod +x providers-install.sh

        export STORAGE_ACCOUNT=${TF_VAR_STORAGE_ACCOUNT_NAME}
        export STORAGE_ACCOUNT_KEY=`az storage account keys list --account-name "${STORAGE_ACCOUNT}" --query [0].value --out tsv`

        terraform init -reconfigure\
          -backend-config="storage_account_name=${STORAGE_ACCOUNT}" \
          -backend-config="container_name=tfstate" \
          -backend-config="access_key=${STORAGE_ACCOUNT_KEY}" \
          -backend-config="key=${TF_VAR_ENV_NAME}.tfstate"

        . terraform-fix.sh
        terraform plan -input=false -lock=false -detailed-exitcode -out ${TF_VAR_ENV_NAME}.plan

        EXIT_CODE=$?
        if [[ "${EXIT_CODE}" = "2" ]]; then
          EXIT_CODE=0
        fi

        if [[ "${EXIT_CODE}" != "0" ]]; then
          exit ${EXIT_CODE}
        fi
    env:
      TF_VAR_CLIENT_ID: $(TF_VAR_CLIENT_ID)
      TF_VAR_CLIENT_SECRET: $(TF_VAR_CLIENT_SECRET)
    displayName: "Terraform plan"

  - bash: |
      set -e -x
      cd ${WORKSPACE}/artifacts
      tar --exclude='./.git' --exclude='./.terraform/providers' -czf terraform.tar.gz -C ${SOURCEDIR} .
      ls -l
    env:
      WORKSPACE: $(Pipeline.Workspace)
      SOURCEDIR: $(System.DefaultWorkingDirectory)
    displayName: "Package terraform plan"

  - task: PublishPipelineArtifact@0
    inputs:
      artifactName: "$(TF_VAR_ENV_NAME)"
      targetPath: "$(Pipeline.Workspace)/artifacts"
    displayName: "Publish terraform plan"

azure-terraform-apply.yml

# https://learn.hashicorp.com/terraform/development/running-terraform-in-automation
# https://www.terraform.io/docs/providers/azurerm/guides/service_principal_client_secret.html

steps:
  - task: ms-devlabs.custom-terraform-tasks.custom-terraform-installer-task.TerraformInstaller@0
    displayName: "Install Terraform latest"

  - bash: |
      if [ -f "$(Pipeline.Workspace)/$(TF_VAR_ENV_NAME)/terraform.tar.gz" ]; then
        echo '##vso[task.setvariable variable=package_available]yes'
      fi
    displayName: "Check terraform plan is available"

  - task: ExtractFiles@1
    inputs:
      archiveFilePatterns: "$(Pipeline.Workspace)/$(TF_VAR_ENV_NAME)/terraform.tar.gz"
      cleanDestinationFolder: false
      destinationFolder: "./"
    condition: and(succeeded(), eq(variables['package_available'], 'yes'))
    displayName: "Extract terraform plan"

  - task: AzureCLI@2
    inputs:
      azureSubscription: $(TF_VAR_SUBSCRIPTION_NAME)
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: |
        echo "Working directory: $(pwd)"
        export ARM_CLIENT_ID=${TF_VAR_CLIENT_ID}
        export ARM_CLIENT_SECRET=${TF_VAR_CLIENT_SECRET}
        export ARM_SUBSCRIPTION_ID=${TF_VAR_SUBSCRIPTION_ID}
        export ARM_TENANT_ID=${TF_VAR_TENANT_ID}
        set -e -x
        chmod +x az.sh

        export TF_LOG_PATH=$(System.DefaultWorkingDirectory)/terraform.log
        terraform init -input=false
        TF_LOG=DEBUG terraform apply -input=false -lock=false -auto-approve ${TF_VAR_ENV_NAME}.plan
    env:
      TF_VAR_CLIENT_ID: $(TF_VAR_CLIENT_ID)
      TF_VAR_CLIENT_SECRET: $(TF_VAR_CLIENT_SECRET)
    condition: and(succeeded(), eq(variables['package_available'], 'yes'))
    displayName: "Terraform apply plan"

  - task: KubectlInstaller@0
    displayName: Kubectl installer
    inputs:
      kubectlVersion: latest

  - task: AzureCLI@2
    displayName: "Kubernetes apply configurations"
    condition: and(succeeded(), eq(variables['package_available'], 'yes'))
    inputs:
      azureSubscription: $(TF_VAR_SUBSCRIPTION_NAME)
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: |
        source ./kubectl-apply.sh
    env:
      ENV_NAME: $(TF_VAR_ENV_NAME)
      ENVIRONMENT_TYPE: $(TF_VAR_ENVIRONMENT_TYPE)
      REGISTRY: $(container-registry)
      RESOURCE_GROUP: $(TF_VAR_RESOURCE_GROUP)

  - task: PublishBuildArtifacts@1
    inputs:
      pathtoPublish: $(System.DefaultWorkingDirectory)/terraform.log
      artifactName: terraform-logs

Answer 1

Blob LeaseIdMissing when trying to update remote state in Azure Blob Storage

Issue would be for two reasons

1. lease state of blob file in remote repository
2. Simultaniously no lease ID is provided to unlock it

Use the CLI command to break lease the blob from the acquire lease

az storage blob lease break --account-key <storageacckey> --account-name samkskvbsstraoeg --blob-name terraform.tfstate --container-name testsampel

you can do this form the terraform commands as well using lockID which we got while appling Acquire lease

terraform force-unlock <LOCK_ID>

State the same while using terraform backend

terraform {
  backend "azurerm" {
    storage_account_name = "<storage_account_name>"
    container_name       = "tfstate"
    key                  = "<env_name>.tfstate"
    access_key           = "<storage_account_key>"
  }
}

But inorder to keep this working we need to update the metadata for the backend configuration

az storage blob metadata update --account-name <storage_account_name> --container-name tfstate --name <tfstate_file_name> --metadata terraformlockid=""

Remove the terraform backend files and rerun the pipeline by using configuration

rm -rf .terraform
terraform init -reconfigure \
  -backend-config="storage_account_name=${STORAGE_ACCOUNT}" \
  -backend-config="container_name=tfstate" \
  -backend-config="access_key=${STORAGE_ACCOUNT_KEY}" \
  -backend-config="key=${TF_VAR_ENV_NAME}.tfstate"

Now update the yaml in terraform apply as mentioned below

az storage blob lease break --account-name ${STORAGE_ACCOUNT} --container-name tfstate --name ${TF_VAR_ENV_NAME}.tfstate --account-key ${STORAGE_ACCOUNT_KEY} 
terraform apply -input=false -auto-approve

This will make sure that the state of blob will be readily available for apply command using CLI

Refer:

https://learn.microsoft.com/en-us/cli/azure/storage/blob/lease?view=azure-cli-latest

https://luke.geek.nz/azure/failed-to-persist-terraform-state-using-an-azure-blob-storage-account/ by Luke Murray