Search code examples
azureazure-devopsazure-cosmosdbazure-rm-templateazure-pipelines-yaml

AzureResourceManagerTemplateDeployment@3 task leads to bad request when deploying Cosmos DB

This one could be related to the Microsoft itself, rather than the task or the pipeline, but if anyone can assist with the error, I'd appreciate.

A week ago, our task 'AzureResourceManagerTemplateDeployment@3' suddenly stopped working, no changes were made.

The error leads to a bad request "API version '2023-04-15' is invalid" for "Microsoft.Azure.Documents.Common"

There were errors in your deployment. Error code: DeploymentFailed.
##[error]At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.
##[error]Details:
##[error]BadRequest: API version '2023-04-15' is invalid
ActivityId: 2b0678f8-af04-4bdb-b84a-eb95117f3127, Microsoft.Azure.Documents.Common/2.14.0
##[error]BadRequest: API version '2023-04-15' is invalid


Deployment details:
- Resource: "my-db-name",
- Type: "Microsoft.DocumentDB/databaseAccounts"
- Status: "Bad request"

Operation details:

{
    "status": "Failed",
    "error": {
        "code": "BadRequest",
        "message": "API version '2023-04-15' is invalid\r\nActivityId: 0b3e12e6-7fc1-4433-a705-2a0e5cbf86bd, Microsoft.Azure.Documents.Common/2.14.0"
    }
}

The task we're using seems right.

    - task: AzureResourceManagerTemplateDeployment@3
      displayName: 'Deploying/Updating resources'
      inputs:
        deploymentScope: 'Resource Group'
        azureResourceManagerConnection: '$(ARMConnection)'
        subscriptionId: '$(AzSubscription)'
        action: 'Create Or Update Resource Group'
        resourceGroupName: '$(resourceGroupName)'
        location: '$(defaultLocation)'
        templateLocation: 'Linked artifact'
        csmFile: 'Infrastructure/AzureResources.json'
        overrideParameters: '-stackName "$(yyStackName)" -appInsightLocation "$(appInsightLocation)" -yyTenantId "$(yyTenantId)" -yyObjectId "$(yyObjectId)" -devSecurityGroup "$(devSecurityGroup)" -frontendUrl "$(frontendUrl)" -environment "${{ parameters.environment }}" -tag.XX-AppId "$(tagXXAppId)" -tag.XX-Env "$(tagXXEnv)" -tag.XX-StackName "$(yyStackName)"'
        deploymentMode: 'Incremental'
        deploymentName: '$(deploymentName)'
        deploymentOutputs: 'MainArmOutputs'

The full log indicates that the template is valid.

Even with System.Debug = true, no further information is available.

Template deployment validation was completed successfully.
Starting Deployment.
...
##[debug][GET]https://management.azure.com/subscriptions/xxxx6efb-9424-4437-bf34-886b2db99f39/resourcegroups/rg-be-xxxx/providers/Microsoft.Resources/deployments/Manual-20230505.15/operationStatuses/08585182901330519018?api-version=2021-04-01
##[debug]Correlation ID from ARM api call response : 1824d088-744e-4f11-ade0-ddfc9c05b6d8
##[debug]Response status code : 200
##[debug][GET]https://management.azure.com/subscriptions/xxxxxb-9424-4437-bf34-886b2db99f39/resourcegroups/rg-be-xxxx/providers/Microsoft.Resources/deployments/Manual-20230505.15?api-version=2021-04-01
##[debug]Correlation ID from ARM api call response : d1f50a07-c232-406e-a9b7-d673eb33800e
There were errors in your deployment. Error code: DeploymentFailed.
##[error]At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.
##[debug]Processed: ##vso[task.issue type=error;]At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.
##[error]Details:
##[debug]Processed: ##vso[task.issue type=error;]Details:
##[error]BadRequest: API version '2023-04-15' is invalid
ActivityId: d95dd2f8-db7a-4942-bf9d-a51e68b4e71f, Microsoft.Azure.Documents.Common/2.14.0

In the "Infrastructure/AzureResources.json" file, Cosmos is defined as follows:

 {
      "type": "Microsoft.DocumentDB/databaseAccounts",
      "apiVersion": "2021-04-15",
      "name": "[variables('cosmosDbName')]",
      "location": "[parameters('location')]",
      "tags": {
        "defaultExperience": "CORE (SQL)",
        "hidden-cosmos-mmspecial": "",
        .....
      },
      "kind": "GlobalDocumentDB",
....
}

Is valid to mention that we are within our service quotas (Provisioned throughput, etc). No resources have been extrapolated.

This issue started on May 3rd and even through the Azure Portal we were not able to publish a database in the Canada East region until Friday May 5th. So I believe it could be something on Microsoft's side.

Solution

  • Precisely, this issue was on Microsoft's side. As we did not make any modifications to our pipelines and were unable to deploy any Cosmos Document DB in Canada East, we opened a ticket with the Cosmos Team who responded:

    The issue is a known issue involving an Outage in the Canada East Region. This was due to an issue created by the latest RP API version, 04-15. The Product group mentioned that the Portal Team is deploying an ongoing fix for the last 48-72 hrs. which should be in Production soon. They have asked to retry this tomorrow and check if they you are still facing this issue. I will also follow up with them and check for updates and will be sure to get back to you on the same. We apologize for the inconvenience caused in the process.

    After their fix, we are finally able to deploy our Cosmos databases in Canada East. No modifications were made on our side, either to the ARM file or the YML file.