Azure aks agic identity refresh credentials

Question

I am deploying an aks cluster with agic (application gateway ingress controller) with bicep

In the aks.bicep I declare this

resource aksCluster 'Microsoft.ContainerService/managedClusters@2021-03-01' = {
  name: 'aks-core-${env}'
  location: resourceGroup().location
  identity: {
    type: 'SystemAssigned'
  }
  properties: {
    ...
    addonProfiles: {
      ...
      ingressApplicationGateway: {
        enabled: true
        config: {
          applicationGatewayId: applicationGatewayId
          effectiveApplicationGatewayId: applicationGatewayId
        }
      }
      
    }
  }
}

However for some reason the identity created by aks seems to need a role in the resource group created for the node pool, because an error "needs contributor role" appears,so I added this:

resource contributorRoleDefinition 'Microsoft.Authorization/roleDefinitions@2020-08-01-preview' existing = {
  scope: subscription()
  name: 'xxx'
}

resource aksfix 'Microsoft.Authorization/roleAssignments@2020-04-01-preview' = {
  name: guid(resourceGroup().id,'aksfix','Contributor')
  scope: resourceGroup()
  properties: {
    description: 'fixes aks cross resource group principal permissions for agic'
    principalId: aksCluster.properties.addonProfiles.ingressApplicationGateway.identity.objectId
    principalType: 'ServicePrincipal'
    roleDefinitionId: contributorRoleDefinition.id
  }
}

Now to be clear this WORKS, but not always. Even if the principal has the appropriate permissions, the same error keeps popping up for 30 min to 1-2 hours and then by magic it works!

I had a similar error when trying to use internal load balancers which requires network contributor to the kubelet principal, and the same behavior appears, it takes an insane amount of time to reflect the role changes, now what seems interesting is a message "if your permissions changed try refreshing your credentials" on the agic pod, is there a way to force a cluster to refresh its credentials (without creating a new service principal)?

Answer 1

You should assign Contributor roles to both the aks magnaged cluster identity to the resource group where the application gateway resources are present and the ingress application gateway identity on the resource group.

So the solution will be better to use user-assigned identity and providing the contributor role and using it in AKS identity like below:

resource aksClusterUserDefinedManagedIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2018-11-30' = {
  name: 'aksClusterUserDefinedManagedIdentityName'
  location: resourceGroup().location
}
resource akscontributorroleassignement 'Microsoft.Authorization/roleAssignments@2020-10-01-preview' = {
  name: guid(concat(resourceGroup().id, aksClusterUserDefinedManagedIdentity.name,aksclustername))
  scope: resourceGroup()
  properties: {
    description: 'Contributor role to the AKS identity to access the AGIC reosurce'
    principalId: aksClusterUserDefinedManagedIdentity.properties.principalId
    principalType: 'ServicePrincipal'
    roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', 'b24988ac-6180-42a0-ab88-20f7382dd24c')
  }
}
resource aksCluster 'Microsoft.ContainerService/managedClusters@2021-03-01' = {
  name: 'aks-core-${env}'
  dependsOn:akscontributorroleassignement
  location: resourceGroup().location
  identity: {
    type: 'UserAssigned',
    userAssignedIdentities: {aksClusterUserDefinedManagedIdentity.id}
  }
  properties: {
    ...
    addonProfiles: {
      ...
      ingressApplicationGateway: {
        enabled: true
        config: {
          applicationGatewayId: applicationGatewayId
          effectiveApplicationGatewayId: applicationGatewayId
        }
      }
      
    }
  }
}


resource contributorRoleDefinition 'Microsoft.Authorization/roleDefinitions@2020-08-01-preview' existing = {
  scope: subscription()
  name: 'xxx'
}

resource aksfix 'Microsoft.Authorization/roleAssignments@2020-04-01-preview' = {
  name: guid(resourceGroup().id,'aksfix','Contributor')
  scope: resourceGroup()
  properties: {
    description: 'fixes aks cross resource group principal permissions for agic'
    principalId: aksCluster.properties.addonProfiles.ingressApplicationGateway.identity.objectId
    principalType: 'ServicePrincipal'
    roleDefinitionId: contributorRoleDefinition.id
  }
}