"Could not retrieve credential from local cache for service principal" when using Azure CLI 2.30.0 credentials in Python SDK on Azure Devops MS agent

Question

I an Azure Pipeline on a self-hosted agent I use this task

      - task: AzureCLI@2
        displayName: Azure CLI task with Python SDK
        inputs:
          azureSubscription: 'SUBSCRIPTION-SERVICE-CONNECTION'
          scriptType: bash
          scriptLocation: inlineScript
          inlineScript: |
            python ./magic-script.py

with that I am able to use the credentials to authenticate Azure Python SDK:

client = get_client_from_cli_profile(GraphRbacManagementClient)

When I transfer this process to a MS hosted agent I get this error:

  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/azure/common/client_factory.py", line 85, in get_client_from_cli_profile
    with_tenant=True,
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/azure/common/credentials.py", line 98, in get_azure_cli_credentials
    cred, subscription_id, tenant_id = profile.get_login_credentials(resource=resource)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/azure/cli/core/_profile.py", line 335, in get_login_credentials
    credential = self._create_credential(account, client_id=client_id)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/azure/cli/core/_profile.py", line 592, in _create_credential
    return identity.get_service_principal_credential(username_or_sp_id)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/azure/cli/core/auth/identity.py", line 185, in get_service_principal_credential
    entry = self._msal_secret_store.load_entry(client_id, self.tenant_id)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/azure/cli/core/auth/identity.py", line 270, in load_entry
    .format(sp_id))
knack.util.CLIError: Could not retrieve credential from local cache for service principal ***. Run `az login` for this service principal.

---

Based on this migration guide I also tried AzureCliCredential like

credential = AzureCliCredential()
client = GraphRbacManagementClient(credential, os.environ["subscriptionId"])   

which get's my script signed in - but when using GraphRbacManagementClient I get this error locally on my dev box and on the agent:

    root_group = [g for g in graph_client.groups.list(
  File "C:\Python38\lib\site-packages\msrest\paging.py", line 143, in __next__
    self.advance_page()
  File "C:\Python38\lib\site-packages\msrest\paging.py", line 129, in advance_page
    self._response = self._get_next(self.next_link)
  File "C:\Python38\lib\site-packages\azure\graphrbac\operations\groups_operations.py", line 336, in internal_paging
    response = self._client.send(request, stream=False, **operation_config)
  File "C:\Python38\lib\site-packages\msrest\service_client.py", line 336, in send
    pipeline_response = self.config.pipeline.run(request, **kwargs)
  File "C:\Python38\lib\site-packages\msrest\pipeline\__init__.py", line 197, in run
    return first_node.send(pipeline_request, **kwargs)  # type: ignore
  File "C:\Python38\lib\site-packages\msrest\pipeline\__init__.py", line 150, in send
    response = self.next.send(request, **kwargs)
  File "C:\Python38\lib\site-packages\msrest\pipeline\requests.py", line 65, in send
    self._creds.signed_session(session)
AttributeError: 'AzureCliCredential' object has no attribute 'signed_session'

For StorageManagementClient this works. Seems to be depending on the SDK client used.

Answer 1

This issue is caused by Azure CLI version 2.30.0 which seemed to be rolled out MS hosted agents recently.

Hence I adapted all my Python scripts running on (MS and self) hosted agents to this model:

def get_graph_client(subscription_id):
    if "tenantId" in os.environ:
        print('using environment variables')
        config_dict = {
            "clientId": os.environ["servicePrincipalId"],
            "clientSecret": os.environ["servicePrincipalKey"],
            "subscriptionId": subscription_id,
            "tenantId": os.environ["tenantId"],
            "activeDirectoryEndpointUrl": "https://login.microsoftonline.com",
            "resourceManagerEndpointUrl": "https://management.azure.com/",
            "activeDirectoryGraphResourceId": "https://graph.windows.net/",
            "managementEndpointUrl": "https://management.core.windows.net/"
        }
        client = get_client_from_json_dict(
            GraphRbacManagementClient, config_dict)
    else:
        print('using CLI credentials')
        credential = AzureCliCredential()
        client = GraphRbacManagementClient(credential, subscription_id)        
    return client

The error AttributeError: 'AzureCliCredential' object has no attribute 'signed_session' I showed in my question was due to corrupt dependencies. Once virtual environment was cleaned up and the correct dependencies where installed, it worked for all Python Azure SDK clients.

This allows me to operate in both modes, with Azure CLI credentials and also with explicit credentials by setting addSpnToEnvironment: true in the corresponding pipeline task:

      - task: AzureCLI@2
        displayName: Azure CLI task with Python SDK
        inputs:
          azureSubscription: 'SUBSCRIPTION-SERVICE-CONNECTION'
          scriptType: bash
          scriptLocation: inlineScript
          addSpnToEnvironment: true
          inlineScript: |
            python ./magic-script.py