Search code examples
azureazure-machine-learning-service

Workspace Authentication: More than one token matches the criteria


I constantly run into problems when working on Azure Compute Instances and trying to connect from the Jupyter Lab to the workspace.

With InteractiveLoginAuthentication I get the following message:

AuthenticationException: AuthenticationException:
    Message: Could not retrieve user token. Please run 'az login'
    InnerException More than one token matches the criteria. The result is ambiguous.
    ErrorResponse 
{
    "error": {
        "code": "UserError",
        "inner_error": {
            "code": "Authentication"
        },
        "message": "Could not retrieve user token. Please run 'az login'"
    }
}

With a Service Principal this one (SP is owner in the ML Workspace):

WorkspaceException: WorkspaceException:
    Message: No workspaces found with name=xxx in all the subscriptions that you have access to.
    InnerException None
    ErrorResponse 
{
    "error": {
        "message": "No workspaces found with name=xxx in all the subscriptions that you have access to."
    }
}

I had another workspace in a different subscription where I could resolve it by giving the tennant as an extra input to the InteractiveLoginAuthentication. This time, no chance.

The funny thing is, though, that I can login to the workspace via InteractiveLoginAuthentication when doing it from my local computer.

I supsected that some old tokens are cached somewhere so I tried to use the "Private browsing" function of my browser. Furthermore, I deleted /home/azureuser/.azure/accessTokens.json but no effect.

Maybe some of you had this problem before and have an idea?

For reference some sites I checked:

Update

When I run this code:

from azureml.core.authentication import InteractiveLoginAuthentication
interactive_auth = InteractiveLoginAuthentication(tenant_id='xxx')

ws = Workspace.get(name='xxx',
                   subscription_id='xxx',
                   resource_group='xxx',
                   auth=interactive_auth)

I get the following trace:

---------------------------------------------------------------------------
AdalError                                 Traceback (most recent call last)
/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/authentication.py in _get_arm_token_with_refresh(profile_object, cloud_type, account_object, config_object, session_object, config_directory, force_reload, resource)
   1820         auth, _, _ = profile_object.get_login_credentials(resource)
-> 1821         access_token = auth._token_retriever()[1]
   1822         if (_get_exp_time(access_token) - time.time()) < _TOKEN_REFRESH_THRESHOLD_SEC:

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/_vendor/azure_cli_core/_profile.py in _retrieve_token()
    525                     return self._creds_cache.retrieve_token_for_user(username_or_sp_id,
--> 526                                                                      account[_TENANT_ID], resource)
    527                 use_cert_sn_issuer = account[_USER_ENTITY].get(_SERVICE_PRINCIPAL_CERT_SN_ISSUER_AUTH)

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/_vendor/azure_cli_core/_profile.py in retrieve_token_for_user(self, username, tenant, resource)
    889         context = self._auth_ctx_factory(self._cloud_type, tenant, cache=self.adal_token_cache)
--> 890         token_entry = context.acquire_token(resource, username, _CLIENT_ID)
    891         if not token_entry:

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/adal/authentication_context.py in acquire_token(self, resource, user_id, client_id)
    144 
--> 145         return self._acquire_token(token_func)
    146 

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/adal/authentication_context.py in _acquire_token(self, token_func, correlation_id)
    127         self.authority.validate(self._call_context)
--> 128         return token_func(self)
    129 

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/adal/authentication_context.py in token_func(self)
    142             token_request = TokenRequest(self._call_context, self, client_id, resource)
--> 143             return token_request.get_token_from_cache_with_refresh(user_id)
    144 

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/adal/token_request.py in get_token_from_cache_with_refresh(self, user_id)
    346         self._user_id = user_id
--> 347         return self._find_token_from_cache()
    348 

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/adal/token_request.py in _find_token_from_cache(self)
    126         cache_query = self._create_cache_query()
--> 127         return self._cache_driver.find(cache_query)
    128 

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/adal/cache_driver.py in find(self, query)
    195                         {"query": log.scrub_pii(query)})
--> 196         entry, is_resource_tenant_specific = self._load_single_entry_from_cache(query)
    197         if entry:

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/adal/cache_driver.py in _load_single_entry_from_cache(self, query)
    123             else:
--> 124                 raise AdalError('More than one token matches the criteria. The result is ambiguous.')
    125 

AdalError: More than one token matches the criteria. The result is ambiguous.

During handling of the above exception, another exception occurred:

AuthenticationException                   Traceback (most recent call last)
/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/authentication.py in wrapper(self, *args, **kwargs)
    288                     module_logger.debug("{} acquired lock in {} s.".format(type(self).__name__, duration))
--> 289                 return test_function(self, *args, **kwargs)
    290             except Exception as e:

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/authentication.py in _get_arm_token(self)
    474         else:
--> 475             return self._get_arm_token_using_interactive_auth()
    476 

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/authentication.py in _get_arm_token_using_interactive_auth(self, force_reload, resource)
    589         arm_token = _get_arm_token_with_refresh(profile_object, cloud_type, ACCOUNT, CONFIG, SESSION,
--> 590                                                 get_config_dir(), force_reload=force_reload, resource=resource)
    591         # If a user has specified a tenant id then we need to check if this token is for that tenant.

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/authentication.py in connection_aborted_wrapper(*args, **kwargs)
    325                 try:
--> 326                     return function(*args, **kwargs)
    327                 except AuthenticationException as e:

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/authentication.py in _get_arm_token_with_refresh(profile_object, cloud_type, account_object, config_object, session_object, config_directory, force_reload, resource)
   1829             raise AuthenticationException("Could not retrieve user token. Please run 'az login'",
-> 1830                                           inner_exception=e)
   1831 

AuthenticationException: AuthenticationException:
    Message: Could not retrieve user token. Please run 'az login'
    InnerException More than one token matches the criteria. The result is ambiguous.
    ErrorResponse 
{
    "error": {
        "code": "UserError",
        "inner_error": {
            "code": "Authentication"
        },
        "message": "Could not retrieve user token. Please run 'az login'"
    }
}

During handling of the above exception, another exception occurred:

AdalError                                 Traceback (most recent call last)
/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/authentication.py in _get_arm_token_with_refresh(profile_object, cloud_type, account_object, config_object, session_object, config_directory, force_reload, resource)
   1820         auth, _, _ = profile_object.get_login_credentials(resource)
-> 1821         access_token = auth._token_retriever()[1]
   1822         if (_get_exp_time(access_token) - time.time()) < _TOKEN_REFRESH_THRESHOLD_SEC:

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/_vendor/azure_cli_core/_profile.py in _retrieve_token()
    525                     return self._creds_cache.retrieve_token_for_user(username_or_sp_id,
--> 526                                                                      account[_TENANT_ID], resource)
    527                 use_cert_sn_issuer = account[_USER_ENTITY].get(_SERVICE_PRINCIPAL_CERT_SN_ISSUER_AUTH)

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/_vendor/azure_cli_core/_profile.py in retrieve_token_for_user(self, username, tenant, resource)
    889         context = self._auth_ctx_factory(self._cloud_type, tenant, cache=self.adal_token_cache)
--> 890         token_entry = context.acquire_token(resource, username, _CLIENT_ID)
    891         if not token_entry:

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/adal/authentication_context.py in acquire_token(self, resource, user_id, client_id)
    144 
--> 145         return self._acquire_token(token_func)
    146 

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/adal/authentication_context.py in _acquire_token(self, token_func, correlation_id)
    127         self.authority.validate(self._call_context)
--> 128         return token_func(self)
    129 

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/adal/authentication_context.py in token_func(self)
    142             token_request = TokenRequest(self._call_context, self, client_id, resource)
--> 143             return token_request.get_token_from_cache_with_refresh(user_id)
    144 

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/adal/token_request.py in get_token_from_cache_with_refresh(self, user_id)
    346         self._user_id = user_id
--> 347         return self._find_token_from_cache()
    348 

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/adal/token_request.py in _find_token_from_cache(self)
    126         cache_query = self._create_cache_query()
--> 127         return self._cache_driver.find(cache_query)
    128 

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/adal/cache_driver.py in find(self, query)
    195                         {"query": log.scrub_pii(query)})
--> 196         entry, is_resource_tenant_specific = self._load_single_entry_from_cache(query)
    197         if entry:

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/adal/cache_driver.py in _load_single_entry_from_cache(self, query)
    123             else:
--> 124                 raise AdalError('More than one token matches the criteria. The result is ambiguous.')
    125 

AdalError: More than one token matches the criteria. The result is ambiguous.

During handling of the above exception, another exception occurred:

AuthenticationException                   Traceback (most recent call last)
<ipython-input-2-fd1276999d15> in <module>
      5                    subscription_id='00c983e5-d766-480b-be75-abf95d1a46c3',
      6                    resource_group='BusinessIntelligence',
----> 7                    auth=interactive_auth)

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/workspace.py in get(name, auth, subscription_id, resource_group)
    547 
    548         result_dict = Workspace.list(
--> 549             subscription_id, auth=auth, resource_group=resource_group)
    550         result_dict = {k.lower(): v for k, v in result_dict.items()}
    551         name = name.lower()

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/workspace.py in list(subscription_id, auth, resource_group)
    637         elif subscription_id and resource_group:
    638             workspaces_list = Workspace._list_legacy(
--> 639                 auth, subscription_id=subscription_id, resource_group_name=resource_group)
    640 
    641             Workspace._process_autorest_workspace_list(

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/workspace.py in _list_legacy(auth, subscription_id, resource_group_name, ignore_error)
   1373                 return None
   1374             else:
-> 1375                 raise e
   1376 
   1377     @staticmethod

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/workspace.py in _list_legacy(auth, subscription_id, resource_group_name, ignore_error)
   1367             # azureml._base_sdk_common.workspace.models.workspace.Workspace
   1368             workspace_autorest_list = _commands.list_workspace(
-> 1369                 auth, subscription_id=subscription_id, resource_group_name=resource_group_name)
   1370             return workspace_autorest_list
   1371         except Exception as e:

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/_project/_commands.py in list_workspace(auth, subscription_id, resource_group_name)
    386         if resource_group_name:
    387             list_object = WorkspacesOperations.list_by_resource_group(
--> 388                 auth._get_service_client(AzureMachineLearningWorkspaces, subscription_id).workspaces,
    389                 resource_group_name)
    390             workspace_list = list_object.value

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/authentication.py in _get_service_client(self, client_class, subscription_id, subscription_bound, base_url)
    155         # in the multi-tenant case, which causes confusion.
    156         if subscription_id:
--> 157             all_subscription_list, tenant_id = self._get_all_subscription_ids()
    158             self._check_if_subscription_exists(subscription_id, all_subscription_list, tenant_id)
    159 

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/authentication.py in _get_all_subscription_ids(self)
    497         :rtype: list, str
    498         """
--> 499         arm_token = self._get_arm_token()
    500         return self._get_all_subscription_ids_internal(arm_token)
    501 

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/authentication.py in wrapper(self, *args, **kwargs)
    293                     InteractiveLoginAuthentication(force=True, tenant_id=self._tenant_id)
    294                     # Try one more time
--> 295                     return test_function(self, *args, **kwargs)
    296                 else:
    297                     raise e

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/authentication.py in _get_arm_token(self)
    473             return self._ambient_auth._get_arm_token()
    474         else:
--> 475             return self._get_arm_token_using_interactive_auth()
    476 
    477     @_login_on_failure_decorator(_interactive_auth_lock)

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/authentication.py in _get_arm_token_using_interactive_auth(self, force_reload, resource)
    588         profile_object = Profile(async_persist=False, cloud_type=cloud_type)
    589         arm_token = _get_arm_token_with_refresh(profile_object, cloud_type, ACCOUNT, CONFIG, SESSION,
--> 590                                                 get_config_dir(), force_reload=force_reload, resource=resource)
    591         # If a user has specified a tenant id then we need to check if this token is for that tenant.
    592         if self._tenant_id and fetch_tenantid_from_aad_token(arm_token) != self._tenant_id:

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/authentication.py in connection_aborted_wrapper(*args, **kwargs)
    324             while True:
    325                 try:
--> 326                     return function(*args, **kwargs)
    327                 except AuthenticationException as e:
    328                     if "Connection aborted." in str(e) and attempt <= retries:

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/authentication.py in _get_arm_token_with_refresh(profile_object, cloud_type, account_object, config_object, session_object, config_directory, force_reload, resource)
   1828         if not token_about_to_expire:
   1829             raise AuthenticationException("Could not retrieve user token. Please run 'az login'",
-> 1830                                           inner_exception=e)
   1831 
   1832     try:

AuthenticationException: AuthenticationException:
    Message: Could not retrieve user token. Please run 'az login'
    InnerException More than one token matches the criteria. The result is ambiguous.
    ErrorResponse 
{
    "error": {
        "code": "UserError",
        "inner_error": {
            "code": "Authentication"
        },
        "message": "Could not retrieve user token. Please run 'az login'"
    }
}
  • azureml-sdk is on version 1.9.0
  • I can connect an authenticate from my local machine. Problems only occur when I want to work on a compute instance.

Solution

  • Okay, here is the answer:

    • You work for company A which is on Azure.
    • You get access to company B's subscription.
    • Problem is: You are associated to A's AAD in ML-Studio.
    • You need to specify the tenant ID in the InteractiveLoginAuthentication like so:
    interactive_auth = InteractiveLoginAuthentication(tenant_id=tenant_id)
    
    workspace = Workspace.get(name=workspace_name,
                              subscription_id=subscription_id,
                              resource_group=resource_group,
                              auth=interactive_auth)
    
    • Now the important part: You need to use company B's tenant_id (I used company A's all the time since I thought that was my authentication point)
    • Of course, this is obvious while you read it...as it is to me now :)

    Hope this helps you. Took me some time but learned a lot ;)