Azure Table Storage query failed with authentication error (Received:Forbidden) from time to time

Question

Environment: Azure app service.

Azure storage SDK: WindowsAzure.Storage (9.3.3)

Invocation (pseudo code):

void QueryAzureTable(){
    while(true){
        var tableClient = new AzureTable();
        var resp = tableClient.Query('table','pk','rk');
        // ...
    }
}

var tasks = new List<Task>();
for (var i = 0; i < 5; i++)
{
    tasks.Add(QueryAzureTable());
}
await Task.WhenAll(tasks).ConfigureAwait(false);

Authorization method for QueryAzureTable: tried with both clientId/secret and managed identity/MSI, same result for both.

Observation:

1. Around half of the requests failed due to anth issue in QueryAzureTable() (see detailed error msg below).
2. If I restart the azure app service instance, the auth error will be gone for ~12 hours.

Error Message:

1. Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
2. Unexpected response code, Expected:OK or NotFound, Received:Forbidden

I have checked and tried with almost every solutions mentioned in this stackoverflow thread, but no luck. Guess this specific auth issue might be related with multi-tasks.

Answer 1

Kind of figured out the solution on my own: adding a retry logic to renew the token.

void query(...){
    int cnt=0;
    while(true){
        try{
            _client.queryTable(...);
        }
        catch(AuthException ex){
            log.error(ex...);
            var token=new Token(...);
            _client = new AzureTableClient(token);
            cnt++;
            if(cnt==3) throw;
        }
    }
}

The first clue to this solution was whenever there was a app service release, deployment or restart of the app service, the query table function worked well for a while, and then after around 12 hours, errors started showing up. But not 100% failure rate.

If there is any explanation or conclusion that helps to root cause this, please share your opinions. Thanks in advance! My blind guess is that it has something to do with muti-tasks: WindowsAzure.Storage (9.3.3) does not do a good job of renewing token for muti-tasks.

Hope this could help you.