Executing queries being timing out after having performed many queries

Question

I am executing queries on a table asynchronously, and everything is fine for a while, but then all of a sudden the queries begin timing out.

In short I have a text file containing a number of IDs, for each of these IDs I want to execute a query in a table. The query itself should not take long, less than 5 seconds, however after having successfully executed many queries, all of a sudden the application starts timing out every single time. If I then restart my application and re-run the IDs that timed out, they now run perfectly without issues.

So say I have a list of 1000 IDs, then it starts timing out after having processed and successfully executed 200 IDs.

I have tried inserting a MaximumExecutionTime and setting it much higher than the standard 5 seconds, but it still times out. I have tried not sharing the CloudTableClients, and simply have each "thread" create a new instance.

The code is part of a larger project, so I will include the relevant bits here and omit some other code. I start by creating one CloudTableClient, this is shared between the different tasks, as if I have understood it correctly it is okay for it to be shared. I then have an ActionBlock that I post the ids to. I have limited the MaxDegreeOfParallelism to how many processors I have, I have had the best experience with this.

CloudStorageAccount cloudStorage = CloudStorageAccount.Parse(_ParsedConnectionString);
CloudTableClient tableClient = cloudStorage.CreateCloudTableClient();

var block = new ActionBlock<int>(
  async id =>
  {
    // Do stuff
    var res = await executeQuery(id, tableClient);
    // Do more stuff
  },
  new ExecutionDataflowBlockOptions
  {
    MaxDegreeOfParallelism = Environment.ProcessorCount
  }
);

StreamReader file = new StreamReader(someFile);
string line;
while ((line = file.ReadLine()) != null)
{
  block.Post(Convert.ToInt32(line));
}
block.Complete();
block.Completion.Wait();

And in the executeQuery function:

CloudTable table = tableClient.GetTableReference(_someTable);

var tableQuery = new TableQuery<SomeEntity>
{
  FilterString = "some query"
};

TableContinuationToken continuationToken = null;
TableRequestOptions options = new TableRequestOptions
{
  MaximumExecutionTime = new TimeSpan(0, 2, 0)
};
OperationContext context = new OperationContext();

do
{
  var tableQueryResult = await table.ExecuteQuerySegmentedAsync(tableQuery, continuationToken, options, context);
  continuationToken = tableQueryResult.ContinuationToken;
  // Do stuff
}
while (continuationToken != null);

The query is fine, as if I re-run the application with an ID it previously timed out with, it runs as expected.

The client could not finish the operation within specified timeout.
at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndExecuteAsync[T](IAsyncResult result) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.cs:line 51
   at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.<>c__DisplayClass2`1.<CreateCallback>b__0(IAsyncResult ar) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Util\AsyncExtensions.cs:line 69
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at App.Parser.<executeQuery>d__25.MoveNext()`

Answer 1

Your queries are running against an AzureStorage according to the exception you get. Some operations on storage are limited by a quota in number/minute.

Basically read/write are almost unlimited but management is.

For instance if in each of your request you first try to acquire storage key then you may hit the limit and afterward azure throttles you (which may leads to timeout as the throttling is per minute).