Search code examples
c#azureazure-table-storageazure-tablequery

Azure.Data.Tables QueryAsync filtering erring with multiple filter conditions


In the old SDK Azure Table you could create FilterConditions like the following

var partitionFilter = TableQuery.GenerateFilterCondition("PartitionKey", QueryComparisons.Equal, symbolRoot);
    var startDateFilter =
        TableQuery.GenerateFilterConditionForDate("TradeDate", QueryComparisons.GreaterThanOrEqual, startDate);
    var endDateFilter =
        TableQuery.GenerateFilterConditionForDate("TradeDate", QueryComparisons.LessThanOrEqual, endDate);

    var filter = TableQuery.CombineFilters(
        TableQuery.CombineFilters(
            partitionFilter,
            TableOperators.And,
            startDateFilter),
        TableOperators.And, endDateFilter);

I'd love to be able to continue with that pattern but....

In the new Azure.Data.Tables SDK a lot has changed I see no methods to create filters with multiple parameters, even though the QueryAsync methods accepts a filter as a parameter. The reference material for it, found here shows how to create a single parameter filter, but not a multiple parameter filter.

This method below fails because the DateTime conversion apparently is not supported within the filtering

public async Task<List<EquityDataEntity>> GetEquityPriceDataXDaysBackAsync(string symbol, int daysBackFromToday)
{
    if (daysBackFromToday > 0)
    {
        daysBackFromToday *= -1; // put it proper form for Add... method
    }
    var data = await TableClient!
        .QueryAsync<EquityDataEntity>(u => u.PartitionKey == symbol.ToUpper()
                                  && Convert.ToDateTime(u.RowKey) >= DateTime.Now.AddDays(daysBackFromToday))  
        //Yes, unfortunately the RowKey is the date
        .ToListAsync();

    return data;
}

"Method ToDateTime not supported."

My two questions are:

  1. Is there a way to generate filters outside the method to use within the method as was done in prior SDK?

  2. How can I accomplish the date comparison in the Query method if the above is not doable and which requires a filter with multiple parameters (PartitionKey and date comparison)?

Update:

To note, the table structure is as follows The PartitionKey is a symbol like GLD, TSLA etc The RowKey is a Date like 2023-03-03

I have tried generating the filter like so

var filter = TableClient.CreateQueryFilter($"PartitionKey eq {symbol.ToUpper()} and RowKey gt {dateDataStarts}");

which produces this

PartitionKey eq 'GLD' and RowKey gt datetime'2021-10-20T17:42:01.3095704Z'

which according to the docs is what the OData filter should look like but it fails.

doing it with LINQ like so

var data = await TableClient!
        .QueryAsync<EquityDataEntity>(u => u.PartitionKey == symbol.ToUpper() && u.RowKey >= DateTime.Now.AddDays(daysBackFromToday).ToString())
        .ToListAsync();
    return data;

creates a VS2022 error

Operator 'operator' cannot be applied to operands of type 'string' and 'string'

replacing >= with gt doesn't work either

Update 2

The filter works now. The issue was the date was in the wrong format. This code now works as expected.

 var dateDataStarts = DateTime.Now.AddDays(daysBackFromToday).ToString("yyyy-MM-dd");
   
 var filter = TableClient.CreateQueryFilter($"PartitionKey eq {symbol.ToUpper()} and RowKey gt {dateDataStarts}");

which generates the OData filter as "PartitionKey eq 'GLD' and RowKey gt '2021-10-20'"


Solution

  • For the 1st one where you are specifying OData filter, you would need to convert your dateDataStarts variable to string. So your code would be something like:

    var filter =
        TableClient.CreateQueryFilter($"PartitionKey eq {symbol.ToUpper()} and RowKey gt {dateDataStarts.ToString("yyyy-MM-dd")}");
    

    For the 2nd one where you are using LINQ, you are correct. You cannot use >= for string comparison. You will need to use String.CompareTo. So your code would be something like:

    TableClient.QueryAsync<TableEntity>(u => u.PartitionKey == symbol && u.RowKey.CompareTo(dateDataStarts.ToString("yyyy-MM-dd")) >= 0)