Consuming an External API that has a complex filter syntax

Question

I need to List/Get/Update/Create/Destroy (ie perform CRUD activites) on data from an external REST API.

This API has a custom filter syntax, which looks like this:

{{BaseUrl}}/V1.0/<Entity>/query?search={"filter":[{"op":"eq","field":"id","value":"68275"}]}

This filter syntax is quite flexible, and basically allows you to do fieldA == x AND/OR fieldB != y queries.

id <= 1000 && Title == "Some title"

{
    "filter": [
        {
            "op": "le",
            "field": "id",
            "value": 1000
        },
        {
            "op": "eq",
            "field": "Title",
            "value": "Some title"
        }
    ]
}

firstname == "john" || lastname != "Jones"

{
    "filter":  [
        {
            "op": "or",
            "items": [
                {
                    "op": "eq",
                    "field": "firstname",
                    "value": "John"
                },
                {
                    "op": "ne",
                    "field": "lastname",
                    "value": "Jones"
                }
            ]
        }
    ]
}  

If you're curious, it's the Autotask API: https://ww3.autotask.net/help/DeveloperHelp/Content/APIs/REST/General_Topics/REST_Swagger_UI.htm

At the moment, I have some classes which can convert to the first example query id <= 1000 && Title == "Some title".

    public interface IAutotaskFilter
    {
        string Field { get; }
        string Value { get; }
        string ComparisonOperator { get; }
    }
    
    public interface IAutotaskQuery
    {
        void AddFilter(IAutotaskFilter autotaskFilter);
        void AddFilters(IList<IAutotaskFilter> filters);
        void RemoveFilter(IAutotaskFilter autotaskFilter);
    }

The problem is that this violates the clean architecture I have.

Clean Architecture Example from Microsoft

If I use these classes (through the above interfaces), then my Application Layer will depend on an implementation detail of my Infrastructure Layer. In other words, my business logic will know how to construct queries for this specific external API.

As far as I can tell, I have a few options:

• Implement a custom LINQ provider that will translate linq queries to this syntax. From what I can tell, this is very difficult. I didn't find any recent information on this. Any libraries that have tried haven't been touched in at least 7 years. If this is possible, I would love to do it. Having that fluent syntax for queries is very appealing to me.

• Suck it up and use as is.

• Cache API entities locally and do cache invalidation using their webhooks 'updated'/'created'/'destroyed' events. That's a lot of processing, data transfer and complexity, so probably not worth it.

I am hoping there's a better option, that I haven't found. Please let me know.

If it helps, the Autotask API provides an OpenAPI v1 doc.

Answer 1

Have a look at the Expression class and ExpressionVisitor in the system.linq.expressions namespace https://learn.microsoft.com/en-us/dotnet/api/system.linq.expressions?view=net-5.0

It is the essence of LINQ, but without the SQL database part. It allows you to write expressions in C# and translate them to any format. See code example below. By assigning a lambda expression like (bo) => bo.MyID == 1 || bo.MyID == 3 to and Expression type the raw expression becomes available in code.

That is a feature of the C# compiler which LINQ uses to capture the query before it is compiled to byte code.

Change the TextWriter part of the code below to write a string in the format required by the API.

class BObject
{
    public int MyID;
}

class MyExpressionVisitor : ExpressionVisitor
{
    private TextWriter writer;

    public MyExpressionVisitor(TextWriter writer)
    {
        this.writer = writer;
    }

    protected override Expression VisitBinary(BinaryExpression node)
    {
        writer.WriteLine("Binary node " + node.NodeType);
        return base.VisitBinary(node);
    }
    protected override Expression VisitConstant(ConstantExpression node)
    {
        writer.WriteLine("Constant node " + node.NodeType + " Value " + node.Value);
        return base.VisitConstant(node);
    }

    protected override Expression VisitMember(MemberExpression node)
    {
        writer.WriteLine("Constant node " + node.NodeType + " Member " + node.Member);
        return base.VisitMember(node);
    }
}

class Program
{
    static void Main(string[] args)
    {
        Expression<Func<BObject, bool>> query = (bo) => bo.MyID == 1 || bo.MyID == 3;
        var visitor = new MyExpressionVisitor(Console.Out);

        visitor.Visit(query.Body);

        Console.ReadKey();
    }
}

Output:

Binary node OrElse
Binary node Equal
Constant node MemberAccess Member Int32 MyID
Constant node Constant Value 1
Binary node Equal
Constant node MemberAccess Member Int32 MyID
Constant node Constant Value 3