Write Dynamic LINQ queries for sorting and projecting with EF Core

Question

I am having troubles trying to get valid expressions for "inner graph" members.

I wrote several code lines based on previous stack overflow answers. but I have issues finding a good "solution approach".

For example trying to express "a string" as a lambda field member...

I had refactored the ToMemberOf extension method that indeed does it BUT will fail for "inner graph" members.

i.e. only works fine only for "direct members".

var order1 = "FullName".ToMemberOf<Player>();   // will be converted to {e => Convert(e.FullName, Object)}

but for "inner graph" members i.e.

var order2 = "Catalog.Photo.FileName".ToMemberOf<Player>()

will fail with System.ArgumentException because the ToMemberOf extension method is not able to handle "dots" and "inner graph" members.

Then I tried to code a new extension method called ToExtendedMemberOf

It seems to work but fails when using EF Core.

i.e. the "inner graph" member "Catalog.Photo.FileName" will be converted to.

var order2 = "Catalog.Photo.FileName".ToExtendedMemberOf<Player>();  // will be converted to {e => new Player() {Catalog = new Catalog() {Photo = new Photo() {FileName = e.Catalog.Photo.FileName}}}}

but when I tried to use it with EF core I got provider specific errors.

i.e. for InMemory provider I got System.InvalidOperationException : Failed to compare two elements in the array

i.e. for SlqServer provider I got System.InvalidOperationException The LINQ expression could not be translated

Indeed I want the evaluation be executed on the server not on the client.

I don't know how can I refactor ToExtendedMemberOf or ToMemberOf to achieve those goals. I need your help here

I am pasting the whole unit text examples to allow a more global vision.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Linq.Expressions;
using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Diagnostics;
using Xunit;

namespace XUnitTestProject
{
    public static class QueryableExtensions
    {
        public static IQueryable<T> Select<T>(this IQueryable queryable, IEnumerable<string> fields) where T : class
        {
            var sourceType = queryable.ElementType;
            var resultType = typeof(T);
            var parameter = Expression.Parameter(sourceType, "e");
            var body = GetNewMember(typeof(T), parameter, fields.Select(f => f.Split('.')));
            var selector = Expression.Lambda(body, parameter);
            return queryable.Provider.CreateQuery<T>(Expression.Call(typeof(Queryable), "Select", new[] { sourceType, resultType }, queryable.Expression, Expression.Quote(selector)));
        }

        public static IQueryable<T> OrderBy<T>(this IQueryable queryable, IEnumerable<string> fields) where T : class
        {
            var sourceType = queryable.ElementType;
            var resultType = typeof(T);
            var parameter = Expression.Parameter(sourceType, "e");
            var body = GetNewMember(typeof(T), parameter, fields.Select(f => f.Split('.')));
            var selector = Expression.Lambda(body, parameter);
            return queryable.Provider.CreateQuery<T>(Expression.Call(typeof(Queryable), "OrderBy", new[] { sourceType, resultType }, queryable.Expression, Expression.Quote(selector))); // {e => new Player() {Catalog = new Catalog() {Photo = new Photo() {FileName = e.Catalog.Photo.FileName}}}}
        }

        private static Expression GetNewMember(Type targetType, Expression source, IEnumerable<string[]> memberPaths, int depth = 0)
        {
            var target = Expression.Constant(null, targetType);
            var bindings = memberPaths.GroupBy(path => path[depth]).Select(memberGroup =>
            {
                var memberName = memberGroup.Key;
                var targetMember = Expression.PropertyOrField(target, memberName);
                var sourceMember = Expression.PropertyOrField(source, memberName);
                var childMembers = memberGroup.Where(path => depth + 1 < path.Length);
                var enumerable = childMembers as string[][] ?? childMembers.ToArray();
                var targetValue = !enumerable.Any() ? sourceMember : GetNewMember(targetMember.Type, sourceMember, enumerable, depth + 1);
                return Expression.Bind(targetMember.Member, targetValue);
            });
            return Expression.MemberInit(Expression.New(targetType), bindings);
        }

    }

    public static class StringExtensions
    {
        public static Expression<Func<T, object>> ToMemberOf<T>(this string name) where T : class
        {
            var parameter = Expression.Parameter(typeof(T), "e");
            var propertyOrField = Expression.PropertyOrField(parameter, name);
            var unaryExpression = Expression.MakeUnary(ExpressionType.Convert, propertyOrField, typeof(object));

            return Expression.Lambda<Func<T, object>>(unaryExpression, parameter);
        }

        public static UnaryExpression ToExtendedMemberOf<T>(this string name) where T : class
        {
            var parameter = Expression.Parameter(typeof(T), "e");
            var body = GetNewExtendedMember(typeof(T), parameter, new[] { name.Split('.').ToArray() });
            var selector = Expression.Lambda(body, parameter);

            return Expression.Quote(selector);
        }

        private static Expression GetNewExtendedMember(Type targetType, Expression source, IEnumerable<string[]> memberPaths, int depth = 0)
        {
            var target = Expression.Constant(null, targetType);
            var bindings = memberPaths.GroupBy(path => path[depth]).Select(memberGroup =>
            {
                var memberName = memberGroup.Key;
                var targetMember = Expression.PropertyOrField(target, memberName);
                var sourceMember = Expression.PropertyOrField(source, memberName);
                var childMembers = memberGroup.Where(path => depth + 1 < path.Length);
                var enumerable = childMembers as string[][] ?? childMembers.ToArray();
                var targetValue = !enumerable.Any() ? sourceMember : GetNewExtendedMember(targetMember.Type, sourceMember, enumerable, depth + 1);
                return Expression.Bind(targetMember.Member, targetValue);
            });
            return Expression.MemberInit(Expression.New(targetType), bindings);
        }
    }

    public class DataContext : DbContext
    {
        public DbSet<Player> Players { get; set; }

        protected override void OnConfiguring(DbContextOptionsBuilder builder)
        {
            base.OnConfiguring(builder);

            if (!builder.IsConfigured)
            {
                builder.UseInMemoryDatabase(Guid.NewGuid().ToString());
                builder.ConfigureWarnings(w => w.Throw(RelationalEventId.QueryClientEvaluationWarning));
            }
        }
    }

    public class Player
    {
        public int Id { get; set; }
        public string FullName { get; set; }
        public int Age { get; set; }
        public Catalog Catalog { get; set; }
    }

    public class Photo
    {
        public int Id { get; set; }
        public string FileName { get; set; }
        public string Format { get; set; }
    }

    public class Catalog
    {
        public int Id { get; set; }
        public string CatalogName { get; set; }
        public string Color { get; set; }
        public Photo Photo { get; set; }
    }

    public class UnitTest
    {
        [Fact]
        public void Test()
        {
            var players = new[]
            {
                new Player
                {
                    Id = 1,
                    FullName = "FullName 01",
                    Age = 1,
                    Catalog = new Catalog
                    {
                        Id = 1,
                        CatalogName = "CatalogName 01",
                        Color = "Color 01",
                        Photo = new Photo {Id = 1, FileName = "FileName 01", Format = "Format 01"}
                    }
                },
                new Player
                {
                    Id = 2,
                    FullName = "FullName 02",
                    Age = 2,
                    Catalog = new Catalog
                    {
                        Id = 1,
                        CatalogName = "CatalogName 02",
                        Color = "Color 02",
                        Photo = new Photo {Id = 1, FileName = "FileName 02", Format = "Format 02"}
                    }
                },
                new Player
                {
                    Id = 3,
                    FullName = "FullName 03",
                    Age = 3,
                    Catalog = new Catalog
                    {
                        Id = 1,
                        CatalogName = "CatalogName 03",
                        Color = "Color 03",
                        Photo = new Photo {Id = 1, FileName = "FileName 03", Format = "Format 03"}
                    }
                },
                new Player
                {
                    Id = 4,
                    FullName = "FullName 04",
                    Age = 4,
                    Catalog = new Catalog
                    {
                        Id = 1,
                        CatalogName = "CatalogName 04",
                        Color = "Color 04",
                        Photo = new Photo {Id = 1, FileName = "FileName 04", Format = "Format 04"}
                    }
                },
                new Player
                {
                    Id = 5,
                    FullName = "FullName 05",
                    Age = 5,
                    Catalog = new Catalog
                    {
                        Id = 1,
                        CatalogName = "CatalogName 05",
                        Color = "Color 05",
                        Photo = new Photo {Id = 1, FileName = "FileName 05", Format = "Format 05"}
                    }
                },
            };

            using (var context = new DataContext())
            {
                context.Players.AddRange(players);
                context.SaveChanges();

                var queryable = context.Players as IQueryable<Player>;

                var result1 = queryable
                    .Select(p => new
                    {
                        p.Id,
                        p.FullName,
                        p.Catalog.CatalogName,
                        p.Catalog.Photo.FileName
                    })
                    .Where(a => a.Id > 1)
                    .OrderBy(a => a.FileName)
                    .ToArray();

                // This is OK to filter and order
                Expression<Func<Player, bool>> filter = p => p.Id > 1;  // will be converted to {p => (p.Id > 1)}
                var order1 = "FullName".ToMemberOf<Player>();           // will be converted to {e => Convert(e.FullName, Object)}
                var result2 = queryable
                    .Select<Player>(new[]
                    {
                        "Id",
                        "FullName",
                        "Catalog.CatalogName",
                        "Catalog.Photo.FileName"
                    })
                    .Where(filter)
                    .OrderBy(order1)
                    .ToArray();

                // HOW TO ENHANCE ToMemberOf to GET SUCH MEMBER
                // This line of code will fail with System.ArgumentException : 'Catalog.Photo.FileName' is not a member of type 'Player'
                // var order2 = "Catalog.Photo.FileName".ToMemberOf<Player>();  

                // I could do as..
                // But I could not use it in on the OrderBy clause below
                var order2 = "Catalog.Photo.FileName".ToExtendedMemberOf<Player>();  // will be converted to {e => new Player() {Catalog = new Catalog() {Photo = new Photo() {FileName = e.Catalog.Photo.FileName}}}}
                var result3 = queryable
                    .Select<Player>(new[]
                    {
                        "Id",
                        "FullName",
                        "Catalog.CatalogName",
                        "Catalog.Photo.FileName"
                    })
                    .Where(filter)
                    //.OrderBy(order2)
                    .ToArray();

                // I could do as.. 
                // But this line of code will with InMemory provider fails with System.InvalidOperationException : Failed to compare two elements in the array. System.ArgumentException : At least one object must implement IComparable.
                // But this line of code will with SqlServer provider fails with System.InvalidOperationException : Error generated for warning 'Microsoft.EntityFrameworkCore.Query.QueryClientEvaluationWarning: The LINQ expression 'orderby new Player() {Catalog = new Catalog() {Photo = new Photo() {FileName = e.Catalog.Photo.FileName}}} asc' could not be translated and will be evaluated locally.'.
                var result4 = queryable
                    .Select<Player>(new[]
                    {
                        "Id",
                        "FullName",
                        "Catalog.CatalogName",
                        "Catalog.Photo.FileName"
                    })
                    .Where(filter)
                    .OrderBy<Player>(new[]
                    {
                        "Catalog.Photo.FileName"
                    })
                    .ToArray();
            }
        }
    }
}

In addition to this.. I intend to use projectiond

I am pasting some images with differences on the returned objects, just to give a better understanding of what I want to achieve.

i.e result1 is a collection of anonymous objects as it should be and result2, result3 and result3 are collections of Player object where not all properties has been populated by EF as i asked to do using projection

See additional related links here below:

Dynamically build select list from linq to entities query

Sorting a list using Lambda/Linq to objects

Answer 1

Regarding building a member selector expression for what you call "inner graph" members (and I call nested members) from dot separated string name (a.k.a. member path).

Your method MemberOf:

public static Expression<Func<T, object>> ToMemberOf<T>(this string name) where T : class
{
    var parameter = Expression.Parameter(typeof(T), "e");
    var propertyOrField = Expression.PropertyOrField(parameter, name);
    var unaryExpression = Expression.MakeUnary(ExpressionType.Convert, propertyOrField, typeof(object));

    return Expression.Lambda<Func<T, object>>(unaryExpression, parameter);
}

can easily be adjusted to handle both direct and nested members. All you need is to change a single line of code:

var propertyOrField = Expression.PropertyOrField(parameter, name);

to

var propertyOrField = name.Split('.')
    .Aggregate((Expression)parameter, Expression.PropertyOrField);

so

var order2 = "Catalog.Photo.FileName".ToMemberOf<Player>();

will be converted to

{e => Convert(e.Catalog.Photo.FileName)}