Combining multiple Select Expressions into dynamic class

Question

I want to create a system that allows a separate assembly (A plugin) to add properties to an object at runtime. The plugins are plugins, so they can be added/removed at any time.

The Base Entity

public class FooDto 
{
    public int Id { get; set; }
    public string Description { get; set; }
    ...
}

This data comes directly from the database using linq projection with EF Core that would look something like this:

DbContext.Foos.Select(foo => new FooDto {
    Id = foo.Id,
    Description = foo.Description,
    ...
});

The Plugin(s)

Plugins can create new tables + relationships that didn't used to exist before. Let's say there is a plugin that create a table called 'Bar' and we want to add Bar's Description to the FooDto.

public class PluginFoo : Foo
{
    public Bar Bar { get; set; }
}

And it would define an select expression like this somewhere:

pluginFoo => new PluginFooDto {
    BarDescription = pluginFoo.Bar.Description
}

I could make it so that each plugin runs completely independently and will fire off its own database query, but I would like to try to combine them all into 1 query.

Basically the only thing the plugin's expressions will share, is that the expression parameter will be of the same base class. The actual class used will likely be a derived class that will contain the additional data the plugin uses (as shown by PluginFoo).

Theoretically it is possible to create a SQL statement that combines both the base Select and the new select expressions.

Here comes my question, is it possible to actually create a system like this?

---

My thought process:

• Instead of doing a select, I'd use a custom extension method called 'ProjectToAndExtend' or something which is where I hook in all the extra data.
• The base DTO would implement an interface that has a property called 'ExtendedValues'. This could either be an IDictionary/object/dynamic.
• Each of the plugins would define a method that returns 'Expression>'
• Inside the 'ProjectToAndExtend', I'd look through each of the plugin expressions and get all the properties/projections and create a runtime (proxy?) class that contains a combination of all the plugin expression's.
• I'd then create a new expression which will project onto the runtime class using the combination of all the provided plugin projections.
• I'd then add this new expression to the original select into the 'ExtendedValues' property.

Now my knowledge of Expression building and runtime class creation is limited. I mostly want to know if something like this is even possible?

I don't expect anyone to actually give me a working example of this if it is possible. I just want to avoid spending hours upon hours learning about Expressions/Reflection.Emit just to find out that this kind of system flat out isn't possible.

If someone does have any better/different idea on how to do this, I'd be eager to listen.

Thanks in advance!

---

Edit:

For the sake of the question, let's assume that the database already has a complete schema for all the plugins. The question isn't meant to be about the step of modifying the schema, it's about querying the data.

I thought it might be helpful to see how the projects query their data, so here's an example of how the DbContexts are setup. Each plugin will have their own independent Context that only deals with the data it uses.

The Base Project

public class BaseDbContext : DbContext 
{
    public DbSet<Foo> Foos { get; set; }
}

In the Plugin Project,

public class PluginDbContext : DbContext 
{
    public DbSet<PluginFoo> Foos { get; set; }
}

The 2 DbContext's point to the exact same database and tables, but they have different scopes on the schema (Foo doesn't know about Bar, but PluginFoo does).

For my question; in the situation where all the schema is correct in the database, is it possible to append Bar's data into a select expression from DbSet<foo>?

If this isn't possible with Ef Core, would it be possible using Linq-Sql directly?

Answer 1

TL;DR It's possible, but I wouldn't ever recommend actually doing something like this. It's overly complex and not manageable. It also requires a bunch of runtime type generation + expression building.

I'm answering this question in case anyone find a single part of this helpful for an unrelated issue.

The Solution

A big issue is because DbSet is a class and not an interface. This means if you try to add an expression to a select that uses a property that isn't on the DbSet's type, it won't generate the sql.

To solve this, we need to generate a bunch of types at run time using TypeBuilder and only do queries through interfaces.

The basic idea is:

• For each DbSet type, create a new runtime type that contains ALL the properties you will ever need to query.
• Create a runtime dbContext that has a DbSet for each of the types created previously.
• Any time you want to query the database use an interface instead of directly accessing the DbSet (Basically the repository pattern).

Implementation Details

Runtime DbSet Types

To be able to generate the runtime types for the DbSets, I define a base type like so:

public class Foo
{
    public int Id { get; set; }
    public string Description { get; set; }
}

Then any plugins that want to extend this type would use an interface like so:

public interface IFooPlugin
{
   int BarId { get; set; }
   Bar Bar { get; set; }
}

You then need some way of linking IFooPlugin to Foo. If you do that, you can create a runtime type that inherits from Foo and implements IFooPlugin. You'd want to do interfaces for the plugins so you have implement multiple plugins in the type you're creating.

Your final dynamic type would look something like this:

public class DynamicFoo : IFooPlugin
{
    public int Id { get; set; }
    public string Description { get; set; }
    public int BarId { get; set; }
    public Bar Bar { get; set; }
}

Querying the dynamic types

In your controllers, you won't be able to use any class to query because you need the covariance you can get from interfaces. If you cast a DbSet<DynamicFoo> to IQueryable<Foo> it will be perfectly happy to generate sql that uses any of the properties on DynamicFoo. If you tried to do the same on a DbSet<Foo> you'd have problems.

So essentially you'll probably want an IDbContext that has a bunch of IRepository<T>, where IRepository<T> would essentially be a wrapper around DbSet.

Adding dynamic data

Now the last hard part is to actually add the dynamic data to your Sql projection. Instead of doing 'Select', create a new extension method called 'SelectAndExtend' or something. This is where we will find all the extra expressions to add.

You'll need some kind of interface defined that returns an Expression<Func<TSource, TResult>>. TSource can be Foo or IFooPlugin, or any type that is implemented by your final dynamic DbSet type. TResult can literally be any class.

For each of these Expression<Func<TSource, TResult>>'s, you'll want to get all the properties that they 'select into'. Using this, you will want to create yet another runtime type with all these properties. This becomes your DTO for the dynamic properties.

You can then build up a new expression using the ExpressionVisitor and copying all the expressions across. Originally I made the IQueryable actually return a new runtime type that has all the required properties, but this causes issues if you do an async method because Task's.

Instead, I added an extra property to the original DTO called 'ExtensionProperties' that is just an object. I can then select the dynamic DTO I generated straight into this new 'ExtensionProperties' property.

The Result

In the end this does actually work, but there are A LOT of problems with doing something like this. I was able to get it to generate a single Sql statement that uses properties on a type that is defined at run time. I could then add/remove Dll's to change the object that gets returned from the server.

I've trivialized a bunch of stuff and skipped over some other issues with it, but basically it isn't really something that'd be worth doing. Keep in mind, this only deals with querying plugin data, actually being able to commit the plugin data would be a completely different beast.

I never actually completed it fully because I satisfied my curiosity about whether it was possible, and also got far enough to realise it wasn't going to be something I'd want to use going forwards.

If anyone has any questions, let me know.