dynamic training/test classes with ML.NET

Question

This is a follow up from the question Dynamic classes/objects ML.net's PredictionMoadel<TInput, TOutput> Train()

My system cannot use a predefined class at compile time, therefore I tried to feed a dynamic class into ML.NET like below

// field data type
public class Field
{
    public string FieldName { get; set; }
    public Type FieldType { get; set; }
}

// dynamic class helper
public class DynamicClass : DynamicObject
{
    private readonly Dictionary<string, KeyValuePair<Type, object>> _fields;

    public DynamicClass(List<Field> fields)
    {
        _fields = new Dictionary<string, KeyValuePair<Type, object>>();
        fields.ForEach(x => _fields.Add(x.FieldName,
            new KeyValuePair<Type, object>(x.FieldType, null)));
    }

    public override bool TrySetMember(SetMemberBinder binder, object value)
    {
        if (_fields.ContainsKey(binder.Name))
        {
            var type = _fields[binder.Name].Key;
            if (value.GetType() == type)
            {
                _fields[binder.Name] = new KeyValuePair<Type, object>(type, value);
                return true;
            }
            else throw new Exception("Value " + value + " is not of type " + type.Name);
        }
        return false;
    }

    public override bool TryGetMember(GetMemberBinder binder, out object result)
    {
        result = _fields[binder.Name].Value;
        return true;
    }
}

private static void Main(string[] args)
{
    var fields = new List<Field>
    {
        new Field {FieldName = "Name", FieldType = typeof(string)},
        new Field {FieldName = "Income", FieldType = typeof(float)}
    };

    dynamic obj1 = new DynamicClass(fields);
    obj1.Name = "John";
    obj1.Income = 100f;

    dynamic obj2 = new DynamicClass(fields);
    obj2.Name = "Alice";
    obj2.Income = 200f;

    var trainingData = new List<dynamic> {obj1, obj2};

    var env = new LocalEnvironment();
    var schemaDef = SchemaDefinition.Create(typeof(DynamicClass));
    schemaDef.Add(new SchemaDefinition.Column(null, "Name", TextType.Instance));
    schemaDef.Add(new SchemaDefinition.Column(null, "Income", NumberType.R4));
    var trainDataView = env.CreateStreamingDataView(trainingData, schemaDef);

    var pipeline = new CategoricalEstimator(env, "Name")
        .Append(new ConcatEstimator(env, "Features", "Name"))
        .Append(new FastTreeRegressionTrainer(env, "Income", "Features"));

    var model = pipeline.Fit(trainDataView);
}

and got the error:

No field or property with name 'Name' found in type 'System.Object

I tried generating the class using Reflection only to run into the same problem.

Is there a workaround?

Answer 1

Dynamic class doesn't actually create a class definition but it rather provides you with dynamic object.

I looked at the code for SchemaDefinition.Create() it needs an actual class definition to build the schema. So your options are to create and load a class definition dynamically.

You can create your class as string with all dynamic properties and compile it using Microsoft compiler services aka Roslyn. See here. This will generate an assembly (in memory as memory stream or on file system) with your dynamic type.

Now you are only half way there. To get your dynamic type from dynamic assembly you need to load it in the App Domain. See this post. Once the assembly is loaded you can use 'Activator.CreateInstance()' if it's same domain or if it's your custom domain then you would need yourDomain.CreateInstanceAndUnwrap() to create the object out of dynamically generated Class and to get the type use Assembly.GetType().

Few sample here, A little out of date but will get you on your feet if you are up for this. See CompilerEngine and CompilerService to compile and load the assembly.

Other options: Refelection.Emit() but it requires a great deal of IL level coding. See this post.