MassTransit Job Consumer message with List<T> member not deserializing

I have successfully set up, configured, and run the MassTransit (v8.0.3) JobConsumers sample and am now trying to customize to my project. After modifying, I'm getting deserialization issues when submitting my job:

Newtonsoft.Json.JsonSerializationException: Cannot deserialize the current JSON array (e.g. [1,2,3]) into type 'MyNamespace.Foo' because the type requires a JSON object (e.g. {"name":"value"}) to deserialize correctly.
To fix this error either change the JSON to a JSON object (e.g. {"name":"value"}) or change the deserialized type to an array or a type that implements a collection interface (e.g. ICollection, IList) like List<T> that can be deserialized from a JSON array. JsonArrayAttribute can also be added to the type to force it to deserialize from a JSON array.
Path 'foos[0]'.
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.EnsureArrayContract(JsonReader reader, Type objectType, JsonContract contract)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, Object existingValue, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateList(IList list, JsonReader reader, JsonArrayContract contract, JsonProperty containerProperty, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, Object existingValue, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.Serialization.JsonSerializerProxy.DeserializeInternal(JsonReader reader, Type objectType)
   at MassTransit.Serialization.JsonConverters.InterfaceProxyConverter.CachedConverter`1.MassTransit.Serialization.JsonConverters.BaseJsonConverter.IConverter.Deserialize(JsonReader reader, Type objectType, JsonSerializer serializer) in /_/src/MassTransit.Newtonsoft/Serialization/JsonConverters/InterfaceProxyConverter.cs:line 31
   at MassTransit.Serialization.JsonConverters.BaseJsonConverter.ReadJson(JsonReader reader, Type objectType, Object existingValue, JsonSerializer serializer) in /_/src/MassTransit.Newtonsoft/Serialization/JsonConverters/BaseJsonConverter.cs:line 15
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.DeserializeConvertable(JsonConverter converter, JsonReader reader, Type objectType, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonSerializer.Deserialize[T](JsonReader reader)
   at MassTransit.Serialization.NewtonsoftObjectDeserializer.DeserializeObject[T](Object value, T defaultValue) in /_/src/MassTransit.Newtonsoft/Serialization/NewtonsoftObjectDeserializer.cs:line 34
   at MassTransit.Serialization.BaseSerializerContext.DeserializeObject[T](Object value, T defaultValue) in /_/src/MassTransit/Serialization/EnvelopeSerializerContext.cs:line 52
   at MassTransit.JobServiceEventExtensions.GetJob[TJob](ConsumeContext`1 context) in /_/src/MassTransit/JobServiceEventExtensions.cs:line 24
   at MassTransit.JobService.FinalizeJobConsumer`1.Consume(ConsumeContext`1 context) in /_/src/MassTransit/JobService/JobService/FinalizeJobConsumer.cs:line 45
   at MassTransit.Middleware.MethodConsumerMessageFilter`2.MassTransit.IFilter<MassTransit.ConsumerConsumeContext<TConsumer,TMessage>>.Send(ConsumerConsumeContext`2 context, IPipe`1 next) in /_/src/MassTransit/Middleware/MethodConsumerMessageFilter.cs:line 27
   at MassTransit.Middleware.LastPipe`1.Send(TContext context) in /_/src/MassTransit.Abstractions/Middleware/Middleware/LastPipe.cs:line 30
   at MassTransit.Consumer.DelegateConsumerFactory`1.Send[TMessage](ConsumeContext`1 context, IPipe`1 next) in /_/src/MassTransit/Consumers/Consumer/DelegateConsumerFactory.cs:line 29
   at MassTransit.Consumer.DelegateConsumerFactory`1.Send[TMessage](ConsumeContext`1 context, IPipe`1 next) in /_/src/MassTransit/Consumers/Consumer/DelegateConsumerFactory.cs:line 39
   at MassTransit.Middleware.ConsumerMessageFilter`2.MassTransit.IFilter<MassTransit.ConsumeContext<TMessage>>.Send(ConsumeContext`1 context, IPipe`1 next) in /_/src/MassTransit/Middleware/ConsumerMessageFilter.cs:line 46

I am using MassTransit.Newtonsoft for JSON support because I use it in my project and due to issues encountered when upgrading to MassTransit v8. It is configured here:

x.UsingRabbitMq((context, cfg) =>
{
    cfg.UseNewtonsoftJsonSerializer();
    cfg.UseNewtonsoftJsonDeserializer();

    ...
}

Here is my message definition (that replaces the sample's ConvertVideo), stripped down to the smallest failing case:

public interface ProcessFoos
{
    List<Foo> Foos { get; }
}

where Foo is defined as:

public class Foo
{
   public string FooID { get; set; } 
}

I'm initiating a new job from a different consumer by calling:

var foos = new List<Foo>
{
    new Foo { FooID = "foo" }
};

Response<JobSubmissionAccepted> response = await _processFoosClient.GetResponse<JobSubmissionAccepted>(new
{
    Foos = foos
});

My job consumer looks like this:

public class ProcessFoosJobConsumer : IJobConsumer<ProcessFoos>
{
    public async Task Run(JobContext<ProcessFoos> context)
    {
        // Never gets here due to the deserialization issue
        var processFoos = context.Job;
        ...
    }
}

Based on the MassTransit docs and everything I've found while researching, it seems like this should be supported and I'm probably just missing something simple.

Solution

When using Entity Framework with the built-in JobServiceDbContext, the Job property in the JobSaga table is serialized to JSON using System.Text.Json – which is not compatible with the serialization of Newtonsoft when it comes to nested object (the job message type is converted to a Dictionary<string,object>).

I don't know of a current fix at this point, since there isn't a way to dynamically specify the serializer based upon the incoming message types for the DbContext.

Or you could create your own mappings, and your own DbContext, with the Newtonsoft serializer methods instead of the ones used in v8.

Your best bet when using job consumers and the job service sagas with Cosmos or EF is to use System.Text.Json.