Search code examples
c#serializationnservicebus

NServiceBus serialization problems - doesn't seem to honoring serializer configuration

So I started having this problem with NServiceBus where it'll apparently accept messages and serialize them, but can't seem to deserialize them for processing. I've tried setting the serializer in settings a few different ways. Here are some of the exceptions (top of the stack) that I pulled from the error queue.

// not setting the serializer. Appears that NServiceBus serializes as XML (as documented) but tries to deserialize using NewtonSoft

"NServiceBus.ExceptionInfo.StackTrace" : "NServiceBus.MessageDeserializationException: An error occurred while attempting to extract logical messages from transport message 7cc546df-09ca-493e-bb6d-a6ce00a6bb10 ---> Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.\u000d\u000a at Newtonsoft.Json.JsonTextReader.ParseValue() in C:\Build\src\NServiceBus.Core\CircuitBreakers\CircuitBreaker.cs:line 0\u000d\u000a at NServiceBus.JsonMessageSerializer.Deserialize(Stream stream, IList`1 messageTypes) in C:\Build\src\NServiceBus.Core\Serializers\Json\JsonMessageSerializer.cs:line 108\u000d\u000a at ...

// with setting the serializer to the normal NServiceBus JsonSerializer (as in the tutorials). Looks like it can't deserialize JObject because it didn't include Newtonsoft in the scan. But why is it JObject at all?

"NServiceBus.ExceptionInfo.StackTrace" : "NServiceBus.MessageDeserializationException: An error occurred while attempting to extract logical messages from transport message ee5c8703-f509-40aa-a187-a6ce00a7fd1e ---> System.Exception: Could not find metadata for 'Newtonsoft.Json.Linq.JObject'.\u000d\u000aEnsure the following:\u000d\u000a1. 'Newtonsoft.Json.Linq.JObject' is included in initial scanning. \u000d\u000a2. 'Newtonsoft.Json.Linq.JObject' implements either 'IMessage', 'IEvent' or 'ICommand' or alternatively, if you don't want to implement an interface, you can use 'Unobtrusive Mode'.\u000d\u000a at NServiceBus.Unicast.Messages.MessageMetadataRegistry.GetMessageMetadata(Type messageType) in ...

// with setting the serializer to NewtonSoftSerializer it appears to be the same error "NServiceBus.ExceptionInfo.StackTrace" : "NServiceBus.MessageDeserializationException: An error occurred while attempting to extract logical messages from transport message 603e07e7-7877-4d71-90b3-a6ce00a9370b ---> System.Exception: Could not find metadata for 'Newtonsoft.Json.Linq.JObject'.\u000d\u000aEnsure the following:\u000d\u000a1. 'Newtonsoft.Json.Linq.JObject' is included in initial scanning. \u000d\u000a2. 'Newtonsoft.Json.Linq.JObject' implements either 'IMessage', 'IEvent' or 'ICommand' or alternatively, if you don't want to implement an interface, you can use 'Unobtrusive Mode'.\u000d\u000a at NServiceBus.Unicast.Messages.MessageMetadataRegistry.GetMessageMetadata(Type messageType) in ...

This feels like one of or all of a couple things. My thoughts based on what I'm seeing are:

  • For some reason, when I set either json serializer, objects are being serialized using the same method (not sure what)
  • Regardless of what serializer I set (or even if I mirror by adding the same serializer to deserializers), it tries to deserialize it using Newtonsoft

Other points I think could be useful:

  • From the headers included in message (from the error queue):
    • When I use either json serializer, the content type is application/json and the inner exception is Newtonsoft.Json.JsonReaderException
    • When I don't set the serializer, the content type is text/xml and the inner exception is Newtonsoft.Json.JsonReaderException
  • This is being hosted in a WebAPI application
    • Maybe this is affecting serializer selection?
  • The message classes are in a shared pcl library. Instead of implementing the IMessage (etc) interface, I've included them in the conventions during config (unobtrusive mode).
  • This was working fine for a while and then just started throwing these errors this week. I can't find anything that I've changed that are in any way related to NServiceBus.
  • I used to have this problem with Commands, but when I added the Newtonsoft nuget library for NServiceBus, commands (Send()) stopped throwing this error. However now events (Publish()) are still throwing the same exception
  • I saw in a different thread, someone suggested looking at the json body in the message to see if the serialized body had a $type property. It does not.

Any insight would be helpful. Why aren't the serializers lining up?

Here are some miscellaneous bits:

Configuration:

public static void Initialize(
  string endpointName, 
  string instanceDescriminator, 
  IWindsorContainer container)
{
    _endpointConfiguration = new EndpointConfiguration(ServiceEndpoint.Surveys);
    _endpointConfiguration.SendFailedMessagesTo("error");
    _endpointConfiguration.MakeInstanceUniquelyAddressable(instanceDescriminator);
    _endpointConfiguration.UseContainer<WindsorBuilder>(customizations =>
    {
        customizations.ExistingContainer(container);
    });

    _endpointConfiguration.UsePersistence<NHibernatePersistence, StorageType.Sagas>();
    _endpointConfiguration.UsePersistence<NHibernatePersistence, StorageType.Subscriptions>();
    _endpointConfiguration.UsePersistence<NHibernatePersistence, StorageType.Timeouts>();
    _endpointConfiguration.UsePersistence<NHibernatePersistence, StorageType.Outbox>();
    _endpointConfiguration.UsePersistence<NHibernatePersistence, StorageType.GatewayDeduplication>();
    _endpointConfiguration.EnableOutbox();
    _endpointConfiguration.UseTransport<SqlServerTransport>();
    _endpointConfiguration.EnableInstallers();

    NServiceBusConventions.SetDasConventions(_endpointConfiguration);
}

Conventions:

public static void SetDasConventions(EndpointConfiguration config)
{
    var conventions = config.Conventions();

    conventions.DefiningCommandsAs(type => type.Namespace != null && type.Namespace.StartsWith("DAS.Infrastructure.Messaging.Command"));
    conventions.DefiningEventsAs(type => type.Namespace != null && type.Namespace.StartsWith("DAS.Infrastructure.Messaging.Event"));
    conventions.DefiningMessagesAs(type => 
        type.Namespace != null && 
        (type.Namespace.StartsWith("DAS.Infrastructure.Messaging.Message") || type.Namespace.StartsWith("DAS.Infrastructure.Messaging")));
    conventions.DefiningEncryptedPropertiesAs(property => property.Name.StartsWith("Encrypted"));
    conventions.DefiningDataBusPropertiesAs(property => property.Name.EndsWith("DataBus"));
    conventions.DefiningExpressMessagesAs(type => type.Name.EndsWith("Express"));
    conventions.DefiningTimeToBeReceivedAs(type => type.Name.EndsWith("Expires") ? TimeSpan.FromSeconds(30) : TimeSpan.MaxValue);
}
Solution

  • As it turns out, the problem seems to be that a downstream subscriber had an older revision of the message that was having a problem. The objects were identical but the assembly revision gets updated on each build on our build server and then pushed to Nuget (e.g. 1.0.0.45621). Supposedly, NServiceBus shouldn't have a problem as long as the major revision has not changed and that it can still find the object with the name in the specified namespace. Evidently, there is some nuance to that, because as soon as I updated the library in the other component, the errors stopped.

    I got the idea from this thread https://groups.google.com/forum/#!topic/particularsoftware/lc7shFVR46k

    Where a dev at Particular mentioned that NSB needs to get away from using FQAN in the headers. I looked at the headers in the message which were included in the error and I noticed that the origin was the service I was working on and the FQAN of the enclosed message types showed a revision number that wasn't the same as the one being used in that service. So I opened up the only other service currently on the bus and found that one matched. I updated it and all was good.

    I followed the issue mentioned there, but it looked like that closed it in lieu of some other thing that I didn't quite get. At any rate, that was my solution. I'm still looking for clarification, or I will open a ticket with them because this is not sustainable in the future. Developers will be working in that shared library constantly, and we can't be updated libraries in every microservice on the platform (which will be upwards of 20 in short time) every time a change is made.