Two-way pipe communication failing with XmlSerializer

Question

I have the following NamedPipe server (running in .NET Framework):

private void PipeListenerThread()
{
    PipeSecurity pipeSecurity = new PipeSecurity();
    pipeSecurity.AddAccessRule(new PipeAccessRule("Everyone", PipeAccessRights.ReadWrite, System.Security.AccessControl.AccessControlType.Allow));
    using(var pipeStream = new NamedPipeServerStream(Common.Constants.PipeName, PipeDirection.InOut, 1, PipeTransmissionMode.Message, PipeOptions.None, 1024, 1024, pipeSecurity, System.IO.HandleInheritability.None))
    {
        var serializer = new XmlSerializer(typeof(Common.Message), new Type[] {
            typeof(Common.ReadRequest),
            typeof(Common.ReadResponse)});

        while (running)
        {
            pipeStream.WaitForConnection();

            try
            {
                Common.Message message = serializer.Deserialize(pipeStream) as Common.Message;
                if (message is Common.ReadRequest readRequest)
                {
                    var result = GetData(readRequest);

                    var reply = new Common.ReadResponse { Value = result };

                    serializer.Serialize(pipeStream, reply);
                    pipeStream.WaitForPipeDrain();
                }
            }
            catch (Exception ex)
            {
                Debug.WriteLine(ex);
            }
            pipeStream.Disconnect();
        }
    }
}

And the following client in LinqPad (.NET 8):

using (var pipe = new NamedPipeClientStream(".", Common.Constants.PipeName, PipeDirection.InOut))
{
    var serializer = new XmlSerializer(typeof(Common.Message), new[] { typeof(Common.ReadRequest), typeof(Common.ReadResponse) });

    var readRequest = new Common.ReadRequest { Foo = 0, Bar = 0};
    pipe.Connect();
    serializer.Serialize(pipe, readRequest);
    pipe.WaitForPipeDrain();
    serializer.Deserialize(pipe).Dump();
}

If I comment out the server's Serialize and WaitForPipeDrain and the client's Deserialize lines, it will send the request just fine, but as soon as I try to send a response through the pipe, the client freezes on its Deserialize line.

What am I doing wrong?

Answer 1

One problem you are having is that, by default, the XmlReader used by XmlSerializer will try to validate that the input stream is a well-formed XML document that has exactly one root element. And to do that, it must try to read past the current object in your PipeStream. But since nothing has yet been written to the stream by your pipe writer, this hangs.

You have a couple of options to avoid this. Firstly, you could use message framing with length prefixing to delimit each XML object in the pipe stream. See e.g.:

• How to create TCP message framing for the stream

With this approach, each read and write would read no more than the amount specified by the length prefix, copying the received bytes into some MemoryStream for subsequent deserialization.

Or, since you are using PipeTransmissionMode.Message, you could copy full incoming message into some MemoryStream, then deserialize that instead.

This approach could be used with any serialization format such as JSON, not just XML.

Alternatively, you could create an XmlReader with ConformanceLevel.Fragment and use ReadSubtree() to read just the next fragment. First create the following extension methods:

public static partial class XmlExtensions
{
    public static T? DeserializeFragment<T>(this XmlSerializer? serializer, Stream stream)
    {
        XmlReaderSettings settings = new()
        {
            ConformanceLevel = ConformanceLevel.Fragment,
            CloseInput = false,
        };
        using var reader = XmlReader.Create(stream, settings);
        return serializer.DeserializeFragment<T>(reader);
    }
    
    public static T? DeserializeFragment<T>(this XmlSerializer? serializer, XmlReader reader)
    {
        serializer ??= new XmlSerializer(typeof(T));
        reader.MoveToContent();
        using var subReader = reader.ReadSubtree();
        return (T?)serializer.Deserialize(subReader);
    }
}

Then use them like so:

var reply = serializer.DeserializeFragment<Common.Message>(pipe);

Notes:

• While DeserializeFragment<T>(this XmlSerializer? serializer, Stream stream) will only attempt to read as far as the end of the current fragment, there is a chance that additional content might be read from the incoming stream due to buffering. Since your pipe stream only contains a single object, this should not be a problem, but if it is you would need to use a single XmlWriter along with the DeserializeFragment<T>(this XmlSerializer? serializer, XmlReader reader) overload.

• I notice you are using the XmlSerializer(Type, Type[]) constructor. If you use this constructor (or any other constructor other than new XmlSerializer(Type) or new XmlSerializer(Type, String)) you must statically cache and reuse the serializer as explained in the documentation. See also Memory Leak using StreamReader and XmlSerializer.

Mockup fiddle demonstrating the problem here and the fix here.