Recreating an UTF-8 StreamWriter on an open NamedPipeClientStream causes extra BOM characters on the server

Question

I have a NamedPipeServerStream that stays open, and a NamedPipeClientStream that stays open. The NamedPipeServerStream has an UTF-8 StreamReader that stays open. The NamedPipeClientStream has an UTF-8 StreamWriter that opens, writes a message, closes and then reopens to write another message. Both the StreamReader and the StreamWriter are opened with the leaveOpen flag set to true, so the StreamWriter doesn't close the NamedPipeClientStream when it is closed. So far, so good.

I have an issue where from the second time the StreamWriter is opened and sends a message, an extra BOM is read by the StreamReader. I have managed to recreate it in a small sample program. It is built in .NET 6. This is the code of the sample program:

using System.IO.Pipes;

namespace NamedPipeTest
{
    internal class EncodingTest
    {
        private static NamedPipeServerStream namedPipeServer = new NamedPipeServerStream("mypipe", PipeDirection.InOut, NamedPipeServerStream.MaxAllowedServerInstances, PipeTransmissionMode.Byte, PipeOptions.Asynchronous);
        private static NamedPipeClientStream namedPipeClient = new NamedPipeClientStream(".", "mypipe", PipeDirection.InOut, PipeOptions.Asynchronous);

        static void Main(string[] args)
        {
            Console.WriteLine("Hello! You are the client. Press any key to send a request.");

            Task[] tasks = new Task[]
            {
                Task.Run(() => Server()),
                Task.Run(() => Client())
            };

            Task.WaitAll(tasks);
        }

        static async Task Server()
        {
            if (!namedPipeServer.IsConnected)
            {
                namedPipeServer.WaitForConnection();
            }

            using (var reader = new StreamReader(namedPipeServer, System.Text.Encoding.UTF8, false, 1024, true))
            {
                while (true)
                {
                    string? line = await reader.ReadLineAsync();
                    Console.WriteLine($"Server: Received request: {line}");
                    if (!string.IsNullOrEmpty(line))
                    {
                        var lineBytes = System.Text.Encoding.UTF8.GetBytes(line);
                        var possibleBOMString = System.Text.Encoding.UTF8.GetString(new byte[] { lineBytes[0], lineBytes[1], lineBytes[2] });
                        if (possibleBOMString.Equals("\uFEFF"))
                        {
                            Console.WriteLine("Server: Whoa, what's up with the extra BOM?");
                        }
                    }
                }
            }
        }

        static async Task Client()
        {
            if (!namedPipeClient.IsConnected)
            {
                namedPipeClient.Connect(6000);
            }

            while (true)
            {
                using (var writer = new StreamWriter(namedPipeClient, System.Text.Encoding.UTF8, 1024, true))
                {
                    Console.ReadKey(true);
                    Console.WriteLine("Client: Received key press, sending request.");
                    string? line = "Hello!";
                    await writer.WriteLineAsync(line);
                    await writer.FlushAsync();
                    Console.WriteLine($"Client: Sent request: {line}");
                }
            }
        }
    }
}

The output of this program is:

Hello! You are the client. Press any key to send a request.
Client: Received key press, sending request.
Server: Received request: Hello!
Client: Sent request: Hello!
Client: Received key press, sending request.
Server: Received request: ?Hello!
Client: Sent request: Hello!
Server: Whoa, what's up with the extra BOM?
Client: Received key press, sending request.
Server: Received request: ?Hello!
Client: Sent request: Hello!
Server: Whoa, what's up with the extra BOM?

And this continues endlessly. So the line does not contain an extra BOM when it is sent, but rather after it has been read by the StreamReader. What is happening here?

Answer 1

1. System.Text.Encoding.UTF8 provides a BOM

It returns a UTF8Encoding object that provides a Unicode byte order mark (BOM).

2. StreamWriter.FlushAsync appends a BOM to the stream.

3. StreamWriter.Dispose will call Flush, so it also appends a BOM.

The solution is to construct a custom UTF8Encoding object:

var enc = new UTF8Encoding(encoderShouldEmitUTF8Identifier: false);
...
using (var writer = new StreamWriter(namedPipeClient, enc, 1024, true))