Strange Behavior in HttpWebRequest when Saving XDocument to Body

Question

I discovered some strange behavior when attempting to sent a POST request using the HttpWebRequest class targeting .NET 4. This may only have to do with XDocument object I'm attempting to send misbehaving, or I'm not understanding the difference between the Save() method and ToString() method of XDocument.

Calling the following will result in the ContentLength of the request to be set to 316 for my specific XDocument when saved to the stream:

using(var stream = request.GetRequestStream())
{
    xDoc.Save(stream, SaveOptions.DisableFormatting);
}

This was causing the server to not understand my request body. Using fiddler I captured the XML being uploaded and then copy & pasted the raw XML into Fiddler's composer window. Upon sending this request to the server it surprisingly responded without issue. When I inspected the headers I noticed that Fiddler calculated 313 for ContentLength.

I attempted a few experiments to get to the bottom of this such as setting the ContentLength value first (since I knew what value should be for this specific request), but saving to the stream would always result in an exception stating that data is longer than the expected ContentLength. Finally, I settled on this code workaround for stringifying the XDocument which causes the HttpWebRequest to calculate the correct ContentLength:

using(var stream = request.GetRequestStream())
using(var writer = new StreamWriter(stream))
{
    writer.Write(xDoc.ToString(SaveOptions.DisableFormatting));
}

The above calculates 313 and allows the server to respond correctly.

Why would saving directly to the stream vs writing data from the ToString() method cause HttpWebRequest to calculate different ContentLength header values? Is this a bug/known issue?

EDIT: As requested here's a MCVE to illustrate this (the API of the program I'm connecting to is awful, my apologies for their mess):

XDocument xDoc = new XDocument(
    new XDeclaration("1.0", "UTF-8", "yes"),
    new XElement("PVDM_HTTPINTERFACE",
        new XElement("FUNCTION",
            new XElement("NAME", "LoginUserEx3"),
            new XElement("PARAMETERS",
                new XElement("ENTITYID", 1),
                new XElement("USERNAME", "testadmin"),
                new XElement("PASSWORD", "12345"),
                new XElement("CLIENTPINGABLE", false),
                new XElement("REMOTEAUTH", false)
            )
        )
    )
);

HttpWebRequest requestWrittenWithString = (HttpWebRequest)WebRequest.Create("http://localhost");
requestWrittenWithString.Method = "POST";
HttpWebRequest requestWrittenWithSave = (HttpWebRequest)WebRequest.Create("http://localhost");
requestWrittenWithSave.Method = "POST";

//Save directly to stream
using(var stream = requestWrittenWithSave.GetRequestStream())
{
    xDoc.Save(stream, SaveOptions.DisableFormatting);
}

long saveContentLength = requestWrittenWithSave.ContentLength; //316

//StreamWriter with ToString() method
using(var stream = requestWrittenWithString.GetRequestStream())
using(var writer = new StreamWriter(stream))
{
    writer.Write(xDoc.ToString(SaveOptions.DisableFormatting));
}

long toStringContentLength = requestWrittenWithString.ContentLength; //313

Answer 1

Based on the incomplete code example provided (which does not even compile, never mind produce exactly the results described, since the second scenario uses an undeclared variable requestDoc and does not emit the XML declaration to the stream), I am reasonably confident that you are in fact getting the UTF8 BOM in the stream in your first scenario.

I'm not aware of a setting in the XDocument class itself that would control this. But it's simple enough to explicitly specify the TextWriter to use for the output, and the default encoding for StreamWriter doesn't include a BOM (as you have already demonstrated in the second scenario).

Something like this should fix your problem in the first scenario:

using (var stream = requestWrittenWithSave.GetRequestStream())
using (TextWriter writer = new StreamWriter(stream))
{
    xDoc.Save(writer, SaveOptions.DisableFormatting);
}

Of course, you could instead just use the XDocument.ToString() method, as in your second example. Either way is fine.

For future reference, please re-read the information at How to create a Minimal, Complete, and Verifiable example to see what is meant by "MCVE". You can save a lot of duplicated effort on the part of answerers, and increase the likelihood of an answer at all, if you provide a good MCVE.