XMLReader to XML file using XMLWriter WriteNode - Very Slow for Large XML

Question

In C# I use SqlCommand ExecuteXmlReader() to call a SQL Server stored procedure which uses 'For XML' to return large (1gb+) complex XML files (multiple hierarchy).

ExecuteXmlReader() returns an XmlReader which I wish to save to an XML file. To do this I use an XmlWriter to stream the data from the XMLReader to the file system.

using (XmlReader xmlFromDatabase = xmlReaderFromDatabase)
{
  var settings = new XmlWriterSettings {Encoding = Encoding.UTF8, Indent = true};
  using (XmlWriter outputXmlFileToDisk = XmlWriter.Create(fileDirectory + fileName, settings))
  {
    outputXmlFileToDisk.WriteNode(xmlFromDatabase, false);
  }
}

Side note: I can't load the entire XML into memory (XDocument) as it is too large.

My problem is that the WriteNode is very slow - it is taking hours to write the file. If I kill my application the XML file written on the disc is partially written since the file is being streamed out to node by node.

Is there a better way to save the XML faster from a XmlReader than a XMLWriter WriteNode?

(I know there is .ReadInnerXml() but this returns a string which is not good for the size of the XML)

After I export the file is need to transform it (I may use Saxon as the .net framework hasn't proved as performant as I'd like) and schema validate it through C#.

Answer 1

I found the solution, using XmlReader/XMLWriter seems to be a good approach to take (https://msdn.microsoft.com/en-us/library/ff647804.aspx) just that the XMLWriter-WriteNode by default validates the XML as it writes. Since we know the XML returned from SQL Server is XML valid and we also XSD validate all XML before sending it we don’t need to have the writer perform the validation as it writes. Passing the Settings object to the XMLWriter with CheckCharacters = false prevents the redundant validation and outputs the file in just a few minutes as apposed to hours.

var settings = new XmlWriterSettings{
            CheckCharacters = false,
            NewLineHandling = NewLineHandling.None,
            Indent = true,
            Encoding = Encoding.UTF8 };

using (var outputXmlFileToDisk = XmlWriter.Create(fileDirectory + fileName, settings))
{