How to use NLog to log unspecified objects to file maintaining ELK Stack compatibility?

I have struggle for weeks to get NLog to log my communication data including unspecific(DataContracts) parameters to file in a format that are ELK-stack compatible. It need to be configured in runtime and its preferred if the output parameters can be limited like MAX chars or depth.

NLog have a built-in JSON serializer but it will only read properties without defaults, fields will be ignored as you can see here. It would be a big job to adapt my data model and I do not really think its the right way to go, NLog should not affect how the data model should look like.

There is a couple of ways to add a custom JSON serializer :

I could use SetupSerialization on each class(Datacontract) like this :

LogManager.Setup().SetupSerialization(s =>
   s.RegisterObjectTransformation<GetEntityViewRequest>(obj => 
       return Newtonsoft.Json.Linq.JToken.FromObject(obj)
   )
);

Because I want to log all communication data the entire data model will have to be registered, its huge job and not effective.

I could use a custom IValueFormatter but it can´t be added to just my communication NLog instance, it have to be added in globally to all loggers like this :
```
NLog.Config.ConfigurationItemFactory.Default.ValueFormatter = new NLogValueFormatter();
```

So the IValueFormatter needs to filter so it is only manipulate data from the communication logger. I probably need to wrap my data in a class with a flag that tells IValueFormatter where the data come from, It do however not feel like a optimal solution.

There are also problems to actually get NLog to put out the data that the ValueFormatter filter as you can see here. The ValueFormatter do still run but its the regular NLog JSON data that will end up in the file.

What I need from NLog is this :

Transform all communication data including parameters from object to string formatted so ELK stack read it.
Serialization depth or string MAX length on parameters to avoid data overflow
Configurable from NLog.config in runtime(as NLog is)
Affect only a specific NLogger instance

My data come in through a IParameterInspector, it is compiled into a special CallInformation class that also holds the parameters(type object). The parameters can be vary complex with several layers. The entire CallInforamtion object is sent to NLog like this :

_comLogger.Log(LogLevel.Info, "ComLogger : {@callInfo}", callInfo);

The Nlog.config looks like this right now :

<logger name="CommunicationLogger" minlevel="Trace" writeto="communicationFileLog"></logger>
<target xsi:type="File"
            name="communicationFileLog"
            fileName="${basedir}/logs/${shortdate}.log"
            maxArchiveDays="5"
            maxArchiveFiles="10">
      <layout xsi:type="JsonLayout" includeAllProperties="true" maxRecursionLimit="1">
      </layout>
</target>

What am I missing? Is there another log library that might support my needs better?

Solution

I think the suggestion of Rolf is the best - create a layout that will use JSON.NET. That one could do all fancy tricks, like serializing fields and handling [JsonIgnore].

A basic version will look like this:

using System.Collections.Generic;
using Newtonsoft.Json;
using NLog;
using NLog.Config;
using NLog.Layouts;

namespace MyNamespace
{
    /// <summary>
    /// Render all properties to Json with JSON.NET, also include message and exception
    /// </summary>
    [Layout("JsonNetLayout")]
    [ThreadAgnostic] // different thread, same result
    [ThreadSafe]
    public class JsonNetLayout : Layout
    {
        public Formatting Formatting { get; set; } = Formatting.Indented; // This option could be set from the XML config

        /// <inheritdoc />
        protected override string GetFormattedMessage(LogEventInfo logEvent)
        {
            var allProperties = logEvent.Properties ?? new Dictionary<object, object>();
            allProperties["message"] = logEvent.FormattedMessage;
            if (logEvent.Exception != null)
            {
                allProperties["exception"] = logEvent.Exception.ToString(); //toString to prevent too much data properties
            }

            return JsonConvert.SerializeObject(allProperties, Formatting);
        }
    }
}

and register the layout, I will use:

Layout.Register<JsonNetLayout>("JsonNetLayout"); // namespace NLog.Layouts

The needed config:

<target xsi:type="File"
            name="communicationFileLog"
            fileName="${basedir}/logs/${shortdate}.log"
            maxArchiveDays="5"
            maxArchiveFiles="10">
    <layout xsi:type="JsonNetLayout" />
</target>

Example

When logging this object:

public class ObjectWithFieldsAndJsonStuff
{
    [JsonProperty]
    private string _myField = "value1";

    [JsonProperty("newname")]
    public string FieldWithName { get; set; } = "value2";

    [JsonIgnore]
    public string IgnoreMe { get; set; } = "value3";
}

And this logger call:

logger
    .WithProperty("prop1", "value1")
    .WithProperty("prop2", objectWithFieldsAndJsonStuff)
    .Info("Hi");

This will result in:

{
  "prop1": "value1",
  "prop2": {
    "_myField": "value1",
    "newname": "value2"
  },
  "message": "Hi"
}

Unit test

All this above in an unit test - using xUnit

        [Fact]
        public void JsonNetLayoutTest()
        {
            // Arrange
            Layout.Register<JsonNetLayout>("JsonNetLayout");

            var xmlConfig = @"
<nlog xmlns=""http://www.nlog-project.org/schemas/NLog.xsd""
      xmlns:xsi=""http://www.w3.org/2001/XMLSchema-instance"" 
      throwExceptions=""true"">

  <targets>
        <target xsi:type=""Memory"" name=""target1"" >
            <layout xsi:type=""JsonNetLayout"" />
        </target>
  </targets>
  <rules>
    <logger name=""*"" minlevel=""Trace"" writeTo=""target1"" />
  </rules>
</nlog>
";

            LogManager.Configuration = XmlLoggingConfiguration.CreateFromXmlString(xmlConfig);

            var logger = LogManager.GetLogger("logger1");

            var memoryTarget = LogManager.Configuration.FindTargetByName<MemoryTarget>("target1");

            // Act
            var objectWithFieldsAndJsonStuff = new ObjectWithFieldsAndJsonStuff();
            logger
                .WithProperty("prop1", "value1")
                .WithProperty("prop2", objectWithFieldsAndJsonStuff)
                .Info("Hi");

            // Assert
            var actual = memoryTarget.Logs.Single();
            var expected =
@"{
  ""prop1"": ""value1"",
  ""prop2"": {
    ""_myField"": ""value1"",
    ""newname"": ""value2""
  },
  ""message"": ""Hi""
}";
            Assert.Equal(expected, actual);
        }