How can I push data written by a Serilog File Sink on the disk to another Serilog sink?

Question

We are trying to load test our infrastructure of logstash/elastic. Since the actual logs are generated by a software that uses hardware, we are unable to simulate it at scale.

I am wondering if we can store the logs using file sink and later write a program that reads the log files and send data through the actual sink. Since, we are trying different setup, it would be great if we can swap different sinks for testing. Say http sink and elastic sink.

I thought of reading the json file one line at a time and then invoking Write method on the Logger. However I am not sure how to get the properties array from the json. Also, it would be great to hear if there are better alternatives in Serilog world for my needs.

Example parsing

            var events=  File.ReadAllLines(@"C:\20210520.json")
                .Select(line => JsonConvert.DeserializeObject<dynamic>(line));

            foreach (var o in objects)
            {
                DateTime timeStamp = o.Timestamp;
                LogEventLevel level = o.Level;
                string messageTemplate = o.MessageTemplate;
                string exception = o.Exception;
                var properties = (o.Properties as JObject);

                List<object> parameters = new List<object>();
                foreach (var property in properties)
                {
                    if(messageTemplate.Contains(property.Key))
                        parameters.Add(property.Value.ToString());
                }

                logInstance.Write(level, messageTemplate, parameters.ToArray());
                count++;
            }

Example Json Event written to the file

{"Timestamp":"2021-05-20T13:15:49.5565372+10:00","Level":"Information","MessageTemplate":"Text dialog with {Title} and {Message} displayed, user selected {Selected}","Properties":{"Title":"Unload Device from Test","Message":"Please unload the tested device from test jig","Selected":"Methods.Option","SinkRepository":null,"SourceRepository":null,"TX":"TX2937-002 ","Host":"Host1","Session":"Host1-2021.05.20 13.12.44","Seq":87321,"ThreadId":3}}

UPDATE Though this works for simple events,

1. it is not able to handle Context properties (there is a work around though using ForContext),
2. also it forces all the properties to be of type string and
3. not to mention that destucturing (@property) is not handled properly

Answer 1

If you can change the JSON format to Serilog.Formatting.Compact's CLEF format, then you can use Serilog.Formatting.Compact.Reader for this.

In the source app:

// dotnet add package Serilog.Formatting.Compact
Log.Logger = new LoggerConfiguration()
  .WriteTo.File(new CompactJsonFormatter(), "./logs/myapp.clef")
  .CreateLogger();

In the load tester:

// dotnet add package Serilog.Formatting.Compact.Reader
using (var target = new LoggerConfiguration()
    .MinimumLevel.Verbose()
    .WriteTo.Console()
    .CreateLogger())
{
    using (var file = File.OpenText("./logs/myapp.clef"))
    {
        var reader = new LogEventReader(file);
        while (reader.TryRead(out var evt))
            target.Write(evt);
    }
}

Be aware though that load testing results won't be accurate for many sinks if you use repeated timestamps. You should consider re-mapping the events you read in to use current timestamps.

E.g. once you've loaded up evt:

var current = new LogEvent(DateTimeOffset.Now,
                           evt.Level,
                           evt.Exception,
                           evt.MessageTemplate,
                           evt.Properties);
target.Write(current);