Serilog forward logs From Unity (or any app) to Elasticsearch instance via ASP.NET Web API

Question

I want to send logs from the Unity app to Elasticsearch. However Serilog.Sinks.Elasticsearch seems complicated (if even possible) to run inside unity. The possible solution is to have an ASP.NET Web API service that can retrieve logs from the app and log them further to Elasticsearch.

I could have Serilog.Sinks.Http in Unity application and Serilog.Sinks.Elasticsearch in Web API. Is it a good solution?

Or maybe simply send it without a logger? Right from the object I'm receiving. Are there special features inside Serilog.Sinks.Elasticsearch?

There must be flexible fields, no need to define the DTO on the server side. What could you recommend?

Answer 1

I decided to have a web server for forwarding logs to Elasticsearch. Then you need an ASP.NET core application with such a controller:

[ApiController]
[Route("[controller]")]
public class LogController : ControllerBase
{
    private readonly ILogger<LogController> _logger;
    readonly ElasticClient _client;
    private readonly string _indexFormat;

    public LogController(IConfiguration configuration, ILogger<LogController> logger)
    {
        _logger = logger;
        _indexFormat = configuration["LogForwarder:Elasticsearch:IndexFormat"];
        var settings = new ConnectionSettings(new Uri(configuration["LogForwarder:Elasticsearch:Endpoint"]))
            .BasicAuthentication(configuration["LogForwarder:Elasticsearch:Username"], configuration["LogForwarder:Elasticsearch:Password"]);

        _client = new ElasticClient(settings);
    }

    [HttpPost]
    public void Post([FromBody] JsonElement[] logs)
    {
        try
        {
            var logEntries = logs.Select(log => JsonConvert.DeserializeObject<LogEntry>(log.ToString())).ToList();

            foreach (var g in logEntries.GroupBy(s => s.TimeStamp.Date))
            {
                string indexName = string.Format(_indexFormat, g.Key);
                var indexExistsResponse = _client.Indices.Exists(indexName);
                if (!indexExistsResponse.Exists)
                {
                    var createIndexResponse = _client.Indices.Create(indexName, c => c
                                                                         .Map<LogEntry>(m => m
                                                                                            .AutoMap()));

                    if (!createIndexResponse.IsValid)
                        _logger.LogError("Failed to create index {IndexName} with error {Error}", indexName, createIndexResponse.ServerError.Error);
                }

                var resp = _client.IndexMany(g, indexName);
                if (!resp.IsValid)
                    _logger.LogError("Failed to index logs with error {ServerError}", resp.ServerError?.Error);
            }
        }
        catch (Exception e)
        {
            _logger.LogError(e, "Failed to process logs");
            throw;
        }
    }
}

Quite dirty, stuff might be in DI instead of created in the constructor. Also, I deserialize it in quite a tricky way with two serializers, because of the HTTP sink format. Feel free to edit this answer if you have a better solution.

then in Unity, somewhere in your code:

    Log.Logger = new LoggerConfiguration()
                 .MinimumLevel.Verbose()
                 .WriteTo.Unity3D() // https://github.com/KuraiAndras/Serilog.Sinks.Unity3D
                 .WriteTo.Http(requestUri: "https://my-logging-endpoint.com/Log", period: TimeSpan.FromSeconds(15), queueLimitBytes: 10_000_000)
                 .CreateLogger();