Log4j2 converting multiline logs stacktraces into single line

Question

I'm trying to push the logs my Elasticserver logs to rsys and then FLuentd. For this the stacktrace error logs should be in one line.

It was multiline before

443 [2022-08-05T07:45:38,068][ERROR][o.e.i.g.GeoIpDownloader  ] [techsrv01] exception during geoip databases update
   444  org.elasticsearch.ElasticsearchException: not all primary shards of [.geoip_databases] index are active
   445      at org.elasticsearch.ingest.geoip.GeoIpDownloader.updateDatabases(GeoIpDownloader.java:137) ~[ingest-geoip-7.17.5.jar:7.17.5]
   446      at org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:284) [ingest-geoip-7.17.5.jar:7.17.5]
   447      at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:100) [ingest-geoip-7.17.5.jar:7.17.5]
   448      at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:46) [ingest-geoip-7.17.5.jar:7.17.5]
   449      at org.elasticsearch.persistent.NodePersistentTasksExecutor$1.doRun(NodePersistentTasksExecutor.java:42) [elasticsearch-7.17.5.jar:7.17.5]
   450      at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) [elasticsearch-7.17.5.jar:7.17.5]
   451      at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.17.5.jar:7.17.5]
   452      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
   453      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
   454      at java.lang.Thread.run(Thread.java:833) [?:?]

After changing the pattern layout in log4j2.properties in the below format. I'm able to get it into two lines. But I'm not able to convert it more into single line.

appender.rolling_old.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}][%node_name] %marker %m %n %throwable{separator(|)}

---

2028     [2022-08-05T11:04:40,810][ERROR][o.e.i.g.GeoIpDownloader  ][techsrv01]  exception during geoip databases update
      2029   ElasticsearchException[not all primary shards of [.geoip_databases] index are active]| at org.elasticsearch.ingest.geoip.GeoIpDownloader.updateDatabases(GeoIpDownloader.java:137)|    at org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:284)|  at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:100)|  at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:46)|   at org.elasticsearch.persistent.NodePersistentTasksExecutor$1.doRun(NodePersistentTasksExecutor.java:42)|   at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777)|  at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)| at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)|   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)|   at java.base/java.lang.Thread.run(Thread.java:833)[2022-08-05T11:04:41,171][INFO ][o.e.c.r.a.AllocationService][techsrv01]  Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.ds-ilm-history-5-2022.07.18-000001][0], [.kibana-event-log-7.17.5-000001][0], [.geoip_databases][0], [.ds-.logs-deprecation.elasticsearch-default-2022.07.18-000001][0]]]).

How can we achieve this using Log4j2 layout pattern?

Answer 1

Instead of converting the logs in one single line using log4j2. I used the default log pattern. And I ditched RSYS instead used FluentD directly to parse the logs, below configuration will only filter warn and error and not info

td-agent.conf

<source>
  @type tail
  path /var/log/elasticsearch/elasticdemo.log
  pos_file /var/log/elasticsearch/elasticdemo.log.pos
  tag elastic_error_self
  <parse>
    @type multiline
    format_firstline /(\d{4})-(\d\d)-(\d\d)/
    format1 /^(?<timestamp>\[.*?\])(?<logLevel>\[.*?\])(?<service>\[.*?\]) (?<node_name>\[.*?\])(?<message>.*)/
  </parse>
</source><filter **>
  @type grep
  <exclude>
    key logLevel
    pattern /INFO/
    # or, to exclude all messages that are empty or include only white-space
  </exclude>
</filter><match elastic**>
  @type elasticsearch
  host elasticIP/lbip/vmip #where elastic is installed
  port 9200
  index_name elastic_error_self
  include_timestamp true #connection configs
    reconnect_on_error true
    reload_on_failure true
    slow_flush_log_threshold 90  # buffer configs
  <buffer>
    @type file
    path /data/opt/fluentd/buffer/elastic_error_self
    chunk_limit_size 32MB
    total_limit_size 20GB
    flush_thread_count 8
    flush_mode interval
    retry_type exponential_backoff
    retry_timeout 10s
    retry_max_interval 30
    overflow_action drop_oldest_chunk
    flush_interval 5s
  </buffer>
</match>