Search code examples
syslog-ng

how can i handle syslog-ng parser failure when using a patterndb

We parse millions of messages a day using syslog-ng, and are in the process of implementing patterndb.

Due to inconsistency in how the messages are composed, in a small percentage of cases, my patterns are insufficient to capture the fields of the message (spacing is off, or sometimes a field is missing altogether).

How can I deal with these cases? Ideally, the parser entry in my log destination would evaluate to false (like a filter) and it would be captured by my fallback log destination.

Solution

  • Try setting drop-unmatched(yes) (needs syslog-ng OSE 3.11 or later): 

    parser pattern_db {
    db-parser(
        file("/opt/syslog-ng/var/db/patterndb.xml")
        drop-unmatched(yes)
    );
    };

    Also, recent syslog-ng versions have several different parsers that might be better for certain log messages than patterndb, for example, JSON and key=value parsers.