Transferring logs using syslog-ng `as is` without timestamp and hostname etc

Question

Background

1. Apache server running on a machine and producing logs into /var/log/httpd/error_log
2. Using syslog-ng to send log to a port 5140
3. Eventually it will be consumed by kafka producer to be send to a topic

Settings

options {                                                                                                                             
    flush_lines (0);                                                                                                                
    time_reopen (10);                                                                                                               
    log_fifo_size (1000);                                                                                                          
    long_hostnames (off);                                                                                                           
    use_dns (no);                                                                                                                   
    use_fqdn (no);                                                                                                                  
    create_dirs (no);                                                                                                               
    keep_hostname (no);                                                                                                             
};

source s_apache2 {
    file("/var/log/httpd/error_log" flags(no-parse));
}


destination loghost {
    tcp("*.*.*.*" port(5140)); 
}

Problem

1. syslog-ng prepends timestamp and hostname to the log data which is undesirable

   <13>Jan 10 11:01:03 hostname [Tue Jan 10 11:01:02 2017] [notice] Digest: generating secret for digest authentication ...
   <13>Jan 10 11:01:03 hostname [Tue Jan 10 11:01:02 2017] [notice] Digest: done
   <13>Jan 10 11:01:03 hostname [Tue Jan 10 11:01:02 2017] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.4.30 mod_ssl/2.2.15 OpenSSL/1.0.0-fips configured -- resuming normal operations
   

2. Desired output (Each log line as is from error_log file)

   [Tue Jan 10 11:01:02 2017] [notice] Digest: generating secret for digest authentication ...
   [Tue Jan 10 11:01:02 2017] [notice] Digest: done
   [Tue Jan 10 11:01:02 2017] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.4.30 mod_ssl/2.2.15 OpenSSL/1.0.0-fips configured -- resuming normal operations
   

Platform

• CentOS release 6.4 (Final)
• syslog-ng @version:3.2

PS Syslog-ng to Kafka Integration : Please let me know if anybody has tried this which will render my java Kafka producer redundant

Answer 1

when you use the flags(no-parse) option in syslog-ng, then syslog-ng does not try to parse the different fields of the message, but puts everything into the MESSAGE field of the incoming log message, and prepends a syslog header. To remove this header, use a template in your syslog-ng destination:

template t_msg_only { template("${MSG}\n"); };
destination loghost {
tcp("*.*.*.*" port(5140) template(t_msg_only) ); 
}

To use the Kafka destination of syslog-ng, you need a newer version of syslog-ng (I'd recommend 3.8 or 3.9). Peter Czanik has written a detailed post about installing new syslog-ng rpm for CentOS.