syslog-ng - Passing FILENAME from client to server when using wildcard_file

Question

I am using syslog-ng, to remote log the application logs of multiple containers of the same image. I am using the source config as below.

source s_wild { wildcard-file(
    base-dir("/var/myapp/logs")
    filename-pattern("*")
    recursive(no)
    flags(no-parse)
    follow-freq(1)
); };

When I am using the logging in the local machine (for testing purposes), using the MACRO, ${FILE_NAME}, it works. But the filename is not being passed on, over network when testing with the remote server.

Aug  3 19:39:46 46fc878e92cf syslog-ng[2320]: Error opening file for writing; filename='', error='Is a directory (21)'

There are around 20-25 files and am looking for auto mapping of the filenames in both client and server side. Is it possible. Not sure how the wildcard_file maps to remote server. Logically it may not be possible. Still wondering on a solution.

I am wondering whether I can avoid manual 1-1 mapping by defining multiple source and destination or using log_prefix.

Answer 1

The $FILE_NAME macro works only if syslog-ng receives messages from a file or a wildcard-file source and it does not work over network(). A couple of options you have here to pass file names over network are:

1. Use the structured-data section of a RFC 5424 syslog message
2. Use template() with json-parser() to send messages from client-side and parse them on server side
3. Use ewmm() (Enterprise-wide message model) which supports delivery of structured messages

In the first method, sending the RFC5424-formatted (IETF-syslog) messages allows you to set the FILE_NAME in the SDATA field. Use the syslog() on the source and destination side instead of network() to send the messages using IETF syslog protocol. The source file wildcard can be defined like this. The whole configuration would be something like below:

syslog-ng client side

source s_wild {
        wildcard-file(
                base-dir("/var/log_syslog")
                filename-pattern("*")
                recursive(no)
                follow-freq(1)
                );

};

rewrite r_set_filename{
    set(
        "$FILE_NAME",
         value(".SDATA.file@18372.4.name")
        );
};

rewrite r_use_basename {
  subst(
        "/var/log_syslog/",
        "",
        value(".SDATA.file@18372.4.name")
        type("string")
        flags("prefix")
       );
};

destination d_container_logs {
        syslog(
              "192.168.10.48"
               transport("tcp")
               port(5141)
        );
};

log {source(s_wild); rewrite(r_set_filename); rewrite(r_use_basename); destination(d_container_logs);};

The r_set_filename gets the absolute path of file and we chop-off the path bit and retains only the filename using r_use_basename

syslog-ng server side

source s_network {
        syslog(
        transport("tcp")
        port(5141)
        keep_hostname(yes)
        );
};


destination d_container_logs {
    file(
        "/var/sys_log/${.SDATA.file@18372.4.name}"
        create_dirs(yes)
        );
};

log {source(s_network); destination(d_container_logs);};