Search code examples
loggingkubernetesstderrfluentd

Using stdout or stderr as a fluentd source?


I currently have a php-fpm container set up in Kubernetes to output error messages, exceptions,... to stderr and I wanted to know how I can use the output from this container as a source/input for fluentd.

Reading the output from a logfile and using f.e. tail as an input plugin, isn't an option in this case, so I'm looking for a different solution.


Solution

  • Docker writes your stdout/stderr logs into /var/lib/docker/containers and Kubernetes symlinks this directory into /var/log/containers.

    In Kubernetes world, the interface of accessing the logs of other containers is only this directory and you can't access their stdout or stderr directly.

    Take a look at Fluentd documentation on deploying to Kubernetes: https://docs.fluentd.org/container-deployment/kubernetes

    On DaemonSet manifests you can see that /var/log is mounted under the same name inside Fluentd container and tail plugin is used to read files inside this directory.

    So I'm saying your concern about having a 12-factor app is completely valid, and while you're writing your logs to stdout/stderr, Docker writes them to files and those files are the way Fluentd must use to access your logs. So use tail plugin and go ahead.

    By the way, this is okay with 12-factor app concerns:

    In staging or production deploys, each process’ stream will be captured by the execution environment, collated together with all other streams from the app, and routed to one or more final destinations for viewing and long-term archival. These archival destinations are not visible to or configurable by the app, and instead are completely managed by the execution environment.

    Also take a look at Kubernetes logging architecture.