How to proxy app log-level without using the one passed with fluentd?

Question

So i have the following Openshift/Origin architecture installed following the official Openshift/Origin documentation We also want to use the Aggregated logging setup that comes out of the box, that's why was set by using strictly the Openshift aggregated logging documentation.

So far the feedback has been excellent but i have another challenge for which i will need some help. Developers want to specify by their own the log level that will be forwarded to the Elasticsearch. Current log level is set only thru FluentD.

Is there a way to set the log level by Deployment variable and make it pass Fluentd unchanged to Elasticsearch ?

The goal is to provide people a way to set by their own the log level that will be forwarded to Elasticsearch.

Answer 1

I afraid there is no way to do it by standard tools without adding a custom FluentD.

First of all, your FluentD in a cluster reading a container logs provided by Docker thru json-file logging driver:

By default, Fluentd reads from /var/log/messages and /var/log/containers/.log for system logs and container logs, respectively.

Even by using SystemD logging you will get the same result - logging level is set by Docker. Kubernetes also using that driver.

For Docker json-file driver you can set log-tags, which, theoretically, can help you filter logs. But it is impossible to set that options for a container in runtime by Kubernetes, so there are now way.

The only way I see how you can do it is to use sidecar container with custom logging agent. It will looks like that:

Using sidecar, you can run FluentD with custom configuration inside it and parse a log of your application with any modification, including of using environment variables as a log level.