How to silence "Configuration property rebalance_cb is a consumer property and will be ignored by this producer instance"

Question

I am running a docker container with alpine, nginx and php. On it I have a symfony application running, which is using "flix-tech/avro-serde-php" and "koco/messenger-kafka". My Docker file looks like this:

[...]

ARG LIBRDKAFKA_GIT_SHA1=1f7417d4796e036b8c19f17373f8290ff5c7561f
RUN apk add --update --no-cache alpine-sdk bash python3 autoconf openssl-dev \
  && git clone -o ${LIBRDKAFKA_GIT_SHA1} https://github.com/edenhill/librdkafka.git /tmp/librdkafka \
  && cd /tmp/librdkafka/  \
  && ./configure \
  && make  \
  && make install

# php-rdkafka should be compiled using the same php module as result we are passing
# --with-php-config /usr/local/bin/php-config
ARG PHPCONF_PATH=/usr/local/etc/php/php.ini.d
ARG RDKAFKA_PHP_GIT_SHA1=abd6b6add8e0b983c27245a59981a9a4b5389139
RUN apk add --update --no-cache pcre-dev \
    && pecl install rdkafka

[...]

That is all fine and dandy. However the container is only sending data to kafka, since it is a producer. It isn't a consumer. So when the data is sent to kafka the following bash entry is generated: Configuration property rebalance_cb is a consumer property and will be ignored by this producer instance which then after some steps end up in datadog, my logging system as an error, which it isn't.

So my question is how I can get rid of this message? Is there a configuration setting, telling the system that it is only a producer? I didn't find the property rebalance_cb anywhere in my project so it is neither part of my code nor of the used libraries. I guess it is part of the compiled librdkafka.

Is there maybe an option to configure the build so that it is only a producer? Or a php.ini setting? I couldn't find anything so far.

Or is there an option for a monolog configuration, which silences such messages?

Answer 1

I circumvented the problem as follows: We use kustomize for deployments and Datadog for logging. I learned that it is possible to filter what is passed on to the logging system by adding to the file kustomize/base/deployment.yaml.

spec:
  template:
    metadata:
      annotations:
        ad.datadoghq.com/tags: '{"team":"acme", "domain":"foo", "application":"bar"}'
        ad.datadoghq.com/strafanzeigen-backend-middleware.logs: >-
          [{
            "log_processing_rules": [{
              "type": "exclude_at_match",
              "name": "exclude_confwarn",
              "pattern" : "rebalance_cb"
            }]
          }]

This blocks any log entries, which contain "rebalance_cb". It is also possible to use regex as pattern. See the Datadog documentation: Filter Datadog Logs

If you have the option to change the Datadog configuration in Datadog itself: It is possible to configure Datadog so that also the log level of log entries, which follow a different pattern, are interpreted correctly. The entries in question looked like this:

%6|1737713699.315|GETSUBSCRIPTIONS|rdkafka#producer-102| [thrd:main]: Telemetry client instance id changed from AAAAAAAAAAAAAAAAAAAAAA to wt3i73zKS4CTVIPCoA8yQA

With the e.g. %6 at the start indicates the log level. A colleague with admin rights on our Datadog instance configured Datadog so that now these entries are interpreted with the correct level.

Both combined solved my log problem.