configuring ca_cert, client_cert, client_key, tls_enabled in kafka table in clickhouse

Question

I want to connect CH to a TLS kafka cluster.

My producer uses the following to push events:

    'kafka://broker0:9092,broker1:9092,broker2:9092?
     ca_cert=redacted&
     client_cert=redacted&
     client_key=some-long-string&
tls_enabled=true&topic_prefix=some-prefix_' 

how to configure the following from the consumer side (ch kafka table): ca_cert client_cert client_key tls_enabled

the docs only cover the basic set up, which is plaintext

Answer 1

All of the following settings can be added to your Kafka configuration in ClickHouse: https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md - but you have to replace the dots (.) with underscores (_) in the property name.

For example, you could create a new kafka.xml file and place it in your config.d directory:

<clickhouse>
   <kafka>
       <security_protocol>SSL</security_protocol>
       <ssl_ca_location>/opt/kafka/my.crt</ssl_ca_location>
       <ssl_certificate_location>/etc/my.pem</ssl_certificate_location>
       <ssl_key_location>my.key</ssl_key_location>
       <ssl_key_password>my_password</ssl_key_password>

   </kafka>
</clickhouse>