Kafka topic configuration is lost after docker-compose restart

Question

I run kafka using docker, have setup storing data on the volume. I setup some source connectors, topics were created automatically with cleanup.policy delete. Using kafka manager I changed policy to compact.

The problem:

After stop/start docker-compose topics are present, but cleanup.policy is reverted back to delete

Question:

How to persist topic configuration after restart?

Additional info

I restart kafka dockers with command:

rm /kafka/data/1/meta.properties; docker-compose down && docker-compose up -d --no-recreate

Docker-compose.yml:

version: '2'
services:
  zookeeper:
    image: debezium/zookeeper:${DEBEZIUM_VERSION}
    ports:
     - 2181:2181
     - 2888:2888
     - 3888:3888
    volumes:
     - /kafka/zookeeper_data:/zookeeper/data
     - /kafka/zookeeper_logs:/zookeeper/logs
     - /kafka/zookeeper_conf:/zookeeper/conf
  kafka:
    image: debezium/kafka:${DEBEZIUM_VERSION}
    ports:
     - 9092:9092
    links:
     - zookeeper
    environment:
     - ZOOKEEPER_CONNECT=zookeeper:2181
     - KAFKA_BROKER_ID=1
     - ADVERTISED_HOST_NAME=172.16.10.187
    volumes:
     - /kafka/data:/kafka/data
     - /kafka/config:/kafka/config

  schema-registry:
    image: confluentinc/cp-schema-registry
    ports:
     - 8181:8181
     - 8081:8081
    environment:
     - SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL=zookeeper:2181
     - SCHEMA_REGISTRY_HOST_NAME=schema-registry
     - SCHEMA_REGISTRY_LISTENERS=http://schema-registry:8081
    links:
     - zookeeper

  connect3:
    build:
      context: debezium-jdbc-es
    ports:
     - 8083:8083
    links:
     - kafka
    environment:
     - BOOTSTRAP_SERVERS=kafka:9092
     - GROUP_ID=1

     - CONFIG_STORAGE_TOPIC=my_connect_configs
     - OFFSET_STORAGE_TOPIC=my_connect_offsets
     - STATUS_STORAGE_TOPIC=my_connect_statuses
     - KEY_CONVERTER=io.confluent.connect.avro.AvroConverter
     - VALUE_CONVERTER=io.confluent.connect.avro.AvroConverter
     - INTERNAL_KEY_CONVERTER=org.apache.kafka.connect.json.JsonConverter
     - INTERNAL_VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter
     - CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL=http://schema-registry:8081
     - CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL=http://schema-registry:8081
     - KAFKA_OPTS=-javaagent:/kafka/jmx_prometheus_javaagent.jar=8080:/kafka/config.yml
     - CONNECT_REST_ADVERTISED_HOST_NAME=connect3
     - JMX_PORT=1976

  prometheus:
    build:
      context: debezium-prometheus
    ports:
     - 9090:9090
    links:
     - connect3
  grafana:
    build:
      context: debezium-grafana
    ports:
     - 3000:3000
    links:
     - prometheus
    environment:
     - DS_PROMETHEUS=prometheus

  restproxy:
    image: confluentinc/cp-kafka-rest
    environment:
      KAFKA_REST_BOOTSTRAP_SERVERS: "kafka:9092"
      KAFKA_REST_LISTENERS: "http://0.0.0.0:8082"      
      KAFKA_REST_HOST_NAME: restproxy
      KAFKA_REST_DEBUG: "true"
      KAFKA_REST_SCHEMA_REGISTRY_URL: 'http://schema-registry:8081'
    ports:
      - 8082:8082
  kafka-ui:
    image: landoop/kafka-connect-ui:latest
    ports:
      - 8000:8000
    links:
      - connect3
      - schema-registry
      - zookeeper
    environment:
    - CONNECT_URL=http://connect3:8083/
  kafka-topic-ui:
    image: landoop/kafka-topics-ui
    links:
      - connect3
    ports:
      - 8001:8000
    environment:
    - KAFKA_REST_PROXY_URL=http://restproxy:8082
    - PROXY=true
  kafka_manager:
    image: hlebalbau/kafka-manager:stable
    ports:
      - "9000:9000"
    environment:
      ZK_HOSTS: "zookeeper:2181"
    links:
      - connect3

Answer 1

Kafka topic configuration is stored in Zookeeper. You can access it using

bin/zookeeper-shell zookeeper_ip_or_fdqn:2181 get /config/topics/yourTopic

One potential problem that comes to mind is that your zookeeper data is not persisted. I see that you have 3 volumes for your zookeeper container but maybe check to see if the DataDir property in zookeeper.properties points to a persisted folder or if there is another reason why zookeeper data is not being persisted.

Hope this was useful.

Best regards.