ksqldb not failing over to standby schema registry

Question

I am trying to test failover scenario for kafka schema registry.

I spanned up two Schema registry docker containers(Primary and standby) and I have a KSQLDB server running in a docker container pointing to primary schema registry. The source kafka connecter is streaming the data from the database to kafka topics. The ksqlDB server is able to validate the schema of the kafka message using primary schema registry. Now I shutdown the primary schema registry. The ksqldb server is not failing over to the stand by schema registry to validate the schema, causing ksqldb server not receiving the data from kafka topics.

How should ksqldb server should know what is the standby schema-registry that it need to connect to when primary is down.

Below is docker-compose.yml file that I have used

schema-registry:
    image: confluentinc/cp-schema-registry:${CP_VERSION}
    depends_on:
      - zookeeper
      - kafka
    ports:
      - "8081:8081"
    container_name: schema-registry
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: PLAINTEXT://kafka:9092
      SCHEMA_REGISTRY_ACCESS_CONTROL_ALLOW_ORIGIN: '*'
      SCHEMA_REGISTRY_ACCESS_CONTROL_ALLOW_METHODS: 'GET,POST,PUT,OPTIONS'
      SCHEMA_REGISTRY_LEADER_ELIGIBILITY : "true"
      SCHEMA_REGISTRY_GROUP_ID : "schema-registry-group"
      SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081
      
  schema-registry-2:
    image: confluentinc/cp-schema-registry:${CP_VERSION}
    depends_on:
      - kafka
      - schema-registry
    ports:
      - "8082:8082"
    container_name: schema-registry-2
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry-2
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: PLAINTEXT://kafka:9092
      SCHEMA_REGISTRY_ACCESS_CONTROL_ALLOW_ORIGIN: '*'
      SCHEMA_REGISTRY_ACCESS_CONTROL_ALLOW_METHODS: 'GET,POST,PUT,OPTIONS'
      SCHEMA_REGISTRY_LEADER_ELIGIBILITY : "true"
      SCHEMA_REGISTRY_GROUP_ID : "schema-registry-group"
      SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8082

 primary-ksqldb-server:
    image: ${KSQL_IMAGE_BASE}confluentinc/ksqldb-server:${KSQL_VERSION}
    hostname: primary-ksqldb-server
    container_name: primary-ksqldb-server
    depends_on:
      - kafka
      - schema-registry
    ports:
      - "8088:8088"
    environment:
      KSQL_CONFIG_DIR: "/etc/ksql"
      KSQL_LISTENERS: http://0.0.0.0:8088
      KSQL_BOOTSTRAP_SERVERS: kafka:9092
      KSQL_KSQL_ADVERTISED_LISTENER : http://localhost:8088
      KSQL_KSQL_SCHEMA_REGISTRY_URL: http://schema-registry:8081
      KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"
      KSQL_KSQL_EXTENSION_DIR: "/usr/ksqldb/ext/"
      KSQL_KSQL_SERVICE_ID: "nrt_"
      KSQL_KSQL_STREAMS_NUM_STANDBY_REPLICAS: 1
      KSQL_KSQL_QUERY_PULL_ENABLE_STANDBY_READS: "true"
      KSQL_KSQL_HEARTBEAT_ENABLE: "true"
      KSQL_KSQL_LAG_REPORTING_ENABLE : "true"
      KSQL_KSQL_QUERY_PULL_MAX_ALLOWED_OFFSET_LAG : 100
      KSQL_LOG4J_APPENDER_KAFKA_APPENDER: "org.apache.kafka.log4jappender.KafkaLog4jAppender"
      KSQL_LOG4J_APPENDER_KAFKA_APPENDER_LAYOUT: "io.confluent.common.logging.log4j.StructuredJsonLayout"
      KSQL_LOG4J_APPENDER_KAFKA_APPENDER_BROKERLIST: localhost:9092
      KSQL_LOG4J_APPENDER_KAFKA_APPENDER_TOPIC: KSQL_LOG
      KSQL_LOG4J_LOGGER_IO_CONFLUENT_KSQL: INFO,kafka_appender
      KSQL_KSQL_QUERY_PULL_METRICS_ENABLED: "true"
      KSQL_JMX_OPTS: >
          -Djava.rmi.server.hostname=localhost
          -Dcom.sun.management.jmxremote
          -Dcom.sun.management.jmxremote.port=1099
          -Dcom.sun.management.jmxremote.authenticate=false
          -Dcom.sun.management.jmxremote.ssl=false
          -Dcom.sun.management.jmxremote.rmi.port=1099

When I stop primary schema registry, ksqldb is supposed to connect to standy schema registry

Answer 1

How would it know the other is available if you don't provide it?

KSQL_KSQL_SCHEMA_REGISTRY_URL: http://schema-registry:8081,http://schema-registry-2:8082

In other words, you shut down schema-registry container, so it will simply not respond. It will not forward requests or update the clients to talk to another server... So, you need to provide a URL-list, or you need to setup an external reverse proxy to round-robin the requests to the active instance.