I am using Kafka Connect and have an independent thread started in my connector plugin that is listening on a port (say "9090"). I want to use this port to allow applications (external to the kafka environment) to communicate with my connector plugin.
I am running Kafka Connect (and the kafka environment) in docker-compose.
Question: how can the port be exposed in docker compose so that external apps can reach my connector plugin thread listener port?
Here is what I have come up with so far, but it seems odd that I would need to expose an external port on a kafka connect component. Is this a sound practice or is there a better way to allow apps to communicate with my connect plugin thread listener port?
Below is a segment of my docker-compose file that shows how I expose port "9090" on connect:
:
:
:
connect:
image: confluentinc/cp-kafka-connect:6.2.0
hostname: connect
container_name: connect
depends_on:
- zookeeper
- kafka
ports:
- 8083:8083
- 9090:9090
environment:
CONNECT_BOOTSTRAP_SERVERS: "kafka:9092"
CONNECT_REST_PORT: 8083
CONNECT_REST_ADVERTISED_HOST_NAME: "connect"
:
:
:
The solution below is very similar to what I had expected, and verified by @OneCricketeer, but I added a range of ports (below, 9090-9094) to the connect docker-compose configuration. I would also make the "listener" thread component listen on a range of ports equal to the number of tasks.max (in the example below, it would be 5).
:
:
:
connect:
image: confluentinc/cp-kafka-connect:6.2.0
hostname: connect
container_name: connect
depends_on:
- zookeeper
- kafka
ports:
- 8083:8083
- 9090-9094:9090-9094
environment:
CONNECT_BOOTSTRAP_SERVERS: "kafka:9092"
CONNECT_REST_PORT: 8083
CONNECT_REST_ADVERTISED_HOST_NAME: "connect"
:
:
: