Search code examples
apache-kafka-connectconfluent-platform

Trying to install a Kafka Connect connector via a jar file using Docker, installation gives errors

I have am using docker-compose along with a Dockerfile to install a connector. I have been successful in installing connectors from Confluent Hub, but not my own jar files.

Here is what I did:

  1. Went to https://search.maven.org/artifact/io.aiven/aiven-kafka-connect-gcs/0.7.0/jar and in the upper right corner, pressed Downloads and clicked on "jar"
  2. Placed this file in the same folder as my Dockerfile
  3. Ran my Dockerfile:

ENV CONNECT_PLUGIN_PATH="/usr/share/java,/usr/share/confluent-hub-components"

USER root
COPY --chown=appuser:appuser aiven-kafka-connect-gcs-0.7.0.jar /usr/share/confluent-hub-components
USER appuser

RUN confluent-hub install --no-prompt aiven/kafka-connect-gcs:0.7.0

I have also tried various confluent-hub install commands, including:

RUN confluent-hub install --no-prompt aiven-kafka-connect-gcs:0.7.0
RUN confluent-hub install --no-prompt confluent-hub-components/aiven-kafka-connect-gcs-0.7.0.jar
RUN confluent-hub install --no-prompt aiven-kafka-connect-gcs-0.7.0.jar

all to no avail. I did try other directories like /etc/kafka-connect/jars and I just keep getting the same issue.

What am I doing wrong? Syntax? Missing additional mounting commands? Something else?

Solution

  • This is how I did it:

    RUN wget -O /set/your/path/here/<connector name here>.tar https://url-of-connector-here/<connector name and version here>.tar
RUN tar -xvf /set/your/path/here/<connector name here>.tar --directory /path/to/connect/plugins/here/

    and it worked.