Search code examples
apache-kafkaapache-kafka-connectsnowflake-connector

Missing required configuration "name" which has no default value when trying a kafka connect to snowflake


I have my config/SF_connect.properties file like below

tasks.max=8
topics=topic1,topic2
snowflake.topic2table.map= topic1:table1,topic2:table2
buffer.count.records=10000
buffer.flush.time=60
buffer.size.bytes=5000000
snowflake.url.name=myorganization-myaccount.snowflakecomputing.com:443
snowflake.user.name=jane.smith
snowflake.private.key=xyz123
snowflake.private.key.passphrase=jkladu098jfd089adsq4r
snowflake.database.name=mydb
snowflake.schema.name=myschema
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=com.snowflake.kafka.connector.records.SnowflakeAvroConverter
value.converter.schema.registry.url=http://localhost:8081
value.converter.basic.auth.credentials.source=USER_INFO
value.converter.basic.auth.user.info=jane.smith:MyStrongPassword

and get the error Missing required configuration "name" which has no default value when trying a kafka connect to snowflake

The official documentation for snowflake states a name is required parameter. So I added a name=somerandomname in the above file. It still doesnt work.

This is the full log indicating even the private key and passphrase is not valid (need to check with admin)

java.util.concurrent.ExecutionException: org.apache.kafka.connect.runtime.rest.errors.BadRequestException: Connector configuration is invalid and contains the following 3 error(s):
Missing required configuration "name" which has no default value.
snowflake.private.key is not valid
snowflake.private.key.passphrase is not valid

Need some help on the above. The official documentation link is here https://docs.snowflake.com/en/user-guide/kafka-connector-install.html#configuring-the-kafka-connector


Solution

  • You still need to follow the section "Using Key Pair Authentication & Key Rotation" in the documentation, and provide snowflake.private.key in the SF_connect.properties and public key in Snowflake to Kafka user. Ex.: alter user jsmith set rsa_public_key='MIIBIjANBgkqh...' ;