I have 2 certificate files, truststore.jks and keystore.jks. keystore.jks contains a full certificate chain for the kafka endpoint I'm using as well as a private key for my application. According to the group who gave me truststore.jks it contains "a new root CA chain" and is essential for getting my application to connect with the kafka endpoint.
I have tried using this script to generate 3 .pem files that I use in my Python code.
#!/bin/bash
srcFolder=$1
keyStore=$1/$2
password=$3
alias=$4
outputFolder=$5
echo $keyStore
echo "Generating certificate.pem"
keytool -exportcert -alias $alias -keystore $keyStore -rfc -file $outputFolder/certificate.pem -storepass $password
echo "Generating key.pem"
keytool -v -importkeystore -srckeystore $keyStore -srcalias $alias -destkeystore $outputFolder/cert_and_key.p12 -deststoretype PKCS12 -storepass $password -srcstorepass $password
openssl pkcs12 -in $outputFolder/cert_and_key.p12 -nodes -nocerts -out $outputFolder/key.pem -passin pass:$password
echo "Generating CARoot.pem"
keytool -exportcert -alias $alias -keystore $keyStore -rfc -file $outputFolder/CARoot.pem -storepass $password
I use the generated CARoot.pem, certificate.pem, and key.pem files in my code like so.
>>> from kafka import KafkaProducer
>>> producer = KafkaProducer(bootstrap_servers='kafka-dev.kafka.com:port', security_protocol="SSL",
... ssl_check_hostname=True,
... ssl_cafile="CARoot.pem",
... ssl_certfile="certificate.pem",
... ssl_keyfile="key.pem",
... ssl_password="")
It gives this error:
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1131)
I have tried combining truststore.jks and keystore.jks like so:
keytool -importkeystore -srckeystore truststore.jks -destkeystore keystore_copy.jks
I then put keystore_copy.jks through the same .pem conversion process as before, but it produces the same error.
For some context, this is how the Kafka endpoint owners suggest to configure the ssl certs in Java, unfortunately I am the first to connect with Python so they haven't been able to help me with it.
props.setProperty(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SSL");
props.setProperty(SslConfigs.SSL_TRUSTSTORE_LOCATION_CONFIG, "<kafka-truststore.jks>");
props.setProperty(SslConfigs.SSL_TRUSTSTORE_PASSWORD_CONFIG, "<password>");
props.setProperty(SslConfigs.SSL_KEYSTORE_LOCATION_CONFIG, "<client-keystore.jks>");
props.setProperty(SslConfigs.SSL_KEYSTORE_PASSWORD_CONFIG, "<password>");
props.setProperty(SslConfigs.SSL_KEY_PASSWORD_CONFIG, "<password>");
Is there something I'm missing? I believe I need to utilize both .jks files in my code somehow but the only approach I can think of taking failed.
Additional things I have tried:
pip install --upgrade certifi based on reports of the same error due to that library being out of date.
I managed to figure out the solution on my own. The key here was realizing that I needed to have the entire cert chain from my truststore.jks file present in CARoot.pem for the code to work. Here are the steps that I took:
Generate CARoot.pem from the trust store:
keytool -importkeystore -srckeystore <<truststore-name.jks>> -destkeystore keystore.p12 -srcstoretype jks -deststoretype pkcs12
openssl pkcs12 -in keystore.p12 -out CARoot.pem
Generate certificate.pem from keystore
keytool -exportcert -alias <<alias>> -keystore ./<<keystore—name.jks>> -rfc -file ./certificate.pem -storepass <<password>>
Extract private key and generate RSAkey.pem from keystore:
keytool -importkeystore -srckeystore <<keystore-name.jks>> -destkeystore keystore.p12 -deststoretype PKCS12
openssl pkcs12 -in keystore.p12 -nodes -nocerts -out RSAkey.pem
Example with Kafka-Python Producer:
producer = KafkaProducer(bootstrap_servers='kafka-dev.kafka.com:port', security_protocol="SSL",
ssl_check_hostname=False,
ssl_cafile="CARoot.pem",
ssl_certfile="certificate.pem",
ssl_keyfile="RSAkey.pem",
ssl_password=<<password>>)