I'm trying to implement a Kafka Producer including SSL configuration. Since we run the service in containers and shipping keystores in the local file system is not possible (using the ssl.keystore.location and ssl.truststore.location options), I have to pass keys and certificates in PEM format using the corresponding/alternate options - as available since version 2.7.0 (I'm running Kafka 3.5.0 and kafka-clients:3.5.0 on JDK8)
Here's my full client configuration (keys are for testing only)
security.protocol=SSL
ssl.enabled.protocols=TLSv1.3
ssl.keystore.type=PEM
ssl.key.password=test1234
ssl.keystore.key=-----BEGIN RSA PRIVATE KEY----- \
MIIEpQIBAAKCAQEAq2UXYa2I/ebR89f9Y7IA7u9PIKOAz2jyK4Y+Zm3lNzIj+7Sf \
1zVFt8KwIG+KSOpvS7i3FoUZhqSr6KnkOZnd5QBuNkAo+20J1FTGSMlHHxxvYSb/ \
03M1Ehwu5FO87jGX4KhZdiFO4glaC+ww7X3Rk9NFcONI+/r7YRS6lmux/yZ45bJb \
NYg7dIkVdl2evZOM3qu+uLbeAU/R7gu2G94XpKPe7OqdIwN5oNch41eoj1Mucm4j \
BT0b4BrJXsB82pHsKtDfpRWw+HjSvysgGhietcj8AfiO9U3SU7frRaFnQg6qrYzd \
6QxQlvBa2T62hoJvfjctXoHTuywpbAQTEzpLlwIDAQABAoIBACMJ+au4ZGczxvxI \
zb6R52rxdYXAp1a/p7KKMTuTf+a203BjzsGaXHRi0sruwqCBfNtIGfX6o+tIwvQ/ \
ty6nbBui6OUiXL553iaQZjD/JRERKRv9cxNxXAolxNhc3iT78oa1JATobk37h3Mk \
iCQlMrE74dhgPs100+tW54ZU4gC9L9OY8M7tQuXFOOK7cydTMmuiC1SMQEHR9WUD \
CJQMyn8zcPxHPGcfyc5Sr7W54FQ2quvWNmxTDGAp80bYl/wuAEbuig3QRyLv055G \
GRMZN0nAM6J3LWbFx/NtWEErN6RoFvc4C0h+H3V+DYiQ2SQNrmBKb3N9gQqs5mXM \
3c/DnYECgYEA+MY3WRDeD1DjHK/q2/NLHLGmSqoQ2hN7ea0j+8VLvu7X4r1TrQJ4 \
qv5j9v4vRnXRW5dCo7tyWmwbY9g93foPGmtYFD6UzaZRu0jO6oyWL2rsEm015NuT \
9Faf5oDMkAlWrVusUTHsnMlwTKCDWRqqa6Jum1wfqsRm0hGGqs7ZiGECgYEAsF+D \
e/ldvxfWdCIK24AQOrKcYgIy0iwTuDrvfkhA0v9I578U15/odBoXLyrUuvJuxWUa \
aQl8oaxvOS5jw/f5nraquOXUECllwMoh1GNkMxX7zTaLtt16JkSHzXZjBZ8QDFAt \
KRRXOqtsDLgrpSvRjHAxltNmPSNKxCzp8cBbdvcCgYEA8cdDV46rItuMPv9CFM6Z \
u6N0aLBsfGXe5Vy+0hzoBwL+UmGLB2SVfiqLYSudKq+q74dMvvCYrQTQgfNR/YDg \
WN7m1ZYU7OGuIYUbhJ1qAV+7XBVEAc7eNuRRt6aKKZrJ4OZtzOPmgGLOf8qonVrr \
MnzFXrzkEgWboKJj7XHnvUECgYEAmZ3HTp6GOV9QyPuxizcCa62PqYmSrB352vdy \
CwJxe5Uf8cErLRc5Bo6G7O0YNe/b7Q+/Lgc2RzaB9ULjYBdGkic1kS+UQbNczcsA \
JhWfeyNFmppMnCxRLK15vsnDO20sNB1NCD+QLiiKRpXOZeFBdVlux5g+rzdi8nTQ \
GmDZCUcCgYEA2X6cJ4uCngOv7tQdILZpzYeZAb2crNa9vlngcfLbi8+7zTyS4Lop \
8Nmn979OrmCtHKgrMOcfFiObIBgnFPSufr+svp/y9Jd5TXE6dVmNLX13rki3HHgS \
uvxYM9UV5XdtSSb4BPYgBYmuXP8QjiK+V7TqU6Cc0ZK/Ma8DoBi3IEY= \
-----END RSA PRIVATE KEY-----
ssl.keystore.certificate.chain=-----BEGIN CERTIFICATE----- \
MIIDODCCAiACFBB23JdRC3qHpoGeo+qFdT2pGgOYMA0GCSqGSIb3DQEBCwUAMEUx \
CzAJBgNVBAYTAmF0MRMwEQYDVQQIDApTb21lLVN0YXRlMSEwHwYDVQQKDBhJbnRl \
cm5ldCBXaWRnaXRzIFB0eSBMdGQwHhcNMjMwODE3MDgyMDQzWhcNMjQwODE2MDgy \
MDQzWjBsMRAwDgYDVQQGEwdVbmtub3duMRAwDgYDVQQIEwdVbmtub3duMRAwDgYD \
VQQHEwdVbmtub3duMRAwDgYDVQQKEwdVbmtub3duMRAwDgYDVQQLEwdVbmtub3du \
MRAwDgYDVQQDEwdVbmtub3duMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC \
AQEAq2UXYa2I/ebR89f9Y7IA7u9PIKOAz2jyK4Y+Zm3lNzIj+7Sf1zVFt8KwIG+K \
SOpvS7i3FoUZhqSr6KnkOZnd5QBuNkAo+20J1FTGSMlHHxxvYSb/03M1Ehwu5FO8 \
7jGX4KhZdiFO4glaC+ww7X3Rk9NFcONI+/r7YRS6lmux/yZ45bJbNYg7dIkVdl2e \
vZOM3qu+uLbeAU/R7gu2G94XpKPe7OqdIwN5oNch41eoj1Mucm4jBT0b4BrJXsB8 \
2pHsKtDfpRWw+HjSvysgGhietcj8AfiO9U3SU7frRaFnQg6qrYzd6QxQlvBa2T62 \
hoJvfjctXoHTuywpbAQTEzpLlwIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQBciKIN \
KCPTnSTPJEMHHunhC12WvbfF//BGqh3gn1eviGHRB1pnylQLPQXBIOJlTtgdlt0h \
0xSYgvWouD4UwctWTZoVRV/yHhMuZ99LXl8OKKCpZKGXfYTB2QjSkdWN+BIBhPz5 \
L7PIsNQJVmY/FpV8Es93lsjS2nH8fqaCOpVLsc604GDVD4cbUJEbCmB6faZ1Fy7x \
GX7yBkqTSEwE9TAJAgsPIWZZNsGD44ZGj0VAy/kRCEHigcToOvxPN+3nVWE3qacu \
QFyxNNX6waCc7rwxiIYarbxiGWxZo9pMEQC5ilxOzu1dShxbZNha8pyk3ClcGpCo \
naBeqW3X6uSM7CMr \
-----END CERTIFICATE----- \
-----BEGIN CERTIFICATE----- \
MIIDazCCAlOgAwIBAgIUEGrJ27wUi2LZ+2dE2DVJyDsPkwowDQYJKoZIhvcNAQEL \
BQAwRTELMAkGA1UEBhMCYXQxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoM \
GEludGVybmV0IFdpZGdpdHMgUHR5IEx0ZDAeFw0yMzA4MTcwODE0MDRaFw0yNDA4 \
MTYwODE0MDRaMEUxCzAJBgNVBAYTAmF0MRMwEQYDVQQIDApTb21lLVN0YXRlMSEw \
HwYDVQQKDBhJbnRlcm5ldCBXaWRnaXRzIFB0eSBMdGQwggEiMA0GCSqGSIb3DQEB \
AQUAA4IBDwAwggEKAoIBAQDbCz4asRwDChWKL6VMX6STMugfn2X+QISSrad21VFH \
G9GHgRY36CpD/dd1AmoJKHChe5idR+Wq/522rg0AWDcSRF0ueI092JIYhEicnLIa \
cu5CehQwrFp+eaWh+ycgun9o1BXoUZVwBz/8I+02PqsgdpPfWdzIEvMd3xGYyI0w \
darsrWrvOxfzmtZcNtj5OWyRX6Gx0ql3eq5i423BHAkKKzMwLw8TLtoIC+7i0fkR \
7BckDexgdEMAPBgLxMwc9/ZI5LPad4BwizNiFxIkA38vwVHpgeNLD58aZNh0wlOB \
c/Vk/Lq8zK3YHHLcXASudUvjR8cDMRNYwcHmrWjl5S1PAgMBAAGjUzBRMB0GA1Ud \
DgQWBBQtJjgJdIYZyJqTACDjVBoD5hPtoTAfBgNVHSMEGDAWgBQtJjgJdIYZyJqT \
ACDjVBoD5hPtoTAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQAQ \
/CS0lo8QiK/6oCyo6rcsh3XjTvbBXTW/h25RNXkQbEDmMErCI9UVHNFA+JtxfTW4 \
YDyp7ccfXdqEVzYGMlqfVeWxVSw3PvC+PCuJeAc2l8PtW9gMfmoM/N6Lg6ckw4Dy \
vEhLy34s0x+mtPQM9owxiPlKMYIzUnGhbdGF4cb+IYDONgrsBIQVa7UHrw8r43DE \
41pVFUqVvgrbHs3Oao8lEPJLyolPLuHdgVafp2urXRCqkWEwM9pTiffWDR2USX5B \
mL3mVNTnVXSpOErpjkQvp73zOjSG/h4CEgGdrLc8kjv1jkSyi3R8c3MRDhKlZnWK \
omAr3CbvLmjrFWnzDN7d \
-----END CERTIFICATE-----
ssl.truststore.type=PEM
ssl.truststore.certificates=-----BEGIN CERTIFICATE----- \
MIIDazCCAlOgAwIBAgIUEGrJ27wUi2LZ+2dE2DVJyDsPkwowDQYJKoZIhvcNAQEL \
BQAwRTELMAkGA1UEBhMCYXQxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoM \
GEludGVybmV0IFdpZGdpdHMgUHR5IEx0ZDAeFw0yMzA4MTcwODE0MDRaFw0yNDA4 \
MTYwODE0MDRaMEUxCzAJBgNVBAYTAmF0MRMwEQYDVQQIDApTb21lLVN0YXRlMSEw \
HwYDVQQKDBhJbnRlcm5ldCBXaWRnaXRzIFB0eSBMdGQwggEiMA0GCSqGSIb3DQEB \
AQUAA4IBDwAwggEKAoIBAQDbCz4asRwDChWKL6VMX6STMugfn2X+QISSrad21VFH \
G9GHgRY36CpD/dd1AmoJKHChe5idR+Wq/522rg0AWDcSRF0ueI092JIYhEicnLIa \
cu5CehQwrFp+eaWh+ycgun9o1BXoUZVwBz/8I+02PqsgdpPfWdzIEvMd3xGYyI0w \
darsrWrvOxfzmtZcNtj5OWyRX6Gx0ql3eq5i423BHAkKKzMwLw8TLtoIC+7i0fkR \
7BckDexgdEMAPBgLxMwc9/ZI5LPad4BwizNiFxIkA38vwVHpgeNLD58aZNh0wlOB \
c/Vk/Lq8zK3YHHLcXASudUvjR8cDMRNYwcHmrWjl5S1PAgMBAAGjUzBRMB0GA1Ud \
DgQWBBQtJjgJdIYZyJqTACDjVBoD5hPtoTAfBgNVHSMEGDAWgBQtJjgJdIYZyJqT \
ACDjVBoD5hPtoTAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQAQ \
/CS0lo8QiK/6oCyo6rcsh3XjTvbBXTW/h25RNXkQbEDmMErCI9UVHNFA+JtxfTW4 \
YDyp7ccfXdqEVzYGMlqfVeWxVSw3PvC+PCuJeAc2l8PtW9gMfmoM/N6Lg6ckw4Dy \
vEhLy34s0x+mtPQM9owxiPlKMYIzUnGhbdGF4cb+IYDONgrsBIQVa7UHrw8r43DE \
41pVFUqVvgrbHs3Oao8lEPJLyolPLuHdgVafp2urXRCqkWEwM9pTiffWDR2USX5B \
mL3mVNTnVXSpOErpjkQvp73zOjSG/h4CEgGdrLc8kjv1jkSyi3R8c3MRDhKlZnWK \
omAr3CbvLmjrFWnzDN7d \
-----END CERTIFICATE-----
However, when I'm trying to run the producer, I'm always getting the following error, indicating that something is wrong with the private key provided but I'm not sure what...
(The trust chain works when I skip client authentication).
org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:837)
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:671)
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:652)
at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:67)
at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:54)
at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
Caused by: org.apache.kafka.common.errors.InvalidConfigurationException: Invalid PEM keystore configs
Caused by: java.io.IOException: overrun, bytes = 925
at java.base/javax.crypto.EncryptedPrivateKeyInfo.<init>(EncryptedPrivateKeyInfo.java:98)
at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$PemStore.privateKey(DefaultSslEngineFactory.java:510)
at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$PemStore.createKeyStoreFromPem(DefaultSslEngineFactory.java:460)
at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$PemStore.<init>(DefaultSslEngineFactory.java:433)
at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.createKeystore(DefaultSslEngineFactory.java:284)
at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.configure(DefaultSslEngineFactory.java:161)
at org.apache.kafka.common.security.ssl.SslFactory.instantiateSslEngineFactory(SslFactory.java:140)
My real-life application set properties at runtime and converts the key from a keystore that is loaded via remote API call (using Bouncy castle)
Key clientPrivateKey = keyStore.getKey(clientCertAlias, keyStorePassword.toCharArray());
try (Writer writer = new StringWriter(); JcaPEMWriter pemWriter = new JcaPEMWriter(writer)) {
pemWriter.writeObject(clientPrivateKey);
pemWriter.flush();
properties.put("ssl.keystore.key", writer.toString());
}
When I use the JKS keystore options with the same keystore (including both the trusted key and my private key), initialization works, so something must be flawed with the way the private key gets exported/converted into PEM format
security.protocol=SSL
ssl.truststore.type=JKS
ssl.truststore.location=/tmp/client.keystore.jks
ssl.enabled.protocols=TLSv1.3
ssl.truststore.password=test1234
ssl.keystore.type=JKS
ssl.keystore.location=/tmp/client.keystore.jks
ssl.keystore.password=test1234
ssl.endpoint.identification.algorithm=
I've found that OpenJDK and lack of support for PBES2-formatted keys in an internal library is the issue.
See also: Kafka won't start with PEM certificate
So, there are two options as of today
1.) Convert the key
Something like
openssl pkcs8 -in old_kafka.key -passout "pass:password" -topk8 -v1 PBE-SHA1-3DES -out kafka.key
should do the trick. My scenario actually included loading a remote keystore, extracting the key and passing it in-memory, so this possibly includes lots of specific command-line action with openssl and keytool such as converting/reimporting the key or initially creating a proper key.
If you have third-party users providing certificates in your Kafka-based stack, you will have to be transparent about this limitation (or convert every incompatible key provided), also be aware of concerns raised in the JDK issue:
If encrypted with legacy PBKDF1+SHA1+3DES the encoding works, but these algorithms are deprecated and insecure.
2.) Use the ssl.keystore/truststore.location options and a temporary keystore file
When using keystores to initialize Kafka, this goes through a different code path in Java libraries and we don't hit javax.crypto.EncryptedPrivateKeyInfo failing to parse a PBES2-encoded key.
My scenario allowed to dump the key into the production containers /tmp filesystem, initialize Kafka and then delete the keystore file to avoid file system growth (or certificates exposed).
KafkaProducer internally does a KeyStore.load and the stores are held in-memory, so you can delete it after your producer was created.
Going forward JDK-8231581, might get fixed but when I checked last (jdk-22+13), it was still the same old code from 2007 (or even before).