Using Kubernetes Secret for PostgreSQL ENV causes application pod CRASH

Question

I have problem with injecting Kubernetes Secret's value into Pod env. I have following pg-secrets.yml:

apiVersion: v1
kind: Secret
metadata:
  name: pg-secrets
type: Opaque
data:
  POSTGRES_USER: cG9zdGdyZXMK
  POSTGRES_PASSWORD: cGFzc3dvcmQK
#   postgres & password

And then I inject POSTGRES_PASSWORD from it to application-deployment.yml ENV:

apiVersion: apps/v1
kind: Deployment
   ...
    spec:
      containers:
        - name: realo
          image: abialiauski/realo
          imagePullPolicy: Always
          ports:
            - containerPort: 8080
          env:
            - name: PG_USERNAME
              valueFrom:
                configMapKeyRef:
                  name: realo-configmap
                  key: PG_USERNAME
            - name: PG_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: pg-secrets
                  key: POSTGRES_PASSWORD
            - name: PG_HOST
              value: postgres

And having this application.yml:

spring:
  application:
    name: realo
  datasource:
    username: ${PG_USERNAME}
    password: ${PG_PASSWORD}
    url: jdbc:postgresql://${PG_HOST}:5432/postgres
    driver-class-name: org.postgresql.Driver

So, after starting PostgreSQL, the backend application crashes refusing the database connection with this exception:

Caused by: org.postgresql.util.PSQLException: Something unusual has occurred to cause the driver to fail. Please report this exception.
    at org.postgresql.Driver.connect(Driver.java:282) ~[postgresql-42.3.8.jar:42.3.8]
    at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138) ~[HikariCP-4.0.3.jar:na]
    at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:364) ~[HikariCP-4.0.3.jar:na]
    at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:206) ~[HikariCP-4.0.3.jar:na]
    at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:476) ~[HikariCP-4.0.3.jar:na]
    at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:561) ~[HikariCP-4.0.3.jar:na]
    at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:115) ~[HikariCP-4.0.3.jar:na]
    at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:112) ~[HikariCP-4.0.3.jar:na]
    at liquibase.integration.spring.SpringLiquibase.afterPropertiesSet(SpringLiquibase.java:266) ~[liquibase-core-4.9.1.jar:na]
    ... 180 common frames omitted
Caused by: java.lang.IllegalArgumentException: Prohibited character

    at org.postgresql.shaded.com.ongres.saslprep.SaslPrep.saslPrep(SaslPrep.java:105) ~[postgresql-42.3.8.jar:42.3.8]
    at org.postgresql.shaded.com.ongres.scram.common.stringprep.StringPreparations$2.doNormalize(StringPreparations.java:55) ~[postgresql-42.3.8.jar:42.3.8]
    at org.postgresql.shaded.com.ongres.scram.common.stringprep.StringPreparations.normalize(StringPreparations.java:65) ~[postgresql-42.3.8.jar:42.3.8]
    at org.postgresql.shaded.com.ongres.scram.common.ScramMechanisms.saltedPassword(ScramMechanisms.java:152) ~[postgresql-42.3.8.jar:42.3.8]
    at org.postgresql.shaded.com.ongres.scram.common.ScramFunctions.saltedPassword(ScramFunctions.java:59) ~[postgresql-42.3.8.jar:42.3.8]
    at org.postgresql.shaded.com.ongres.scram.client.ScramSession$ClientFinalProcessor.<init>(ScramSession.java:196) ~[postgresql-42.3.8.jar:42.3.8]
    at org.postgresql.shaded.com.ongres.scram.client.ScramSession$ClientFinalProcessor.<init>(ScramSession.java:163) ~[postgresql-42.3.8.jar:42.3.8]
    at org.postgresql.shaded.com.ongres.scram.client.ScramSession$ServerFirstProcessor.clientFinalProcessor(ScramSession.java:130) ~[postgresql-42.3.8.jar:42.3.8]
    at org.postgresql.jre7.sasl.ScramAuthenticator.processServerFirstMessage(ScramAuthenticator.java:147) ~[postgresql-42.3.8.jar:42.3.8]
    at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:816) ~[postgresql-42.3.8.jar:42.3.8]
    at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:180) ~[postgresql-42.3.8.jar:42.3.8]
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:235) ~[postgresql-42.3.8.jar:42.3.8]
    at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49) ~[postgresql-42.3.8.jar:42.3.8]
    at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:223) ~[postgresql-42.3.8.jar:42.3.8]
    at org.postgresql.Driver.makeConnection(Driver.java:402) ~[postgresql-42.3.8.jar:42.3.8]
    at org.postgresql.Driver.connect(Driver.java:261) ~[postgresql-42.3.8.jar:42.3.8]
    ... 188 common frames omitted

If I inspect ENV everything will be OK, but backend still fails.

kubectl exec -i -t pod/realo-66856b59b4-bp27r -- /bin/sh -c 'echo "username: $PG_USERNAME\npassword: $PG_PASSWORD\n"'

output:

username: postgres\npassword: password

Answer 1

You used

$ echo postgres | base64
cG9zdGdyZXMK

to generate your base64 encoded secret.

The problem with that is echo adds a linefeed to its output, you should call it like this:

$ echo -n postgres | base64
cG9zdGdyZXM=

You need to do the same with password too:

$ echo -n password | base64
cGFzc3dvcmQ=