Spring in Kubernetes tries to reach DB at pod IP

Question

I'm facing an issue while deploying a Spring API which should connect to a MySQL database.

I am deploying a standalone MySQL using the [bitnami helm chart][1] with the following values:

        primary:
          service:
            type: ClusterIP
          persistence:
            enabled: true
            size: 3Gi
            storageClass: ""
          extraVolumes:
          - name: mysql-passwords 
            csi:
              driver: secrets-store.csi.k8s.io
              readOnly: true
              volumeAttributes:
                secretProviderClass: mysql-spc
          extraVolumeMounts:
          - name: mysql-passwords
            mountPath: "/vault/secrets"
            readOnly: true
          configuration: |-
            [mysqld]
            default_authentication_plugin=mysql_native_password
            skip-name-resolve
            explicit_defaults_for_timestamp
            basedir=/opt/bitnami/mysql
            plugin_dir=/opt/bitnami/mysql/lib/plugin
            port=3306
            socket=/opt/bitnami/mysql/tmp/mysql.sock
            datadir=/bitnami/mysql/data
            tmpdir=/opt/bitnami/mysql/tmp
            max_allowed_packet=16M
            bind-address=0.0.0.0
            pid-file=/opt/bitnami/mysql/tmp/mysqld.pid
            log-error=/opt/bitnami/mysql/logs/mysqld.log
            character-set-server=UTF8
            collation-server=utf8_general_ci
            slow_query_log=0
            slow_query_log_file=/opt/bitnami/mysql/logs/mysqld.log
            long_query_time=10.0

            [client]
            port=3306
            socket=/opt/bitnami/mysql/tmp/mysql.sock
            default-character-set=UTF8
            plugin_dir=/opt/bitnami/mysql/lib/plugin

            [manager]
            port=3306
            socket=/opt/bitnami/mysql/tmp/mysql.sock
            pid-file=/opt/bitnami/mysql/tmp/mysqld.pid
        auth:
          createDatabase: true  
          database: api-db
          username: api
          usePasswordFiles: true
          customPasswordFiles:
            root: /vault/secrets/db-root-pwd
            user: /vault/secrets/db-pwd
            replicator: /vault/secrets/db-replica-pwd
        serviceAccount:
          create: false
          name: social-app

I use the following deployment which runs a spring API (with Vault secret injection):

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: social-api
  name: social-api
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  selector:
    matchLabels:
      app: social-api
  template:
    metadata:
      labels:
        app: social-api
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: 'social'

    spec:
      serviceAccountName: social-app
      containers:
      - image: quay.io/paulbarrie7/social-network-api
        name: social-network-api
        command:
          - java
        args:
          - -jar
          - "-DSPRING_DATASOURCE_URL=jdbc:mysql://social-mysql.default.svc.cluster.local/api-db?useSSL=false"
          - "-DSPRING_DATASOURCE_USERNAME=api"
          - "-DSPRING_DATASOURCE_PASSWORD=$(cat /secrets/db-pwd)"
          - "-DJWT_SECRET=$(cat /secrets/jwt-secret)"
          - "-DS3_BUCKET=$(cat /secrets/s3-bucket)"
          - -Dlogging.level.root=DEBUG
          - -Dspring.datasource.hikari.maximum-pool-size=5
          - -Dlogging.level.com.zaxxer.hikari.HikariConfig=DEBUG
          - -Dlogging.level.com.zaxxer.hikari=TRACE
          - social-network-api-1.0-SNAPSHOT.jar
        resources:
          limits:
            cpu: 100m
            memory: 100Mi
          requests:
            cpu: 100m
            memory: 100Mi
        ports:
        - containerPort: 8080
        volumeMounts:
        - name: aws-credentials
          mountPath: "/root/.aws"
          readOnly: true
        - name: java-secrets
          mountPath: "/secrets"
          readOnly: true
      volumes:
      - name: aws-credentials
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: aws-secret-spc
      - name: java-secrets
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: java-spc

Identifier are ok, when I run an interactive mysql pod I can connect to the database. However name resolution for the Spring API is wrong since I get the error:

java.sql.SQLException: Access denied for user 'api'@'10.24.0.194' (using password: YES)

which is wrong since 10.24.0.194 is the API pod address and not the mysql pod or service address, and I cant solve why.

Any idea? [1]: https://artifacthub.io/packages/helm/bitnami/mysql

Answer 1

Thanks to David's suggestion I succeeded in solving my problem.

Actually there were two issues in my configs.

First the secrets were indeed misinterpreted, then I've changed my command/args to:

command:
          - "/bin/sh"
          - "-c"
        args:
          - |
            DB_USER=$(cat /secrets/db-user)
            DB_PWD=$(cat /secrets/db-pwd)
            JWT=$(cat /secrets/jwt-secret)
            BUCKET=$(cat /secrets/s3-bucket)
            java -jar  \
            -DSPRING_DATASOURCE_URL=jdbc:mysql://social-mysql.default.svc.cluster.local/api-db?useSSL=false \
            "-DSPRING_DATASOURCE_USERNAME=$DB_USER" \
            "-DSPRING_DATASOURCE_PASSWORD=$DB_PWD" \
            "-DJWT_SECRET=$JWT" \
            "-DS3_BUCKET=$BUCKET" \
            -Dlogging.level.root=DEBUG \
            social-network-api-1.0-SNAPSHOT.jar

And the memory resources set were also too low, so I have changed them to:

resources:
 limits:
  cpu: 100m
  memory: 400Mi
 requests:
  cpu: 100m
  memory: 400Mi