Volumes in k8s pod/Container ... loosing some dirs from the Conatiner Image

Question

I have a custom Container Image for postgresql and try to run this as a Stateful kubernetes application

The image knows 2 Volumes which are mounted into

1. /opt/db/data/postgres/data (the $PGDATA directory of my postgres intallation)
2. /opt/db/backup

the backup Volume contains a deeper folder structure which is defined in the Dockerfile

Dockerfile

(excerpts)

...
...
# Environment variables required for this build (do NOT change)
# -------------------------------------------------------------
...
ENV PGDATA=/opt/db/data/postgres/data
ENV PGBASE=/opt/db/postgres
...
ENV PGBACK=/opt/db/backup/postgres/backups
ENV PGARCH=/opt/db/backup/postgres/archives


# Set up user and directories
# ---------------------------
RUN mkdir -p $PGBASE $PGBIN $PGDATA $PGBACK $PGARCH && \
    useradd -d /home/postgres -m -s /bin/bash --no-log-init --uid 1001 --user-group postgres && \
    chown -R postgres:postgres $PGBASE $PGDATA $PGBACK $PGARCH && \
    chmod a+xr $PGBASE

# set up user env
# ---------------
USER postgres
...
...
# bindings
# --------
VOLUME ["$PGDATA", "$DBBASE/backup"]
...
...
# Define default command to start Database.
ENTRYPOINT ["/docker-entrypoint.sh"]
CMD ["postgres", "-D", "/opt/db/data/postgres/data"]

When I run this as a container or single pod in kubernetes without any volumeMounts all is good and the folder structure looks like it should

find /opt/db/backup -ls
2246982      4 drwxr-xr-x   3 root     root         4096 Feb 18 09:00 /opt/db/backup
2246985      4 drwxr-xr-x   4 root     root         4096 Feb 18 09:00 /opt/db/backup/postgres
2246987      4 drwxr-xr-x   2 postgres postgres     4096 Feb 11 14:59 /opt/db/backup/postgres/backups
2246986      4 drwxr-xr-x   2 postgres postgres     4096 Feb 11 14:59 /opt/db/backup/postgres/archives

However once I run this based on the Statefulset below (which mounts to Volumes into the pod @ /opt/db/data/postgres/data & /opt/db/backup (which includes folder structure that goes deeper then the mount point, as listed above) this is not bein carried out as intended

find /opt/db/backup –ls
2      4 drwxr-xr-x   3 postgres postgres     4096 Feb 17 16:40 /opt/db/backup
11     16 drwx------   2 postgres postgres    16384 Feb 17 16:40 /opt/db/backup/lost+found

/opt/db/backup/postgres/backups & /opt/db/backup/postgres/archives, inherit in the Image are gone.

Can anybody point me to where to start looking for a solution in this?

Statefulset

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  namespace: postgres-stateful
  name: postgres-stateful
  labels:
    app.kubernetes.io/name: postgres
    app.kubernetes.io/component: database
    app.kubernetes.io/part-of: postgres
    app: postgres
spec:
  serviceName: "postgres"
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: postgres
      app.kubernetes.io/component: database
      app.kubernetes.io/part-of: postgres
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
        app.kubernetes.io/name: postgres
        app.kubernetes.io/component: database
        app.kubernetes.io/part-of: postgres
    spec:
      serviceAccountName: default
      initContainers:           # Give `postgres` user (id 1001) permissions to mounted volumes
      - name: take-volume-mounts-ownership
        image: dev-local.dev.dlz.net/postgresql:14.2-deb11
        securityContext:
          readOnlyRootFilesystem: true
        env:
          - name: "PGDATA"
            value: "/opt/db/data/postgres/data"
        command: [ "/bin/sh" ]
        args: ["-c", "chown -R 1001:1001 /opt/db/data/postgres /opt/db/backup /tmp" ]
        volumeMounts:
        - name: pv-data
          mountPath: /opt/db/data/postgres
        - name: pv-backup
          mountPath: /opt/db/backup     # /opt/db/backup/postgres
        - name: emptydir-tmp
          mountPath: /tmp
      containers:
      - name: postgres
        image: dev-local.dev.dlz.net/postgresql:14.2-deb11
        imagePullPolicy: Always
        readinessProbe:
          exec:
            command: ["pg_isready", "-q"]
          periodSeconds: 10
          initialDelaySeconds: 7
          timeoutSeconds: 2
        livenessProbe:
          exec:
            command: ["psql", "-q", "-c", "SELECT 1 WHERE 1=0"]
          periodSeconds: 15
          initialDelaySeconds: 20
          timeoutSeconds: 2
        env:
          - name: "PGDATA"
            value: "/opt/db/data/postgres/data"
        envFrom:
        - configMapRef:
            name: postgres-configuration
        ports:
        - containerPort: 5432
          name: postgresdb
        resources:
          requests:
            memory: "256Mi"
            cpu: "50m"
          limits:
            memory: "1Gi"
            cpu: "1"
        securityContext:
          privileged: false
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1001
          allowPrivilegeEscalation: false
        volumeMounts:
        - name: pv-data
          mountPath: /opt/db/data/postgres/data   # /var/lib/postgresql/data
        - name: pv-backup
          mountPath: /opt/db/backup     # /opt/db/backup/postgres
        - name: emptydir-tmp
          mountPath: /tmp
      volumes:
      - name: pv-data
        persistentVolumeClaim:
          claimName: pgdata-ina-pvc
      - name: pv-backup
        persistentVolumeClaim:
          claimName: pgbackup-ina-pvc
      - name: emptydir-tmp
        emptyDir: {}

Answer 1

The directory which you had created in Dockerfile will be overlaid when you mount persistent volume to the same path. You can re-construct the directory structure in your "take-volume-mounts-ownership" container:

...
initContainers:
- name: take-volume-mounts-ownership
  ...
  env:
  - name: PGDATA
    value: /opt/db/data/postgres/data
  - name: PGBASE
    value: /opt/db/postgres
  - name: PGBACK
    value: /opt/db/backup/postgres/backups
  - name: PGARCH
    value: /opt/db/backup/postgres/archives
  ...
  args: ["-c", "mkdir -p $PGBASE $PGBIN $PGDATA $PGBACK $PGARCH && chown -R 1001:1001 /opt/db/data/postgres /opt/db/backup /tmp" ]
  ...