Logs says that heap dump is created in gc persistent volume but when entering in the pod there is no heap dump file

Question

I have created the following persistent volume in GoogleCloud by deploying the following yml file:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: staging-filestore
  labels:
    filestore: standard
spec:
  capacity:
    storage: 1T
  accessModes:
  - ReadWriteMany
  mountOptions:
  - lookupcache=positive #Disables caching to make all writes be sync to the server
  nfs:
    path: /staging
    server: 10.64.16.130
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: staging-filestore-pvc
spec:
  selector:
    matchLabels:
      filestore: standard
  accessModes:
  - ReadWriteMany
  storageClassName: ""
  resources:
    requests:
      storage: 1T

The volume is created successfully and mounted in thein the service below:

apiVersion: v1
kind: Service
metadata:
  name: demo
  labels:
    app: demo
    vendor: rs
spec:
  selector:
    app: demo
    vendor: rs
  ports:
  - port: 4010
    name: internal
    targetPort: 4010
  - port: 80
    name: external
    targetPort: 4010
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo
  labels:
    app: demo
    vendor: rs
spec:
  selector:
    matchLabels:
      app: demo
  replicas: 1
  revisionHistoryLimit: 2
  template:
    metadata:
      labels:
        app: demo
        vendor: rs
    spec:
      imagePullSecrets:
      - name: dockerhub-secret

      containers:

      - name: demo
        image: rs/demo:latest
        imagePullPolicy: Always
        resources:
          requests:
            memory: "1Gi"
          limits:
            memory: "2Gi"
        volumeMounts:
          - mountPath: /dumps/heap-dump
            name: dumpstorage
        readinessProbe:
          httpGet:
            path: /_health
            port: http
          initialDelaySeconds: 5
          periodSeconds: 5
          failureThreshold: 5
        livenessProbe:
          httpGet:
            path: /_health
            port: http
          initialDelaySeconds: 25
          periodSeconds: 5
          failureThreshold: 10
        envFrom:
        - configMapRef:
            name: rs-config
        env:
        - name: JAVA_TOOL_OPTIONS
          value: " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=\"/dumps/demo-heap-dump\""
        ports:
        - containerPort: 4010
          name: http
      volumes:
        - name: dumpstorage
          persistentVolumeClaim:
            claimName: staging-filestore-PVC
            readOnly: false

I want that the heap dump generated when the application has a memory overflow error to be stored and the staging-filestore-PVC volume. To test this, I set the heap size to 1MB with the option -Xmx16m. The application crashes causing also the pod to crash. I can see in the logs that say the heap dump is created:

2022-08-08 15:34:55.767 CESTDumping heap to /dumps/demo-heap-dump ...
2022-08-08 15:34:55.990 CESTHeap dump file created [26757864 bytes in 0.223 secs]

I deploy the service yml file again with the -Xmx1G option. And enter inside the pod terminal with the command kubectl exec -it demo-77f84d9957-vxrsz -- sh, go to the /dumps directory there is no demo-heap-dump file.

I am struggling to understand why the file is not present although no error is thrown and the logs say that the file is created. Thank you for the help.

Regards, Rando.

Answer 1

Try:

...
env:
- name: JAVA_TOOL_OPTIONS
  value: " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=\"/dumps/heap-dump\""  # <-- update the path
...