Taking Thread dump/ Heap dump of Azure Kubernetes pods

Question

We are running our kafka stream application on Azure kubernetes written in java. We are new to kubernetes. To debug an issue we want to take thread dump of the running pod.

Below are the steps we are following to take the dump.

1. Building our application with below docker file.

   FROM mcr.microsoft.com/java/jdk:11-zulu-alpine
   RUN apk update && apk add --no-cache gcompat
   RUN addgroup -S user1 && adduser -S user1 -G user1
   USER user1
   WORKDIR .
   COPY target/my-application-1.0.0.0.jar .
   

2. Submitting the image with below deployment yaml file

apiVersion: apps/v1 kind: Deployment metadata: name: my-application-v1.0.0.0 spec: replicas: 1 selector: matchLabels: name: my-application-pod app: my-application-app template: metadata: name: my-application-pod labels: name: my-application-pod app: my-application-app spec: nodeSelector: agentpool: agentpool1 containers: - name: my-application-0 image: myregistry.azurecr.io/my-application:v1.0.0.0 imagePullPolicy: Always command: ["java","-jar","my-application-1.0.0.0.jar","input1","$(connection_string)"] env: - name: connection_string valueFrom: configMapKeyRef: name: my-application-configmap key: connectionString resources: limits: cpu: "4" requests: cpu: "0.5"

3. To get a shell to a Running container you can run the command below:

   kubectl exec -it <POD_NAME> -- sh

4. To get thread dump running below command

   jstack PID > threadDump.tdump

but getting permission denied error

Can some one suggest how to solve this or steps to take thread/heap dumps. Thanks in advance

Answer 1

Since you likely need the thread dump locally, you can bypass creating the file in the pod and just stream it directly to a file on your local computer:

kubectl exec -i POD_NAME -- jstack 1 > threadDump.tdump

If your thread dumps are large you may want to consider piping to pv first to get a nice progress bar.