Metabase on K8s (GKE) get Readiness probe failed

Question

I try to deploy Matabase on my GKE cluster but I got Readiness probe failed.

I build on my local and get localhost:3000/api/health i got status 200 but on k8s it's not works.

Dockerfile. I create my own for push and build to my GitLab registry

FROM metabase/metabase:v0.41.6
EXPOSE 3000
CMD ["/app/run_metabase.sh" ]

my deployment.yaml

# apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: metaba-dev
spec:
  selector:
    matchLabels:
      app: metaba-dev
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 50%
      maxSurge: 100%
  template:
    metadata:
      labels:
        app: metaba-dev
    spec:
      restartPolicy: Always
      imagePullSecrets:
        - name: gitlab-login
      containers:
        - name: metaba-dev
          image: registry.gitlab.com/team/metabase:dev-{{BUILD_NUMBER}}
          command: ["/app/run_metabase.sh" ]
          livenessProbe:
            httpGet:
              path: /api/health
              port: 3000
            initialDelaySeconds: 60
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /api/health
              port: 3000
            initialDelaySeconds: 60
            periodSeconds: 10

          imagePullPolicy: Always
          ports:
            - name: metaba-dev-port
              containerPort: 3000



      terminationGracePeriodSeconds: 90

I got this error from

kubectl describe pod metaba-dev

  Warning  Unhealthy                61s (x3 over 81s)      kubelet                                Readiness probe failed: Get "http://10.207.128.197:3000/api/health": dial tcp 10.207.128.197:3000: connect: connection refused
  Warning  Unhealthy                61s (x3 over 81s)      kubelet                                Liveness probe failed: Get "http://10.207.128.197:3000/api/health": dial tcp 10.207.128.197:3000: connect: connection refused

kubectl logs

Picked up JAVA_TOOL_OPTIONS: -Xmx1g -Xms1g -Xmx1g
Warning: environ value jdk-11.0.13+8 for key :java-version has been overwritten with 11.0.13
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
2022-01-28 15:32:23,966 INFO metabase.util :: Maximum memory available to JVM: 989.9 MB
2022-01-28 15:33:09,703 INFO util.encryption :: Saved credentials encryption is ENABLED for this Metabase instance. 🔐
 For more information, see https://metabase.com/docs/latest/operations-guide/encrypting-database-details-at-rest.html

---

Here Solution

I add initialDelaySeconds: to 1200 and check logging it's cause about network mypod cannot connect to database and when i checking log i did not see that cause i has been restart and it's was a new log

Answer 1

Try to change your initialDelaySeconds: 60 to 100 And you should always set the resource request and limit in your container to avoid the probe failure, this is because when your app starts hitting the resource limit the kubernetes starts throttling your container.

containers:


- name: app
    image: images.my-company.example/app:v4
    resources:
      requests:
        memory: "128Mi"
        cpu: "250m"
      limits:
        memory: "100Mi"
        cpu: "500m"