Can not configure hazelcast monitoring center health check (different port) on GKE

Question

I'm kind of new with K8s things, and now run into the problem that I can not configure health check on Google Cloud ingress/load balancer. The hazelcast man-center has health check on port 8081, but the working port is 8080. When I'm deploying the app into GKE, the health checks are done on working nodePort that refers to the working port (8080). In documentation here and here is said that I need to add custom BackendConfig and configure in service for the health check nodePort that refers to the app health check port (8081). I did so, but it did not override default health check on work port. So it fails.

These are my yaml files:

hazelcast-mc-backend-config.yaml

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: hazelcast-mc-backend-config
spec:
  healthCheck:
    checkIntervalSec: 60
    timeoutSec: 60
    healthyThreshold: 1
    unhealthyThreshold: 10
    type: HTTP2
    requestPath: /hazelcast-mc/health
    port: 31111
  timeoutSec: 86400
  connectionDraining:
    drainingTimeoutSec: 30
  customRequestHeaders:
    headers:
      - "X-Client-Region:{client_region}"
      - "X-Client-City:{client_city}"
      - "X-Origin-Request-Header:{origin_request_header}"
      - "X-TLS-Version:{tls_version}"

hazelcast-mc-service.yaml

apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/backend-config: '{"ports": {"8081":"hazelcast-mc-backend-config"}}'
    cloud.google.com/app-protocols: '{"hazelcast-management-center":"HTTP2"}'
  name: hazelcast-management-center
spec:
  ports:
    - name: hazelcast-management-center
      port: 443
      targetPort: 8080
    - name: hazelcast-management-center-health
      port: 8081
      targetPort: 8081
      nodePort: 31111
  selector:
    app: hazelcast-management-center
    tier: backend
  type: NodePort

hazelcast-mc-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hazelcast-management-center
  labels:
    app: hazelcast-management-center
    tier: backend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hazelcast-management-center
  strategy:
    rollingUpdate:
      maxSurge: 3
      maxUnavailable: 50%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: hazelcast-management-center
        tier: backend
    spec:
      containers:
        - name: hazelcast-management-center
          image: hazelcast/management-center:5.1.3
          imagePullPolicy: "Always"
          securityContext:
            runAsUser: 2  # non-root user
            allowPrivilegeEscalation: false
          resources:
            requests:
              memory: 128Mi
              cpu: 20m
            limits:
              memory: 256Mi
              cpu: 200m
          env:
            - name: MC_ADMIN_USER
              value: admin
            - name: MC_ADMIN_PASSWORD
              value: random
            - name: MC_CONTEXT_PATH
              value: /hazelcast-mc
            - name: MC_HEALTH_CHECK_ENABLE
              value: "true"
          ports:
            - name: mancenter
              containerPort: 8080
            - name: health
              containerPort: 8081
          livenessProbe:
            httpGet:
              path: /hazelcast-mc/health
              port: 8081
            initialDelaySeconds: 45
            periodSeconds: 10
            timeoutSeconds: 5
            successThreshold: 1
            failureThreshold: 10
          readinessProbe:
            httpGet:
              path: /hazelcast-mc/health
              port: 8081
            initialDelaySeconds: 45
            periodSeconds: 10
            timeoutSeconds: 1
            successThreshold: 1
            failureThreshold: 3

What did I do wrong ?

Answer 1

The problem was solved by configuring Backendconfig as default config to the service. The healthcheck for external load balancer is using nodePort, so I had to specify it.

apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/backend-config: '{"default": "hazelcast-mc-backend-config"}'
  name: hazelcast-management-center
spec:
  ports:
    - name: hazelcast-management-center
      port: 80
      targetPort: 8080
    - name: hazelcast-management-health
      port: 8081
      targetPort: 8081
      nodePort: 31111
  selector:
    app: hazelcast-management-center
    tier: backend
  type: NodePort