Health checks for service returning 301 after updating deployment

Question

We recently updated the deployment of a dropwizard service deployed using Docker and Kubernetes.

It was working correctly before, the readiness probe was yielding a healthcheck ping to internal cluster IP getting 200s. Since we updated the healthcheck pings are resulting in a 301 and the service is considered down.

I've noticed that the healthcheck is now Default kubernetes L7 Loadbalancing health check for NEG. (port is set to 80) where it was previously Default kubernetes L7 Loadbalancing health check. where the port was configurable.

The kube file is deployed via CircleCI but the readiness probe is:

kind: Deployment
metadata:
  name: pes-${CIRCLE_BRANCH}
  namespace: ${GKE_NAMESPACE_NAME}
  annotations:
    reloader.stakater.com/auto: 'true'
spec:
  replicas: 2
  selector:
    matchLabels:
      app: ***
  template:
    metadata:
      labels:
        app: ***
    spec:
      containers:
        - name: ***
          image: ***
          envFrom:
            - configMapRef:
                name: ***
            - secretRef:
                name: ***
          command: ['./gradlew', 'run']
          resources: {}
          ports:
            - name: pes
              containerPort: 5000
          readinessProbe:
            httpGet:
              path: /api/healthcheck
              port: pes
            initialDelaySeconds: 15
            timeoutSeconds: 30

---
apiVersion: v1
kind: Service
metadata:
  name: ***
  namespace: ${GKE_NAMESPACE_NAME}
spec:
  ports:
    - name: pes
      port: 5000
      targetPort: pes
      protocol: TCP
  selector:
    app: ***
  type: LoadBalancer

Any ideas on how this needs to be configured in GCP?

I have a feeling that the new deployment has changed from legacy health check to non legacy but no idea what else needs to be set up for it to work. Does the kube file handle creating firewall rules or does that need to be done manually?

Reading the docs at https://cloud.google.com/load-balancing/docs/health-check-concepts?hl=en

EDIT: Issue is now resolved. After GKE version was updated it is now creating a NEG healthcheck by default. We disabled this by adding below annotation to service deployment file.

metadata: annotations: cloud.google.com/neg: '{"ingress":false}'

Answer 1

Issue is now resolved. After GKE version was updated it is now creating a NEG healthcheck by default. We disabled this by adding below annotation to service deployment file.

metadata: annotations: cloud.google.com/neg: '{"ingress":false}'