Kubernetes Error When Referencing Service in Different Namespace Despite ReferenceGrant

Question

Issue Description

I am encountering an error when trying to reference a Kubernetes Service in a different namespace using an HTTPRoute, despite having a ReferenceGrant in place. The error occurs only when the Service is referenced from a different namespace. Referencing the same Service from the same namespace works without issues.

Error Message:

Error cause: reference-not-permitted: Error GWCER104: HTTPRoute "backbone-meta-green/backbone-meta-green-http-route" is misconfigured, err: cannot use backend backbone-meta-blue/httpbin, references to backends in other namespaces are not supported.

Manifest Configuration

Below is the manifest configuration I am using:

apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
    name: cross-ns-gateway
    namespace: gateway-infra
spec:
    addresses:
        - type: NamedAddress
          value: cross-ns-gateway-ip
    gatewayClassName: gke-l7-global-external-managed
    listeners:
        - allowedRoutes:
              namespaces:
                  from: All
          name: https
          port: 443
          protocol: HTTPS
          tls:
              mode: Terminate
              certificateRefs:
                  - name: gateway-cert-secret

---
apiVersion: apps/v1
kind: Deployment
metadata:
    name: httpbin
    namespace: cross-ns-1
spec:
    replicas: 1
    selector:
        matchLabels:
            app: httpbin
    template:
        metadata:
            labels:
                app: httpbin
        spec:
            containers:
                - name: httpbin
                  image: kennethreitz/httpbin
                  ports:
                      - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
    name: httpbin
    namespace: cross-ns-1
spec:
    selector:
        app: httpbin
    ports:
        - protocol: TCP
          port: 8080
          targetPort: 80
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
    name: cross-ns-reference-grant
    namespace: cross-ns-1
spec:
    from:
        - group: gateway.networking.k8s.io
          kind: HTTPRoute
          namespace: cross-ns-2
    to:
        - group: ''
          kind: Service
          name: httpbin
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
    name: cross-ns-http-route
    namespace: cross-ns-2
spec:
    hostnames:
        - cross-ns-test.com
    parentRefs:
        - group: gateway.networking.k8s.io
          kind: Gateway
          name: cross-ns-gateway
          namespace: gateway-infra
    rules:
        - backendRefs:
              - group: ''
                kind: Service
                port: 8080
                name: httpbin
                namespace: cross-ns-2
          matches:
              - path:
                    type: PathPrefix
                    value: /

Expected Behavior

I expect the HTTPRoute in cross-ns-2 to successfully reference the Service httpbin in cross-ns-1 with the ReferenceGrant in place.

Actual Behavior

The HTTPRoute is unable to reference the Service in a different namespace, resulting in the aforementioned error.

Steps to Reproduce

1. Deploy the Gateway, Service, and Deployment as per the provided manifests.
2. Apply the ReferenceGrant.
3. Deploy the HTTPRoute.
4. Observe the error.

Additional Context

Cloud Provider: Google Cloud Platform

Assistance Required

I would appreciate any guidance or suggestions on resolving this issue, especially if there are additional configurations or steps I may have missed that allow for cross-namespace service references in this context.

Answer 1

ReferenceGrant is currently not supported by GKE gateway. We have below workarounds, so choose the best which will help you to resolve your issue:

1. You can use "RequestMirror" to support traffic mirroring for the same request.

2. There is a new feature in a multi-cluster called multi-cluster Gateways which is still in preview . You can enable it using this official GCP document and deploy it using this doc. You will have to consider the requirements in this document before you are able to enable and use this feature. Please notice that using this feature in Preview means it carries no technical support.

The issue is that public documentation doesn't mention the cross namespace routing option. Follow the suggestions in the blog by Glen while using the official doc. The example in the blog is for a single cluster, so you have to use the official doc but make the changes suggested in the blog. The only major change is the way you create the name space. There is an additional label to add shared-gateway-access: "true" to your namespace YAML.

If the above workaround does not help feel free to raise a feature request.