Connection problems with APISIX in GCP Cloud Run

Question

I would like to set up APISIX in the Google Cloud Platform on a Cloud Run instance and configure other Cloud Run instances as upstreams.

My APISIX setup:

I configure my APISIX Cloud Run container with the following files:

# Dockerfile

FROM apache/apisix

COPY ./configs/config.yaml /usr/local/apisix/conf/config.yaml
COPY ./configs/apisix.yaml /usr/local/apisix/conf/apisix.yaml

configs/apisix.yaml

routes:
  - 
    uri: /hello
    plugins:
      proxy-rewrite:
        uri: /hello
    upstream:
      nodes:
        "<CLOUD_RUN_URL>.run.app": 1
      scheme: https
#END

configs/config.yaml

deployment:
  role: data_plane
  role_data_plane:
    config_provider: yaml
#END

I build and push the Docker Image to GCPs Artifact Registry and then I build both Cloud Run Services using the following Terraform Code:

# main.tf

# APISIX Docker Image
data "google_artifact_registry_docker_image" "apisix_image" {
  project       = var.project_id
  location      = "europe-west3"
  repository_id = <ARTIFACT_REGISTRY>
  image_name    = "<APISIX_IMAGE>:latest"
}

# Cloud Run APISIX
resource "google_cloud_run_v2_service" "apisix" {
  project  = var.project_id
  location = "europe-west3"
  name     = "apisix"
  ingress  = "INGRESS_TRAFFIC_INTERNAL_LOAD_BALANCER"

  template {
    containers {
      image = data.google_artifact_registry_docker_image.apisix_image.self_link
      ports {
        container_port = 9080
      }
      env {
        name  = "APISIX_STAND_ALONE"
        value = true
      }
    }
    vpc_access {
      egress = "ALL_TRAFFIC"
      network_interfaces {
        network    = "<NETWORK_NAME>"
        subnetwork = "<SUBNETWORK_NAME>"
      }
    }
  }
}

# Cloud Run Upstream API
resource "google_cloud_run_v2_service" "api" {
  project  = var.project_id
  location = "europe-west3"
  name     = "upstream-api"
  ingress  = "INGRESS_TRAFFIC_INTERNAL_LOAD_BALANCER"

  template {
    containers {
      image = "us-docker.pkg.dev/cloudrun/container/hello"
    }
  }
}

However, if I deploy the whole thing in GCP I get a 404 error and the request never reaches my upstream cloud run container. Both Cloud Run services are in the same GCP project.

Error Log:: When I try to access the Cloud Run Upstream URL, I get the following error log in the APISIX Cloud Run Service:

<APISIX_Cloud_Run_URL> "GET /hello HTTP/1.1" 404 36 0.050 "-" "curl" 199.36.153.9:443 404 0.019 "<APISIX_Cloud_Run_URL>/hello"

It can be seen that APISIX can correctly determine the IP of the upstream to be called ...

It is also interesting that I can reach the upstream cloud run container from another cloud run container running a FastAPI service (with the Python requests package). This FastAPI Cloud Run Service is configured in exactly the same way as the APISIX Cloud Run Service. So the whole thing sounds to me very much like a problem with APISIX itself.

Do any of you have experience with this or a similar setup? If you need more details, I am happy to provide more information.

Thanks in advance for any advice and help!

Additional info: I also found out the following - If I put an internal load balancer in front of the upstream cloud run service and configure the internal IP of the load balancer (alternatively also a private DNS domain) as an upstream node in APISIX, I can reach the backend. This means that, in principle, internal IPs/URLs can be resolved/reached by APISIX. However, for some reason it does not work with the Cloud Run URL.

Answer 1

Here’s my thoughts regarding your issue with APISIX service. APISIX might not handle certain host headers or HTTPS-based upstreams correctly out of the box. This could result in misrouting or failure to resolve upstreams.

Add a proxy-rewrite plugin configuration in your apisix.yaml to explicitly set the Host header to match the upstream's Cloud Run URL.

routes:
  -
    uri: /hello
    plugins:
      proxy-rewrite:
        uri: /get
        headers:
          Host: "<CLOUD_RUN_URL>.run.app"
    upstream:
      nodes:
        "<CLOUD_RUN_URL>.run.app": 1
      scheme: https

Hope this one works!