Connecting to CloudSQL through cloud-sql-proxy running on a bastion vm

Question

I'm working on a more secure iteration of GCP project setup where there is a bastion host, and the cloudsql instance doesn't have a public IP.

In the current setup cloud-sql-proxy is being used locally to get to CloudSQL instances. But in the new iteration I've got terraform starting cloud-sql-proxy on the bastion vm;

module "bastion-host" {
  source  = "terraform-google-modules/bastion-host/google"
  version = "6.0.0"

  project = var.project
  zone    = var.zone
  network = var.network
  subnet  = var.subnet
  members = var.members
  
  additional_ports = [
    "22",
    "3306",
  ]

  startup_script = <<EOT
    #!/bin/bash
    sudo apt-get update
    $(gcloud info --format="value(basic.python_location)") -m pip install numpy
    URL="https://storage.googleapis.com/cloud-sql-connectors/cloud-sql-proxy/v2.6.1"
    curl "$URL/cloud-sql-proxy.linux.amd64" -o cloud-sql-proxy
    chmod +x cloud-sql-proxy
    ./cloud-sql-proxy --private-ip ${var.project}:${var.db_instance_region}:${var.db_instance_name}
  EOT
  
}

resource "google_project_iam_member" "bastion" {
  for_each = toset(["roles/editor"])
  project  = var.project
  role     = each.value
  member   = "serviceAccount:${module.bastion-host.service_account}"
}

The firewall is configured to allow ports 22 & 3306 for the IAP IP range (35.235.240.0/20). I've verified that cloud-sql-proxy is running on 3306 and ran another instance on another port and connected it to the database instance.

From my machine I can open an SSH tunnel to the bastion vm, but I can't bind port 3306 on there to my machine in order to access the database through that machine. I've found a few things written about this general topic, but nothing that combines the bastion host running cloud-sql-proxy itself for people to route traffic through.

What I want to do is use IAP to bind a local port to 3306 on the bastion vm where cloud-sql-proxy is running, but I get the following;

❯ gcloud compute start-iap-tunnel bastion-vm 3306 --local-host-port=localhost:3310 --zone=europe-west2-a --project=my-project

Testing if tunnel connection works.
ERROR: (gcloud.compute.start-iap-tunnel) While checking if a connection can be made: Error while connecting [4003: 'failed to connect to backend']. (Failed to connect to port 3306)

I can however get an SSH connection to the vm and run cloud-sql-proxy manually;

❯ gcloud compute ssh --zone europe-west2-a bastion-vm --tunnel-through-iap --project my-project

Linux bastion-vm 5.10.0-28-cloud-amd64 #1 SMP Debian 5.10.209-2 (2024-01-31) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Tue Apr  9 09:45:12 2024 from 35.225.242.4
mwalker@bastion-vm:~$ cloud-sql-proxy --private-ip my-project:europe-west2:my-project-8-b7d9f5df
2024/04/09 12:36:07 Authorizing with Application Default Credentials
2024/04/09 12:36:07 [my-project:europe-west2:my-project-8-b7d9f5df] Listening on 127.0.0.1:3307
2024/04/09 12:36:07 The proxy has started successfully and is ready for new connections!

I've also got tinyproxy running on the vm and I can get visibility of kubernetes though that which is the other part of the puzzle of putting everything behind private IPs.

❯ gcloud compute ssh bastion-vm \
    --tunnel-through-iap \
    --project=alldam-production \
    --zone=europe-west2-a \
    --ssh-flag="-4 -L8888:localhost:8888 -N -q -f"

❯ export HTTPS_PROXY=localhost:8888
kubectl get ns
NAME              STATUS   AGE
default           Active   13d
kube-node-lease   Active   13d
kube-public       Active   13d
kube-system       Active   13d

Answer 1

This turned out to be a classic case of missing the simple/obvious.

The startup script was launching cloud-sql-proxy with the default parameters;

./cloud-sql-proxy --private-ip ${var.project}:${var.db_instance_region}:${var.db_instance_name}

This meant that it was listening out on localhost:3306.

Updating this to bind to all network interfaces means I can then connect to it from my local machine.

./cloud-sql-proxy --private-ip --address 0.0.0.0 --port 3306 ${var.project}:${var.db_instance_region}:${var.db_instance_name}

And port added so that it's explicit.