Search code examples
google-cloud-platformgoogle-cloud-functionsgoogle-secret-manager

Cloud Secret Manager global endpoint not redirecting to working region

I have Google Cloud Secrets which are replicated in multiple Europe regions. The Europe West 9 region is currently down and my services located in Europe West 1 region cannot access my secrets ! Even if those secrets are replicated between multiple regions.

Any tricks to make my programs running in Cloud Function (Europe West 1) access my secrets?

Seems that the design of the replication of GCP Secrets is a failure because the global endpoint does not redirect to a working region.

Solution

  • We had this problem where our datacenter in AWS paris, and as such it was trying to route through google's datacenter which is currently underwater/on fire, even though our secrets themselves don't live there. As Google does not use a regional endpoints, only global, we need to not route through the paris endpoint. We want to trick google by hitting another IP not originally in Paris. We were able to work around this by specifying one of Google Secret Manager's IP address directly instead of letting Google resolve it.

    nslookup secretmanager.googleapis.com

    Name:   secretmanager.googleapis.com
Address: 142.251.45.42
Name:   secretmanager.googleapis.com
Address: 142.251.45.74
Name:   secretmanager.googleapis.com
Address: 142.250.68.138
Name:   secretmanager.googleapis.com
Address: 142.251.32.170
Name:   secretmanager.googleapis.com
Address: 142.251.33.42
Name:   secretmanager.googleapis.com
Address: 192.178.48.138
Name:   secretmanager.googleapis.com
Address: 142.251.40.74
Name:   secretmanager.googleapis.com
Address: 142.251.46.138
Name:   secretmanager.googleapis.com
Address: 142.250.138.95
Name:   secretmanager.googleapis.com
Address: 142.250.115.95
Name:   secretmanager.googleapis.com
Address: 142.250.114.95
Name:   secretmanager.googleapis.com
Address: 142.250.113.95
Name:   secretmanager.googleapis.com
Address: 142.251.32.202
Name:   secretmanager.googleapis.com
Address: 142.251.32.234
Name:   secretmanager.googleapis.com
Address: 142.251.35.202

    We picked the first one, IP in Washington DC. Once we updated our config to force to resolve to this IP, in our case overriding /etc/hosts in k8s via this api, https://kubernetes.io/docs/tasks/network/customize-hosts-file-for-pods/, it worked.

    This is obviously a temporary solution, but is good enough for the duration of google's datacenter being underwater.