We have troubles with the configuration of two GCP components (the GCP Cloud Composer, the GCP Functions) under VPC Service Controls for a single project. We can configure them separately with VPC Service Controls, but as soon as we add the second component we experience troubles.
Once we have configured the Firewall rules and DNS settings for the Cloud Function, the Composer starts to fail. The Composer monitoring tool (for webserver, SQL server, etc.) shows that the composer is unhealthy.
What are we doing wrong?
The docs we've used:
https://cloud.google.com/functions/docs/securing/using-vpc-service-controls
https://cloud.google.com/composer/docs/configuring-vpc-sc
For the Composer we use these settings:
The APIs what we have enabled for VPC Service Controls:
The FireWall settings (based on the documentation):
| Name | Type | Targets | Filters | Protocols / Ports | Action | Priority |
|---|---|---|---|---|---|---|
| composer-egress-gke-cluster-all-port | Egress | Apply to all | IP ranges: 10.10.0.0/16 | tcp, udp | Allow | 1000 |
| composer-egress-healthcheck | Egress | Apply to all | IP ranges: 130.211.0.0/22, 35.191.0.0/16 | tcp:80,443 | Allow | 1000 |
| composer-egress-port53 | Egress | Apply to all | IP ranges: 0.0.0.0/0 | tcp:53, udp:53 | Allow | 1000 |
| composer-egress-webserver | Egress | Apply to all | IP ranges: 172.31.251.0/24 | tcp:3306,3307 | Allow | 1000 |
| restricted-google-apis | Egress | Apply to all | IP ranges: 199.36.153.4/30 | tcp:443 | Allow | 1000 |
| composer-deny-all | Egress | Apply to all | IP ranges: 0.0.0.0/0 | all | Deny | 2000 |
| composer-ingress-helthcheck | Ingress | Apply to all | IP ranges: 130.211.0.0/22, 35.191.0.0/16 | tcp:80,443 | Allow | 1000 |
| ingress-iap | Ingress | Apply to all | IP ranges: 35.235.240.0/20 | tcp | Allow | 1000 |
| gke-europe-west1-composer-dns-xxx-xxx-all | Ingress | gke-europe-west1-composer-dns-xxx-xxx-node | IP ranges: 10.124.0.0/14 | tcp;udp;esp;ah;sctp;icmp | Allow | 1000 |
| gke-europe-west1-composer-dns-xxx-xxx-master | Ingress | gke-europe-west1-composer-dns-xxx-xxx-node | IP ranges: 172.16.8.0/28 | tcp:10250,443 | Allow | 1000 |
| gke-europe-west1-composer-dns-xxx-xxx-vms | Ingress | gke-europe-west1-composer-dns-xxx-xxx-node | IP ranges: 10.10.0.0/16 | tcp:1-65535;udp:1-65535;icmp | Allow | 1000 |
The DNS settings (based on the documentation):
DNS name: googleapis.com. Type: Private
| DNS name | Type | TTl(Seconds) | Data |
|---|---|---|---|
| *.googleapis.com. | CNAME | 300 | restricted.googleapis.com. |
| googleapis.com. | SOA | 21600 | ns-gcp-private.googledomains.com. cloud-dns-hostmaster.google.com. 1 21600 3600 259200 300 |
| googleapis.com. | NS | 21600 | ns-gcp-private.googledomains.com. |
DNS name: cloudfunctions.net. Type: Private
| DNS name | Type | TTl(Seconds) | Data |
|---|---|---|---|
| *.cloudfunctions.net. | A | 300 | 199.36.153.4,199.36.153.5,199.36.153.6,199.36.153.7 |
| cloudfunctions.net | NS | 21600 | ns-gcp-private.googledomains.com. cloud-dns-hostmaster.google.com. 1 21600 3600 259200 300 |
| cloudfunctions.net. | SOA | 21600 | ns-gcp-private.googledomains.com. |
DNS name: pkg.dev. Type: Private
| DNS name | Type | TTl(Seconds) | Data |
|---|---|---|---|
| *.pkg.dev. | CNAME | 300 | pkg.dev. |
| pkg.dev. | SOA | 21600 | ns-gcp-private.googledomains.com. cloud-dns-hostmaster.google.com. 1 21600 3600 259200 300 |
| pkg.dev. | NS | 21600 | ns-gcp-private.googledomains.com. |
| pkg.dev. | A | 300 | 199.36.153.4,199.36.153.5,199.36.153.6,199.36.153.7 |
DNS name: gcr.io. Type: Private
| DNS name | Type | TTl(Seconds) | Data |
|---|---|---|---|
| *.gcr.io. | CNAME | 300 | gcr.io. |
| gcr.io. | SOA | 21600 | ns-gcp-private.googledomains.com. cloud-dns-hostmaster.google.com. 1 21600 3600 259200 300 |
| gcr.io. | NS | 21600 | ns-gcp-private.googledomains.com. |
| gcr.io. | A | 300 | 199.36.153.4,199.36.153.5,199.36.153.6,199.36.153.7 |
Thank you for any idea or help what you can give.
Meanwhile we've figured out the problem:
A type DNS record was missing from googleapis.com. which contains the IP addresses.The two documentations obviously does not cover this special case when two of these services should be used together. After fixing the issues above it has started to work.
The problems what we still experience: