My co-workers are launching GKE clusters and managing them from a pair of centralized VMs. The vms are in us-east4
When they launch GKE clusters in the same region (us-east4), all is well. They can access both the worker nodes and also the GKE Master addresses via the peering connection. However, they could not access the master nodes of a GKE cluster built in europe-west3. I built a VM in that region, and was successfully able to connect to port 443 of the master node IPs. Global routing is enabled for the VPC network and inter-region access of VMs and other services is no problem.
Seems very clear that GKE master nodes can only be accessed in the same region. But is this documented somewhere? I did open a support case on Monday, but having little luck getting any reasonable information back.
It seems like this is an expected behavior. Since I have reviewed here, I understood the following information about it, but you are right, there is nothing like this on it:
The private IP address of the master in a regional cluster only could be reachable from the subnetworks that are in the same region, or from on-premises devices that are connected to the same region.
Now, based on this, I would recommend you to set up a proxy on the same region where your GKE master is, in order to make all the requests coming from a different region, look like they come from the reachable region.
Please review this, it is an example about how to reach your master from a cluster in another region.