I am trying to connect to my GCP Sql instance using a docker cloud-sql container. I also have a service that depends on cloudsql to connect to the database in GKE. Together, these two create a pod.
The connection always seems to fail about 3 or 4 times and then connects successfully. this happens every time want to deploy a new version of my service by creating an updated kubernetes pod.
My kubernetes deployment that describe this code is:
...
- name: cloudsql-proxy
image: gcr.io/cloudsql-docker/gce-proxy:1.12
command:
- /cloud_sql_proxy
- -instances=my-project:europe-west1:my-instance=tcp:5432
- -credential_file=/secrets/cloudsql/credentials.json
...
I expect cloud-sql to connect immediately to my instance but I get this error a couple of times on my logs
couldn't connect to "my-project:europe-west1:my-instance": Post https://www.googleapis.com/sql/v1beta4/projects/my-project/instances/my-instance/createEphemeral?alt=json&prettyPrint=false: oauth2: cannot fetch token: Post https://oauth2.googleapis.com/token: net/http: TLS handshake timeout
This error indicates an overloaded pod or a slow network startup time. Since you are connecting to a Google OAuth endpoint, I will rule out the other side being the problem.
If you have the cluster or a pod is overloaded (hitting memory limits, CPU at 100%, etc) network response can start failing.