I'm porting a django management command to a new [private] GKE cluster configured with service accounts & workload identity. This command uses the kubernetes API to change settings on the autoscaler for the cluster.
It looks like the API connection requires a token and a certificate. These are bundled up to create the configuration;
configuration = kubernetes.client.Configuration()
configuration.api_key["authorization"] = token
configuration.api_key_prefix["authorization"] = "Bearer"
configuration.host = server
configuration.ssl_ca_cert = cert
api = kubernetes.client.AutoscalingV1Api(
kubernetes.client.ApiClient(configuration)
)
The existing project that I'm porting this command from uses defaults for token and certificate which are defined as;
parser.add_argument(
"--cert",
action="store",
dest="cert",
default="/var/run/secrets/kubernetes.io/serviceaccount/ca.crt",
help="File containing valid certificate to make request",
)
parser.add_argument(
"--token",
action="store",
dest="token",
type=argparse.FileType("r"),
default="/var/run/secrets/kubernetes.io/serviceaccount/token",
help="File containing token to make request",
)
I've noticed that these aren't added by GKE by default. And looking at the pods for the existing project, I can see that /var/run/secrets doesn't exist. So I think this cluster is able to use this API via it's default service account, whereas this new cluster doesn't use that SA.
The error I'm seeing come from attempts to run this command point at the missing certificate;
HTTPSConnectionPool(host='10.255.240.1', port=443): Max retries exceeded with url: /apis/autoscaling/v1/namespaces/staging/horizontalpodautoscalers/draft-nginx (Caused by SSLError(FileNotFoundError(2, 'No such file or directory')))
I found the google docs on how I can mount a token. So the helm for that is in my templates and I've verified the token in a pod;
containers:
- name: scale-workloads
image: {{ .Values.gke_registry }}/base_python:{{ .Values.global.build }}
imagePullPolicy: Always
command:
- python -m django
args:
- scale_workloads
- --namespace={{ .Release.Namespace }}
- --appserver={{ .Values.pods.appserver.minReplicas | default 1 }}
- --nginx={{ .Values.pods.nginx.minReplicas | default 1 }}
env:
{{- include "proj.sharedEnv" $ | nindent 16 }}
- name: DJANGO_SETTINGS_MODULE
value: {{ .Values.django_settings_module }}
resources:
requests:
cpu: 1000m
memory: 500Mi
volumeMounts:
- mountPath: /etc/config
name: configs
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: ksa-token
volumes:
- name: configs
projected:
defaultMode: 420
sources:
- secret:
name: proj-secrets
- name: ksa-token
projected:
sources:
- serviceAccountToken:
path: ksa-token
expirationSeconds: 86400
audience: some-oidc-audience
But can't find any similar docs on mounting a certificate that the cluster either is, or could, be using.
The stacktrace from manually running this management command shows the following;
File "/usr/src/app/drafty/core/management/commands/scale_workloads.py", line 198, in scale_pods
api.patch_namespaced_horizontal_pod_autoscaler(
File "/usr/local/lib/python3.11/site-packages/kubernetes/client/api/autoscaling_v1_api.py", line 983, in patch_namespaced_horizontal_pod_autoscaler
return self.patch_namespaced_horizontal_pod_autoscaler_with_http_info(name, namespace, body, **kwargs) # noqa: E501
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/kubernetes/client/api/autoscaling_v1_api.py", line 1098, in patch_namespaced_horizontal_pod_autoscaler_with_http_info
return self.api_client.call_api(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api
return self.__call_api(resource_path, method,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
response_data = self.request(
^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 407, in request
return self.rest_client.PATCH(url,
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/kubernetes/client/rest.py", line 296, in PATCH
return self.request("PATCH", url,
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/kubernetes/client/rest.py", line 169, in request
r = self.pool_manager.request(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/urllib3/request.py", line 78, in request
return self.request_encode_body(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/urllib3/request.py", line 170, in request_encode_body
return self.urlopen(method, url, **extra_kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/urllib3/poolmanager.py", line 376, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 826, in urlopen
return self.urlopen(
^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 826, in urlopen
return self.urlopen(
^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 826, in urlopen
return self.urlopen(
^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 798, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='10.255.240.1', port=443): Max retries exceeded with url: /apis/autoscaling/v1/namespaces/staging/horizontalpodautoscalers/draft-nginx (Caused by SSLError(FileNotFoundError(2, 'No such file or directory')))
I've solved this by getting the certificate from the cluster;
import base64
import kubernetes.client
from google.auth import compute_engine
from google.cloud.container_v1 import ClusterManagerClient
from kubernetes.client.rest import ApiException
from python_hosts import Hosts, HostsEntry
...
# Get cluster details
credentials = compute_engine.Credentials()
cluster_manager_client = ClusterManagerClient(credentials=credentials)
cluster = cluster_manager_client.get_cluster(
name=f"projects/{settings.GCP_PROJECT}/locations/{settings.GCP_REGION}/clusters/{settings.GCP_PROJECT}"
)
# Save cluster certificate for SSL verification
cert = base64.b64decode(cluster.master_auth.cluster_ca_certificate)
cert_filename = "cluster_ca_cert"
with open(cert_filename, "wb") as cert_file:
cert_file.write(cert)
# Configure hostname for SSL verification
hosts = Hosts()
hosts.add(
[
HostsEntry(
entry_type="ipv4",
address=cluster.endpoint,
names=["kubernetes"],
)
]
)
hosts.write()
configuration = kubernetes.client.Configuration()
configuration.host = f"https://{cluster.endpoint}:443"
configuration.api_key["authorization"] = token
configuration.api_key_prefix["authorization"] = "Bearer"
configuration.ssl_ca_cert = cert_filename
kubernetes.client.Configuration.set_default(configuration)
api = kubernetes.client.AutoscalingV1Api(
kubernetes.client.ApiClient(configuration)
)