SSH to GKE node from POD

Question

Created the basic GKE with spot instance to save cost

gcloud container clusters create test-cluster \
    --machine-type=e2-medium \
    --num-nodes=1 \
    --spot \
    --zone=us-central1-c \
    --workload-pool=<Project-id>.svc.id.goog \
    --release-channel rapid \
    --addons HttpLoadBalancing

Created the Service Account and added the Roles necessary, image attached below

IAM looks something like this, so ServiceAccount now have Access too, added Owner as last option

This my GKE workload

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    iam.gke.io/gcp-service-account: "node-ssh-access@auto-x-244507.iam.gserviceaccount.com"
  name: daemonset-access
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: command
data:
  command.sh: |
    #!/bin/bash
    echo "running sh script on node..!"
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-gke
  labels:
    k8s-app: fluentd-logging
spec:
  selector:
    matchLabels:
      name: node-gke
  template:
    metadata:
      labels:
        name: node-gke
    spec:
      serviceAccountName: daemonset-access
      securityContext:
        runAsUser: 0
      containers:  
      - name: node-gke
        image: gcr.io/google.com/cloudsdktool/google-cloud-cli:latest
        command: [ "/bin/sh" , "-c", "tail -f /dev/null" ]
        volumeMounts:
        - name: commandfile
          mountPath: /test/command.sh
          subPath: command.sh
        - name: script-dir
          mountPath: /test
        securityContext:
          runAsUser: 0
          privileged: true
          allowPrivilegeEscalation: true
      volumes:
      - name: commandfile
        configMap:
          name: command
          defaultMode: 0777
      - name: script-dir
        hostPath:
          path: /var/log/data
          type: DirectoryOrCreate

As of now the workload is an image with gcloud installed, as ServiceAccount is attached i am able to Access GCP API & list the Instances running.

i am trying to do the SSH from inside of above workload POD with gcloud to GKE node, however i getting errors.

The command i am tried :

gcloud compute ssh gke-test-cluster-default-pool-0e59e91e-q5x6 --zone us-central1-c --impersonate-service-account=node-ssh-access@auto-x-244507.iam.gserviceaccount.com

gcloud compute ssh gke-test-cluster-default-pool-0e59e91e-q5x6 --zone us-central1-c --impersonate-service-account=node-ssh-access@auto-x-244507.iam.gserviceaccount.com --container=3a7aed0fce5f2

 gcloud compute ssh gke-test-cluster-default-pool-0e59e91e-q5x6 --zone us-central1-c --impersonate-service-account=node-ssh-access@auto-x-244507.iam.gserviceaccount.com --container=3a7aed0fce5f2 --internal-ip

Not able to do the SSH to GKE node from POD. Able to do SSH from a local machine with serviceAccount

The error i am seeing inside POD

i know we can set the SSH-Key to Node, save it in K8s secret and use it inside the POD but not looking for that way as workload identity there, plus i think there is no OS-login enabled on my GKE or project level too.

Local :

Answer 1

i wanted the above case as Log service daemon on node needs to be restarted once there is any change in Log config (configmap/secret) to apply ASAP.

It worked Maybe there was some other issues with script without any error worked like a charm and was able to SSH with

gcloud compute ssh gke-test-cluster-default-pool-0e59e91e-q5x6 --zone us-central1-c --impersonate-service-account=node-ssh-access@auto-x-244507.iam.gserviceaccount.com

inside of the POD.

Another option is privileged POD which is similar to i shared in answer : https://stackoverflow.com/a/72485840/5525824 as suggested by @boredabdel

If you are on older cluster version you might hit on PodSecurityPolicy to make sure there is no PSP set, however, it is deprecated in the newer releases of K8s added as an admission controller part now.