google-cloud-platform google-compute-engine spring-cloud-gcp

GCP's SSH terminal not working after stopping and starting vm instance

I am using gcp vm machine instance N1-standard 8V-30GB and N1-standard 4V-15GB

os-Debian
version - Debian GNU/Linux 10(buster)

this issue i am facing from last 1 month. public access permission denied is one of message i am seeing while trying to access from cloud shell I had run command chmod 777 <home directory> earlier.

Solution

I've tried to reproduce your steps and was able to solve this issue.

Please have a look at my steps below:

create VM instances:

gcloud compute instances create instance-1 --zone=europe-west3-a --machine-type=e2-medium --image=ubuntu-1804-bionic-v20200701 --image-project=ubuntu-os-cloud
gcloud compute instances create instance-2 --zone=europe-west3-a --machine-type=e2-medium --image=ubuntu-1804-bionic-v20200701 --image-project=ubuntu-os-cloud

change permissions recursively on my home directory at the VM instance instance-1:

instance-1:~$ chmod -R 777 ~
instance-1:~$ ls -la
...
drwxrwxrwx 2 username username 4096 Jul 15 07:50 .ssh

create snapshot of the VM instance instance-1 boot disk:

gcloud compute disks snapshot instance-1 --snapshot-names instance-1-snapshot --zone=europe-west3-a

create a new disk with the snapshot:

gcloud compute disks create instance-1-snapshot-disk --zone=europe-west3-a --source-snapshot=instance-1-snapshot

attach created disk instance-1-snapshot-disk to the VM instance instance-2:

instance-2:~$ ls -l /dev/ | grep sd
brw-rw---- 1 root disk      8,     0 Jul 15 07:39 sda
brw-rw---- 1 root disk      8,     1 Jul 15 07:39 sda1
brw-rw---- 1 root disk      8,    14 Jul 15 07:39 sda14
brw-rw---- 1 root disk      8,    15 Jul 15 07:39 sda15
instance-2:~$ mount | grep sda
/dev/sda1 on / type ext4 (rw,relatime)
/dev/sda15 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)

then

gcloud compute instances attach-disk instance-2 --disk=instance-1-snapshot-disk --zone=europe-west3-a

after that

instance-2:~$ ls -l /dev/ | grep sd
brw-rw---- 1 root disk      8,     0 Jul 15 07:39 sda
brw-rw---- 1 root disk      8,     1 Jul 15 07:39 sda1
brw-rw---- 1 root disk      8,    14 Jul 15 07:39 sda14
brw-rw---- 1 root disk      8,    15 Jul 15 07:39 sda15
brw-rw---- 1 root disk      8,    16 Jul 15 08:04 sdb
brw-rw---- 1 root disk      8,    17 Jul 15 08:04 sdb1
brw-rw---- 1 root disk      8,    30 Jul 15 08:04 sdb14
brw-rw---- 1 root disk      8,    31 Jul 15 08:04 sdb15
instance-2:~$ sudo mkdir /mnt/instance-1-snapshot-disk
instance-2:~$ sudo mount /dev/sdb1 /mnt/instance-1-snapshot-disk
instance-2:~$ ls -la /mnt/instance-1-snapshot-disk
total 104
drwxr-xr-x 23 root root  4096 Jul 15 07:56 .
drwxr-xr-x  3 root root  4096 Jul 15 08:05 ..
drwxr-xr-x  2 root root  4096 Jul  1 19:14 bin
drwxr-xr-x  4 root root  4096 Jul  1 19:19 boot
drwxr-xr-x  4 root root  4096 Jul  1 19:11 dev
drwxr-xr-x 93 root root  4096 Jul 15 07:55 etc
drwxr-xr-x  4 root root  4096 Jul 15 07:50 home
lrwxrwxrwx  1 root root    30 Jul  1 19:18 initrd.img -> boot/initrd.img-5.3.0-1030-gcp
lrwxrwxrwx  1 root root    30 Jul  1 19:18 initrd.img.old -> boot/initrd.img-5.3.0-1030-gcp
drwxr-xr-x 22 root root  4096 Jul  1 19:17 lib
drwxr-xr-x  2 root root  4096 Jul  1 19:01 lib64
drwx------  2 root root 16384 Jul  1 19:13 lost+found
drwxr-xr-x  2 root root  4096 Jul  1 19:01 media
drwxr-xr-x  2 root root  4096 Jul  1 19:01 mnt
drwxr-xr-x  2 root root  4096 Jul  1 19:01 opt
drwxr-xr-x  2 root root  4096 Apr 24  2018 proc
drwx------  3 root root  4096 Jul 15 07:36 root
drwxr-xr-x  4 root root  4096 Jul  1 19:19 run
drwxr-xr-x  2 root root  4096 Jul  1 19:17 sbin
drwxr-xr-x  6 root root  4096 Jul 15 07:36 snap
drwxr-xr-x  2 root root  4096 Jul  1 19:01 srv
drwxr-xr-x  2 root root  4096 Apr 24  2018 sys
drwxrwxrwt  7 root root  4096 Jul 15 07:56 tmp
drwxr-xr-x 10 root root  4096 Jul  1 19:01 usr
drwxr-xr-x 13 root root  4096 Jul  1 19:12 var
lrwxrwxrwx  1 root root    27 Jul  1 19:18 vmlinuz -> boot/vmlinuz-5.3.0-1030-gcp
lrwxrwxrwx  1 root root    27 Jul  1 19:18 vmlinuz.old -> boot/vmlinuz-5.3.0-1030-gcp

change permissions:
- .ssh directory: 700 drwx------
- public key (.pub file): 644 -rw-r--r--
- private key (id_rsa): 600 -rw-------
- lastly your home directory should not be writeable by the group or others: 755 drwxr-xr-x

instance-2:~$ chmod -R 755 /mnt/instance-1-snapshot-disk/home/username/
instance-2:~$ chmod -R 700 /mnt/instance-1-snapshot-disk/home/username/.ssh/
instance-2:~$ chmod 644 /mnt/instance-1-snapshot-disk/home/username/.ssh/authorized_keys

unmount the disk when you finish:

instance-2:~$ sudo umount /mnt/instance-1-snapshot-disk/

detach disk instance-1-snapshot-disk from the VM instance instance-2:

gcloud compute instances detach-disk instance-2 --disk=instance-1-snapshot-disk --zone=europe-west3-a

create a new instance from the repaired disk:

gcloud compute instances create instance-3 --zone=europe-west3-a --machine-type=e2-medium --disk=name=instance-1-snapshot-disk

check SSH connection to at the VM instance instance-1.

In addition, please have a look at the documentation Troubleshooting SSH section Inspect the VM instance without shutting it down to find more details.