Search code examples
linuxsshgoogle-compute-enginedisk

Unable to increase disk size on file system

I'm currently trying to log in to one of the instances created on google cloud, but found myself unable to do so. Somehow the machine escaped my attention and the hard disk got completely full. Of course I wanted to free some disk space and make sure the server running could restart, but I am facing some issues.

First off, I have found the guide on increasing the size of the persistent disk (https://cloud.google.com/compute/docs/disks/add-persistent-disk). I followed that and already set it 50 GB which should be fine for now.

However, on file system level because my disk is full I cannot make any SSH connection. The error is simply a timeout caused by the fact that there is absolutely no space for the SSH deamon to write to its log. Without any form of connection I cannot free some disk space and/or run the "resize2fs" command.

Furthermore, I already tried different approaches.

  1. I seem to not be able to change the boot disk to something else.
  2. I created a snapshot and tried to increase the disk size on the new instance I created from that snapshot, but it has the same problem (filesystem is stuck at 15GB).
  3. I am not allowed to mount the disk as an additional disk in another instance.

Currently I'm pretty much out of ideas. The important data on the disk was back-upped but I'd rather have the settings working as well. Does anyone have any clues as where to start?

[EDIT] Currently still trying out new things. I have also tried to run shutdown- and startup scripts that remove /opt/* in order to free some temporary space but the script either don't run or provide some error I cannot catch. It's pretty frustrating working nearly blind I must say.

The next step for me would be to try and get the snapshot locally. It should be doable using the bucket but I will let you know.

[EDIT2] Getting a snapshot locally is not an option either or so it seems. Images from the google cloud instances can only be created or deleted, but not downloaded.

I'm now out of ideas.

Solution

  • So I finally found the answer. These steps were taken:

    • In the GUI I increased the size of the disk to 50 GB.
    • In the GUI I detached the drive by deleting the machine whilst ensuring that I did not throw away the original disk.
    • In the GUI I created a new machine with a sufficiently big harddisk.
    • On the command line (important!!) I attached the disk to the newly created machine (the GUI option has a bug still ...)

    After that I could mount the disk as a secondary disk and perform all the operations I needed.

    Keep in mind: By default google cloud solutions do NOT use logical volume management, so pvresize/lvresize/etc. is not installed and resize2fs might not work out of the box.