I have a VM instance that I'm trying to access from Notebook AI. It's been functional for about 2 months, but when I tried to start it today it gets hung up on "Setting up proxy to JupyterLab." It's been stuck on that for about 30min and I also can't SSH in to see what's going on. Any thoughts?
Edit: this is what I see in serial port log 1
Welcome to [1mDebian GNU/Linux 9 (stretch)[0m!
[ 3.754088] systemd[1]: No hostname configured.
[ 3.755514] systemd[1]: Set hostname to <localhost>.
[ 3.958833] systemd[1]: Listening on Journal Socket (/dev/log).
[[0;32m OK [0m] Listening on Journal Socket (/dev/log).
[ 3.974149] systemd[1]: Listening on fsck to fsckd communication Socket.
[[0;32m OK [0m] Listening on fsck to fsckd communication Socket.
[ 3.986195] systemd[1]: Listening on udev Kernel Socket.
[[0;32m OK [0m] Listening on udev Kernel Socket.
[ 3.988524] systemd[1]: Listening on udev Control Socket.
[[0;32m OK [0m] Listening on udev Control Socket.
[ 4.002213] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[[0;32m OK [0m] Started Forward Password Requests to Wall Directory Watch.
[[0;32m OK [0m] Set up automount Arbitrary Executab…rmats File System Automount Point.
[[0;32m OK [0m] Listening on Syslog Socket.
[[0;32m OK [0m] Reached target Swap.
[[0;32m OK [0m] Reached target Remote File Systems.
[[0;32m OK [0m] Listening on /dev/initctl Compatibility Named Pipe.
[[0;32m OK [0m] Listening on Journal Socket.
[[0;32m OK [0m] Listening on Journal Audit Socket.
[[0;32m OK [0m] Started Dispatch Password Requests to Console Directory Watch.
[[0;32m OK [0m] Reached target Encrypted Volumes.
[[0;32m OK [0m] Created slice User and Session Slice.
[[0;32m OK [0m] Created slice System Slice.
Mounting POSIX Message Queue File System...
Starting Create list of required st…ce nodes for the current kernel...
Mounting Huge Pages File System...
Mounting Debug File System...
[[0;32m OK [0m] Reached target Slices.
[[0;32m OK [0m] Created slice system-getty.slice.
Starting Journal Service...
[[0;32m OK [0m] Created slice system-serial\x2dgetty.slice.
[[0;32m OK [0m] Created slice system-systemd\x2dfsck.slice.
Starting Load Kernel Modules...
[ 4.224846] gve: loading out-of-tree module taints kernel.
Starting Remount Root and Kernel File Systems...
[ 4.237141] EXT4-fs (sda1): re-mounted. Opts: (null)
[[0;32m OK [0m] Mounted Huge Pages File System.
[[0;32m OK [0m] Mounted POSIX Message Queue File System.
[[0;32m OK [0m] Mounted Debug File System.
[[0;32m OK [0m] Started Journal Service.
[[0;32m OK [0m] Started Create list of required sta…vice nodes for the current kernel.
[[0;32m OK [0m] Started Load Kernel Modules.
[[0;32m OK [0m] Started Remount Root and Kernel File Systems.
Starting Load/Save Random Seed...
Starting udev Coldplug all Devices...
Starting Apply Kernel Variables...
Starting Create Static Device Nodes in /dev...
Starting Flush Journal to Persistent Storage...
[[0;32m OK [0m] Started Load/Save Random Seed.
[[0;32m OK [0m] Started Apply Kernel Variables.
[ 4.408785] systemd-journald[485]: Received request to flush runtime journal from PID 1
[[0;32m OK [0m] Started Flush Journal to Persistent Storage.
[[0;32m OK [0m] Started udev Coldplug all Devices.
[[0;32m OK [0m] Started Create Static Device Nodes in /dev.
[[0;32m OK [0m] Reached target Local File Systems (Pre).
Starting udev Kernel Device Manager...
[[0;32m OK [0m] Started udev Kernel Device Manager.
[ 4.656986] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input3
[ 4.658353] ACPI: Power Button [PWRF]
[ 4.659040] input: Sleep Button as /devices/LNXSYSTM:00/LNXSLPBN:00/input/input4
[ 4.660982] ACPI: Sleep Button [SLPF]
[[0;32m OK [0m] Found device /dev/ttyS0.
[ 4.677261] sd 0:0:1:0: Attached scsi generic sg0 type 0
[ 4.678504] sd 0:0:2:0: Attached scsi generic sg1 type 0
[ 4.697847] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 10737418240 ms ovfl timer
[ 4.699520] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules
[ 4.703833] RAPL PMU: hw unit of domain package 2^-0 Joules
[ 4.705136] RAPL PMU: hw unit of domain dram 2^-16 Joules
[ 4.723655] random: crng init done
[ 4.724722] random: 7 urandom warning(s) missed due to ratelimiting
[[0;32m OK [0m] Found device PersistentDisk.
Starting File System Check on /dev/…58d-07f1-406b-93de-fdcf9c254ceb...
[[0;32m OK [0m] Started File System Check Daemon to report status.
[ 4.929677] EDAC MC: Ver: 3.0.0
[ 4.933734] EDAC sbridge: Ver: 1.1.1
[[0;32m OK [0m] Started File System Check on /dev/d…3b58d-07f1-406b-93de-fdcf9c254ceb.
Mounting /mnt/disks/datadisk...
[ 5.010494] EXT4-fs (sdb): Unrecognized mount option "NOFAIL_OPTION" or missing value
[[0;1;31mFAILED[0m] Failed to mount /mnt/disks/datadisk.
See 'systemctl status mnt-disks-datadisk.mount' for details.
[[0;1;33mDEPEND[0m] Dependency failed for Local File Systems.
Starting Raise network interfaces...
[[0;32m OK [0m] Reached target Timers.
[[0;32m OK [0m] Closed Syslog Socket.
[[0;32m OK [0m] Reached target Login Prompts.
[[0;32m OK [0m] Reached target Sockets.
[[0;32m OK [0m] Started Emergency Shell.
[[0;32m OK [0m] Reached target Emergency Mode.
Starting Create Volatile Files and Directories...
[[0;32m OK [0m] Reached target Paths.
[[0;32m OK [0m] Started Create Volatile Files and Directories.
[[0;32m OK [0m] Reached target System Time Synchronized.
Starting Update UTMP about System Boot/Shutdown...
[[0;32m OK [0m] Started Update UTMP about System Boot/Shutdown.
Starting Update UTMP about System Runlevel Changes...
[[0;32m OK [0m] Started Update UTMP about System Runlevel Changes.
[[0;32m OK [0m] Started Raise network interfaces.
[[0;32m OK [0m] Reached target Network.
[[0;32m OK [0m] Reached target Network is Online.
You are in emerg
Cannot open access to console, the root account is locked.
See sulogin(8) man page for more details.
The issue was because of a mounted disk which I had deleted and, upon doing so, learned that I setup the fstab
wrong for that disk. This made the OS unbootable.
Solution was to delete the instance (but not the boot disk), create a temporary instance with the boot disk attached, fix the fstab
, delete temporary instance, then create a new instance again but with the now-fixed disk as the boot disk.