Search code examples
ubuntuvagrantbootmountsupervisord

Vagrant fails to properly boot box after host reboot, causes mount errors


Summary

I have a Vagrant box with Ubuntu 14.04 on my Arch Linux host. It can't boot properly when supervisord is running at the guest and the host kills the virtualbox process which is done at every host shutdown if don't manually use vagrant halt.

Problem

When i create the machine, vagrant up and vagrant halt && vagrant up work properly. I mount a shared folder which is used by some services in the box. Here is the successful output:

==> default: Checking if box 'ubuntu/trusty64' is up to date...
==> default: Clearing any previously set forwarded ports...
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
    default: Adapter 2: hostonly
==> default: Forwarding ports...
    default: 22 => 2222 (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
    default: Warning: Connection timeout. Retrying...
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
==> default: Setting hostname...
==> default: Configuring and enabling network interfaces...
==> default: Mounting shared folders...
    default: /myproject => /home/myuser/myproject
==> default: Machine already provisioned. Run `vagrant provision` or use the `--provision`
==> default: to force provisioning. Provisioners marked to run always will still run.

When i reboot the host vagrant refuses to mount the folder anymore and gives me the following error:

==> default: Checking if box 'ubuntu/trusty64' is up to date...
==> default: Clearing any previously set forwarded ports...
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
    default: Adapter 2: hostonly
==> default: Forwarding ports...
    default: 22 => 2222 (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
    default: Warning: Connection timeout. Retrying...
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
==> default: Setting hostname...
==> default: Configuring and enabling network interfaces...
==> default: Mounting shared folders...
    default: /myproject => /home/myuser/myproject
Failed to mount folders in Linux guest. This is usually because
the "vboxsf" file system is not available. Please verify that
the guest additions are properly installed in the guest and
can work properly. The command attempted was:

mount -t vboxsf -o uid=`id -u vagrant`,gid=`getent group vagrant | cut -d: -f3` myproject /myproject
mount -t vboxsf -o uid=`id -u vagrant`,gid=`id -g vagrant` myproject /myproject

Here is the Vagrantfile: http://pastebin.com/DCMMrUym

Detailed Output

Here are the last lines of dmesg of successful boots (vagrant up or vagrant halt && vagrant up):

[    4.721766] vboxvideo: Unknown symbol drm_open (err 0)
[    4.721768] vboxvideo: Unknown symbol drm_poll (err 0)
[    4.721770] vboxvideo: Unknown symbol drm_pci_init (err 0)
[    4.721772] vboxvideo: Unknown symbol drm_ioctl (err 0)
[    4.721774] vboxvideo: Unknown symbol drm_vblank_init (err 0)
[    4.721775] vboxvideo: Unknown symbol drm_mmap (err 0)
[    4.721776] vboxvideo: Unknown symbol drm_pci_exit (err 0)
[    4.721778] vboxvideo: Unknown symbol drm_release (err 0)
[    4.986464] init: udev-fallback-graphics main process (889) terminated with status 1
[    5.051943] vboxsf: Successfully loaded version 4.3.10_Ubuntu (interface 0x00010004)
[    6.992684] init: plymouth-upstart-bridge main process (180) killed by TERM signal

Full output: http://pastebin.com/cktKJBZT


Then i grep for "virtualbox" and kill the process of that machine. Or i simply reboot my host without using vagrant halt, which is the main use case.

Here is the dmesg output of the next vagrant up:

[    5.408971] vboxvideo: Unknown symbol drm_open (err 0)
[    5.408973] vboxvideo: Unknown symbol drm_poll (err 0)
[    5.408974] vboxvideo: Unknown symbol drm_pci_init (err 0)
[    5.408977] vboxvideo: Unknown symbol drm_ioctl (err 0)
[    5.408978] vboxvideo: Unknown symbol drm_vblank_init (err 0)
[    5.408980] vboxvideo: Unknown symbol drm_mmap (err 0)
[    5.408981] vboxvideo: Unknown symbol drm_pci_exit (err 0)
[    5.408983] vboxvideo: Unknown symbol drm_release (err 0)
[    5.694825] init: udev-fallback-graphics main process (889) terminated with status 1

Full Output: http://pastebin.com/mgECtZBH

As you can see the last two lines are missing:

vboxsf: Successfully loaded version 4.3.10_Ubuntu (interface 0x00010004)
init: plymouth-upstart-bridge main process (180) killed by TERM signal

When i open the box via the VirtualBox GUI, i get a login prompt at the first attempts. After killing the virtualbox process (or rebooting the host), it hangs at the following lines without giving me the login prompt. Here are the last lines of output:

* Starting early crypto disks...
  ...done.
* Starting AppArmor profiles
Skipping profile inn /etc/apparmor.d/disable: usr.sbin.rsyslogd
  ...done.
* modprobe vboxvideo failed. Please use 'dmesg' to find out why
  ...fail!
* Setting up X socket directories...
  ...done.
ssty: standard input: Input/output error
* Not starting NFS kernel daemon: no exports

And then it just waits infinitely.

First Approach

I thought that maybe supervisor is causing this and i removed the initial start of supervisord. After killing the process (or rebooting the host) and got no problems or errors at all and everything booted and mounted just fine. Without starting supervisord everything works as expected.

The supervisord process starts several services:

  1. Redis
  2. Mongodb
  3. Nginx
  4. Elasticsearch
  5. PostgreSQL
  6. Tornado
  7. Celery

These processes use the aforementioned shared folder on my host.

I didn't try yet to isolate one of the services yet.

Any insights or ideas how to solve this riddle?


Solution

  • I tried to deactivate each of the used services and narrowed it down to nginx. Specifically my custom nginx.conf.

    The key was to turn on the daemon parameter. Simply replace daemon off; with daemon on;.