Search code examples
amazon-web-servicesamazon-ec2bitnami

AWS Bitnami EC2 and EC2 fails to start with Service Unavailable

I currently have an AWS Bitnami EC2 Wordpress instance with 10gb of general purpose storage attached. Everything was running fine until recently, when I received the following error on the page:

Service Unavailable
The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

Additionally, a 503 Service Unavailable error was encountered while trying to use an ErrorDocument to handle the request.

I went ahead and checked the system log and got this:

[  151.389085] bitnami[1159]: Please see below for details
[  151.392234] bitnami[1159]: http://docs.fedoraproject.org/en-US/Fedora/13/html/SELinux_FAQ/index.html#id3037154
[  151.396103] bitnami[1159]: ## 2021-02-08 10:21:07+00:00 ## INFO ## Running /opt/bitnami/var/init/post-start/040_update_welcome_file...
[  151.400843] bitnami[1159]: ## 2021-02-08 10:21:07+00:00 ## INFO ## Running /opt/bitnami/var/init/post-start/050_bitnami_credentials_file...
[  151.416962] bitnami[1159]: ## 2021-02-08 10:21:07+00:00 ## INFO ## Running /opt/bitnami/var/init/post-start/060_start_gonit...
[  151.426036] bitnami[1159]: Starting gonit daemon
[  151.435518] bitnami[1159]: write /var/lib/gonit/state: no space left on device
[  151.438242] bitnami[1159]: write /var/lib/gonit/state: no space left on device
[  151.440614] bitnami[1159]: write /var/lib/gonit/state: no space left on device
[  151.444201] bitnami[1159]: ## 2021-02-08 10:21:07+00:00 ## INFO ## Running /opt/bitnami/var/init/post-start/070_clean_metadata...
[[0;1;31mFAILED[0m] Failed to start LSB: bitnami init script.
See 'systemctl status bitnami.service' for details.
[[0;32m  OK  [0m] Reached target Multi-User System.
         Starting Execute cloud user/final scripts...
[[0;32m  OK  [0m] Reached target Graphical Interface.
         Starting Update UTMP about System Runlevel Changes...
[[0;32m  OK  [0m] Started Update UTMP about System Runlevel Changes.
[  151.981648] cloud-init[2275]: OSError: [Errno 28] No space left on device
[  151.985418] cloud-init[2275]: During handling of the above exception, another exception occurred:
[  151.988142] cloud-init[2275]: Traceback (most recent call last):
[  151.996571] cloud-init[2275]:   File "/usr/bin/cloud-init", line 9, in <module>
[  151.999040] cloud-init[2275]:     load_entry_point('cloud-init==19.2', 'console_scripts', 'cloud-init')()
[  152.001985] cloud-init[2275]:   File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 893, in main
[  152.005271] cloud-init[2275]:     get_uptime=True, func=functor, args=(name, args))
[  152.008071] cloud-init[2275]:   File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2525, in log_time
[  152.011326] cloud-init[2275]:     ret = func(*args, **kwargs)
[  152.020732] cloud-init[2275]:   File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 648, in status_wrapper
[  152.023872] cloud-init[2275]:     atomic_helper.write_json(status_path, status)
[  152.026501] cloud-init[2275]:   File "/usr/lib/python3/dist-packages/cloudinit/atomic_helper.py", line 46, in write_json
[  152.032550] cloud-init[2275]:     omode="w", mode=mode)
[  152.034569] cloud-init[2275]:   File "/usr/lib/python3/dist-packages/cloudinit/atomic_helper.py", line 39, in write_file
[  152.038544] cloud-init[2275]:     raise e

It's a lot more than this but I keep getting a "No space left on device" error. There's a lot more to the log if anyone willing to assist is interested.

I should also state that due to a problem I no longer have access to the ssh key, though I feel strongly that it wouldnt work since I created an image of this instance and volume that returned a "Connection timed out" with every ssh attempt.

I'm at my wits end here. Any help would be appreciated.

Solution

  • Bitnami Engineer here,

    As you mentioned, the problem is that there is not space left on the machine so you need to remove the data you have or increase the disk size.

    First of all, let's get more information about the folders with more data inside the Bitnami installation

    cd /opt/bitnami
du -h -d 1

    https://docs.bitnami.com/aws/faq/troubleshooting/troubleshoot-server-performance/

    If MySQL is the folder with more space, you can try to disable the binary logging in the database to save some space

    • Stop the database
    sudo /opt/bitnami/ctlscript.sh stop mysql
    • Disable the binary logging in the MySQL's configuration
    [mysqld]
disable_log_bin
...
    • Restart the server and purge the old files from the instance

    More info here: https://docs.bitnami.com/aws/apps/wordpress/troubleshooting/disable-binary-logging-mysql/

    However, if you can't remove any data in the instance, you will need to increase the disk size in the instance. You will need to access the AWS EC2 console, identify the volume the instance is using and increase the size of the volume

    enter image description here enter image description here

    You have more info here: https://docs.bitnami.com/aws/how-to/modify-ebs-volume/