Search code examples
amazon-web-servicesamazon-ec2ubuntu-16.04amazon-ami

Running out of disk space in Amazon EC2, can't find what I am using my storage for


I am Running an AWS ami using a T2.large instance using the US East. I was trying to upload some data and I ran in the terminal:

df -h

and I got this result:

Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G     0  3.9G   0% /dev
tmpfs           799M  8.6M  790M   2% /run
/dev/xvda1      9.7G  9.6G   32M 100% /
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs           799M     0  799M   0% /run/user/1000

I know I have not uploaded 9.7 GB of data to the instance, but I don't know what /dev/xvda1 is or how to access it.

I also assume that all the tmpfs are temporal files, how can I erase those?

Answering some of the questions in the coments, I runned

sudo du -sh /*

And I got:

16M /bin
124M    /boot
0   /dev
6.5M    /etc
2.7G    /home
0   /initrd.img
0   /initrd.img.old
4.0K    /jupyterhub_cookie_secret
16K /jupyterhub.sqlite
268M    /lib
4.0K    /lib64
16K /lost+found
4.0K    /media
4.0K    /mnt 
562M    /opt
du: cannot access '/proc/15616/task/15616/fd/4': No such file or directory
du: cannot access '/proc/15616/task/15616/fdinfo/4': No such file or directory
du: cannot access '/proc/15616/fd/4': No such file or directory
du: cannot access '/proc/15616/fdinfo/4': No such file or directory
0   /proc
28K /root
8.6M    /run
14M /sbin
8.0K    /snap 
8.0K    /srv
0   /sys
64K /tmp
4.7G    /usr
1.5G    /var
0   /vmlinuz
0   /vmlinuz.old

Solution

  • /dev/xvda1 is your root volume. The AMI you listed has a default root volume size of 20GB as you can see here:

    Describe the image and get it's block device mappings:

    aws ec2 describe-images --image-ids ami-3b0c205e --region us-east-2 | jq .Images[].BlockDeviceMappings[]
    

    Look at the volume size

    {
      "DeviceName": "/dev/sda1",
      "Ebs": {
        "Encrypted": false,
        "DeleteOnTermination": true,
        "VolumeType": "gp2",
        "VolumeSize": 20,
        "SnapshotId": "snap-03341b1ff8ee47eaa"
      }
    }
    {
      "DeviceName": "/dev/sdb",
      "VirtualName": "ephemeral0"
    }
    {
      "DeviceName": "/dev/sdc",
      "VirtualName": "ephemeral1"
    }
    

    When launched with the correct volume size of 20GB there is plenty of free space (10GB)

    root@ip-10-100-0-64:~# df -h
    Filesystem      Size  Used Avail Use% Mounted on
    udev            488M     0  488M   0% /dev
    tmpfs           100M  3.1M   97M   4% /run
    /dev/xvda1       20G  9.3G   11G  49% /
    tmpfs           496M     0  496M   0% /dev/shm
    tmpfs           5.0M     0  5.0M   0% /run/lock
    tmpfs           496M     0  496M   0% /sys/fs/cgroup
    tmpfs           100M     0  100M   0% /run/user/1000
    

    It appears the issue here is the instance was launched with 10GB (somehow, I didn't think this was possible) of storage instead of the default 20GB.