Disks & Storage

"No space left on device" is one of the most common reasons a server stops working: the database cannot write, logs cannot grow, uploads fail. In this lesson you will learn to check disk space, track down what is filling it, see which disks and partitions a machine has, and attach a new disk so it is available at every boot.

The examples come from Priya's cloud server, shop-server, which has a 25 GB system disk and a 50 GB extra volume she has just added for website data.

df: how much space is free?

df (disk free) shows every mounted filesystem with its size, use and free space. -h makes the numbers readable, and -x tmpfs hides the small in-memory filesystems that would otherwise clutter the list:

df -h -x tmpfs

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        24G  5.9G   18G  25% /
/dev/sda16      881M  112M  708M  14% /boot
/dev/sda15      105M  6.2M   99M   6% /boot/efi

The Mounted on column tells you which part of the directory tree each filesystem holds. Here the root filesystem / is a quarter full. Start worrying when Use% goes above about 85%, and act fast near 100%. df -h /var shows only the filesystem that contains /var, which is a quick way to answer "where would this file go?".

A filesystem can also run out of inodes, the records that describe each file, even when gigabytes are free. This happens with millions of tiny files, such as old PHP session files. df -i shows inode use.

du: what is using the space?

du (disk usage) adds up the size of files and folders. -s gives one total per argument and -h makes it readable. To find the biggest folders, list one level deep and sort by size:

du -sh /var/log
sudo du -h --max-depth=1 /var | sort -h

412M	/var/log
4.0K	/var/local
4.0K	/var/opt
16K	/var/mail
48M	/var/backups
96M	/var/www
412M	/var/log
1.1G	/var/cache
1.9G	/var/lib
3.6G	/var

sort -h understands sizes like 96M and 1.9G. Repeat the command one level deeper in the biggest folder (/var/lib, then perhaps /var/lib/docker) until you find the cause. Common culprits are old logs, the apt cache (cleared with sudo apt clean), Docker images, old backups and journal files.

lsblk: which disks are there?

Disks appear as files in /dev: sda, sdb for SATA, SCSI and many cloud disks, nvme0n1 for NVMe drives and vda on some virtual machines. Partitions add a number: sda1, nvme0n1p1. lsblk shows them as a tree:

lsblk

NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda       8:0    0   25G  0 disk
├─sda1    8:1    0   24G  0 part /
├─sda14   8:14   0    4M  0 part
├─sda15   8:15   0  106M  0 part /boot/efi
└─sda16 259:0    0  913M  0 part /boot
sdb       8:16   0   50G  0 disk

The new 50 GB volume sdb has no partitions and no mount point: it is blank. lsblk -f adds the filesystem type, label and UUID of each device.

Formatting and mounting a new disk

Before a blank disk can store files it needs a filesystem, the structure that organises files on it. ext4 is the standard choice on Ubuntu. Creating a filesystem is called formatting, and it erases everything on the device.

sudo mkfs.ext4 -L shopdata /dev/sdb
sudo mkdir -p /srv/data
sudo mount /dev/sdb /srv/data
df -h /srv/data

Discarding device blocks: done
Creating filesystem with 13107200 4k blocks and 3276800 inodes
...
Writing superblocks and filesystem accounting information: done
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb         49G   24K   47G   1% /srv/data

Mounting attaches a filesystem to a folder, the mount point. From now on, anything saved in /srv/data is stored on the new disk. A small part of the space is reserved for the filesystem itself, which is why 50 GB shows as 49G. To detach it again, run sudo umount /srv/data (note the spelling: no "n"). Unmounting fails with "target is busy" while a program or a shell is using a file or folder on it.

Mounting at boot: /etc/fstab

A mount command lasts only until the next reboot. To make it permanent, add a line to /etc/fstab (filesystem table). Refer to the disk by its UUID, a unique ID stored in the filesystem, because device names like sdb can change when disks are added or removed.

sudo blkid /dev/sdb

/dev/sdb: LABEL="shopdata" UUID="5b1e7c1e-3f0a-4c8e-9d2b-6a7f4e1c0d93" BLOCK_SIZE="4096" TYPE="ext4"

Then add this line to /etc/fstab with sudo nano /etc/fstab:

UUID=5b1e7c1e-3f0a-4c8e-9d2b-6a7f4e1c0d93  /srv/data  ext4  defaults,nofail  0  2

The six fields are: device, mount point, filesystem type, options, a legacy dump flag (0), and the order for the boot-time filesystem check (2 for data disks, 1 only for /). The option nofail lets the server boot even if this disk is missing, which matters on cloud machines.

Test the file before rebooting:

sudo umount /srv/data
sudo findmnt --verify
sudo systemctl daemon-reload
sudo mount -a
findmnt /srv/data

Success, no errors or warnings detected
TARGET    SOURCE   FSTYPE OPTIONS
/srv/data /dev/sdb ext4   rw,relatime

findmnt --verify checks fstab for mistakes, daemon-reload lets systemd pick up the change (systemd turns fstab lines into mount units), and mount -a mounts everything listed that is not mounted yet. If mount -a succeeds, the next boot will too. A broken fstab line can stop a server from booting, so never skip this test.

Memory and swap

free -h reports RAM rather than disk, but it belongs here because swap is disk space used as overflow memory:

free -h

               total        used        free      shared  buff/cache   available
Mem:           3.8Gi       702Mi       2.1Gi       1.0Mi       1.2Gi       3.1Gi
Swap:             0B          0B          0B

Look at available, not free: Linux uses spare RAM as a disk cache (buff/cache) and gives it back when programs need it. Many cloud images have no swap; a small swap file can prevent out-of-memory kills on a 1 GB server.

Common mistakes

• Formatting the wrong device. Always confirm with lsblk first.
• Using /dev/sdb in fstab. Device names can change; use UUID=.
• Rebooting without testing fstab. Run findmnt --verify and mount -a first.
• Deleting big files that are still open. The space is freed only when the program closes them; restart the service or truncate the file instead.
• Reading "free" memory as "available". Cached memory is not wasted.

Key points

• df -h shows free space per filesystem; df -i shows inode use.
• du -sh sizes a folder; du -h --max-depth=1 | sort -h finds the biggest ones.
• lsblk lists disks and partitions; blkid shows UUIDs and filesystem types.
• mkfs.ext4 formats (and erases) a device; mount and umount attach and detach it.
• Make mounts permanent in /etc/fstab with UUID= and nofail, then test with findmnt --verify and mount -a.

df -h /
sudo du -h --max-depth=1 -x / 2> /dev/null | sort -h | tail -n 4
sudo du -h --max-depth=1 /var | sort -h | tail -n 3
sudo apt clean
sudo journalctl --vacuum-size=200M

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        24G   23G  1.1G  96% /
2.1G	/usr
2.4G	/home
17G	/var
23G	/
6.2G	/var/backups
9.8G	/var/log
17G	/var