Logs & journalctl

When a website stops responding at 2 a.m. or someone keeps trying to log in to your server, the answer is almost always written down somewhere. Linux records what services, the kernel and users do in logs. Knowing where they are and how to filter them quickly is one of the most valuable skills an administrator has.

Ubuntu keeps logs in two places: the systemd journal, read with journalctl, and classic text files under /var/log. This lesson covers both, plus kernel messages and log rotation.

The systemd journal

The journal is a structured log database run by the systemd-journald service. Every service started by systemd sends its output there automatically, along with kernel messages and the system log. Each entry records the time, the host, the program, its PID, a priority and more, which makes powerful filtering possible.

journalctl -n 3
journalctl -u ssh --since today

Oct 30 11:20:03 shop-server systemd[1]: Finished backup-notes.service - Back up Priya's notes folder.
Oct 30 11:24:51 shop-server sudo[3502]:    priya : TTY=pts/0 ; PWD=/home/priya ; USER=root ; COMMAND=/usr/bin/apt update
Oct 30 11:24:51 shop-server sudo[3502]: pam_unix(sudo:session): session opened for user root(uid=0) by priya(uid=1000)
Oct 30 06:12:01 shop-server sshd[1201]: Failed password for root from 192.0.2.55 port 51122 ssh2
Oct 30 06:13:40 shop-server sshd[1230]: Failed password for invalid user admin from 198.51.100.89 port 40210 ssh2
Oct 30 07:02:15 shop-server sshd[1302]: Accepted publickey for priya from 203.0.113.7 port 55012 ssh2: ED25519 SHA256:q3Xr9mJb0eL2vKcR8sTnYpWz4uHaD1fG6iOkB7xVyZw

-n 3 shows the last three entries. -u ssh shows only the SSH service (its unit is ssh.service on Ubuntu). Normal users can read their own entries; members of the adm or sudo groups can read the whole journal, otherwise use sudo journalctl. Output opens in a pager; press q to leave, or add --no-pager.

Filtering the journal

The real power is in combining filters:

journalctl -p err -b
journalctl -u nginx --since "2026-10-30 09:00" --until "2026-10-30 10:00"
journalctl -u sweets-api -f

Oct 30 08:01:20 shop-server sweets-api[912]: sqlite3.OperationalError: unable to open database file
Oct 30 08:01:20 shop-server systemd[1]: sweets-api.service: Main process exited, code=exited, status=1/FAILURE
Oct 30 08:01:20 shop-server systemd[1]: sweets-api.service: Failed with result 'exit-code'.

The first command, "errors since boot", is the best starting point when something is wrong after a restart. Here it shows that Priya's API could not open its database file, so the fix is a path or permission problem, not a code bug. The last command follows one service live while you test it in a browser.

Journal storage and size

On Ubuntu the journal is persistent: it is stored in /var/log/journal/ and survives reboots, which is why -b -1 works. journald limits its own size automatically (by default to a share of the disk), but you can check and trim it:

journalctl --disk-usage
sudo journalctl --vacuum-time=2weeks

Archived and active journals take up 96.0M in the file system.

--vacuum-time deletes archived entries older than two weeks; --vacuum-size=200M keeps the total under a size instead.

Classic log files in /var/log

Many programs still write plain text files, and on Ubuntu the rsyslog service also copies journal messages into traditional files. These are easy to search with the tools you already know.

sudo grep "Failed password" /var/log/auth.log | grep -oE 'from [0-9.]+' | sort | uniq -c | sort -rn
sudo tail -n 20 /var/log/nginx/error.log
zgrep "Accepted" /var/log/auth.log.2.gz

38 from 192.0.2.55
12 from 198.51.100.89

The first pipeline counts failed SSH logins per IP address: exactly the information a tool like fail2ban uses, as you will see in Basic Server Hardening. Files ending in .1, .2.gz and so on are older, rotated copies.

Kernel messages: dmesg

The kernel reports hardware events, driver problems, disk errors and out-of-memory kills. sudo dmesg prints its message buffer; -T shows readable times and -w follows live. journalctl -k gives the same messages with the journal's filters.

sudo dmesg -T | grep -i -E "error|oom" | tail -n 3

[Fri Oct 30 09:14:07 2026] Out of memory: Killed process 2875 (python) total-vm:2209344kB, anon-rss:1630216kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:3592kB oom_score_adj:0

An "Out of memory: Killed process" line explains a service that "randomly" stopped: the server ran out of RAM and the kernel's OOM killer ended the biggest process.

Keeping logs small: logrotate

Text logs grow forever unless something trims them. logrotate runs daily (from a systemd timer) and, for each log, renames the current file, compresses old copies and deletes the oldest. Packages install their rules in /etc/logrotate.d/. A rule for Priya's own app log might be /etc/logrotate.d/sweets-api:

/var/log/sweets-api/*.log {
    weekly
    rotate 8
    compress
    delaycompress
    missingok
    notifempty
}

This keeps eight weeks of logs, gzips all but the newest old copy, and does not complain if a file is missing or rotate it when empty. Test a rule without changing anything with sudo logrotate --debug /etc/logrotate.d/sweets-api.

Common mistakes

• Reading only the last line. The real cause is often a few lines earlier. Look at 30 to 50 lines of context.
• Forgetting which boot. After a crash and reboot, the evidence is in journalctl -b -1, not -b.
• "Permission denied" on logs. Use sudo, or add your user to the adm group.
• Deleting log files to free space. A running program may keep writing to the deleted file, so the space is not freed. Use logrotate or journalctl --vacuum-* instead.

Key points

• systemd services log to the journal; read it with journalctl.
• Filter with -u unit, -p err, -b, --since/--until, -n, -g, and follow with -f.
• Classic text logs live in /var/log: syslog, auth.log, kern.log, apt/history.log, nginx/.
• dmesg -T and journalctl -k show kernel messages such as hardware errors and OOM kills.
• logrotate rotates and compresses text logs; journalctl --vacuum-time trims the journal.

journalctl -b -1 -p err --no-pager
journalctl -u ssh --since "2 hours ago"
grep "$(date +%F)" -A 4 /var/log/apt/history.log
journalctl -k -b -1 -g "Out of memory"

Oct 30 02:41:55 shop-server kernel: Out of memory: Killed process 2875 (python) total-vm:2209344kB, anon-rss:1630216kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:3592kB oom_score_adj:0