Project: Production Server with Monitoring & Backups

In Project: Deploy a Website on a VPS you published Priya's static site with HTTPS. Now Sharma Sweets takes orders online, through a small Python API that stores orders in an SQLite database. A static site that goes down is embarrassing; an order system that goes down, or loses a day of orders, costs real money. Production work is less about installing software and more about everything around it: least privilege, limits, monitoring, alerts, backups and a tested way back.

In this project you will take the server shop-server from the first project and make it production-ready. Each step uses a lesson from this course, so follow the links when something is new. Commands run on the server as Priya unless a step says otherwise.

The target

By the end, the server will look like this:

Step 1: a dedicated user and a safe place for secrets

The API must never run as Priya or as root. Create a system user that cannot log in, install the app under /opt, and keep secrets in a file only that user can read:

sudo useradd --system --home-dir /opt/sweets-api --shell /usr/sbin/nologin sweets
sudo mkdir -p /etc/sweets-api
printf 'APP_ENV=production\nSECRET_KEY=%s\n' "$(openssl rand -hex 32)" | sudo tee /etc/sweets-api/env > /dev/null
sudo chown root:sweets /etc/sweets-api/env
sudo chmod 640 /etc/sweets-api/env
ls -l /etc/sweets-api/env

-rw-r----- 1 root sweets 95 Nov  2 11:02 /etc/sweets-api/env

openssl rand -hex 32 generates a random secret that never appears on screen or in shell history, because tee writes it straight to the file. Root owns the file and the sweets group can only read it, so even a bug in the app cannot change its own configuration. Copy the application code to /opt/sweets-api (for example with rsync) and create its Python virtual environment there, owned by root and readable by sweets.

Step 2: a sandboxed, limited systemd service

The first project used a simple unit. Production units also say what the service is not allowed to do. Create /etc/systemd/system/sweets-api.service:

[Unit]
Description=Sharma Sweets order API
After=network-online.target
Wants=network-online.target
OnFailure=shop-alert@%n.service

[Service]
User=sweets
Group=sweets
WorkingDirectory=/opt/sweets-api
EnvironmentFile=/etc/sweets-api/env
ExecStart=/opt/sweets-api/.venv/bin/gunicorn --bind 127.0.0.1:8000 --workers 2 app:app
Restart=on-failure
RestartSec=5
StateDirectory=sweets-api

NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX

MemoryHigh=400M
MemoryMax=512M
TasksMax=64

[Install]
WantedBy=multi-user.target

gunicorn is a production web server for Python apps; binding to 127.0.0.1 means only programs on this machine, such as nginx, can reach the API. StateDirectory=sweets-api creates /var/lib/sweets-api, owned by sweets, where the app keeps orders.db. The second block is the sandbox: ProtectSystem=strict makes the whole filesystem read-only for this service except its state directory, ProtectHome hides users' home folders, PrivateTmp gives it its own empty /tmp, NoNewPrivileges stops it from gaining privileges through set-UID programs, and RestrictAddressFamilies allows only normal network and local sockets. The third block applies the limits from Resource Control: cgroups, ulimit & OOM. OnFailure= starts an alert unit, built in Step 5, if the service ends up failed.

sudo systemctl daemon-reload
sudo systemctl enable --now sweets-api
curl -s http://127.0.0.1:8000/api/health
sudo ss -tlnp | grep 8000

{"status":"ok"}
LISTEN 0      2048       127.0.0.1:8000      0.0.0.0:*    users:(("gunicorn",pid=2214,fd=5),("gunicorn",pid=2213,fd=5),("gunicorn",pid=2210,fd=5))

The API answers, and ss confirms it listens only on loopback. Now ask systemd to grade the sandbox: systemd-analyze security sweets-api lists every protection setting with a tick or a cross and ends with an overall exposure score from 0 (very protected) to 10 (unprotected). A plain unit scores near 9; this one should land well below 5. If the app breaks after adding a protection, journalctl -u sweets-api usually shows a "Read-only file system" or "Permission denied" error that names the path to allow.

Step 3: nginx as a reverse proxy with limits and headers

A reverse proxy receives visitors' requests and forwards some of them to a backend application. nginx stays the only public entry point: it handles HTTPS, serves static files fast and protects the app. First define a rate limit zone in the http context, in a new file /etc/nginx/conf.d/ratelimit.conf:

limit_req_zone $binary_remote_addr zone=orders:10m rate=10r/m;
limit_req_status 429;
server_tokens off;

This remembers clients by IP address in 10 MB of shared memory and allows 10 requests per minute each; extra requests get status 429 (Too Many Requests). Then add these lines inside the HTTPS server block of /etc/nginx/sites-available/shop, which certbot created in the first project:

client_max_body_size 2m;

add_header Strict-Transport-Security "max-age=31536000" always;
add_header X-Content-Type-Options "nosniff" always;
add_header X-Frame-Options "DENY" always;
add_header Referrer-Policy "strict-origin-when-cross-origin" always;

location = /api/order {
    limit_req zone=orders burst=5 nodelay;
    proxy_pass http://127.0.0.1:8000;
    include /etc/nginx/proxy_params;
}

location /api/ {
    proxy_pass http://127.0.0.1:8000;
    include /etc/nginx/proxy_params;
    proxy_read_timeout 30s;
}

proxy_pass forwards the request unchanged, and Ubuntu's proxy_params file sets the headers that tell the app the real host, visitor IP and scheme (X-Real-IP, X-Forwarded-For, X-Forwarded-Proto). The order endpoint alone gets the rate limit, with a burst of 5 so a customer who double-clicks is not punished. The security headers tell browsers to use HTTPS only, not to guess file types, and not to show the shop inside other sites' frames. client_max_body_size rejects huge uploads early. Test, reload and try to flood the order endpoint:

sudo nginx -t && sudo systemctl reload nginx
for i in $(seq 1 8); do curl -s -o /dev/null -w '%{http_code} ' -X POST -H 'Content-Type: application/json' -d '{"item":"kaju katli","qty":1}' https://shop.example.in/api/order; done; echo
curl -sI https://shop.example.in/ | grep -i -E 'strict-transport|x-frame'

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
201 201 201 201 201 201 429 429
strict-transport-security: max-age=31536000
x-frame-options: DENY

One request plus a burst of five got through; the rest were refused before reaching the app. (Delete the test orders afterwards.) On RHEL-family servers, SELinux blocks nginx from connecting to the app until you run sudo setsebool -P httpd_can_network_connect on, as explained in SELinux & AppArmor.

Step 4: confirm the network surface

Check from the outside in. The firewall from the first project should still allow only SSH and the web ports, and nothing private should listen on a public address:

sudo ufw status | grep ALLOW
sudo ss -tlnp | awk 'NR > 1 {print $4}' | sort -u
sudo sshd -T | grep -E '^(permitrootlogin|passwordauthentication) '

22/tcp (OpenSSH)           ALLOW       Anywhere
80,443/tcp (Nginx Full)    ALLOW       Anywhere
22/tcp (OpenSSH (v6))      ALLOW       Anywhere (v6)
80,443/tcp (Nginx Full (v6)) ALLOW       Anywhere (v6)
0.0.0.0:22
0.0.0.0:443
0.0.0.0:80
127.0.0.1:8000
127.0.0.53%lo:53
127.0.0.54:53
[::]:22
[::]:443
[::]:80
permitrootlogin no
passwordauthentication no

Only 22, 80 and 443 are public; the API and the local DNS resolver listen on loopback addresses. Do this check after every new installation, because some packages open ports you did not expect. Use Firewalls: nftables, firewalld & iptables if you manage this server with nftables or firewalld instead of ufw.

Step 5: health checks and alerts

Monitoring answers one question every few minutes: is the shop healthy, and if not, why? Save this script as /usr/local/bin/shop-health.sh and make it executable:

#!/usr/bin/env bash
# shop-health.sh: exit non-zero and explain why if the shop is unhealthy
set -Euo pipefail

URL=${URL:-https://shop.example.in/api/health}
LOG=${LOG:-/var/log/nginx/shop.access.log}
DISK_LIMIT=85
MEM_MIN_MB=200
problems=()

curl -fsS -m 5 -o /dev/null "$URL" || problems+=("API health check failed: $URL")

while read -r use mount; do
    (( ${use%\%} > DISK_LIMIT )) && problems+=("disk $mount is ${use} full")
done < <(df --output=pcent,target -x tmpfs -x devtmpfs -x overlay | tail -n +2)

avail=$(awk '/^MemAvailable:/ {print int($2 / 1024)}' /proc/meminfo)
(( avail < MEM_MIN_MB )) && problems+=("only ${avail} MB memory available")

failed=$(systemctl --failed --plain --no-legend | awk '{print $1}' | paste -sd' ')
[[ -n $failed ]] && problems+=("failed units: $failed")

errors=$(tail -n 500 "$LOG" | awk '$9 >= 500 {n++} END {print n + 0}')
(( errors > 10 )) && problems+=("$errors server errors in the last 500 requests")

if (( ${#problems[@]} )); then
    printf 'UNHEALTHY: %s\n' "${problems[@]}"
    exit 1
fi
echo "healthy"

It checks the API from outside through nginx and HTTPS, every real disk, available memory, failed systemd units and the share of 5xx errors among recent requests, using the awk idea from awk in Depth: Reports from Logs. It deliberately leaves out set -e: one failed check must not stop the others, so every problem is collected into an array and reported together. Now run it from a timer, and connect failures to an alert. Create three units in /etc/systemd/system/:

# shop-health.service
[Unit]
Description=Sharma Sweets health check
OnFailure=shop-alert@%n.service

[Service]
Type=oneshot
DynamicUser=yes
SupplementaryGroups=adm
ExecStart=/usr/local/bin/shop-health.sh

# shop-health.timer
[Unit]
Description=Run the health check every 5 minutes

[Timer]
OnBootSec=2min
OnUnitActiveSec=5min

[Install]
WantedBy=timers.target

# shop-alert@.service
[Unit]
Description=Send an alert about %i

[Service]
Type=oneshot
EnvironmentFile=/etc/shop-alert.env
ExecStart=/usr/local/bin/shop-alert.sh %i

DynamicUser=yes runs the check as a temporary user created just for that run, and the adm group lets it read nginx logs. When the check exits with an error, OnFailure= starts shop-alert@shop-health.service.service: the part after @ (%i) carries the name of the unit that failed. The same alert template is already attached to sweets-api. The alert script sends the last journal lines of the failed unit to a webhook, such as a team chat channel or a self-hosted notification service:

#!/usr/bin/env bash
# shop-alert.sh UNIT: post the unit's recent log to the alert webhook
set -Eeuo pipefail
unit=$1
msg="$(hostname): $unit failed at $(date '+%F %T')
$(journalctl -u "$unit" -n 15 --no-pager -o cat)"
curl -fsS -m 10 --data-binary "$msg" "$ALERT_URL"

Put ALERT_URL=... in /etc/shop-alert.env with mode 600, never in the script. Enable the timer and test the whole chain by stopping the API on purpose:

sudo systemctl daemon-reload
sudo systemctl enable --now shop-health.timer
sudo systemctl stop sweets-api
sudo systemctl start shop-health.service
journalctl -u shop-health -o cat | grep -E 'UNHEALTHY|OnFailure'
sudo systemctl start sweets-api

UNHEALTHY: API health check failed: https://shop.example.in/api/health
shop-health.service: Triggering OnFailure= dependencies.

nginx answered 502 because the backend was gone (curl's own error line is also in the journal), the check failed, and systemd triggered the alert unit; your chat channel should show the message within seconds. Test alerts regularly: an alert path that silently broke is worse than none, because you trust it. Finally, enable sysstat history as in Performance Troubleshooting, so that after an alert you can see what led up to it. For dashboards across many servers, the next step is Prometheus with the prometheus-node-exporter package, allowed through the firewall only from the monitoring server.

Step 6: keep logs long enough

The CERT-In directions mentioned in Basic Server Hardening require keeping system logs for 180 days. Make room for that. Give the journal enough space with a drop-in, /etc/systemd/journald.conf.d/retention.conf:

[Journal]
SystemMaxUse=2G

Then change rotate 14 to rotate 180 in /etc/logrotate.d/nginx so nginx keeps 180 daily files, and apply with sudo systemctl restart systemd-journald. Check after a few weeks with journalctl --disk-usage and ls /var/log/nginx | wc -l. Logs on the server can be deleted by an attacker who gets root, so for real compliance also ship copies to the backup server, for example with rsyslog forwarding or the nightly pull in the next step.

Step 7: backups and a restore drill

Following Backups & Restores, the plan is: database every hour (RPO one hour), configuration every night, copies pulled to a separate backup server, and a monthly restore drill. Save /usr/local/bin/shop-backup.sh:

#!/usr/bin/env bash
# shop-backup.sh: hourly database copy, nightly config archive, local retention
set -Eeuo pipefail
trap 'echo "backup failed at line $LINENO: $BASH_COMMAND" >&2' ERR

dest=/srv/backups
stamp=$(date +%F-%H%M)
mkdir -p "$dest/db" "$dest/config"

sqlite3 /var/lib/sweets-api/orders.db ".backup '$dest/db/orders-$stamp.db'"
[[ $(sqlite3 "$dest/db/orders-$stamp.db" 'PRAGMA integrity_check;') == ok ]]
gzip "$dest/db/orders-$stamp.db"

if [[ $(date +%H) == 02 ]]; then
    tar -czf "$dest/config/config-$(date +%F).tar.gz" --exclude=/opt/sweets-api/.venv \
        /etc/nginx /etc/sweets-api /etc/systemd/system /etc/letsencrypt /opt/sweets-api
    dpkg --get-selections > "$dest/config/packages-$(date +%F).txt"
fi

find "$dest/db" -name 'orders-*.db.gz' -mmin +$((48 * 60)) -delete
find "$dest/config" -type f -mtime +14 -delete
echo "backup ok: orders-$stamp.db.gz"

SQLite's .backup command copies the database safely while the API keeps writing, unlike cp. PRAGMA integrity_check proves the copy is sound before it is kept; if it does not print ok, the [[ ]] test fails, set -e stops the script and the ERR trap names the line. At 2 a.m. the script also archives configuration, certificates and the installed-package list, everything needed to rebuild the server. Run it from shop-backup.service and a timer with OnCalendar=hourly, with the same OnFailure=shop-alert@%n.service line. The folder contains secrets, so give it to the backup user's group only: sudo chgrp backup /srv/backups and sudo chmod 2750 /srv/backups. The backup server then pulls /srv/backups every hour as that user, with the restricted read-only rsync key from the backups lesson.

Once a month, prove that a restore works, on the backup server or a scratch VM, never over production:

latest=$(ls -1t /srv/backups/db/orders-*.db.gz | head -n 1)
zcat "$latest" > /var/tmp/restore-test.db
sqlite3 /var/tmp/restore-test.db 'PRAGMA integrity_check; SELECT COUNT(*), MAX(created_at) FROM orders;'

ok
1342|2026-11-02 10:58:41

The integrity check passes, the order count is plausible and the newest order is less than an hour old, which matches the promised RPO. Write the steps and the time they took in /root/RUNBOOK.txt on both servers, so whoever is on call at 2 a.m. does not need to remember this page.

Final checklist

Ideas to extend the project

• Add a cooldown to shop-alert.sh (a timestamp file) so a long outage sends one alert every 30 minutes, not every 5.
• Move the database to MariaDB or PostgreSQL and switch the backup to mysqldump --single-transaction or pg_dump.
• Put the API in a container and compare the systemd sandbox with container isolation.
• Collect metrics with Prometheus and node exporter and draw dashboards.

Key points

• Run each application as its own system user, with secrets in a root-owned file only that user can read.
• systemd sandboxing (ProtectSystem=strict, NoNewPrivileges, PrivateTmp) and limits (MemoryMax, TasksMax) shrink the damage a bug can do; check with systemd-analyze security.
• nginx is the only public entry point: reverse proxy to 127.0.0.1, security headers, rate limits on sensitive endpoints.
• A timer-driven health check plus OnFailure= alert units turns silent failures into messages; test the alert path.
• Hourly verified database copies, nightly config archives, off-server pulls and monthly restore drills make the RPO and RTO real.

cert_ok() {
    echo | openssl s_client -connect shop.example.in:443 -servername shop.example.in 2> /dev/null |
        openssl x509 -noout -checkend $((14 * 24 * 3600)) > /dev/null
}
cert_ok || problems+=("TLS certificate for shop.example.in expires within 14 days")

sudo systemctl start shop-health.service
journalctl -u shop-health -n 1 -o cat

healthy