Logs, Metrics & Prometheus

A cluster that runs well today will misbehave one day: a Pod restarts every hour, response times creep up, a node fills its disk. You find out what is happening through three kinds of signals. Logs tell you what a program said, metrics are numbers measured over time, and events are Kubernetes' own record of what it did. Together they are called observability.

In this lesson you will read logs across many Pods, understand where logs go and why you need a log collector, use events and kubectl top, and install Prometheus and Grafana with one Helm chart.

Logs: what kubectl gives you

Containers should write logs to standard output and standard error, not to files. The container runtime stores that output on the node, and kubectl logs reads it. For several Pods at once, use a label selector and --prefix to see which line came from where:

kubectl logs -l app=node-agent --prefix --tail=1

[pod/node-agent-7vxqp/agent] node-agent alive on kind-worker2
[pod/node-agent-hkz5c/agent] node-agent alive on kind-worker
[pod/node-agent-q2z8m/agent] node-agent alive on kind-control-plane

These are the DaemonSet Pods from DaemonSets. Other useful flags: -f to follow, --since=15m for recent lines only, --previous for the run before a crash, --all-containers for every container in a Pod, and kubectl logs deploy/menu to pick a Pod of a Deployment.

The limits of kubectl logs matter in production:

• Logs live on the node. The kubelet rotates them (by default at 10 MiB, keeping 5 files per container), and kubectl logs shows only the latest file.
• When a Pod is deleted or its node dies, its logs are gone.
• You cannot search across all Pods for "payment failed" in the last week.

Kubernetes has no built-in log storage. The standard answer is a node logging agent: a DaemonSet such as Fluent Bit, Fluentd, Vector or the OpenTelemetry Collector that reads every container's log files on each node and ships them to a store such as Loki, Elasticsearch or OpenSearch, or a cloud logging service.

Write logs that are easy to search

Structured logs, one JSON object per line, can be filtered by field instead of by guessing text. A good log line from Farhan's payment API looks like this:

{"time":"2026-11-02T10:42:07.318+05:30","level":"error","msg":"payment failed","order_id":"ORD-48213","amount_inr":1299,"gateway":"upi","duration_ms":842,"trace_id":"4bf92f3577b34da6"}

Include a level, a clear message and the IDs you will search by. Never log passwords, OTPs, full card numbers or personal data such as Aadhaar numbers; logs are copied to many places and kept for a long time.

Events

Events are short records that Kubernetes components write about objects: scheduled, image pulled, probe failed, OOM killed, quota exceeded. They are the first stop when a Pod will not start:

kubectl events -n shop-prod --types=Warning
kubectl describe pod <name> -n shop-prod

Events are kept for only about an hour by default, so export them with your logs if you need history.

Metrics: from kubectl top to Prometheus

You installed metrics-server in Autoscaling with the HPA. It powers kubectl top, which is great for a quick look:

kubectl top pods -n shop-prod --sort-by=memory
kubectl top nodes

But metrics-server only keeps the current value. To answer "how much memory did the API use at 9 p.m. yesterday?" or "alert me when errors exceed 1%", you need a time-series database. In Kubernetes that is almost always Prometheus, a CNCF graduated project.

Prometheus pulls (scrapes) metrics over HTTP from targets every few seconds and stores them as time series. Apps expose numbers in a simple text format, usually at /metrics, with labels in braces:

# HELP http_requests_total Requests handled, by route and status.
# TYPE http_requests_total counter
http_requests_total{route="/api/orders",status="200"} 48213
http_requests_total{route="/api/orders",status="500"} 37

Client libraries for every major language produce this for you. You query the data with PromQL. Two everyday queries, the request rate per second over the last 5 minutes and the share of server errors:

sum(rate(http_requests_total[5m]))
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

Install the whole stack with one chart

The community chart kube-prometheus-stack installs the Prometheus Operator, Prometheus, Alertmanager, Grafana, node-exporter (node metrics) and kube-state-metrics (the state of Kubernetes objects), along with ready-made dashboards and alert rules:

helm install monitoring oci://ghcr.io/prometheus-community/charts/kube-prometheus-stack -n monitoring --create-namespace
kubectl port-forward svc/monitoring-grafana -n monitoring 3000:80

Open http://localhost:3000 for Grafana's dashboards of nodes, namespaces, workloads and Pods. The chart stores Grafana's admin login in a Secret in the monitoring namespace; read the chart's notes and values to find it, and change it. This stack is heavy for a laptop, so give Docker enough memory.

To scrape your own app, the Prometheus Operator uses a ServiceMonitor: "scrape every Service with this label, on this port":

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: payments-api
  namespace: shop-prod
  labels:
    release: monitoring
spec:
  selector:
    matchLabels:
      app: payments-api
  endpoints:
    - port: http
      path: /metrics
      interval: 30s

The release: monitoring label matters: by default the chart's Prometheus only picks up ServiceMonitors labelled with its release name.

Alerts

Alertmanager receives alerts from Prometheus rules, groups them and sends them to email, Slack, Microsoft Teams, PagerDuty or a webhook. Good alerts describe symptoms users feel (error rate above 1% for 10 minutes, checkout latency above 2 seconds), not every internal hiccup. A team that is paged ten times a night soon ignores every page.

Common mistakes

• Writing logs to files inside the container. They are invisible to kubectl logs and log collectors, and fill the container's disk.
• Relying on kubectl logs in production. Deleted Pods take their logs with them. Ship logs off the node.
• Using metrics-server as monitoring. It has no history. Use Prometheus.
• High-cardinality labels such as user IDs or order IDs in metrics. Each value creates a new time series and can overload Prometheus. Put such IDs in logs.

Key points

• Containers log to stdout and stderr; kubectl logs reads the latest rotated file on the node, with -l, --prefix, -f and --previous.
• Kubernetes stores no logs centrally; a DaemonSet log agent ships them to a log store.
• Events explain what Kubernetes did and are the first stop for Pods that will not start.
• metrics-server gives current CPU and memory for kubectl top and autoscaling; Prometheus scrapes and stores metrics over time and is queried with PromQL.
• kube-prometheus-stack installs Prometheus, Alertmanager, Grafana and exporters; ServiceMonitors add your own apps.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: exam-results
  namespace: exam-prod
  labels:
    release: monitoring
spec:
  selector:
    matchLabels:
      app: exam-results
  endpoints:
    - port: metrics
      path: /metrics
      interval: 15s

sum(rate(http_requests_total{status="500"}[5m]))