Advanced Debugging: Slim Images & Sidecars

The method in Debugging & Troubleshooting Containers (state, logs, inspect, look inside) solves most problems. It starts to fail on well-built production images. A distroless or hardened image has no shell, no curl, no ps and no netstat, so "look inside" seems impossible. That is a good thing for security, and it does not have to make you blind.

This lesson shows how professionals debug such containers without changing the image: by attaching a debug sidecar that shares the app's network or processes, by reading health logs and file changes from the outside, and by using host tools on a Linux server.

Look from outside first

Several Docker commands need nothing at all inside the container.

Healthcheck history. When a container is unhealthy, Docker keeps the output of the last few checks:

docker inspect --format '{{json .State.Health}}' shop-api

{"Status":"unhealthy","FailingStreak":4,"Log":[{"Start":"2026-11-02T10:41:07.512Z","End":"2026-11-02T10:41:07.598Z","ExitCode":1,"Output":"Error: connect ECONNREFUSED 10.0.2.15:3306\n"}]}

The output is shortened to one entry. It tells you the healthcheck itself fails because the database refuses connections, which is a very different problem from the app crashing.

File changes. docker diff lists files added (A), changed (C) or deleted (D) since the container started, compared with its image:

docker diff shop-api

C /tmp
A /tmp/upload-7f3a.jpg
C /app
A /app/sessions.json

An app writing sessions.json into /app is a bug waiting to happen: that file disappears whenever the container is replaced, and it fails completely with --read-only. docker cp shop-api:/app/sessions.json . copies it out for a closer look.

Events over time. docker events --filter container=shop-api --since 30m shows every start, die, kill, OOM and health change for the last half hour, which is ideal for "it restarted at 3 a.m., why?".

The sidecar trick: share namespaces

A container's isolation comes from Linux namespaces: one for the network, one for the process list, and so on. Docker lets a new container join another container's namespaces. So you start a throwaway container full of tools that shares the network or process space of the broken one, while the broken one stays untouched.

The community image nicolaka/netshoot is popular for this, because it bundles network tools such as curl, dig, ss, tcpdump and iperf3:

docker run --rm -it --network container:shop-api nicolaka/netshoot

Inside, localhost is now the app's localhost, and the debug shell sees exactly what the app sees:

ss -tlnp
curl -s http://127.0.0.1:3000/api/health
dig +short db

State  Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0      511                *:3000            *:*
{"status":"ok"}
172.20.0.3

The app listens on port 3000 on all addresses, answers its health route, and the name db resolves. If curl works here but not through the published port, the problem is the port mapping or the proxy, not the app.

To see the app's processes, share its PID namespace instead:

docker run --rm -it --pid container:shop-api --cap-add SYS_PTRACE alpine:3.22 sh
ps aux
ls /proc/1/root/app

ps now lists the app's processes. /proc/1/root is the app container's file system as seen through its PID 1, so you can read its files with the tools from Alpine. SYS_PTRACE is needed because Linux only lets a process look inside another process's files with that permission. Remove the debug container as soon as you are done; it has powerful access.

docker debug on Docker Desktop

Docker Desktop has a built-in command that does the sidecar trick for you:

docker debug shop-api

It opens a shell attached to the container, with a toolbox of common tools (such as curl, htop and vim) that is not part of the image. It also works on images and stopped containers; in that case, changes are thrown away when you exit. Extra tools can be added to your own toolbox with its install command. docker debug is a Docker Desktop feature, introduced in version 4.27 and generally available since 4.33, and Docker has tied it to certain subscription plans, so check what your plan includes. On a Linux server with only Docker Engine, use the sidecar method.

Host tools on a Linux server

On the server itself, nsenter runs a host program inside a container's namespaces. Find the container's main process ID on the host, then enter only its network namespace:

pid=$(docker inspect --format '{{.State.Pid}}' shop-api)
sudo nsenter --target "$pid" --net ss -tlnp

This runs the host's own ss in the container's network namespace. It is useful when you cannot pull an extra image, for example on a locked-down server with no internet access.

A crash loop with no shell

A container that restarts every few seconds is hard to examine. Slow it down and preserve the evidence:

1. Stop the restarts: docker update --restart no shop-api.
2. Read the logs of the last runs: docker logs --since 10m shop-api.
3. Save the stopped container's file system as an image to inspect at leisure: docker commit shop-api shop-api-crash:debug. Then explore it with docker debug shop-api-crash:debug, or export it with docker export shop-api > crash.tar and look through the archive with tar -tf crash.tar.

docker commit is a debugging tool here, not a way to build images. Never ship a committed image; fix the Dockerfile instead.

Common mistakes

• Rebuilding the image with a shell "just to debug". You change the thing you are investigating. Attach a sidecar instead.
• Forgetting that --network container:X shares localhost. In the sidecar, 127.0.0.1 is the app, not the sidecar's own services.
• Leaving debug containers running with SYS_PTRACE or shared namespaces.
• Ignoring health logs. .State.Health.Log often names the exact failing dependency.
• Shipping images made with docker commit. They cannot be rebuilt or reviewed.

Key points

• Start outside: health logs in docker inspect, docker diff, docker cp and docker events need no tools in the image.
• --network container:NAME and --pid container:NAME let a tool-filled sidecar such as nicolaka/netshoot see the app's network or processes.
• /proc/1/root in a PID-sharing sidecar (with SYS_PTRACE) shows the app's files.
• docker debug on Docker Desktop attaches a toolbox shell to containers and images; check your plan.
• On Linux servers, nsenter --target PID --net runs host tools in a container's network namespace; preserve crash loops with docker update --restart no and docker commit.

docker inspect --format '{{json .State.Health}}' fees-api
docker run --rm -it --network container:fees-api nicolaka/netshoot
ss -tlnp
curl -s http://127.0.0.1:8080/health