Vibe Engines
YouTube

Vibe Engines is free, interactive AI engineering & system design — runs in your browser. Pick a learning path

Linux Debugging Toolkit

When a process is slow, stuck, or leaking — the commands that tell you what it is actually doing: strace, lsof, perf, and friends.

Debugging a live process means asking the kernel, which already tracks everything the process does. strace shows the system calls it makes, lsof and ss show the files and sockets it holds, perf samples where its CPU time goes, and /proc/PID/ exposes its memory, limits and state as plain files.

Work from the outside in: find which resource is saturated (CPU, memory, disk or network), then narrow to the process, then to the code. Most tools need root or the same user as the target, and strace slows a process badly, so attach briefly in production.

CPUMemoryDiskNetwork1 saturated?vmstat 1 (r, wa)free -hiostat -xz 1sar -n DEV 12 which PID?pidstat 1ps --sort=-rsspidstat -d 1ss -tnp3 doing what?perf record -gpmap -x PIDstrace -T -e filess -ti · tcpdumpfirst, always: uptime · dmesg -T | tail
Work down a column only after its first row shows saturation. Most slow-box hunts end on row one: the resource that is maxed out names itself before any per-process tool is needed.

The first 60 seconds

uptime
1, 5 and 15-minute load averages — rising or falling? Linux load also counts D-state tasks
dmesg -T | tail
kernel messages with readable times: OOM kills, segfaults, disk and NIC errors
vmstat 1
r run queue vs CPU count, si/so swapping, wa I/O wait — a line per second
mpstat -P ALL 1
per-CPU usage: one core pinned at 100% means a single-threaded bottleneck
pidstat 1
per-process CPU each second; -d disk I/O, -r memory, -w context switches
iostat -xz 1
per-device %util, await and queue size; -z hides idle devices
free -h
trust the available column — “used” includes reclaimable page cache
sar -n DEV 1
network throughput per interface — compare it with the link’s capacity
sar -n TCP,ETCP 1
TCP connections per second and retransmits — retransmits point at the network

What is it doing? (syscalls)

strace -p PID
attach to a running process and print its syscalls live
strace -f ./program
trace from launch, following child processes and threads
strace -c -p PID
summary table of syscall counts and time — find the hot call; Ctrl-C prints it
strace -e trace=network
one class only: network, file, process, or a list like openat,connect
strace -T -tt
time spent inside each syscall plus microsecond timestamps — spot the slow one
strace -s 200 -o out.txt
show 200 bytes of string arguments and write the trace to a file
strace -y
print the file path or socket behind each file descriptor number
ltrace -p PID
like strace, but for library calls such as libc

What is it holding open?

lsof -p PID
every file, socket and pipe the process holds open
lsof -i :8080
which process owns a port
lsof -iTCP -sTCP:LISTEN
every listening TCP socket; add -nP to skip DNS and port-name lookups
lsof +L1
deleted files still held open — why df stays full after you delete a log
lsof +D /var/log
processes with files open under a directory (slow on big trees)
ss -ltnp
listening TCP sockets with the owning process; -u for UDP
ss -s
socket summary — counts by state, including TIME-WAIT pile-ups
ss -tn state established
only established TCP connections; add -p for the owning process
ls /proc/PID/fd | wc -l
open descriptor count; compare with Max open files in /proc/PID/limits

Where is the CPU going?

top -H -p PID
live per-thread CPU for one process — find the hot thread (or use htop)
perf top -p PID
live view of the hottest functions
perf record -g -p PID
sample with call stacks until Ctrl-C, then perf report
perf record -F 99 -a -g
sample every CPU at 99 Hz; end with -- sleep 30 to stop after 30 seconds
perf stat -p PID
hardware counters: cycles, instructions, cache misses, context switches
perf script
→ FlameGraph’s stackcollapse-perf.pl and flamegraph.pl → an interactive SVG
gdb -p PID
attach, then thread apply all bt for every thread’s stack; detaching resumes it
py-spy dump --pid PID
Python stacks without restarting the process; py-spy top is a live profiler
cat /proc/PID/stack
kernel stack of a process stuck inside a syscall (needs root)

Memory & OOM

ps aux --sort=-rss
processes by resident memory, biggest first; --sort=-%cpu for CPU
/proc/PID/status
VmRSS resident memory, VmSwap, Threads — quick facts for one process
pmap -x PID
memory per mapping: heap, stacks, libraries, anonymous regions
/proc/PID/smaps_rollup
RSS and PSS totals — PSS splits shared pages fairly between processes
dmesg | grep -i killed
was it OOM-killed? The log names the victim and its memory at the time
journalctl -k -p err
kernel errors from the journal; -b -1 looks at the previous boot
cat /proc/meminfo
full breakdown: MemAvailable, Dirty, Slab, huge pages

Disk & I/O

iotop -o
only processes doing I/O right now (root); without iotop use pidstat -d 1
df -h · df -i
space · inodes per filesystem — a full inode table looks like a full disk
du -xh --max-depth=1 /
biggest directories on one filesystem; -x stays off other mounts
ps -eo stat,pid,cmd
state D = uninterruptible sleep, usually stuck on disk or NFS; Z = zombie
cat /proc/pressure/io
share of time tasks stalled on I/O (kernel 4.20+); also cpu and memory

Network

tcpdump -i any port 443
capture packets; -nn skips name lookups, -w out.pcap saves for Wireshark
dig +short host
what DNS returns; getent hosts host shows what the system resolver gives apps
nc -zv host 443
can a TCP connection open at all? Separates firewall problems from app problems
mtr host
traceroute and ping per hop — shows where latency or loss begins
ip -s link
per-interface packets, errors and drops; ip addr and ip route for addressing

/proc, cgroups & services

cat /proc/PID/limits
the limits actually in force — open files, processes, core size
/proc/PID/environ
the environment it started with: tr '\0' '\n' < /proc/PID/environ
memory.events
cgroup v2: oom_kill count for a container; memory.max is its limit
cpu.stat
cgroup v2: nr_throttled and throttled_usec reveal CPU-limit throttling
systemctl status svc
state, main PID and recent logs; journalctl -u svc -f follows its log

Worked example: Slow API, idle CPU

Latency on a 4-CPU database host has doubled, yet top shows the CPUs far from busy. uptime reports a load average of 6.1. Load above the CPU count with idle CPUs means tasks are waiting on something else. Columns below are trimmed to the ones that matter.

$ vmstat 1 3          # ignore the first line: averages since boot
 r  b  si  so     bi   bo  us sy id wa
 1  5   0   0 182400  310  12  6 37 45

Only 1 task is runnable (r) but 5 are blocked (b) in uninterruptible sleep, the CPUs spend 45% of their time in I/O wait, and nothing swaps. The blocked tasks count towards the load average, which explains 6.1 on an idle-looking box. Next, which disk:

$ iostat -xz 1 2
Device    r/s     rkB/s  r_await  aqu-sz  %util
sdb    1420.0  181760.0    38.50   54.70   99.80

sdb is saturated: 99.8% busy, about 55 requests queued, and each read waits 38.5 ms where a few milliseconds would be normal. 181,760 kB/s over 1,420 reads/s is 128 kB per read, the signature of a sequential scan. Which process:

$ pidstat -d 1 1
  UID    PID    kB_rd/s  kB_wr/s  Command
    0  48211  179200.00     0.00  tar
  999   1312     310.00   280.00  postgres
  1. A backup tar started at the top of the hour is reading about 175 MiB/s and starving Postgres of disk.
  2. ps -eo stat,pid,wchan:20,cmd | awk '$1 ~ /^D/' lists the blocked Postgres workers, all waiting on disk I/O.
  3. Fix: move the backup off-peak, or cap it with a cgroup I/O limit such as systemd’s IOReadBandwidthMax=. Rerun iostat afterwards to confirm r_await falls.

Recipes

The first 60 seconds on a slow Linux box

uptime
dmesg -T | tail
vmstat 1 5
mpstat -P ALL 1 3
pidstat 1 3
iostat -xz 1 3
free -h
sar -n DEV 1 3
sar -n TCP,ETCP 1 3
top

This sequence, popularized by Brendan Gregg, checks load, kernel errors, CPU, disk, memory and network in turn, so the saturated resource usually names itself within a minute. mpstat, pidstat, iostat and sar come from the sysstat package.

Find out why a process is stuck

ps -o pid,stat,wchan:32,cmd -p 1234
cat /proc/1234/stack
strace -p 1234
gdb -p 1234 -batch -ex 'thread apply all bt'

State D with a disk or NFS function in wchan means it is waiting on the kernel; if strace shows it parked in futex or read, the gdb thread stacks show which lock or socket.

Disk is full but du disagrees

df -h /var
du -xh --max-depth=1 /var | sort -h
lsof +L1

When du totals less than df reports, a process still holds a deleted file open. Restart it, or truncate the file through /proc/PID/fd/N to free the space at once.

Record a CPU flame graph

perf record -F 99 -g -p 1234 -- sleep 30
perf script > out.perf
./stackcollapse-perf.pl out.perf > out.folded
./flamegraph.pl out.folded > flame.svg

Sample at 99 Hz for 30 seconds, then fold the stacks with Brendan Gregg’s FlameGraph scripts; the widest bars are where the time goes. Without frame pointers or debug symbols the stacks come out shallow.

Check whether a container is throttled or OOM-killed

cat /sys/fs/cgroup/cpu.stat
cat /sys/fs/cgroup/memory.events
cat /sys/fs/cgroup/memory.max

Run these inside the container on cgroup v2. A climbing nr_throttled means the CPU limit is too low even when average usage looks fine, and any oom_kill above 0 means the memory limit was hit.

When to use it, and when not

These tools answer questions about one live machine you can log into, while the problem is happening or can be reproduced. They need no prior setup, which is why they are the first resort in an incident.

  • Many hosts or a past incident: use metrics, logs and a continuous profiler. A shell session cannot see last night, and a process that already died leaves only logs and, if enabled, a core dump (coredumpctl on systemd).
  • Hot production paths: prefer sampling (perf) or eBPF tools such as bpftrace and the bcc collection (execsnoop, opensnoop, biolatency), which cost far less than strace.
  • Containers often lack every tool here. Run them from the host against the container’s PID, enter its namespaces with nsenter -t PID -n -p, or use kubectl debug.

Trade-offs and limits

The tools differ most in what they cost the process you are inspecting:

ToolOverheadNeeds
strace, ltraceHigh: the process stops at every traced callRoot, or same user where kernel.yama.ptrace_scope allows
gdb -pProcess frozen while attachedSame as strace; symbols for readable stacks
perf at 99 HzLow: sampling, not tracingRoot or a permissive kernel.perf_event_paranoid; symbols or frame pointers
bpftrace / bccLow: runs in the kernelRoot and a reasonably recent kernel
tcpdumpModerate; captures fill disksRoot or CAP_NET_RAW
/proc, vmstat, ssNegligibleNothing for your own processes

With Yama’s ptrace_scope at 1, the default on Ubuntu, an unprivileged user can trace only its own child processes, so strace -p fails even on a process you own.

Common mistakes

  • Reading the first line of vmstat or iostat. It reports averages since boot, not now. Watch the lines after it, or use iostat -y to skip it.
  • Piping strace into grep. strace writes to stderr, so strace -p PID | grep open filters nothing. Use 2>&1 |, -o file, or narrow it at the source with -e trace=openat.
  • Reading 250% CPU as an error. top counts per core by default, so a process at 250% is using two and a half cores. Press H or use top -H -p PID to see which threads.
  • Profiling without symbols. perf shows hex addresses or [unknown] without debug symbols or frame pointers. JIT runtimes need a perf map: Node has --perf-basic-prof.
  • Leaving strace attached to a busy service. Every syscall now costs a round trip through the tracer, and request timeouts follow. Attach for seconds with -c or a narrow -e, then detach.

Check yourself

vmstat 1 on a 4-CPU machine shows r at 12 and id at 0. What does that tell you?

The CPUs are saturated: 12 runnable tasks share 4 CPUs, so about 8 are waiting for a turn at any moment. Find the consumers with pidstat 1.

df -h shows 30% used, yet writes fail with “No space left on device”. What do you check?

df -i. The filesystem has run out of inodes, usually from millions of tiny files such as a session or cache directory.

A container’s cpu.stat reads nr_periods 1000 and nr_throttled 400. What does that mean?

It hit its CPU quota in 400 of 1,000 enforcement periods: throttled 40% of the time. With the default 100 ms period, it ran out of quota partway through those periods and waited for the next one. Raise the limit or smooth the bursts.

Dozens of processes show state Z with the same parent. Will kill -9 clear them?

No. Zombies have already exited; only their exit status is left, waiting for the parent to call wait(). Fix or restart the parent; once it dies, init adopts the zombies and reaps them.

Questions about Linux Debugging Toolkit Cheat Sheet

What is the difference between strace and ltrace?

strace intercepts system calls, the requests a process makes to the kernel such as open, read, connect and futex, so it shows how a program interacts with files, the network and other processes. ltrace intercepts calls into shared libraries, such as malloc or strlen in libc, which shows what a program does in user space. Both rely on ptrace and slow the traced process considerably, sometimes by an order of magnitude or more, so attach them briefly in production or use a sampling tool such as perf instead.

Why is my disk still full after deleting a large file?

Deleting a file only removes its name from the directory. If a process still has the file open, such as a daemon writing to a log, the kernel keeps the data until the last descriptor closes, so df keeps counting the space while du can no longer find the file. Run lsof +L1 to list deleted files that are still open, then restart the process or signal it to reopen its logs. To free the space immediately, truncate the file through the process’s descriptor under /proc/PID/fd.

What does the D state mean in ps or top?

D is uninterruptible sleep: the process is waiting inside the kernel, usually for disk or network file system I/O, and cannot be interrupted, not even by kill -9, until that operation finishes. A few processes briefly in D is normal under load. Many processes stuck in D for long periods point to a struggling disk, a hung NFS mount or a storage driver problem, and they also inflate the load average, which on Linux counts D-state tasks alongside runnable ones. Check iostat, dmesg and the wchan column to see what they wait on.

How do I find which process is using a port on Linux?

Run ss -ltnp to list listening TCP sockets with the owning process name and PID; add -u to cover UDP. lsof -i :8080 does the same for a single port and also works on macOS. Run either one as root to see processes owned by other users, otherwise the process column can come back empty. If something is listening but connections still fail, check whether the service is bound to 127.0.0.1 instead of all interfaces, which ss shows in its local address column.