Linux Debugging Toolkit

Q: What is the difference between strace and ltrace?

strace intercepts system calls, the requests a process makes to the kernel such as open, read, connect and futex, so it shows how a program interacts with files, the network and other processes. ltrace intercepts calls into shared libraries, such as malloc or strlen in libc, which shows what a program does in user space. Both rely on ptrace and slow the traced process considerably, sometimes by an order of magnitude or more, so attach them briefly in production or use a sampling tool such as perf instead.

Q: Why is my disk still full after deleting a large file?

Deleting a file only removes its name from the directory. If a process still has the file open, such as a daemon writing to a log, the kernel keeps the data until the last descriptor closes, so df keeps counting the space while du can no longer find the file. Run lsof +L1 to list deleted files that are still open, then restart the process or signal it to reopen its logs. To free the space immediately, truncate the file through the process’s descriptor under /proc/PID/fd.

Q: What does the D state mean in ps or top?

D is uninterruptible sleep: the process is waiting inside the kernel, usually for disk or network file system I/O, and cannot be interrupted, not even by kill -9, until that operation finishes. A few processes briefly in D is normal under load. Many processes stuck in D for long periods point to a struggling disk, a hung NFS mount or a storage driver problem, and they also inflate the load average, which on Linux counts D-state tasks alongside runnable ones. Check iostat, dmesg and the wchan column to see what they wait on.

Q: How do I find which process is using a port on Linux?

Run ss -ltnp to list listening TCP sockets with the owning process name and PID; add -u to cover UDP. lsof -i :8080 does the same for a single port and also works on macOS. Run either one as root to see processes owned by other users, otherwise the process column can come back empty. If something is listening but connections still fail, check whether the service is bound to 127.0.0.1 instead of all interfaces, which ss shows in its local address column.

When a process is slow, stuck, or leaking — the commands that tell you what it is actually doing: strace, lsof, perf, and friends.

Debugging a live process means asking the kernel, which already tracks everything the process does. strace shows the system calls it makes, lsof and ss show the files and sockets it holds, perf samples where its CPU time goes, and /proc/PID/ exposes its memory, limits and state as plain files.

Work from the outside in: find which resource is saturated (CPU, memory, disk or network), then narrow to the process, then to the code. Most tools need root or the same user as the target, and strace slows a process badly, so attach briefly in production.

The first 60 seconds

uptime: 1, 5 and 15-minute load averages — rising or falling? Linux load also counts D-state tasks
dmesg -T | tail: kernel messages with readable times: OOM kills, segfaults, disk and NIC errors
vmstat 1: r run queue vs CPU count, si/so swapping, wa I/O wait — a line per second
mpstat -P ALL 1: per-CPU usage: one core pinned at 100% means a single-threaded bottleneck
pidstat 1: per-process CPU each second; -d disk I/O, -r memory, -w context switches
iostat -xz 1: per-device %util, await and queue size; -z hides idle devices
free -h: trust the available column — “used” includes reclaimable page cache
sar -n DEV 1: network throughput per interface — compare it with the link’s capacity
sar -n TCP,ETCP 1: TCP connections per second and retransmits — retransmits point at the network

What is it doing? (syscalls)

strace -p PID: attach to a running process and print its syscalls live
strace -f ./program: trace from launch, following child processes and threads
strace -c -p PID: summary table of syscall counts and time — find the hot call; Ctrl-C prints it
strace -e trace=network: one class only: network, file, process, or a list like openat,connect
strace -T -tt: time spent inside each syscall plus microsecond timestamps — spot the slow one
strace -s 200 -o out.txt: show 200 bytes of string arguments and write the trace to a file
strace -y: print the file path or socket behind each file descriptor number
ltrace -p PID: like strace, but for library calls such as libc

What is it holding open?

lsof -p PID: every file, socket and pipe the process holds open
lsof -i :8080: which process owns a port
lsof -iTCP -sTCP:LISTEN: every listening TCP socket; add -nP to skip DNS and port-name lookups
lsof +L1: deleted files still held open — why df stays full after you delete a log
lsof +D /var/log: processes with files open under a directory (slow on big trees)
ss -ltnp: listening TCP sockets with the owning process; -u for UDP
ss -s: socket summary — counts by state, including TIME-WAIT pile-ups
ss -tn state established: only established TCP connections; add -p for the owning process
ls /proc/PID/fd | wc -l: open descriptor count; compare with Max open files in /proc/PID/limits

Where is the CPU going?

top -H -p PID: live per-thread CPU for one process — find the hot thread (or use htop)
perf top -p PID: live view of the hottest functions
perf record -g -p PID: sample with call stacks until Ctrl-C, then perf report
perf record -F 99 -a -g: sample every CPU at 99 Hz; end with -- sleep 30 to stop after 30 seconds
perf stat -p PID: hardware counters: cycles, instructions, cache misses, context switches
perf script: → FlameGraph’s stackcollapse-perf.pl and flamegraph.pl → an interactive SVG
gdb -p PID: attach, then thread apply all bt for every thread’s stack; detaching resumes it
py-spy dump --pid PID: Python stacks without restarting the process; py-spy top is a live profiler
cat /proc/PID/stack: kernel stack of a process stuck inside a syscall (needs root)

Memory & OOM

ps aux --sort=-rss: processes by resident memory, biggest first; --sort=-%cpu for CPU
/proc/PID/status: VmRSS resident memory, VmSwap, Threads — quick facts for one process
pmap -x PID: memory per mapping: heap, stacks, libraries, anonymous regions
/proc/PID/smaps_rollup: RSS and PSS totals — PSS splits shared pages fairly between processes
dmesg | grep -i killed: was it OOM-killed? The log names the victim and its memory at the time
journalctl -k -p err: kernel errors from the journal; -b -1 looks at the previous boot
cat /proc/meminfo: full breakdown: MemAvailable, Dirty, Slab, huge pages

Disk & I/O

iotop -o: only processes doing I/O right now (root); without iotop use pidstat -d 1
df -h · df -i: space · inodes per filesystem — a full inode table looks like a full disk
du -xh --max-depth=1 /: biggest directories on one filesystem; -x stays off other mounts
ps -eo stat,pid,cmd: state D = uninterruptible sleep, usually stuck on disk or NFS; Z = zombie
cat /proc/pressure/io: share of time tasks stalled on I/O (kernel 4.20+); also cpu and memory

Network

tcpdump -i any port 443: capture packets; -nn skips name lookups, -w out.pcap saves for Wireshark
dig +short host: what DNS returns; getent hosts host shows what the system resolver gives apps
nc -zv host 443: can a TCP connection open at all? Separates firewall problems from app problems
mtr host: traceroute and ping per hop — shows where latency or loss begins
ip -s link: per-interface packets, errors and drops; ip addr and ip route for addressing

/proc, cgroups & services

cat /proc/PID/limits: the limits actually in force — open files, processes, core size
/proc/PID/environ: the environment it started with: tr '\0' '\n' < /proc/PID/environ
memory.events: cgroup v2: oom_kill count for a container; memory.max is its limit
cpu.stat: cgroup v2: nr_throttled and throttled_usec reveal CPU-limit throttling
systemctl status svc: state, main PID and recent logs; journalctl -u svc -f follows its log

Recipes

The first 60 seconds on a slow Linux box

uptime
dmesg -T | tail
vmstat 1 5
mpstat -P ALL 1 3
pidstat 1 3
iostat -xz 1 3
free -h
sar -n DEV 1 3
sar -n TCP,ETCP 1 3
top

This sequence, popularized by Brendan Gregg, checks load, kernel errors, CPU, disk, memory and network in turn, so the saturated resource usually names itself within a minute. mpstat, pidstat, iostat and sar come from the sysstat package.

Find out why a process is stuck

ps -o pid,stat,wchan:32,cmd -p 1234
cat /proc/1234/stack
strace -p 1234
gdb -p 1234 -batch -ex 'thread apply all bt'

State D with a disk or NFS function in wchan means it is waiting on the kernel; if strace shows it parked in futex or read, the gdb thread stacks show which lock or socket.

Disk is full but du disagrees

df -h /var
du -xh --max-depth=1 /var | sort -h
lsof +L1

When du totals less than df reports, a process still holds a deleted file open. Restart it, or truncate the file through /proc/PID/fd/N to free the space at once.

Record a CPU flame graph

perf record -F 99 -g -p 1234 -- sleep 30
perf script > out.perf
./stackcollapse-perf.pl out.perf > out.folded
./flamegraph.pl out.folded > flame.svg

Sample at 99 Hz for 30 seconds, then fold the stacks with Brendan Gregg’s FlameGraph scripts; the widest bars are where the time goes. Without frame pointers or debug symbols the stacks come out shallow.

Check whether a container is throttled or OOM-killed

cat /sys/fs/cgroup/cpu.stat
cat /sys/fs/cgroup/memory.events
cat /sys/fs/cgroup/memory.max

Run these inside the container on cgroup v2. A climbing nr_throttled means the CPU limit is too low even when average usage looks fine, and any oom_kill above 0 means the memory limit was hit.

Questions about Linux Debugging Toolkit Cheat Sheet

What is the difference between strace and ltrace?

strace intercepts system calls, the requests a process makes to the kernel such as open, read, connect and futex, so it shows how a program interacts with files, the network and other processes. ltrace intercepts calls into shared libraries, such as malloc or strlen in libc, which shows what a program does in user space. Both rely on ptrace and slow the traced process considerably, sometimes by an order of magnitude or more, so attach them briefly in production or use a sampling tool such as perf instead.

Why is my disk still full after deleting a large file?

Deleting a file only removes its name from the directory. If a process still has the file open, such as a daemon writing to a log, the kernel keeps the data until the last descriptor closes, so df keeps counting the space while du can no longer find the file. Run lsof +L1 to list deleted files that are still open, then restart the process or signal it to reopen its logs. To free the space immediately, truncate the file through the process’s descriptor under /proc/PID/fd.

What does the D state mean in ps or top?

D is uninterruptible sleep: the process is waiting inside the kernel, usually for disk or network file system I/O, and cannot be interrupted, not even by kill -9, until that operation finishes. A few processes briefly in D is normal under load. Many processes stuck in D for long periods point to a struggling disk, a hung NFS mount or a storage driver problem, and they also inflate the load average, which on Linux counts D-state tasks alongside runnable ones. Check iostat, dmesg and the wchan column to see what they wait on.

How do I find which process is using a port on Linux?

Run ss -ltnp to list listening TCP sockets with the owning process name and PID; add -u to cover UDP. lsof -i :8080 does the same for a single port and also works on macOS. Run either one as root to see processes owned by other users, otherwise the process column can come back empty. If something is listening but connections still fail, check whether the service is bound to 127.0.0.1 instead of all interfaces, which ss shows in its local address column.

Go deeper

Linux internals handbook →Bash cheatsheet →