Linux Debugging Toolkit
When a process is slow, stuck, or leaking — the commands that tell you what it is actually doing: strace, lsof, perf, and friends.
What is it doing? (syscalls)
- Trace syscalls
strace -p PID· attach to a running process- Only file/network
strace -e trace=openat,connect -p PID- Count + time syscalls
strace -c -p PID— summary table, find the hot call- Follow children
strace -f ./program- Library calls
ltrace -p PID— like strace but for libc calls
What is it holding open?
- Open files/sockets
lsof -p PID- Who owns a port
lsof -i :8080· orss -ltnp- What deletes free space
lsof +L1— deleted-but-open files (disk not freed)- Files under a dir
lsof +D /var/log
Where is the CPU going?
- Live top by thread
top -H -p PID· orhtop- Sample a profile
perf top -p PID— live hottest functions- Record + report
perf record -g -p PIDthenperf report- Flame graph
perf script→ FlameGraph tools → interactive SVG- Quick stack dump
cat /proc/PID/stack·gdb -p PIDthenbt
Memory, disk, kernel
- Process memory
cat /proc/PID/status(VmRSS) ·pmap -x PID- System memory
free -h·vmstat 1- Disk I/O by process
iotop· overall:iostat -x 1- Kernel messages
dmesg -T | tail— OOM kills, segfaults, hardware- Was it OOM-killed?
dmesg | grep -i "killed process"