Bash & The Terminal

Pipes, process control, the text-tool trinity and the shell one-liners that replace small programs.

Bash is the glue of Unix: small programs that each do one thing, wired together with pipes. Three ideas carry every row below. Each program has three streams, stdin (0), stdout (1) and stderr (2), and redirection rewires them. The shell expands $var, globs and $(cmd) before the command runs, so quoting decides what the program actually receives. And the exit status is the control flow: 0 means success, anything else is failure, which is what &&, ||, if and set -e all test.

Syntax is bash with GNU tools. macOS ships bash 3.2 and BSD tools, so a few flags differ; those are marked.

Pipes & redirection

cmd1 | cmd2: stdout → stdin; the whole philosophy in one character
> · >> · 2>&1: overwrite · append · send stderr where stdout goes
cmd >/dev/null 2>&1: silence everything; order matters — 2>&1 >/dev/null still prints errors
&> out.log: bash shorthand for >out.log 2>&1; not POSIX, so avoid it under #!/bin/sh
2> err.log: send only stderr to a file; stdout still reaches the terminal
tee -a log.txt: write to a file AND pass through: cmd | tee -a log.txt; drop -a to overwrite
$(cmd): command substitution — use the output inline; trailing newlines are stripped
<(cmd): process substitution — output as a file: diff <(sort a) <(sort b)
<<'EOF': heredoc; quoting the delimiter stops $var expansion — a bare <<EOF expands it
xargs: stdin → arguments: ls *.log | xargs rm; splits on spaces, so pair -print0 with xargs -0
xargs -P 4 -n 1: run up to 4 commands in parallel, one argument each

grep

grep -rn "x" src/: recursive search with line numbers; -v invert, -E regex, -c count
grep -i · -w · -F: ignore case · whole words only · fixed string, so . and * are literal
grep -l · -o: list matching files only · print just the matched text, one per line
grep -C 3: 3 lines of context around each hit; -A after only, -B before only
grep -r --include='*.py': search only matching file names; --exclude-dir=node_modules skips a directory

sed, awk & friends

sed 's/old/new/g': stream replace; sed -n '5,10p' prints lines 5–10; -E enables extended regex
sed -i.bak 's/a/b/' f: edit in place with a backup — the one spelling both GNU and macOS sed accept
awk '{print $2}': column extraction; -F, sets delimiter; $NF = last field. Single quotes keep the shell off $2
awk '{s+=$1}END{print s}': sum a column — END runs once after the last line
awk -F, '$3 > 100': filter rows: a bare condition prints every line where it is true
sort | uniq -c | sort -rn: the frequency-count pipeline; uniq only merges adjacent duplicates, hence the first sort
sort -t, -k2,2n: sort by the 2nd comma-separated field, numerically; -u dedupes, -h sorts sizes like 2K
cut -d: -f1: lighter column cutter for delimited data
tr -d '\r': strip Windows CR line endings; tr a-z A-Z upper-cases
head / tail -f: first lines / follow a growing log live; -F also survives log rotation

Files & find

find . -name '*.log': find by name; quote the glob or the shell expands it first. -iname ignores case
find . -mtime +7: modified over 7 days ago; -mmin -30 = changed in the last 30 minutes
find . -size +100M: files over 100 MB; add -type f for files only, -type d for directories
find … -exec cmd {} +: run a command on results in batches; \; instead runs it once per file
find … -delete: delete matches; run once without -delete to check the list first
du -sh * | sort -h: size of each item here, smallest to largest; * skips dotfiles
df -h · df -i: free space · free inodes — “disk full” with space left usually means inodes
tar -czf a.tgz dir/: create a gzip tarball; -xzf extracts, -tzf lists contents
rsync -av src/ dst/: sync a tree; the trailing slash on src/ copies its contents, not the folder
mktemp -d: unique temp directory; remove it with trap 'rm -rf "$tmp"' EXIT

Variables & expansion

"${var:-default}": default values; ALWAYS quote "$var"
${var:?msg}: abort with msg when unset or empty — guards required input
${f%.txt} · ${f#*/}: strip shortest suffix · prefix match; doubled %% / ## strip the longest
${var//old/new}: replace every match; a single slash replaces only the first
${var,,} · ${var^^}: lower-case · upper-case (bash 4+; macOS ships bash 3.2)
$((a + b)): integer arithmetic only — use bc or awk for decimals
"$@": all arguments, each kept intact; unquoted $* re-splits on spaces
$? · $! · $$: last exit status · last background PID · this shell’s PID
arr=(a b c): array: expand with "${arr[@]}", count with ${#arr[@]}

Script survival

set -euo pipefail: exit on error, unset vars and pipe failures; -e is ignored in if tests and &&/|| chains
trap 'cleanup' EXIT: run cleanup however the script ends — success, error or Ctrl-C
if [[ -f file ]]: file exists; -d dir · -z empty string
[[ $s == *.txt ]]: glob match; =~ does regex: [[ $s =~ ^[0-9]+$ ]]. Prefer [[ ]] over [ ] in bash
for f in *.txt; do … done: the loop you write weekly; with no match it runs once on the literal *.txt
while IFS= read -r line: read a file line by line intact: … done < file; plain read mangles backslashes
case "$1" in: branch on patterns: start) …;; with *) as the default; close with esac
command -v jq: exits non-zero when the tool is missing — check dependencies before you need them
bash -x script.sh: trace every command as it runs; bash -n checks syntax without running
shellcheck script.sh: linter that catches quoting and logic bugs; worth running in CI

Processes & jobs

ctrl-z · bg · fg: suspend · resume in background · bring back
cmd &: run in the background; jobs lists them, wait blocks until all finish
ps aux | grep x · kill -9: find and kill; try plain kill (TERM) before -9
pgrep -f · pkill -f: find or signal processes by full command line — no grep -v grep needed
nohup cmd &: survive terminal close; check jobs
lsof -i :3000: what is holding that port
timeout 10 cmd: kill the command after 10 seconds; exits 124 on timeout (GNU coreutils)
time cmd: wall/user/sys — first stop for "why slow"

History & keys

ctrl-r: reverse history search — the highest-ROI keybinding in Unix
!! · sudo !!: repeat the last command — usually to re-run it with sudo
!$ · alt-.: last argument of the previous command; alt-. inserts it, repeat to go further back
ctrl-u · ctrl-k · ctrl-w: cut to start · to end · previous word; ctrl-y pastes it back
ctrl-x ctrl-e: open the current command line in $EDITOR for long one-liners
cd -: jump back to the previous directory

Recipes

Find the 10 largest files under a directory

find . -type f -exec du -k {} + 2>/dev/null | sort -rn | head -n 10

Sizes are in KiB and 2>/dev/null hides permission errors. To rank directories too, use du -ah | sort -rh | head.

Count the most frequent IPs in an access log

awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -n 20

The frequency-count pipeline: uniq -c only merges adjacent duplicates, so the first sort is not optional.

Rename a string across a repo, safely

grep -rl --null 'oldName' src/ | xargs -0 sed -i.bak 's/oldName/newName/g'
git diff --stat
find src/ -name '*.bak' -delete

--null with xargs -0 survives spaces in file names, and -i.bak works on GNU and macOS sed alike. Review the diff before deleting the backups.

A script skeleton that fails loudly and cleans up

#!/usr/bin/env bash
set -euo pipefail
[[ $# -eq 1 ]] || { echo "usage: $0 FILE" >&2; exit 2; }
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
: > "$tmp/out.txt"
while IFS= read -r line; do
  if [[ -n $line ]]; then
    printf '%s\n' "$line" >> "$tmp/out.txt"
  fi
done < "$1"
sort -u "$tmp/out.txt"

It stops on bad input, removes its temp directory on success, error or Ctrl-C, and reads every line intact. Replace the loop body with the real work.

Compress week-old logs in parallel

find /var/log/myapp -name '*.log' -mtime +7 -print0 | xargs -0 -P 4 -n 1 gzip

-P 4 runs four gzip processes at once; -print0 and -0 keep unusual file names intact.

Questions about Bash & The Terminal Cheat Sheet

What does set -euo pipefail do in a bash script?

It turns on three safety switches. -e exits the script as soon as a command fails instead of carrying on with bad state. -u treats an unset variable as an error, which catches typos such as $flie. -o pipefail makes a pipeline fail when any stage fails; by default only the last command counts, so a failing grep piped into sort looks successful. The catch is that -e is deliberately ignored in some places: the condition of an if or while, every command except the last in an and/or chain, and a failure hidden inside local x=$(cmd). Check those exit codes yourself.

Why should I always quote variables in bash?

Because an unquoted expansion is split on whitespace and then glob-expanded before the command ever sees it. If file holds my report.txt, rm $file tries to delete two files named my and report.txt, and a value containing an asterisk expands to every file in the directory. Writing "$file" passes exactly one argument with the value intact. The same rule applies to "$@" when forwarding arguments and to command substitution. Cases where you really want splitting should be deliberate and commented. Running shellcheck on a script flags every unquoted expansion it finds.

What is the difference between [ and [[ in bash?

[ is the POSIX test command, so its arguments go through normal word splitting and globbing: an empty or spaced variable breaks it unless quoted, and combining tests needs separate brackets. [[ is bash syntax parsed by the shell itself. Variables inside it are not split, so [[ -n $x ]] is safe unquoted, and it adds glob matching with ==, regex matching with =~, and and/or operators inside the brackets. Use [[ in bash scripts for safety and readability, and [ only when the script must also run under plain sh, such as dash on Debian and Ubuntu.

Why does my bash script behave differently on macOS?

macOS ships bash 3.2 from 2007, because later versions are GPLv3, and it uses BSD versions of the core tools rather than GNU ones. So bash 4 features such as associative arrays, mapfile and case conversion are missing, and flags differ: sed -i requires a suffix argument, grep -Z means decompress rather than null-terminate, and find has no -printf. The fixes are to install a current bash with Homebrew and start scripts with #!/usr/bin/env bash, to install GNU coreutils for the g-prefixed tools, or to stick to POSIX features and portable spellings such as sed -i.bak.

Go deeper

Agentic Coding handbook (your agent lives here) →