Sharing what
isn't enough.

An operating system is two things at once: a resource manager that shares scarce hardware — CPU, memory, disk, devices — among competing programs, and an abstraction layer that hides the messy hardware behind clean ideas: a process (a running program with its own memory), a file, a socket. Every program you run believes it has the whole machine. The OS maintains that fiction.

The fiction rests on two central scarcities. There's one CPU (or a few cores) but many programs want to run — so the OS schedules, deciding who runs now and for how long. And there's limited RAM but programs collectively want more — so the OS uses virtual memory, giving each process a private address space larger than physical memory by keeping only hot pages in RAM. Master these two — time-sharing the CPU and space-sharing memory — and you understand the beating heart of every OS. Everything else (filesystems, drivers, system calls) is built on top.

A CPU core can run exactly one instruction stream at a time, yet your machine runs hundreds of processes "simultaneously." The trick is speed: the OS runs a process for a tiny slice, then switches to another, fast enough that all of them appear to progress at once. The simplest fair policy is round-robin: give each ready process a fixed slice — a quantum — then rotate to the next, sending the interrupted one to the back of the queue.

Every rotation costs a context switch: the OS saves the running process's registers and state, and loads the next process's — pure overhead, no useful work done during it. That sets up the central tension of scheduling. A large quantum means fewer switches (less overhead) but worse responsiveness — a short interactive task waits behind long ones, degrading toward first-come-first-served. A small quantum means snappy responsiveness but more time lost to switching. Real schedulers (Linux's CFS, priority multilevel queues) are far more sophisticated, but they're all refinements of the same idea round-robin makes plain: rotate the CPU so no one starves, and pay for fairness in context-switch overhead.

Virtual memory gives every process a large, private, contiguous address space — an illusion, because physical RAM is small and shared. Memory is cut into fixed-size pages; the OS maps each virtual page to a physical frame, keeping only the actively used pages in RAM and leaving the rest on disk. When a process touches a page that isn't resident, the hardware raises a page fault, and the OS fetches it from disk — evicting another page if RAM is full.

Because a disk fetch is millions of times slower than a RAM hit, the whole game is minimizing page faults, and that's what the replacement policy decides. LRU — evict the least-recently-used page — bets on locality: a page used recently is likely to be used again soon, so the coldest page is the safest to drop. Most real programs have strong temporal locality, so LRU keeps fault rates low. Its exact form is costly to track, so kernels approximate it (clock/second-chance), but the principle is the same one behind every cache: keep what's hot, evict what's cold.

Round-robin serves each process a quantum in rotation; a process finishes when its accumulated slices cover its burst, so short jobs leave the rotation first. Paging counts a fault whenever a referenced page isn't resident:

Bursts A=4,B=2,C=1 with quantum 2 finish in order B, C, A — B and C complete in round one, A drags to the end.

Page faults fall (weakly) as frames rise, because more RAM holds a larger working set — and the hit ratio is one minus the fault fraction:

The reference stream 1,2,3,4,1,2,5,1,2,3,4,5 causes 10 faults with 3 frames but only 8 with 4 — more RAM, fewer trips to disk. The runnable version below simulates both.

A handful of terms unlock most OS discussions:

The through-line is isolation and mediation: processes can't touch each other's memory or the hardware directly — they ask the kernel, which arbitrates. That's why a buggy app crashes alone instead of taking the system with it, and why the CPU has a hardware-enforced kernel/user boundary at all. Scheduling and paging are how the kernel shares the CPU and memory; the rest of the vocabulary names the machinery that keeps the sharing safe.

Everything above this point is the kernel's view of memory: pages, frames, faults, evictions. It is a wholesale view. The kernel's smallest unit of memory is a page — 4 KiB on x86-64 Linux and Windows, 16 KiB on Apple Silicon, with 2 MiB and 1 GiB huge pages available on request — and the only way to get one is a system call.

But no program allocates a page. Programs allocate a 24-byte tree node, a 40-byte string header, a three-element array. A busy service does that millions of times a second and frees most of it just as fast. Between "the kernel deals in 4096-byte units, via syscalls" and "my code wants 24 bytes, now" sits a piece of software almost nobody thinks about: the allocator, the code behind malloc(). Understanding it is what turns "my process is using 6 GB and I don't know why" from a mystery into a diagnosis.

Why not just ask the kernel every time?

Because asking is expensive twice over. Getting memory from the OS means mmap or brk — a system call, which crosses the user/kernel boundary, and on any kernel with Spectre/Meltdown mitigations enabled that crossing costs on the order of hundreds of nanoseconds to a couple of microseconds. Worse, the call doesn't even give you memory: it gives you mappings. The pages it returns are not backed by physical frames yet, so the first touch of each one raises a minor page fault and takes another trip into the kernel.

A fast-path malloc() that pops a block off a per-thread free list costs tens of nanoseconds and touches nothing but memory the process already owns. That's roughly two orders of magnitude. So allocators buy wholesale and sell retail: one syscall grabs a large slab, and thousands of small requests are served out of it with pure in-process bookkeeping.

The shelf, in glibc's actual numbers

glibc's allocator (ptmalloc2) grows the main heap with brk, padding each extension by M_TOP_PAD so it doesn't have to come back immediately. Requests at or above M_MMAP_THRESHOLD skip the heap entirely and get their own private mmap. Everything smaller is cut out of the slabs the allocator already holds, tracked in size-classed free lists:

Two maintenance moves keep that shelf usable. Splitting: when the smallest adequate free block is much bigger than the request, the allocator carves off what it needs and returns the remainder to the shelf. Coalescing: when a freed block is adjacent to another free block, they're merged into one larger block — a chunk header bit (PREV_INUSE) lets free() check its lower neighbour in O(1) without walking the heap.

One subtlety worth knowing: the mmap threshold is dynamic. When you free an mmap'd block of size S, glibc raises the threshold to S (capped at 32 MiB) on the theory that you'll ask again. That's why a loop allocating and freeing 1 MiB buffers is slow on its first few iterations — two syscalls and 256 fresh page faults each — and fast forever after.

Why free() doesn't shrink your process

free() almost never talks to the kernel. It flips the chunk to free, drops it into the right bin, maybe coalesces with a neighbour — and stops. The pages are still mapped; the kernel still considers them yours. That's precisely what makes the next malloc() of the same size instant, and it's why top can show a process holding 6 GB after it freed 5.9 GB of it.

There are only two ways memory goes back. An mmap'd block (≥ the threshold) is munmap'd on free. And the main heap can be trimmed with brk — but brk only moves the top. If a single live 32-byte chunk sits above a 10 GiB region you just freed, all 10 GiB stays mapped, because there is no way to move the top past a live object. This is the single most common source of "my long-running service leaks" reports that turn out not to be leaks at all.

Modern allocators route around it with madvise(MADV_DONTNEED) or MADV_FREE: keep the virtual mapping, but tell the kernel to reclaim the physical frames. RSS falls; virtual size doesn't. jemalloc and tcmalloc do this continuously on a decay timer measured in seconds; glibc mostly does it only when you call malloc_trim(0) yourself.

That also gives you a clean way to separate the two classic diagnoses. A leak is a bookkeeping failure: a chunk you were handed and never returned, usually because the last pointer to it was lost. To the allocator it is simply still in use, forever. Fragmentation is a layout failure: plenty of total free bytes, scattered as gaps too small to satisfy the next request. malloc_info() tells them apart — if in-use bytes track your expectations but system bytes are far larger, you have fragmentation or arena bloat; if in-use bytes climb monotonically, you have a leak. See the glibc malloc internals and mallopt(3) for the full knob set; the per-thread caching story connects directly to locking and contention, since the whole arena design exists to avoid one global heap lock.

And note where this hands back to the pager: virtual memory is nearly free, physical memory is spent on first touch. calloc(1, 1<<30) on a fresh mapping costs almost no resident memory, because the kernel points every one of those 262,144 pages at a single shared zero page, copy-on-write — the same copy-on-write mechanism the fork() video at the end of this page walks through. RSS grows only as you write. Remember that phrase, first touch decides. It comes back with teeth two sections from now.

The allocator asks the kernel for anonymous pages — pages backed by nothing but swap. But a page table entry can point at something else: a file. Once you see that, the whole read-a-file-into-a-buffer ritual starts to look like unnecessary work.

The conventional path is read(). The kernel pulls blocks off storage into the page cache, then copies them again into a buffer you allocated. Two copies, and your memory use is proportional to how much of the file you want resident at once. Want random access across a 50 GB file? Classically, you need a 50 GB buffer, or you hand-roll a paging scheme of your own — which is to say, you reimplement the thing described in section 03, badly.

What the mapping actually costs

mmap() inverts the arrangement: instead of copying file data into memory you own, you declare that a range of your address space is that file. The kernel records a VMA (a virtual memory area: start, length, file, offset, permissions) and hands back a pointer. That's it. Zero bytes of I/O. Mapping a 50 GB file takes microseconds, because mapping is bookkeeping, not reading. You have plenty of room for it: a 64-bit process on x86-64 gets 128 TiB of user address space under 4-level paging (128 PiB with 5-level), so a 50 GB mapping consumes about 0.04% of it.

Data moves on demand. Touch a byte, the hardware finds no valid PTE, and you take a page fault — the exact mechanism from section 03, now doing double duty as a file-read trigger. Two flavours, and the distinction is the whole performance story:

So reading one byte 40 GB into a 50 GB file pulls in one 4 KiB page (plus whatever fault-around picks up for free), not the 40 GB before it. That is the entire trick, and it is just section 03's pager pointed at a file instead of swap.

Why two processes get the same physical page

Those pages land in the systemwide page cache, not in a private buffer. mmap doesn't add a caching layer — it lets your process address the one the kernel already maintains. So if two unrelated processes map the same file and touch the same offset, they are looking at the same physical frame. This is why libc.so is loaded once and its text pages are shared by every process on the machine, and why summing RSS across ps output routinely exceeds the machine's physical RAM: shared pages are counted once per process. PSS in /proc/<pid>/smaps is the number that actually adds up.

Writes split into two modes. MAP_SHARED makes your stores visible to everyone else mapping the file, and dirty pages are written back to disk by the kernel's writeback threads (or on demand via msync). MAP_PRIVATE gives you copy-on-write: reads share the page cache, but the first store to a page triggers a fault that copies it into a private anonymous frame. That is the very same COW machinery fork() uses — the difference is only what sits behind the page.

The failure modes

mmap's cost is that it converts I/O errors and I/O latency into things the CPU does silently, inside ordinary load instructions. Four consequences engineers hit for real:

• SIGBUS instead of an error code. read() returns -EIO or a short read and you handle it. A load from a mapped page has no return value. Truncate a file out from under a mapping, or fill the filesystem while writing to a sparse MAP_SHARED region, and the process takes SIGBUS at an arbitrary instruction. Handling it means a signal handler plus sigsetjmp — which is to say, most programs simply die.
• Uncontrollable stalls. Any load can block for milliseconds, with no way to express it asynchronously, no timeout, no backpressure. In an N-worker thread pool, N simultaneous major faults stall all N workers, and your p99 detonates while the CPU graph looks idle.
• Page-table overhead. Fully touching 50 GB with 4 KiB pages means ~13 million PTEs — roughly 100 MB of page tables per process, plus the TLB misses to walk them. Huge pages cut the entry count by 512×, which is why madvise(MADV_HUGEPAGE) matters on large mappings. Unmapping is worse: munmap must shoot down TLB entries on every core that has them, via inter-processor interrupts, and that cost grows with core count.
• You don't control eviction. The kernel's LRU approximation decides which of your pages to drop, using global memory pressure you can't see. A database that wants to guarantee its hot index stays resident cannot express that through mmap — which is the core of the CIDR 2022 argument in Are You Sure You Want to Use MMAP in Your DBMS?. Feel the eviction dynamics yourself in the cache eviction lab.

So the honest rule: mmap wins for read-mostly, random-access, larger-than-RAM data that many processes share — shared libraries, read-only indexes, embedded stores like LMDB, analytics scanning columnar files. It loses for write-heavy transactional storage that needs precise durability and eviction control, which is why most serious storage engines manage their own buffer pool instead (compare the access patterns in the B-tree vs LSM lab). The same page-cache-sharing idea, incidentally, is what makes sendfile/splice zero-copy sends possible on the network path. Details in mmap(2) and madvise(2).

Three assumptions have been quietly load-bearing so far. The pager assumed any free frame is as good as any other. The allocator assumed a byte is a byte, wherever it came from. mmap assumed the page cache is one shared pool. On a laptop all three are true. On a two-socket server none of them are.

A big machine has several CPU packages, each with its own memory controller and its own DIMMs wired directly to it. A socket plus its attached memory is a NUMA node. The kernel still presents one flat physical address space, so nothing in your code changes — but physically, memory is now several piles, and each pile is near one set of cores and far from the rest. Non-Uniform Memory Access is the admission that a physical frame has a location.

The actual numbers

On a typical modern two-socket server, an idle-latency load from the local memory controller lands around 80–90 ns; the same load to the other socket's memory crosses the inter-socket interconnect and lands around 130–150 ns. Call it 1.5×–2× worse latency. That ratio is the number everyone quotes, but bandwidth is usually the number that hurts: local access gets the socket's full complement of memory channels, while remote traffic is capped by the interconnect and shares it with cache-coherence messages. An all-remote placement typically achieves roughly half the streaming bandwidth of an all-local one, and a cache line ping-ponging between sockets costs far more than either.

Firmware publishes its own opinion in the ACPI SLIT table, which numactl --hardware prints as a distance matrix: 10 for local, commonly 21 for a remote node. Treat those as relative hints, not measurements.

And "one socket" no longer means "one node". AMD EPYC can split a single socket into 1, 2, or 4 nodes via the NPS BIOS setting (NPS1/NPS2/NPS4); Intel's Sub-NUMA Clustering does the same. Cloud instances inherit whatever the hypervisor exposes as vNUMA. Check with numactl --hardware or lscpu before assuming.

First touch, and the trap it sets

Remember first touch decides. Linux's default memory policy doesn't place a page when you allocate it — allocation is just a VMA — it places the page on the node of whichever core takes the first write fault on it. Usually that's the right guess. Here is the case where it is catastrophically wrong:

The classic fix is parallel first touch: spawn the workers first, pin each to the node it will run on, and have each thread memset only the slice it will own. The pages then land on the node that will read them — the same code, the same allocation, a different thread doing the first write.

The second trap needs no mistake on your part at all. The scheduler balances load by migrating threads across cores, and it will happily move a thread to the other socket. The data does not follow. Nothing about your program changed — same allocation, same pointers, same code path — and every access that was local is now a round trip across the interconnect. This is why "we moved to a box with twice the cores and it got slower" is a real and common bug report: more cores meant more sockets, more sockets meant a wider spread, and a workload that fit one node's memory now bounces across two.

It is also miserable to diagnose, because the code path is identical to the fast run — the difference is only stall cycles. perf stat with the remote-DRAM load events, numastat -p <pid>, and /proc/<pid>/numa_maps will show you the per-node split; perf c2c finds the cache lines being fought over, which is the false-sharing variant of the same disease.

Placing memory on purpose

AutoNUMA is the kernel's own counter-move: it periodically strips PTEs to provoke "NUMA hinting" faults, samples which node is actually touching each page, then migrates pages toward threads or threads toward pages. It genuinely helps sloppy workloads. But those sampling faults are real faults, and the migrations are real copies, so it shows up as periodic latency spikes — which is exactly why database and cache tuning guides so often tell you to disable it and place memory explicitly instead. See the kernel's NUMA memory policy docs and numa(7).

One last interaction to keep in mind: transparent huge pages cut page-table and TLB cost dramatically, but they coarsen placement — a 2 MiB page lives entirely on one node, so a hot structure that straddles a huge page can't be split between the threads that use it. Every layer in this handbook trades granularity against overhead, and this is that trade one more time.

Step back and the escalation is a single argument. The pager gave you the illusion that memory is infinite; malloc gave you the illusion that it is fine-grained; mmap gave you the illusion that files are memory; NUMA is the point where the last illusion, that memory is uniform, finally breaks — and it breaks exactly where performance matters most. If you want the kernel-side machinery behind all of it, keep going with Linux internals.

The first practical trap is thrashing: when the working set of active pages is larger than physical RAM, the system spends nearly all its time paging in and out instead of computing — every reference faults, evicts a page that's about to be needed, and faults again. Performance falls off a cliff. The fix is more RAM or fewer concurrent processes; the lesson is that virtual memory's magic ends the moment your hot data doesn't fit, and it ends catastrophically, not gracefully.

Two more that trip up application engineers. Ignoring context-switch cost: spawning thousands of threads doesn't multiply throughput on a few cores — past the core count you just add switching overhead, which is why async I/O and thread pools exist (don't out-thread your CPUs). And assuming memory access is uniform: it isn't — an L1 cache hit is nanoseconds, a RAM hit tens of nanoseconds, a page fault to disk milliseconds, a spread of a million to one. Data-structure and access-pattern choices that respect locality (arrays over linked lists, sequential over random) can dwarf algorithmic constants, because they turn faults into hits. You don't write a scheduler or a pager, but you live inside theirs: understand time-slicing and paging and you'll know why your program is fast, slow, or thrashing. The whole handbook is one line: the OS shares scarce time and space, and your performance is decided by how well your work fits the slices and the frames.