mmap: How to Read a 50GB File Without 50GB of RAM
How memory-mapped files let a process address a file directly instead of copying it into a buffer — mapping loads nothing up front, and the OS pulls in only the 4KB page you actually touch, via a page fault and the shared page cache.
mmap maps a file into your address space so it behaves like an array, but nothing loads until you touch a byte — the CPU raises a page fault and the OS pulls in just that page from disk (demand paging). Multiple processes mapping the same file share the same physical pages via the OS page cache, which is how a shared library loads once for every app and how databases operate on files bigger than RAM.
Transcript
You open a fifty-gigabyte file and read a single byte near the very end — instantly, using almost no memory. No read loop, no huge buffer. How? You never read the file. You mapped it — so it acts like a giant array — and the OS loads only the pages you touch, like map tiles.
Normally, to read a file you call read, which copies bytes from disk into a buffer you allocate. To reach a byte far in, you seek and read, chunk by chunk. And to hold a fifty-gig file in memory that way, you'd need fifty gigs of RAM. You're stuck managing buffers and copies by hand.
mmap flips it. You ask the OS to map the file into your address space — and the file appears as a range of memory, an array you index into. No read calls, no copying into your own buffer. The file and your memory become the same thing. But here's the trick: so far, nothing has actually loaded.
The OS set up the address range but loaded zero bytes. The instant you touch a byte, the CPU raises a page fault — this page isn't here yet. The OS catches it, loads just that one four-kilobyte page from disk, and your access continues as if nothing happened. You only load the pages you touch.
And because the mapping is backed by the OS's page cache, many programs can map the SAME file and share the exact same pages in RAM — one copy for everyone. That's how a shared library loads once for every app, and how databases run over files far bigger than memory. Writes can flush back to disk.
So mmap is a deal with the OS: treat this file as memory, and I'll fetch each page only when you reach for it — lazily, on demand. So next time you're about to read a giant file, ask if you should map it instead. Don't load the whole thing — touch only the page you need.