Run the model
on your machine.

Running an LLM locally is appealing for concrete reasons: privacy (your data never leaves the machine), no per-token cost, offline operation, and full control over the model and its version. The tooling has gotten easy — with ollama you pull and run a model in one command. So the question is rarely "how do I run it?" but "can I run it — will it fit?"

And "fit" is almost entirely about memory. The model's weights have to live in your GPU's VRAM (fast) or your system RAM (slower, CPU inference), alongside the KV cache and some overhead. Compute matters for speed, but memory is the hard wall: if the weights don't fit, the model simply won't load. The good news is that memory is trivially predictable — there's a single formula, and once you know it, hardware planning for local LLMs becomes arithmetic you can do in your head.

Every weight in a model is a number stored in some number of bits. So the memory for the weights is just: how many weights, times how many bits each, converted to bytes. That's the whole equation — memory = params × bits / 8 — and it tells you nearly everything about what fits.

Work an example. A 7-billion-parameter model at 16-bit (half precision, the usual training format) is 7×10⁹ × 16 / 8 = 14 GB just for weights. That won't fit an 8GB consumer GPU. The same model at 4-bit is 7×10⁹ × 4 / 8 = 3.5 GB — comfortable, with room for the KV cache. A 70B model at 4-bit is 35GB, needing a big GPU or CPU offload. Nothing here is mysterious: read off the parameter count and the bit-width, multiply, divide by eight, and you know. The lever that turns "doesn't fit" into "fits" is the bit-width — which is quantization.

Quantization stores each weight in fewer bits. A model trained in 16-bit can be compressed to 8-bit, 4-bit, or even lower, and because memory scales linearly with bit-width, dropping from 16 to 4 bits cuts the footprint to a quarter. That's the difference between a model that won't load and one that runs smoothly on a laptop GPU.

There's a quality cost — cruder weights lose a little precision — but modern 4-bit schemes (like the QLoRA-era NF4 and the GGUF K-quants) are remarkably good, and 4-bit is the default for local inference because the trade is usually excellent. Lower still (3-bit, 2-bit) exists for squeezing the largest models onto limited hardware, with steeper quality loss. The practical workflow: pick the largest model whose 4-bit size fits your memory, and step up the bit-width only if you have room to spare and want the last bit of quality.

The weight-memory formula, with parameters P and bits per weight b:

Linear in both size and bit-width. Halve the bits, halve the memory; double the parameters, double the memory.

To decide if a model fits, compare weight memory (times an overhead factor for the KV cache, activations, and runtime) against your available VRAM:

7B 4-bit ≈ 3.5GB × 1.2 = 4.2GB fits an 8GB GPU; 7B fp16 ≈ 16.8GB doesn't. The runnable version below computes model memory, checks the fit, and shows quantization's savings.

The local ecosystem is a few layers, from raw engine to friendly app:

The practical path: use ollama to run a model with one command when you want convenience; drop to llama.cpp (or vLLM for serving) when you want control over quantization, context length, or throughput. On Apple silicon, unified memory means the "VRAM" is your system RAM, so the same formula applies to a larger pool. Whatever the runtime, the memory equation is identical — it's a property of the model and the bit-width, not the tool. And because local runtimes expose an OpenAI-compatible API, pointing an app at a local model is often just changing the base URL, which pairs neatly with routing cheap/private traffic to a local model and hard queries to a hosted one.

The first surprise is forgetting the KV cache and context: the weight formula gives the floor, but a long context grows the KV cache, which also lives in memory — a model that fits at 2K context can OOM at 32K. Budget headroom (the overhead factor), and remember longer context costs more memory, not just more compute. The second is over-quantizing: 2-bit or aggressive 3-bit can degrade quality sharply, so don't chase the smallest file — 4-bit (Q4_K_M-class) is usually the floor for good quality, and if it doesn't fit, prefer a smaller model at 4-bit over a bigger model at 2-bit.

Two more. CPU-only is slow: it works (llama.cpp runs on CPU), but tokens-per-second drops a lot without a GPU — fine for light use, painful for heavy. And expecting frontier quality: a 7B–14B local model is genuinely useful, but it won't match the largest hosted models on the hardest tasks, so match the model to the job (local for privacy, cost, and offline; hosted for peak capability). The whole discipline reduces to the one equation: know a model's size and bit-width, compute the memory, compare to your hardware, and you'll never be surprised by a model that won't load again.