Vibe Engines is free, interactive AI engineering & system design — runs in your browser. Pick a learning path
NPU Model-Fit Calculator
An on-device AI planner keyed to the NPU, not the GPU. Enter your device’s TOPS (e.g. a 75-TOPS Snapdragon X2) and unified RAM, pick a model size and quantization, and see two things the VRAM-fit calculators miss: whether the weights fit in memory, and how fast the NPU can prefill a prompt — the compute-bound step that TOPS actually governs. Get an escalate-to-cloud verdict when a prompt is too long to stay responsive locally.
Fit is a RAM question; prefill speed is a TOPS question. Decode (per-token generation) is memory-bandwidth-bound and deliberately out of scope — these are order-of-magnitude planning estimates, not runtime benchmarks.
Running an LLM on a laptop or phone NPU comes down to two separate questions: do the weights fit in memory, and can the NPU get through the prompt fast enough. This calculator answers both. Memory decides whether the model loads at all; the NPU's TOPS rating decides how long a user waits for the first token.
What it computes
weights_GB = params_B × bytes_per_param
need_GB = weights_GB × 1.2 + 1.5
fits = need_GB ≤ RAM_GB
prefill = TOPS × 10^12 × util ÷ (2 × params_B × 10^9) tokens/s
TTFT = prompt_tokens ÷ prefill seconds
- bytes_per_param is 2 for FP16, 1 for Q8 and 0.5 for Q4.
- 1.2 and 1.5 add 20% headroom for the KV cache and activations, plus a 1.5 GB floor for the operating system and runtime.
- TOPS is the NPU's INT8 rating in trillions of operations per second, and util is the fraction of it you actually sustain (default 30%, floored at 2%).
- 2 × params is the work per prompt token: about one multiply and one add for every parameter.
The verdict: if it does not fit, requantize to 4-bit when Q4 would fit, otherwise escalate to the cloud. If it fits, TTFT above 10 s means escalate long prompts, above 3 s is usable but noticeable, and 3 s or less runs comfortably on-device.
A worked example
The defaults are a 75 TOPS NPU, 16 GB of RAM, a 7B model at Q4, a 4,096-token prompt and 30% utilization.
- Weights:
7 × 0.5 = 3.5 GB. - Need:
3.5 × 1.2 + 1.5 = 5.7 GB, under 16 GB, so it fits. - Prefill:
75 × 10^12 × 0.30 ÷ (2 × 7 × 10^9) = 22.5 × 10^12 ÷ (14 × 10^9)≈ 1,607 tok/s. - TTFT:
4,096 ÷ 1,607≈ 2.5 s, which is 3 s or less, so the verdict is Runs comfortably on-device.
| Model at Q4 | Need | Prefill | TTFT, 4,096 tokens | Verdict on 16 GB |
|---|---|---|---|---|
| 3B | 3.3 GB | 3,750 tok/s | 1.1 s | Runs comfortably |
| 7B | 5.7 GB | 1,607 tok/s | 2.5 s | Runs comfortably |
| 13B | 9.3 GB | 865 tok/s | 4.7 s | Usable, TTFT noticeable |
| 30B | 19.5 GB | 375 tok/s | 10.9 s | Too big, escalate to cloud |
Switching the 7B model to FP16 raises the need to 18.3 GB, which does not fit, so the verdict becomes Requantize to 4-bit. Q8 needs 9.9 GB and fits, with the same prefill speed.
Assumptions and limits
- Prefill only. Decode is limited by memory bandwidth, not TOPS, and is out of scope.
- Attention is ignored. The 2 × params rule leaves out attention work, which grows with prompt length, so very long prompts will be slower than shown.
- One TOPS figure for every precision. Prefill speed does not change with quantization here, while real NPUs usually run higher precisions below their INT8 peak.
- Utilization is a guess. Runtime, kernels and memory movement decide it, and it moves the result in direct proportion.
- Memory is approximate. 4-bit formats also store scales, long contexts need more KV cache than 20%, and other apps share unified RAM.
How to use the result
- Size the model to the device. Choose the largest size and quantization that fits with room to spare, then check TTFT at your typical prompt length.
- Turn TTFT into a routing rule. At the defaults, a 7B model stays under 3 s up to about 4,800 prompt tokens and crosses 10 s near 16,000. Route anything longer to the cloud.
- Requantize before you give up. A requantize verdict means 4-bit would fit on the same device.
- Calibrate with one measurement. Time a real prefill on the target device and adjust utilization until the tool matches it.
When to use it, and when not
Use it when you choose which model size and quantization to ship for a class of device, and when you need a rule for which prompts stay local and which go to the cloud. Both gates are simple enough to run in your head once you have seen them, which makes the tool useful in a design review where nobody has the device to hand.
Do not use it to promise generation speed: decode depends on memory bandwidth, which the tool never asks for. Do not use it to rank NPUs from different vendors on their spec sheets alone either. A TOPS figure quoted at lower precision or with sparsity is not the dense INT8 number this formula assumes. Once you have the hardware, one timed prefill is worth more than any estimate.
Trade-offs and limits
Both gates are deliberately coarse. This is roughly how far each simplification can move the answer:
| Simplification | How far off it can be |
|---|---|
| Attention left out of the prefill work | For a 7B model with 32 layers and a hidden size of 4,096, attention adds at most about 8% to the work at 4,096 tokens and about 31% at 16,384 (an upper bound that counts every token as seeing the whole prompt) |
| Utilization fixed at 30% | Proportional: at 15% the default TTFT doubles from 2.5 s to 5.1 s; at 100% it would be 0.76 s |
| 20% memory headroom | For an 8B model at Q4 the headroom is 0.8 GB, while a 16,384-token FP16 cache for Llama 3.1 8B is 2 GB, so long contexts can need 1.2 GB more than shown |
| Speed independent of quantization | FP16 is shown as fast as Q4, though NPUs usually run higher precisions below their INT8 peak |
The errors do not cancel. Long prompts make attention, the KV cache and the TTFT all grow together, so the estimate is weakest exactly where the verdict is most likely to say escalate.
Common mistakes
- Counting all the RAM as free. Unified memory is shared with the operating system, the GPU and every other open app. The 1.5 GB floor covers the runtime, not a browser with forty tabs.
- Expecting TOPS to speed up generation. A faster NPU cuts the wait for the first token. The tokens after it are paced by how fast weights and cache stream from memory.
- Testing with a short prompt. TTFT grows in line with prompt length. The default 7B model answers a 512-token prompt in 0.3 s and a 16,384-token one in 10.2 s, so a demo with a one-line question says little about a document chat.
- Assuming peak utilization. Setting it to 100% makes the default look 3.3× faster than at 30%. Few runtimes sustain anything close on real kernels.
- Sizing by the fit gate alone. On a 32 GB laptop a 30B model at Q4 fits with room to spare, at 19.5 GB, yet a 4,096-token prompt takes 10.9 s to prefill at the defaults. The largest model that loads is often too slow to be the default.
Check yourself
An 8B model at Q8 on the default 75 TOPS, 16 GB device with a 4,096-token prompt: what is the verdict?
Weights 8 × 1 = 8 GB, need 8 × 1.2 + 1.5 = 11.1 GB, so it fits. Prefill 22.5 × 10^12 ÷ (16 × 10^9) ≈ 1,406 tok/s, TTFT ≈ 2.9 s. Just under 3 s, so it runs comfortably on-device, with almost no margin.
A 13B model at FP16 on a 32 GB laptop: what does the tool say?
Need 26 × 1.2 + 1.5 = 32.7 GB, just over 32, so it does not fit. At Q4 it needs 9.3 GB, which fits, so the verdict is requantize to 4-bit.
A 70B model at Q4 on a 64 GB machine at the default TOPS and utilization: does it fit, and how long is TTFT for 4,096 tokens?
Need 35 × 1.2 + 1.5 = 43.5 GB, so it fits. Prefill 22.5 × 10^12 ÷ (140 × 10^9) ≈ 161 tok/s, so TTFT ≈ 25.5 s: fits, but escalate long prompts.
What utilization would bring a 13B Q4 model under 3 s for 4,096 tokens at 75 TOPS?
It needs 4,096 ÷ 3 ≈ 1,365 tok/s. Utilization = 1,365 × 2 × 13 × 10^9 ÷ (75 × 10^12) ≈ 47%, well above the 30% default. A shorter prompt or a smaller model is the more realistic fix.
How it works
- Fit gate: weights = params × bytes per param; need = weights × 1.2 + 1.5 GB.
- Speed gate: prefill tok/s = TOPS × utilization ÷ (2 × params); TTFT = prompt ÷ prefill rate.
- Decode is memory-bandwidth-bound and scoped out on purpose.
- Verdict: run on-device, requantize to 4-bit, or escalate long prompts to the cloud.
Frequently asked questions
Will a 7B model run on a 16 GB laptop with an NPU?
At 4-bit, comfortably, according to this model. A 7B model at Q4 has 3.5 GB of weights and needs about 5.7 GB once headroom for the KV cache and the operating system is added, well inside 16 GB. A 75 TOPS NPU at 30% utilization prefills it at about 1,607 tokens per second, so a 4,096-token prompt reaches the first token in about 2.5 seconds. At FP16 the same model needs 18.3 GB and does not fit, so the verdict becomes requantize to 4-bit.
How much RAM does a local LLM need?
The calculator multiplies the parameter count by the bytes per parameter (2 for FP16, 1 for Q8, 0.5 for Q4), then adds 20% headroom for the KV cache and activations and a 1.5 GB floor for the operating system and runtime. At 4-bit that gives 3.3 GB for a 3B model, 5.7 GB for 7B, 9.3 GB for 13B, 19.5 GB for 30B and 43.5 GB for 70B. Remember that other apps share unified memory too.
How do you estimate prefill speed from NPU TOPS?
Prefill, processing the prompt before the first output token, is compute-bound at roughly two operations per parameter per token. So prefill tokens per second is TOPS times 10 to the 12th times utilization, divided by 2 times the parameter count. Utilization is the share of peak compute you actually get, and the default is 30%. Time to first token is then the prompt length divided by that rate. Treat the result as a planning estimate, not a benchmark.
Why does this calculator lead with TOPS instead of VRAM?
Most NPUs share unified system RAM rather than having their own VRAM, and memory only answers whether the weights fit. TOPS answers the other question: how long the user waits before the first token on a long prompt. A model can fit easily and still feel slow. On the default device, a 13B model at 4-bit fits in 9.3 GB but takes about 4.7 seconds to prefill 4,096 tokens, and a 16,384-token prompt on a 7B model takes about 10.2 seconds.
Does the calculator estimate decode (generation) speed?
No, and on purpose. Decode, generating each output token after the first, is limited by memory bandwidth, because every new token has to read the weights and the KV cache from memory. A high TOPS rating does not make that faster. This tool scopes TOPS to prefill, where compute genuinely binds, and treats fit as a RAM question. For tokens per second during generation you need a bandwidth-based estimate instead.
When should a local AI app escalate a prompt to the cloud?
The verdict follows fixed rules. If the model does not fit in RAM at the chosen quantization, it suggests requantizing to 4-bit when that would fit, and escalating to the cloud when even 4-bit would not. If it fits, a time to first token above 10 seconds means escalate long prompts, 3 to 10 seconds is usable but noticeable, and 3 seconds or less runs comfortably. For a 7B model at the defaults, the 10-second line falls near 16,000 prompt tokens.