Vibe Engines is free, interactive AI engineering & system design — runs in your browser. Pick a learning path
NPU Model-Fit Calculator
An on-device AI planner keyed to the NPU, not the GPU. Enter your device’s TOPS (e.g. a 75-TOPS Snapdragon X2) and unified RAM, pick a model size and quantization, and see two things the VRAM-fit calculators miss: whether the weights fit in memory, and how fast the NPU can prefill a prompt — the compute-bound step that TOPS actually governs. Get an escalate-to-cloud verdict when a prompt is too long to stay responsive locally.
Fit is a RAM question; prefill speed is a TOPS question. Decode (per-token generation) is memory-bandwidth-bound and deliberately out of scope — these are order-of-magnitude planning estimates, not runtime benchmarks.
Running an LLM on a laptop or phone NPU comes down to two separate questions: do the weights fit in memory, and can the NPU get through the prompt fast enough. This calculator answers both. Memory decides whether the model loads at all; the NPU's TOPS rating decides how long a user waits for the first token.
What it computes
weights_GB = params_B × bytes_per_param
need_GB = weights_GB × 1.2 + 1.5
fits = need_GB ≤ RAM_GB
prefill = TOPS × 10^12 × util ÷ (2 × params_B × 10^9) tokens/s
TTFT = prompt_tokens ÷ prefill seconds- bytes_per_param is 2 for FP16, 1 for Q8 and 0.5 for Q4.
- 1.2 and 1.5 add 20% headroom for the KV cache and activations, plus a 1.5 GB floor for the operating system and runtime.
- TOPS is the NPU's INT8 rating in trillions of operations per second, and util is the fraction of it you actually sustain (default 30%, floored at 2%).
- 2 × params is the work per prompt token: about one multiply and one add for every parameter.
The verdict: if it does not fit, requantize to 4-bit when Q4 would fit, otherwise escalate to the cloud. If it fits, TTFT above 10 s means escalate long prompts, above 3 s is usable but noticeable, and 3 s or less runs comfortably on-device.
A worked example
The defaults are a 75 TOPS NPU, 16 GB of RAM, a 7B model at Q4, a 4,096-token prompt and 30% utilization.
- Weights:
7 × 0.5 = 3.5 GB. - Need:
3.5 × 1.2 + 1.5 = 5.7 GB, under 16 GB, so it fits. - Prefill:
75 × 10^12 × 0.30 ÷ (2 × 7 × 10^9) = 22.5 × 10^12 ÷ (14 × 10^9)≈ 1,607 tok/s. - TTFT:
4,096 ÷ 1,607≈ 2.5 s, which is 3 s or less, so the verdict is Runs comfortably on-device.
| Model at Q4 | Need | Prefill | TTFT, 4,096 tokens | Verdict on 16 GB |
|---|---|---|---|---|
| 3B | 3.3 GB | 3,750 tok/s | 1.1 s | Runs comfortably |
| 7B | 5.7 GB | 1,607 tok/s | 2.5 s | Runs comfortably |
| 13B | 9.3 GB | 865 tok/s | 4.7 s | Usable, TTFT noticeable |
| 30B | 19.5 GB | 375 tok/s | 10.9 s | Too big, escalate to cloud |
Switching the 7B model to FP16 raises the need to 18.3 GB, which does not fit, so the verdict becomes Requantize to 4-bit. Q8 needs 9.9 GB and fits, with the same prefill speed.
Assumptions and limits
- Prefill only. Decode is limited by memory bandwidth, not TOPS, and is out of scope.
- Attention is ignored. The 2 × params rule leaves out attention work, which grows with prompt length, so very long prompts will be slower than shown.
- One TOPS figure for every precision. Prefill speed does not change with quantization here, while real NPUs usually run higher precisions below their INT8 peak.
- Utilization is a guess. Runtime, kernels and memory movement decide it, and it moves the result in direct proportion.
- Memory is approximate. 4-bit formats also store scales, long contexts need more KV cache than 20%, and other apps share unified RAM.
How to use the result
- Size the model to the device. Choose the largest size and quantization that fits with room to spare, then check TTFT at your typical prompt length.
- Turn TTFT into a routing rule. At the defaults, a 7B model stays under 3 s up to about 4,800 prompt tokens and crosses 10 s near 16,000. Route anything longer to the cloud.
- Requantize before you give up. A requantize verdict means 4-bit would fit on the same device.
- Calibrate with one measurement. Time a real prefill on the target device and adjust utilization until the tool matches it.
How it works
- Fit gate: weights = params × bytes per param; need = weights × 1.2 + 1.5 GB.
- Speed gate: prefill tok/s = TOPS × utilization ÷ (2 × params); TTFT = prompt ÷ prefill rate.
- Decode is memory-bandwidth-bound and scoped out on purpose.
- Verdict: run on-device, requantize to 4-bit, or escalate long prompts to the cloud.
Frequently asked questions
Will a 7B model run on a 16 GB laptop with an NPU?
At 4-bit, comfortably, according to this model. A 7B model at Q4 has 3.5 GB of weights and needs about 5.7 GB once headroom for the KV cache and the operating system is added, well inside 16 GB. A 75 TOPS NPU at 30% utilization prefills it at about 1,607 tokens per second, so a 4,096-token prompt reaches the first token in about 2.5 seconds. At FP16 the same model needs 18.3 GB and does not fit, so the verdict becomes requantize to 4-bit.
How much RAM does a local LLM need?
The calculator multiplies the parameter count by the bytes per parameter (2 for FP16, 1 for Q8, 0.5 for Q4), then adds 20% headroom for the KV cache and activations and a 1.5 GB floor for the operating system and runtime. At 4-bit that gives 3.3 GB for a 3B model, 5.7 GB for 7B, 9.3 GB for 13B, 19.5 GB for 30B and 43.5 GB for 70B. Remember that other apps share unified memory too.
How do you estimate prefill speed from NPU TOPS?
Prefill, processing the prompt before the first output token, is compute-bound at roughly two operations per parameter per token. So prefill tokens per second is TOPS times 10 to the 12th times utilization, divided by 2 times the parameter count. Utilization is the share of peak compute you actually get, and the default is 30%. Time to first token is then the prompt length divided by that rate. Treat the result as a planning estimate, not a benchmark.
Why does this calculator lead with TOPS instead of VRAM?
Most NPUs share unified system RAM rather than having their own VRAM, and memory only answers whether the weights fit. TOPS answers the other question: how long the user waits before the first token on a long prompt. A model can fit easily and still feel slow. On the default device, a 13B model at 4-bit fits in 9.3 GB but takes about 4.7 seconds to prefill 4,096 tokens, and a 16,384-token prompt on a 7B model takes about 10.2 seconds.
Does the calculator estimate decode (generation) speed?
No, and on purpose. Decode, generating each output token after the first, is limited by memory bandwidth, because every new token has to read the weights and the KV cache from memory. A high TOPS rating does not make that faster. This tool scopes TOPS to prefill, where compute genuinely binds, and treats fit as a RAM question. For tokens per second during generation you need a bandwidth-based estimate instead.
When should a local AI app escalate a prompt to the cloud?
The verdict follows fixed rules. If the model does not fit in RAM at the chosen quantization, it suggests requantizing to 4-bit when that would fit, and escalating to the cloud when even 4-bit would not. If it fits, a time to first token above 10 seconds means escalate long prompts, 3 to 10 seconds is usable but noticeable, and 3 seconds or less runs comfortably. For a 7B model at the defaults, the 10-second line falls near 16,000 prompt tokens.