NPU Model-Fit Calculator
An on-device AI planner keyed to the NPU, not the GPU. Enter your device’s TOPS (e.g. a 75-TOPS Snapdragon X2) and unified RAM, pick a model size and quantization, and see two things the VRAM-fit calculators miss: whether the weights fit in memory, and how fast the NPU can prefill a prompt — the compute-bound step that TOPS actually governs. Get an escalate-to-cloud verdict when a prompt is too long to stay responsive locally.
Fit is a RAM question; prefill speed is a TOPS question. Decode (per-token generation) is memory-bandwidth-bound and deliberately out of scope — these are order-of-magnitude planning estimates, not runtime benchmarks.
The on-device-AI question is not “does the model fit in VRAM” — most NPUs share unified system RAM, and the constraint that actually bites is compute. This calculator keys off the NPU’s TOPS rating, the axis the GPU-VRAM tools skip. It splits the decision into two independent gates. The fit gate is memory: a model’s weights take params × bytes-per-parameter (2 bytes at FP16, 1 at INT8, 0.5 at 4-bit), plus room for the KV cache and the OS, so a 7B model at 4-bit needs roughly 4–6 GB and fits a 16 GB device easily, while the same model at FP16 needs ~18 GB and does not.
The speed gate is where TOPS earns its place. Prefill — processing the prompt before the first token comes out — is compute-bound at about 2 × parameters FLOPs per token, so sustained prefill throughput is roughly TOPS × utilization ÷ (2 × params). A 75-TOPS part prefills a 7B model at a comfortable rate but crawls on a 70B one, and even a small model’s time-to-first-token climbs linearly with prompt length. That is the number that decides whether a long-context request feels instant or makes the user wait, and it is invisible to a fit-only calculator.
The output is a verdict, not just a table: run it locally, drop to a smaller quant, or escalate to the cloud when the prompt is too long to prefill responsively. Decode speed — generating each subsequent token — is a separate, memory-bandwidth-bound story that a TOPS number does not capture, so this tool scopes the TOPS axis to prefill on purpose and is honest that its throughput figures are order-of-magnitude planning estimates, not runtime benchmarks.
How it works
- RAM gate: do the weights (params × bytes-per-param) fit with headroom?
- TOPS gate: prefill tok/s ≈ TOPS × util ÷ (2 × params).
- Decode is bandwidth-bound — scoped out on purpose.
- Verdict: run local, requantize, or escalate long prompts to cloud.
Frequently asked questions
Why lead with TOPS instead of VRAM?
Because on an NPU the two constraints are separate and TOPS is the one the other calculators ignore. RAM decides whether the weights fit at all; TOPS decides how fast the device can process the prompt (prefill), which is the compute-bound step and the main driver of time-to-first-token on a long input. A model can fit comfortably in RAM and still be unusable because a 75-TOPS part takes tens of seconds to prefill an 8K prompt. This tool shows both gates so you plan against the one that actually binds.
How is prefill speed estimated?
Prefill is roughly compute-bound at about 2 × parameters FLOPs per token, so sustained prefill tokens/sec ≈ TOPS × utilization ÷ (2 × params). Utilization is well under 100% in practice (memory movement, attention, kernel overhead), so the tool applies a conservative default you can adjust. The result is an order-of-magnitude planning number, not a benchmark — real throughput depends on the runtime, the model, and the specific NPU.
What about decode (generation) speed?
Decode — generating one token at a time — is memory-bandwidth-bound, not TOPS-bound, so a high-TOPS NPU does not necessarily decode fast. That is a different constraint governed by memory bandwidth and KV-cache size. This tool deliberately scopes the TOPS axis to prefill, where TOPS genuinely binds, and treats fit as the RAM question. For decode throughput you want a bandwidth-based estimate, not a TOPS one.
When should I escalate to the cloud?
When the model does not fit in RAM at any acceptable quant, or when prefill for your typical prompt length pushes time-to-first-token past what your UX tolerates. The tool flags both. A common pattern is a hybrid: run short, latency-sensitive prompts on-device for privacy and offline use, and route long-context or large-model requests to the cloud — the escalation threshold is exactly what this calculator surfaces.