Fine-Tuning Cost Estimator
Pick a method (QLoRA / LoRA / full AdamW), model size, dataset and GPU, and get the napkin answer: training memory, how many cards you need to fit, GPU-hours from the 6·P·tokens FLOPs rule, wall-clock time and the rental bill — with the QLoRA-vs-full memory gap made painfully visible.
The 6·P·T rule
Training compute has a famous napkin law: one training pass costs about 6 FLOPs per parameter per token — 2 for the forward pass, 4 for the backward. Multiply by your dataset (× epochs), divide by what your GPU actually delivers (its dense BF16 TFLOPs times a realistic ~35% utilization), and you have GPU-hours before you have rented anything. It is the same arithmetic labs use to budget frontier runs, scaled down to your weekend project.
Memory picks your hardware; FLOPs pick your bill
The two costs are separable, and people constantly confuse them. Memory decides which GPUs you can use: full AdamW fine-tuning holds ~16 bytes/param (weights + gradients + two optimizer moments), LoRA drops that to ~2.5 by freezing the base in 16-bit, QLoRA to ~0.75 by freezing it in 4-bit. FLOPs decide how long you run: nearly identical across all three methods, because every method still runs the full forward and backward through the base model.
The QLoRA punchline
Slide the model to 70B and flip between methods: full fine-tuning wants ~1,120 GB of training memory — a 16-GPU cluster with all its parallelism pain — while QLoRA wants ~52 GB, one big card or two consumer ones. Same dataset, same epochs, nearly the same GPU-hours. The revolution wasn't making fine-tuning faster; it was making the hardware reachable.
Where the estimate lies to you
Napkin math has edges: multi-GPU runs lose time to communication (add 10–30%), tiny batch sizes waste utilization, very long sequences inflate activation memory beyond the bytes/param model, and data pipelines stall GPUs in ways no formula predicts. Treat the output as a ±40% planning number — and remember the real budget line is usually the five failed runs before the one that works.
How it works
- GPU-hours = 6 · params · tokens ÷ (TFLOPs × 35% utilization).
- Memory: full ≈ 16 B/param, LoRA ≈ 2.5, QLoRA ≈ 0.75.
- GPUs-to-fit uses 85% usable VRAM; parallelism overhead not included.
- Prices are typical cloud rental rates — swap in your own mentally.
Frequently asked questions
How are GPU-hours estimated?
From the standard training-compute rule: FLOPs ≈ 6 × parameters × training tokens (forward + backward), divided by the GPU’s dense BF16 throughput at ~35% realized utilization. It is napkin math — real runs vary with sequence length, parallelism efficiency and data pipeline stalls — but it lands within ±40% of practice.
Why does QLoRA need so much less memory than full fine-tuning?
Full fine-tuning with AdamW holds weights, gradients and two optimizer moments — roughly 16 bytes per parameter. QLoRA freezes the base model in 4-bit (0.5 bytes/param) and trains only tiny 16-bit adapters, landing near 0.75 bytes/param overall. Same FLOPs, ~20× less memory: that is why a 70B full fine-tune wants a cluster while QLoRA wants two cards.
Does LoRA/QLoRA train faster than full fine-tuning?
Not meaningfully — the forward and backward passes still run through the whole base model, so FLOPs are similar. What parameter-efficient methods buy is memory (fewer, cheaper GPUs) and tiny artifacts, not fewer training operations. The cost win comes from renting less hardware, not fewer hours.
How many epochs should I plan for?
Instruction fine-tunes typically converge in 1–3 epochs; more usually overfits small datasets. If you are budgeting, 3 epochs over your dataset is the honest default — and if quality is not there by then, the fix is almost always better data, not more passes.