GPU Numbers Every AI Engineer Should Know
VRAM, bandwidth, TFLOPs and rental prices for the cards that matter — plus the rules of thumb they imply.
The cards (BF16 dense · memory · bandwidth)
- RTX 4090
- 24 GB · ~1.0 TB/s · 165 TFLOPs · ~$0.5/hr rented
- L4
- 24 GB · 300 GB/s · 121 TFLOPs · ~$0.7/hr — the cheap inference card
- A100 80GB
- 80 GB · 2.0 TB/s · 312 TFLOPs · ~$2.5/hr
- H100 80GB
- 80 GB · 3.35 TB/s · 989 TFLOPs · ~$3.5/hr
- H200
- 141 GB · 4.8 TB/s · 989 TFLOPs — same compute, more memory
- M-series Mac
- 36-192 GB unified · 100-800 GB/s — why laptops run 70B
Rules of thumb
- What fits (FP16)
- 24 GB → 8-13B · 80 GB → 70B tight (INT4) · 2×80 GB → 70B comfy
- Decode tok/s
- ≈
bandwidth ÷ model bytes— 8B INT4 on 4090 ≈ 200 tok/s - Real utilization (MFU)
- training 30-45% of peak; marketing numbers assume sparsity
- Interconnect
- NVLink ~900 GB/s · PCIe 5 ~64 GB/s — sharding across PCIe hurts
- Power
- H100 ~700W · 8×H100 node ~10 kW — datacenter math starts here
Cost anchors
- Fine-tune 8B (QLoRA)
- ~20 GPU-hours on A100 ≈ $50
- Pretrain 8B (Chinchilla)
- 6·8e9·160e9 FLOPs ≈ 22K H100-hours ≈ $80K
- DeepSeek-V3 (671B MoE)
- 2.79M H800-hours ≈ $5.6M — the efficiency benchmark
- Serving 8B
- one L4 ≈ $500/mo ≈ millions of requests — self-host math