GPU Numbers Every AI Engineer Should Know

VRAM, bandwidth, TFLOPs and rental prices for the cards that matter — plus the rules of thumb they imply.

Three numbers describe a GPU for AI work: memory capacity (what fits), memory bandwidth (how fast it generates tokens) and tensor-core FLOPs (how fast it trains and processes prompts). The rows quote dense BF16 throughput, the honest figure; spec sheets often headline sparse or FP8 numbers that are 2× or 4× larger. Prices are rough on-demand rental ranges that move quickly, so treat them as orders of magnitude.

The rules of thumb turn these figures into the questions you actually ask: will it fit, how fast will it run, and what will it cost.

The cards (BF16 dense · memory · bandwidth)

RTX 4090: 24 GB · ~1.0 TB/s · 165 TFLOPs · ~$0.5/hr rented
RTX 5090: 32 GB GDDR7 · ~1.8 TB/s — the 2025 consumer flagship
RTX 3090: 24 GB · ~0.94 TB/s — the used-market 24 GB card
T4: 16 GB · 320 GB/s · 65 TFLOPs FP16 (no BF16) — the free Colab card
L4: 24 GB · 300 GB/s · 121 TFLOPs · ~$0.7/hr — the cheap inference card
A10: 24 GB · 600 GB/s · 125 TFLOPs; AWS g5 instances use the A10G variant
V100: 16 or 32 GB · 900 GB/s · 125 TFLOPs FP16 (no BF16) — the 2017 generation
A100 80GB: 80 GB · 2.0 TB/s · 312 TFLOPs · roughly $1.5–2.5/hr
H100 80GB: 80 GB · 3.35 TB/s · 989 TFLOPs · roughly $2–4/hr, more on hyperscalers
H200: 141 GB · 4.8 TB/s · 989 TFLOPs — same compute, more memory
B200: ~180 GB · ~8 TB/s · ~2.25 PFLOPs BF16 · ~1 kW per GPU
MI300X: 192 GB · 5.3 TB/s · ~1.3 PFLOPs BF16 — AMD’s H100 rival
M-series Mac: unified memory up to 128 GB (laptops) or 512 GB (Mac Studio) · ~100–800 GB/s

Number formats

FP32: 4 bytes — master weights and optimizer states in mixed-precision training
BF16 · FP16: 2 bytes; BF16 keeps FP32’s range (no loss scaling), FP16 is more precise but overflows
TF32: A100+ tensor-core mode for FP32 math: FP32 range, 10-bit mantissa, much faster
FP8: 1 byte on H100 and newer, 2× BF16 throughput; E4M3 forward, E5M2 for gradients
INT8 · INT4: 1 and 0.5 bytes per weight for inference; 4-bit cuts memory 4× versus FP16
FP4: Blackwell tensor-core format: another 2× over FP8 for inference

Rules of thumb

What fits: 24 GB: 8B FP16 or ~30B 4-bit · 80 GB: 70B INT8, tight · 2×80 GB: 70B FP16
Decode tok/s: ≈ bandwidth ÷ model bytes — 8B INT4 on 4090 ≈ 200 tok/s
Prefill throughput: compute-bound: ≈ FLOPs × MFU ÷ (2 × params) — tens of thousands of tok/s for 8B on H100
Quantization speedup: decode reads fewer bytes per token, so 4-bit can be up to ~4× faster than FP16
Ridge point: peak FLOPs ÷ bandwidth ≈ 300 FLOPs per byte on H100; below that, you are memory-bound
Real utilization (MFU): training 30-45% of peak; marketing numbers assume sparsity
Training memory: ≈ 16 bytes per param with AdamW mixed precision — 7B full fine-tune ≈ 112 GB plus activations
QLoRA reach: fine-tune a 65B model on one 48 GB GPU — the QLoRA paper’s headline result
Overhead: budget 10–20% above weights + KV cache for activations, CUDA context and fragmentation
Power: H100 ~700W · 8×H100 node ~10 kW — datacenter math starts here

Interconnect & clusters

NVLink (H100): ~900 GB/s per GPU, both directions — keep tensor parallelism inside one NVLink domain
NVLink 5 (B200): ~1.8 TB/s per GPU, twice Hopper’s
PCIe 5.0 x16: ~64 GB/s each way (~128 GB/s total), about 7× below H100 NVLink
InfiniBand NDR: 400 Gb/s ≈ 50 GB/s per port; H100 training nodes usually carry one per GPU
8×H100 node: 640 GB of HBM, NVSwitch all-to-all, about 10 kW
GB200 NVL72: 72 Blackwell GPUs in one NVLink domain — a rack that behaves like one giant GPU

Cost anchors

Fine-tune 8B (QLoRA): ~20 GPU-hours on A100 ≈ $50
Pretrain 8B (Chinchilla): 6·8e9·160e9 FLOPs ≈ 6K H100-hours at 35% MFU ≈ $21K at ~$3.5/hr
Llama 3 8B (actual): about 1.3M H100-hours for 15T tokens (Meta model card)
Llama 3.1 405B: about 30.8M H100-hours (Meta model card) — tens of millions of dollars
DeepSeek-V3 (671B MoE): 2.79M H800-hours ≈ $5.6M at $2/hr — the final training run only
Serving 8B: one L4 ≈ $500/mo ≈ millions of requests — self-host math

Inside the GPU

SMs: H100 SXM 132 · A100 108 — the units every kernel is spread across
L2 cache: H100 50 MB · A100 40 MB — tiny next to HBM, so data reuse matters
Shared memory: up to 228 KB per SM on H100; FlashAttention tiles attention to stay in it
SRAM vs HBM: on-chip SRAM is roughly an order of magnitude faster — fast kernels move fewer bytes
Tensor cores: matrix units; peak speed needs BF16/FP8 data and dimensions in multiples of 8 or 16

Recipes

Pick a GPU for serving a 70B model

Weights at 4-bit:  70e9 × 0.5 B                  ≈ 35 GB
KV cache:          8K context × 320 KiB per token  ≈ 2.5 GiB per sequence
16 concurrent:     16 × 2.5 GiB                    ≈ 43 GB
Overhead:          +15%                            ≈ 12 GB
Total:                                             ≈ 90 GB → one H200, or two 80 GB cards

With long contexts and many users the KV cache, not the weights, decides the card; halving the context or the concurrency fits a single 80 GB GPU.

Estimate decode speed for an 8B model on an H100

Bandwidth:      3.35 TB/s
FP16 weights:   16 GB   → 3,350 ÷ 16  ≈ 210 tokens/s per sequence, at best
INT4 weights:   ~4.5 GB → 3,350 ÷ 4.5 ≈ 740 tokens/s per sequence, at best
Batch of 32:    each weight read serves 32 sequences → far higher total throughput

The ceiling assumes the weights are read exactly once per token; KV-cache reads and kernel overheads keep real numbers below it.

Fine-tuning memory for a 7B model

Full fine-tune:  7e9 × 16 B   ≈ 112 GB + activations → 2–4× 80 GB GPUs with sharding
LoRA:            7e9 × 2 B    ≈ 14 GB base + small adapters → one 24 GB card
QLoRA:           7e9 × 0.5 B  ≈ 3.5 GB base → a 16 GB T4-class card works

The frozen base model dominates LoRA memory, which is why quantizing it, the Q in QLoRA, is the step that moves fine-tuning onto consumer GPUs.

Training cost from first principles

Model and data:   8e9 params, 160e9 tokens (Chinchilla 20:1)
FLOPs:            6 × 8e9 × 160e9         = 7.7e21
Effective H100:   989e12 × 0.35 MFU       = 3.5e14 FLOP/s
GPU-hours:        7.7e21 ÷ 3.5e14 ÷ 3,600 ≈ 6,100
Cost at ~$3.5/hr:                         ≈ $21,000

MFU is the uncertain input: at 50% the bill falls by almost a third, and at 25% it rises by 40%.

Questions about GPU Numbers Every AI Engineer Should Know Cheat Sheet

What GPU do I need to run a 70B model locally?

At 4-bit quantization the weights take about 35 to 40 GB, so you need roughly 48 GB of GPU memory once the KV cache and overhead are included: two 24 GB cards such as RTX 3090s or 4090s, a single 48 GB workstation card, or an Apple silicon Mac with 64 GB or more of unified memory. At 8-bit it needs about 70 GB plus cache, which fits an 80 GB A100 or H100 tightly. Full FP16 needs around 140 GB, which means two 80 GB data-center GPUs.

What is the difference between the H100 and the A100?

The H100, NVIDIA’s Hopper generation from 2022, delivers roughly three times the A100’s dense BF16 throughput, about 989 versus 312 teraFLOPs, adds FP8 tensor cores that double that again, and raises memory bandwidth from about 2.0 to 3.35 TB/s in the 80 GB SXM versions. In practice training runs two to three times faster, and token generation improves by about the bandwidth ratio, around 1.6 times. The A100 is much cheaper to rent and remains a sensible choice for inference and smaller fine-tunes.

Why does memory bandwidth matter more than FLOPs for LLM inference?

Because generating text is a sequence of small steps, each of which reads every weight from memory to produce one token per sequence while doing only about two floating-point operations per weight. GPUs compute far faster than they can fetch bytes, so during decoding they wait on memory, and tokens per second scale with bandwidth divided by model size. That is why an H200, with the same compute as an H100 but more bandwidth, generates faster. FLOPs matter more for training and for processing long prompts, which are compute-bound.

What is MFU in GPU training?

Model FLOPs utilization is the share of a GPU’s peak throughput that a training run actually spends on the model’s useful arithmetic: the model’s FLOPs per second divided by the hardware’s dense peak. Communication, memory stalls, data loading and recomputation all pull it down. Large, well-tuned runs typically report 30 to 50 percent, and a run in the teens usually signals a bottleneck worth fixing. MFU is the input that turns a FLOPs estimate into GPU-hours, so it dominates the uncertainty in any training cost estimate.

Go deeper

GPU VRAM Calculator →Fine-Tuning Cost Estimator →On-Device AI handbook →LLM Serving handbook →