GPUs: the bottleneck is never where you think.

A CPU is a latency processor: a few heroic cores with deep caches and branch predictors, built to finish one thread as fast as possible. A GPU is a throughput processor: ~100+ streaming multiprocessors (SMs) hosting tens of thousands of lightweight threads, built to finish all the work per second — no single thread is fast, the swarm is.

The execution model is SIMT — single instruction, multiple threads: threads run in lockstep groups of 32 called warps, one instruction driving 32 data elements. This is why GPUs devour uniform work (every element gets the same treatment — exactly what a matmul is) and why branch divergence hurts: if half a warp takes the if and half takes the else, the hardware runs both paths serially. Deep learning happens to be the most uniform large-scale workload ever invented — the marriage wasn't luck, it was fit.

The compute units can chew through data far faster than HBM can deliver it. So the central discipline of GPU performance is arithmetic intensity: FLOPs per byte moved. Each GPU has a break-even ratio (peak FLOPs ÷ bandwidth ≈ ~300 FLOPs/byte on H100 BF16). Above it, you're compute-bound — the cores are the wall. Below it, memory-bound — the cores idle while bytes crawl. That one ratio — the roofline — explains most "why is this slow" mysteries in AI.

Tensor cores are why the compute side got so tall: dedicated units performing small matrix multiply-accumulates in hardware, with throughput that roughly doubles each precision step down (BF16 → FP8 → FP4 on the newest parts) — the hardware reason low-precision training and serving keep winning. And MFU (model FLOPs utilization) is the honesty metric: real training jobs hitting 35-45% of peak is good; the marketing number assumes a workload nobody runs.

A kernel is a function launched across a grid of thread-blocks — the unit of GPU work. Almost nobody hand-writes them anymore; the stack has layers:

The practical division of labor: use the framework; take the torch.compile win; profile before believing anything; and reach for Triton only when the profiler shows a genuinely unfused hot spot. Writing CUDA is a specialization — reading a profile is table stakes.

Models outgrew single GPUs, so the memory-hierarchy story repeats one level up: NVLink (~900 GB/s GPU-to-GPU inside a node) and InfiniBand (~400 Gb/s between nodes) are to the cluster what HBM is to the chip — the slow layer everything must be tiled around. The primitive is the collective: all-reduce (sum gradients everywhere), all-gather, reduce-scatter, moved by NCCL and overlapped with compute so communication hides behind math.

The three parallelism strategies in one line each: data parallel (copy the model, split the batch, all-reduce gradients), tensor parallel (split individual matrices across GPUs — needs NVLink-class bandwidth), pipeline parallel (split layers into stages — cheap on bandwidth, pays a bubble). Real frontier runs compose all three; that composition is its own handbook, coming next.