Vibe Engines is free, interactive AI engineering & system design — runs in your browser. Pick a learning path →
GPU vs TPU: flexible, or purpose-built?
A TPU can be more efficient than a GPU at the exact operation that dominates transformer training — matrix multiplication — because it was built around nothing else. That specialization is also exactly why it's the wrong choice the moment your workload steps outside it.
The one distinction that decides everything
A GPU (Nvidia, dominant via the CUDA ecosystem) is a general-purpose massively-parallel processor — originally built for graphics, now the default for nearly all AI workloads. A TPU (Google's Tensor Processing Unit) is an ASIC — application-specific silicon — purpose-built around a systolic array optimized specifically for matrix multiplication, available only on Google Cloud.
Need framework flexibility, custom kernels, or multi-cloud portability? GPU — that's the entire industry's default. Already on GCP running large, well-understood matmul-heavy training at scale? TPU is worth evaluating.
Head to head
| Dimension | GPU | TPU |
|---|---|---|
| Design | General-purpose parallel compute (SIMT cores) | Purpose-built systolic array for matmul |
| Availability | Every cloud, on-prem, consumer cards | Google Cloud only |
| Software ecosystem | CUDA — universal framework support, largest ecosystem | XLA/JAX-first; PyTorch support improved but secondary |
| Best at | Flexible — training, inference, custom kernels, research | Large-scale, matmul-heavy training at Google-scale |
| Interconnect | NVLink (node), InfiniBand (cross-node) | Custom high-bandwidth pod interconnect, built for pod scale |
| Custom architectures / kernels | High flexibility — CUDA lets you write anything | Lower — best within XLA-compilable ops |
| Who actually uses it | Essentially the entire industry outside Google | Primarily Google, plus GCP customers who buy in |
When to use each
Reach for GPU
- Default choice for almost everyone
- Need framework flexibility and custom architectures
- Multi-cloud portability matters
- Ecosystem (pretrained checkpoints, tutorials, tooling) assumes CUDA
Reach for TPU
- Already on GCP
- Training well-understood, transformer-shaped workloads at large scale
- XLA compilation overhead pays off at your scale
- Want Google’s specific pod-scale interconnect economics
Not really a per-run combination — but often a per-stack one
A single training run happens on one or the other, not both. But many teams DO use both across their overall stack — train on TPU pods for cost/scale where the workload fits, while keeping GPU infrastructure for portability, edge deployment, or the parts of the pipeline (research iteration, custom kernels) that don't fit TPU's constraints well. Frameworks like JAX increasingly aim for code portability across both, turning this into more of an infra/cost decision made per workload than a full rewrite each time.
Most organizations outside Google default to GPU for everything, because the ecosystem cost of stepping outside CUDA usually outweighs a TPU's raw efficiency advantage on any single workload.
Why the systolic array is specifically good at matmul
The architectural reason a TPU is efficient for large matrix multiplies is the systolic array: a grid of multiply-accumulate units where data flows through the grid and each loaded weight gets reused across many computations before being evicted, rather than being fetched from memory again for each use. Since matrix multiplication IS mostly repeated multiply-accumulate operations over the same operands, this design minimizes the memory-bandwidth pressure for exactly the operation that dominates transformer training and inference — and memory bandwidth, not raw compute, is very often the actual bottleneck in large-scale training.
A GPU's tensor cores also accelerate matmul well, but the GPU as a whole is a more general SIMT (single-instruction-multiple-thread) processor that wasn't built around ONLY that operation — which is exactly why it remains better for the long tail of non-matmul, custom, research-oriented workloads: novel attention variants, custom loss functions, sparse operations, anything that needs a hand-written CUDA kernel rather than compiling cleanly through XLA. The TPU's specialization is a real efficiency win precisely where the workload matches it, and a real limitation the moment it doesn't.
A TPU trades general-purpose flexibility for efficiency at one specific operation — worth it exactly when your workload IS that operation, at scale, and a liability the moment your workload needs something XLA can't compile cleanly.
A worked scenario: research iteration, then production-scale pretraining
A lab designing a new transformer architecture with experimental custom attention kernels wants GPUs — during this phase, the ability to write and iterate on custom CUDA kernels while the architecture itself is still changing is worth more than raw matmul efficiency, because the workload isn't yet a fixed, well-understood shape a systolic array is optimized for.
Once that architecture is settled and the lab is running repeated, large, well-understood pretraining passes of the now-finalized architecture — a workload that IS mostly large matrix multiplies, at Google-infrastructure scale — TPU pods' cost/performance for that specific, now-stable, matmul-heavy workload becomes genuinely attractive, if the lab is already on GCP. The right accelerator tracks how settled and matmul-shaped the workload has become, not a fixed rule about which is "better."
Flexibility matters most while the workload is still changing. Specialization pays off once the workload has stopped changing and matches exactly what the specialized hardware was built for.
The flip point: sustained utilization
A TPU pod bills for the whole slice whether or not you saturate it, and its systolic array only reaches its low cost-per-FLOP on large, dense, statically-shaped matmuls. So a TPU is cheaper than a GPU only above the utilization where its price-per-effective-FLOP actually beats the GPU’s — peak price advantage means nothing at the FLOPs you leave idle.
Suppose a TPU is ~2× cheaper per peak FLOP (so 0.5× the GPU’s cost per peak FLOP) but your workload only keeps it 40% utilized. Your effective cost is 0.5 / 0.40 = 1.25× a well-fed GPU — you are now paying more. The break-even utilization here is 50%: below it the GPU wins on cost, above it the TPU pulls ahead. The flip point is therefore sustained utilization on supported ops: large-batch, standard-architecture training saturates a TPU and it wins; dynamic shapes, custom kernels, smaller or varied jobs, and the CUDA ecosystem keep you on the GPU. Plug in your real price ratio and measured utilization; the crossover is the utilization at which their product favors the TPU.
The picture: why a systolic array reuses every weight
Cost and trade-offs: where each chip wins or loses time
Read Time as how fast you get a working, efficient run. Read Space as what it costs you in money, compute or lock-in. The utilisation rows extend the page’s own example: the TPU at 0.5× the GPU’s cost per peak FLOP.
| Case | Time | Space | Why |
|---|---|---|---|
| New attention variant or custom op | GPU: write a CUDA or Triton kernel and run it the same day | TPU: it must compile through XLA, or be written in JAX’s Pallas, with far fewer ready-made kernels | This is the research phase in the page’s scenario. The flexibility is worth more than per-FLOP price here. |
| Shapes change every batch | XLA compiles a program per distinct input shape, so a new shape means a recompile | Padding to a fixed shape wastes compute: an average of 1,300 tokens padded to 2,048 is 63.5% useful work | Bucket sequence lengths into a few fixed sizes. GPUs run dynamic shapes eagerly with no compile step. |
| TPU at 25% utilisation | Runs, but the array mostly waits on memory | 0.5 ÷ 0.25 = 2× a well-fed GPU’s cost per useful FLOP | Small jobs, small batches or a slow input pipeline. Utilisation decides it, not list price. |
| TPU at 70% utilisation | Large, dense, static matmuls keep the array full | 0.5 ÷ 0.7 ≈ 0.71× the GPU’s cost | Above the 50% break-even, the specialisation pays off. |
| Serving one request at a time | A batch of 1 means 1 use per weight read (see the diagram) | Both chips run far below peak, and a latency target caps how far you can batch | Inference efficiency comes mostly from batching concurrent requests together. That works on either chip, so the batcher matters more than the chip. |
| One codebase for both | JAX code compiles through XLA on GPUs and TPUs alike | PyTorch-first code needs PyTorch/XLA to reach a TPU, and every custom CUDA kernel needs a replacement | The page’s “per-stack, not per-run” pattern is cheapest when the model is written in JAX from the start. |
| Scaling to thousands of chips | GPU: NVLink inside a node, InfiniBand between nodes | TPU: the pod interconnect is part of the product, built for this scale | At pod scale the network, not the chip, often sets step time. That is part of the TPU’s pitch. |
| Moving clouds or going on-prem | GPU code moves with you | TPU: Google Cloud only, so the code and the contract both stay put | Lock-in is a cost you pay later, at renewal or migration time. |
| Hiring and debugging | Most engineers, profilers and forum answers assume CUDA | A TPU team needs XLA and JAX depth that fewer people have | This is the “ecosystem cost” the page says usually outweighs a per-workload efficiency win. |
Two rows decide most real choices. The shape row: can your workload be made static and dense? The utilisation rows: will you keep the chips busy? If both answers are yes and you are already on GCP, the TPU math works. If either is no, the GPU’s flexibility is cheaper than it looks on a price sheet. The cheap way to find out is a short pilot: port one stable training job, measure step time and utilisation on both chips at the same global batch size, and multiply each by its real hourly price. Count the engineering days the port took as part of the bill. That pilot answers in a week what a spec-sheet comparison cannot answer at all.
Common mistakes
- Comparing price per peak FLOP
A spec sheet shows peak throughput, but you pay for utilisation. The page’s example shows a chip 2× cheaper per peak FLOP costing 1.25× more at 40% busy. Measure model FLOPs utilisation on your own training step before you compare prices: the FLOPs your model needs per step, divided by step time, divided by the chip’s peak. That one ratio, times the hourly price, is the number to compare.
- Benchmarking at the wrong batch size
Reuse in a systolic array equals the batch size (see the diagram). A large-batch training benchmark tells you little about serving one request at a time, and the reverse is true too. Benchmark at the batch sizes you will actually run, for training and for inference separately.
- Feeding XLA dynamic shapes
Variable sequence lengths, data-dependent control flow or a ragged last batch each trigger a recompile. Training then appears to stall for no clear reason. Pad to a few fixed buckets, drop or pad the last batch, and use the compile cache so restarts do not pay again.
- Starving the accelerator
If the host CPU cannot decode, augment and tokenise data as fast as the chip consumes it, the chip waits. A faster accelerator only makes that wait longer. Profile step time against input-pipeline time first. This failure looks like low utilisation on either chip, and it is often blamed on the chip. Preprocess offline where you can, prefetch the next batch while the current one runs, and add host CPU before you add accelerators.
- Porting by changing the device string
With PyTorch/XLA, an op that XLA cannot lower can end up running on the host CPU, and one such op inside the training step can erase the gain. Check the op-level metrics after porting. Budget the engineering time to replace custom CUDA kernels, not only the cloud bill.
Which would you pick?
Three situations. Pick the side you'd actually build — the explanation follows.
A research team is still changing model architecture week to week and writing custom kernels.
Specialisation pays off once a workload has stopped changing and matches what the hardware was built for. While the architecture is still moving, a TPU's efficiency advantage is exactly the thing you cannot rely on — and splitting across both means maintaining two toolchains for a workload that has not settled.
A team already on GCP runs large, stable, matmul-heavy training that has not changed in a year.
Stable, large and matmul-dominated on a cloud that offers them is the narrow case where specialisation genuinely pays. "Always default to GPU" is a good heuristic that stops being useful when the exact conditions it hedges against — changing workloads and ecosystem lock-in — no longer apply. Moving the entire stack is a far larger bet than the accelerator choice.
A benchmark shows a TPU is substantially more efficient than a GPU on your training step. Is that decisive?
The page is blunt that most organisations outside Google default to GPU precisely because a per-workload efficiency win rarely covers the cost of stepping outside CUDA — tooling, libraries, portability and hiring. A benchmark measures the one thing that favours specialisation and none of the things that usually decide it.
Quick answers
GPU or TPU — which should I use?
Default to GPU — it’s available everywhere, has the largest software ecosystem, and supports custom architectures and kernels. Consider TPU only if you’re already on Google Cloud and training a large, well-understood, matmul-heavy workload where XLA compilation overhead pays off.
Why is a TPU efficient specifically for AI training?
Its systolic array architecture reuses each loaded weight across many multiply-accumulate operations before evicting it, minimizing memory-bandwidth pressure for matrix multiplication — the operation that dominates transformer training and inference.
Can I run PyTorch on a TPU?
Yes, PyTorch/XLA support has improved substantially, but TPUs remain a JAX/XLA-first ecosystem. GPU’s CUDA ecosystem still has broader, more mature framework and tooling support across the board.
Is a TPU always faster than a GPU for AI?
No — only for workloads that are large-scale, matmul-heavy, and XLA-compilable. For custom kernels, novel architectures still being iterated on, or workloads outside that shape, a GPU’s general-purpose flexibility wins.
GPU vs TPU: flexible, or purpose-built?
Explore the topic
See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.
More Handbooks
- The Prompting HandbookA friendly, hands-on field guide for everyday humans — learn the CRISP framework, spot bad prompts, practice with real recipes, play a drag-and-drop game, and test yourself with a quiz. No code required.Read →
- The Senior AI Engineer Interview Handbook60 questions across architecture, production incidents, agentic systems, RAG, evals, cost, safety, and leadership — what staff-level AI interviewers actually probe for.Read →
- The LLM Serving HandbookHow to serve large language models fast and cheap — prefill vs decode, the KV cache, continuous batching and PagedAttention (vLLM), quantization, speculative decoding, and the latency-vs-throughput tradeoffs that decide your inference bill.Read →
- The Transformers HandbookThe architecture behind every LLM, built up from scratch — tokens and embeddings, positional encoding, self-attention (query/key/value), multi-head attention, the transformer block (FFN, residuals, layer norm), and how a decoder-only model generates text autoregressively.Read →
- The Fine-Tuning HandbookWhen to fine-tune vs RAG vs prompting, full fine-tuning vs parameter-efficient methods, how LoRA and QLoRA make it cheap enough for a single GPU, why the data matters more than the method, catastrophic forgetting, and how to evaluate and decide.Read →
- The On-Device AI HandbookSmall models, everywhere: why 1–14B models got frontier-adjacent (distillation + textbook data), the napkin law of local inference (tokens/sec ≈ bandwidth ÷ bytes), quantization and the llama.cpp / GGUF / Ollama / MLX stack, the honest local-vs-cloud decision table, four hybrid patterns that ship, and the evals craft of choosing a small model that actually fits the device.Read →