Vibe Engines
YouTube

Vibe Engines is free, interactive AI engineering & system design — runs in your browser. Pick a learning path

Handbooks  /  GPU vs TPU
Engineering~8 min readComparison
Head to Head

GPU vs TPU: flexible, or purpose-built?

GPUvsTPU

A TPU can be more efficient than a GPU at the exact operation that dominates transformer training — matrix multiplication — because it was built around nothing else. That specialization is also exactly why it's the wrong choice the moment your workload steps outside it.

01

The one distinction that decides everything

A GPU (Nvidia, dominant via the CUDA ecosystem) is a general-purpose massively-parallel processor — originally built for graphics, now the default for nearly all AI workloads. A TPU (Google's Tensor Processing Unit) is an ASIC — application-specific silicon — purpose-built around a systolic array optimized specifically for matrix multiplication, available only on Google Cloud.

→ The rule

Need framework flexibility, custom kernels, or multi-cloud portability? GPU — that's the entire industry's default. Already on GCP running large, well-understood matmul-heavy training at scale? TPU is worth evaluating.

02

Head to head

DimensionGPUTPU
DesignGeneral-purpose parallel compute (SIMT cores)Purpose-built systolic array for matmul
AvailabilityEvery cloud, on-prem, consumer cardsGoogle Cloud only
Software ecosystemCUDA — universal framework support, largest ecosystemXLA/JAX-first; PyTorch support improved but secondary
Best atFlexible — training, inference, custom kernels, researchLarge-scale, matmul-heavy training at Google-scale
InterconnectNVLink (node), InfiniBand (cross-node)Custom high-bandwidth pod interconnect, built for pod scale
Custom architectures / kernelsHigh flexibility — CUDA lets you write anythingLower — best within XLA-compilable ops
Who actually uses itEssentially the entire industry outside GooglePrimarily Google, plus GCP customers who buy in
03

When to use each

Reach for GPU

  • Default choice for almost everyone
  • Need framework flexibility and custom architectures
  • Multi-cloud portability matters
  • Ecosystem (pretrained checkpoints, tutorials, tooling) assumes CUDA

Reach for TPU

  • Already on GCP
  • Training well-understood, transformer-shaped workloads at large scale
  • XLA compilation overhead pays off at your scale
  • Want Google’s specific pod-scale interconnect economics
04

Not really a per-run combination — but often a per-stack one

A single training run happens on one or the other, not both. But many teams DO use both across their overall stack — train on TPU pods for cost/scale where the workload fits, while keeping GPU infrastructure for portability, edge deployment, or the parts of the pipeline (research iteration, custom kernels) that don't fit TPU's constraints well. Frameworks like JAX increasingly aim for code portability across both, turning this into more of an infra/cost decision made per workload than a full rewrite each time.

→ The practical reality

Most organizations outside Google default to GPU for everything, because the ecosystem cost of stepping outside CUDA usually outweighs a TPU's raw efficiency advantage on any single workload.

05

Why the systolic array is specifically good at matmul

The architectural reason a TPU is efficient for large matrix multiplies is the systolic array: a grid of multiply-accumulate units where data flows through the grid and each loaded weight gets reused across many computations before being evicted, rather than being fetched from memory again for each use. Since matrix multiplication IS mostly repeated multiply-accumulate operations over the same operands, this design minimizes the memory-bandwidth pressure for exactly the operation that dominates transformer training and inference — and memory bandwidth, not raw compute, is very often the actual bottleneck in large-scale training.

A GPU's tensor cores also accelerate matmul well, but the GPU as a whole is a more general SIMT (single-instruction-multiple-thread) processor that wasn't built around ONLY that operation — which is exactly why it remains better for the long tail of non-matmul, custom, research-oriented workloads: novel attention variants, custom loss functions, sparse operations, anything that needs a hand-written CUDA kernel rather than compiling cleanly through XLA. The TPU's specialization is a real efficiency win precisely where the workload matches it, and a real limitation the moment it doesn't.

→ The trade you’re actually making

A TPU trades general-purpose flexibility for efficiency at one specific operation — worth it exactly when your workload IS that operation, at scale, and a liability the moment your workload needs something XLA can't compile cleanly.

06

A worked scenario: research iteration, then production-scale pretraining

A lab designing a new transformer architecture with experimental custom attention kernels wants GPUs — during this phase, the ability to write and iterate on custom CUDA kernels while the architecture itself is still changing is worth more than raw matmul efficiency, because the workload isn't yet a fixed, well-understood shape a systolic array is optimized for.

Once that architecture is settled and the lab is running repeated, large, well-understood pretraining passes of the now-finalized architecture — a workload that IS mostly large matrix multiplies, at Google-infrastructure scale — TPU pods' cost/performance for that specific, now-stable, matmul-heavy workload becomes genuinely attractive, if the lab is already on GCP. The right accelerator tracks how settled and matmul-shaped the workload has become, not a fixed rule about which is "better."

→ The pattern generalizes

Flexibility matters most while the workload is still changing. Specialization pays off once the workload has stopped changing and matches exactly what the specialized hardware was built for.

★

The flip point: sustained utilization

A TPU pod bills for the whole slice whether or not you saturate it, and its systolic array only reaches its low cost-per-FLOP on large, dense, statically-shaped matmuls. So a TPU is cheaper than a GPU only above the utilization where its price-per-effective-FLOP actually beats the GPU’s — peak price advantage means nothing at the FLOPs you leave idle.

→ The number

Suppose a TPU is ~2× cheaper per peak FLOP (so 0.5× the GPU’s cost per peak FLOP) but your workload only keeps it 40% utilized. Your effective cost is 0.5 / 0.40 = 1.25× a well-fed GPU — you are now paying more. The break-even utilization here is 50%: below it the GPU wins on cost, above it the TPU pulls ahead. The flip point is therefore sustained utilization on supported ops: large-batch, standard-architecture training saturates a TPU and it wins; dynamic shapes, custom kernels, smaller or varied jobs, and the CUDA ecosystem keep you on the GPU. Plug in your real price ratio and measured utilization; the crossover is the utilization at which their product favors the TPU.

The picture

The picture: why a systolic array reuses every weight

weights loaded once, held in the cells inputs8 vectors partial sums flow down and out systolic reuse16 weight reads128 MACs: 8 per read no reuse (naive)128 weight reads1 MAC per read reuse = batch size 4×4 cells, 8 input vectors: counts computed, sizes illustrative
A weight-stationary systolic array, the shape the page describes. Each cell holds one weight. Input values enter from the left and march one cell per clock to the right. Each cell multiplies, adds the result to the partial sum arriving from above, and passes it down. A weight is read from memory once and then used by every input vector that streams past. With 8 vectors that is 8 uses per read; with 1 vector it is 1. Double the vectors and you double the work each memory read pays for. That is the mechanical reason behind the page’s flip point. Large batches fill the array and it runs near its peak, while small or ragged batches leave it waiting on memory. A GPU’s tensor cores reuse operands too, by tiling through on-chip memory. The difference is that the TPU dedicates most of its die to this one pattern.
Trade-offs

Cost and trade-offs: where each chip wins or loses time

Read Time as how fast you get a working, efficient run. Read Space as what it costs you in money, compute or lock-in. The utilisation rows extend the page’s own example: the TPU at 0.5× the GPU’s cost per peak FLOP.

CaseTimeSpaceWhy
New attention variant or custom opGPU: write a CUDA or Triton kernel and run it the same dayTPU: it must compile through XLA, or be written in JAX’s Pallas, with far fewer ready-made kernelsThis is the research phase in the page’s scenario. The flexibility is worth more than per-FLOP price here.
Shapes change every batchXLA compiles a program per distinct input shape, so a new shape means a recompilePadding to a fixed shape wastes compute: an average of 1,300 tokens padded to 2,048 is 63.5% useful workBucket sequence lengths into a few fixed sizes. GPUs run dynamic shapes eagerly with no compile step.
TPU at 25% utilisationRuns, but the array mostly waits on memory0.5 ÷ 0.25 = 2× a well-fed GPU’s cost per useful FLOPSmall jobs, small batches or a slow input pipeline. Utilisation decides it, not list price.
TPU at 70% utilisationLarge, dense, static matmuls keep the array full0.5 ÷ 0.7 ≈ 0.71× the GPU’s costAbove the 50% break-even, the specialisation pays off.
Serving one request at a timeA batch of 1 means 1 use per weight read (see the diagram)Both chips run far below peak, and a latency target caps how far you can batchInference efficiency comes mostly from batching concurrent requests together. That works on either chip, so the batcher matters more than the chip.
One codebase for bothJAX code compiles through XLA on GPUs and TPUs alikePyTorch-first code needs PyTorch/XLA to reach a TPU, and every custom CUDA kernel needs a replacementThe page’s “per-stack, not per-run” pattern is cheapest when the model is written in JAX from the start.
Scaling to thousands of chipsGPU: NVLink inside a node, InfiniBand between nodesTPU: the pod interconnect is part of the product, built for this scaleAt pod scale the network, not the chip, often sets step time. That is part of the TPU’s pitch.
Moving clouds or going on-premGPU code moves with youTPU: Google Cloud only, so the code and the contract both stay putLock-in is a cost you pay later, at renewal or migration time.
Hiring and debuggingMost engineers, profilers and forum answers assume CUDAA TPU team needs XLA and JAX depth that fewer people haveThis is the “ecosystem cost” the page says usually outweighs a per-workload efficiency win.

Two rows decide most real choices. The shape row: can your workload be made static and dense? The utilisation rows: will you keep the chips busy? If both answers are yes and you are already on GCP, the TPU math works. If either is no, the GPU’s flexibility is cheaper than it looks on a price sheet. The cheap way to find out is a short pilot: port one stable training job, measure step time and utilisation on both chips at the same global batch size, and multiply each by its real hourly price. Count the engineering days the port took as part of the bill. That pilot answers in a week what a spec-sheet comparison cannot answer at all.

Watch for

Common mistakes

  • Comparing price per peak FLOP

    A spec sheet shows peak throughput, but you pay for utilisation. The page’s example shows a chip 2× cheaper per peak FLOP costing 1.25× more at 40% busy. Measure model FLOPs utilisation on your own training step before you compare prices: the FLOPs your model needs per step, divided by step time, divided by the chip’s peak. That one ratio, times the hourly price, is the number to compare.

  • Benchmarking at the wrong batch size

    Reuse in a systolic array equals the batch size (see the diagram). A large-batch training benchmark tells you little about serving one request at a time, and the reverse is true too. Benchmark at the batch sizes you will actually run, for training and for inference separately.

  • Feeding XLA dynamic shapes

    Variable sequence lengths, data-dependent control flow or a ragged last batch each trigger a recompile. Training then appears to stall for no clear reason. Pad to a few fixed buckets, drop or pad the last batch, and use the compile cache so restarts do not pay again.

  • Starving the accelerator

    If the host CPU cannot decode, augment and tokenise data as fast as the chip consumes it, the chip waits. A faster accelerator only makes that wait longer. Profile step time against input-pipeline time first. This failure looks like low utilisation on either chip, and it is often blamed on the chip. Preprocess offline where you can, prefetch the next batch while the current one runs, and add host CPU before you add accelerators.

  • Porting by changing the device string

    With PyTorch/XLA, an op that XLA cannot lower can end up running on the host CPU, and one such op inside the training step can erase the gain. Check the op-level metrics after porting. Budget the engineering time to replace custom CUDA kernels, not only the cloud bill.

Your call

Which would you pick?

Three situations. Pick the side you'd actually build — the explanation follows.

A research team is still changing model architecture week to week and writing custom kernels.

A team already on GCP runs large, stable, matmul-heavy training that has not changed in a year.

A benchmark shows a TPU is substantially more efficient than a GPU on your training step. Is that decisive?

Frequently asked

Quick answers

GPU or TPU — which should I use?

Default to GPU — it’s available everywhere, has the largest software ecosystem, and supports custom architectures and kernels. Consider TPU only if you’re already on Google Cloud and training a large, well-understood, matmul-heavy workload where XLA compilation overhead pays off.

Why is a TPU efficient specifically for AI training?

Its systolic array architecture reuses each loaded weight across many multiply-accumulate operations before evicting it, minimizing memory-bandwidth pressure for matrix multiplication — the operation that dominates transformer training and inference.

Can I run PyTorch on a TPU?

Yes, PyTorch/XLA support has improved substantially, but TPUs remain a JAX/XLA-first ecosystem. GPU’s CUDA ecosystem still has broader, more mature framework and tooling support across the board.

Is a TPU always faster than a GPU for AI?

No — only for workloads that are large-scale, matmul-heavy, and XLA-compilable. For custom kernels, novel architectures still being iterated on, or workloads outside that shape, a GPU’s general-purpose flexibility wins.

▶  Watch it explained

GPU vs TPU: flexible, or purpose-built?

GPU vs TPU · Vibe Engines · 2026
Finished this one? 0 / 209 Handbooks done
NextQuantization vs Distillation Next on the “Serving models, and paying for them” ladder · rung 5 of 13

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

More Handbooks