Behind every ChatGPT token, every Sora frame, and every LLM you run in Ollama, there is an accelerator performing the same calculation trillions of times: matrix multiplication. The two ways of doing this in silicon — GPU (NVIDIA) and TPU (Google) — start from opposite philosophies. This text draws on the video GPU vs TPU: How AI Chips Actually Work, by Pouria Taj.
The video uses the H100 and the TPU v5e as teaching examples. In 2026 the factory floor is already Blackwell on one side and Trillium/Ironwood on the other — the architecture lesson has not changed.
Why AI is MatMul
An LLM does not spit out a paragraph all at once. Generation is autoregressive: the prompt becomes embeddings, the Transformer produces a distribution over the next token, the chosen token goes back into the input, and the cycle repeats.
In diagrams, neurons and arrows. In silicon, weight matrices. GPT-3 (175 B, 96 layers) already stacks tens of thousands of them; Claude Opus and GPT-4 rise to hundreds of billions to trillions of parameters. Almost all the cost is the same MAC: row × column, add, write the output cell.
Hardware's good fortune: each cell of the resulting matrix is independent. You do not need to wait for [0,0] to calculate [500,200]. That is why a CPU with a few fast cores loses badly — and why GPUs and TPUs exist.
Inside the GPU: H100
The GH100 die (Hopper) in the H100 SXM: 80 billion transistors, 8 GPCs, 132 SMs, 80 GB of HBM3 at 3,35 TB/s, TDP up to 700 W.
GH100 (H100 SXM)
80 B transistores | 8 GPC | 132 SM | 80 GB HBM3
SM × 132
4 warp schedulers
128 CUDA cores (FP32/INT32)
4 Tensor Cores (MatMul 4×4, FP8/FP16/BF16)
256 KB registradores | até 228 KB shared/L1
- 16.896 CUDA cores: general-purpose ALUs. At ~1,98 GHz that yields ~67 TFLOPS FP32 (one FMA per cycle).
- 528 Tensor Cores (4 per SM): dense algebra hardware. A matrix block per instruction, FP16/BF16/FP8. Marketing peak: ~2 PFLOPS dense FP8, ~4 PFLOPS with 2:4 sparsity.
The host dispatches a kernel. A 4096×4096 MatMul has 16,7 million cells → millions of threads, packed into blocks (up to 1024). The GigaThread Engine spreads the blocks across the 132 SMs. Within an SM, threads run in warps of 32, the same instruction in the same cycle (SIMT).
The bottleneck: the memory wall
A MAC in registers: ~4 cycles. Bringing the number from HBM3 to the core: hundreds of cycles. Writing the result: another trip. And thousands of neighboring cores simultaneously fetch the same row and column the core next door just loaded.
Much of a GPU's energy goes not to math but to moving data. It is the same wall the DRAM industry tries to break through with HBM and LPDDR (see CXMT in LPDDR6).
The TPU bet: a systolic array
Google's question: what if the core passed the number it just used directly to its neighbor, without sending it back to memory?
The TPU is a tensor-only ASIC. In v5e, each TensorCore has four MXUs with 128×128 MACs — 16.384 units per array, almost the H100's total CUDA core count, but welded together. No register in between. The name comes from systole: data pulses to the clock's rhythm.
X0 X1 X2
| | |
v v v
Y0→ MAC → MAC → MAC →
| | |
v v v
Y1→ MAC → MAC → MAC →
| | |
v v v
S0 S1 S2
- Weights stay put: they leave HBM, enter VMEM (SRAM), and stay fixed in each array cell.
- Inputs flow: the prompt enters from the left and advances one step per cycle.
- Accumulation goes down: each cell multiplies the passing value by the local weight, adds what came from above, and sends it downward.
- The result leaves at the edge — column by column, without a round trip.
HBM is touched twice: load weights, read results. Almost 100% of energy goes into calculation, not transit. v5e: ~197 TFLOPS BF16, 16 GB HBM, 819 GB/s — lower peak than H100, more joules per MAC in dense MatMul. In 2026 the same design grew: Trillium (v6e) moved to 256×256; Ironwood (TPU7x) reaches ~4,6 PFLOPS FP8 and 192 GB of HBM3e per chip, still only on GCP.
In 2026: Blackwell on one side, Ironwood on the other
H100 and v5e remain the best teaching examples — the design is clean and the numbers fit in your head. But someone buying today does not buy either. It is worth seeing what that same logic became when pushed to the limit.
On NVIDIA's side, the Blackwell architecture abandoned the single die. A Blackwell GPU packs 208 billion transistors in two reticle-limit dies, stitched together by a chip-to-chip interconnect of 10 TB/s that makes the two behave as one GPU. The second-generation Transformer Engine dropped to FP4: fewer bits per number, twice the throughput in the same area. In a DGX B200, each GPU delivers around 9 dense FP4 PFLOPS, with 180 GB of HBM3e and ~8 TB/s bandwidth — versus H100's 80 GB and 3,35 TB/s. And the sales unit is no longer the card: the GB200 NVL72 is a liquid-cooled rack with 36 Grace CPUs and 72 Blackwell GPUs in a single NVLink domain.
On Google's side, TPU7x Ironwood followed the opposite path from flexibility: two TensorCores and four SparseCores per chip, 256×256 MXUs, 4.614 TFLOPS FP8, 192 GB of HBM3E at 7.380 GB/s, and a superpod of 9.216 chips connected by optical switching. It was announced in April 2025 and became generally available on 31 March 2026.
Notice that both converged on numeric format — FP4 and FP8 — and diverged in scope. NVIDIA stacks general-purpose silicon and solves scaling with NVLink; Google keeps the rigid array and solves scaling with mirrors and lasers. The underlying calculation, MAC, is the same on both sides.
No longer a duel: the third bench
In 2017 the question “GPU or TPU?” made sense because there were only those two answers. In 2026 there are more players on the map.
- AMD. The Instinct line (MI300X, MI325X, and successors) puts plenty of HBM on each card and runs on Linux with ROCm, which is open. The bottleneck was never silicon: it is the gap between a kernel that compiles in CUDA and the same kernel in HIP. That gap shrinks with every release, but still exists.
- AWS Trainium. Amazon built its own ASIC and an entire data center around it — the subject of the post on Project Rainier. Same thesis as the TPU, different owner.
- NPUs in processors. Intel, AMD, and ARM SoCs already include inference blocks on the CPU die. They do not train anything, but handle small models running locally without firing up the GPU.
The pattern is clear: anyone with a predictable workload and enough scale to amortize a chip design builds an ASIC. Everyone else rents GPUs.
GPU vs TPU, side by side
| NVIDIA H100 (GPU) | Google TPU v5e | |
|---|---|---|
| Type | Programmable SIMT | Tensor ASIC |
| MatMul | CUDA + Tensor Cores, through registers/cache | 128×128 systolic MXU |
| Memory | HBM all the time | data flows cell to cell |
| Peak | ~2 PFLOPS dense FP8 / 700 W | ~197 TFLOPS BF16 |
| Where it runs | AWS, Azure, GCP, on-prem, PC | Google only (GCP / pods) |
| Software | CUDA, PyTorch, Triton, TensorRT | JAX, TensorFlow, XLA, PyTorch-XLA |
| 2026 generation | Blackwell B200 / GB200 NVL72 (FP4) | TPU7x Ironwood (FP8, superpod of 9.216) |
Why the world still buys GPUs
If a TPU uses fewer joules for the same calculation, why does NVIDIA's queue never end?
- ASIC rigidity. The TPU is fast because its silicon embodies today's Transformer mathematics. A new model that does not revolve around dense MatMul leaves an ASIC data center stranded.
- The GPU did not make that bet. Any kernel — physics, molecules, hybrids, the next paper — can be implemented in software.
- CUDA's moat. Two decades of libraries and people who already know them. New code runs on NVIDIA's driver the same day. For companies, adoption risk decides the purchase — not the MXU's TDP.
And on your machine?
None of that changes the fact that, on your Linux desktop, the accelerator is the GPU in the PCIe slot. It is worth knowing how to read what you have:
# o que está no barramento
lspci | grep -Ei "vga|3d|display"
# NVIDIA: modelo, VRAM, driver, uso em tempo real
nvidia-smi
# AMD: o equivalente do ROCm
rocm-smi
# Intel/Arc e integradas
sudo intel_gpu_top
The number that decides whether a model fits is not TFLOPS — it is VRAM. A 7 B model at Q4 quantization needs around 5 GB; a 70 B model at the same Q4 exceeds 40 GB and already requires two cards or offload to RAM, with the accompanying speed drop. That is the calculation the post about refurbished server performs in practice, and the reason llama.cpp and Ollama emphasize quantized GGUF so much.
You do not install a TPU: it only exists inside Google, rented by the hour on GCP or free on Colab and Kaggle. If your plan is running models locally, the path is CUDA or ROCm — and this comparison explains why things work out that way, rather than helping you choose between the two.
Timeline
- 2006 — NVIDIA launches CUDA and turns the gaming card into a general-purpose processor.
- 2012 — AlexNet wins ImageNet by training on two GTX 580 cards and proves neural networks are a GPU problem.
- 2015 — Google puts TPU v1 into internal production, INT8 inference only.
- 2017 — the TPU v1 paper comes out; v2 brings training and bfloat16; the Transformer is born on TPU pods.
- 2017 — the Volta architecture introduces Tensor Cores at NVIDIA: the GPU gains dedicated MatMul hardware.
- 2020–2022 — Ampere and Hopper consolidate CUDA's moat; A100 and then H100 become bargaining chips.
- 2021 — TPU v4 introduces optical circuit switching (OCS) and the 3D torus in the pod.
- 2023 — AMD launches the Instinct MI300 series with open ROCm; the race is no longer a single-vendor affair.
- 2024 — TPU v6e Trillium (256×256 MXU) and Blackwell announcement, with two dies in one package.
- 2025 — GB200 NVL72 reaches data centers; Anthropic secures access to up to 1 million TPUs.
- abr/2026 — TPU7x Ironwood becomes generally available: 4.614 TFLOPS FP8 and a superpod of 9.216 chips.
References
- Video: Pouria Taj — GPU vs TPU: How AI Chips Actually Work
- NVIDIA — Hopper architecture in-depth
- Google Cloud — TPU v5e
- How to Scale Your Model — How to think about TPUs
- NVIDIA — Blackwell architecture
- Google Cloud — TPU7x (Ironwood)
- Google TPUs: architecture, generations, and models
- Project Rainier: AWS's data center without NVIDIA
- How DeepSeek rewrote the Transformer
- llama-model: GGUF on Linux without the Ollama daemon
Extreme specialization versus generality: the TPU shows the efficiency ceiling when the calculation is known; the GPU sells insurance that tomorrow's calculation will still fit in the same rack. On everyday Linux, what you have on your desk is still the GPU — and the CUDA that came with it.