Episode summary
How GPT, Claude, and Gemini are actually trained and served – Reiner Pope
Dwarkesh Podcast
· · 2 hr 14 min · Hosts: Dwarkesh Patel · Guests: Reiner Pope
Summary
Reiner Pope argues that frontier-model serving economics are largely explained by a few roofline equations linking compute throughput, memory bandwidth, batch size, and context length. The central result is that latency falls and cost per token improves as batch size rises, because weight reads are amortized across more sequences; at the same time, the KV cache and context length can push systems from compute-bound to memory-bound. He says the practical batch size for frontier models is often around 300 times the model’s sparsity ratio, which for DeepSeek-style routing lands in the low-thousands of sequences.
The discussion then turns to mixture-of-experts layout and why rack-scale communication matters. Reiner says expert parallelism maps naturally onto an all-to-all pattern within a rack, while crossing rack boundaries makes communication much slower and becomes a bottleneck. That leads to the claim that larger scale-up domains help frontier labs because they can fit more experts, more parameters, and enough KV cache inside one fast interconnect domain.
Pipeline parallelism is presented as useful mainly for reducing memory capacity requirements, not latency or compute cost. Reiner argues that, during inference, most frontier systems should stay within a single scale-up domain, use expert parallelism heavily, and use only limited pipelining when model size forces it.
Key takeaways
- Reiner Pope says the key serving trade-off is batch size: bigger batches lower cost per token but raise queuing latency.
- He estimates practical frontier-model batches at roughly 300 times sparsity, implying around 2,000 sequences for DeepSeek-like models.
- He argues that expert parallelism fits the rack-level all-to-all network well, while crossing racks makes MoE communication much more expensive.
- He says pipeline parallelism mainly helps memory capacity, not runtime, so it is secondary for inference.
- He claims larger scale-up domains unlocked bigger models because they can hold more weights and KV cache inside one low-latency interconnect domain.
Notable quotes
“the big effect is batch size. What we're going to do now is quantify exactly what that looks like and what its implications are on latency and cost”
“if you do not batch together many users, the cost and the economics you get can be a thousand times worse than if you do batch many users together”
“What we actually want to plot is the cost versus batch size, which is t over B versus batch size. This is the cost per token”
“batch size needs to be bigger than approximately 300 times sparsity. For example, in DeepSeek I activate 32 out of 256 experts, so this would be 8 for DeepSeek”
“The standard practice here, and it is the best solution, is to use expert parallelism. That means different experts go on different GPUs”
How the episode opens
Today, I'm interviewing Reiner Pope, who is the CEO of MatX, which is a new chip startup. Previously, he was doing TPU architecture and many other things at Google. This is a very different format from my usual interviews.
This is going to be a blackboard lecture. We're going to get up in a second. We in fact built this whole new studio with specifically this format in mind, so it's a pleasure to get to inaugurate it with you.
We're going to be talking about model architecture, ML infra, and many other things. The reason I think it's an important topic is because once you understand how training and inference work in a cluster, a lot of things—about why AI is the way it is, why AI architectures are the way they are, why API prices are the way they are, and fundamentally why AI progress is the way it is—start making sense. You need to understand the details to get there, and you need a blackboard to understand the details.
Reiner, thank you so much for doing this. Very happy to be here. Full disclosure, I am an angel investor in MatX, but that's unrelated to this podcast.
Reiner, to kick us off I'll ask this question. We have a couple of companies like Claude and Codex and Cursor offering something like Fast Mode, where for 6x the price, they'll stream you tokens at 2.5x the speed. Mechanically, I'm curious what's going on here.
Why is it the case that you can pay more to get faster latency? Two, could you keep going? Could you pay 100x more and somehow get much faster speeds?
Three, could you go the other way? Could you have something like Claude Code "Slow Mode", where if you are willing to wait for minutes on end, you could get even cheaper prices?
From the episode’s captions: roughly the first two minutes, lightly cleaned. Keep watching on YouTube →
Topics
- llm training
- llm serving
- moe gpu layout
- pipeline parallelism
- api pricing long-context costs
Companies and tickers mentioned
- GOOGL
- GOOG
- OpenAI
- Anthropic
- MatX
Watch the episode
Watch “How GPT, Claude, and Gemini are actually trained and served – Reiner Pope” on YouTube
Summary, takeaways and quote selection by Octagon. Quotes are short excerpts; the full conversation is in the episode.
More from Dwarkesh Podcast
-
What Robert Caro learned about power after studying it for 50 years -
What will computer chips look like in 2045? -
How did a few hundred Spanish soldiers topple two empires? – Si Sheppard -
OpenAI researcher on agent swarms & recursive self-improvement -
AI researchers debate how close we are to recursive self-improvement -
Ajeya Cotra – Inside the OpenAI agent swarm that hacked Hugging Face
Dwarkesh Podcast, this episode and its artwork belong to Dwarkesh Patel. Octagon is not affiliated with the show; this page is an independent summary.
