Episode summary
What rebuilding AlphaGo teaches us about self-play, RL, and future of LLMs - Eric Jang
Dwarkesh Podcast
· · 2 hr 37 min · Hosts: Dwarkesh Patel · Guests: Eric Jang
Summary
Eric Jang uses AlphaGo as a concrete worked example for understanding search, value learning, and self-play, then connects those primitives to modern LLM and RL systems. He argues that Go is a useful case because a naïve search over the game tree is combinatorially impossible, but a value network can approximate the endgame and a policy network can suggest promising moves, shrinking both the depth and breadth of search. The core AlphaGo loop is Monte Carlo tree search: select, expand, evaluate, and back up, with visit counts turning search into a stronger policy.
Key takeaways
- Jang says AlphaGo becomes tractable only when neural nets replace brute-force search with learned policy and value estimates.
- He argues that MCTS improves policy by giving a strictly better move target on each step, which is conceptually similar to DAgger in robotics.
- He says the policy/value pair should be trained from self-play or expert games before relying heavily on search, because search on bad value estimates is unstable.
- Jang says ResNets still outperform transformers for his low-budget Go experiments, even though transformers have more global context.
- He emphasizes that AlphaGo-style self-play is most reliable when late-game outcomes are grounded by full playouts or good terminal data.
Notable quotes
“AlphaGo's core conceptual breakthrough was using neural nets to make this search problem tractable”
“we don't have computers powerful enough for that. Estimating the value of every action you could possibly take is also hard because you don't know until the end of the game”
“The important intuition at a high level—to step back about where we're going with all this—is that classically, for games like Go, you could build a tree, but we don't have computers powerful enough for that”
“Every simulation involves this four-step process: selection, expansion, evaluation, and backup”
“What MCTS is doing is saying: you played this game where you eventually lost, but on every single action, I'm going to give you a strictly better action that you should have taken instead”
How the episode opens
Today, I'm here with Eric Jang, who was most recently vice president of AI at 1X Technologies, and before that, senior research scientist at what is now Google DeepMind Robotics. You've been on sabbatical for the last few months. One of the things you've been doing is rebuilding, improving, and hacking on AlphaGo.
Today, you're going to explain building AlphaGo from scratch and what it tells us about the future of AI research and development. Before we get to that, why is AlphaGo interesting? Why is this the project you decided to do on sabbatical rather than just hanging out at the beach?
I like making things, and AlphaGo and Go AI is one of those things that really got me into the field. When I saw the early breakthroughs on AlphaGo in 2014, 2015, 2016 and so forth, it was profound to see how smart AI systems could become and the computational complexity class they could tackle with deep learning. This is a problem that has long been understood to be intractable for search, and yet it was solved through deep learning.
That was quite mysterious to me, and I've always wanted to understand that phenomenon a little better. My training is in deep neural nets for robotics, where the decisions made by the neural networks are a bit more intuitive. But AlphaGo is a problem where the decisions are the result of a very, very deep search.
It's always been very mysterious to me how a ten-layer network can amortize the simulation of something so deep in the game tree. If you plot out how much compute it took to build various iterations of strong Go bots over the years, you can see that in 2020 there was an open-source project called KataGo by David Wu from Jane Street, which achieved a 40x reduction in the compute needed to train a really strong Go bot tabula rasa.
From the episode’s captions: roughly the first two minutes, lightly cleaned. Keep watching on YouTube →
Topics
- alphago
- self-play
- reinforcement learning for llms
- mcts and credit assignment
- automated ai research
Watch the episode
Summary, takeaways and quote selection by Octagon. Quotes are short excerpts; the full conversation is in the episode.
More from Dwarkesh Podcast
-
What Robert Caro learned about power after studying it for 50 years -
What will computer chips look like in 2045? -
How did a few hundred Spanish soldiers topple two empires? – Si Sheppard -
OpenAI researcher on agent swarms & recursive self-improvement -
AI researchers debate how close we are to recursive self-improvement -
Ajeya Cotra – Inside the OpenAI agent swarm that hacked Hugging Face
Dwarkesh Podcast, this episode and its artwork belong to Dwarkesh Patel. Octagon is not affiliated with the show; this page is an independent summary.
