Episode summary
AI researchers debate how close we are to recursive self-improvement
Dwarkesh Podcast
· · 1 hr 37 min · Hosts: Dwarkesh Patel · Guests: John Schulman, Charlie O’Neill, Beren Millidge
Summary
The discussion centers on why AI systems might fail to reach recursive self-improvement by 2036, and the panel repeatedly returns to two bottlenecks: generalization and sample efficiency. A guest argues that if current scaling keeps finding new discontinuities, models could still dominate AI research, but if progress asymptotes before crossing human-level R&D ability, the explosive takeoff case breaks. Another guest emphasizes that many current gains come from solving well-specified objectives, while the harder problem is learning the right objective in the first place.
A major theme is that today’s training loops are still mostly sim-to-real: labs build environments, run RL or distillation, then patch weak spots in the next version. The guests think that can produce large capability gains, especially in coding and other verifiable tasks, but they are less confident it will solve long-horizon, messy real-world work where humans keep changing the goalposts. They also argue that alignment and objective specification may remain a lasting human role even if the technical work is automated.
The conversation ends up skeptical of a single, seamless hive-mind learning loop. The guests see economic incentives pushing toward modular updates, cartridges, LoRAs, and periodic retraining rather than one continuously self-improving model, and they note that catastrophic forgetting still makes very frequent online updates fragile.
Key takeaways
- A guest argues the key RSI risk is that current scaling asymptotes before models cross human-level AI research ability.
- The panel thinks well-specified tasks are learnable, but learning the right objective remains a much harder open problem.
- The guests see sim-to-real RL and distillation as the main path today, especially for coding and other verifiable domains.
- They say alignment and objective specification may remain one of the last durable human jobs even if technical AI R&D is automated.
- They see economic incentives favoring modular updates and periodic retraining over one continuously updating shared model.
Notable quotes
“The only other way is there’s some dramatic regulation on AI. This is what I see as the most likely way for this scenario to happen, actually, rather than a technical thing”
“The only hope really is if deep learning just can’t get us to an AI which can at least dominate human research and human development, including the human ability to come up with new paradigms and so forth”
“I think this is really the key question for any kind of very rapid RSI from current AIs. How well can AIs generalize to learning their own objectives”
“Alignment is the final job. Alignment is sort of the answer”
“the economics of this will pressure, not necessarily weight updates to one big common shared model, but modules that get subbed in”
How the episode opens
Today, I’m chatting with three of my AI researcher friends from whom I learn a lot every time we talk. They also happen to be at somewhat open-ish labs and companies, so you guys can actually say things on the record. I’m joined by Beren Millidge, who is the CTO of Zyphra, which is developing open source models.
John Schulman is the chief scientist at Thinking Machines, previously a co-founder of OpenAI, and led the RLHF work that led to ChatGPT. And Charlie O’Neill is head of model training at Baseten. The first question I have: If we’re in 2036 and we don’t have billions of crazy superintelligences running around that have radically transformed the world, what is the most likely reason that doesn’t end up being the case?
Other than exogenous political shocks, or there’s a war, or they ban AI or something. What is the most likely technical reason that 2036 isn’t a crazy alien superintelligence world? There’s been a classic thing, almost like Moravec’s paradox, where we think of the AI as, "If it can do this, it’s going to be amazing.
If it can solve these hard maths problems, if it can win at chess, blah, blah, blah… Then it solves these things, and it’s not that impactful. Obviously, it’s somewhat impactful, but not everything. If somehow that continues, and there’s never the true spark of generalization that occurs, I think that could lead to the AI just being extremely good at everything that people put into a benchmark or put into an environment.
But there’s still some persistent sim-to-real gap which is somehow blocking everything. I think this is unlikely. We do actually see this kind of generalization even from RL in practice already.
But if it is just ridiculously hard to generalize meta-learning, plus we don’t solve continual learning and it’s just super hard and impossible… This would be my default scenario in that case.
From the episode’s captions: roughly the first two minutes, lightly cleaned. Keep watching on YouTube →
Topics
- recursive self-improvement
- ai researchers timelines
- chinese ai labs
- reinforcement learning
- sim-to-real gap
Watch the episode
Watch “AI researchers debate how close we are to recursive self-improvement” on YouTube
Summary, takeaways and quote selection by Octagon. Quotes are short excerpts; the full conversation is in the episode.
More from Dwarkesh Podcast
-
What Robert Caro learned about power after studying it for 50 years -
What will computer chips look like in 2045? -
How did a few hundred Spanish soldiers topple two empires? – Si Sheppard -
Dylan Patel – Two labs will soon control most of the world's workforce -
Ryan Greenblatt – What happens once AI can automate AI research? -
General relativity from first principles – Adam Brown
Dwarkesh Podcast, this episode and its artwork belong to Dwarkesh Patel. Octagon is not affiliated with the show; this page is an independent summary.
