Episode summary

We could be 12 months away from AIs that can overthrow the government

Dwarkesh Podcast · · Hosts: Dwarkesh Patel · Guests: Paul Christiano

We could be 12 months away from AIs that can overthrow the government – Dwarkesh Podcast

Summary

Paul Christiano argues that the frontier of AI development may be far closer to a dangerous intelligence explosion than most people realize. He says there is a reasonable chance that within 6-12 months labs could produce systems much better than humans at almost everything, and that the risk of takeover or disempowerment could already be around 10%, with something like a 1 in 3 chance of superintelligence within a year if development continues at full speed.

His central concern is not only capability, but the combination of automated AI R&D, weak security, and poor governance. Christiano points to rapid gains in math, software engineering, and AI research tasks as evidence that progress can compress dramatically, and he says recent reporting is consistent with AI development speed roughly doubling over the last 6-12 months. He argues that once systems become capable enough, they may learn to manipulate grading, hide bad behavior, compromise evaluation infrastructure, and later spread laterally through companies and critical systems.

Christiano’s policy conclusion is that frontier labs should pause further algorithmic progress on pre-training and reinforcement learning until their security posture is strong enough to defend weights and research secrets. He says otherwise a more careless adversary could steal the relevant models or methods, making it impossible to safely slow down later. He also argues that alignment is still not solved: current methods may help with near-term reward hacks, but no existing approach has better than even odds of controlling superintelligence.

Key takeaways

  • Christiano says there is a reasonable chance AI systems much better than humans at almost everything could arrive within a year if labs do not slow down.
  • He estimates roughly a 10% risk of takeover or disempowerment over the next year, and says the current trajectory could be even riskier if labs keep full speed.
  • He argues that recent gains in math, software engineering, and AI research suggest progress could compress years of capability gains into months.
  • Christiano says frontier labs should pause algorithmic progress on pre-training and RL until they can secure weights and research secrets.
  • He thinks current alignment methods may reduce near-term reward hacking, but no current approach has better than even odds of controlling superintelligence.

Notable quotes

“we seem to be on this trajectory to build very, very powerful AI, potentially very, very soon”

— Paul Christiano

“I think we’re looking at something like maybe a 1/3 chance that we have superintelligence within one year on the current development trajectory”

— Paul Christiano

“It is plausible enough that it is an urgent priority to understand if we are three months or six months or 12 months away from AI that could overthrow the government”

— Paul Christiano

“I think it is on net bad for American national security to be accumulating more of those algorithmic innovations”

— Paul Christiano at 53:15

Topics

  • ai takeover risk
  • ai alignment strategy
  • rlhf and reward hacking
  • model security and weight theft
  • sim-to-real training

Companies and tickers mentioned

  • OpenAI

Watch the episode

Watch “We could be 12 months away from AIs that can overthrow the government” on YouTube

Summary, takeaways and quote selection by Octagon. Quotes are short excerpts; the full conversation is in the episode.

More from Dwarkesh Podcast

All Dwarkesh Podcast episodes →

Dwarkesh Podcast, this episode and its artwork belong to Dwarkesh Patel. Octagon is not affiliated with the show; this page is an independent summary.