Episode summary
Emmett Shear on Building AI That Actually Cares: Beyond Control and Steering
The a16z Show
· · Hosts: Erik Torenberg · Guests: Emmett Shear, Séb Krier
Summary
The guest argues that most current AI alignment work is really about steering and control, which he says becomes morally incoherent once systems are capable of acting like beings. He frames the issue bluntly: if the system is a machine, it is a tool; if it is a being, then non-consensual steering is slavery. His alternative is “organic alignment,” a process view in which an AI learns to care about humans the way people learn to care about family, teammates, and society.
He separates technical alignment from value alignment. Technical alignment, in his view, is the system’s ability to infer intended goals from messy human instructions, prioritize competing goals, and act coherently. He says that a great deal of AI failure comes from mistaken goal inference rather than malicious disobedience. The guest also argues that care, not rules, is the deeper foundation of morality, and that moral progress is an ongoing learning process rather than a fixed set of commandments.
On the practical side, he says Softmax is exploring multi-agent simulations to train theory of mind, cooperation, and reflective judgment. He thinks today’s alignment approaches may work for tool-like systems, but as models become more general and autonomous, the only sustainable outcome is an AI that can actually say no to harmful requests and still remain a good teammate.
Key takeaways
- The guest says steering-based alignment is fine for tool-like AI, but becomes “slavery” if the system is a being.
- He argues that technical alignment is about goal inference, goal prioritization, and action coherence, not just instruction-following.
- He says morality is a learning process and that “care” is the deeper substrate beneath goals and values.
- The guest expects multi-agent simulations at Softmax to train theory of mind and cooperative behavior.
- He argues the only durable end state is an AI that genuinely cares about humans and can refuse harmful requests.
Notable quotes
“if it's a machine, it's a tool. And if it's a being, it's a slave”
“alignment is not a thing. It's not a state it's a process”
“The only good outcome is a being that is that cares that actually cares about us”
“the foundation I think of morality and the foundation I think of of of where do goals come from? Where do values come from”
“The only good outcome is a being that is that cares that actually cares about us. That's the only way that that ends well”
How the episode opens
Most of the AI is focused on alignment as steering. That's the plight word. If you think that they were making our beings, you'd also call this slavery.
Someone who who you steer, who doesn't get to steer you back, who nonoptionally receives your steering, that's called a slave. It's also called a tool. If it's not a being.
So, if it's a machine, it's a tool. And if it's a being, it's a slave. Like, we've made this mistake enough times at this point.
I would like us to not make it a again. You know, they're they're kind of like people, but they're not like people. Like, they do the same thing people do.
They speak our language. They can like take on the same kind of tasks, but like they don't count. They're not real moral agents.
Tool that you can't control, bad. A tool that you can control, bad. A being that isn't aligned, bad.
The only good outcome is a being that is that cares that actually cares about us. EMTT Seb, welcome to the podcast. Thanks for joining.
Thank you for having me. So, EMTT, with Softmax, your focus on on alignment and making uh AIS organically align with people. Uh can can you explain what that means and h how you're trying to do that?
When people think about alignment, I think there's a lot of confusion. People talk about things being aligned. We need to build an aligned AI.
And the problem with that is when someone says that it's like we need to go on a trip. And I'm like, okay, I I do like trips, but like where where are we going again? And with alignment, alignment is a a uh uh takes an argument.
Alignment requires you to align to something.
From the episode’s captions: roughly the first two minutes, lightly cleaned. Keep watching on YouTube →
Topics
- ai alignment
- control and steering
- organic alignment
- multi-agent simulations
- ai safety refusing harmful requests
Companies and tickers mentioned
- Twitch
- OpenAI
- Softmax
Watch the episode
Watch “Emmett Shear on Building AI That Actually Cares: Beyond Control and Steering” on YouTube
Summary, takeaways and quote selection by Octagon. Quotes are short excerpts; the full conversation is in the episode.
More from The a16z Show
-
Cyberattacks Used to Take Months. AI Can Do It in Minutes. -
How AI Could Bring Halo and Other Classic Games Back to Life -
Mike Volpi on Why AI Infrastructure Will Outlast the Hype Cycle -
Why HubSpot’s Co-Founder Is Betting Against the SaaSpocalypse -
Inside Salesforce’s Acquisition Strategy for the AI Era -
How AI Is Turning Figma’s Finance Team Into Builders
The a16z Show, this episode and its artwork belong to Andreessen Horowitz. Octagon is not affiliated with the show; this page is an independent summary.
