Episode summary
What It Takes to Build an AI That Actually Knows You
Summary
Ken Liu argues that AI agents become much more useful when they have a durable model of the person they serve, not just a chat history. First Person Labs’ product, Second, is designed as a personal context layer that learns preferences, priorities, commitments, relationships, and work patterns, then shares that context with other agents so users do not have to re-explain themselves every time.
A major part of the discussion is the systems problem behind that idea. Liu says Second analyzes the raw stream of computer activity, plus connectors like messages and apps, to infer what the user is doing and what might matter next. The host highlights proactive behavior such as surfacing a weather-related change in an event plan or catching context the user would have forgotten. Liu frames this as moving from reactive assistance to reflective assistance that connects dots over time.
Security and trust are central to the product design. Liu says users can bring their own tokens, run analysis locally, or use confidential inference so personal data is handled inside a provable secure environment. The longer-term vision is an ambient, portable “second” that becomes the trusted store for personal context while other specialized agents come and go around it.
Key takeaways
- Ken Liu says Second is meant to model a person’s memories, priorities, preferences, goals, values, commitments, and intent, then put that person in control.
- The host says Second can proactively surface changes and reminders the user did not ask for, which he describes as unusually thoughtful and useful.
- Liu says the system watches a raw stream of computer events and uses agents to infer higher-level meaning across apps, typing, and switching behavior.
- Security is built around local data control, bring-your-own tokens, and confidential inference rather than a simple policy promise, Liu says.
- Liu expects the product to become a trusted, portable context layer that other agents query instead of a standalone chatbot.
Notable quotes
“We focus on essentially creating a model of a person and then put that person in control”
“You no longer have to explain anything to your coding agents. You can just screenshot anything”
“The question then is like, well, can you just like analyze the heck out of it”
“if you don't trust let's say us in principle you should be able to bring your own tokens to then analyze data”
“I think right now the the fact that we have many personal agents is probably transitory”
Topics
- ai agents
- personal context
- proactive ai
- security architecture
- llm privacy
Companies and tickers mentioned
- First Person Labs
- a16z
Watch the episode
Watch “What It Takes to Build an AI That Actually Knows You” on YouTube
Summary, takeaways and quote selection by Octagon. Quotes are short excerpts; the full conversation is in the episode.
More from The a16z Show
-
What History Tells Us About the AI Revolution | Tyler Cowen & Alex Imas -
Cyberattacks Used to Take Months. AI Can Do It in Minutes. -
How AI Could Bring Halo and Other Classic Games Back to Life -
Mike Volpi on Why AI Infrastructure Will Outlast the Hype Cycle -
Why HubSpot’s Co-Founder Is Betting Against the SaaSpocalypse -
Inside Salesforce’s Acquisition Strategy for the AI Era
The a16z Show, this episode and its artwork belong to Andreessen Horowitz. Octagon is not affiliated with the show; this page is an independent summary.