Episode summary

What It Takes to Build an AI That Actually Knows You

The a16z Show · · Hosts: Martin Casado, Joel de la Garza · Guests: Ken Liu

What It Takes to Build an AI That Actually Knows You – The a16z Show

Summary

Ken Liu argues that AI agents become much more useful when they have a durable model of the person they serve, not just a chat history. First Person Labs’ product, Second, is designed as a personal context layer that learns preferences, priorities, commitments, relationships, and work patterns, then shares that context with other agents so users do not have to re-explain themselves every time.

A major part of the discussion is the systems problem behind that idea. Liu says Second analyzes the raw stream of computer activity, plus connectors like messages and apps, to infer what the user is doing and what might matter next. The host highlights proactive behavior such as surfacing a weather-related change in an event plan or catching context the user would have forgotten. Liu frames this as moving from reactive assistance to reflective assistance that connects dots over time.

Security and trust are central to the product design. Liu says users can bring their own tokens, run analysis locally, or use confidential inference so personal data is handled inside a provable secure environment. The longer-term vision is an ambient, portable “second” that becomes the trusted store for personal context while other specialized agents come and go around it.

Key takeaways

  • Ken Liu says Second is meant to model a person’s memories, priorities, preferences, goals, values, commitments, and intent, then put that person in control.
  • The host says Second can proactively surface changes and reminders the user did not ask for, which he describes as unusually thoughtful and useful.
  • Liu says the system watches a raw stream of computer events and uses agents to infer higher-level meaning across apps, typing, and switching behavior.
  • Security is built around local data control, bring-your-own tokens, and confidential inference rather than a simple policy promise, Liu says.
  • Liu expects the product to become a trusted, portable context layer that other agents query instead of a standalone chatbot.

Notable quotes

“We focus on essentially creating a model of a person and then put that person in control”

— Ken Liu

“You no longer have to explain anything to your coding agents. You can just screenshot anything”

— Ken Liu

“The question then is like, well, can you just like analyze the heck out of it”

— Ken Liu

“if you don't trust let's say us in principle you should be able to bring your own tokens to then analyze data”

— Ken Liu at 0:50

“I think right now the the fact that we have many personal agents is probably transitory”

— Ken Liu at 10:27

Topics

  • ai agents
  • personal context
  • proactive ai
  • security architecture
  • llm privacy

Companies and tickers mentioned

  • First Person Labs
  • a16z

Watch the episode

Watch “What It Takes to Build an AI That Actually Knows You” on YouTube

Summary, takeaways and quote selection by Octagon. Quotes are short excerpts; the full conversation is in the episode.

More from The a16z Show

All The a16z Show episodes →

The a16z Show, this episode and its artwork belong to Andreessen Horowitz. Octagon is not affiliated with the show; this page is an independent summary.