Episode summary
Jev Creator on Will US Open Models Catch Chinese | The Cost of Intelligence & Tokens | Diogo Almeida
Summary
Diogo Almeida argues that the main barrier to AI automation is not raw capability but reliability: models can be brilliant most of the time and catastrophically wrong the rest, which makes them unsuitable for high-stakes background work. He says Jev is built around “reliability of intelligence,” not speed or cost, and that the field has over-optimized for human preference and benchmarks instead of real automation. In his view, the industry’s big lesson is that data matters more than compute, and that the most important question is choosing the right task before worrying about model size or algorithmic tricks.
He is skeptical of synthetic-data hype, open-ended benchmark chasing, and most “neo labs,” which he calls wasteful. He argues that the best data is task-shaped data that pushes the frontier models in the right distribution, not generic enterprise data lakes. He also says open-source and Chinese models are often compared on misleading benchmarks, and that many labs and copycat products are optimizing for optics rather than true intelligence.
On the commercial side, Diogo thinks incumbents like SaaS companies have an advantage because they know workflows and can engineer around them, but he rejects the idea that every company should “own its own intelligence” via bespoke models. He expects decision-model usage to expand dramatically and believes background automation will eventually dwarf token-heavy chatbot and coding-agent use cases, if the field can solve reliability and efficiency.
Key takeaways
- Diogo Almeida says reliability of intelligence is the real bottleneck, because models that are 95% right and 5% wildly wrong cannot safely automate important work.
- He argues the field’s “bitterest lesson” is that data matters more than compute, and the right task matters even more than the data itself.
- Diogo calls benchmarks “total bullshit” and says they create perverse incentives that hide real model quality.
- He thinks most neo labs are wasteful, while better ROI comes from task-specific engineering and data aimed at the frontier use case.
- He believes incumbents like SaaS firms have an edge because they understand workflows, but he doubts that most companies should train their own intelligence models.
- He expects decision-model usage to scale far beyond current LLM token usage if reliability enables real background automation.
Notable quotes
“Almost everyone who's paying attention to AI knows reliability is the absolute largest bottleneck”
“I would argue that the way to do that um oh man I wish I could like bring up my slides here but I will I'll I'll try to use my hands I guess”
“benchmarks are total [ __ ] All benchmarks are bad”
“I am concerned for the world. I think I think that throwing money into the furnace is a bad thing for the world”
Topics
- ai reliability
- training data
- synthetic data
- decision models
- ai market bubbles
Companies and tickers mentioned
- TypeSafe AI
- OpenAI
- Sequoia
- Andreessen Horowitz
- Harvey
- Neo Labs
Watch the episode
Summary, takeaways and quote selection by Octagon. Quotes are short excerpts; the full conversation is in the episode.
More from 20VC
-
Sequoia Leaves London, Close Scout Program & End Evergreen Fund | NASDAQ Loses $2.5T in 24 hours -
Mistral CEO: Why OpenAI & Anthropic Will Come for Your Business | Are Chinese Models Truly Safe? -
Inside WashU's Endowment Fund | SpaceX Made Up 20% of Out AUM | Scott Wilson -
Google Loses $500B in 12 Hours | DoorDash Buys Instacart for $17B | Kalshi Raises $1B -
Modal CEO: Has Open-Source LLM Inference Peaked? | What Happens to the Cost of Compute -
David Tisch: Why No Manager Wants to Stay Small & Why Competing with Mega Platforms is a Losing Game
20VC, this episode and its artwork belong to Harry Stebbings. Octagon is not affiliated with the show; this page is an independent summary.