Episode summary
Modal CEO: Has Open-Source LLM Inference Peaked? | What Happens to the Cost of Compute
20VC
· · 1 hr 2 min · Hosts: Harry Stebbings · Guests: Erik Bernhardsson
Summary
Erik Bernhardsson argued that inference is becoming a new cloud layer, but not a single commodity product. He thinks the market will fragment into many specialized products—low-latency inference, batch inference, sandboxes, post-training, reinforcement learning, and agentic workflows—and that providers will win by bundling enough of those pieces together to create switching costs and capture margin.
He framed Modal’s strategy as building infrastructure that makes compute feel like one pooled resource. Rather than relying on Kubernetes or Docker, Modal built its own container files, filesystem, runtime, and scheduler, which Erik said lets the company aggregate customer demand and manage GPU capacity more efficiently. He described GPU sourcing as still chaotic, but said demand is growing about 30% to 40% month over month and that the company is willing to keep buying capacity as long as customers can pay more.
On competition, Erik was bullish on both Nvidia and specialized chip vendors, and skeptical that CUDA alone is the real moat. He rejected the idea that open-source inference has peaked, saying both open and frontier models can grow several times over the next year. He also said software engineers are not going away soon; in his view, better tools and agents mostly expand demand for software.
Key takeaways
- Erik Bernhardsson said inference is turning into "the new cloud," but expects the market to fragment into many specialized products rather than one winner-take-all layer.
- Erik Bernhardsson believes Modal can expand margins by bundling sandboxes, post-training, batch jobs, and agentic workflows around core inference.
- Erik Bernhardsson said Modal is already profitable on a P&L basis, while cash flow remains negative because the company keeps buying GPUs and capacity.
- Erik Bernhardsson said demand has been growing about 30% to 40% month over month, which he uses to forecast GPU purchases 3 to 4 months ahead.
- Erik Bernhardsson dismissed the idea that open-source LLM inference has peaked, arguing both open and frontier model usage can still grow 5x to 20x over the next 12 months.
Notable quotes
“I think there is an opportunity to create a hundred billion dollar company with a lot of specialized models in inference”
“I think inference is going to be an enormously large market. It's like the new cloud basically”
“if you're just selling like tokens for dollars, there's no switching cost, you know? People can just switch a different provider”
“I actually think in the long term there is so much gross margin opportunity in the space”
“I think we're very early. I think you're saying the same thing”
How the episode opens
I think there is an opportunity to create a hundred billion dollar company with a lot of specialized models in inference. There's going to be a pullback. I'm okay with that because I think the long-term opportunity is so big.
It's like so large. Today, I'm thrilled to welcome Eric, co-founder and CEO of Modal. Now, Modal just hit a billion in AR.
They just raised $750 million led by Excel and they're priced at $15.75 billion today. But today we dig into the inference market, what to come, what no one sees that everyone should see, and so much more.
We've lost new grats going to, you know, some of the labs for like 20 plus million. I mean, these are exceptional people. Exceptional. I don't want to sell this company cuz I would just go and start Modal 2.
Ready to go. Eric, listen, dude. It is so great to have you on the show. You know, I was sitting downstairs with you chatting before and I was just like, you can feel when it's going to be a good show because people are themselves. And that was an amazing conversation and we're going to have a lot of fun. So, thank you for joining me.
It's so exciting to be here. I mean, I've listened to so many episodes. It's it's kind of surreal to be inside of it right now. — That's really awesome. As I said to you, you you could have text me when you started the company and maybe I would have been a little bit richer.
Yeah, but you probably didn't know about me then. No, I didn't. But I would have welcomed investing in another suite. Um, I wanted to start with I was talking to Zach Frankle last night and he said about your childhood and like early indicators of success and like exceptional ability.
From the episode’s captions: roughly the first two minutes, lightly cleaned. Keep watching on YouTube →
Topics
- open source llm inference
- compute cost
- gpu pricing
- open models commodity inference
- ai infrastructure market fragmentation
Companies and tickers mentioned
- NVDA
- Modal
- Spotify
- Accel
- Nvidia
- OpenAI
- Jev
- TurboPuffer
- Nvidia CUDA
- OpenRouter
Watch the episode
Summary, takeaways and quote selection by Octagon. Quotes are short excerpts; the full conversation is in the episode.
More from 20VC
-
Sequoia Leaves London, Close Scout Program & End Evergreen Fund | NASDAQ Loses $2.5T in 24 hours -
Mistral CEO: Why OpenAI & Anthropic Will Come for Your Business | Are Chinese Models Truly Safe? -
Inside WashU's Endowment Fund | SpaceX Made Up 20% of Out AUM | Scott Wilson -
Instinct Worth 30BN? | Microsoft Cuts 27,000 Jobs While Meta Taps Equity Markets for $95B CapEx -
Satya Nadella on Open vs Closed Models: Who Wins | Why Companies Need to Own Their Own Intelligence -
Inference Will Be $10TRN Market: What No One Sees | Software Investors are F***ed | Naveen Rao
20VC, this episode and its artwork belong to Harry Stebbings. Octagon is not affiliated with the show; this page is an independent summary.
