Episode summary
This Is How To Tell if Writing Was Made by AI | Odd Lots
Odd Lots
· · 51 min · Hosts: Tracy Alloway, Joe Weisenthal · Guests: Max Spero
Summary
The episode argues that AI-generated writing is already widespread enough to be a practical problem for journalism, teachers, platforms, and ordinary readers. The hosts say AI text is often polished and grammatically strong, but still has a recognizable feel, which creates both skepticism and new reputational risks for people whose work is judged by style. They also argue that simple heuristics are breaking down, because high-quality prose no longer reliably implies a serious human author.
Max Spero, founder and CEO of Pangram Labs, explains how his company tries to detect AI text by training on millions of human examples paired with AI-generated counterparts. He says Pangram’s model looks for subtle decision patterns in word choice, phrasing, and structure, and that its published system is designed to distinguish human writing from AI-assisted writing as well as fully generated text. He says the company’s false positive rate is about 1 in 10,000 and its false negative rate is around 1% in the general case.
The discussion also turns to the scale of the problem. Spero estimates about 40% of the internet is AI-written from a page perspective, with much higher rates on some platforms and lower rates on others, such as Reddit. He says the biggest fix may be norms: people should stop treating undisclosed AI output as an acceptable substitute for original writing.
Key takeaways
- Max Spero says Pangram Labs sees about 1 in 10,000 false positives on human text and around a 1% false negative rate in the general case.
- The hosts argue that AI writing is often good enough that grammar and polish no longer reliably signal human authorship.
- Spero says Pangram trains on millions of paired human and AI examples to learn subtle decision patterns rather than hard-coded tells.
- Spero estimates about 40% of the internet is AI-written from a page perspective, with Reddit at a little over 10% today.
- Spero says the biggest solution may be a social norm against undisclosed AI output rather than technology alone.
Notable quotes
“It's about 40%. Oh, how do you. Based on what you just read. How'd you get that number”
“the majority of the population, like, doesn't know where to put a comma within the sentence”
“we've built out, really comprehensive evals. Okay. And so our evaluations, there's two kinds of errors”
“our number right now is about 1 in 10,000. Okay. So if we scan, 10,000 documents, up on average one”
“we're learning the patterns and how, like these frontier models make these decisions”
How the episode opens
Using your model, are you able to quantify like what percentage of the internet at the moment is I slop? It's about 40%. Oh, how do you.
Based on what you just read. How'd you get that number? So a lot of the internet is just like SEO written articles and like much of that.
Yeah, it's articles written for for search, basically, so that, your website comes up more often. Search because it's targeting certain keywords. And a lot of that industry has switched over to using AI because then instead of having to pay writers, you could churn out articles for pennies on the dollar.
But I think that kind of results in a lot of the internet being AI written. It's a little bit. It is also kind of platform dependent.
So this is just it's about 40% from like a internet page perspective. About a year and a half ago, we looked at medium and found that over 50% of newly written medium articles were AI generated, which was a crazy high number. What about Reddit?
Reddit about it was 7% a year ago, I believe. A little over 10% today. Hello and welcome to another episode of the Odd Lots podcast.
I'm Joe Weisenthal and I'm Tracy Alloway. So, Tracy, you know, you ever come across some writing you can't articulate exactly why, but you're like, I'm pretty sure I wrote this. Does this happen too much?
So full disclosure, I haven't really thought about it that much. Yeah, because the thing is, if I, I probably should think about it more, but there's a lot of bad writing out there, and I've become sort of inured to it. And I also think that, I don't know, trying to figure out whether or not something was generated by AI.
From the episode’s captions: roughly the first two minutes, lightly cleaned. Keep watching on YouTube →
Topics
- ai writing detection
- false positives false negatives
- chatbots
- student cheating
- internet content moderation
Companies and tickers mentioned
- Pangram Labs
Watch the episode
Watch “This Is How To Tell if Writing Was Made by AI | Odd Lots” on YouTube
Summary, takeaways and quote selection by Octagon. Quotes are short excerpts; the full conversation is in the episode.
More from Odd Lots
-
How a Chicago Restaurant Has Stayed Open For Nearly Three Decades | Odd Lots -
How Airlines Actually Hedge Higher Fuel Prices | Odd Lots -
Robert Friedland on the World's Monumental Shortage of Copper | Odd Lots -
Brad Setser on the US's Unusual Japanese Yen Intervention | Odd Lots -
The Tungsten Market Is Warning of an Upcoming War | Odd Lots -
These Are the Sharps Actually Making Money on Prediction Markets | Odd Lots
Odd Lots, this episode and its artwork belong to Bloomberg. Octagon is not affiliated with the show; this page is an independent summary.
