Episode summary
I Used AI to Data Mine 900,000 Financial Backtests: Here's What Happened
The Acquirers Podcast
· · 6 min · Hosts: Tobias Carlisle, Jake Taylor
Summary
The discussion centers on how AI changes backtesting by letting the host run a massive search over portfolio combinations and parameters. He says he fed his backtest data into Claude, connected it to his backtesting software, and let it run 916,000 backtests across 1,217 parameters. The point was not to find a magical answer, but to see how fragile optimization can be when the sample is split into training and test periods.
The host argues that most apparently optimal portfolios do not hold up out of sample. He describes testing different mixes of assets across decades and countries, then comparing which allocations ranked well in the first half of the data versus the second half. Even when a portfolio looks best in one period, it often falls far down the rankings in another. The guest pushes on the same idea by joking that the computer is doing with an earth mover what a person used to do with a spoon.
They extend that skepticism to classic diversification frameworks from Ray Dalio and Harry Browne. The guest notes that simple, broad mixes like the permanent portfolio can do reasonably well, but he also points out how little truly catastrophic market history there is to test against. The broader conclusion is that backtests can inform judgment, but they cannot eliminate uncertainty or prove that a strategy will survive the next regime shift.
Key takeaways
- The host says Claude ran 916,000 backtests across 1,217 parameters, turning manual testing into a large-scale data mining exercise.
- The host argues that splitting data into in-sample and out-of-sample periods still leaves a big gap between what optimizes and what persists.
- The guest says simple all-weather mixes can do fine, but there is no clean answer once you test across many countries, decades, and objectives.
- The guest warns that major crash data are too sparse to support strong conclusions about how portfolios will behave in the next true drawdown.
Notable quotes
“I have lots of back test data and I fed it all into Claude”
“It ran 916,000 back tests testing 1,217 different parameters. Like it's a data mine”
“One of them is that you data mine in the first half of the set and then you test in the second half of the set”
“There's no good answer”
“there just haven't been that many of them. And so to draw too many conclusions from a small sample sizes is bad stats, right”
How the episode opens
So, I think it's interesting in the context of back testing. So, I was saying to to Jake before he came on that I have lots of back test data and I fed it all into Claude. And then I asked Claude to run these various back tests and I gave it access to my back testing software as well.
And where I would run dozens, maybe, over the course of an afternoon. It ran 916,000 back tests testing 1,217 different parameters. Like it's a data mine.
Yeah. — And then I said to it, "It's a data mine." — You've been data mining with a with a spoon. — Right. — Now you've been given a, you know, an earth mover. — Right. Exactly right. — Yeah. — And so but I I have some rules for data mining. One of them is that you data mine in the first half of the set and then you test in the second half of the set. That's got its own problems as well, but it's better than nothing. And it's very hard to find any thing that persists from one set to the next.
Mhm. — I think that's that's kind of interesting, but the things that are optimal it optimizes something in the first set it doesn't work in the second set. So, I was looking at it in terms of asset allocation. Because you know, Ray Dalio has that view that you take uncorrelated return streams and that's you get enough of them and then you you spread your bets over those and you There's always something that's working.
Lever the bejesus out of it. And — That's right. — And then uh there's the Harry Browne who wrote uh "Fail-Safe Investing" and out of that comes the permanent portfolio.
From the episode’s captions: roughly the first two minutes, lightly cleaned. Keep watching on YouTube →
Topics
- ai backtesting
- market crash preparedness
- value investing
- portfolio hedging
- rebalancing and drawdowns
Watch the episode
Watch “I Used AI to Data Mine 900,000 Financial Backtests: Here's What Happened” on YouTube
Summary, takeaways and quote selection by Octagon. Quotes are short excerpts; the full conversation is in the episode.
More from The Acquirers Podcast
-
How Artificial Intelligence is Changing Biotech & Drug Discovery -
The AI Singularity Is Coming. Is This $400 Stock Is Still Mispriced? -
The Quality Growth Playbook for Biotech | David Ferreiro -
Entropy, Disney, and Nvidia: Is AI Circular Financing a Trap? -
Small Value is Back. Is Mag 7 Finally Rolling Over? -
The Best Business You've Never Heard Of? | Todd Wenning
The Acquirers Podcast, this episode and its artwork belong to Tobias Carlisle. Octagon is not affiliated with the show; this page is an independent summary.
