Short answer: Using an Artificial Analysis chart, Ivan Lee says agents only started working with Claude 4.5 in December 2025, and every open weights model shown clears that bar; only 5-10% of tasks need top frontier models.
Watch this answer from Episode 56 (starts at 9:01) · Episode 56 notes and transcript
From Episode 56: He Built Siri at Apple. Now His Private AI Runs Inside the FBI, Google, and Netflix., starting at 9:01.
Ivan Lee [9:00]: Like, we've had clients who have bought millions of dollars of new hardware just to run their private AI solutions, but I think that cost just keeps plummeting over the next couple of years as you get more powerful models that can run on increasingly commoditized hardware. Let me take a moment. I wanna actually share a slide that kind of speaks to the gap in model performance. This slide is a snapshot of a very powerful kind of benchmarking site called artificial analysis, but I wanna draw your attention to the two black lines. First of all, the lower black line, this is GPT four o. This is a model that was deprecated at the beginning of this year, but a lot of people lamented because they actually really liked it. It was that first model that really started feeling kinda human and really started being able to answer questions legitimately. Legitimately. There are so many tasks that can be solved with the four o model, and you can see how low that is sitting here in amongst its peers. Right?
Ivan Lee [9:58]: The second model is... Here in Silicon Valley, we've been selling agents for a couple of years. But realistically, pragmatically, agents didn't start working until Claude 4.5, until December 2025, and that's the second bar. Everything above that is already at 4.5 or smarter. Right? So here's another view of the same picture, but every single blue model is open weights. That means that DeepSeek 4.1, MIMO... I don't think this, when I took the screenshot, Meta's MUSE hadn't been tested yet, but every single one of these clears that by a significant margin. So, again, there will be 5%, 10% of tasks that should be sent to 5.5 max, fable, astral, whatever, mythos. But there's so many things for a lot of these clients, a lot of low and middle hanging fruit. We could pause all model development now for the next decade, and we would still be gathering a lot of improvements across the enterprise just using these open weights models alone.
