Short answer: Ivan Lee says an average chatbot query uses 200 to 2,000 tokens while an agentic task uses 3 million. He argues chatbots have crossed the chasm but agents have not, and when they do, token usage will skyrocket.
Watch this answer from Episode 56 (starts at 11:02) · Episode 56 notes and transcript
From Episode 56: He Built Siri at Apple. Now His Private AI Runs Inside the FBI, Google, and Netflix., starting at 11:02.
Jordan Metzner [11:01]: Okay. Then tell us the Jevons paradox here because then what happens? Right?
Ivan Lee [11:06]: So that's where I think there is... I personally feel like there is a lot more to... There's a lot more tokens to be consumed on the enterprise front. There are two things that are happening simultaneously. One, tokens right now, if you... Your average chatbot query is 200 to 2,000 tokens. Your average agentic task is 3,000,000 tokens. It's over three orders of magnitude more than your ChatGPT query. Right? That is why Jensen and Sam and Dario love h s. Right? But they really are that much more capable. And so you have significantly more token consumption alongside token maxing. That's why we had this kinda like... We actually got into a GPU glut earlier this year where Anthropic had to slow down its services. OpenAI had to stop working with, offering on Gemini, on GCP. We got into this glut because people were overspending. So on the one hand, as more agentic adoption occurs, we're going to need a lot more tokens. And then in parallel to that, we're still... Are you familiar with the crossing the chasm paradigm?
Jordan Metzner [12:21]: Maybe not.
Ivan Lee [12:23]: Crossing the chasm is this kind of, like, MBA chart. I'll pull up a version of it here. There's this MBA chart that everybody alludes to where every technology has to have this adoption curve where, essentially, every technology, whether you want to talk about the Internet, mobile, the microwave, the television, everything goes through innovators, early adopters. Then there's a chasm, and some companies... And some technologies never cross this chasm. There's a early majority, late majority, then think of your, friends who were the last to get smartphones. There's the laggards. So you can actually chart every single technology adoption across this same curve. Now I would argue that chatbots are here. We're we're at a billion monthly active users for ChatGPT and Gemini, and I think we're already in that early majority. It's already crossed that chasm. Contrast this with something like VR where it's still at the early adopter stage and may never cross that chasm in its current state. Right? Now I think agents, on the other hand, agents are at the early adopter stage and we haven't crossed that chasm yet. But when we do, this is absolutely going to skyrocket token usage even further because a lot more companies are going to start adopting this over their Microsoft Copilot ChatGPT tool right now, and then they're gonna use way more tokens, and it's going...
Ivan Lee [13:46]: And that same shock that hit Silicon Valley earlier this year is going to hit the rest of the world, really.
Jordan Metzner [13:52]: I totally agree with you. I think... I mean, Sam and I have both been using Rockbot a lot. I've gone... Kinda gone through some of the agents, but what I realized is it also collapses some of my software spend. Right? And so that's like a shift of, like, CPU to GPU. Right? Essentially, the same thing, like, kind of to your mentioning. Like, I don't need to pay for some software that was doing some specific thing for me in, like, some outbound sales marketing tool, and I can have my agent do it for me. And so it's just like CPU away, all shift to tokens. But again, like, does that make the token... As the tokens get cheaper to your point running on these Macs or what... Whatever the next iteration of that is. But I guess gross token consumption of mine is... Will significantly go up. Right? And same with the enterprise.
Ivan Lee [14:39]: It's it's a really interesting kind of macroeconomic situation with all these variables shifting. Right? Everybody's looking to make the models not only more intelligent but more efficient. There's obviously new GPU and hardware, research as well. There's so much money being thrown at this that, like, the scales are being tipped on both sides.
