Short answer: Jordan Metzner says users keep putting more into prompts. Mitesh Agrawal says Google's TurboQuant compaction paper moved memory stocks, but by the Jevons paradox cheaper context just means people want more, especially with many agents.
Watch this answer from Episode 37 (starts at 12:06) · Episode 37 notes and transcript
From Episode 37: Why AI Inference Is So Expensive (And How Positron Is Solving It), starting at 12:06.
Jordan Metzner [12:06]: Alright. Let's jump into Positron a little bit. So, yeah, as you were saying, kind of these inference use cases in which large memory is needed, and obviously, like, memory is relative as you're saying. Like, people are trying to throw the whole code base into the prompt. Right? You know? And then, obviously, I mean, the next wave of that is throwing, like, kind of like the entire Python, best practices. You know? I mean, it's just like, whatever. It just keeps going, like, deeper and deeper into, like, how big you can throw into this memory prompt. Like, it seems that, users are hungry for more memory in prompt, and that's not going to stop because, it allows for either easier usage from the user or better performance or a combination of the two, maybe at the sacrifice to your point of, like, power or cost or something else. Right?
Mitesh Agrawal [12:55]: Yeah. A 100%. And Jordan, like, one of the interesting kind of conversations that are happening in the current world, is around kind of compaction. Right? And we are seeing the TurboQuant from Google capture people's attention, in fact, so much that, freaking memory prices dropped the next day, and I was like, oh my god, that is insane that a research paper driving down stock prices, it's like in the current world we live in, it's both awesome to see, a little bit scary how quickly people react to it, but the point there is actually, know, it's actually really good for memory prices or memory stocks or memory consumption. Know people know fundamentally this has been like beaten to death around Jevons paradox around how if tokens get cheaper, people consume more and obviously improve it. It's the same logic, know. If you figure out how to do software compaction on how much cache you can fit in or you know, how much bigger prompt you can fit in, people, to your point, Jordan, they'll just want more. Like, okay, I'm uploading my entire code repository, great, then I wanna after that I wanna upload what are the best practices. Then after that I wanna scrape every single GitHub and see what are the best code written and then upload or that for a different make bigger and bigger context length or things like these.
Mitesh Agrawal [14:08]: Right? And then you have all this we haven't even talked around the agentic workflows, right, where you have each individual running tens, hundreds of agents
Jordan Metzner [14:15]: Concurrency, yeah.
