Short answer: Jordan Metzner says price matters more as AI spend scales. Mitesh Agrawal says Positron gives users a slider between fast interactivity at higher cost and the largest models with bigger batches at lower cost.
Watch this answer from Episode 37 (starts at 16:13) · Episode 37 notes and transcript
From Episode 37: Why AI Inference Is So Expensive (And How Positron Is Solving It), starting at 16:13.
Jordan Metzner [16:13]: Yeah. It's super interesting. Obviously, I mean, not and not every use case has like the same need to have like the best at the most expensive and the most powerful, like, a lot of use cases are offline or can be done slowly or in the evening time where you still want a lot of context, but you're not focused on speed and need trade off speed for price or for other types of trade off. So I think like Yeah. As we start to see more and more, real world applications and implementations. Right now, I mean, like, for our personally, excuse me, we just try to go to, like, whatever the best model is today, because, like, we're just hungry for power and use case and implementation, but also because, like, our aggregate spend across models is low enough that, like, we can have that. But, like, as our business grows or even as, like, we're implementing AI and other businesses, like, as you start to see that scale, then, like, price consciousness becomes, like, much more integral to the decisions you're making. Right?
Mitesh Agrawal [17:08]: And this is the key point that you're making, Jordan, is Positron, like fundamentally, we are not saying that either fast speed is bad or low price is good or vice versa. Low price is bad and fast speed is great or low price. What we wanna create and what we are creating is we wanna give the slider bar to the end user. Like, tomorrow, let's say that, you come out and say, like, look, I only care about interactivity, which is the speed per user per token. Right? Like, want, like, a thousand or 5,000 tokens per second for a gpt-oss model. Great. That's what we're building at Positron. So if you want that, great. You'll just have to pay more because you'll have to use more chips to get that interactively for per user. But the other way around, you're saying, well, I'm running all of this agentic, hundreds of agentic workflows, and they need to use the best larger model, the largest model out there with maximum context size, but I wanna keep the budget to, I don't know, a 100,000 or 10,000, whatever the budget that you wanna keep to. So then you have to have a slider bar of like, okay, we need to keep it within that budget, so we need to figure out what the interactivity what is the speed per user or per agent, what is the speed and number of tokens that we can output, that we can keep it within the budget, and that's the important factor that Positron AI is really addressing that only to my knowledge, only GPUs can do that today. If you look at specific inference ASICs or custom silicon, they're all like, yep, we're only gonna be super fast, but it's gonna be super expensive, and that's the only option you get. Or, other way around. But for us, it's actually a slider bar in terms of the way our architecture works is you can go after super fast interactivity, but it'll cost more, or you can go after very large models, but lower interactivity if you wanted, higher batch sizes, but you're optimizing for cost if you wanna do that.
Mitesh Agrawal [19:00]: And that's that's kind of the interesting always, give and take there from
Jordan Metzner [19:05]: From Yeah. Awesome. Yeah. Well, I think not just optimizing for cost. I think we'll find use cases where both are applicable irregardless of cost. I mean, it might be just a better use case to operate, that those types of jobs that way as they become, like, better and more efficient or whatnot. Yeah. Okay. Let's jump into the news, Sam. What do you think?
