Short answer: Sam Nadler's simulation shows GPU costs rising sharply at scale. Mitesh Agrawal says energy and supply chain constrain inference, and Positron can go into air-cooled data centers and output two to five times more tokens per unit of energy.
Watch this answer from Episode 37 (starts at 6:32) · Episode 37 notes and transcript
From Episode 37: Why AI Inference Is So Expensive (And How Positron Is Solving It), starting at 6:32.
Jordan Metzner [6:31]: Alright, Sam. Let's jump into your tool, and let's jump back to Positron in a little bit.
Sam Nadler [6:34]: Yeah. Let's let's do this quickly. I'm afraid it's gonna be pretty juvenile here. But, yeah. What is the inference problem? Training this is just a little walk through to, get our users on the same page. While training models get all the headlines, inference is where 90% of the real world AI costs happen. If any of this is inaccurate, please call me out on that. So traditional infrastructure breaks. GPUs were built for graphics, not text generation. So you end up paying for idle compute cycles, yada. The real constraint on AI, it's an economics problem. It's not limited by intelligence. It's limited by how efficiently we can run it. Okay. What you're about to see, we're going to start a real world AI application. We'll push through traditional GPU infrastructure, and then we'll switch to Positron. So start the simulation. You have, your typical chat prompt type. And let's you know, at a lower scale, it seems like it could make sense. You have a positive effective margin, but as you creep up and scale oop. Let me get this little dashboard going. It becomes and gets really expensive. This problem exasperates if you go to something like image generation.
Sam Nadler [7:54]: I can get, like, extremely if not getting the dashboard extremely expensive here. But if I system's breaking red alert. If I switch, you have a significantly less compute load, average latency, etcetera. I know this is a little silly, but I wanted to try and do something to highlight, the real difference between GPUs and what you're building.
Mitesh Agrawal [8:20]: Yeah. First of all, let me just qualify. It's not silly at all. This is actually very close to like an internal thing that we also prepare in terms of performance. And, if you have seen Inference X, which is SemiAnalysis inference, it used to be called Inference Max, but it's a SemiAnalysis token tool. They do comparison of different things, and we have something similar, but this is actually very, very good. I wonder where the data you're pulling from is. It's for the of our website or things like those, but what I will highlight is two things to your point. Look, the when you look at inference, alongside, obviously, the efficiency in the chip, it's it's it's constrained by the real world factors of energy. Right? How much energy do you have available to do it? And then second is supply chain production. Right? Like, can you produce enough chips to run it? And if you can drive efficiency or alternative supply chain, that's important.
Mitesh Agrawal [9:13]: So in your example, if you see energy usage is the one that I see, and that's one of the critical ones. We only found out by going to market, by our GTM motion, where people are like, hey, you're you know, you can go into data centers that the new way GPUs can't go into, like air cool systems, or you can for same amount of watt consumption, like, let's say you're consuming, one kilowatt hour, you can drive four times or five times or two times more tokens, well, that's way more useful to us. I think that's very, very critical to understand, in terms of having a product out there. Right? And so that's that's very very this is actually really really cool.
