Short answer: Mitesh Agrawal says Positron is making the first terabyte-plus memory chip, up to two terabytes. A 10 trillion parameter model needs about five terabytes for its weights, which fits in one Positron system instead of many GPUs.
Watch this answer from Episode 37 (starts at 1:07) · Episode 37 notes and transcript
From Episode 37: Why AI Inference Is So Expensive (And How Positron Is Solving It), starting at 1:07.
Mitesh Agrawal [1:07]: Yeah. Sam, Jordan, thank you so much for having me. Super excited to actually chat with you guys and see what you have built. That's always the fun part. But look, Positron AI is is really, really focused in the world of silicon. So we are designing and building silicon for inference applications. And, like, if you're following the news, if you've not been under a rock and have been seeing, inference is where all the spend is going and kind of where all the applications are growing, whether that's code generation on the enterprise side, obviously, world building models and video models have really come up and shot to the top in terms of discussion forums, in terms of where the next wave of compute is gonna be used in inference. And then in general, just, the big news this week, and as is AI, there's every week there's a big news, the potential launch of Anthropic's Mythos model, right, which is rumored to be a 10,000,000,000,000 parameter model, which is very relevant to Positron AI, because, look, what we are designing and building, how it is different is, we are really focused on the kind of memory scaling problem, or memory wall problem, both from the speed of the memory, or what is called the bandwidth, and then the capacity of the memory attached with each chip, so how much memory per chip that you have.
Mitesh Agrawal [2:24]: And Positron is making the world's first terabyte plus memory chip, so having capacity of memory up to two terabytes. So just in terms of what that translates into is when you're talking about your phone or your laptop and you say, oh, I have this much storage, have this much memory, obviously more memory means more tabs can be opened, more more things can be done, and faster memory means processing is faster. Similarly in the world of silicon inference use cases, more memory means like, if it's a 10,000,000,000,000 parameter model, even at quantized version of it, that's five terabytes of space that you need or memory that you need just to store those 10,000,000,000,000 parameter, the weights of the 10,000,000,000,000 parameter, just the weights of those. Right? So if you need five terabytes, well, if you're using, let's say, a GPU which has few 100 gigabytes of memory, so point 3.4 giga terabytes per GPU, you need to spread that out among that many more GPUs.
Mitesh Agrawal [3:21]: Whereas, if you have a two terabyte system card that the Positron has in a our system is, like, over nine terabytes of attached memory capacity to the card, you can put that into a single system. Right? So that's kind of it can give you a huge amount of cost advantage there. Now, as people who run inference will tell you, and there's a lot of metrics that go into inference, memory capacity. So number of parameters, number of batch size, speed of tokens, speed of interactivity. So all those things has to be kinda mashed and those results into different type of cost structure, but that's kind of our key innovation is like how are we utilizing the speed that is available to the maximum effect. So memory bandwidth utilization, MBU, how are we maximizing that? And then how are we maximizing memory capacity? So in summary, that's what Positron AI is building and we are really excited. We've already launched our first gen product that proved out our MBU part of the story, how are we leveraging maximum memory bandwidth utilization.
