Short answer: Mitesh Agrawal says HBM is the best memory, but a startup cannot get it behind Nvidia, TPU and AMD. Positron uses commodity phone and laptop memory, whose contract prices rose 2 to 2.5x, but capacity is available.
Watch this answer from Episode 37 (starts at 14:17) · Episode 37 notes and transcript
From Episode 37: Why AI Inference Is So Expensive (And How Positron Is Solving It), starting at 14:17.
Mitesh Agrawal [14:16]: Exactly talking to each other, feeding information into each other's context, right, so it's never ending thing. That's why even us having that one, two terabytes of memory, I think people will want even more, and I think the more important thing there is for us is the way we have achieved that is we are not behind the lines of Nvidia and TPU and AMD to get HBM. Right? There is I cannot say it with a straight face to you here and say that, oh, yeah, if you're using the same type of memory that NVIDIA is, that we can get access to it. Even if you take me at my face value that our chip is phenomenal and our architecture is phenomenal, great. Can you fabricate it? Can you actually get the components to it? And if I'm using the same memory components as NVIDIA, like, good luck. You basically say thank you and good luck to you, and then, you're not there. But right? But that's a big thing. We are not using the HBM and CoWoS that NVIDIA uses. We are using commodity memory, the things that go into your phone and to your laptops. Now even there the prices have fluctuated so much, they've gone up by in the contractual pricing world two and a half x, so our pricing will get impacted, but at least allocation is there, capacity is there, and more fabs are coming online for it. So this is kind of the interesting balance that you have to do between fundamentally technology maximization and really physical constraint.
Mitesh Agrawal [15:39]: Is HBM the best memory in terms of performance? No doubt about it. What NVIDIA is using is the best memory out there, but first of all, you can't get access to it. Second of all, by using that memory, you can't have as much of memory capacity as you can have by using some of the commodity memory things that we are using, and then through architectural means we're trying to make that commodity memory perform better and get it closer in terms of performance per dollar and performance per watt perspective to that best memory out there. Right? So these are kind of interesting topics that go into description of it, of designing a product here.
