Short answer: Mitesh Agrawal calls Nvidia one of the world's smartest companies, but says very large models, long context, image and video generation and code generation, where 90% of tokens are caching, allow 2x to 4x gains.
Watch this answer from Episode 37 (starts at 9:51) · Episode 37 notes and transcript
From Episode 37: Why AI Inference Is So Expensive (And How Positron Is Solving It), starting at 9:51.
Mitesh Agrawal [9:51]: I do wanna know, like, what the data is because, like, obviously, I see, like, effective cost per million is, like, $10,000 on a GPU, 3,600 on Positron AI. So this is a great advertisement for us, so thank you, Sam, for building it. But I think one thing I would wanna also say, like, look, I wanna be grounded about this pack, I think to your point that you kind of hinted on certain use cases is where we drive those efficiencies. Right? I think, look, let's be humble. This sense, NVIDIA is the world's largest company in the sense of market valuation, but in my mind, they're also, if not the world's smartest company, one of the world's smartest companies. Right? So so they are building out something that is out there for training applications and for majority of inference applications, but there are certain types of inference applications where their architecture is not the most efficient. It will work without a question, but, you can drive more efficiencies. I talked about very large models, very large context lengths, video generation or image generation where the output is so much larger where you need a lot of memory to store it.
Mitesh Agrawal [10:57]: Those kind of applications where you can drive that Pareto curve of efficiency to be much better, and that's how we are targeting. It's like few applications where we can really drive two x, three x, four x performance improvements over the latest Nvidia. I mean, even Nvidia finally kind of validated the Inference chip market a little bit by their acquisition, by their announcement off of specific chips. Right? So I think you are seeing that more and more, and I think I just wanted to add that to this, but you correctly highlighted, like, you look at code generation is massively caching. Like, 90% of total tokens is just caching. So you need massive amount of memory where you can do hot caching to drive fast performance, and that's kind of where, we can really stand out, for example.
Sam Nadler [11:42]: Planning a company off-site? Don't waste weeks juggling hotels, flights, and agendas. Offsiteio handles everything from location scouting to activities, all at a simple upfront price per person. Save time, save money, and give your team an unforgettable experience. Go to.com and get your free proposal today.
