Short answer: Amar Goel says Nvidia still leads for training while others catch up on inference. Jordan Metzner suspects Amazon held back sales because supply is limited and sees cheaper chips handling simple tasks routed through tools like Bedrock.
Watch this answer from Episode 23 (starts at 23:38) · Episode 23 notes and transcript
From Episode 23: OpenAI Code Red? The New AI Reviewer Changing Everything, starting at 23:38.
Sam Nadler [23:37]: Well, yeah, talking about GPUs. Amazon releases the new AI chip, at, re:Invent, I think, yesterday, was it, or two days ago, the Trainium three, supposedly four x faster and a cheaper way to train big models. I'm a very bullish Nvidia bull, and I think most people are, to be honest. But, what are your thoughts on Amazon's new chip? Jordan, by the way, Amar is a, Amazon alum, so he's got some, usually, some strong feelings about, whatever's going on at Amazon.
Amar Goel [24:18]: Got it. Interesting. Yeah. I mean, I think if you take a step back, there's a number of folks now. I mean, Google's also got, the TPUs. And so, I think, like, they also now have gone from saying, hey. We're not gonna just offer this in Google Cloud. We're gonna also offer this to other people to set up in their own data centers. Like, supposedly, they're talking to Meta about a, multibillion dollar kind of deal. AMD is, of course, out there as well. So, more and more people are, trying to peel off a piece of that GPU market. I guess I'm a little surprised it took AWS and Google Cloud, like, so long to, like, say, hey. We'll make these chips available to anyone, which then makes me wonder maybe they're not that great at the price performance curve, compared to NVIDIA chips. I mean, generally, what I've seen is that heard is, like, from when you're doing training, NVIDIA chips are still, like, blow everybody else out of the water. But when it comes to inference, other people are starting to catch up, and the inference workloads are growing, very, very dramatically.
Amar Goel [25:39]: I mean, the sense I still get is that TPUs are still well ahead of, like, the training and chips from Amazon. But then, of course, you've also got folks like Groq with a q and Cerebras and SambaNova that are trying to do really fast. So so, anyways, I think, like, it feels like you would think in the next year if NVIDIA's dominance or market share was to come down that it would happen in the next year. But so far, they've done a good job of kinda keeping a lock on the marketplace.
Jordan Metzner [26:10]: Yeah. So I agree with, like, kinda most of your points there. I would say that probably Amazon, didn't start to sell it because they didn't have any. Right? Like, these chips are three nanometers. Right? And so, like, how long does it take to, like, fabricate them and, like, get them in data centers and, like, test them and make sure they even work for their own workloads first and, how much money do they save by offloading, third party workloads to other chips, to their own chips. And so, I would guess that there's probably some supply, plus tariffs. Right? So there's probably some actual supply chain issues that, like, actually cause them to get these things, like, up and running. I do think that, like, as we see this logarithmic curve of these models that, I don't think any of these LLMs have done a good job of, I would say, like, routing, taking basically, like, simple tasks and sending it to, like, the cheapest model possible if it's, like, cheap enough and fast enough. And over time, I know, Amazon has Bedrock which is kind of like this, like, LLM kind of routing platform. Right? So you can start to see over time, like, for certain tasks, you can probably offload those tasks to a cheaper processor, to a cheaper GPU, right, and be able to save a significant amount of money especially if it's things like maybe like ETL pipelines and things like that where, like maybe they become really common tasks like AWS alternatives. My guess is like they, again, they just don't have enough.
Jordan Metzner [27:31]: It doesn't seem like anybody has enough. I mean we try to buy GPUs online and it's very they're very hard to come by and they're hard to find. Whether Amazon's, like, TPU is gonna be able to compete with, like, NVIDIA's top line chips, it doesn't look like it. But I don't know if that's really their goal in the short term. I mean, maybe they can just offline tasks that are, being run by servers that are very expensive today that can move over. Right?
