Positron AI CEO Mitesh Agrawal on why inference is expensive, why his chips hold up to 2 TB of commodity memory, and his speed vs cost slider for users.
AI inference is expensive because the biggest models and longest contexts need huge amounts of fast memory, and GPUs spread that memory across many chips. On Episode 37 of Built This Week, Mitesh Agrawal, CEO of Positron AI and former cofounder and COO of Lambda, explains how Positron designs inference chips with up to two terabytes of commodity memory each, and why he wants customers to choose their own trade-off between speed and cost.
Mitesh says inference is where the spend is going: code generation in the enterprise, plus video and world-building models. Positron designs silicon for inference and focuses on what he calls the memory wall, both the speed of the memory (bandwidth) and how much memory sits on each chip (capacity).
And Positron is making the world's first terabyte plus memory chip, so having capacity of memory up to two terabytes.
— Mitesh Agrawal, Episode 37
His example was the rumored 10 trillion parameter Anthropic model in that week's news. Even quantized, he says, that is about five terabytes just to store the weights. On GPUs with a few hundred gigabytes each, the model has to be spread across many more chips. Positron's system, which he says has over nine terabytes of attached memory, can hold it in a single system, which is where the cost advantage comes from. According to Mitesh, the first-generation chip proved out memory bandwidth utilization, and the second chip, due to tape out later this year, targets both bandwidth and capacity.
Watch Mitesh Agrawal on GPU vs inference chip costs in Episode 37 (starts at 8:20) · Episode 37 notes and transcript
Positron's original cofounders are Thomas Sohmers, a Thiel Fellow who Mitesh says built his first chip at 17, and Edward Kmett, a software programmer. Mitesh joined in January 2025 from Lambda, where he was COO and cofounder.
He gave two reasons. He sees AI as built on three technology stacks: energy, models and data, and silicon, networking and compute. Cloud services sit on top of those, and he wanted to go down a layer to work on the core technology. The second reason, which he called a little selfish, is that in his view much of the value in AI has gone to that third layer: Nvidia and AMD, networking companies like Credo and Astera, and power companies.
For the episode, Sam built a simple AI inference cost simulator that runs a chat app and then an image generation app on GPU infrastructure, watches effective margin collapse as usage scales, and then switches to Positron. Mitesh said it was close to something Positron prepares internally, and compared it to SemiAnalysis's InferenceX tool.
He added the real-world constraints. Inference is limited by energy, meaning how much power you have available, and by supply chain, meaning whether you can produce enough chips. Positron learned through its go-to-market that customers care about running in data centers GPUs can't go into, such as air-cooled facilities, and about getting more tokens from the same energy. Mitesh says that for the same power, Positron can produce two to five times more tokens.
Mitesh was careful here. He called Nvidia, if not the world's smartest company, one of them, and said its chips work for training and most inference. But for a few workloads he says the architecture is not the most efficient: very large models, very long contexts, and image or video generation where the output is large. That is where he says Positron targets two to four times the performance of Nvidia's latest. He also noted that Nvidia's own moves into inference-specific chips have helped validate the market.
Code generation is a big one, because it relies so heavily on caching.
90% of total tokens is just caching. So you need massive amount of memory where you can do hot caching to drive fast performance
— Mitesh Agrawal, Episode 37
Jordan pointed out that people keep putting more into the prompt: first the whole code base, then best practices, and so on. Users are hungry for more memory in the prompt because it makes tools easier to use or perform better.
Mitesh agreed and brought up Google's TurboQuant compaction paper, which he says drew so much attention that memory stocks dropped the next day. He thinks that reaction misses the point. By the Jevons paradox, if you can fit more cache into the same memory, people will just want more: the repository, then best practices, then every good piece of code on GitHub. Add agentic workflows, where one person runs tens or hundreds of agents feeding each other's context, and he expects demand for memory to keep growing even beyond Positron's two terabytes.
The fastest memory, HBM, goes to Nvidia, Google's TPUs and AMD first. Mitesh says a startup in that line could have a phenomenal chip and still not get the parts to build it.
Is HBM the best memory in terms of performance? No doubt about it. What NVIDIA is using is the best memory out there, but first of all, you can't get access to it.
— Mitesh Agrawal, Episode 37
So Positron uses commodity memory, the kind that goes into phones and laptops. Mitesh says contract prices for it have gone up about two and a half times, which will affect Positron's pricing, but the allocation and capacity are there and more fabs are coming online. Commodity memory also lets Positron put more capacity on each chip, and the architecture is designed to close the gap on performance per dollar and per watt.
Jordan noted that not every job needs the fastest, most expensive setup. Many run offline or overnight and need a lot of context but not speed. Today we just use whatever model is best because our total spend is low enough, but he says price becomes much more important as AI use scales across a business.
What we wanna create and what we are creating is we wanna give the slider bar to the end user.
— Mitesh Agrawal, Episode 37
At one end, a customer who only cares about interactivity, say thousands of tokens per second per user on a gpt-oss model, pays more because it takes more chips. At the other, a team running hundreds of agentic workflows on the largest model with maximum context can set a budget and accept lower speed with bigger batches. Mitesh says other inference chips tend to offer only one end of that range, and that to his knowledge only GPUs offer the slider today.
In the news segment, Mitesh also explained why agents are bringing back the CPU. He says Positron is one of the first accelerators integrated with Arm's own CPU, that Intel has raised CPU prices twice in six months and still sells out, and that as people go from tens of agents to thousands, the CPU is back in business. For more on chips and data centers, see our AI in semiconductors and AI infrastructure pages, and Episode 56 on running open weights models on your own hardware.
Mitesh also gives his take on OpenAI's secondary market and Arm's first CPU in Episode 37 of Built This Week. The full transcript is on the episode page.
