Episode
59

1M+ Data Points: How Snorkel Builds the Data That Frontier AI Labs Train On

Published on:
Nov 20, 2026
Listen on:
Watch this Built This Week episode on video
Subscribe on YouTubeA new AI build every week

Kartik Mathur: We are past that time where, you know, you can just scan the web and give the data. That's not good enough. Models are already much smarter than that. To push the models further, you need really, really higher quality data.Song: Built This Week, breaking it down. Built This Week, we show you how. A fresh idea, a clever tweak you locked in. You built this week.Sam Nadler: Hey, everyone, and welcome to Built This Week, the podcast where we share what we're building, how we're building it, and what it means for the world of AI and startups. I'm Sam Nadler, cofounder here at Rise Labs. And each and every week, I'm joined by my friend, cohost, business partner, Metzner. How you doing today, Jordan?Jordan Metzner: Hey, Sam. How's it going? Another exciting week coming on the edges of another new model coming out from OpenAI this week. At time of recording, we got Anthropic IPO as well coming up tomorrow. So, yeah, another crazy, exciting, diverse week in the world of AI, but even more excited about today's guest.Sam Nadler: Super excited about today's guest. And before I introduce him, please don't forget to like and subscribe. I believe we just crossed over 70,000 subscribers on YouTube. Exciting milestone for us. So hit that notification button.Like and subscribe. New episodes out every Friday. And with that, I'd like to introduce Kartik, the director of engineering at Snorkel. We're gonna dive into Snorkel, understand what they do, how they're doing it. But before I do, Kartik, why don't you just give us a a brief introduction of who you are?Kartik Mathur: Hey, Sam. Hey, Jordan. Very excited to be here. I am Kartik. I am director of engineering at Snorkel here.I am currently leading the AI infrastructure team here at Snorkel. Little bit about me. I started my career at AMD. I was building the compute infrastructure for chip design and verification, and then I switched to couple of startups. I worked at Blue Data building the training AI and deployment infrastructure, which was acquired by HPE.Then I worked at Domino, where I led the model serving and monitoring team. And most recently, right now, I am at Snorkel AI, building, the, you know, agent tech factory for Snorkel's, you know, data platform. Great. So letSam Nadler: me let me ask some super basic high level questions about what Snorkel actually does. I think, you know, do you guys... Snorkel's taking a different approach than, you know, the Mercos and Micro ones come... Almost coming selling the full dataset. But kind of what what is it exactly that Snorkel does and how does it different from the other players in the market?Kartik Mathur: Yeah. So Snorkel is a research like Data Lab. Like, we have always been in the space of generating high quality datasets for AI. And right now we are doing it for Frontier AI. So what that means is datasets evaluation environments benchmark, you know, to push the Frontier in.We almost act like a sort of research design partner. So we work very closely with them. And it's what we call is like a recursive self improvement loop where, you know, we... Our agent tech platform and the, you know, subject matter experts, they work on the platform to curate and deliver these datasets. There is constant feedback, and then we work closely with the customer, in delivering,Sam Nadler: you know, these datasets to push the Frontier. And what's hot right now? Like, what is the datasets that are, you know, hot for the Frontier Labs right now? And you know, there's all these ads for businesses to provide their, you know, historic code bases. Is it is it really niche specialized?Like what what's kind of top of mind November 2026?Kartik Mathur: Yeah. Lot of... So we play mostly in a very specialized domains like obviously coding software engineering. We are constantly seeing more and more demands for more complex long horizon kind of use cases. Then the multimodal is becoming a big thing like multimedia kind of realty datasets.And then lot of data around the finance industry, legal. Right? Lot of very, very sophisticated data is being asked by Frontier. Physical AI is another thing that we are starting to hear like, you know, more real world kind of robotic kind of use cases and that... Those are the things.Sam Nadler: Yeah. Interesting. And like, how has the shift from humans providing the data and humans providing the feedback to AI creating it? Is it shifting rapidly? Is it...How has it evolved over the last, yeah, two years, year and a half?Kartik Mathur: Yeah. I think at Snorkel, have always been like human will be in the loop, because, like, you know, they are kind of the specialist as a decision maker, but lot of AI and agents are complementing human. Right? That's what we are building like the agent tech factory, like, what we call Snorkel's data platform is like, data factory. So to deliver high quality data points is is a very sophisticated process like with lot of moving parts.You know, you have to set up series of evaluators, you know, a lot of infrastructure components, and a lot of things can go wrong. So it requires a lot of hand holding right now. So that's what the agent tech factory is doing. We... That we are building a fleet of agents.It's a self serve platform that you can build agents to automate a lot of these things. And then while you are working on building a data point, you can reason with these agents. You can push the agents. You can provide feedback with the agent, and these agent take that feedback and then continue to work in the background. To do that at scale is a challenge.Right? We are now talking about the live hundreds of thousands, millions of data points. So that's where, you know, the agent directory is quite critical.Jordan Metzner: Okay. And and then, Kartik, all the... All this data for what? Is this to train models? Is this to use open source models so that they they can be more efficient?Like, you know, all all of this data for what? Right?Kartik Mathur: It's it's all of it, to improve the quality of model agents, RL environments, you know, whatever the the Frontier lab needs, you know, we kind of partner with them and then we provide these datasets and evaluators to, you know, improve the quality of their products and models and agents and RL environments.Jordan Metzner: And and how fast are we churning through datasets? Like, I mean, I know there was the period of time they were scanning all the books, but, you know, we're not scanning books anymore because I've kind of gone through that. You know, there's the time of, like Sam mentioned, of, like, buying all these businesses to understand their workflows. Like, how long are each of these capture windows before they move on to the next dataset or, you know, what is what is the next frontier? Yeah.Kartik Mathur: Yeah. That's a very subjective, like, depends on the problem that they are they want their model to solve. So like I have seen projects where like for a very very complex legal case it could take like hours and hours you know generating this data because there is a lot of experimentation going back and forth. Right? This is like we are past that time where you know you can just scan and the web and give the data.That's not good enough. Models are already much smarter than that, right? So now to push the models further you need like really really higher quality data. So it's a lot of, you know, iteration experimentation. That's what the platform does.Jordan Metzner: Okay. Cool. And, is there is there type of, you mentioned earlier like robotics and and some other types of fields. Is there any type of specialization where you guys have found kind of opportunity to excel or has it just been kind of across the board?Kartik Mathur: Across the board right now. This is still very niche and early days for that, but we are starting to see some data around that. But most of our focus in creation of benchmarks have been lot around, you know, coding environments, RL environments, multimedia kind of environments.Jordan Metzner: Okay. And can you give us some snippets of some things you guys have learned? Maybe, obviously, not anything confidential, but some interesting data facts or points you've learned just kind of seeing the data from a 30,000 view.Kartik Mathur: Like the type of data you're seeing?Jordan Metzner: Just... I mean, yeah, in general. I mean, you know, obviously, have you have this unique opportunity to see this, like, voluminous set of data. Yeah.Kartik Mathur: I can talk, like, a little bit about the coding kind of data. We have different benchmark like a software engineering benchmark, seniors, speed benchmark. So those tasks are very complicated. You know, you have to use whole bunch of tools. You have to sometimes go to your terminal.You have to do a lot of research and then, you know, you have to keep going back and forth. So, you know, lot of that published data for the benchmark is already available. It's Snorkel's website, you know, to solve like really really complex tasks for software engineering. That's like, you know, one type of data.Jordan Metzner: Okay. And in case of software engineering, obviously, you know, there's today's data. But, I mean, you know... And obviously, all the historical GitHub repos or whatever whatever and it keeps going back. ButKartik Mathur: Yeah. I think in in today's day and age, the task is not about code generation. Right? Code generation is kind of a solved problem. It's like the solving the entire problem that an engineer would solve.Right? By lot of experimentation testing in real environment. So that requires a lot of infrastructure component because you want to simulate that environment, test it against various failure modes, you know. So for an agent to be able to do all that, you need that kind of data and harnesses. That's what kind of we deliver.Jordan Metzner: That's pretty awesome. Rise Labs builds production grade AI for the world's most ambitious companies. Supporting engineers.Song: Data scientists. AI architects. All working togetherKartik Mathur: on your problem.Jordan Metzner: While others talk about AI.Kartik Mathur: Rice Labs is already implementing it.Sam Nadler: Kartik, this may be a silly question, but, you know, obviously, there's some some big big businesses take... Tackling different ends of this space. But why aren't... Why can't the labs do this themselves? Is it just their priorities building the model, and this is, you know, this is just a better way for them to get to better models faster?Or to some degree, they're doing it themselves?Kartik Mathur: Yeah. I think I can give my take on that. I think they... Some of it, they sometimes try to do, but a they want diversity of data so they take data from different sources and b this is kind of now become a very specialized field. Right?This is not just onboard, some humans to do it. Right? This requires this entire platform you know, writing these evaluators specializing this all of it. So that's kind of they they have some trusted partners that they work with, and I'm sure they must be doing for some niche field by themselves as well. So I think it's a mix is what I'm seeing.Sam Nadler: Awesome. Kartik, I don't know if you have anything to show us. Is there anything you could demo or that would provide a little bit more color? I know it's kind of a product that may be hard to demo, but anything that you could walk us through.Kartik Mathur: I think I can quickly demo what my team has been recently working on. Okay so this is sort of the recursive loop that I was mentioning before that you know you have these fleet of agents, you have human experts generating lot of data. Right? And we are talking here about hundreds of thousands millions of data points, hundreds of active agents. Right?So to enable all this we need a very standardized way to build agents. Right? And that's what, the agent tech factory that we are building here and why we are building the platform. Like there are a lot of out of the box solutions to build agents but we experimented and prototype with a bunch but then we quickly realized that you know we needed more control, we need better governance, auditability, what actions an agent is taking, you know a lot of that. And then we want it to be like a flexible open platform we don't want to be logged into you know certain types of models or with a vendor.We are starting to see use cases where we'll work with some regulated industry so we might have to you know host this into a more controlled environment. So for all that flexibility we are building this agentic factory. So let's take an example of a technical user, like an engineer or a forward deploy engineer, you know, who's in their cloud code session at terminal, you know, and they are working on experimenting like you know, how the data points that are being generated for a given project is doing. They can list, like, what agents are available to them. Right?Then they can start reasoning with the agent. And imagine, like, for a given project where you are generating thousands of data points, there are different persona of people who are interacting with these agents providing feedback to the agent. So this is like a consistent layer where agent is constantly learning and everybody who's working on that data point of project is benefiting from them. Right? So couple of you interesting examples like, you know, a review data point of view agent like it can tell you, okay, this data point, this seems off, then you can start providing some feedback and you improve it.Why is this very fire working like this? You know, and imagine any kind of a research world where you are experimenting. But here because it's a shared agent, like everybody will benefit and the agent will learn and it will do better with, you know, future data points. Couple of example like token cost is becoming super critical now with, you know, all the LLMs usage. This is obviously placeholder data, but you can, you know, query the agent like for a given project.What is the cost? Why is it rising? What happened last week, yesterday? You know, if given evaluator is kind of costing you more and then, you know, you can go deeper into that. So you can imagine like in a complex, task generation, you have to look into so many different moving parts.So you have all of these agents at your disposal. The most effective way we have identified to roll these out is, in users own workspace. Right? So it could be their cloud port session. It could be their cloud chatbot.It could be their IDE. It could be Slack. So we want to, like, enable all these agents, you know, into all of those workspaces. Here, another example, same agent like another user, nontechnical, who's just, using their cloud app to kind of, you know, reason with the agent and ask kind of the same questions. So, yeah, that's kind ofSam Nadler: where we are. Super interesting. Kartik, you know, before we jump into our next section, just where can people find you, how to reach out to You know, what... Just to get in touch and talk about Snorkel.Kartik Mathur: Yeah. I'm I'm available on LinkedIn, so, you know, I can share my LinkedIn and people can reach out to me. Snorkel, people can email me. Happy to chat about any of those things.Jordan Metzner: Awesome. Great.Sam Nadler: Alright, guys. I'm gonna queue us up for, you know, I don't know how exciting this is, but could be interesting. I'm sure both of you are using some of these new agentic tools like GrockBot, maybe Muse, apparently Google, which I haven't used. I use Gemini quite a lot, but I I haven't used their tasks, which I think is is now... Or was originally launched as Gemini Spark to help, you know, execute calls, purchases, etcetera, integrate very easily to your emails if you're if you're a Gmail user.Curious how you're using some of these agentic tools in your in your personal lives.Kartik Mathur: Yeah. I'm very excited with this announcement. Like you said, I have also used the other ones, and this is pretty recent. So I'm definitely going to give it a try. Think I this one is a little bit more exciting to me because like Google already has a good ecosystem like you know we are deeply integrated with Google for a lot of our personal calendars, know Google Pay is supported now.The most know the most exciting thing here is like you know it can make phone calls so I have like a long list of to do things at home so I'm quickly going to give it a try and see how it is. I think the interesting part will be the failure modes like the happy path will work like you know it does something but like you know there is sometimes a lot of time, decision making where, you know, it does not know enough context of what you provided for for day to day personal tasks. So, you know, I'm very excited about this. So let's see how it goes. But this this space is becoming very interesting now.Jordan Metzner: I'm super excited about this mostly because, you know, we run our business on Google Stack, and then my personal emails on Google Stack. And, I've had great success with Grock Bot as well with Cloud for certain features, but Grock Bot really just like getting access directly and having its own server, being able to do a lot of features, sending emails automatically and things like that. It does feel like Google has a superpower here for the for the Google stack. I do know that, like, Copilot on Microsoft side is sending emails automatically, responding automatically. So it does feel like this is a little behind from Google's perspective.I am gonna give it a try. But it does feel like a huge opportunity for Google to kinda take place here to have kind of this task manager agent that kind of can do a lot of things.Sam Nadler: Yeah. The calling feature sounds exciting to me. I did get a call from Jordan from his, agent yesterday telling me to work for that.Song: You got it.Sam Nadler: Yeah. So, it did work.Jordan Metzner: Yeah. Just as... I mean, Mike Rockbot has called, like, United Airlines, Delta Airlines. If I get a credit from an airline, it automatically calls and, like, gets me the credit. I haven't had it call, like, people very much except that I had to call Sam, but he didn't pick up so it left a voice mail.Sam Nadler: It left me a voice mail telling me to work harder.Jordan Metzner: I guess I guess that worked.Kartik Mathur: I'm looking for a feature where it can start joining video calls. That'll be interesting.Jordan Metzner: Well, I know Grock Bot. I heard that they can join video calls, and Sam's been trying to get Claude meetings to join the video calls. I don't think we've had success yet, but, you know, the demo has been pretty awesome. So it does feel like AgenTek is coming to meetings, like, very, very soon. Right?Sam Nadler: Yeah. Yeah. Yeah. Well, Kartik, great episode. Thank you for joining.And, yeah, any any last final words? And, otherwise, we'll sign off.Kartik Mathur: Thanks a lot. Yeah. Really excited about what's happening in the world of AI and then, you know, really enjoyed the conversation today.Sam Nadler: Great. Thanks everyone.

Hosted by
Sam
Hosted by
Jordan

More podcast