0:00Ben Lerner, CEO of Espresso AI, says transformer models understand and work with code far more deeply than earlier models. Sam Nadler demos a quiz built for Espresso AI estimating a scale-up could save up to $8,000 a month.
2:44Ben Lerner says FinOps means doing work to review budgets and dashboards, while Espresso plugs into your data warehouse and optimizes in the background, aiming to cut a Snowflake bill in half without customers touching anything.
6:00Ben Lerner says Espresso uses ML models to rewrite code, shift workloads between clusters and run the right machines at the right time, like Kubernetes powered by ML. Today it works with Snowflake and Databricks SQL.
8:19Ben Lerner says it is not via marketplaces. Espresso gets an account to monitor and resize warehouses, and routes queries in real time through a proxy on the customer side, for example running work from 10 clusters on seven.
9:58Ben Lerner says Espresso plans to look at GPU inference, since any large compute with unpredictable demand can benefit from statistics and detailed analysis of which workloads can safely share a machine.
11:16Ben Lerner says Espresso started about six months after ChatGPT came out. Its models read incoming SQL and compare it to previous runs to predict whether adding a query will queue, add capacity or slow existing queries.
13:56Ben Lerner says Codex writes Espresso's dashboards and throwaway pandas scripts, but models are not good enough for serious back end work. He estimates engineers are 10 to 30% more productive, while Claude Code costs far less than a third of an engineer.
17:17Ben Lerner, who worked on natural language processing at Google, says Siri has not gotten substantially better in years and the upgrade to Gemini is exciting. He notes many people from that part of Google moved to Apple.
20:23Ben Lerner says Espresso investor Daniel Gross leads part of Meta's effort; his fund's Andromeda cluster bought $100 million in GPUs for portfolio companies in 2023. Ben suspects Meta will keep the compute internal.
24:19Ben Lerner invites anyone wanting to cut their Snowflake or Databricks SQL bill to visit espresso.ai or email him at ben@espresso.ai.
Cut Snowflake Costs with ML: How Espresso AI Does It: the key questions from this episode, answered with quotes and timestamps.
Ben Lerner [0:00]: The really phenomenal thing about transformers and these transformer based models is that they can understand and work with code in a much deeper way than previous models.
Sam Nadler [0:22]: Hey, everyone, and welcome to Built This Week, the podcast where we share what we're building, how we're building it, and what it means for the world of AI and startups. I'm Sam Nadler, cofounder here at Ryz Labs, and each and every week, I'm joined by my friend, business partner, and cohost, Jordan Metzner. And this week, we have a special guest, the CEO of Espresso dot ai, Ben. We're gonna jump all into espresso.ai, but first, we're gonna demo a little product we built with Ben in mind. This is intended to show the value of Espresso AI to, your potential clients, and they go through a quick questionnaire to really see how much they're wasting by not integrating Espresso AI. So let's just go through this start small test. Let's actually let's choose where a scale up, and you'll see we'll just have some quick questions. Question number one is how often does your team use SELECT * in production queries? We'll probably do it a little often. Do you know your top five most expensive queries or dashboards?
Sam Nadler [1:30]: Not really. I have a probably a handle on the top two. How often do dashboards refresh when compared to how often they're viewed? Way more often. And are multiple teams querying the same raw data independently? Yes. Definitely. And who is responsible for warehouse performance and cost? It's a shared responsibility. And as you can see, we're now getting a key observation for Espresso AI. On the top, you can see the CFO BPM is a 168. Your CFO is going crazy. We're very over extracted in terms of our Espresso rating. Key observations, receipt of sins, actually. We could be saving up to $8,000 a month with espresso AI, some key observations, queries are driving up costs, lack of visibility, refreshing dashboards significantly more often than necessary, multiple teams querying log data. All of this can be solved when we optimize with Espresso AI.
Ben Lerner [2:43]: I don't know if the clapping comes through at all on the mic. This is phenomenal. This is really, really well. Yeah. This is great. This is a super fun quiz. It's the, like, sort of thing that people should be thinking about. I think pretty ironically, like, this is a something that points to, like, a really common set of products in the space, but the solutions are, like, very different from what we actually do. So, like, this starts touching on, like, FinOps and accounting best practices and all the kind of stuff that, like, comes to mind with really, like, any expensive bill in a business or even, like, at home. Right? You're like, well, you should look at your budget. You should figure out what you're doing and why. You should figure out if you can be doing it better. Maybe we really don't need that dashboard to refresh, like, 10 times a day if it's checked once a week. You know? Oh, like, honey, like, we haven't watched Netflix in six months. We should just shut it off. Right? So it's it's those sorts of things that go into, like, financial discipline and FinOps best practices. In a lot of ways, this is, like, exactly the opposite of what we actually do.
Ben Lerner [3:49]: So we plug into your data warehouse, and we shift everything around. Like, we're not opinionated on, like, what you're running or why. We try to do everything in the background to make things as cheap and efficient as possible. And I think that, like, you're paying for convenience is, like, pretty accurate. But the flip side of that is always, like, how much are you paying? Right? Like, everyone we talk to is, like, overworked. Like, this, like, this is a great quiz. And, like, if this bill number is right, like, this is probably a team with two data engineers. They have way too much on their plate. And what we do is we come in and we give you a button, and we're like, okay. Well, you're you know, can we get your CFO's BPM down by just cutting your bill in half without you having to think about anything and without you having to touch anything? And we kind of, like one of the things I say to people is, like, yeah. You can do all this work to save money, but, like, man, like, if your snowflake bill were 50% lower, like, would you care, or would you do other stuff? So a, this is, like, awesome.
Ben Lerner [4:53]: I'm, like, very impressed that you guys put this together. Thank you. But, b, I think it's, like, a nice call out of, like, you can go do a bunch of work to lower your bills. And to some extent, what we try to tell people is, like, you can and you should do that, and at scale, you will need to. But you can also not do a bunch of work, and we can lower your bills for you using AI magic.
Jordan Metzner [5:14]: No. I think that's that's a great way to look at it. I know, like, we just had a GPU cluster that stayed on, and we had some AWS overage, and, it almost feels like, man, I wish I would have, like, I wish I would have had some protection against that and put some, better guardrails in place, so that wouldn't happen next time type of thing.
Ben Lerner [5:35]: Yeah. So, like, the analogy there is, like, we can't help you if, like, your ML engineers are running up their GPU clusters, but we can definitely help you if they forgot to shut them off because we could tell there's you know, they're underutilized and we'll help with that. So, yeah, I think it's different ways to get to the same goal. But, like, not doing work is usually better than doing work.
Jordan Metzner [5:59]: Yeah. I would love to. I'd love to just get, like, the one minute spiel about you and your business.
Ben Lerner [6:04]: Yes. I'm I'm Ben, one of the founders. I'm the CEO of Espresso AI. The goal of Espresso is to use machine learning to optimize compute. So we use ML models to rewrite code and make it faster. We use it to shift workloads around between different clusters to make sure things are running as efficiently as possible. We use it to make sure that you're running the right machines at the right time to get the performance that you want, but without overpaying for unused or underutilized capacity. So for the more technical folks in the audience, you can think of it like Kubernetes but powered by ML. Today, we work specifically for Snowflake and for Databricks SQL, but we're planning to expand to other data warehouses and other compute generally in the future.
Jordan Metzner [6:52]: Got it. And so, essentially, so so if I understand this straight, let's give a case of somebody on Snowflake. The reason they're able to save is that they're buying they're they were previously overpaying for how much, like, compute they need for their data warehouse. And by optimizing that through, like, scalable some type of cluster system, they're able to save significantly and then just buy and I don't know enough about it, but just buy basically on a license basis versus, like, on a hosted basis. Is that, like, how the switchover happens per se?
Ben Lerner [7:20]: No. So we hook up directly to people's Snowflake accounts. And Interesting. Really, everything that we're doing is stuff that our customers could do directly, but we're doing very fine grained piloting of exactly how much compute do you need at any given second, where should this query be running versus, like you know, instead of running all of your BI tooling on one cluster, sometimes you may be able to shift it around. And, like, I wouldn't necessarily say people are overpaying. I think it's more that, like, people have been trying to make computers run faster and more efficiently since, basically, since we've had computers, and this is kind of a next step in that and really leveraging ML to get even more performance out of it. So, like, once you know how to do something better, you can look at the old way and say, oh, yeah. Well, this is more effective. But it's not like people are doing something wrong per se. It's just that the technology's gotten better.
Jordan Metzner [8:15]: Yeah. Cool. Okay. Now I totally get it. That's awesome. And does that serve through, like, these through their marketplaces, or how do customers normally, like, integrate Espresso, like, on top of, like, whether it's Databricks or Snowflake? Like, how does the integration normally work?
Ben Lerner [8:32]: Yeah. So it's not through the marketplaces are designed for you know, I wrote an app that is gonna give you maybe some dashboarding or some other features on top of some data. It's not designed for, let me rewrite and redirect all of your traffic in real time. Those are it's definitely, like, a bit of an unusual use case. So we sit on the customer side of things, and we basically integrate in two ways. One is we'll get an account in Snowflake or Databricks, so that lets us monitor things from kind of a bird's eye view, and it lets us make changes like in Snowflake and Databricks, what most people will call clusters, they refer to as warehouses. So it lets us say, oh, let's make this warehouse larger. Let's make it smaller. Let's add a little bit of capacity here because there's a spike in traffic and so on. The rest of our tooling runs over a proxy. So we actually, again, sit on the customer side before their workloads hit Snowflake or Databricks. And we can, like, in real time, basically do, like, the flight traffic controller thing of, like, this is gonna go over here. This is gonna go over there. And, you used to run this on 10 clusters, but, actually, if you're really granular about it, maybe you can run it on seven.
Jordan Metzner [9:42]: Oh, that's great. So the real time routing of, like, basically queries against this data warehouse, of optimizing your compute to serve those queries while not having too much compute or, like, unused compute on the other side, which makes total sense. Yeah. No. This is super cool. I could see this happening on, like, other types of things like GPUs as well because it's it seems like a very similar use case.
Ben Lerner [10:04]: Yeah. So one of the things that we're planning to take a look at is GPU inference specifically. So it any sort of large compute where, like, you don't know exactly how much you're gonna need ahead of time, and there's a little bit of a question of how do things, like, fit together onto physical machines, and is there a more efficient way to pack them? All of that is probably gonna benefit from the kinds of techniques that we use, which is basically really understanding that at a very detailed granular level. I mean, yeah, GPU inference is obviously, like, blowing up and absolutely something where, like, you have some rough ideas about how things are gonna grow, but you can do a lot of statistics to get a better idea of that. You can do a lot of detailed analysis of what workloads pair together and how do they work, as well as just, like, smart, like, CS algorithms, smart models, have a better idea of, like, okay. Well, you're running these two things together on the same machine, and, it doesn't quite have the resources that the two would need if they both burst at the same time, but, actually, they're not correlated, and so it's totally safe to run them together and occasionally spike capacity.
Jordan Metzner [11:15]: Yeah. Totally makes sense to me. Maybe you can talk a little bit about how kind of AI, like the LLMs or whether that's impactful on your business or not, and kind of, like, obviously, I mean, just from a coding perspective, how amazing Opus is or whatnot, or just Claude Code, and kind of a little bit about how that's impacted kind of the way you guys build. Because I think that'll be most interesting to our audience, and then they can also talk a little bit about kind of how AI's had the impact on your business.
Ben Lerner [11:40]: Yeah. So I think we're a little bit so one answer is, like, AI massively impacts and informs our business. And we started this business because of the newer generation of models and the sorts of things that we thought they could do and that we have figured out that they, in fact, can do. It's little bit off the beaten path, so I think that there's a lot of interest right now in, like, vibe coding and taking vibe coding to the next level. So we don't benefit as much from that, and I'll touch on that in a second. But for the actual business, like, the really phenomenal thing about transformers and these transformer based models is that they can understand and work with code in a much deeper way than previous models. Right? And so this is the fundamental thing that's powering things like cursor and Claude Code and codex and so on. And you can leverage that understanding of code in different ways. So one is obviously with, like, autocomplete and, like, natural language prompts and so on.
Ben Lerner [12:37]: What we're doing is quite different. What we're doing is using those models to understand your workloads and how they scale, right, and how much like, what resources they actually need. So we actually have models that will look at your queries as they come in, like, read the SQL, and effectively compare them to previous runs and understand, okay. There's seven queries running in parallel on this machine now. What happens if I add an eighth one? Is it going to queue because the cluster's full? Is it gonna cause us to spin up more capacity? And if so, is that good or bad? Is it gonna slow down the existing queries because it's you know, there's enough capacity to run it, but it's slicing into what they're doing or not? All of these things are really, really difficult to unsolved problems. There's a bunch of literature on people who've tried it before. And, obviously, there's, like, some heuristics and some understanding of cost, but it's pretty coarse compared to the kinds of results that you can get out of transformers and out of modern, like, LM based models. So that is one of the things that makes our business possible. And, we basically started this, like, six months or so after ChatGPT came out. So this is very much focused on the modern generation models.
Ben Lerner [13:55]: The day to day, like, I think we're using models like anyone like any other company is, and, they're really, really helpful for some things. A lot of our dashboarding and a lot of our, like, throwaway, like, pandas scripts are definitely coming out of prompts from Codex. I think the flip side is, like, the models are definitely not good enough to do very serious back end infrastructure work. And so they can I'll just talk about, like, my own experience. Like, if I'm trying to, like, interface with some API I have in route or whatever, like, it's transformational. Right? Like, it's faster. It's more accurate. I don't need to go, look at Stack Overflow. I probably haven't visited Stack Overflow in, like, a year plus. But, like, I can't just, prompt at something and then throw it out into production and expect to run, like, tens of millions of dollars of Snowflake workloads through it. Like, it's it's just not that good yet. So these models are getting better every month. I'm, like, excited to see what happens, over 2026. But mostly, we're, like you know, I'm still writing a lot of code by hand.
Ben Lerner [15:02]: I do hit tab a whole lot. There's a lot of good autocomplete. But, yeah, I think that yeah. Like, I think the full vibe coding feature, there are, like, 25 Claude Code agents running in the background for serious back end and for, like, we're just not there yet.
Jordan Metzner [15:18]: Yeah. Totally agree. But it still seems like even though we are not there, and then maybe we'll, like, never get there, but it's just makes, like, someone who's not an engineer better than never being an engineer, and someone who is, like, a great engineer, a better you know, like, everybody gets the superpower. Right? So even for the best engineers, they're still getting this superpower where, even yes. I agree with you. It's not like, go build this whole thing and, like, put it into production, but, like, even they get that advantage. And even as you said, like, whether it's auto complete or whether it's, like, hey, go do this. Like, I have you know, one of our back end engineers hates front end tasks, and, like, now he can just get those done really quickly so they get out of his task out of his way rather than, like, codepend on somebody else, for example, previously or something.
Ben Lerner [16:00]: Yeah. I think that that's definitely very true. And, like, I would say everyone is probably ten, twenty, 30% more productive, which is huge. Right? Like, we pay a lot less for Claude Code than, like, a third of an engineer per engineer. So that's that's a really substantial change. And then, like, we don't have a dedicated front end engineer. That just those tasks tend to just go into our on call rotation because it's easy enough. We also we don't have a massive front end. It's it's just some dashboarding for our end users, so it's also relatively simple. But, yeah, like, we just kind of rely on the robot to do some of that stuff, which is really cool.
Jordan Metzner [16:39]: Yeah. That's amazing.
Ben Lerner [16:40]: We outsource. The design is a little bit more of a bottleneck because we have some customers where we're saving them millions of dollars, and they're, paying a good amount. And I think they want something that looks a little bit better than, like, we don't have a designer on staff. So interestingly, like, that's you know, I think, like, design might get more humanized as the like, it's complement, like, the actual front end engineering gets easier.
Jordan Metzner [17:11]: Yeah. Super cool. Okay. Well, I think we talked a lot about Espresso. We got two new stories, so let's get into it. But I think the first one is, Apple signed the deal with Google, and obviously, now Gemini is gonna be powering Siri, and obviously probably powering Android as well through Google. But, yeah, I just wanna talk a little bit about Gemini, Apple, Android, maybe even a little bit too. But, yeah, just, and obviously, I mean, we use all the models here internally as well, but I think we've seen, some things where Gemini has performed well, and other places, it obviously hasn't been a top tier model yet compared to some of the other models on the market. Love to hear your thoughts there.
Ben Lerner [17:49]: I think so I have opinions on this. I used to work on natural language processing at Google, and a lot of the folks that I work with well, it's been a few years, but some of those folks and the code that I used to work with transformed into Gemini eventually. And, like, I'm very, very excited to see this. I think Siri is one of the voice assistants with a lot of mind share and a lot of you know, like, lots of usage. Like, lots of people use it, but realistically doesn't seem to be something that's gotten substantially better over the last few years. So, like, I don't have so much of an opinion on, like, should they have gone with Gemini? Should they have gone with Anthropic, I think, or, OpenAI? I think all of these models are keeping pace with each other. And, any given month, like, one might be a little bit ahead. But, like, I think the upgrade from Siri to Gemini is is super exciting.
Ben Lerner [18:41]: If they really leverage it, I think there's gonna be a lot of cool stuff. And I also have a little bit of inside baseball, which is all of this is public. Like, a lot of the folks from that part of Google moved over to Apple. So I kind of assume that they're of, yeah, we can do things internally and try to catch up, but maybe not that, but we can also just sign this partnership, which like, I think it's gonna be great for Apple users and, talking on AirPods into a MacBook. So, like, I will probably benefit in this as well.
Jordan Metzner [19:14]: No. I totally agree with you. I think it's obvious that, like, things like speech to text on an iPhone is pretty poor. And, I mean, almost every LLM app has a much better, speech to text model and transcription and duration, and it's unfortunate. And, like, Siri's not smart at all or hasn't been historically. I think the same almost the same thing happened with, Amazon and Alexa. Right? Like, they just kind of lost market shares. They weren't able to keep up with, AI at the edge. Right?
Ben Lerner [19:44]: Yeah. For sure. I would, maybe politely phrase that as well as really great at shipping hardware. So excited to see, like yeah. I think it'll just be awesome to have, like, a smarter voice assistant, like, actually work in your headphones and your phone and all that.
Jordan Metzner [20:00]: With a friend about something, we said, like, okay. Well, we'll see what happens in a year. And I would just I told Siri to, like, remind me in the year, but she couldn't really, like, do that well, like, within our text message chain. You know? So, yeah, I think, those things with well integration like into the Apple ecosystem, obviously, will just create for a much more, enjoyable experience on probably all Apple devices, but definitely an iPhone. Okay. Cool. Let's jump into the other story, which I think is like pretty interesting, which is about Facebook's announcement about this megawatt capacity. And I think, I've been saying for a while it seems obvious that they should kind of do the AWS thing and like offer some of their GPUs as a service. I think what's like most interesting about this is the way they finance this thing, because it shows like an ability for Facebook to kind of scale these things without having to leverage their balance sheet, and that obviously can, provide for even more scale.
Jordan Metzner [20:52]: But, yeah, obviously, we've seen the mega salaries they've done. We've seen the Scale AI acquisition, etcetera. But, I would love to hear your thoughts about Facebook and building their own kind of mega compute, and whether they make it available or not. Still is not clear yet, but seems like it will be.
Ben Lerner [21:09]: Yeah. It is. It would be interesting to have another hyperscaler competing. I think that, like, Google has really phenomenal technology. In a way, like, sales is really, like, a core part of that, and being late to the race is really difficult, which is definitely something that GCP learned. Maybe, like, really focusing on GPUs is a way to get into the market now, like CoreWeave IPO recently. So there's definitely more more room there. Generally so the in this article, the person leading up part of this effort is Daniel Gross. He's an investor of ours and was part of the big, like, meta acquisition. And interestingly, like, the way that we connected with him is in 2023, his fund announced the Andromeda cluster, which was a, small purchase of a $100,000,000 in GPUs to make available to their portfolio companies. And there was, like at the time, there's just, no GPU anywhere. Like, it was if you were not committing to, millions of dollars in spend, which as a start up, we definitely couldn't, He just like, we ended up, like, buying GPUs off Amazon, basically, and, like, just running them in our office. So, like, I think that this is, like, an awesome project that is totally up his alley and something that he has a lot of experience in. So, like, really excited to see where it goes. The overall, like, should like, yeah, like, the meta huge salaries thing is kind of like Mark Zuckerberg seems to take really big bets, and, some of them play out and some of them don't.
Ben Lerner [22:45]: I think that given how important ML is, like, it's not crazy. I probably no one would have done it. Like, no one else would have maybe done it exactly in this way, but the idea of, like, we have all this money. How do we give ourselves a shot at being a at the leader in the AI race, especially when, they're not OpenAI and they're not Google and maybe haven't been doing quite that much research into LLM specifically is, like, it's it's it's fun to watch. And I like, it's it's interesting to see what comes out of there. I'd also be, like, excited to have, like, more GPU availability and, like, hopefully, cheaper and better availability. But, honestly, I kinda suspect they'll set it all up internally. The flip side is maybe, like, if they haven't gone onto the market as a, cloud provider, like, why would they do it now? But who knows? Like, Zuck could wake up one day and decide that's what they're doing, and then that's that's what they'll do.
Jordan Metzner [23:42]: Yeah. I definitely agree with many of your points. I mean, if it seems like historically to date that, ads or, AI and ads are AI is one of the best use case or ads are one the best use case for AI, etcetera. So, yeah, obviously, I mean, a business built on advertising, like Facebook and Google both, obviously perform well using AI. Yeah. We'll see as, NVIDIA brings on, TPUs with its Groq acquisition and the impact on, all these types of chips at the inference level. And to your point, like, yeah, Zuck is willing to take a lot of risk and make some bets, and, it'll be exciting to see what happens there.
Sam Nadler [24:19]: Alright, Jordan. Thanks, Ben. This is a great episode. If you haven't already, please hit the like and subscribe button for Built This Week. And if you get some time, please check out Espresso AI.
Ben Lerner [24:31]: Yeah. This is really fun. Thank you, guys. Sam, that was awesome. Like, that was a really fun demo. Thanks for putting that together. If you're interested in cutting a bunch on your Snowflake bill or your Databricks SQL bill, come visit our website. It's espresso.ai, or you can just email me. I'm Ben@espresso.ai. Happy to talk to you about you know, if you're stressed about your snowflake bill, we can probably help either directly or just happy to chat about best practices.
Jordan Metzner [24:56]: Thanks so much. Have a great one. Bye, Ben.
