Short answer: Ben Lerner says it is not via marketplaces. Espresso gets an account to monitor and resize warehouses, and routes queries in real time through a proxy on the customer side, for example running work from 10 clusters on seven.
Watch this answer from Episode 27 (starts at 8:19) · Episode 27 notes and transcript
From Episode 27: We Let AI Control Our Data Warehouse — The Results Were SHOCKING, starting at 8:19.
Jordan Metzner [8:19]: And does that serve through, like, these through their marketplaces, or how do customers normally, like, integrate Espresso, like, on top of, like, whether it's Databricks or Snowflake? Like, how does the integration normally work?
Ben Lerner [8:32]: Yeah. So it's not through the marketplaces are designed for you know, I wrote an app that is gonna give you maybe some dashboarding or some other features on top of some data. It's not designed for, let me rewrite and redirect all of your traffic in real time. Those are it's definitely, like, a bit of an unusual use case. So we sit on the customer side of things, and we basically integrate in two ways. One is we'll get an account in Snowflake or Databricks, so that lets us monitor things from kind of a bird's eye view, and it lets us make changes like in Snowflake and Databricks, what most people will call clusters, they refer to as warehouses. So it lets us say, oh, let's make this warehouse larger. Let's make it smaller. Let's add a little bit of capacity here because there's a spike in traffic and so on. The rest of our tooling runs over a proxy. So we actually, again, sit on the customer side before their workloads hit Snowflake or Databricks. And we can, like, in real time, basically do, like, the flight traffic controller thing of, like, this is gonna go over here. This is gonna go over there. And, you used to run this on 10 clusters, but, actually, if you're really granular about it, maybe you can run it on seven.
Jordan Metzner [9:42]: Oh, that's great. So the real time routing of, like, basically queries against this data warehouse, of optimizing your compute to serve those queries while not having too much compute or, like, unused compute on the other side, which makes total sense. Yeah. No. This is super cool.
