Short answer: Jordan Metzner says nobody knows the true cost of tokens yet and Codex computer use drove a spend spike that took two and a half weeks to trace. Sam Nadler says unclear billing dashboards made tracing spend slow.
Watch this answer from Episode 45 (starts at 19:35) · Episode 45 notes and transcript
From Episode 45: Your Pipeline Is Lying to You. This AI Fixes It After Every Sales Call, starting at 19:35.
Jordan Metzner [19:35]: I don't know if I agree with the Uber CEO or with Anis. I think, I understand kind of capping it from the sense of kind of limiting your max outbound pain or spend. But the real question is, is like, how much tokens should a developer use? And, I think that's like I think you're totally right about like, I'm a crappy developer and I use a lot of tokens. And we see like, our best developers also use a lot of tokens and then some in the middle. There are some that are, like, top developers that don't use a lot of tokens actually that we've seen also that type of behavior where, like, they use it, like, more infrequently and only in the cases of, like, probably, like, performance improvements and things like that. Like, less, like, kinda core writing, I would say. But I guess the point is is that, like, you know and this all comes from this Uber article in the first place, was that, like, they blew through their token budget. Question is like, when do they mail the token budget? These models came out in, December and now we've seen new updates.
Jordan Metzner [20:35]: And I think the other thing is that like, we don't know how we're spending using these tokens. They don't know how we're spending. I mean, I think like, we've seen from OpenAI, like, the Codex dashboard has changed like almost every day for the past like fourteen days. So I think they're also trying to figure it out. And we're in this like kind of wide open space where no one knows the true cost of like tokens, how much tokens you need to use. And then the models are getting better, faster, cheaper with like more, features inside the harnesses. Right? So, is like using sub agents more efficient than like running like, multi threads. Right? Like is that an efficient gain and that's a better way to use tokens or should sub agents never be used because like it's an excessive use. And, I didn't go back and forth on this, and I've probably been rambling a little bit. But, the Codex computer use feature over the past few weeks, like, definitely killed us on spend on tokens. And it took us like two and a half weeks to just figure out what it was that was causing all the spend. So does feel like the early days, almost kind of like when Internet was metered, and you could only get so many gigabytes of Internet on your phone or on your house, you know. And I don't know.
Jordan Metzner [21:38]: I mean, the people that just kinda let it rip end up kind of taking a little bit of a competitive advantage. But anyway, that's kinda where I feel. But, anyway, I'd love to hear your guys' feedback.
Sam Nadler [21:46]: Yeah. The only thing I'm gonna say is, like, I do feel like it's early days. It Anis, maybe you're more sophisticated than us, but it did take us about a week and a half on a few occasions to really trace where the spend was coming from. It wasn't very transparent on their dashboards. Was it API usage? Was it did the billings, like, where the charges were coming from were all coded differently and, like, they were the where there was the console, there's the Codex console. There's like, there's it took, a lot of time just to piece it all together, and that was frustrating. But, as Jordan mentioned, we're we're are, like, debating with, like, is it time yet to try and, like, control their usage, or do we just deal with it and just monitor it? And if something gets out of control, then we'll, rein it in. But in the meantime, it's an ongoing discussion over here.
