Short answer: Jim Olsen says model-level or user-level token views are not useful to the business; ModelOp tracks what each use case costs, such as a support chatbot, using data imported from Kong Gateway or tracked directly.
Watch this answer from Episode 55 (starts at 3:28) · Episode 55 notes and transcript
From Episode 55: Every Client Is Fortune 500. What Do They Know About AI That You Don’t?, starting at 3:28.
Sam Nadler [3:27]: Amazing. I would love to see a demo. And while you're pulling something up, I... My understanding is, at least one feature of ModelOp is that you're... From the token usage use case, you're not only looking at token usage from the user, which I think, a lot of us are trying to understand, but also from almost the business unit or what's it trying to achieve. Did I understand that correctly?
Jim Olsen [3:49]: Yes. That's correct. We can look at there's the traditional view of looking at it from, like, for instance, the model consumption overall, but that's not really useful to the business or from the user. That's not really useful to the business because a user using Claude Code to do something and develop some software is a very different use case than the user using the same model to perhaps, answer customer questions or these kinds of things. So we need to actually understand what is the solution costing me. So for instance, like if I have a customer support chatbot that's out there, I wanna know what that particular business use case is costing me as a business on there, and also any potential, like, issues with it. So for instance, here we're integrating with Kong Gateway to pull in information about guardrails. It's imposed in blocks, specific requests on there. So understanding that over time as to how many exceptions you're getting, but then also what is the solution costing me. Because if it's just like a general HR query tool and it's costing me thousands per month, that might not be worth it.
Jim Olsen [4:55]: But if it's a critical item like customer care support, etcetera, and it's saving new things, that's a very different cost equation. So being able to have that kind of information at your fingertips, so to speak, to say, yes, this is worth the spend. Does that make sense?
Sam Nadler [5:10]: It does. And what am I looking at here?
Jim Olsen [5:13]: So this is actually tracking a business use case, a hypothetical one here in this case, a customer support chatbot that, our... We actually track all kinds of information about it, the information about the business use case, but also, potentially the implementation and stuff, which we'll show an example of in a second. But this is tracking the token usage of... And actually, in this case, imported from Kong Gateway. We integrate with them as a partner. We can also track it directly. We'll see that with some of our agentic use case examples so that we can see that that cost over time of what it's actually providing as well as an overall dashboard of just, all of the information about the different use cases, models, etcetera, and what they are consuming in token so we can tie it all together and get that view.
