Short answer: Arun Kalaiselvan says an Anthropic connection change made one bill six times higher than expected. Anand Chandrasekaran says newer models turn many multi shot tasks into single shot, allow more precise prompts, and Gemini 3 and 2.5 Flash are very fast.
Watch this answer from Episode 24 (starts at 13:51) · Episode 24 notes and transcript
From Episode 24: This AI Fills Healthcare Shifts in Minutes, starting at 13:51.
Jordan Metzner [13:50]: I guess maybe just last follow-up on, like, the AI stuff in regards to this, and obviously, like, through Google and Bedrock, etcetera. And obviously, like, the future providers, whatever Microsoft rolls out type in the same. But, really, I guess, like, what have these new models in their deployment inside of these platforms enabled you guys to do that wasn't able to be done, as of yesterday per se? So, whether it's, Gemini three or whether it's, Claude through Bedrock or whether it's even just like OpenAI through Bedrock or even some of the Amazon hosted models, like, are obviously really inexpensive if you run them on Titanium, if maybe they're doing things like data processing. So, yeah, it'd just be interesting to hear, as the new models have been rolled out, how have they made an impact on kind of new things you've been able to
Arun Kalaiselvan [14:33]: It's I'm I'm glad you touched on the expense of things. Our last bill was about six x for what we would expect it to be because there was a change in Anthropic and how much time it took our Lambda to actually have that connection open with Anthropic. So it was kind of insane. We're like, where did this come from? We thought it was because we launched a new customer, one of our largest ones. And then we're like, wait. No. That's not it. This was literally our AI calls to generate details, a human readable detail of a shift that is being texted to people, and this thing increased our bill six x.
Anand Chandrasekaran [15:16]: Yeah.
Arun Kalaiselvan [15:16]: We've we've been trying to fix that.
Anand Chandrasekaran [15:18]: Yeah. To touch up on that. Right? Like, of the biggest advantages that I am seeing as of today with using the latest models is we are time bound in most of the things that we do. And we have SLAs sometimes with, like, five minutes with our customers when they need a feedback. And with AI, when you do real time stuff, it takes time. Like, putting we only see a chatbot being put on, like, you're getting Claude or chatbot responding very fast. But at the same time, if you're doing, like, document processing, compliance processing, it takes a lot of time. Like, it can take multiple sometimes you have to do multi shot instead of single shot. But if you look when I run some analytics between using Claude three point like, 3.5 and what it was 1.5 to all those things, a lot of the multi shots are now single shots. That's like the biggest advantage. And also the prompt itself can be very precise. Instead of writing tons and tons of examples, you don't have to do that anymore. Whereas the structured output and all, like, it's more straightforward. That means that your token length is concise, so hallucination is less.
Anand Chandrasekaran [16:30]: And the speed at which it responds, especially Gemini three and the Gemini 2.5 flash, they are just too good. My personal thing that I'm consider currently talking with Google is the Flash-Lite. But I need that to be a little bit more capable than what it is right now. So but that's that's the kind of the fun talk that's going on right now. Right? Like that you can do a lot of things with single shot for most of the applications and that reduces the time and that has a big impact for our customers. So that's that's one of the biggest wins I'm seeing right now.
Jordan Metzner [17:04]: Yeah. That's great. Like, I'm no more, like, exclusionary kind of, like, don't do this. Don't do all those rules, like, I mean, you would just, like, run iterations and then it would do this random thing and then you'd be, like, never ever do this. And then, if it's not high enough in the ex in the exclusion rules, then, like, it'll start to do it. Right?
Arun Kalaiselvan [17:21]: That's thing, though, Jordan. We still have to do that. Because if you think about it, the restrictions that we have in place are very, very different from using a chatbot that that learns from what you're talking to. So we actually don't allow any kind of long term memory when we sent any prompts to our AI. Right? So we actually have to put in all of the exclusions, all of the context, everything in every single chat that or every single LLM query that goes out there. So, yes, a little bit less work in terms of not having to do multi shots, but at the same time, we do still have to maintain all of the context on our end. And the AI itself actually is not maintaining any kind of memory on any data that we send it.
