Short answer: Jordan Metzner says Amazon tested many small changes like star-rating designs that often didn't matter, Google has barely changed its homepage, and e-commerce has the clearest feedback loop for constant testing.
Watch this answer from Episode 35 (starts at 25:57) · Episode 35 notes and transcript
From Episode 35: AI Agents Are Replacing Entire Marketing Teams, starting at 25:57.
Jordan Metzner [25:56]: Yeah. This is a huge unlock. I think I was talking with someone yesterday just about this idea that, historically, you kind of like build a website, you put it up, and then you focus most on your web application. If you build a new feature, you go back to your website and update it. But for the most part, kind of they stay static for a pretty long time until kind of you do a refresh or whatnot. And, obviously there's always been tools to like a b test buttons and copy and things like that. But they've been, one not of high value. Changing a button, from blue to green, I think is like an over optimization where people find that, the results, even if they come in statistically significant, like at the end of the day, like if you didn't change it, you'd end up getting the same type of results. And I learned A lot of That in Amazon. I think like they tried so many things, so many tests of like changing the shape of you know, the design of the star ratings and whether that would have an impact on sales. And you know, just so much kind of historical like web lab type testing where you know, you run this experiment, you give it its time, you get enough, you get a coefficient, and then you know, and then you kind of decide like, okay, now we decided we're gonna go with b or whatever. Right? And so this idea of this like constant experiment where like you don't even know what's happening on your website because it's constantly testing every single variable and all these other variables.
Jordan Metzner [27:10]: So, even customer that comes in today sees a different website than one that comes in an hour from now, even one that comes in tomorrow, I think that's obviously incredibly cool. The real question is like does that actually mean something for the like the net impact to your business? Like does optimizing your website over and over again have, significant lift over time? That probably has an impact on what is the business. Right? I mean Google's probably the classic of this, but like they haven't changed their homepage in forever. Right? I mean, almost like since the beginning of time. And you know, they make small changes and add a microphone here or there, but you know, they've been very delicate on kind of changing that core kind of Google launch page experience and obviously have done incredibly success been incredibly successful. It does seem like e commerce is probably where it's like this opportunity lies the most kind of like because, e commerce like people check out and so you're able to check like and get a real feedback loop of whether or not like this has an impact on customer behavior versus just like, in a b to b setting to your point, Ilya, like on your software, like customers might visit your website, but it might take them, a longer period of time to make a purchasing decision, etcetera.
Jordan Metzner [28:14]: So but it just thinks I just think this is like more democratization of AI and machine learning, and overall just giving more capabilities into like developers hands, and I think that's that's incredible.
Sam Nadler [28:25]: Cool, guys. Great episode. Ilya, thanks for joining. Where can they find you? I know it's juma.ai, but, if someone, is desperate for this product, what's what's the best way to reach out?
Iliya Valchanov [28:37]: Yeah. Juma.ai, they can start free trial. They can check the product. It's a bit and they can check the product. They can see if it's useful to them. Or on LinkedIn, I'm posting regular LinkedIn, checking my inbox there. So if they wanna reach me, Iliya Valchanov on LinkedIn is the best place.
Sam Nadler [28:56]: Amazing. Well, thanks for joining, and, Jordan, we'll see you next week.
Jordan Metzner [29:00]: Thanks, everyone. Thanks, Ilya. Thanks, Sam. Great episode. And, yeah, exciting times in AI, looking forward to what's coming out next week. See you this week.
