Short answer: Angie Westbrook says traditional tests split about 40 stores for 12 to 16 weeks, while some Standard AI customers reached statistical significance within three weeks using only a couple of stores.
Watch this answer from Episode 21 (starts at 11:43) · Episode 21 notes and transcript
From Episode 21: Not your STANDARD Market, starting at 11:43.
Jordan Metzner [11:42]: That's super cool, Angie. Just a few follow-up quick questions. How does this impact kind of speed? Kinda how fast are retailers able to get access to this data compared to, how long it would take them to get access to something like checkout data or other types of data to make actionable behavior?
Angie Westbrook [11:59]: I'm like a operations nerd. Okay. So the thing that I preach over and over again to teams over the years is the importance of leading indicators and metrics over lagging. And so when you think about the delay, it's not just the delay in getting the transaction log data, which can be weeks down the road. It's also just what is it's it's such a lagging indicator of what's happening that you can't get that crisp crystallized feedback on how the shopper is responding. And so not only can we deliver it faster, but you can get to statistical significance way faster than you could by trying to use sales data. And a great example of that is if you talk to almost any retailer, if they're trying to set up an AB test, it's very, very difficult because, to set have a controlled environment. I you know, perhaps there's nothing more dynamic than physical retail. It's also why I love it, but very, very dynamic. So oftentimes, retailers will set aside a store set of, like, 40 stores, let's say.
Angie Westbrook [13:08]: And they'll try to make 20 a control, 20 of the experiment. They have to run that often for something like twelve to sixteen weeks to get enough data to be able to analyze it. And even then, you talk to any anybody in this space, and they'll tell you kinda have to squint and really study hard to see if you can correlate a lift or any happen directly with your decision. And you think you know, with ours, we've had some of our retailers and our customers be able to get to statistical significance on feedback of a change within three weeks and using only a couple of stores in some instances because they have so much confidence in what that change is doing, seeing the reaction that shoppers have versus only relying on sales data that they can then scale those insights across the board.
Jordan Metzner [14:01]: That's awesome. So they can experiment faster and then deploy the successful results of the experiment faster, thus, like, yielding higher results?
Angie Westbrook [14:08]: Exactly. And then you think about the importance of that time factor, in the case where the change that you made was successful and it made the desired effect, waiting sixteen weeks to implement it is, a bummer. Right? But the bigger problem is is what if it created an adverse effect or had a negative impact? Now you've waited sixteen weeks to see that, and even in your stores that you're testing in, it was a negative experience for those customers during that time. So, we think that providing this level of, like, rapid feedback in the physical retail environment is total totally new unlock that they haven't been able to do before.
