How to Prioritize Experiments
If you’re running a brand doing $20M or $50M, you know the drill. Your backlog isn't a strategy; it’s a graveyard of good intentions. You’ve got the CEO who saw a cool widget on a competitor’s site at 2 AM, the Lead Engineer who is obsessed with a technical spec no customer understands, and a Marketing team that thinks a new font is going to save the quarter.
Most agencies will tell you to "just test everything." They’ll sell you a retainer, run sixteen color-change tests on a button nobody clicks, and send you a pretty slide deck at the end of the month that says a whole lot of nothing.
At Mobile1st, we stopped playing that game. We realized that if we wanted to actually move the needle for brands in that $3M to $100M "growth trap," we had to stop being "yes men" and start being the bouncers of the roadmap. We prioritize experiments like we’re triaging a battlefield. If it doesn’t meet the criteria, it doesn’t get on the site. Period.
Here is exactly how we decide what lives and what dies.
Rule 1: The 60% Revenue Floor (The "Who Cares?" Filter)
I’m going to say the unsaid: Most of the stuff your team wants to change simply doesn't matter.
If an experiment only impacts a tiny corner of your site, like a "refer-a-friend" link buried in the footer or a specialized landing page that gets 2% of your traffic, we don’t care. I don’t care how "strategic" it feels.
At Mobile1st, if an experiment doesn't touch at least 60% of your total revenue flow, it goes to the bottom of the pile. We call this the "Who Cares?" filter.
Why 60%? Because if you’re doing $50M a year and you find a 5% lift on a page that only 10% of your users see, you haven't moved the needle. You’ve just wasted three weeks of development time to buy yourself an expensive coffee. You need to be hunting where the money is. We look at the checkout flow, the high-traffic PDPs, and the global navigation. If it doesn't have the scale to actually change your bank account balance, we aren't touching it.
Rule 2: The Hierarchy of Insights (Data Over Ego)
The second way we kill bad ideas is by looking at the quality of the insight behind them.
Every brand has a "Random Idea Generator." Usually, it’s the person with the highest salary in the room. They say, "I think we should move the search bar."
We don't care what you "think." We care about our Hierarchy of Insights. (If you haven't seen it, check it out here: https://mobile1st.com/news/the-hierarchy-of-e-commerce-insights).
If your "insight" is just an opinion, it’s at the bottom. If it’s a best practice from a blog post? Still at the bottom. But if you show me a heat map where users are rage-clicking a non-functional element, or a user recording where a customer literally says, "I can't find the shipping price," now we’re talking. If we have alrady tested a similar concept on a different part of the site, we make the experiment important. That is the best insight we could ever ask for.
When you have high-quality insightsthat experiment jumps to the top of the stack. We aren't testing to see "if" something works; we're testing to solve a problem we know exists.
Rule 3: The 70/30 Split (Optimization vs. Transformation)
This is where most experimentation programs go to die: they get stuck in the "Optimization Loop."
You change a headline. You move a picture. You tweak a button color. You get a 1% lift here and a 2% lift there. It feels good, but it doesn't change the trajectory of the business.
We believe that 70% of your experiments should be optimization. You have to protect the house. You have to keep refining the current experience to squeeze every cent out of your traffic.
But the other 30%? Those need to be "Business Changing Opportunities."
These are the big swings. The experiments that challenge your entire business model or how you present your value proposition. These are the tests that make people in the office nervous.
I once had a Lead Engineer try to kill an experiment because he thought the specs he wrote were the hero of the page. He spent months on the cooling fins of a product. I told him nobody cared about the fins; they cared if it would fit in their car. We deleted his "innovative" specs and replaced them with a "What's in the Box" list. RPV went up 23%. That wasn't just an optimization; it was a realization that the way the company talked about itself was wrong.
Rule 4: Value Over Speed (The Anti-Friction Rule)
Here is where we really start to piss off the "CRO Gurus."
For the last decade, the industry has been obsessed with "reducing friction." Every blog post tells you to make the checkout faster, remove steps, and get them to the "Thank You" page in three seconds.
We do the opposite. If an experiment is just about reducing friction to help people get to checkout faster, we deprioritize it.
Wait, what?
Think about it: Making a bad experience faster just helps people leave your site sooner. Speed doesn't build brands. Value builds brands.
If an experiment improves the perceived value of your product or helps a customer find the exact right product for their specific needs, it goes straight to the top of the priority list. We want to spend our time helping the customer understand why they should pay you full price, not just making it easier for them to click "Buy" on something they aren't sure about.
Similarly, if we can personalize the experience for almost all users—not just a tiny segment—we do everything we can to test that right away. Personalization that scales is a revenue multiplier.
Stop worrying about making the "Buy" button 0.2 seconds faster. Start worrying about why the customer is hovering over it and not clicking. Are they confused? Do they feel the value? Do they know this is the right version for them? Solve for value, and the conversion will follow.
The Cold Truth
If you’re leading an e-commerce team right now and you’re still building your plans around "hacks" and "speed tricks," you're playing a game you've already lost. The "vending machine" era of e-commerce—where you put a dollar in Meta and get three out—is dead.
You win now by being more disciplined than your competition. You win by being the person who says "no" to the low-impact friction tweaks so you can say "yes" to the 60% revenue drivers and the big value-based swings.
If you’re tired of the "invisible grind" of running tests that don't matter and managing an agency that’s just a "yes man" for your CEO’s bad ideas, maybe it’s time for a different approach.
We aren't here to be your friends in the boardroom. We're here to be the ones who tell you when your baby is ugly so we can help you build something that actually grows.
Let’s stop "optimizing" and start winning.
Ready to grow RPV by prioritizing the right experiments Reach out.
Check out what else we’ve been writing about or send Justin a message on LinkedIn.
Be a guest on Checkin to Checkout
Want to tell the story of your e-commerce journey, and give your insights? We’d love to hear what you have to say. Reach out, to be a guest on our show.
Want help? Reach out.
How to Use Segmentation to Grow Revenue with Anna Christiansen
Listen to the latest episode of season two of Checkin to Checkout.
