Tips for Successful Experimentation Strategies

When you're launching something new, you want to be sure it's going to work. Running in-market experiments prior to launch confirms hypotheses before you commit resources. Just as important,  experiments can often prevent big missteps. Here are four rules of thumb that make for powerful experimentation: 1. Test more than one concept or proposition with more than one target market segment. Sure, you can test just one concept with just one target, but you'll only learn if it succeeded or failed. If you test several concepts in parallel with more than one target, you can compare performance by audience and start to understand the drivers of success across concepts. 2. Make sure that tested concepts are distinct and differentiated. Each concept should be unique because the goal is to learn as much as possible. If you only test three shades of blue, you'll never learn that people actually want red. 3. Test more than once. As you see 'hot spots' form between concept and audience, test variations of your winning concept. Let’s say, for example, that you test three distinct versions of your new product concept—let’s call them Red, Yellow, and Blue. In the first experiment, Red tests well with all three of your target audience segments. In the next experiment, test three versions of Red with all three segments. This next experiment might explore value propositions or particular features or positioning. It’s a way to generate additional learning about strategy: →What problem does Red solve for customers? →Which features drive interest in Red? →Which positioning helps to interest people in Red? 4. Be aware of your testing environment and how it creates bias (or not) for your experiment. I prefer real-life in-market experiments, with just enough exposure to generate statistically valid results; others prefer ‘lab-based’ testing. Either way, think about how representative your environment is of your eventual launch. The next time you’re making a big move, remember: experiments are a powerful way to reduce risk, whether you are launching a new product, repositioning a brand, or prioritizing a product pipeline. Happy experimenting! #LIPostingDayJune

The companies that grow the fastest scale their experimentation programs. These are the 3 keys: 1. Trustworthy experiments 2. Institutional memory 3. Data culture Let me explain each. — PILLAR 1: TRUSTWORTHY EXPERIMENTS Three challenges block trust. Here’s how to solve them: Challenge 1: Outlier Customers One enterprise client can skew data like 200 average users. Results warp. You build for the 1%, not the majority. Solution: Use stratified sampling. Balance test groups by customer size. Turn outliers into insights, not noise. Challenge 2: Novelty Effects Week 1 shows amazing results. By Week 6, you're back to baseline. This classic trap wastes months on temporary wins. Solution: Track metrics over weeks, not days. Create holdout groups to measure true impact. Don't celebrate until you see sustained value. Challenge 3: Consistency Issues Different teams get contradictory results. Trust collapses. Progress stalls. Solution: Standardize methodology across teams. Create unified playbooks. — PILLAR 2: INSTITUTIONAL MEMORY Most companies run experiments but fail to build lasting knowledge. Here are the 3 elements you need: Element 1: Batting Average View Track your success rate (industry average: 33%). Measure your average lift (typically 8%). Focus on high-probability experiments instead of random testing. Element 2: Frictionless Documentation Documentation fails when it's manual work. Automate capturing rationale, setup, and results. When documentation is automatic, it actually happens. Element 3: Cross-Team Learning Growth, marketing, product—each runs valuable experiments. Insights often die in silos. Build shared repositories. New hires gain years of wisdom instantly. — PILLAR 3: DATA CULTURE Even perfect experiments fail without the right cultural foundation. These 3 elements create that foundation: Element 1: Standardized Definitions Create a metrics dictionary everyone follows: Revenue = Monthly recurring revenue only Engagement = Sessions >2 min with 3+ page views When everyone measures the same way, results become comparable. Element 2: Truth Over Gaming Value right actions over being right. Create safe spaces for negative results. Element 3: Statistical Literacy Help teams understand error margins. Separate signal from noise. No advanced degrees required. Just enough knowledge to make good decisions. — LEARN MORE In my deepdive (free, no paywall thanks to Statsig): https://lnkd.in/etAGf7Nu — THE BOTTOM LINE The cost of not building this system? Testing the same ideas repeatedly. Forgetting what you've learned. Seeing competition pull ahead. What pillar do you need to focus on?

The old adage of “measure twice, cut once” is great when you are buying expensive fabric. But it puts the incentive on getting it right the first time, which goes against best practices for innovation. The solution is get cheaper fabric – and here’s what that looks like: 🚧 Put guardrails around your R&D work to help teams feel more comfortable failing. Innovation should not look like expensive, long-drawn out experiments with no results. Expect some sort of learnings (results) in a shorter amount of time, with a tighter budget. 🔍 Expect your teams to prototype and be smart about testing assumptions. Think like a scientist and design a smart experiment that can get you answers quicker, faster, cheaper. If you don’t need silk to prove that your shirt design fits the model, then use the cheaper fabric before sinking cost. This is hard to learn as an organization, and hard to do generally because we are wired to add on more and more, “yes, and”ing into infinity until we’ve added so much bloat and scope that the costs rise high. We need to learn a lean and subtractive mindset that narrows in on the factors that indicate success, and develop smart tests that help us understand if our core assumptions make sense. Leaders can get wary of innovation work because typically we assume it’s a big money and time sink without much ROI, but we’re doing it wrong. We have to create safe spaces for teams to experiment and fail (and learn), that the organization is comfortable with the investment, and test more ideas through cheaper, faster experiments. These are smart guardrails that we can establish to cultivate a more innovation-friendly environment. #innovation #design #leadership #designthinking This is the 2nd of 3 posts this week on innovation. Connect with me if this resonates ~

Had the incredible opportunity to moderate an amazing panel session with 5 of the top experimentation experts in the field: Steph Le Prevost, Gabriela Florea, Dave Gowans, Marc Uitterhoeve, and Will Laurenson. Our discussion lasted 2 hours, but we certainly could have talked for many more! Distilled down for you, from 2 hours, to less than 500 words, here's the top-5 take aways revealed: 1️⃣ KNOW YOUR AUDIENCE It's absolutely crucial you are completely tapped into who your audience is, what their needs are, what resonates with them, and why they'll buy -- or they won't buy from you! Use data-driven research techniques, including AI-based data mining to deeply understand who your audience is and what makes them tick. Then, and only then, apply this knowledge to formulate and execute your test ideas and parameters. 2️⃣ THINK LOCAL It's not enough to just know your audience. You also need to be able to intricately cater to their specific needs, based on local, geographical, and cultural factors. Don't treat all users alike, and recognize that you are not your user. Segment audiences based on their region, or cohort, and appeal to each distinct group accordingly. 3️⃣ FOLLOW A ROBUST EXPERIMENTATION FRAMEWORK Whether it's the AIDA model, the ALARM protocol, the DEXTER method, or another experimentation framework, use data to develop an informed hypothesis. Then, clearly define the test parameters with your audience in mind. Strike a balance between the MDE and MVP of the test design. Ensure both the lift and the dev/build are reasonable to justify the resources involved. Don't fool yourself. Go in expecting the test will lose. Then, use rigour and critical discernment to explore alternative executions that could optimize upon your current ideas and execution even more. 4️⃣ BEST PRACTICES & TEMPLATES ARE A GREAT LAUNCHING PAD There's nothing wrong with using website templates and best practices as the starting point for building and optimizing your site. In fact, doing so makes efficient use of existing resources and helps ensure a smooth, "prototypical" or expected user experience. The problems occur when you only rely on best practices and blindly copy competitors. Don't do that! Instead, take templates and use competitor sites as inspiration, finding ways to optimize and innovate for your specific situation and audience. 5️⃣ DON'T JUMP THE SHARK Effective test design requires a very linear, methodical process that goes: Data --> Hypothesis --> Test design --> Development --> QA --> Launch --> Result Remove any one of these important steps and you've created a gashing whole in the process. This gap can cause discordance that may result in lost revenue opportunities, causing you to crash, burn, or get eaten alive. Don't fall victim. Curious to hear more? Check out this free, incredible 2-hour masterclass moderated by me and put on by the award-winning Convert.com platform. Link to masterclass replay below ⬇