User Experience Metrics That Highlight Areas For Improvement

Most teams pick metrics that sound smart… But under the hood, they’re just noisy, slow, misleading, or biased. But today, I'm giving you a framework to avoid that trap. It’s called STEDII and it’s how to choose metrics you can actually trust: — ONE: S — Sensitivity Your metric should be able to detect small but meaningful changes Most good features don’t move numbers by 50%. They move them by 2–5%. If your metric can’t pick up those subtle shifts , you’ll miss real wins. Rule of thumb: - Basic metrics detect 10% changes - Good ones detect 5% - Great ones? 2% The better your metric, the smaller the lift it can detect. But that also means needing more users and better experimental design. — TWO: T — Trustworthiness Ever launch a clearly better feature… but the metric goes down? Happens all the time. Users find what they need faster → Time on site drops Checkout becomes smoother → Session length declines A good metric should reflect actual product value, not just surface-level activity. If metrics move in the opposite direction of user experience, they’re not trustworthy. — THREE: E — Efficiency In experimentation, speed of learning = speed of shipping. Some metrics take months to show signal (LTV, retention curves). Others like Day 2 retention or funnel completion give you insight within days. If your team is waiting weeks to know whether something worked, you're already behind. Use CUPED or proxy metrics to speed up testing windows without sacrificing signal. — FOUR: D — Debuggability A number that moves is nice. A number you can explain why something worked? That’s gold. Break down conversion into funnel steps. Segment by user type, device, geography. A 5% drop means nothing if you don’t know whether it’s: → A mobile bug → A pricing issue → Or just one country behaving differently Debuggability turns your metrics into actual insight. — FIVE: I — Interpretability Your whole team should know what your metric means... And what to do when it changes. If your metric looks like this: Engagement Score = (0.3×PageViews + 0.2×Clicks - 0.1×Bounces + 0.25×ReturnRate)^0.5 You’re not driving action. You’re driving confusion. Keep it simple: Conversion drops → Check checkout flow Bounce rate spikes → Review messaging or speed Retention dips → Fix the week-one experience — SIX: I — Inclusivity Averages lie. Segments tell the truth. A metric that’s “up 5%” could still be hiding this: → Power users: +30% → New users (60% of base): -5% → Mobile users: -10% Look for Simpson’s Paradox. Make sure your “win” isn’t actually a loss for the majority. — To learn all the details, check out my deep dive with Ronny Kohavi, the legend himself: https://lnkd.in/eDWT5bDN

When I was interviewing users during a study on a new product design focused on comfort, I started to notice some variation in the feedback. Some users seemed quite satisfied, describing it as comfortable and easy to use. Others were more reserved, mentioning small discomforts or saying it didn’t quite feel right. Nothing extreme, but clearly not a uniform experience either. Curious to see how this played out in the larger dataset, I checked the comfort ratings. At first, the average looked perfectly middle-of-the-road. If I had stopped there, I might have just concluded the product was fine for most people. But when I plotted the distribution, the pattern became clearer. Instead of a single, neat peak around the average, the scores were split. There were clusters at both the high and low ends. A good number of people liked it, and another group didn’t, but the average made it all look neutral. That distribution plot gave me a much clearer picture of what was happening. It wasn’t that people felt lukewarm about the design. It was that we had two sets of reactions balancing each other out statistically. And that distinction mattered a lot when it came to next steps. We realized we needed to understand who those two groups were, what expectations or preferences might be influencing their experience, and how we could make the product more inclusive of both. To dig deeper, I ended up using a mixture model to formally identify the subgroups in the data. It confirmed what we were seeing visually, that the responses were likely coming from two different user populations. This kind of modeling is incredibly useful in UX, especially when your data suggests multiple experiences hidden within a single metric. It also matters because the statistical tests you choose depend heavily on your assumptions about the data. If you assume one unified population when there are actually two, your test results can be misleading, and you might miss important differences altogether. This is why checking the distribution is one of the most practical things you can do in UX research. Averages are helpful, but they can also hide important variability. When you visualize the data using a histogram or density plot, you start to see whether people are generally aligned in their experience or whether different patterns are emerging. You might find a long tail, a skew, or multiple peaks, all of which tell you something about how users are interacting with what you’ve designed. Most software can give you a basic histogram. If you’re using R or Python, you can generate one with just a line or two of code. The point is, before you report the average or jump into comparisons, take a moment to see the shape of your data. It helps you tell a more honest, more detailed story about what users are experiencing and why. And if the shape points to something more complex, like distinct user subgroups, methods like mixture modeling can give you a much more accurate and actionable analysis.

Compare designs to show improvement and build trust. Design is about understanding and managing change for users and stakeholders. If you change something too much, it might overwhelm users or lead to negative feedback. If you only slightly change an underperforming screen or page, the improvement might not generate the lift stakeholders seek. In the past, understanding a stakeholder’s needs was often enough to add value to design. But now, with established design patterns and increased specialization, designers need to answer a more specific question: How much did this design improve? Lately, I’ve been posting a lot about measuring design. Measuring design helps build trust and transparency in the process, but it’s only helpful if you have something to compare your design. Here are 11 ways to compare your work, also known as UX benchmarking. We use Helio to test. 1. Competitors See how your metrics compare to similar features in competing products. This will show you where you’re strong and where you can improve. 2. Iterations Track metrics across design versions to see if changes make the user experience better or worse. 3. Timeline Look at metrics over time to find patterns, like seasonal changes or long-term trends. 4. Segments Break down metrics by user groups (like age or location) to understand different experiences and make targeted improvements. 5. Journeys Check metrics at each user journey stage to see where users get the most value or run into issues. 6. Platforms/Devices Compare across devices (like mobile vs. desktop) to spot and fix issues specific to each platform. 7. User Goals/Tasks Focus on specific tasks (like completing a task vs. exploring) to see if the product supports what users want to do. 8. Feature Usage Review metrics for individual features to prioritize improvements for high-value or underperforming areas. 9. Geographies Compare by region to see if user experience differs in various parts of the world. 10. User Lifecycle Look at new vs. experienced users to understand adoption patterns and loyalty. 11. Behavioral Triggers Examine how specific actions (like seeing a tutorial) affect user satisfaction and behavior. If these ideas excite you, DM me–we’re focused on finalizing Glare, our open UX metrics framework, for its public 1.0 release (https://glare.helio.app/). We've been refining the ways to benchmark UX design work to support individual product designers and teams. #productdesign #productdiscovery #userresearch #uxresearch