Evaluating User Experience in SaaS Usability Testing

How do you figure out what truly matters to users when you’ve got a long list of features, benefits, or design options - but only a limited sample size and even less time? A lot of UX researchers use Best-Worst Scaling (or MaxDiff) to tackle this. It’s a great method: simple for participants, easy to analyze, and far better than traditional rating scales. But when the research question goes beyond basic prioritization - like understanding user segments, handling optional features, factoring in pricing, or capturing uncertainty - MaxDiff starts to show its limits. That’s when more advanced methods come in, and they’re often more accessible than people think. For example, Anchored MaxDiff adds a must-have vs. nice-to-have dimension that turns relative rankings into more actionable insights. Adaptive Choice-Based Conjoint goes further by learning what matters most to each respondent and adapting the questions accordingly - ideal when you're juggling 10+ attributes. Menu-Based Conjoint works especially well for products with flexible options or bundles, like SaaS platforms or modular hardware, helping you see what users are likely to select together. If you suspect different mental models among your users, Latent Class Models can uncover hidden segments by clustering users based on their underlying choice patterns. TURF analysis is a lifesaver when you need to pick a few features that will have the widest reach across your audience, often used in roadmap planning. And if you're trying to account for how confident or honest people are in their responses, Bayesian Truth Serum adds a layer of statistical correction that can help de-bias sensitive data. Want to tie preferences to price? Gabor-Granger techniques and price-anchored conjoint models give you insight into willingness-to-pay without running a full pricing study. These methods all work well with small-to-medium sample sizes, especially when paired with Hierarchical Bayes or latent class estimation, making them a perfect fit for fast-paced UX environments where stakes are high and clarity matters.

Recently, someone shared results from a UX test they were proud of. A new onboarding flow had reduced task time, based on a very small handful of users per variant. The result wasn’t statistically significant, but they were already drafting rollout plans and asked what I thought of their “victory.” I wasn’t sure whether to critique the method or send flowers for the funeral of statistical rigor. Here’s the issue. With such a small sample, the numbers are swimming in noise. A couple of fast users, one slow device, someone who clicked through by accident... any of these can distort the outcome. Sampling variability means each group tells a slightly different story. That’s normal. But basing decisions on a single, underpowered test skips an important step: asking whether the effect is strong enough to trust. This is where statistical significance comes in. It helps you judge whether a difference is likely to reflect something real or whether it could have happened by chance. But even before that, there’s a more basic question to ask: does the difference matter? This is the role of Minimum Detectable Effect, or MDE. MDE is the smallest change you would consider meaningful, something worth acting on. It draws the line between what is interesting and what is useful. If a design change reduces task time by half a second but has no impact on satisfaction or behavior, then it does not meet that bar. If it noticeably improves user experience or moves key metrics, it might. Defining your MDE before running the test ensures that your study is built to detect changes that actually matter. MDE also helps you plan your sample size. Small effects require more data. If you skip this step, you risk running a study that cannot answer the question you care about, no matter how clean the execution looks. If you are running UX tests, begin with clarity. Define what kind of difference would justify action. Set your MDE. Plan your sample size accordingly. When the test is done, report the effect size, the uncertainty, and whether the result is both statistically and practically meaningful. And if it is not, accept that. Call it a maybe, not a win. Then refine your approach and try again with sharper focus.

You know that feeling when you use a product or app and it just clicks? Everything seems intuitive, easy to navigate, and you get what you need done effortlessly. But then there are those other experiences that make you want to tear your hair out in frustration. What's the difference? It all comes down to usability. The products that just "get you" have gone through rigorous usability testing during their development. Creating a solid usability test script is absolutely important for building an amazing user experience. It goes beyond simply checking if users can navigate your interface; it's about understanding their thought process, struggles, and successes. Here's how to create a script that gets you out of the build trap and into the minds of your users: 1. Define Your Objectives: Start with the end in mind. What do you want to learn from this usability test? Are you testing a new feature, the overall workflow, or the clarity of your content? Be specific. Your objectives will shape your script and ensure you're measuring the right things. 2. Craft Realistic Scenarios: Put your users in the driver's seat with scenarios that mimic real-life tasks. This isn't about leading them to the 'right' answer; it's about observing their natural behavior. What paths do they take? Where do they stumble? Their journey is a goldmine of insights. 3. Ask Open-Ended Questions: Your script should encourage users to think aloud. Ask questions that prompt explanation, not just yes or no answers. "What are you thinking right now?" "How does this feature make you feel?" These questions reveal the 'why' behind user actions. 4. Include Probing Questions: Be ready to dig deeper. When a user hesitates or expresses frustration, that's your cue to explore. "Can you tell me more about that?" "What were you expecting to happen?" These moments are where you'll find the most valuable feedback. 5. Stay Neutral: Your script is not a sales pitch. Avoid leading questions that sway users toward a particular response. You're there to learn from them, not to validate your own assumptions. 6. Pilot Your Script: Test your script with a colleague or friend before going live with users. This dry run will help you refine your questions and ensure the flow feels natural. Remember, the script is not set in stone. If during the test you spot an opportunity to dive deeper into a user's thought process, go for it. The script is a guide, not a script. Stay curious, stay flexible. A great usability test script is the difference between finding out that your users are struggling and understanding why they're struggling. It's the difference between making assumptions and making informed decisions. Your users—and your product—deserve it. Download our comprehensive PDF guide below on crafting your own usability test script to conduct effective tests and elevate your product's user experience. #UsabilityTesting #ProductDevelopment #ProductInstitute