Evaluating User Satisfaction on Social Media Platforms

Brains aren’t calculators (they really aren’t). People compare, not score, so why do we keep asking for numbers when their minds work in stories and snapshots? I used to rely heavily on rating questions in UX studies. You’ve probably used them too. Rate the ease of a task from 1 to 7 or indicate satisfaction on a scale from 1 to 10. These questions feel measurable and look neat in reports, but after running enough sessions, I started noticing a pattern. A participant would finish a task and pause when asked for a score. They’d hesitate, look unsure, and eventually say something like, “Maybe a six?” followed by, “I’m not really sure what that means.” That hesitation is not about the experience itself. It’s about the format of the question. Most people do not evaluate their experiences using numbers. They judge by comparing, whether against other apps, past expectations, or familiar interactions. When I started asking questions like “How did that compare to what you’re used to?” or “Was that easier or harder than expected?” the responses became clearer and more useful. Participants shared what stood out, what surprised them, and what felt better or worse. Their answers were grounded in real impressions, not guesses. This shift from rating questions to comparison questions changed how I run research. Rating scales flatten experiences into abstract numbers. Comparison questions surface preference, context, and emotion. They help users express themselves in the way they naturally reflect on experiences. And they help researchers hear the parts of the experience that actually drive behavior. There is strong support for this in cognitive science. Tversky’s Elimination by Aspects model shows that people decide by gradually filtering out options that lack something important. Prototype theory explains that we judge how well something matches our internal image of what “good” looks like. Both models show that people think in relative terms, not fixed scores. Even heuristic evaluation in usability relies on comparing designs to expected norms and mental shortcuts, not isolated measurement. These models all point to the same idea. People understand and evaluate experiences through contrast. Asking them to rate something on a scale often hides what they really feel. Asking them to compare helps them express it. I still use quantitative data when needed. It helps with tracking and reporting. But when I want to understand why something works or fails, I ask comparison questions. Because users don’t think in scores. They think in reference points, in expectations, and in choices. That is what we should be listening to.

❓Are we thinking wrongly about user satisfaction 😁? Very frequently in UX and marketing we find that the distribution of user satisfaction ratings is ‘J’ shaped - most of the ratings are positive, very few are in the middle and some are negative. Sometimes this shape is taken at face value and even considered to be canonical or the “ground truth” for how satisfaction distributions should look like. 🛑 But there is a major caveat. Users are self-selecting to answer in-product surveys and rate products and experiences. Users that are very satisfied and very dissatisfied are more likely to respond and that results in the ‘J’ shape we see. Two recent articles I came across, demonstrate this point beautifully. ✅ In “The Polarity of Online Reviews: Prevalence, Drivers and Implications” researchers examined 280 million reviews from 25 major online platforms and showed that platforms that have more reviews submitted per user on average, have less ‘J’ shaped and more normally distributed ratings (e.g. Netflix, IMDB, RateBeer). That is because more reviews per user means users are more likely to review all products and not just the ones they really liked or really disliked. They complimented these findings with a survey where they either asked people to review the last restaurant they visited (forced review condition) or review a restaurant they would be most likely to write a review for (self-selection condition). Lo and behold, the ratings in the forced review condition were normally distributed and the ratings in the self-selection condition were ‘J’ shaped. https://lnkd.in/gGYnM4vD ✅ Another group of researchers made a similar finding using machine learning in “Positivity Bias in Customer Satisfaction Ratings”. They analyzed online chat logs of over 170,000 sessions from Samsung’s customer service using a neural network to predict a satisfaction rating for all sessions. They then compared the distribution of these predicted ratings to the distribution of self-reported satisfaction by users for a fraction of the sessions. The self-report distribution was ‘J’ shaped while the predicted satisfaction rating had a similar number of satisfied and dissatisfied users (they use a binary categorization in the neural network). https://lnkd.in/gaKedKNb 💡 How can we change our thinking and practices with satisfaction ratings? First, don’t assume ‘J’ shape is canonical or ground truth. Just stop doing it right now. Second, favor data collection methods that yield more normal distributions of Satisfaction (e.g. platforms, question types, using panels). Third, incentivize users to respond less selectively if possible (e.g. ask about the last experience, provide small compensation or encouragement). #data #datascience #ux #uxresearch #surveys #marketing

User experience surveys are often underestimated. Too many teams reduce them to a checkbox exercise - a few questions thrown in post-launch, a quick look at average scores, and then back to development. But that approach leaves immense value on the table. A UX survey is not just a feedback form; it’s a structured method for learning what users think, feel, and need at scale- a design artifact in its own right. Designing an effective UX survey starts with a deeper commitment to methodology. Every question must serve a specific purpose aligned with research and product objectives. This means writing questions with cognitive clarity and neutrality, minimizing effort while maximizing insight. Whether you’re measuring satisfaction, engagement, feature prioritization, or behavioral intent, the wording, order, and format of your questions matter. Even small design choices, like using semantic differential scales instead of Likert items, can significantly reduce bias and enhance the authenticity of user responses. When we ask users, "How satisfied are you with this feature?" we might assume we're getting a clear answer. But subtle framing, mode of delivery, and even time of day can skew responses. Research shows that midweek deployment, especially on Wednesdays and Thursdays, significantly boosts both response rate and data quality. In-app micro-surveys work best for contextual feedback after specific actions, while email campaigns are better for longer, reflective questions-if properly timed and personalized. Sampling and segmentation are not just statistical details-they’re strategy. Voluntary surveys often over-represent highly engaged users, so proactively reaching less vocal segments is crucial. Carefully designed incentive structures (that don't distort motivation) and multi-modal distribution (like combining in-product, email, and social channels) offer more balanced and complete data. Survey analysis should also go beyond averages. Tracking distributions over time, comparing segments, and integrating open-ended insights lets you uncover both patterns and outliers that drive deeper understanding. One-off surveys are helpful, but longitudinal tracking and transactional pulse surveys provide trend data that allows teams to act on real user sentiment changes over time. The richest insights emerge when we synthesize qualitative and quantitative data. An open comment field that surfaces friction points, layered with behavioral analytics and sentiment analysis, can highlight not just what users feel, but why. Done well, UX surveys are not a support function - they are core to user-centered design. They can help prioritize features, flag usability breakdowns, and measure engagement in a way that's scalable and repeatable. But this only works when we elevate surveys from a technical task to a strategic discipline.