Mobile User Experience

Don't just blindly use LLMs, evaluate them to see if they fit into your criteria. Not all LLMs are created equal. Here’s how to measure whether they’re right for your use case👇 Evaluating LLMs is critical to assess their performance, reliability, and suitability for specific tasks. Without evaluation, it would be impossible to determine whether a model generates coherent, relevant, or factually correct outputs, particularly in applications like translation, summarization, or question-answering. Evaluation ensures models align with human expectations, avoid biases, and improve iteratively. Different metrics cater to distinct aspects of model performance:  Perplexity quantifies how well a model predicts a sequence (lower scores indicate better familiarity with the data), making it useful for gauging fluency. ROUGE-1 measures unigram (single-word) overlap between model outputs and references, ideal for tasks like summarization where content overlap matters. BLEU focuses on n-gram precision (e.g., exact phrase matches), commonly used in machine translation to assess accuracy. METEOR extends this by incorporating synonyms, paraphrases, and stemming, offering a more flexible semantic evaluation. Exact Match (EM) is the strictest metric, requiring verbatim alignment with the reference, often used in closed-domain tasks like factual QA where precision is paramount. Each metric reflects a trade-off: EM prioritizes literal correctness, while ROUGE and BLEU balance precision with recall. METEOR and Perplexity accommodate linguistic diversity, rewarding semantic coherence over exact replication. Choosing the right metric depends on the task—e.g., EM for factual accuracy in trivia, ROUGE for summarization breadth, and Perplexity for generative fluency. Collectively, these metrics provide a multifaceted view of LLM capabilities, enabling developers to refine models, mitigate errors, and align outputs with user needs. The table’s examples, such as EM scoring 0 for paraphrased answers, highlight how minor phrasing changes impact scores, underscoring the importance of context-aware metric selection. Know more about how to evaluate LLMs: https://lnkd.in/gfPBxrWc Here is my complete in-depth guide on evaluating LLMs: https://lnkd.in/gjWt9jRu Follow me on my YouTube channel so you don't miss any AI topic: https://lnkd.in/gMCpfMKh

Founders who actually use their own product and become part of their target audience get to really understand the pains. Being a founder who uses your own product puts you in your customers' shoes. You see firsthand what works, what doesn’t and where the pain points are. This insider view is priceless because you really understand the needs and frustrations of your audience. When you live your users experience, you build real empathy. You feel their struggles and can create solutions that truly help. This goes beyond just data and survey — it’s about living the same experience. Using your product often helps you spot small but important fixes that might get missed otherwise. These little tweaks can really boost user satisfaction and product quality. Plus being an active user lets you connect with your community better. You join conversations and get direct feedback, keeping you in touch with your users' changing needs. So scratch your own itch and solve problems that you’ve personally experience because this can be a huge competitive advantage.

Evaluations are extremely important for any AI application. That is, how do you know which models to use, if things are working optimally, etc? Today, we’re sharing a bit about our eval stack.   Behind every Decagon AI agent is a rigorous model evaluation engine built for the highest-stakes customer interactions.   When your agents are handling complex, customer-facing use cases, you need more than just promising model outputs. You need a framework that continuously and precisely measures real performance at scale.   In our latest blog post, we break down the core components of that evaluation framework: 🧠 LLM-as-judge evaluation – scoring real-world interactions across relevance, correctness, empathy, and naturalness, with human validation to catch edge cases 📊 Ground truth benchmarking – using curated, expert-labeled datasets to measure factuality and intent coverage 🚦 Live A/B testing – deploying variants in production and measuring their impact on real business outcomes like CSAT and resolution rate   This evaluation doesn’t stop once the latest version of an AI agent ships. Every insight feeds back into prompts, retrieval, and agent logic. The result: continuous improvement in the quality of customer experiences.   Check out the full blog in the comments.

Here is why leaderboards can fool you (and what to do instead) 👇 Benchmarks are macro averages, and your application is a micro reality. A model that’s top-3 on MMLU or GSM-Plus might still bomb when asked to summarize legal contracts, extract SKUs from receipts, or answer domain-specific FAQs. That’s because: 👉 Benchmarks skew toward academic tasks and short-form inputs. Most prod systems run multi-turn, tool-calling, or retrieval workflows the benchmark never sees. 👉 Scores are single-shot snapshots. They don’t cover latency, cost, or robustness to adversarial prompts. 👉 The “average of many tasks” hides mode failures. A 2-point gain in translation might mask a 20-point drop in structured JSON extraction. In short, public leaderboards tell you which model is good in general, not which model is good for you . 𝗕𝘂𝗶𝗹𝗱 𝗲𝘃𝗮𝗹𝘀 𝘁𝗵𝗮𝘁 𝗺𝗶𝗿𝗿𝗼𝗿 𝘆𝗼𝘂𝗿 𝘀𝘁𝗮𝗰𝗸 1️⃣ Trace the user journey. Map the critical steps (retrieve, route, generate, format). 2️⃣ Define success per step. Example metrics: → Retrieval → document relevance (binary). → Generation → faithfulness (factual / hallucinated). → Function calls → tool-choice accuracy (correct / incorrect). 3️⃣ Craft a golden dataset. 20-100 edge-case examples that stress real parameters (long docs, unicode, tricky entities). 4️⃣ Pick a cheap, categorical judge. “Correct/Incorrect” beats 1-5 scores for clarity and stability 5️⃣ Automate in CI/CD and prod. Gate PRs on offline evals; stream online evals for drift detection. 6️⃣ Iterate relentlessly. False negatives become new test rows; evaluator templates get tightened; costs drop as you fine-tune a smaller judge. When you evaluate the system, not just the model, you’ll know exactly which upgrade, prompt tweak, or retrieval change pushes the real-world metric that matters: user success. How are you’re tailoring evals for your own LLM pipeline? Always up to swap notes on use-case-driven benchmarking Image Courtesy: Arize AI ---------- Share this with your network ♻️ Follow me (Aishwarya Srinivasan) for more AI insights and resources!

Look at what they do, not just what they say. User behavior is how users interact with and use software. It includes things like: → how people navigate the interface → which features people use most often → the order in which people perform tasks → how much time people spend on activities → how people react to prompts or feedback Product managers and designers must understand these behaviors. Analyzing user behavior can enhance the user experience, simplify processes, spot issues, and make the software more effective. Discovering the "why" behind user actions is the key to creating great software. In many of my sales discussions with teams, I notice that most rely too heavily on interviews to understand user problems. While interviews are a good starting point, they only cover half of the picture. What’s the benefit of going beyond interviews? → See actual user behavior, not just reported actions → Gain insights into unspoken needs in natural settings → Minimize behavior changes by observing discreetly → Capture genuine interactions for better data → Document detailed behaviors and interactions → Understand the full user journey and hidden pain points → Discover issues and opportunities users miss → Identify outside impacts on user behavior Most people don't think in a hyper-rational way—they're just trying to fit in. That's why when we built Helio, we included task-based activities to learn from users' actions and then provided follow-up questions about their thoughts and feelings. User behaviors aren't always rational. Several factors contribute to this: Cognitive Biases ↳ Users rely on mental shortcuts, often sticking to familiar but inefficient methods. Emotional Influence ↳ Emotions like stress or frustration can lead to hasty or illogical decisions. Habits and Routine ↳ Established habits may cause users to overlook better options or new features. Lack of Understanding ↳ Users may make choices based on limited knowledge, leading to seemingly irrational actions. Contextual Factors ↳ External factors like time pressure or distractions can impact user behavior. Social Influence ↳ Peer pressure or the desire to conform can also drive irrational choices. Observing user behavior, especially in large sample sizes, helps designers see how people naturally use products. This method gives a clearer and more accurate view of user behavior, uncovering hidden needs and issues that might not surface in interviews. #productdesign #productdiscovery #userresearch #uxresearch

In the last 90 days I spoke to 12 CXO.  They all said one thing: GenAI doesn't deliver business value. The reason? It’s not because of model choice. Not because of bad prompts. But because they skip the most important part: LLM evaluation This is why evals matter. In one Datali project, testing took us from 60% to 92% accuracy. Not by luck and blind trying. But by building a rigorous, automated testing pipeline. Here’s the boring but harsh truth: You don’t write a perfect system prompt and test it. You write tests first and discover prompts that pass them. This what you get: 1//  You gain crystal clear visibility - the perfect picture of what works and what doesn’t. You see how your system behaves across real-world inputs. You know where failures happen and why. You can plan risk mitigation strategies early 2// You iterate faster. Once you're testing thoroughly, you can run more experiments, track their results and revisit what worked best. Even months later. You catch problems early. You refine prompts, add data or fine-tune with confidence. You iterate faster from PoC → MVP → production, adjusting to user feedback without guesswork. 3// You build better products in less time. The better means here: Higher accuracy → less hallucination, better task handling. More stability → no surprises in production, fewer user complaints. 4// You reach the desired business impact: ROI, KPIs and cost savings. This is the combined result of previous actions. They drive your KPIs. If your system is accurate, stable and aligned to the user’s goals - that’s everything you need. Shorter development cycles = faster time to market Fewer bugs = lower support costs Focused iterations = less wasted dev time It’s priceless. But you can get it only with the right approach.

Tools create a path of least resistance through the way they are designed - some things become easier (encouraging users to do them) and some things become comparatively harder (discouraging that behavior). When it comes to AI chatbots, the design encourages users to trust the AI's outputs. Unfortunately, all LLMs hallucinate - it's only a matter of when. And as users get used to relying on the machine, their ability and willingness to spot these errors deteriorates. In the exact situation where the human in the loop is necessary, that human is unequipped to step in. Blaming the user for this is irresponsible. The problem is caused by the way these tools are designed - so it's up to us, as designers, to fix it. https://lnkd.in/eSmv_8yv

Why most software fails? It wasn’t built for the people using it. It was built for a spec. For a stakeholder. For a presentation slide. But not for the person who opens it every morning and just wants things to work. We’ve seen it happen: → Systems that look sleek, but no one understands → Dashboards that report everything, but say nothing → Tools packed with features, and full of frustration That’s not progress. That’s noise. Here’s what we do instead: We start with the user. → We watch how they actually work → We ask what they avoid, and why → We test early, not just at the end → We cut what’s confusing → We refine what’s unclear → We keep it honest, simple over clever If your team needs a tutorial to use it, we built it wrong. Good software feels obvious. Comfortable. Almost invisible. That’s the goal. If you’ve been burned by “great tools” that never landed with your team, Let’s build something they’ll actually want to use. Because that’s where the real ROI lives.

Are we benchmarking LLMs the wrong way? Why we need more LLM arenas. Benchmarks are essential for evaluating the performance of Large Language Models (LLMs). But are they capturing what really matters—the user experience? A new study suggests that we might be missing the mark. Researchers collected 1,863 real-world use cases from 712 participants across 23 countries. They call this the User Reported Scenarios (URS) dataset. Using URS, they benchmarked 10 LLM services on their ability to satisfy user needs across 7 different intent categories. The results are showing that the benchmark scores aligned well with user-reported experiences, highlighting a critical oversight in current evaluation practices: subjective scenarios. The study proposes a paradigm shift in how we evaluate LLMs, moving from predefined abilities to a user-centric perspective. By benchmarking LLMs based on authentic, diverse user needs, we can ensure that these powerful tools are truly serving their intended purpose—collaborating with and assisting users in the real world. ↓ Liked this post? Follow the link under my name and never miss a paper highlight again 💡

Traditional usability tests often treat user experience factors in isolation, as if different factors like usability, trust, and satisfaction are independent of each other. But in reality, they are deeply interconnected. By analyzing each factor separately, we miss the big picture - how these elements interact and shape user behavior. This is where Structural Equation Modeling (SEM) can be incredibly helpful. Instead of looking at single data points, SEM maps out the relationships between key UX variables, showing how they influence each other. It helps UX teams move beyond surface-level insights and truly understand what drives engagement. For example, usability might directly impact trust, which in turn boosts satisfaction and leads to higher engagement. Traditional methods might capture these factors separately, but SEM reveals the full story by quantifying their connections. SEM also enhances predictive modeling. By integrating techniques like Artificial Neural Networks (ANN), it helps forecast how users will react to design changes before they are implemented. Instead of relying on intuition, teams can test different scenarios and choose the most effective approach. Another advantage is mediation and moderation analysis. UX researchers often know that certain factors influence engagement, but SEM explains how and why. Does trust increase retention, or is it satisfaction that plays the bigger role? These insights help prioritize what really matters. Finally, SEM combined with Necessary Condition Analysis (NCA) identifies UX elements that are absolutely essential for engagement. This ensures that teams focus resources on factors that truly move the needle rather than making small, isolated tweaks with minimal impact.