Analyzing User Behavior in Voice-Activated Systems

A lot of us still rely on simple trend lines or linear regression when analyzing how user behavior changes over time. But in recent years, the tools available to us have evolved significantly. For behavioral and UX data - especially when it's noisy, nonlinear, or limited - there are now better methods to uncover meaningful patterns. Machine learning models like LSTMs can be incredibly useful when you’re trying to understand patterns that unfold across time. They’re good at picking up both short-term shifts and long-term dependencies, like how early frustration might affect engagement later in a session. If you want to go further, newer models that combine graph structures with time series - like graph-based recurrent networks - help make sense of how different behaviors influence each other. Transformers, originally built for language processing, are also being used to model behavior over time. They’re especially effective when user interactions don’t follow a neat, regular rhythm. What’s interesting about transformers is their ability to highlight which time windows matter most, which makes them easier to interpret in UX research. Not every trend is smooth or gradual. Sometimes we’re more interested in when something changes - like a sudden drop in satisfaction after a feature rollout. This is where change point detection comes in. Methods like Bayesian Online Change Point Detection or PELT can find those key turning points, even in noisy data or with few observations. When trends don’t follow a straight line, generalized additive models (GAMs) can help. Instead of fitting one global line, they let you capture smooth curves and more realistic patterns. For example, users might improve quickly at first but plateau later - GAMs are built to capture that shape. If you’re tracking behavior across time and across users or teams, mixed-effects models come into play. These models account for repeated measures or nested structures in your data, like individual users within groups or cohorts. The Bayesian versions are especially helpful when your dataset is small or uneven, which happens often in UX research. Some researchers go a step further by treating behavior over time as continuous functions. This lets you compare entire curves rather than just time points. Others use matrix factorization methods that simplify high-dimensional behavioral data - like attention logs or biometric signals - into just a few evolving patterns. Understanding not just what changed, but why, is becoming more feasible too. Techniques like Gaussian graphical models and dynamic Bayesian networks are now used to map how one behavior might influence another over time, offering deeper insights than simple correlations. And for those working with small samples, new Bayesian approaches are built exactly for that. Some use filtering to maintain accuracy with limited data, and ensemble models are proving useful for increasing robustness when datasets are sparse or messy.

People often say what they think they should say. I had a great exchange with 👋 Brandon Spencer, who highlighted the challenges of using qualitative user research. He suggested that qual responses are helpful, but you have to read between the lines more than you do when watching what they do. People often say what they think they should be saying and do what they naturally would. I agree. Based on my digital experiences, there are several reasons for this behavior. People start with what they know or feel, filtered by their long-term memory. Social bias ↳ People often say what they think they should be saying because they want to present themselves positively, especially in social or evaluative situations. Jakob's Law ↳ Users spend most of their time on other sites, meaning they speak to your site/app like the sites they already know. Resolving these issues in UX research requires a multi-faceted approach that considers what users say (user wants) and what they do (user needs) while accounting for biases and user expectations. Here’s how we tackle these issues: 1. Combine qualitative and quantitative research We use Helio to pull qualitative insights to understand the "why" behind user behavior but validate these insights with quantitative data (e.g., structured behavioral questions). This helps to balance what users say with what they do. 2. Test baselines with your competitors Compare your design with common patterns with which users are familiar. Knowing this information reduces cognitive load and makes it easier for users to interact naturally with your site on common tasks. 3. Allow anonymity  Allow users to provide feedback anonymously to reduce the pressure to present themselves positively. Helio automatically does this while still creating targeted audiences. We also don’t do video. This can lead to more honest and authentic responses. 4. Neutral questioning We frame questions to reduce the likelihood of leading or socially desirable answers. For example, ask open-ended questions that don’t imply a “right” answer. 5. Natural settings Engage with users in their natural environment and devices to observe their real behavior and reduce the influence of social bias. Helio is a remote platform, so people can respond wherever they want. The last thing we have found is that by asking more in-depth questions and increasing participants, you can gain stronger insights by cross-referencing data. → Deeper: When users give expected or socially desirable answers, ask follow-up questions to explore their true thoughts and behaviors. → Wider: Expand your sample size (we test with 100 participants) and keep testing regularly. We gather 10,000 customer answers each month, which helps create a broader and more reliable data set. Achieving a more accurate and complete understanding of user behavior is possible, leading to better design decisions. #productdesign #productdiscovery #userresearch #uxresearch

I’ve open-sourced a key component of one of my latest projects: Voice Lab, a comprehensive testing framework that removes the guesswork from building and optimizing voice agents across language models, prompts, and personas. Speech is increasingly becoming a prominent modality companies employ to enable user interaction with their products, yet the AI community is still figuring out systematic evaluation for such applications. Key features: (1) Metrics and analysis – define custom metrics like brevity or helpfulness in JSON format and evaluate them using LLM-as-a-Judge. No more manual reviews. (2) Model migration and cost optimization – confidently switch between models (e.g., from GPT-4 to smaller models) while evaluating performance and cost trade-offs. (3) Prompt and performance testing – systematically test multiple prompt variations and simulate diverse user interactions to fine-tune agent responses. (4) Testing different agent personas, from an angry United Airlines representative to a hotel receptionist who tries to jailbreak your agent to book all available rooms. While designed for voice agents, Voice Lab is versatile and can evaluate any LLM-based agent. ⭐️ I invite the community to contribute and would highly appreciate your support by starring the repo to make it more discoverable for others. GitHub repo (commercially permissive) https://lnkd.in/gAaZ-tkA