Writing Survey Questions That Minimize Confusion

Remember that bad survey you wrote? The one that resulted in responses filled with blatant bias and caused you to doubt whether your respondents even understood the questions? Creating a survey may seem like a simple task, but even minor errors can result in biased results and unreliable data. If this has happened to you before, it's likely due to one or more of these common mistakes in your survey design: 1. Ambiguous Questions: Vague wording like “often” or “regularly” leads to varied interpretations among respondents. Be specific—use clear options like “daily,” “weekly,” or “monthly” to ensure consistent and accurate responses. 2. Double-Barreled Questions: Combining two questions into one, such as “Do you find our website attractive and easy to navigate?” can confuse respondents and lead to unclear answers. Break these into separate questions to get precise, actionable feedback. 3. Leading/Loaded Questions: Questions that push respondents toward a specific answer, like “Do you agree that responsible citizens should support local businesses?” can introduce bias. Keep your questions neutral to gather unbiased, genuine opinions. 4. Assumptions: Assuming respondents have certain knowledge or opinions can skew results. For example, “Are you in favor of a balanced budget?” assumes understanding of its implications. Provide necessary context to ensure respondents fully grasp the question. 5. Burdensome Questions: Asking complex or detail-heavy questions, such as “How many times have you dined out in the last six months?” can overwhelm respondents and lead to inaccurate answers. Simplify these questions or offer multiple-choice options to make them easier to answer. 6. Handling Sensitive Topics: Sensitive questions, like those about personal habits or finances, need to be phrased carefully to avoid discomfort. Use neutral language, provide options to skip or anonymize answers, or employ tactics like Randomized Response Survey (RRS) to encourage honest, accurate responses. By being aware of and avoiding these potential mistakes, you can create surveys that produce precise, dependable, and useful information. Art+Science Analytics Institute | University of Notre Dame | University of Notre Dame - Mendoza College of Business | University of Illinois Urbana-Champaign | University of Chicago | D'Amore-McKim School of Business at Northeastern University | ELVTR | Grow with Google - Data Analytics #Analytics #DataStorytelling

Imagine this: you’re filling out a survey and come across a question instructing you to answer 1 for Yes and 0 for No. As if that wasn't bad enough, the instructions are at the top of the page, and when you scroll to answer some of the questions, you’ve lost sight of what 1 and 0 means. Why is this an accessibility fail? Memory Burden: Not everyone can remember instructions after scrolling, especially those with cognitive disabilities or short-term memory challenges. Screen Readers: For people using assistive technologies, the separation between the instructions and the input field creates confusion. By the time they navigate to the input, the context might be lost. Universal Design: It’s frustrating and time-consuming to repeatedly scroll up and down to confirm what the numbers mean. You can improve this type of survey by: 1. Placing clear labels next to each input (e.g., "1 = Yes, 0 = No"). 2. Better yet, use intuitive design and replace numbers with a combo box or radio buttons labeled "Yes" and "No." 3. Group the questions by topic. 4. Use headers and field groups to break them up for screen reader users. 5. Only display five or six at a time so people don't get overwhelmed and bail out. 6. Ensure instructions remain visible or are repeated near the question for easy reference. Accessibility isn’t just a "nice to have." It’s critical to ensure everyone can participate. Don’t let bad design create barriers and invalidate your survey results. Alt: A screen shot of a survey containing numerous questions with an instructing you to answer 1 for Yes and 0 for No. The instruction is written at the top and it gets lost when you scroll down to answer other questions. #AccessibilityFailFriday #AccessibilityMatters #InclusiveDesign #UXBestPractices #DigitalAccessibility

It’s arguably never been easier to run surveys, but that doesn’t mean we’re getting better insights from them. Often this stems back to two main issues 1) lack of clear objectives and hypotheses upfront, and 2) poorly written questions. ✅ Start with a list of hypotheses you want to investigate. Think of these as statements you believe to be true and want to confirm. This should not be a list of stats you’d like to generate from the survey. Instead, what are the ideal “headlines” you’d love to report on? For example, rather than seeking a stat like “60% of Gen Z discover new products on social media compared to 20% of Gen X”, think of the overall insight you want to gain, like “the shopping experience has changed and brands need to adapt their marketing strategy: a majority of Gen Z now use social media to discover new products, while a minority of Gen X shoppers discover products this way”. ⁉️ Now, what questions help you get to these insights? One of the most frequent question pitfalls I see is asking two questions in one question. Don’t ask a question with “or” in the middle. Each question should have a single point to it. E.g. “Which of the below channels do you use for product discovery?” If you want to also learn about channels that they are more likely to convert from, ask it in a different question. Define all terms you are using. What do you mean by “discovery”? Are all the channels you list easily understood? Questions should be as simple and specific as possible: as few words as possible, no fancy vocab. Then test your questions with a few users. Do they all understand and interpret the questions in the same way? If people tell you multiple meanings, be more simple, and more specific. To put these points together, add a sentence to your survey draft above each question (or in some cases, a set of questions) with the headline you ideally want to share.    💡 To summarize, before running a survey, what insights do you want to take from it? And do you have the right question to get you there?