Writing Survey Questions for Online Polls

Remember that bad survey you wrote? The one that resulted in responses filled with blatant bias and caused you to doubt whether your respondents even understood the questions? Creating a survey may seem like a simple task, but even minor errors can result in biased results and unreliable data. If this has happened to you before, it's likely due to one or more of these common mistakes in your survey design: 1. Ambiguous Questions: Vague wording like “often” or “regularly” leads to varied interpretations among respondents. Be specific—use clear options like “daily,” “weekly,” or “monthly” to ensure consistent and accurate responses. 2. Double-Barreled Questions: Combining two questions into one, such as “Do you find our website attractive and easy to navigate?” can confuse respondents and lead to unclear answers. Break these into separate questions to get precise, actionable feedback. 3. Leading/Loaded Questions: Questions that push respondents toward a specific answer, like “Do you agree that responsible citizens should support local businesses?” can introduce bias. Keep your questions neutral to gather unbiased, genuine opinions. 4. Assumptions: Assuming respondents have certain knowledge or opinions can skew results. For example, “Are you in favor of a balanced budget?” assumes understanding of its implications. Provide necessary context to ensure respondents fully grasp the question. 5. Burdensome Questions: Asking complex or detail-heavy questions, such as “How many times have you dined out in the last six months?” can overwhelm respondents and lead to inaccurate answers. Simplify these questions or offer multiple-choice options to make them easier to answer. 6. Handling Sensitive Topics: Sensitive questions, like those about personal habits or finances, need to be phrased carefully to avoid discomfort. Use neutral language, provide options to skip or anonymize answers, or employ tactics like Randomized Response Survey (RRS) to encourage honest, accurate responses. By being aware of and avoiding these potential mistakes, you can create surveys that produce precise, dependable, and useful information. Art+Science Analytics Institute | University of Notre Dame | University of Notre Dame - Mendoza College of Business | University of Illinois Urbana-Champaign | University of Chicago | D'Amore-McKim School of Business at Northeastern University | ELVTR | Grow with Google - Data Analytics #Analytics #DataStorytelling

A good survey works like a therapy session. You don’t begin by asking for deep truths, you guide the person gently through context, emotion, and interpretation. When done in the right sequence, your questions help people articulate thoughts they didn’t even realize they had. Most UX surveys fall short not because users hold back, but because the design doesn’t help them get there. They capture behavior and preferences but often miss the emotional drivers, unmet expectations, and mental models behind them. In cognitive psychology, we understand that thoughts and feelings exist at different levels. Some answers come automatically, while others require reflection and reconstruction. If a survey jumps straight to asking why someone was frustrated, without first helping them recall the situation or how it felt, it skips essential cognitive steps. This often leads to vague or inconsistent data. When I design surveys, I use a layered approach grounded in models like Levels of Processing, schema activation, and emotional salience. It starts with simple, context-setting questions like “Which feature did you use most recently?” or “How often do you use this tool in a typical week?” These may seem basic, but they activate memory networks and help situate the participant in the experience. Visual prompts or brief scenarios can support this further. Once context is active, I move into emotional or evaluative questions (still gently) asking things like “How confident did you feel?” or “Was anything more difficult than expected?” These help surface emotional traces tied to memory. Using sliders or response ranges allows participants to express subtle variations in emotional intensity, which matters because emotion often turns small usability issues into lasting negative impressions. After emotional recall, we move into the interpretive layer, where users start making sense of what happened and why. I ask questions like “What did you expect to happen next?” or “Did the interface behave the way you assumed it would?” to uncover the mental models guiding their decisions. At this stage, responses become more thoughtful and reflective. While we sometimes use AI-powered sentiment analysis to identify patterns in open-ended responses, the real value comes from the survey’s structure, not the tool. Only after guiding users through context, emotion, and interpretation do we include satisfaction ratings, prioritization tasks, or broader reflections. When asked too early, these tend to produce vague answers. But after a structured cognitive journey, feedback becomes far more specific, grounded, and actionable. Adaptive paths or click-to-highlight elements often help deepen this final stage. So, if your survey results feel vague, the issue may lie in the pacing and flow of your questions. A great survey doesn’t just ask, it leads. And when done right, it can uncover insights as rich as any interview. *I’ve shared an example structure in the comment section.

On survey items and publication (or get it right or get out of here!) As an author & an editor, one of the most damning indictments of a paper is a reviewer saying "the items do not measure what the authors claim to study." When I see that criticism, I typically flip through the paper, look at the items, & more often than I would like, the reviewer is right. Leaving little choice, re-do the study or have it rejected. This is frustrating, bc designing effective measures is within the reach of any author. While one can spend a lifetime studying item development, there are also simple guides, like this one offered by Pew (https://lnkd.in/ei-7vzfz), that, if you pay attention, can help you pre-empt many potential criticisms of your work. But. It takes time. Which is time well-spent, because designing effective survey questions is a necessary condition for conducting high impact research. Why? Because poorly written questions lead to confusion, biased answers, or incomplete responses, which undermine the validity of a study's findings. When well-crafted, a survey elicits accurate responses, ensures concepts are operationalized properly, & create opportunities to provide actionable insights. So how to do it? According to Pew Research Center, good surveys have several characteristics: Question Clarity: Questions are simple, use clear language to avoid misunderstandings, & avoid combining multiple issues (are not double-barreled questions). Use the Right Question Type: Use open-ended questions for detailed responses & closed-ended ones for easier analysis. Match the question type to your research question. Avoid Bias: Craft neutral questions that don’t lead respondents toward specific answers. Avoid emotionally charged or suggestive wording. Question Order: Arrange questions logically to avoid influencing responses to later questions. Logical flow ensures better data quality. Have Been Pretested: Use pilot tests to identify issues with question wording, structure, or respondent interpretation before finalizing your survey. Use Consistent Items Over Time: Longitudinal studies should use consistent wording & structure across all survey iterations to track changes reliably. Questionnaire Length: Concise surveys reduce respondent fatigue & elicit high-quality responses. Cultural Sensitivity: Be mindful of cultural differences. Avoid idioms or terms that may not translate well across groups. Avoid Jargon: Avoid technical terms or acronyms unless they are clearly defined. Response Options: Provide balanced & clear answer choices for closed-ended questions, including “Other” or “Don’t know” when needed. So why post a primer on surveys & items? BC badly designed surveys not only get your paper to reject, but they also waste your participants' time - neither of which is a good outcome. So take time your time, get the items right, get the survey right, and you be far more likely to find a home for your work. #researchdesign

It’s arguably never been easier to run surveys, but that doesn’t mean we’re getting better insights from them. Often this stems back to two main issues 1) lack of clear objectives and hypotheses upfront, and 2) poorly written questions. ✅ Start with a list of hypotheses you want to investigate. Think of these as statements you believe to be true and want to confirm. This should not be a list of stats you’d like to generate from the survey. Instead, what are the ideal “headlines” you’d love to report on? For example, rather than seeking a stat like “60% of Gen Z discover new products on social media compared to 20% of Gen X”, think of the overall insight you want to gain, like “the shopping experience has changed and brands need to adapt their marketing strategy: a majority of Gen Z now use social media to discover new products, while a minority of Gen X shoppers discover products this way”. ⁉️ Now, what questions help you get to these insights? One of the most frequent question pitfalls I see is asking two questions in one question. Don’t ask a question with “or” in the middle. Each question should have a single point to it. E.g. “Which of the below channels do you use for product discovery?” If you want to also learn about channels that they are more likely to convert from, ask it in a different question. Define all terms you are using. What do you mean by “discovery”? Are all the channels you list easily understood? Questions should be as simple and specific as possible: as few words as possible, no fancy vocab. Then test your questions with a few users. Do they all understand and interpret the questions in the same way? If people tell you multiple meanings, be more simple, and more specific. To put these points together, add a sentence to your survey draft above each question (or in some cases, a set of questions) with the headline you ideally want to share.    💡 To summarize, before running a survey, what insights do you want to take from it? And do you have the right question to get you there?

You've probably seen hundreds of surveys in your day... but do you actually know 𝘩𝘰𝘸 to set up questions in a way that is useful to get information about your target audience? Earlier this week, I attended a fantastic workshop called "𝘜𝘯𝘭𝘰𝘤𝘬𝘪𝘯𝘨 𝘊𝘰𝘯𝘴𝘶𝘮𝘦𝘳 𝘐𝘯𝘴𝘪𝘨𝘩𝘵𝘴" led by Hailey Mortimore of Sage Outcomes, a boutique marketing research firm. Here are 4 unique survey tips to get good quality data in DIY surveys: 1. 𝗞𝗲𝗲𝗽 𝗶𝘁 𝟰 𝗺𝗶𝗻𝘂𝘁𝗲𝘀 𝗼𝗿 𝗹𝗲𝘀𝘀. Instead of focusing on # of questions... THINK: 𝘩𝘰𝘸 𝘭𝘰𝘯𝘨 𝘥𝘰𝘦𝘴 𝘪𝘵 𝘵𝘢𝘬𝘦 𝘴𝘰𝘮𝘦𝘰𝘯𝘦 𝘵𝘰 𝘧𝘪𝘭𝘭 𝘰𝘶𝘵? People's attention spans are like a goldfish, so ask the absolute more important questions. 2. 𝗣𝘂𝘁 𝘆𝗼𝘂𝗿 𝗱𝗲𝗺𝗼𝗴𝗿𝗮𝗽𝗵𝗶𝗰 𝗾𝘂𝗲𝘀𝘁𝗶𝗼𝗻𝘀 𝗮𝘁 𝘁𝗵𝗲 𝗲𝗻𝗱.     I'm talking gender, age, race, income, etc. People subconsciously don't like to feel like they are answering on behalf of their identities and communities. 3. 𝗨𝘀𝗲 𝗹𝗼𝘁𝘀 𝗼𝗳 𝗺𝘂𝗹𝘁𝗶𝗽𝗹𝗲 𝗰𝗵𝗼𝗶𝗰𝗲 𝗼𝗿 𝘀𝗰𝗮𝗹𝗲 𝗾𝘂𝘀𝘁𝗶𝗼𝗻𝘀, 𝗻𝗼𝘁 𝘁𝗼𝗼 𝗺𝗮𝗻𝘆 𝗼𝗽𝗲𝗻 𝗿𝗲𝘀𝗽𝗼𝗻𝘀𝗲.     If you drop in a bunch of fill-in questions, especially back to back, the mental load to fill out your survey is high and you risk them calling it quits. Plus, it's harder to analyze. ❌ "𝘛𝘦𝘭𝘭 𝘮𝘦 𝘢𝘣𝘰𝘶𝘵 𝘸𝘩𝘺 𝘺𝘰𝘶 𝘨𝘰 𝘵𝘰 𝘵𝘩𝘦 𝘴𝘵𝘰𝘳𝘦."  vs. ✅ "𝘞𝘩𝘢𝘵 𝘪𝘴 𝘺𝘰𝘶𝘳 𝘱𝘳𝘪𝘮𝘢𝘳𝘺 𝘳𝘦𝘢𝘴𝘰𝘯 𝘧𝘰𝘳 𝘨𝘰𝘪𝘯𝘨 𝘵𝘰 𝘵𝘩𝘦 𝘨𝘳𝘰𝘤𝘦𝘳𝘺 𝘴𝘵𝘰𝘳𝘦?" - 𝘍𝘰𝘳 𝘸𝘦𝘦𝘬𝘭𝘺 𝘩𝘰𝘶𝘴𝘦𝘩𝘰𝘭𝘥 𝘨𝘳𝘰𝘤𝘦𝘳𝘪𝘦𝘴 - 𝘛𝘰 𝘣𝘶𝘺 𝘴𝘱𝘦𝘤𝘪𝘢𝘭𝘪𝘵𝘺 𝘪𝘵𝘦𝘮𝘴 𝘧𝘰𝘳 𝘢 𝘳𝘦𝘤𝘪𝘱𝘦 - 𝘍𝘰𝘳 𝘱𝘭𝘦𝘢𝘴𝘶𝘳𝘦 You can use them occassionally, but intentionally. 𝟰. 𝗥𝗲𝗿𝗲𝗮𝗱 𝘆𝗼𝘂𝗿 𝗾𝘂𝗲𝘀𝘁𝗶𝗼𝗻𝘀 𝘁𝗼 𝗺𝗮𝗸𝗲 𝘀𝘂𝗿 𝗲𝘁𝗵𝗲𝘆 𝗮𝗿𝗲 𝗻𝗼𝘁 "𝗹𝗲𝗮𝗱𝗶𝗻𝗴 𝗾𝘂𝗲𝘀𝘁𝗶𝗼𝗻𝘀." Would someone feel pressured to answer a certain way because of how you wrote the question? ❌ "𝘗𝘦𝘰𝘱𝘭𝘦 𝘰𝘧𝘵𝘦𝘯 𝘨𝘦𝘵 𝘴𝘤𝘢𝘳𝘦𝘥 𝘢𝘵 𝘵𝘩𝘦 𝘥𝘦𝘯𝘵𝘪𝘴𝘵. 𝘞𝘩𝘢𝘵 𝘢𝘣𝘰𝘶𝘵 𝘵𝘩𝘦 𝘥𝘦𝘯𝘵𝘪𝘴𝘵 𝘢𝘳𝘦 𝘺𝘰𝘶 𝘢𝘧𝘳𝘢𝘪𝘥 𝘰𝘧?" ✅ Surveys are more powerful that we realize. You can collect lotssss of insight in a much shorter time than other research methods. And following those tips will make your insights even more useful. Shoutout to Prodigy & Co for hosting the session, Tulsa Tech and Gradient: Tulsa's Hub for Innovation