Examples of Biased Survey Questions

1. The Leading Question

One of the biggest mistakes survey creators make is creating a question that leads respondents to give the “correct” answer. Leading questions negate your survey results, so you want to stay away from them at all costs.

You don’t want to word a survey question in a way that will sway your reader to a particular side. To do this, you must use neutral wording. Here are some examples of leading and biased questions:

Leading question: How dumb is (insert politician) when it comes to foreign policy? This immediately brings a negative connotation to the question. Instead, you might ask the question: Please describe your politician’s position on foreign policy.

Leading question: Should concerned dog owners vaccinate their pets? By using the word concerned, you put pet owners who don’t vaccinate their pets on the defensive, thus creating bias. Instead, ask it this way: Do you think dogs should be required to be vaccinated?

2. The Loaded Question

With the loaded question, you basically force people into answering the question in a particular way. You keep them from explaining their own opinions. The loaded question has the potential to lead to survey drop-out and unclear results. Here is an example:

Loaded question: Where do you like to party? Well, what if the respondents don’t like to party? What if they are homebodies? Instead, you could ask it like this: What do you like to do on weekend evenings?

Avoid loaded questions so you get the most truthful survey answers.

3. The Double-Barreled Question

Another very common survey mistake, the double-barreled question, forces your respondents to answer two questions at once.

You’ll easily destroy your survey results with the double-barreled question. You want each one of your survey questions to only answer one thing. One subject per question is the rule for accurate, measurable surveys. Here is an example:

Double-barreled question: How happy or unhappy are you with the rate of current school board funding and the common-core curriculum? Wow! This is asking a lot of your respondents. Some might answer both questions, but many others will concentrate on the one that means the most to them. Instead, you could ask it like this: How happy or unhappy are you with the rate of current school board funding? And, next question: What do you think of the common core?

Always break questions into singles so your survey is succinct and measurable.

4. The Absolute Question

Yes or no answers can keep respondents from leaving unbiased feedback. (tweet this) It creates a bias because you aren’t getting the whole story with this type of question.

The absolute question usually only has the option of a yes or no answer. It also commonly includes words such as all, always, ever, and every.

Consider this question: Do you always shower before bed? The obvious answer for most people is no. You’ve basically backed respondents into a corner.

But, what if you sell shower gel, and you want to know specific showering habits for marketing purposes?

This is where you want to offer respondents some options. You might ask the question this way: How many nights a week do you shower before bed?

You’ve turned an inflexible, biased question into one that provides you with valuable information. Your answer options might be: every day, 5-6 days, 3-4 days, 1-2 days, usually shower in the morning.

5. The Unclear Question

You want clear, concise answers, right? Then you need to pose clear, concise questions that avoid terms your respondents might not know.

Tech jargon and acronyms create bias because only some of the people in your audience know what you are talking about. It’s important to make it as easy as possible for someone to answer your question.

For example, you want to know how many of your survey respondents own a smartphone, yet you ask them if they have an iPhone. Just because you carry an iPhone doesn’t mean everyone else does. You might ask: Do you have a smartphone (i.e., iPhone, Android, Windows, etc.).

On the flip side, if you are surveying aeronautical engineers, feel free to ask technical questions.

The bottom line? Know your audience and avoid any question that makes people uncomfortable because you’ve shown bias or asked them something they don’t know.

6. The Multiple Answer Question

Consider your multiple choice questions. When posing them, you want to make sure that there is only one answer.

For example, what if in the question above about showering, you offered these choices: every day, 5-6 days, 4-5 days, 3-4 days and 1-3 days. By using the same numbers over again, you’ve made it quite difficult to get an accurate answer.

7. Prefer Not to Answer

It’s a good idea to always include “prefer not to answer” in your answer choices if at all possible. Many people will drop-out of a survey if they are uncomfortable with a particular question.

8. Include All Possible Answers

Not including all possible answers also creates bias. If you are unsure of all the options, you can always add “other” as a choice.

9. Use Accurate Scales

When asking people to rate your question, you want to offer options ranging from bad to excellent to avoid bias.

For example, you ask people about their experience with your customer service team. If you leave off “poor” as an option, you’ve biased the survey. A great example of just the opposite is the NPS survey question, which has a standardized question with a rating of 1-10 no matter where or when it is served to visitors.

10. Survey Structure

The way you structure your questions from one to the other can also bias respondents. Study and test your survey to root out poor structure. For example, ask your more personal or in-depth questions at the end to avoid survey dropout.