What is Survey Bias, and How Do You Avoid It?
Market research aims to accurately take a pulse of a population or sector through surveys. However, survey bias happens when a researcher selects or encourages one outcome or answer over others, sometimes unknowingly or inadvertently. Many quality checks occur in market research to protect from fraud and bad data, and being aware of survey bias is another way to protect the integrity of research performed.
Survey bias is a deviation of feedback based on certain influences by the surveyor and respondent. Below, we break down the most common types of sampling and non-sampling survey biases and how you can identify and avoid them.
Types of Survey Bias
How to Avoid Survey Bias
How Does PureSpectrum Prevent Survey Bias?
Sampling Bias
Also known as selection bias, sampling bias arises when researchers fail to connect their survey with the right type of people. Not having the proper respondents in a survey can severely impact the data quality. Imagine running a study about baby gear, not specifically targeting new and expecting parents. The data you would get back would probably not represent your target market. Sampling bias can also arise in studies when researchers cannot capture an accurate representation of the general population.
Additionally, research participants should be chosen at random to avoid bias. The survey’s validity can be seriously impacted if respondents aren’t randomly selected. Non-random selection rarely reflects the more significant population, and people may become aware of why or who chose them.
A way to avoid sampling bias is to multi-source sample from a wide variety of panels, not just one or two. This blending can easily be achieved by utilizing a sample marketplace like PureSpectrum. In our article, The Importance of Sample Source Transparency, you can read more about multi-sourcing and sample aggregation.
Non-Sampling Bias
Non-sampling errors occur when there are problems with the way the survey is designed or carried out. Below we cover the most common types of non-sampling bias and how you can avoid them.
Wording Bias
Wording bias, also referred to as question-wording bias, develops when the words or phrasing in a questionnaire influence the responses. This bias is specific to how questions are written or what is missing from a study question. This can occur when performing research for a particular brand and you use phrases that show favor to your brand that may sway how a participant responds, thus creating survey bias.
Examples of Wording Bias
Leading Respondents
If you are studying vitamin brands and you ask, “How important are these essential minerals to you?” you are leading the respondent to believe that the minerals are indeed essential. The rewrite for this question could be “Which of the following minerals are important to you?”
Leading Respondents Fix: Eliminate adjectives or descriptions that elicit influence. Keep your questions straightforward and factual.
Missing Responses
Using the same example above, “Which of the following minerals are important to you?” A survey may provide the following multiple-choice selections.
- Calcium
- Magnesium
- Chromium
But without the inclusion of additional selections like
4. None of the above
The question will elicit a negative outcome because a person must choose an option that might not be accurate to continue the survey.
Missing Responses Fix: Ensure you provide enough responses for your respondent to collect accurate insights. Include answers like “all of the above,” “none of the above,” and “prefer not to answer” in your multiple-choice questions.
Double Bar
Double bar questions are biased because they ask respondents for two pieces of feedback, not just one. A biased question gauging employee happiness might ask, “Do you like your CEO and his policies?” Here the interviewee may like the CEO but not his policies, and because of this double bar, the response will be biased. When rewriting this question, you should break apart the question into two. “Do you like your CEO?” and “Do you approve of your CEO’s policies?”
Double Bar Fix: Only solicit one piece of feedback per question.
Loaded Question
A loaded question written with a controversial or unjustified assumption. Aside from potentially being incorrect, such questions may be used as a rhetorical tool: the question attempts to limit direct replies to those that serve the questioner’s agenda. An example is, “Have you stopped drinking too much?” By posing the question in this loaded manner, the respondent might immediately assume their alcohol consumption has been categorized as negative, which may bias their responses.
Loaded Question Fix: Make sure your questions are free from emotion or descriptive terms.
Response Bias
Response bias develops as a reaction to the way a question is asked, phrased, or presented to a respondent. There are several reasons why a survey participant might provide inaccurate responses, from a desire to comply with social desirability and answer in a way the respondent thinks they ‘should’ to the nature of the survey and the questions asked.
Response Bias Example
Response bias can happen when a question is presented incorrectly. For example, a question might be posed as multiple choice but should be written as a scale rating. It could also arise when a question is presented as open-ended but is just a simple yes or no answer.
Response bias can be avoided by:
- Matching responses appropriately with the question type you have selected.
- Asking one question at a time so that a respondent is only responsible for providing one answer.
- Providing enough options for a respondent to give a truthful response. It is also helpful to provide the opportunity of “I prefer not to answer” when asking personal questions.
- Writing your survey in clear and concise language that is easy to read and comprehend.
Interviewer Bias
Interviewer bias relates to how researchers ask questions or respond to provided answers. This type of bias occurs most often during in-person quantitative interviews. In addition to body language, facial expressions (and other non-linguistic aspects of communication), interviewer bias may arise from the respondents’ perception of the interviewer’s demographic characteristics, such as age, gender, ethnicity, social class, professional background, and more. Whenever seemingly related to the interview topic, those characteristics may influence the way participants respond to interview questions.
Three Ways to Avoid Interviewer Bias:
- Remain neutral in demeanor and vocal cadence when asking questions.
- There should be no judgment or emotion shown when responses are given.
- Ideally, the interviewer should not identify whom they work for and what brand (if any) is behind a survey when making introductions.
How Do You Avoid Survey Bias?
It is hard to entirely avoid survey bias because everyone has their own experiences and prejudices. As a survey writer, though, it is always the goal to prevent these biases as much as possible. Here is a short checklist to keep in mind when designing your study.
- Phrase your questions neutrally and in the proper format.
- Make sure responses match the question asked and allow a person to check “prefer not to answer” if they wish.
- Sample anonymously, ideally from multiple sources or panels.
- Keep your survey as short and straightforward as possible to avoid respondent fatigue.
- Remove your brand so that interviewees are not aware of who is collecting their responses.
How Does PureSpectrum Prevent Survey Bias?
The PureSpectrum Marketplace sources respondents from multiple panels and sources. Not only does this mean surveys will match their quotas quickly it also ensures a greater chance that a survey will be matched to its target market. The multisource technology found on the PureSpectrum platform provides inclusion and representation and aims to eliminate under-coverage that is inherent to single sourcing.
Sourcing survey respondents from an online marketplace can help add the appropriate anonymity to prevent voluntary response bias. By collecting sample from multiple panels, people are unaware of who is fielding a survey and what type of respondents they are looking for, avoiding self-selection.
Our easy-to-use platform allows stratified sampling by setting quotas, CPI, and click balance. Click balancing helps researchers see how many clicks it will take to achieve their given incidence rate. Stratified sampling assures that you are casting a wide net and not sampling with bias.
The PureSpectrum platform supports any research objective you or your business may have. Our expert support team is available to help with your project or answer any questions you may have. We offer Platform Access, Access with Support, and Full-Service solutions.
Ready to get started? Reach out to our team today:
About PureSpectrum
PureSpectrum offers a complete end-to-end market research and insights platform, helping insights professionals make decisions more efficiently, and faster than ever before. Awarded MR Supplier of the Year at the 2021 Marketing Research and Insight Excellence Awards, PureSpectrum is recognized for industry-leading data quality. PureSpectrum developed the respondent-level scoring system, PureScore™, and believes their continued success stems from their talent density and dedication to simplicity and quality.
In the few years since its inception, PureSpectrum has been named one of the Fastest Growing Companies in North America on Deloitte’s Fast 500 since 2020, and ranked for three years in a row on the GRIT Top 50 Most Innovative List and the Inc. 5000 lists.