Who We Are

We are your guide to timely, fact-based insights and actionable solutions. We connect the dots between the research and your business objectives.

Who We Work With

Some of the most iconic brands in the world have trusted us since 1983 and we are proud to have clients that continue to work with us after 30 years of successful business together.

What We Do

We deliver on our promise. Our promise to provide the highest quality marketing research insights with a reliable team of industry experts that you can count on every step of the way.

How We Do It

We care about the quality of our deliverables. We care about the impact that our insights and solutions have on your organization. We care about our relationships with our clients and with each other.

Let’s Connect

We know that trust must be built between us over time. So, let’s get started.

Privacy Policy
Copyright © 2022 KS&R, Inc.
All Rights Reserved

Conducting an online survey? How to make sure you don’t get bad data

With nearly 25 years of global market research experience, Chris has a broad consulting background guiding clients in technology, risk management, retail and insurance with latent customer needs identification, new product / service development, fraud and financial crime assessment and optimal customer experiences. He is skilled at qualitative and quantitative research methods and has extensive experience as a professional moderator and ethnographer applying creative approaches to unlock functional and emotional drivers of customer behaviors and decision processes. Chris has authored several White Papers, including those highlighting the ongoing fraud risks to retailers, e-commerce merchants, financial services, and lending firms. Outside of the office, Chris continues to tackle the game of golf and all its frustrating joys. Being a lifelong Red Sox fan has taught him how to be patient with success.

Connect with Chris

Conducting an online survey? How to make sure you don’t get bad data Hero Image

Skipped questions. Contradictory responses. Irrelevant answers.

These kinds of responses are bad for business. They skew survey results, so companies can’t get the answers they need. And, unfortunately, survey responses like these are becoming increasingly common as online surveys gain in popularity.

This doesn’t mean you should stop conducting online surveys, but it does mean that market researchers need to take extra steps to ensure data integrity.

When data is bad, either the survey results need to be thrown out, or companies mistakenly use the results and make misinformed business decisions. Either way, getting bad data is a waste of time, energy and money.

Online surveys: Great for market research, but added challenges

The ability to conduct online surveys has been a major advancement in market research. Quickly reaching thousands of respondents, data is easily collected and tabulated. Surveys can be tailored automatically based on real-time responses. Cost and time savings are immediate.

However, market researchers do miss out on some of the advantages of the “old days,” particularly the ability to directly observe and react to the respondent. For example, during a telephone interview, if a respondent was having trouble understanding a question, the interviewer could immediately step in to clarify. The researchers could also monitor calls and identify any issues with the survey questions.

With online surveys, it’s much more difficult to get that real-time interaction. The respondent opens up the survey, alone, on a computer monitor, while he/she is at home or at work. Perhaps there is an acronym the survey taker doesn’t know, or a product description isn’t clear enough for the respondent to provide meaningful feedback. He/she attempts to answer nonetheless, which often leads to nonresponsive answers. In addition, it’s more difficult to screen respondents when they’re recruited online. Some examples of survey takers who can provide bad data include:

  • Fraudulent respondents – misrepresent themselves and provide inaccurate information, especially demographic information
  • Inattentive respondents – not attentive when taking the survey; do not provide thoughtful answers to survey questions 
  • Professional survey takers – may belong to more than one panel or take numerous surveys each month to collect incentives

Ensuring data integrity

Given these challenges, how do market researchers make sure the data is accurate? Here are three methods that KS&R uses to ensure that our clients get accurate data from their online surveys. 

1. Conduct a Pre-Test 

Pre-tests are invaluable. While they require extra time and money, the costs of not pre-testing can be much greater. To pre-test, the survey is given to a small group of respondents (in person or on the phone) while researchers observe and follow-up to uncover any issues. Did they encounter any problems? What was unclear, ambiguous or confusing? How can the survey be improved? Using that feedback, we then re-write survey directions and questions to ensure that it’s intuitive and relevant to respondents.

2. Set-up Traps 

By asking innovative questions, we can catch the “bad” survey takers (those fraudulent, inattentive and/or professional respondents). Here are a few examples:

  • Ask demographic questions twice. Collecting demographic information at the beginning and again at the end of the survey gives fraudulent respondents an opportunity to trip up.
  • Add “red herring” answer choices. Including answer choices that are not legitimate options – i.e., non-existent products, retailers, medical conditions – is very effective in identifying “bad” survey takers. For example:

Which of the following IT vendors provides your company with Intel-based servers? 

  1. HP
  2. Technology Frontier
  3. IBM 
  4. Dell

In this example, Technology Frontier does not exist. Respondents who select it should be considered for removal.

  • Tell the respondent to pick a certain answer. To help detect inattentive respondents, a simple question included more than halfway through the survey can help flag them. For example:

Please select the number three in the list below. 

  1. 2
  2. 3

Respondents who do not select 3 should be considered for removal.

3. Establish Data Collection Red Flags

Because online surveys can be programmed and quickly analyzed, it’s a great opportunity to identify “red flag” data. Some examples:

  • Quick takers: Using a time stamp, it is easy to see if respondents sped through the survey. Survey takers who complete a survey in less than 1/3 of the average time typically provide unusable data.
  • Duplicate entries: Collecting an IP address from respondents identifies respondents who took the survey twice.
  • Straight-lining: When a respondent selects the same response (i.e., all A’s, all “very satisfied”) for many questions right in a row, it might indicate a fraudulent survey taker. While the responses can be valid, a more detailed examination can identify respondents who are either fraudulent or simply rushing through the survey.
  • Inconsistent or contradictory responses: When respondents consistently fall for the “trap” questions listed above; provide open-ended responses that are not meaningful; and have outlying answers, it’s important to analyze their completed surveys for possible data contamination.

Better data is worth the effort

Setting up pre-tests, adding trap questions, and screening and analyzing for red flags are just a few of the methods we use to identify data that could muddy your survey results. A clean data set is a critical starting point for the information and insights you need to move your company forward.

Do you have questions about preserving the data integrity for your online surveys? Contact KS&R to learn more.