Survey Reversals Explained (with examples)

In this post, we’ll explain survey reversals, and how they negatively impact both market research companies and respondents. We’ll also cover the key reasons triggering reversals, issues with false positives and some tips and tricks for how to avoid losing your rewards.

Survey reversals can happen for a variety of reasons, and it is important for both respondents and market research companies to understand the metrics that can lead to reversals, as well as the consequences of these rejections.

4 Key reasons for survey reversals

A market research company issues reversals after an analysis of the survey answers. This assessment of data quality is usually done through four lenses:

• Are they real?
• Are they who they say they are?
• Are they unique?
• Are they attentive?

In this article, we will discuss four scenarios where survey reversals can be justified.

Are they real?

One of the first questions that market research companies need to answer is whether the respondents are real people or bots. Bots are automated programs that can fill out surveys with random or predetermined answers, often for fraudulent purposes. Bots can skew the survey results and waste the resources of the researchers and clients.

One way to detect bots is to use a captcha or a honeypot question at the beginning or end of the survey. A captcha is a test that requires human intelligence to solve, such as identifying images or typing letters. A honeypot question is a hidden question that only bots can see and answer, such as “Please leave this field blank”. If a respondent fails the captcha or answers the honeypot question, they are likely to be a bot.

Are they who they say they are?

Another question that market research companies need to answer is whether the respondents are who they claim to be. Some respondents may provide false or inaccurate demographic information, such as age, gender, income, education, etc., either to qualify for a survey that they are not eligible for or to receive incentives that they are not entitled to. This can also distort the survey results and undermine the representativeness of the sample.

One way to prevent this is to use survey reversals within the demographic section of the survey. Survey reversals can help detect respondents who are lying or guessing about their personal information. For example, if a respondent reports their age as 25 in one question and 35 in another question with reversed wording, they are likely to be dishonest or careless.

Are they unique?

A third question that market research companies need to answer is whether the respondents are unique or duplicates. Some respondents may complete the same survey multiple times using different devices, email addresses, IP addresses, etc., either to increase their chances of winning a prize or to manipulate the survey outcomes. This can inflate the sample size and reduce the diversity and validity of the survey data.

One way to avoid this is to use cookies or tokens to track and limit the number of times a respondent can access and complete a survey. Cookies are small pieces of data that are stored on the respondent’s browser and can identify whether they have already participated in a survey. Tokens are unique codes that are assigned to each respondent and can prevent them from taking a survey more than once.

Another way to avoid this is to use survey reversals across different surveys that target the same population or segment. Survey reversals can help identify respondents who are taking multiple surveys with similar or overlapping questions. For example, if a respondent gives different answers to the same question with reversed wording in two different surveys, they are likely to be a duplicate or a fraudulent respondent.

Are they attentive?

A fourth question that market research companies need to answer is whether the respondents are attentive or distracted. Some respondents may not pay attention to the survey questions or instructions, either because they are bored, tired, multitasking, or rushing. They may skip questions, choose the same or random answers, or provide nonsensical or irrelevant responses. This can affect the reliability and completeness of the survey data and introduce noise and errors.

One way to measure this is to use attention checks or trap questions throughout the survey. Attention checks are questions that test whether the respondent is reading and understanding the survey questions, such as “Please select the third option from the list below”. Trap questions are questions that have obvious or absurd answers, such as “How many hours are there in a day?”. If a respondent fails an attention check or a trap question, they are likely to be an inattentive or a careless respondent.

Another way to measure this is to use survey reversals within the main body of the survey. Survey reversals can help assess whether the respondent is consistent and coherent in their answers and whether they are influenced by response biases or tendencies. For example, if a respondent agrees with both “I like chocolate ice cream” and “I dislike chocolate ice cream”, they are likely to be an inattentive or an acquiescent respondent.

False survey reversals causing respondent issues

The 4 reasons tactics described above to assess survey answers are often deployed by automated platforms based on machine learning and AI. These systems are not perfect and may sometimes flag legitimate respondents as reversals, resulting in false positives.

False positives can have negative impacts on both survey sites and respondents. For survey sites, false positives can reduce the quality and quantity of data collected, damage their reputation and trustworthiness, and increase operational costs and legal risks. For respondents, this can cause frustration, inconvenience, loss of rewards or incentives, and privacy concerns.

Some of the main issues that contribute to false positives from AI-based fraud detection systems used by survey sites are:

• Lack of data: AI-based fraud detection systems rely on large and diverse datasets to train their models and improve their accuracy. However, some survey sites may not have enough data or may have data that is outdated, incomplete, or biased. This can limit the performance of the AI models and increase the chances of misclassifying respondents.
• Lack of explainability: AI-based fraud detection systems often use complex and opaque algorithms that are difficult to understand and interpret. This can make it hard for survey sites to justify their decisions and provide feedback to respondents. It can also make it challenging for respondents to appeal or dispute false positives and restore their accounts or rewards.
• Lack of standardization: AI-based fraud detection systems may vary widely in their design, implementation, and evaluation across different survey sites. There is no common framework or benchmark to compare and validate the effectiveness and fairness of these systems. This can create inconsistency and confusion for respondents who participate in multiple survey sites and face different criteria and rules for fraud detection.
• Complexity: Issues such as inattentiveness can manifest in different ways, making it challenging to create a comprehensive set of rules or features for an AI system to identify accurately. Inattentiveness can range from occasional lapses in attention to consistent careless responses, making it a nuanced behaviour that may require more sophisticated techniques to detect effectively.
• Client pressure: Survey companies might feel pressure from clients to “turn up the dial” on their AI-based fraud detection system to make it more strict. Stricter criteria to detect inattentiveness may result in a higher false-positive rate, which can frustrate genuine survey takers. Achieving the right balance is a challenging task for AI models.
• Lack of helpdesk support: Survey companies are technology-driven, with a focus on automating the majority of their tasks. Their helpdesk teams are usually not scaled or equipped to deal with complex investigations into why their algorithm flagged a particular respondent. Complaints are often answered with standard templated emails, which adds to the feeling of respondents being treated unfairly.

How can you avoid survey reversals and account suspensions?

When a survey reversal or account suspension occurs, you may lose the reward that you earned for completing the survey, or even worse, lose access to your account and all the rewards that you accumulated. This can be very frustrating and discouraging, especially if you invested a lot of time and energy into taking surveys.

Here are some tips for how to avoid survey reversals:

• Always read and follow the instructions and terms and conditions of the survey site and the individual surveys.
• Always answer the survey questions honestly and carefully. Do not rush or skip questions.
• Do not provide random or false answers.
• Always check your profile and update it regularly. Make sure that your information is accurate and consistent with your answers to the survey questions.
• Do not use any software or tools that may interfere with your IP address or location.
• Do not use multiple devices or browsers to take surveys.
• Do not create multiple accounts or use fake identities.
• Do not share your account information with anyone else.
• Do not participate in surveys that you are not interested in or qualified for.
• Do not try to trick the system or cheat the survey site or its clients.

By following these tips, you can increase your chances of having a positive and rewarding experience with online survey sites. You can also avoid the hassle and disappointment of survey reversals and account suspensions. Happy surveying!