Should you create an online survey?

From time to time, I’m asked by clients if they should create a survey to make decisions about their business or organization and if so, how should they conduct the survey? My response is always the same. If you want to have a survey that is representative and accurate, sending out a Google poll will never give you good results.

If you want to conduct a survey with results that actually mean something, you have to hire a professional polling company to create, administer, and interpret the results.

(And yes, this very expensive.)

Let me break down why this is the case and why you really shouldn’t rely on Google surveys or Doodle polls to make decisions for your business or organization.

It’s very easy to create a Google survey and send it out in an email. And it’s free! So why shouldn’t you just do that instead of hiring a polling company?

Self-made surveys and polls do not collect accurate data

This is the number one most important takeaway from this blog post that I want everyone to understand. A Google survey will not get you reliable, accurate results, and the results you do get will not come from a representative sample of the demographic you need.

Sampling is the process of using a smaller group of people to create an interpretation of a larger group. This is super important in your polling. You may send your survey to 100 people, get back 100 results, but if those 100 people are all people that you already know agree with you, then the survey is useless. Surveys of your friends and family do not accurately represent the broader range of demographic you are probably looking for.

“All good polling relies on statistical adjustment called ‘weighting’ to make sure that samples align with the broader population on key characteristics.“

Pew Research

The question I get on this topic is generally along the lines of “I’ll send the survey to everyone in town or in my neighborhood or send it to all my customers and that will be an accurate sample.” And no, it will not. Because unless you can guarantee that every person you sent the poll to completes the survey, that data will be skewed. People who like you are more likely to complete your poll. People who are aggressively pro or con the issue at hand are much more likely to complete that survey. People who don’t complete surveys: folks who are busy with their day-to-day living, lower income people, those with less access to technology. You will miss many key demographics if your sole survey method is sending out a Google link in an email.

Imagine if the US Census was conducted with an online link that went out in an email. It just doesn’t work if accurate results are important to you.

Inherent bias in your questions

One thing I mentioned above was not just hiring a company to conduct the survey, but they need to create the survey as well, and that is because if you write your own survey, your questions will be inherently biased to get you the answer you desire.

The way the questions are worded can play a huge role in the results you get, and professional survey companies know how to prepare surveys to remove your inherent bias. (And yes, you have one. We all do.)

Have you ever taken a survey and been annoyed because they asked you the same question multiple times? There’s a reason for that. They ask the same questions several times to see if you answer the same way. Sometimes they change the wording to help gauge the results. You may have thought the creator of the poll forgot they asked that question, but there’s actual science behind this process.

There are so many ways in which your questions can be biased to get the answers you desire. Some are more obvious than others. One of those is by having a limited pool of choices for a question. Being unclear in your phrasing. Playing on people’s emotions with your questions. And just the choice of words you use.

If you want an accurate survey, you cannot write the questions yourself. (And I’d venture to say you shouldn’t have any kind of personal relationship with the person writing the questions.)

Intentional or not, if you write the questions, interpret the data, and share the results, you will more often than not get the answers you were looking for in your survey.

Data Interpretation

Analyzing and interpreting the data you get in a survey or poll is very important, and if you aren’t a skilled or professional poll taker, there’s a good chance you are going to get it wrong. You might be tempted to look at just the raw data rather than the sample data. For example, if 100 people took the survey, and 70 of those people say X and 30 say Y. So it would be very easy for someone to say that 70% agree with X, so that is the majority and it wins.

And that’s not an inaccurate response! It’s just unbalanced. If you don’t sample your respondents based on a variety of different factors (race, socioeconomic, education, to name a few) then your results are only true superficially. When you use a sampling versus a majority, you are likely to get different results.

I love data. And I am easily influenced by data. When someone tells me “80% agree with this!” then that can be a very compelling argument. Not just for me, but for lots of people. When you blend an inaccurate interpretation of the data with marketing tactics like the bandwagon effect, you can easily use a bad survey to influence your demographic. This is especially evident in online surveys.

“Bandwagon effects are most applicable in the digital marketing arena, where the actions of audience peers are most visible. The crowd appeal or virality of your promotion correlates directly to the exposure you get on Facebook, in Twitter, and in Google search.”

Cheryl Conner, Forbes

I’m not saying you can’t create a Twitter poll or Facebook poll to get instant results or create some fun interactions for your audience. There is a time and a place for that kind of survey. But if your company or organization is seeking to gather useful information with which to make critical decisions, then it is absolutely essential to hire a professional company, one with people who are skilled and educated in the best practices for survey creation, administration, and interpretation. Otherwise, the results you think you got may lead you to make decisions that could not be as beneficial as you hoped.