Survey Research Analysis Method
(What, Where, How, Why it’s used, advantages & disadvantages).
A field of applied statistics of human research surveys, survey methodology studies the sampling of individual units from a population and the associated survey data collection techniques, such as questionnaire construction and methods for improving the number and accuracy of responses to surveys[1].
Survey research is one of the most important areas of measurement in applied social research. The broad area of survey research encompasses any measurement procedures that involve asking questions of respondents. A “survey” can be anything forms a short paper-and-pencil feedback form to an intensive one-on-one in-depth interview.
We’ll begin by looking at the different types of surveys that are possible. These are roughly divided into two broad areas: Questionnaires and Interviews. Next, we’ll look at how you select the survey method that is best for your situation. Once you’ve selected the survey method, you have to construct the survey itself. Here, we will be address a number of issues including: the different types of questions; decisions about question content; decisions about question wording; decisions about response format; and, question placement and sequence in your instrument. We turn next to some of the special issues involved in administering a personal interview. Finally, we’ll consider some of the advantages and disadvantages of survey methods[2].
Types of Surveys
Surveys can be divided into two broad categories: the questionnaire and the interview. Questionnaires are usually paper-and-pencil instruments that the respondent completes. Interviews are completed by the interviewer based on the respondent says. Sometimes, it’s hard to tell the difference between a questionnaire and an interview. For instance, some people think that questionnaires always ask short closed-ended questions while interviews always ask broad open-ended ones. But you will see questionnaires with open-ended questions (although they do tend to be shorter than in interviews) and there will often be a series of closed-ended questions asked in an interview.
Survey research has changed dramatically in the last ten years. We have automated telephone surveys that use random dialing methods. There are computerized kiosks in public places that allow people to ask for input. A whole new variation of group interview has evolved as focus group methodology. Increasingly, survey research is tightly integrated with the delivery of service. Your hotel room has a survey on the desk. Your waiter presents a short customer satisfaction survey with your check. You get a call for an interview several days after your last call to a computer company for technical assistance. You’re asked to complete a short survey when you visit a web site. Here, I’ll describe the major types of questionnaires and interviews, keeping in mind that technology is leading to rapid evolution of methods. We’ll discuss the relative advantages and disadvantages of these different survey types in Advantages and Disadvantages of Survey Methods.
Questionnaires
When most people think of questionnaires, they think of the mail survey. All of us have, at one time or another received a questionnaire in the mail. There are many advantages to mail surveys. They are relatively inexpensive to administer. You can send the exact same instrument to a wide number of people. They allow the respondent to fill it out at their own convenience. But there are some disadvantages as well. Response rates from mail surveys are often very low. And, mail questionnaires are not the best vehicles for asking for detailed written responses.
A second type is the group administered questionnaire. A sample of respondents is brought together and asked to respond to a structured sequence of questions. Traditionally, questionnaires were administered in group settings for convenience. The researcher could give the questionnaire to those who were present and be fairly sure that there would be a high response rate. If the respondents were unclear about the meaning of a question they could ask for clarification. And, there were often organizational settings where it was relatively easy to assemble the group (in a company or business, for instance).
What’s the difference between a group administered questionnaire and a group interview or focus group? In the group administered questionnaire, each respondent is handed an instrument and asked to complete it while in the room. Each respondent completes an instrument. In the group interview or focus group, the interviewer facilitates the session. People work as a group, listening to each other’s comments and answering the questions. Someone takes notes for the entire group — people don’t complete an interview individually.
A less familiar type of questionnaire is the household drop-off survey. In this approach, a researcher goes to the respondent’s home or business and hands the respondent the instrument. In some cases, the respondent is asked to mail it back or the interview returns to pick it up. This approach attempts to blend the advantages of the mail survey and the group administered questionnaire. Like the mail survey, the respondent can work on the instrument in private, when it’s convenient. Like the group administered questionnaire, the interviewer makes personal contact with the respondent — they don’t just send an impersonal survey instrument. And, the respondent can ask questions about the study and get clarification on what is to be done. Generally, this would be expected to increase the percent of people who are willing to respond.
Interviews
Interviews are a far more personal form of research than questionnaires. In the personal interview, the interviewer works directly with the respondent. Unlike with mail surveys, the interviewer has the opportunity to probe or ask follow-up questions. And, interviews are generally easier for the respondent, especially if what is sought is opinions or impressions. Interviews can be very time consuming and they are resource intensive. The interviewer is considered a part of the measurement instrument and interviewers have to be well trained in how to respond to any contingency.
Almost everyone is familiar with the telephone interview. Telephone interviews enable a researcher to gather information rapidly. Most of the major public opinion polls that are reported were based on telephone interviews. Like personal interviews, they allow for some personal contact between the interviewer and the respondent. And, they allow the interviewer to ask follow-up questions. But they also have some major disadvantages. Many people don’t have publicly-listed telephone numbers. Some don’t have telephones. People often don’t like the intrusion of a call to their homes. And, telephone interviews have to be relatively short or people will feel imposed upon[3].
Selecting the Survey Method
Selecting the type of survey you are going to use is one of the most critical decisions in many social research contexts. You’ll see that there are very few simple rules that will make the decision for you — you have to use your judgment to balance the advantages and disadvantages of different survey types. Here, all I want to do is give you a number of questions you might ask that can help guide your decision.
Population Issues
The first set of considerations have to do with the population and its accessibility.
- Can the population be enumerated?
For some populations, you have a complete listing of the units that will be sampled. For others, such a list is difficult or impossible to compile. For instance, there are complete listings of registered voters or person with active drivers licenses. But no one keeps a complete list of homeless people. If you are doing a study that requires input from homeless persons, you are very likely going to need to go and find the respondents personally. In such contexts, you can pretty much rule out the idea of mail surveys or telephone interviews.
- Is the population literate?
Questionnaires require that your respondents can read. While this might seem initially like a reasonable assumption for many adult populations, we know from recent research that the instance of adult illiteracy is alarmingly high. And, even if your respondents can read to some degree, your questionnaire may contain difficult or technical vocabulary. Clearly, there are some populations that you would expect to be illiterate. Young children would not be good targets for questionnaires.
- Are there language issues?
We live in a multilingual world. Virtually every society has members who speak other than the predominant language. Some countries (like Canada) are officially multilingual. And, our increasingly global economy requires us to do research that spans countries and language groups. Can you produce multiple versions of your questionnaire? For mail instruments, can you know in advance the language your respondent speaks, or do you send multiple translations of your instrument? Can you be confident that important connotations in your instrument are not culturally specific? Could some of the important nuances get lost in the process of translating your questions?
- Will the population cooperate?
People who do research on immigration issues have a difficult methodological problem. They often need to speak with undocumented immigrants or people who may be able to identify others who are. Why would we expect those respondents to cooperate? Although the researcher may mean no harm, the respondents are at considerable risk legally if information they divulge should get into the hand of the authorities. The same can be said for any target group that is engaging in illegal or unpopular activities.
- What are the geographic restrictions?
Is your population of interest dispersed over too broad a geographic range for you to study feasibly with a personal interview? It may be possible for you to send a mail instrument to a nationwide sample. You may be able to conduct phone interviews with them. But it will almost certainly be less feasible to do research that requires interviewers to visit directly with respondents if they are widely dispersed.
Sampling Issues
The sample is the actual group you will have to contact in some way. There are several important sampling issues you need to consider when doing survey research.
- What data is available?
What information do you have about your sample? Do you know their current addresses? Their current phone numbers? Are your contact lists up to date?
- Can respondents be found?
Can your respondents be located? Some people are very busy. Some travel a lot. Some work the night shift. Even if you have an accurate phone or address, you may not be able to locate or make contact with your sample.
- Who is the respondent?
Who is the respondent in your study? Let’s say you draw a sample of households in a small city. A household is not a respondent. Do you want to interview a specific individual? Do you want to talk only to the “head of household” (and how is that person defined)? Are you willing to talk to any member of the household? Do you state that you will speak to the first adult member of the household who opens the door? What if that person is unwilling to be interviewed but someone else in the house is willing? How do you deal with multi-family households? Similar problems arise when you sample groups, agencies, or companies. Can you survey any member of the organization? Or, do you only want to speak to the Director of Human Resources? What if the person you would like to interview is unwilling or unable to participate? Do you use another member of the organization?
- Can all members of population be sampled?
If you have an incomplete list of the population (i.e., sampling frame) you may not be able to sample every member of the population. Lists of various groups are extremely hard to keep up to date. People move or change their names. Even though they are on your sampling frame listing, you may not be able to get to them. And, it’s possible they are not even on the list.
- Are response rates likely to be a problem?
Even if you are able to solve all of the other population and sampling problems, you still have to deal with the issue of response rates. Some members of your sample will simply refuse to respond. Others have the best of intentions, but can’t seem to find the time to send in your questionnaire by the due date. Still others misplace the instrument or forget about the appointment for an interview. Low response rates are among the most difficult of problems in survey research. They can ruin an otherwise well-designed survey effort.
Question Issues
Sometimes the nature of what you want to ask respondents will determine the type of survey you select.
- What types of questions can be asked?
Are you going to be asking personal questions? Are you going to need to get lots of detail in the responses? Can you anticipate the most frequent or important types of responses and develop reasonable closed-ended questions?
- How complex will the questions be?
Sometimes you are dealing with a complex subject or topic. The questions you want to ask are going to have multiple parts. You may need to branch to sub-questions.
- Will screening questions be needed?
A screening question may be needed to determine whether the respondent is qualified to answer your question of interest. For instance, you wouldn’t want to ask someone their opinions about a specific computer program without first “screening” them to find out whether they have any experience using the program. Sometimes you have to screen on several variables (e.g., age, gender, experience). The more complicated the screening, the less likely it is that you can rely on paper-and-pencil instruments without confusing the respondent.
- Can question sequence be controlled?
Is your survey one where you can construct in advance a reasonable sequence of questions? Or, are you doing an initial exploratory study where you may need to ask lots of follow-up questions that you can’t easily anticipate?
- Will lengthy questions be asked?
If your subject matter is complicated, you may need to give the respondent some detailed background for a question. Can you reasonably expect your respondent to sit still long enough in a phone interview to ask your question?
- Will long response scales be used?
If you are asking people about the different computer equipment they use, you may have to have a lengthy response list (CD-ROM drive, floppy drive, mouse, touch pad, modem, network connection, external speakers, etc.). Clearly, it may be difficult to ask about each of these in a short phone interview.
Content Issues
The content of your study can also pose challenges for the different survey types you might utilize.
- Can the respondents be expected to know about the issue?
If the respondent does not keep up with the news (e.g., by reading the newspaper, watching television news, or talking with others), they may not even know about the news issue you want to ask them about. Or, if you want to do a study of family finances and you are talking to the spouse who doesn’t pay the bills on a regular basis, they may not have the information to answer your questions.
- Will respondent need to consult records?
Even if the respondent understands what you’re asking about, you may need to allow them to consult their records in order to get an accurate answer. For instance, if you ask them how much money they spent on food in the past month, they may need to look up their personal check and credit card records. In this case, you don’t want to be involved in an interview where they would have to go look things up while they keep you waiting (they wouldn’t be comfortable with that).
Bias Issues
People come to the research endeavor with their own sets of biases and prejudices. Sometimes, these biases will be less of a problem with certain types of survey approaches.
- Can social desirability be avoided?
Respondents generally want to “look good” in the eyes of others. None of us likes to look like we don’t know an answer. We don’t want to say anything that would be embarrassing. If you ask people about information that may put them in this kind of position, they may not tell you the truth, or they may “spin” the response so that it makes them look better. This may be more of a problem in an interview situation where they are face-to face or on the phone with a live interviewer.
- Can interviewer distortion and subversion be controlled?
Interviewers may distort an interview as well. They may not ask questions that make them uncomfortable. They may not listen carefully to respondents on topics for which they have strong opinions. They may make the judgment that they already know what the respondent would say to a question based on their prior responses, even though that may not be true.
- Can false respondents be avoided?
With mail surveys it may be difficult to know who actually responded. Did the head of household complete the survey or someone else? Did the CEO actually give the responses or instead pass the task off to a subordinate? Is the person you’re speaking with on the phone actually who they say they are? At least with personal interviews, you have a reasonable chance of knowing who you are speaking with. In mail surveys or phone interviews, this may not be the case.
Administrative Issues
Last, but certainly not least, you have to consider the feasibility of the survey method for your study.
- costs
Cost is often the major determining factor in selecting survey type. You might prefer to do personal interviews, but can’t justify the high cost of training and paying for the interviewers. You may prefer to send out an extensive mailing but can’t afford the postage to do so.
- facilities
Do you have the facilities (or access to them) to process and manage your study? In phone interviews, do you have well-equipped phone surveying facilities? For focus groups, do you have a comfortable and accessible room to host the group? Do you have the equipment needed to record and transcribe responses?
- time
Some types of surveys take longer than others. Do you need responses immediately (as in an overnight public opinion poll)? Have you budgeted enough time for your study to send out mail surveys and follow-up reminders, and to get the responses back by mail? Have you allowed for enough time to get enough personal interviews to justify that approach?
- personnel
Different types of surveys make different demands of personnel. Interviews require interviewers who are motivated and well-trained. Group administered surveys require people who are trained in group facilitation. Some studies may be in a technical area that requires some degree of expertise in the interviewer.
Clearly, there are lots of issues to consider when you are selecting which type of survey you wish to use in your study. And there is no clear and easy way to make this decision in many contexts. There may not be one approach which is clearly the best. You may have to make tradeoffs of advantages and disadvantages. There is judgment involved. Two expert researchers may, for the very same problem or issue, select entirely different survey methods. But, if you select a method that isn’t appropriate or doesn’t fit the context, you can doom a study before you even begin designing the instruments or questions themselves[4].
Constructing the Survey
Constructing a survey instrument is an art in itself. There are numerous small decisions that must be made — about content, wording, format, placement — that can have important consequences for your entire study. While there’s no one perfect way to accomplish this job, we do have lots of advice to offer that might increase your chances of developing a better final product.
First of all you’ll learn about the two major types of surveys that exist, the questionnaire and the interview and the different varieties of each. Then you’ll see how to write questions for surveys. There are three areas involved in writing a question:
- determining the question content, scope and purpose
- choosing the response format that you use for collecting information from the respondent
- figuring out how to word the question to get at the issue of interest
Finally, once you have your questions written, there is the issue of how best to place them in your survey.
You’ll see that although there are many aspects of survey construction that are just common sense, if you are not careful you can make critical errors that have dramatic effects on your results[5].
Benefits and Weaknesses of Survey Research
Survey research is quickly becoming the number one tool that market researches use to gather data. The advent of online survey tools has led to widespread use of quantitative surveys in order to collect, analyze, and use data that can contribute to a more effective business model, better marketing strategies, improved customer service and more.
Survey research methods have been shown time and time again to benefit market researchers and improve ROI. Yet depending on the type of research you are doing, survey research may not be as appropriate. Here are the advantages and disadvantages of surveys:
Advantages of Survey Research
- Cost
Surveys – particularly online surveys – have a very small cost per participant. Even with incentives, the cost per response is often far less than the cost of running a focus group or a phone interview, while the responses you receive can number into the thousands.
- Data Collection
Data collection is much easier with surveys, which tend to use easy to read data sources that can be compiled and analyzed as needed for your market research needs. The qualitative research techniques of other market research methods may yield some interesting answers, but the ability to analyze themes becomes a much more difficult (and possibly inaccurate) process.
- Sample Size
Sample size is one of the most underrated aspects of all types of research, including market research. Far too many companies make the mistake of assuming their small sample of only a few dozen participants is enough to make any firm conclusions. Surveys allow you to reach thousands of possible participants if necessary, which ensures a more accurate sample in which to draw conclusions.
- Candid Responses
The anonymity of surveys allows people to feel more candid with their responses. To get accurate data, you need your participants to be as honest as possible with their answers. Surveys provide more honest responses than other types of research methodology, especially if it is clear that the answers will remain confidential.
Disadvantages of Survey Research
- Sample Choice
It’s up to the user to ensure they have a representative sample from which to collect data. Survey research is prone to researcher error, where assumptions are made about the sample that may not be accurate. For example, if you are researching low income communities, many may not have email access, so their voice won’t be heard in your data.
- Rigidity
Another weakness of survey research is rigidity. Before you develop your survey, you need to remember to account for all possible answers and program the survey accordingly. If you fail to account for all possible answers you may be missing out on data. For example, if the participant wants to tell you something about your product, but doesn’t have space to do so, you will lose out on that data.
Survey Research for Marketing
Of all of the ways to perform market research, survey research is one of the most reliable and far reaching. It is, however, prone to user error. If the researcher is not prepared for running survey research, the data collected may not be as valuable. However, if the marketer understands how to use survey research effectively and accurately, it has been shown to be the best available way to collect and analyze data[6].
Conclusion
A single survey is made of at least a sample (or full population in the case of a census), a method of data collection (e.g., a questionnaire) and individual questions or items that become data that can be analyzed statistically. A single survey may focus on different types of topics such as preferences (e.g., for a presidential candidate), opinions (e.g., should abortion be legal?), behavior (smoking and alcohol use), or factual information (e.g., income), depending on its purpose. Since survey research is almost always based on a sample of the population, the success of the research is dependent on the representativeness of the sample with respect to a target population of interest to the researcher. That target population can range from the general population of a given country to specific groups of people within that country, to a membership list of a professional organization, or list of students enrolled in a school system (see also sampling (statistics) and survey sampling). The persons replying to a survey are called respondents, and depending on the questions asked their answers may represent themselves as individuals, their households, employers, or other organization they represent.
Survey methodology as a scientific field seeks to identify principles about the sample design, data collection instruments, statistical adjustment of data, and data processing, and final data analysis that can create systematic and random survey errors. Survey errors are sometimes analyzed in connection with survey cost. Cost constraints are sometimes framed as improving quality within cost constraints, or alternatively, reducing costs for a fixed level of quality. Survey methodology is both a scientific field and a profession, meaning that some professionals in the field focus on survey errors empirically and others design surveys to reduce them. For survey designers, the task involves making a large set of decisions about thousands of individual features of a survey in order to improve it.
Bibliography
- Groves, R.M.; Fowler, F. J.; Couper, M.P.; Lepkowski, J.M.; Singer, E.; Tourangeau, R. (2009). Survey Methodology. New Jersey: John Wiley & Sons. ISBN 978-1-118-21134-2.
2. Lynn, P. (2008) “The problem of non-response”, chapter 3, 35-55, in International Handbook of Survey Methodology (ed.s Edith de Leeuw, Joop Hox & Don A. Dillman). Erlbaum. ISBN 0-8058-5753-2
3. Bogen, Karen (1996). “THE EFFECT OF QUESTIONNAIRE LENGTH ON RESPONSE RATES — A REVIEW OF THE LITERATURE” (PDF). Proceedings of the Section on Survey Research Methods. American Statistical Association: 1020–1025. Retrieved 2013-03-19.
4. “Respondent engagement and survey length: the long and the short of it”. research. April 7, 2010. Retrieved 2013-10-03.
5. Eisinga, R.; Te Grotenhuis, M.; Larsen, J.K.; Pelzer, B. (2011). “Interviewer BMI effects on under- and over-reporting of restrained eating. Evidence from a national Dutch face-to-face survey and a postal follow-up”. International Journal of Public Health. 57 (3): 643–647. PMC 3359459. PMID 22116390. doi:10.1007/s00038-011-0323-z.
- https://explorable.com/handling-survey-data, 14.6.2017, 01.00 PM.
[1] https://www.google.com/?gws_rd=ssl#q=what+is+Survey+Research+Analysis+Method, 12.6.2017, 10 AM.
[2] http://www.socialresearchmethods.net/kb/survey.php, 12.6.2017, 11 AM.
[3] http://www.socialresearchmethods.net/kb/survtype.php, 12.6.2017, 11.30 AM.
[4] http://www.socialresearchmethods.net/kb/survsel.php, 12.6.2017, 12.30 PM.
[5] http://www.socialresearchmethods.net/kb/survwrit.php, 12.6.2017, 01.00 PM.
[6] https://surveymethods.com/blog/benefits-and-weaknesses-of-survey-research-2/, 13.6.2017, 10 AM.