Southern Online Journal of Nursing Research

www.snrs.org

Issue 2, Vol. 5
May 2004

Accruing the Sample in Survey Research

Saunjoo L. Yoon, PhD, RN¹; Claydell H. Horne, PhD, RN²

 ¹Assistant Professor; ²Adjunct Associate Professor

University of Florida College of Nursing, Gainesville

Abstract

Although sampling is a challenge in any type of research, survey research is particularly vulnerable to bias based upon the sample. The purposes of this article are to discuss 1) the use of random sampling in survey research, 2) methods to enhance acquisition of the sample, and 3) the strengths and limitations of four types of sampling methods including telephone, mail, face-to-face (personal) interview, and electronic (Internet) surveys. Examples of survey types are presented and strategies are proposed to improve the sampling process and yield more generalizable findings.

Keywords: survey research, sample acquisition, mail survey, telephone survey, interview survey, electronic survey

______________________________________

Data from an entire population of interest are rarely feasible to obtain; therefore, careful sampling is necessary to obtain unbiased estimates of the desired population. Selection of the study population should be based on previous research and the current research question. Frequently, the findings from survey research are not generalizable beyond the selected sample either because a convenience sample is used or because the response rates from the sample are low. There are, however, ways to make the findings of survey research more generalizable and, therefore, more beneficial. Thus, the purposes of this article are threefold: 1) to discuss the use of random sampling in survey research, 2) to discuss methods to enhance acquisition of the sample, and 3) to discuss the strengths and limitations of four types of sampling methods including telephone, mail, face-to-face (personal) interview, and electronic (Internet) surveys. Examples of survey types are presented and for each type, strategies are proposed to improve the sampling process and yield more generalizable findings. Although important, the calculation of sample size is not a focus of this discussion.

Sampling Framesfor Survey Research

Random sampling and convenience sampling are the two most common sampling techniques used by researchers to conduct quantitative studies, while purposive or judgmental sampling is used for qualitative studies. Random sampling can be defined as “a selection process in which each element in the population has an equal, independent chance of being selected.”^1p295

In contrast, convenience sampling or accidental sampling, a type of nonprobability sampling, involves a use of “the most conveniently available people as study participants.”^1p292 Random sampling allows the researcher to estimate the degree of expected error and to more accurately generalize the findings to the population of interest, thereby, increasing the credibility of research findings in survey research.^2-3 Despite the advantages of using random sampling, many nursing researchers use convenience sampling because, as its name suggests, it is convenient, economical, and less time-consuming. Another deterrent to random sampling includes ethical decisions that place participants at risk if they are randomized into a control group.² The advantages of convenience sampling must be weighed against its disadvantages, these include possible sampling bias, a less representative sample of the population, and a limited generalizability of the results.^1-2  Therefore, for non-experimental research such as a survey design, it is well worth the effort and cost for researchers to use random sampling methods whenever possible.

Sample Sources for Survey Research

Once researchers have identified their population of interest, they should determine what sources they will use for sample selection. Public records such as national or state licensing agencies, Departments of Safety and Motor Vehicles, voter registration lists, and telephone directories are usually available for a minimal charge at national, state, and local levels.  

When choosing a source for subject selection, researchers should be knowledgeable of the types of information provided. It is important to identify sample limitations, such as cost, unavailable demographic data, and questions about whether the sample composition is representative of the study population. Although the cost of obtaining the listing may be a limitation, most publicly available lists are reasonably priced according to cost to the agency. Researchers should determine that the necessary demographic information such as age, gender, or race is available and that the individuals on the list can be sorted by demographic characteristics. For example, the voter registration office in a given area can supply information on the percentage of registered voters, as well as on age, gender, and race. Thus, researchers can be confident that subject selection is representative in these areas. If a researcher is studying adult women over 18 years of age in a given geographicalarea and decides to use the Department of Safety and Motor Vehicles as a source, it would be important to know the percent of women over 18 years of age without a current driver’s license. Also, the researcher would need to determine whether the cohort of people with driver’s licenses has a racial representation similar to the population of interest.

If a mailed survey is to be used, the source should provide a current address, which is often problematic in our mobile society. Programs such as www.cleanlist.com can be purchased to validate addresses for large lists of individuals. It is important to understand that unidentified, undeliverable addresses may adversely affect response rates. Researchers often cannot tell the difference between a non-respondent and a person who did not receive the survey because of an outdated address.

If a phone survey is planned, the researcher must know a current phone number, the percent of residences in the United States without telephones, and the percent of residences with unlisted telephone numbers. Information on telephone access in the United States and in individual states is readily available from the U.S. Census Bureau  (See Table 1). Telephone directory data tend to be a representative source, since only 1.2% of owner occupied homes and 4.9% of renter-occupied homes within the U.S. are without telephones.⁴ A future concern related to dependence on telephone directories for generalizable data is the trend towards mobile phones instead of conventional landline phones since cell phone usage continues to rise.^5-6

Issues Related to Non-Response to Survey Research

Non-response is a major concern to researchers conducting survey research using any survey mode because the non-response rate may affect the representativeness of the sample. Although there is no documented minimum acceptable response rate in the literature, the Office of Management of Budget of the federal government requests that government contracted surveys achieve a minimum of 75% response rate.^7p42 However, wide ranges of response rates are recognized in different survey studies.³ Dillman^3p150 suggests and discusses five elements for survey studies to achieve high response rates.They are as follows:

1.      a respondent-friendly questionnaire,

2.      up to five contacts with the questionnaire recipient,

3.      inclusion of stamped return envelopes,

4.      personalized correspondence, and

5.      a token financial incentive that is sent with the survey request.

Other elements affecting response rates relate to the type of survey mode used such as mail, telephone, face-to-face (personal interview), and Internet (web) surveys; geographical locations (e.g., rural vs. urban); and salience.^7-9 Each type of survey mode has its own strengths and limitations; therefore, a mixture of these survey modes may increase the response rate.^3,7 When survey modes are mixed, one must consider time, cost, and compatibility of the survey modes. To increase the response rate in survey research, it is imperative for the researcher to choose the best survey mode(s) for the selected sample from the very beginning of the study design.

Selected Surveys Modes: Strategies to Increase Response Rates

Telephone Survey

Telephone survey can be an economical, convenient mode to access a large number of persons in a short time frame, is flexible, and is not geographically bound. The phone numbers of subjects can be from a generated list or by random-digit-dialing (RDD). An advantage to the generated list is that if addresses are known, an advance letter can be mailed prior to the telephone call and small advance incentives can be given. It is difficult to achieve a high response rate with RDD.^7p47 Limitations to telephone surveys include the distaste of the general public to calls from persons they do not know, the high number of calls (6-10) needed to one household to make contact,^7p46 information is limited to that appropriate for telephone conversations, privacy may be limited, and persons with certain disabilities may be excluded. Specifically, persons who are hard of hearing (a large percentage of the oldest old) and persons with mild cognitive impairment are disadvantaged with telephone surveys; whereas, these groups may be able to effectively complete a mailed survey or a face-to-face interview survey.

There are ways to increase response rates with telephone surveys. If possible, use a combined mail and phone mode with an advance letter and incentive. Quickly identify the caller as the person who sent the advance letter so that the caller will not be falsely viewed as a telemarketer. If RDD is used, start conversation with identification and succinct purpose for the call. Be courteous, do not start with “small talk” such as “how are you today?,” speak clearly and slowly, give an approximate time that the telephone interview will take, and ask permission to proceed. Use simplicity and give explicit directions for responses before asking the question. The survey should be formatted for short answers such as yes or no or the end points of a Likert scale; a full Likert scale is difficult for most telephone respondents. Response rates for telephone interviews are highly dependent upon the interviewer’s skills.

Mailed Survey

A common approach to survey research is the mailed survey. There are numerous strengths to this mode with some being ease of accessibility to the participant, written questions are easier to communicate than verbal questions, responses can be more detailed and contemplated, and the participant can respond at a more convenient time. However, the mailed survey is not without limitations such as convincing the person to participate without the human element of the interviewer, inability to differentiate between undelivered mail and non-respondents, and need of the respondent to have skills in reading and writing.

Response rates to mailed surveysusing random sampling are usually low (around 30%^1,12-13); therefore, additional measures are needed to increase this rate. The more done to encourage the respondent to complete and mail the survey immediately, the more likely it is to be returned.  Dillman reports response rates over 70% on mailed surveys when some of the following techniques are considered.³

Time Considerations

The researcher should choose a time for mailing surveyswhen people are more likely to respond and less likely to put the survey aside to never resurface. Year-end holidays are not good times for surveys to arrive in the mail; therefore, it is poor planningto mail surveys during the months of November or December. Also, times of high volume mail such as the first of the year or the first of the month should be avoided. A strategic plan is to mail the surveys so that they will arrive on a Tuesday or Wednesday. If people do not complete the survey immediately, they frequently use the weekend to catch-up on mail that arrives in the middle of the week. Generally, summer is not an ideal time to mail surveys, because people are often away on vacation. They are not as likely to complete a survey that comes with a volume of other mail upon their return home. Researchers need to know the migratory trends of the geographical area they are surveying, since some people spend months at a time in another location.

Method of Mailing and Preparation.

First-class mail should be used for greater efficiency. Although bulk mail is less expensive, it may not be delivered for up to three weeks, and busy people are not as likely to open bulk mail upon receipt or sometimes even at all. First-class mail carries a sense of increased importance. A postage stamp that is attractive and not controversial will generate more response than metered mail. To encourage prompt return, a self-addressed, stamped envelope should be included. Again, a postage stamp is preferred for the return mail. A request should be made for the return of mail that cannot be forwarded or for the forwarded mail address to be sent to the researcher so that address records can be updated and the researcher can account for undelivered mail that would otherwise have to be considered a non-response.

The introductory letter is of great importance. Rather than a generic letter, use the person’s name. Databases and computer merge capabilities make this personal touch easy. Another personal touch is added by individually signing each letter. The letter should be short while still informative. Be clear and concise about your request and include a convincing reason for the person to participate. Also mention the approximate time it will take the recipient to complete the survey. State a reasonable time for return, such as two weeks after mailing. If too much time is allowed, there is greater likelihood that the response will be put off and forgotten. If too little time is given and the person cannot immediately complete and mail the survey, this too may discourage response. Finally, anonymity or confidentiality of information must be assured. If possible, the letter will get better response if it is written or sent by someone whose name the respondent recognizes. Of course the letter should be on official letterhead, typed, neat, and professionally written. A small incentive included with the mailing will also increase the likelihood of a returned survey. Some suggestions are a miniature pencil to use in survey completion, a gift certificate, a small laminated calendar or other useful tool such as a calorie counter, a small book of postage stamps, or a small amount of cash.

Survey Characteristics

The survey must be attractive and professional in appearance. Simplicity is the best rule. In addition to providing quality measurements, the directions and questions must be clearly and simply stated.^3,7The length of the questionnaire should be short. If the questionnaire appears to be long, create appropriate subsets of questions for certain subsamples of your respondents.⁷ Surveys should be as short as possible to enhance response rates and should include only those questions essential for addressing the hypotheses and research questions. If items do not address the purpose, hypotheses, and research questions, they should not be included. Item choices should have little narrative with the option of “not applicable” or “other” included when appropriate. Begin the survey with simple non-threatening information. Some of the more sensitive demographics, such as age, income, ethnicity, and religious preference, should be placed at the end of the survey. After completing most of the survey, a respondent who may have initially hesitated to answer sensitive demographic information will be more likely to complete the survey if this information is at the end; if such information is at the beginning, however, a person may not even start the survey. A non-threatening method of asking age is to ask date of birth; income information is better asked by giving income categories for selection. A good model for race information is the census survey.¹⁰ Allow a write-in space for race so that each person can feel included in the manner in which they choose. Sometimes, it is the little things that make a person feel excluded and refuse to participate. Provide simple directions for returning the survey. Otherwise, a completed survey may never be put into the mail for return.

Anonymous versus Confidential Surveys

Researchers must determine if the survey will be anonymous or confidential. In anonymous surveys, the respondents remain unidentifiable; in confidential surveys, the respondents are identifiable but are assured that information will be kept private. The most important issue is that the participants understand the privacy issues and are clear about the researcher’s intent. Researchers must be honest and forthright in the issues of anonymity and confidentiality. If the information sought is of a sensitive nature, anonymity will likely result in better response rates; however, the disadvantage of anonymity is that the first respondents cannot be identified and omitted insubsequent second mail-outs. Fowler⁷ suggests an alternative strategy for follow-up when anonymity is desired.A postcard with the respondent identifier is sent with the survey and a request is made to place the card in the mail when the survey is completed. Fowler’s card reads as follows:

“Dear Researcher, I am sending this postcard at the same time that I am putting my completed questionnaire in the mail. Since my questionnaire is completely anonymous, this postcard will tell you that you need not send me a further reminder to return the questionnaire.”^7p50

Fowler reports that the number of postcards usually equals the number of returned surveys.

Subsequent Mail-Outs

If participants can be identified by returned surveys, Dillman³ recommends that a postcard reminder be mailed to all non-respondents about 10 days following the first mail-out. Then ten days after the postcard, a letter, questionnaire, and a return stamped envelope are sent to non-respondents.If the survey is confidential, an identification code is used, rather than the recipient’s name, to identify participants. The code and its purposeare briefly explained to participants. If the surveys are anonymous, it may be necessary to send a reminder card to all potential participants, remembering to thank those who returned the survey. It may also be advantageous to send another copy of the survey to all, in case the person has misplaced the first mailing. However, since respondents are not identified in anonymous surveys and each individual must receive the survey again, the disadvantages of re-sending the survey are added cost and the chance that some subjects may unintentionally submit two surveys.

Prediction of Response Rates From Mailed Surveys

An average response rate from a random sample for mailed surveys has been reported to be around 30%.^12-13 The suggestions provided above can increase this rate. There tend to be fast responders, scattered respondents during the return period, and a few responders after the requested time for return. This pattern repeats itself with the second mailing, but in smaller numbers. The history of survey research for the authors yields some predictions for response rates. About 30 % of the total number of respondents will return the surveys within the first week after mailing. The time between the end of week one and the effects of the second mail-out will yield 50% of the total return. The second mailing of the survey to non-respondents will yield another 20%. A close projection of the total response rate may be made from the early respondents. For example if 1,000 surveys are mailed and 180 are returned within the first week, you can project that your return rate will be about 60% (see Figure 1).

Figure 1.         Equation for Predicting Response Rate

Step 1 calculation: (180 * 100% = 30% * x) or (180 = .3x) thus x = 600 [ratio and proportion