Sign the Resolution
Contents | Feedback | Search
DRCNet Home | Join DRCNet
DRCNet Library | Schaffer Library | Government Publications

Sourcebook of Criminal Justice Statistics

Appendix 6

Public opinion survey sampling procedures

Note: Information on Gallup survey sampling procedures was excerpted from George H. Gallup, The Gallup Poll, Public Opinion 1934-1971, Vol. 1, 1935-1948 (New York: Random House, 1972), pp. vi-viii; George H. Gallup, The Gallup Opinion Index, Report No. 162 (Princeton, NJ: The Gallup Poll, January 1979), pp. 29, 30; George Gallup, The Sophisticated Poll Watcher's Guide (Princeton, NJ: Princeton Opinion Press, 1976), p. 102; and from information provided to SOURCEBOOK staff from The Gallup Organization, Inc. Information on the Harris Poll survey sampling procedures was provided to SOURCEBOOK staff by Louis Harris and Associates, Inc.; similar procedures used in earlier surveys are described in Louis Harris and Associates, Inc., The Harris Yearbook of Public Opinion 1970: A Compendium of Current American Attitudes (New York: Louis Harris and Associates, Inc., 1971), pp. 511-514. Information on the survey procedures employed by the National Opinion Research Center was excerpted from the National Opinion Research Center, General Social Surveys, 1972-1996: Cumulative Codebook (Chicago: National Opinion Research Center, University of Chicago, 1996), pp. v-vii, 54, 965, 966, 1184-1186. Information on the Phi Delta Kappa/Gallup Poll was excerpted from Stanley M. Elam and Lowell C. Rose, "The 27th Annual Phi Delta Kappa/Gallup Poll of the Public's Attitudes Toward the Public Schools," Phi Delta Kappan (September 1995), p. 56; Stanley M. Elam, Lowell C. Rose, and Alec M. Gallup, "The 28th Annual Phi Delta Kappa/Gallup Poll of the Public's Attitudes Toward the Public Schools," Phi Delta Kappan (September 1996), p. 58; Lowell C. Rose, Alec M. Gallup, and Stanley M. Elam, "The 29th Annual Phi Delta Kappa/Gallup Poll of the Public's Attitudes Toward the Public Schools," Phi Delta Kappa [Online]. Available: http://www.pdkintl.org/kappan/kpoll97.htm [Dec.31, 1997]; and Lowell C. Rose and Alec M. Gallup, "The 30th Annual Phi Delta Kappa/Gallup Poll of the Public's Attitudes Toward the Public Schools," Phi Delta Kappa [Online]. Available: http://www.pdkintl.org/kappan/kp9809-a.htm [Jan. 5, 1999]. Information also was excerpted from material provided by The Pew Research Center for The People & The Press. Information on the Youth Risk Behavior Surveillance System was excerpted from Laura Kann et al., "Youth Risk Behavior Surveillance--United States, 1995," CDC Surveillance Summaries, Morbidity and Mortality Weekly Report 47 SS-3 (Washington, DC: USGPO, Aug. 14, 1998).

The sampling procedures of six public opinion surveys or survey organizations are presented in this appendix: The Gallup Poll, the Gallup/Phi Delta Kappa Education Poll, the Harris Survey, the National Opinion Research Center, The Pew Research Center for The People & The Press, and the Youth Risk Behavior Surveillance System.

GALLUP POLLS

All Gallup polls since 1950, excluding certain special surveys, have been based on a national probability sample of interviewing areas. Refinements in the sample design have been introduced at various points in time since then. However, over this period the design essentially has conformed to the current procedure, as described in the following paragraphs.

The United States is divided into seven size-of-community strata: cities of population 1,000,000 and over, 250,000 to 999,999, and 50,000 to 249,000, with the urbanized areas of all these cities forming a single stratum; cities of 2,500 to 49,999; rural villages; and farm or open country rural areas. Within each of these strata, the population is further divided into seven regions: New England, Middle Atlantic, East Central, West Central, South, Mountain, and Pacific Coast. Within each size-of-community and regional stratum the population is arrayed in geographic order and zoned into equal sized groups of sampling units. Pairs of localities in each zone are selected with probability of selection proportional to the size of each locality's population--producing two replicated samples of localities.

Within selected cities for which population data are reported by census tracts or enumeration districts, these sample subdivisions are drawn with probability of selection proportional to the size of the population. For other cities, minor civil divisions, and rural areas in the sample for which population data are not reported by census tracts or enumeration districts, small, definable geographic areas are drawn, with the probability of selection proportional to size where available data permit; otherwise with equal probability.

A block or block cluster is drawn with probability of selection proportional to the number of dwelling units from within each subdivision selected for which block statistics are available. In cities and towns for which block statistics are not available, blocks are drawn at random, that is, with equal probability. In subdivisions that are rural or open country in character, segments approximately equal in size of population are delineated and drawn with equal probability.

In each cluster of blocks and each segment so selected, a randomly selected starting point is designated on the interviewer's map of the area. Starting at this point, interviewers are required to follow a given direction in the selection of households, taking households in sequence, until their assigned number of interviews has been completed. Within each occupied dwelling unit or household reached, the interviewer asks to speak to the youngest man 18 or older at home, or if no man is at home, the oldest woman 18 or older. This method of selection within the household has been developed empirically to produce an age distribution by men and women separately which compares closely with the age distribution of the population. It increases the probability of selecting younger men, who are at home relatively infrequently, and the probability of reaching older women in the household who tend to be under-represented unless given a disproportionate chance of being drawn from among those at home. The method of selection among those at home within the household is not strictly random, but it is systematic and objective, and eliminates interviewer judgment in the selection process. Interviewing is conducted at times when adults are most likely to be at home, which means on weekends or if on weekdays, after 4 p.m. for women and after 6 p.m. for men. Allowance for persons not at home is made by a "times-at-home" weighting procedure rather than by "call-backs." This procedure is a standard method for reducing the sample bias that would otherwise result from under-representation of persons who are difficult to find at home.

The pre-stratification by regions is routinely supplemented by fitting each obtained sample to the latest available U.S. Bureau of the Census estimates of the regional distribution of the population. Also minor adjustments of the sample are made by educational attainment (for men and women separately), based on the annual estimates of the U.S. Bureau of the Census derived from their Current Population Survey. The sample procedure described is designed to produce an approximation of the adult civilian population living in the United States, except for those persons in institutions such as prisons or hospitals. The four regions of the country, as reported in Gallup public opinion surveys, have been defined in the following manner:

East--Maine, New Hampshire, Vermont, Massachusetts, Rhode Island, Connecticut, New York, New Jersey, Pennsylvania, Maryland, Delaware, West Virginia, District of Columbia;

Midwest--Ohio, Michigan, Indiana, Illinois, Wisconsin, Minnesota, Iowa, Missouri, North Dakota, South Dakota, Nebraska, Kansas;

South--Virginia, North Carolina, South Carolina, Georgia, Florida, Kentucky, Tennessee, Alabama, Mississippi, Arkansas, Louisiana, Oklahoma, Texas; and

West--Montana, Arizona, Colorado, Idaho, Wyoming, Utah, Nevada, New Mexico, California, Oregon, Washington, Hawaii, Alaska.

Urbanization--Central cities have populations of 50,000 and above. Suburbs constitute the fringe and include populations of 2,500 to 49,999. Rural areas are those that have populations of under 2,500.

Race/ethnicity--Nonwhite is comprised of individuals who report themselves as any combination of the following classifications: Hispanic, American Indian, Other Indian, Oriental, and Black. Black and Hispanic are subcategories of Nonwhite. However, due to variation in respondent reporting the category White may also include some Hispanics.

According to Gallup policy, if the interviewee does not hear or does not understand a question, the interviewer repeats the question and if on the second reading the person does not understand or does not get the point of the question, the interviewer checks the "no opinion" box. It should also be noted that seldom more than 10% of all those contacted refuse to be interviewed. Gallup Poll Surveys of a nationwide sample usually include approximately 1,000 respondents.

Sampling error

All sample surveys are subject to sampling error, that is, the extent to which the results may differ from those that would be obtained if the entire population surveyed had been interviewed. The size of sampling errors depends largely on the number of interviews. The following table may be used in estimating sampling error. The computed allowances have taken into account the effect of the sample design upon sampling error. They may be interpreted as indicating the range (plus or minus the figure shown) within which the results of repeated samplings in the same time period could be expected to vary, 95% of the time, assuming the same sampling procedure, the same interviewers, and the same questionnaire.

Recommended allowance for sampling error (plus or minus) at 95% confidence level

(Percent)

PHI DELTA KAPPA/GALLUP POLL

The Phi Delta Kappa/Gallup polls are modified probability samples of adults 18 years of age and older living in the United States. The 1995 sample was comprised of 1,311 adults; interviewing took place May 25-June 15, 1995. The 1996 sample was comprised of 1,329 adults; interviewing took place May 2-22, 1996. The 1997 sample was comprised of 1,517 adults, including 1,017 parents of public school children; interviewing took place June 3-22, 1997. The 1998 sample was comprised of 1,151 adults; interviewing took place June 5-23, 1998. The data collection design employed the Gallup Organization's standard national telephone sample, i.e., an unclustered, directory-assisted, random-digit telephone sample, based on a proportionate stratified sampling design. "Nonpublic school parents" includes parents of students who attend parochial schools, private schools, or independent schools. For further information on the survey sampling procedures see Lowell C. Rose and Alec M. Gallup, "The 30th Annual Phi Delta Kappa/Gallup Poll of the Public's Attitudes Toward the Public Schools," Phi Delta Kappa [Online]. Available: http://www.pdkintl.org/kappan/kp9809-a.htm [Jan. 5, 1999].

HARRIS SURVEYS

Harris surveys are based on a national sample of the civilian population of the continental United States. Alaska and Hawaii are not represented in the sample, nor are those in prisons, hospitals, or religious and educational institutions. The sample is based on census information on the population of each State in the country, and on the population living in standard metropolitan areas and in the rest of the country. These population figures are updated by intercensal estimates produced annually by the U.S. Bureau of the Census, and sample locations are selected biennially to reflect changes in the country's demographic profile.

National samples are stratified in two dimensions--geographic region and metropolitan (and non-metropolitan) residence. Stratification insures that the samples will reflect, within 1%, the actual proportions of those living in the country in different regions and metropolitan (and non-metropolitan) areas. Within each stratum the selection of the ultimate sampling unit is achieved through a series of steps, a process which is technically called multi-stage unclustered sampling. Each sampling unit yields one interview. First States, then counties, and then minor civil divisions (cities, towns, townships) are selected with probability proportional to census estimates of their respective household populations.

The Harris Survey has four of these national samples, and they are used in rotation from study to study. The specific sample locations in one study generally are adjacent to those used in the next study. For most surveys covering the entire country, more than one national sample may be employed. Harris Surveys of a nationwide sample usually include approximately 1,250 respondents.

All interviews prior to 1978 were conducted in person, in the homes of respondents. At each household the respondent was chosen by means of a random selection pattern, geared to the number of adults of each sex who live in the household. Interviews lasted approximately 1 hour. Almost all interviews conducted as of 1978 have been telephone interviews. Respondents are selected on the basis of random digit dialing. When the completed interviews are received in New York, a subsample of the respondents are re-contacted to verify that the data have been accurately recorded. Questionnaires are edited and coded in the New York office. The Harris sampling procedure is designed to produce a national cross-section that accurately reflects the actual population of the country 18 years of age and older living in private households. This means that the results of a survey among a national sample can be projected as representative of the country's civilian population 18 years of age and older.

Harris Survey national results are reported for the East, Midwest, South, and West regions of the country, defined as follows:

East--Maine, New Hampshire, Vermont, New York, Massachusetts, Rhode Island, Connecticut, Pennsylvania, Maryland, New Jersey, Delaware, West Virginia;

Midwest--North Dakota, South Dakota, Nebraska, Kansas, Minnesota, Iowa, Missouri, Wisconsin, Illinois, Michigan, Indiana, Ohio;

South--Kentucky, Virginia, Tennessee, North Carolina, South Carolina, Georgia, Alabama, Mississippi, Florida, Louisiana, Arkansas, Oklahoma, Texas; and

West--Washington, Oregon, California, Idaho, Nevada, Utah, Arizona, Montana, Wyoming, Colorado, New Mexico.

Sampling error

The results of the surveys are subject to sampling error, i.e., the difference between the results obtained from the sample and those that would be obtained by surveying the entire population. The size of a possible sampling error varies to some extent with the size of the sample and with the percentage giving a particular answer. The following table sets forth the range of error in samples of different sizes and at different percentages of response.

For example, if the response for a sample size of 1,200 is 30%, in 95 cases out of 100 the response in the population will be between 27% and 33%. This error accounts only for sampling error. Survey research also is susceptible to other errors, such as data handling and interview recording.

Recommended allowance for sampling error
(plus or minus) at 95% confidence level

NATIONAL OPINION RESEARCH CENTER

The National Opinion Research Center (NORC) maintains a national probability sample. The General Social Surveys (GSS) are interviews administered to the NORC national samples using a standard questionnaire. They have been conducted during February, March, and April from 1972 to 1978, 1980, 1982 to 1991, 1993, 1994, and 1996. There are a total of 35,284 completed interviews (1,613 in 1972; 1,504 in 1973; 1,484 in 1974; 1,490 in 1975; 1,499 in 1976; 1,530 in 1977; 1,532 in 1978; 1,468 in 1980; 1,506 and an oversample of 354 blacks in 1982; 1,599 in 1983; 1,473 in 1984; 1,534 in 1985; 1,470 in 1986; 1,466 and an oversample of 353 blacks in 1987; 1,481 in 1988; 1,537 in 1989; 1,372 in 1990; 1,517 in 1991; 1,606 in 1993; 2,992 in 1994; and 2,904 in 1996). Sampling frames are based on 1970 census information for surveys conducted in 1972-78, 1980, and 1982. For all interviews conducted from 1984-91, the national sampling frame was based on 1980 census information. A split sample transition design was used in the 1983 survey. One-half of the sample was drawn from the 1970 frame and one-half from the 1980 frame. Again in 1993, a split sample transaction design was employed on the 1993 survey to measure the effect of switching from the 1980 sample frame to the 1990 sample frame. Half the sample was drawn from each frame. Since 1973, the median length of the interview has been about one and a half hours. This study employed standard field procedures for national surveys, including interviewer hiring and training by area supervisors in interviewing locations when necessary.

Each survey is an independently drawn sample of English-speaking persons 18 years of age and older, living in non-institutional arrangements within the United States. Alaska and Hawaii are not included in samples drawn from the 1970 sampling frame, but are represented in one-half of the 1983 surveys and all those conducted from 1984-96. Block quota sampling was used in the 1972, 1973, and 1974 surveys and for half of the 1975 and 1976 surveys. Full probability sampling was employed in half of the 1975 and 1976 surveys and in all of the surveys conducted subsequent to 1976.

The sample is a multi-stage area probability sample to the block or segment level. At the block level, quota sampling is used with quotas based on sex, age, and employment status. The cost of the quota samples is substantially less than the cost of a full probability sample of the same size, but there is, of course, the chance of sample biases mainly due to not-at-homes which are not controlled by the quotas. However, in order to reduce this bias, the interviewers are given instructions to canvass and interview only after 3:00 p.m. on weekdays or during the weekend or holidays. The first stage of sample selection includes selection of the Primary Sampling Units (PSUs). The PSUs employed are Standard Metropolitan Statistical Areas (SMSAs) or nonmetropolitan counties selected in NORC's Master Sample. These SMSAs and counties were stratified by region, age, and race before selection. The units of selection of the second stage were block groups (BGs) and enumeration districts (EDs). These EDs and BGs were stratified according to race and income. The third stage of selection was that of blocks, which were selected with probabilities proportional to size. In places without block statistics, measures of size for the blocks were obtained by field counting. The average cluster size is five respondents per cluster.

The quotas call for approximately equal numbers of men and women with the exact proportion in each segment determined by the 1970 census tract data. For women, the additional requirement is imposed that there be the proper proportion of employed and unemployed women in the location. Again, these quotas are based on the 1970 census tract data. For men, the added requirement is that there be the proper proportion of men over and under age 35 in the location. Past experience suggests that, for most purposes, this quota sample of 1,500 could be considered as having about the same efficiency as a simple random sample of 1,000 cases.

The 1975 and 1976 studies were conducted with a traditional sample design, one-half full probability and one-half block quota. The sample was divided into two parts for several reasons: (1) to provide data for possibly interesting methodological comparisons; and (2) on the chance that there are some differences over time, that it would be possible to assign these differences to either shifts in sample designs, or changes in response patterns. Having allowed for the appearance of all items in the transitional sample design, the GSS then switched to a full probability sample beginning with the 1977 survey.

Rotation

Since its inception the GSS employed a rotation design under which most of its items appeared on two out of every three surveys. While this design proved to be useful for both monitoring change and augmenting the content of the GSS, it had the disadvantage of irregularly spacing the data and allowing gaps in the time series. This situation was particularly acute during 1978-82 because of the lack of funding for surveys in 1979 and 1981. At that juncture 4-year gaps regularly appeared in the data and 6-year lapses existed for bivariate correlations between items from different rotations. Even with annual surveys 2-year gaps and 3-year intervals for bivariate correlations occur. To reduce this imbalance in the time series and reduce the length of intervals, in 1988 the rotation, across-time design previously used was changed to a split-ballot design. Under this design rotations 1, 2, and 3 occur across random sub-samples within each survey rather than across surveys (and years). Each sub-sample (known as ballots) consists of 1/3 of the sample. Permanent items are not affected by this switch. They continue to appear on all cases for all surveys. Rotating items now appear on all surveys and are asked of two-thirds of respondents on each survey. Over a 3-year cycle the same number of respondents are asked the "rotating" items as before (3,000), but instead of coming in two segments of 1,500 each from two surveys, they appear in three segments of 1,000 each from three surveys.

The 1993 GSS was the last survey conducted under this design. In 1994 two major innovations were introduced to the GSS.

First, the traditional core was substantially reduced to allow for the creation of mini-modules (i.e. blocks of about 15 minutes devoted to some combination of small- to medium-sized supplements). The mini-modules space provides greater flexibility to incorporate innovations and to include important items proposed by the social science community.

Second, a new biennial, split-sample design was used. The sample consists of two parallel sub-samples of approximately 1,500 cases each. The two sub-samples both contain the identical core. The A sample also contains a standard, topical module, the mini-modules, and an International Social Survey Program (ISSP) module (on women, work, and the family). The B sample has a second topical module, mini-modules, and an ISSP module (on the environment). In effect, one can think of the A sample as representing a traditional GSS for 1994 and the B sample representing a traditional GSS for 1995. Rather than being fielded separately in two different years they are fielded together.

In 1996 (and in subsequent even-numbered years), the same design described for 1994 was repeated. In addition, in 1994 only, a transitional design was utilized to calibrate any impact of deletions from the core.

Survey results are reported for four regional categories, with the States classified in the following way:

Northeast--Connecticut, Maine, Massachusetts, New Hampshire, New Jersey, New York, Pennsylvania, Rhode Island, Vermont;

North Central--Illinois, Indiana, Iowa, Kansas, Michigan, Minnesota, Missouri, Nebraska, North Dakota, Ohio, South Dakota, Wisconsin;

South--Alabama, Arkansas, Delaware, District of Columbia, Florida, Georgia, Kentucky, Louisiana, Maryland, Mississippi, North Carolina, Oklahoma, South Carolina, Tennessee, Texas, Virginia, West Virginia;

West--Alaska, Arizona, California, Colorado, Hawaii, Idaho, Montana, Nevada, New Mexico, Oregon, Utah, Washington, Wyoming.

THE PEW RESEARCH CENTER FOR THE PEOPLE & THE PRESS

Both the May/June and November/December 1996 surveys were conducted by Princeton Survey Research Associates for The Pew Research Center for The People & The Press. The survey results are based on telephone interviews among nationwide samples of adults 18 years of age and older. The samples are random digit dialing samples of telephone numbers selected from telephone exchanges in the continental United States. Both listed and unlisted numbers (including not-yet-listed numbers) are represented. The telephone exchanges were selected with probabilities proportional to their size. The first eight digits of the sampled telephone numbers (area code, exchange, bank number) were selected to be proportionally stratified by county and by telephone exchange within county. That is, the number of telephone numbers randomly sampled from within a given county is proportional to that county's share of telephone households in the United States. Estimates of the number of telephone households within each county are derived from 1990 Census data on residential telephone incidence that have been updated with State-level information on new telephone installations and county-level projections of the number of households.

At least four attempts were made to complete an interview at every sampled telephone number. The calls were staggered over times of the day and days of the week to maximize the chances of making a contact with a potential respondent. All interview breakoffs and refusals were recontacted at least once in order to attempt to convert them to completed interviews. In each contacted household, interviewers asked to speak with the "youngest male 18 or older who is at home." If there was no eligible male at home, interviewers asked to speak with "the oldest woman 18 or older who lives in the household." This systematic respondent selection technique has been shown empirically to produce samples that closely mirror the population in terms of age and gender. To compensate for potential biases in survey-derived estimates the sample data have been weighted in the analysis. The demographic weighting parameters were derived from an analysis of the most recently available U.S. Bureau of the Census' Current Population Survey (March 1994). This analysis produced population parameters for the demographic characteristics of households with adults 18 years of age and older, which are then compared with the sample characteristics to construct sample weights. The analysis included only households in the United States that contained a telephone.

YOUTH RISK BEHAVIOR SURVEILLANCE SYSTEM

The Youth Risk Behavior Surveillance System (YRBSS) is conducted by the U.S. Department of Health and Human Services, Centers for Disease Control and Prevention and monitors priority health risk behaviors among youth and adults. The 1997 national school-based survey, a component of the YRBSS, employed a three-stage cluster sample design to produce a nationally representative sample of students in grades 9 through 12. The first-stage sampling frame contained 1,719 primary sampling units (PSUs), consisting of large counties or groups of smaller, adjacent counties. From the 1,719 PSUs, 54 were selected from 16 strata formed on the basis of the degree of urbanization and the relative percentage of black (non-Hispanic) and Hispanic students in the PSU. The PSUs were selected with probability proportional to school enrollment size. At the second sampling stage, 191 schools were selected with probability proportional to school enrollment size. To enable separate analysis of black and Hispanic students, schools with substantial numbers of black (non-Hispanic) and Hispanic students were sampled at higher rates than were all other schools. The third stage of sampling consisted of randomly selecting one or two intact classes of a required subject (e.g., English or social studies) from grades 9 through 12 at each chosen school. All students in the selected classes were eligible to participate in the study. The school response rate was 70% and the student response rate was 87%, for an overall response rate of 69%. A total of 16,262 questionnaires were completed in 151 schools.

A weighting factor was applied to each student record to adjust for nonresponse and for the varying probabilities of selection, including those resulting from the oversampling of black (non-Hispanic) and Hispanic students. Numbers of students in racial/eth- nic groups other than white (non-Hispanic), black (non-Hispanic), and Hispanic were too low for meaningful analysis. The weights were scaled so that the weighted count of students was equal to the total sample size and so that the weighted proportions of students in each grade matched national population proportions.

The data are representative of students in grades 9 through 12 in public and private schools in the 50 States and the District of Columbia.

Sourcebook of Criminal Justice Statistics