Contents | Feedback | Search | DRCNet Home Page | Join DRCNet
DRCNet Library | Schaffer Library | Government Publications | GAO Publications
Statement of Frederick Mulhauser, Assistant Director, Program Evaluation and Methodology Division
GAO/T-PEMD-94-4
October 5, 1993
United States General Accounting Office
GAO Testimony
Before the Subcommittee on Legislation and National Security, Committee on Government Operations,
House of Representatives
Mr. Chairman and Members of the Subcommittee:
I am pleased to respond to your invitation to testify about the work GAO has conducted for the committee in the area of drug use measurement.' We examined three nationally prominent drug use studies: the National Household Survey on Drug Abuse (NHSDA), the High School Senior Survey (HSSS), and the Drug Use Forecasting (DUF) study of booked arrestees, each cited in the President's National Drug Control Strategy.
Let me discuss our findings from three perspectives: what the studies have found, what confidence we should have in those findings, and how drug use measurement studies could be improved, with a particular focus on high-risk populations. In brief, all three studies show a decreasing trend in the overall rate of marijuana use. Since 1985, NHSDA and HSSS have also estimated a decreasing trend in the overall use of cocaine. DUF data, on the other hand, have described an overall stable, high rate of cocaine use among booked arrestees during the first 4 years of the study (1987-90). The use patterns for heroin and other opiates are less clearly identified, due to the problems experienced in obtaining access to users of these drugs. Thus, the three studies produce information regularly on marijuana and cocaine use, but we still do not have adequate prevalence measures of heroin usage in the United States. This is a serious problem in light of recent increases in heroin availability and purity, and decreases in the price charged for this drug.
What confidence can we have in the studies' estimates? We believe that both NHSDA and HSSS have been important in the development of national drug control strategies, since they represent our most highly sophisticated drug prevalence studies to date. We did, however, find a number of limitations. For NHSDA, we found exclusion of groups at high risk for drug use, problematic measurement of heroin and cocaine use, and reliance on subject self-reports. We found that HSSS excludes dropouts and absentees, yields questionable estimates of the drug use patterns of nonwhite populations, does not adequately measure heroin use, and also relies on subject self-reports. Finally, although DUF employs an objective drug use detection technique-urinalysis--in addition to self-reports, the study's results cannot be generalized either to the booking facilities studied or the geographic areas specified in DUF publications. DUF also lacks standardization across study sites, which minimizes comparability across these sites. In summary, we judge NHSDA and HSSS national prevalence estimates to be conservative measures of actual drug use patterns. In addition, our confidence in DUF results is limited since we cannot determine the extent to which DUF drug use findings actually characterize the population of booked arrestees housed in each locality.
What recommendations can we make for improving drug prevalence measurement? We focused our efforts along two lines of work: (1) enhancing the NHSDA, HSSS, and DUF studies; and (2) developing new methods for studying high-risk groups. Promising methodologies, such as hair analysis, deserve exploration as less intrusive means than urinalysis to validate self-reports. Hair analysis can also determine drug use over a more extended period of time, while urinalysis detection is typically restricted to the initial 12 to 96 hours following use. Expanding the subsamples of current surveys and conducting new studies aimed at hard-to-reach groups should improve the coverage of underrepresented, high-risk target populations.
Let me now give you the details of our findings and recommendations, as well as highlight specific implications for the development of a national drug control strategy.
DRUG USE PATTERNS OF TARGETED GROUPS IN NHSDA, HSSS, AND DUF
As already noted, each of these surveys reported a decline in the overall rate of marijuana use. Between 1979 and 1990, the NHSDA past year rate decreased from 18 to 10 percent, while the HSSS yearly rate decreased from 51 to 27 percent. During 1987-90, the DUF booked-arrestee marijuana rate declined from 36 to 19 percent, based on urinalysis test results.
NHSDA and HSSS have demonstrated declines in the overall rate of cocaine use between 1985 and 1990. The NHSDA yearly rate decreased from 6 to 3 percent, and the HSSS yearly rate decreased from 13 to 5 percent. Cocaine use among booked arrestees participating in the DUF study, however, remained high during the period 1987 to 1990. In 1987, 46 percent of all those tested by urinalysis procedures were found to be positive. Three years later, the rate was still above 40 percent (43 percent). This points up the need to develop appropriate cocaine treatment strategies for specific target populations in our national drug control strategies.
NHSDA has not been a useful tool for tracking heroin use, nor was it expected to be given that heroin users frequently do not live in typical household settings. Through 1990, the National Institute on Drug Abuse (NIDA) judged both the pastmonth and past-year use estimates to be too imprecise to publish. HSSS publishes separate heroin and other opiate use rates on an ongoing basis. For 1979-90, the high school senior rate of pastyear heroin use remained stable at less than 1 percent; the rate for other opiate use varied around the 5-percent level. DUF provides a combined score for heroin and other opiate use, based on urinalysis testing. During the period of study, 1987-90, the DUF heroin and other opiate use rate dropped from 14 to 10 percent.
STRENGTHS AND LIMITATIONS OF THE THREE STUDIES
National Household Survey on Drug Abuse
Six particular strengths are emphasized in our report: (1) project management skills; (2) national population coverage; (3) high screening and interview completion rates; (4) inclusion of all major illicit drug types, as well as alcohol and tobacco; (5) statistical determination of drug use patterns and trends; and (6) use of cognitive laboratory procedures to assess more effective, user-friendly ways of acquiring drug use data.
Four selected limitations are discussed here in greater detail because of their implications for drug use measurement and the development of a national drug control policy.
The Accuracy of Subject Self-Reports
NHSDA findings rely entirely on the honesty of subject self reports. Recent studies (1985-91) provide mixed evidence regarding the accuracy of drug use self-reports. Some studies have confirmed the validity of the self-report method. Others found particular groups providing inaccurate responses (for example, arrestees, pregnant females, and discharged clients). Underreporting of the more socially disapproved drugs has also been noted.
The accuracy of self-reported responses for household respondents in NHSDA has never been effectively evaluated, despite the high price tag associated with the study (approximately $12 million in 1991). We therefore cannot be sure of the veracity of subject responses, particularly as related to the more stigmatized, or socially disapproved, drugs (for example, heroin and cocaine).
Problems In Heroin Measurement
NHSDA includes few subjects who have indicated use of heroin either during the past month or during the past year. In part, this reflects the relatively low prevalence rate of heroin use in this country. But it is also a function of the fact that heroin users are frequently not situated in the household environment or are excluded from the sample because they are transients. In 1990, only 1 Hispanic male, 7 black males, and 13 white males indicated using heroin in the past year. During the same time period, there were only 4 Hispanic females, 3 black females, and 4 white females who indicated heroin use. The generalization of NHSDA results to various subgroups of the population is therefore impractical given these limited numbers.
Nevertheless, in 1991, 142,000 heroin users were nationally estimated based on projections from one 79-year-old woman participating in the survey, and 32 percent of the annual heroin users were judged to be older than age 60 based on projections from just seven heroin-using subjects.
Given national concerns about a potential increase in the heroin use rate, it is imperative that additional data sets be integrated to track the prevalence and incidence rates of this drug. Policy planners cannot rely on NHSDA for good documentation of heroin use.
Problems in Cocaine Measurement
In the 1990 NHSDA, the number of individuals said to be using cocaine "once a week" or more was estimated at 662,000. In 1991, this estimate increased to 855,000, indicating a jump of almost 200,000 weekly cocaine users within a single year. This sharp increase has been taken as one indicator of the need to fight a "two-front drug war": one with casual, recreational users; the other with more hard-core, frequent users. However, the 855,000 estimate was subsequently shown to be incorrect. Due to imputation problems, the National Institute on Drug Abuse (NIDA) revised the 1991 user population estimate, first from 855,000 down to 654,000 and then to 625,000, implying that the frequency of weekly cocaine users in NHSDA had, in fact, not risen between 1990 and 1991. obviously, greater quality control procedures need to be in place to catch errors of such significant policy relevance.
Questions have also been raised about the validity of national estimates of "frequent" cocaine users. In 1990, 41 percent of the survey's past-month cocaine users did not initially indicate use of the drug during this time period. The contractor subsequently modified their responses to eliminate contradictory results. However, the procedures for modifying a subject's response pattern are not believed to be entirely justifiable.
Frequency of NHSDA Survey Administration
The Anti-Drug Abuse Act of 1988 requires that the extent of alcohol and drug abuse among the general population be assessed annually. (NIDA had been conducting the NHSDA every 2 to 3 years previous to 1990.) A yearly data collection strategy is questionable for three reasons: (1) the 1991 total cost allocation was sizable, at $11.5 million; (2) prevalence rate changes have been minimal between survey administrations; and (3) hard-core, frequent drug users are often not found in the household environment.
The High School Senior Survey
Five particular strengths are highlighted in our report:
(1) HSSS is managed by a distinguished group of social scientists at the University of Michigan who have guided the project since its inception in 1975; (2) a sophisticated design is utilized to obtain between 15,000 and 19,000 participating seniors each year, drawn from between 120 and 140 public and private schools; (3) the refusal rate among available students has consistently been less than 1 percent; (4) all illicit drug types, as well as alcohol, tobacco, and steroids, are considered; and (5), as in NHSDA, statistical tests have been used to determine whether there have been significant score changes between survey administrations.
Five limitations potentially affecting the validity of HSSS, and thereby the development of national drug policy, are highlighted in our report.
The Accuracy of Subject Self-Reports
Like NHSDA, HSSS has not assessed the accuracy of selfreported drug use against objective criteria. While the study investigators provide reasonable self-report correlative evidence that the findings may be valid, an "honesty" survey question shows that high school senior lifetime drug use estimates for marijuana, heroin, and the amphetamines may be understated. As in the case of NHSDA, objective confirmation of the accuracy of HSSS is needed.
Exclusion of High School Dropouts
By design, dropouts have not been included in the sampling frame of HSSS since the inception of the study in 1975. Since they are thought to have higher rates of drug use than students in school, this implies some underestimation of drug use rates among the age cohort of high school seniors. Correction factors introduced by project officials have not been universally accepted. In fiscal year 1992, NIDA awarded a 5-year grant to HSSS investigators to study drug use among dropouts.
Exclusion of School Absentees
The HSSS field staff does not engage in follow-up school visits because of costs and logistics. As a result, students absent on the day of the survey administration are excluded from participation. According to the coprincipal investigators, absentees constitute approximately 17 to 23 percent of enrolled students. HSS is therefore missing about one in five students because of absenteeism. Drug correction factors have been based on the assumption that absenteeism is a fairly random event and thus the absentee patterns of those present (at the time of the survey) represent those who are absent. This assumption, while plausible, has not been confirmed.
School Nonparticipation and Replacement
The school nonparticipation rate has tended to be between 20 and 40 percent. Since school officials do not indicate school drug problems as a reason for nonparticipation, the assumption has been made that no drug bias occurs in the school replacement process. This conclusion too has never been empirically proven.
Minority Drug Use Estimates
Drug use estimates for nonwhite seniors have traditionally not been reported on a yearly basis in HSSS press releases and publications. Nor have safeguard procedures been adopted to ensure the representativeness of minority groups in the HSSS survey sample. As a result, annual nonwhite drug use patterns among high school seniors over the years have remained unclear.
Drug Use Forecasting
Four strengths of DUF are highlighted in our report: (1) establishment of a much needed drug use data base at the local level; (2) resort to an objective drug use criteria, rather than reliance on subject self-reports; (3) high urinalysis participation rates; and (4) use of a centralized laboratory to minimize potential specimen test biases.
I will discuss three of the principal DUF limitations in my testimony today. The full range of significant limitations are detailed in our report.
Geographic Site Variations
DUF collects self-report interviews and urinalysis specimens from booked arrestees at central booking facilities. These participating facilities, however, often serve very different geographical areas. Some facilities serve an entire city; others part of a city, a central city plus additional cities, an entire county, or parts of a county. These differences are not made clear in DUF publications. Based on annual reports, the impression is created that drug use findings are generated for entire cities, when in fact this is not always the case. Criminal justice planners and decision makers must therefore exercise caution in using these data for the development of overall city and county drug policies.
Subject Sampling Procedure
DUF has not been able to demonstrate a representative sample of booked arrestees either at the central booking facility level or in the geographic locales highlighted in DUF reports. The sampling procedures, as described in the DUF procedures manual, are not being followed closely by all sites.
Privacy in the Interview Situation
Not all facilities use a private office for arrestee interviews. Subjects have been interviewed in hallways traversed regularly by police department personnel; in small alcoves, with a police officer standing guard at the entrance; or through the bars of a holding cell, in close proximity to other arrestees. Under such conditions, there is a potential for underreporting drug use, particularly since arrestees are awaiting arraignment before a judge. (Though pledges of confidentiality are given, arrestees may not fully believe them.) Urinalysis rates, however, should not be affected.
RECOMMENDATIONS FOR THE IMPROVEMENT OF DRUG PREVALENCE ESTIMATES
Enhancing the NHSDA, HSSS, and DUF Studies
In our report, we recommend to the Congress:
-- That part A of title V of the Public Health Service Act be amended to provide that the Secretary of Health and Human Services collect survey data only biennially, rather than each year, on the national prevalence of the various forms of substance abuse among high school students and among the general population. But if local or regional indicators portend an increase in drug use, then the Secretary should have the authority to initiate new or augment current studies to determine the nature and degree of the problem.
Implementation of this recommendation can result in an NHSDA savings of approximately $13 million per year, based on 1993 cost estimates, if the current methodology is maintained. A portion of this cost-savings could be usefully applied toward studying the drug use patterns of those high-risk groups disproportionately impacting criminal justice and health system resources, field-testing new methodologies, and validating our current measures and findings.
We also recommended that the Secretary of Health and Human Services:
-- Give high priority to validating self-reports of the use of illicit drugs, particularly focusing on objective techniques such as hair testing. Hair testing has received publicity both in the United States and abroad because of its potential to distinguish the use of illicit drugs in hair specimens for extended periods of time. At present, we conclude that hair analysis has sufficient scientific merit to justify its use in self-report validation field trials. (This should not be construed to mean that we also support hair analysis in employment testing and court testimony, for which maximal precision would be required.) A hair validation study of four drug types, involving 2,000 subjects, using both a hair screen and a confirmatory test, would cost approximately $150,000 (including subject payment incentives), a relatively small allocation compared to the $13 million cost of conducting just one round of NHSDA.
-- Develop or improve supplementary data sources to more appropriately determine heroin and cocaine prevalence patterns and trends.
In 1991, NHSDA was expanded beyond just a household survey to include the sheltered homeless, civilians on military bases, and residents of college dormitories. These target populations, however, do not encompass the totality of heroin and cocaine users. If we are to more accurately judge the prevalence of heroin and cocaine use in this country, we must come up with ingenious ways of identifying and accessing the relevant population groups (that is, those living in institutional and noninstitutional quarters and on the streets), as well as devise ways to prevent double counting (for example, of a homeless person formerly living in public housing).
-- Incorporate methodological design changes into HSSS so that nonwhite individuals are adequately sampled.
Nonwhite annual drug use rates have typically not been discussed in HSSS reports because of methodological study concerns. We therefore know little about minority drug use rates at the high school level and their contribution to overall national drug prevalence patterns. If we are to plan prevention and treatment strategies aimed at this diverse subgroup, we must certainly ascertain the nature and extent of their drug use on a more timely basis.
-- Retain the current design of NHSDA to provide national estimates only (and not expand the design to provide state-level estimates of drug use).
Expanding NHSDA to the state level would be too costly, given other drug area priorities. The Department of Health and Human Services and NIDA have estimated a cost of about $110 million for state-level survey expansion. Such an effort would also duplicate other already existing studies being conducted in multiple states.
Further, we recommended that the Director of the National Institute of Justice review the practicality of improving the DUF design, such as by using a standardized methodology across sites; and give priority to creating a DUF arrestee data base that can be generalized to booked arrestees in the geographic areas surveyed.
The development of generalizable findings requires several stages of decision making. First, the geographic unit of study must be clarified. (For example, if cities are the relevant unit, then county data from outside city boundaries must be excluded.) Second, central booking facilities must be chosen that adequately represent a cross section of arrestees being detained in that geographic unit. (This may involve the selection of one or more booking facilities.) Third, a sample of arrestees must be obtained from each booking facility to yield an appropriate cross section of that facility's arrestees.
Developing New Methods for Studying High-Risk Groups
Since NHSDA and HSSS do not sufficiently measure drug use among high-risk target groups, supplementary methods must be conceptualized, field-tested, and implemented if we are to better understand drug prevalence rates and trends among these groups.
Until recently, only modest efforts were being made in this area. But momentum is building. We have already discussed NIJ's pioneering work with booked arrestees. Particularly over the past 3 years, NIDA has been active in sponsoring studies aimed at identifying, gaining access to, and interviewing individuals at high risk for substance abuse. The Washington, D.C., Metropolitan Area Drug Study (DC*MADS) was designed to develop prototype methodologies in the Washington, D.C., area for replication in other metropolitan areas. High-risk groups in the study include the homeless and transient populations, school dropouts, juvenile and adult offenders, and the institutionalized. NIDA intends to publish methodological, substantive reports in 1993 describing the procedures adopted in the field experiments, success levels achieved, and resultant drug use findings. It is premature at this time for us to comment on the utility of the various methodologies for obtaining drug use data from these high-risk groups. However, transference to other metropolitan areas is certainly going to be a function of study costs. Budget submissions in September 1990 to the Office of Management and Budget for initial study were quite high (for example, for the homeless and transient study, $883,628; for the school dropout study, $576,033; and for the adult offender study, $577,550).
NIDA funded a 5-year grant award to the University of Michigan's HSSS investigators aimed at following up a national cohort of 8th and 10th graders every 2 years to learn more about the drug use of school dropouts. It is also too soon to comment on the effectiveness of this study, given that the award was made in fiscal year 1992. The direct costs of this study range from $319,000 in the first year of the grant to $619,000 in the fifth year.
The Senate Committee on the Judiciary and the Office of National Drug Control Policy have sought to estimate the number of hard-core and heavy cocaine users through secondary analyses of existing data bases. Ethnographic street studies have been incorporated in high-risk group prevalence estimation efforts, as have nominative techniques and a wide range of traditional operations research procedures. These types of studies are still in their early stages of development with respect to drug use, requiring much more elaboration and specificity, but are certainly worthy of continued attention and funding.
To undertake such work on high-risk groups in an orderly, scientific manner, we recommended that the Secretary of Health and Human Services
-- conduct a systematic program for the study of drug prevalence rates among underrepresented, high-risk groups.
It is not sufficient for agencies to engage ad hoc in singular studies of specific high-risk groups. Given the impingement of these groups on the health care delivery system, policymakers and health officials must have comprehensive data bases from which to plan needed prevention and intervention strategies.
Mr. Chairman, this concludes my remarks. I would be happy to answer any questions you or Members of the Subcommittee may have.