This is the accessible text file for GAO report number GAO-06-818 
entitled 'ONDCP Media Campaign: Contractor's National Evaluation Did 
Not Find That the Youth Anti-Drug Media Campaign Was Effective in 
Reducing Youth Drug Use' which was released on August 25, 2006. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to the Subcommittee on Transportation, Treasury, the Judiciary, 
Housing and Urban Development, and Related Agencies, Committee on 
Appropriations, U.S. Senate: 

United States Government Accountability Office: 

GAO: 

August 2006: 

ONDCP Media Campaign: 

Contractor's National Evaluation Did Not Find That the Youth Anti-Drug 
Media Campaign Was Effective in Reducing Youth Drug Use: 

Youth Anti-Drug Media Campaign: 

GAO-06-818: 

GAO Highlights: 

Highlights of GAO-06-818, a report to the Chairman and Ranking Member, 
Subcommittee on Transportation, Treasury, the Judiciary, Housing and 
Urban Development, and Related Agencies, Committee on Appropriations, 
U.S. Senate 

Why GAO Did This Study: 

Between 1998 and 2004, Congress appropriated over $1.2 billion to the 
Office of National Drug Control Policy (ONDCP) for the National Youth 
Anti-Drug Media Campaign. The campaign aimed to prevent the initiation 
of or curtail the use of drugs among the nation’s youth. In 2005, 
Westat, Inc., completed a multiyear national evaluation of the 
campaign. 

GAO has been mandated to review various aspects of the campaign, 
including Westat’s evaluation which is the subject of this report. 
Applying generally accepted social science research standards, GAO 
assessed (1) how Westat provided credible support for its findings and 
Westat’s findings about (2) attitudes, beliefs, and behaviors of youth 
and parents toward drug use and (3) youth self-reported drug use. 

What GAO Found: 

GAO’s review of Westat’s evaluation reports and associated 
documentation leads to the conclusion that the evaluation provides 
credible evidence that the campaign was not effective in reducing youth 
drug use, either during the entire period of the campaign or during the 
period from 2002 to 2004 when the campaign was redirected and focused 
on marijuana use. By collecting longitudinal data—i.e., multiple 
observations on the same persons over time—using generally accepted and 
appropriate sampling and analytic techniques, and establishing reliable 
methods for measuring campaign exposure, Westat was able to produce 
credible evidence to support its findings about the relationship 
between exposure to campaign advertisements and both drug use and 
intermediate outcomes. In particular, Westat was able to demonstrate 
that its sample was not biased despite sample coverage losses, 
maintained high follow-up response rates of sampled individuals to 
provide for robust longitudinal analysis, established measures of 
exposure that could detect changes in outcomes on the order of 
magnitude that ONDCP expected for the campaign and that could reliably 
measure outcomes, and used sophisticated statistical methods to isolate 
causal effects of the campaign. 

Westat’s findings on the effects of exposure on intermediate 
outcomes—theorized precursors of drug use—were mixed. Specifically, 
although sampled youth and parents’ recall of campaign advertisements 
increased over time, they had good impressions of the advertisements, 
and they could identify the specific campaign messages, exposure to the 
advertisements generally did not lead youth to disapprove of using 
drugs and may have promoted perceptions among exposed youth that 
others’ drug use was normal. Parents’ exposure to the campaign led to 
changes in beliefs about talking about drug use with their children and 
the extent to which they had these conversations with their children. 
However, exposure did not appear to lead to increased monitoring of 
youth. Moreover, the evaluation was unable to demonstrate that changes 
in parental attitudes led to changes in youth attitudes or behaviors 
toward drug use. 

Westat’s evaluation indicates that exposure to the campaign did not 
prevent initiation of marijuana use and had no effect on curtailing 
current users’ marijuana use, despite youth recall of and favorable 
assessments of advertisements. Although general trend data derived from 
the Monitoring the Future survey and the Westat study show declines in 
the percentage of youth reportedly using marijuana from 2002 to 2004, 
the trend data do not explicitly take into account exposure to the 
campaign, and therefore, by themselves, cannot be used as evidence of 
effectiveness. In Westat’s evaluation of relationships between exposure 
and marijuana initiation the only significant finding was of small 
unfavorable effects of the campaign exposure on marijuana initiation 
during some periods of data collection and in some subgroups. 

What GAO Recommends: 

Given that Westat’s evaluation stated the campaign did not reduce youth 
drug use nationally, Congress should consider limiting appropriations 
for the campaign, beginning in the 2007 fiscal year budget until ONDCP 
provides credible evidence of a media campaign approach that 
effectively prevents and curtails youth drug use. ONDCP’s written 
comments on our report generally disagreed with the findings. 
Specifically, ONDCP does not believe the Westat findings reflect the 
campaign’s effectiveness. We believe the Westat study is sound. 

[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-818]. 

To view the full product, including the scope and methodology, click on 
the link above.
For more information, contact Laurie Ekstrand, (202) 512-8777, 
ekstrandl@gao.gov. 

[End of Section] 

Contents: 

Letter: 

Results in Brief: 

Background: 

Westat's Evaluation Design, Use of Generally Accepted and Appropriate 
Sampling and Analytic Techniques, and Reliable Methods for Measuring 
Campaign Exposure Produced Credible Evidence to Support Its Findings: 

The Phase III Evaluation Provided Mixed Evidence of the Campaign's 
Effectiveness on Intermediate Outcomes, but It Found No Effect of the 
Campaign on Parental Monitoring of Youth: 

The Phase III Evaluation Found No Significant Effects of Exposure to 
the Campaign on Youth Drug Use Outcomes Other than Limited Unfavorable 
Effects on Marijuana Initiation: 

Conclusions: 

Matter for Congressional Consideration: 

Agency Comments and Our Evaluation: 

Appendix I: Westat's Methods for Addressing Evaluation Implementation 
Issues: 

Appendix II: Comments from the Office of National Drug Control Policy: 

Appendix III: GAO Contacts and Staff Acknowledgments: 

Tables: 

Table 1: Appropriations for the National Youth Anti-Drug Media 
Campaign, Fiscal Years 1998 through 2006: 

Table 2: NSPY Survey Rounds and Response Rates, Sampled and Surveyed 
Youth: 

Figures: 

Figure 1: Data Collection Rounds and Waves of the NSPY: 

Abbreviations: 

CPS: Current Population Survey: 
GRP: gross rating points: 
MTF: Monitoring the Future: 
NIDA: National Institute on Drug Abuse: 
NIP: National Immunization Program: 
NIS: National Immunization Survey of Children: 
NLSY: National Longitudinal Survey of Youth: 
NSDUH: National Survey on Drug Use and Health: 
NRC: National Research Council: 
NSPY: National Survey of Parents and Youth: 
OMB: Office of Management and Budget: 
ONDCP: Office of National Drug Control Policy: 
PART: Performance Assessment Rating Tool: 
PATS: Partnership for a Drug Free America's Attitude Tracking Survey: 
PDFA: Partnership for a Drug Free America: 
PME: Performance Measures of Effectiveness: 
YRBSS: Youth Risk Behavior Surveillance System: 

[End of section] 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. However, because 
this work may contain copyrighted images or other material, permission 
from the copyright holder may be necessary if you wish to reproduce 
this material separately. 

United States Government Accountability Office: 
Washington, DC 20548: 

August 25, 2006: 

The Honorable Christopher Bond: 
Chairman: 
The Honorable Patty Murray: 
Ranking Member: 
Subcommittee on Transportation, Treasury, the Judiciary, Housing and 
Urban Development, and Related Agencies: 
Committee on Appropriations: 
United States Senate: 

Congressionally mandated under the Treasury and General Government 
Appropriations Act of 1998,[Footnote 1] the National Youth Anti-Drug 
Media Campaign had the primary goals of preventing the initiation of 
drug use--particularly the use of entry-level drugs marijuana and 
inhalants--among the nation's youth and stopping youth that have begun 
using drugs from continuing their use. Administered through the Office 
of National Drug Control Policy (ONDCP) and implemented in three 
phases, the campaign featured as its centerpiece a paid advertising 
effort in which campaign funds were used to purchase media time and 
space for advertisements that delivered anti-drug messages to the 
campaign's target audience--youth aged 9 to 18 and their parents-- 
through strategic placement of anti-drug advertisements on television 
and radio and in print media. In addition to the advertising, the 
campaign included community outreach, work with the entertainment 
industry to encourage the accurate depiction of the consequences of 
drug use, outreach to faith-based organizations, and work with youth 
organizations. The campaign's first two phases, which ran from January 
1998 through the summer of 1999, were pilot phases that focused 
primarily on informing the planning and development for phase III and 
included a 12-city pilot (phase I) and nationwide advertising (phase 
II). Phases I and II aimed to increase public awareness of anti-drug 
messages. Phase III of the campaign, which began in mid-1999, continued 
the nationwide advertising campaign begun during phase II and 
integrated the advertising with outreach efforts. From fiscal year 1998 
through fiscal year 2006, Congress appropriated over $1.4 billion to 
support the campaign. For fiscal year 2007, the President's budget 
requested $120 million for the campaign, an increase over the fiscal 
year 2006 appropriation, to purchase additional media time and space to 
increase the reach and frequency of the campaign's messages, which 
would restore appropriations to their fiscal year 2005 level. 

Congress first authorized funding for the campaign in fiscal year 1998 
with the expectation that demonstrable changes in youth drug behaviors 
would be apparent within 3 years, and Congress required ONDCP to assess 
whether the campaign's efforts have been effective in changing the drug 
use behaviors of America's youth. ONDCP also indicated that it 
anticipated that it would take 2 to 3 years for the campaign to affect 
drug use behavior, although ONDCP also indicated that it was with the 
implementation of phase III of the campaign, beginning in mid-1999, 
that ONDCP expected to see improvements in anti-drug attitudes that 
would lead to decreases in youth drug use within 3 years. We previously 
reported that ONDCP's evaluations of the first two phases of the 
campaign produced inconclusive results because of various evaluation 
implementation problems and limitations of the analyses used to support 
findings about effects during these pilot phases.[Footnote 2] In 
particular, we noted that the impact evaluations of phases I and II did 
not adequately gauge the overall level of anti-drug awareness generated 
by the campaign--the principal outcome measure for these two phases-- 
and we identified site selection problems, unknown parent response 
rates, low school response rates, and data analysis issues contributing 
to the inconclusive results. 

To implement the phase III evaluation, ONDCP entered into an 
interagency agreement with the National Institute on Drug Abuse (NIDA), 
which in turn awarded contracts to Westat, Inc., through June 2005 for 
$42.7 million to conduct the evaluation. Westat subcontracted with the 
Annenberg School for Communication at the University of Pennsylvania, 
and staff from Westat and Annenberg were coprincipal investigators for 
the study.[Footnote 3] Westat's phase III evaluation covered the period 
from September 1999 through June 2004 and studied the impact of the 
effectiveness of the nationwide campaign in reaching its target 
audience; affecting youth beliefs, attitudes, intentions, and behaviors 
with regard to drug use; and affecting their parents' beliefs and 
attitudes toward drug use and affecting parents' behaviors associated 
with interacting with their children and monitoring their activities. 

Westat evaluated the campaign using a longitudinal panel survey--the 
National Survey of Parents and Youth (NSPY)--that aimed to measure the 
campaign's effectiveness by assessing changes in various outcomes 
within individuals over time in relation to their exposure to campaign 
messages. Westat's evaluation assessed the effect of exposure to the 
campaign on youth drug use and on several key intermediate outcomes-- 
such as youth and parent attitudes toward and beliefs about drug use 
and parental involvement with their children--that were believed to 
influence youth drug use. In conducting the evaluation, Westat 
submitted several interim reports that were used in part to inform 
decisions about the direction of the campaign. Westat submitted a draft 
of the final evaluation report to NIDA in February 2005.[Footnote 4] 

Both Congress and ONDCP recognized the need for a separate evaluation 
of the campaign, because of limitations associated with existing 
national surveys of drug use. Congressional conferees acknowledged 
their intention to rely on the evaluation to gauge the impact of the 
campaign, and also indicated that if the campaign failed to show its 
effectiveness, they would be compelled to reevaluate the use of 
taxpayer money to support it. ONDCP acknowledged that existing national 
surveys of drug use would not be able to answer the critical question 
of whether changes in drug use behavior and attitudes were the result 
of the campaign. These surveys do not ask respondents about their 
exposure and reactions to the messages of the campaign that can then be 
linked to their drug-related attitudes and behavior. For example, in a 
2001 report on youth drug use and the campaign, ONDCP officials noted 
that while national surveys of youth drug use showed flattening or 
declining youth marijuana use in 1999 and 2000 and these trends 
suggested that the campaign may be having the desired impact, it was 
necessary to await the results of the campaign's independent evaluation 
before drawing any definitive conclusions regarding the campaign's 
contribution to changes in youth drug use. 

In a committee report for the fiscal year 2004 appropriations cycle, 
the Senate Appropriations Committee directed us to review how 
consultants were used in support of the media campaign.[Footnote 5] 
This is the second of two reports responding to this mandate.[Footnote 
6] The first report provided information concerning ONDCP's use of 
consultants in the campaign. This second report addresses three 
questions related to Westat's evaluation of phase III of the media 
campaign: (1) How did Westat ensure that it could report credible 
results in its evaluation of the campaign? (2) What did the evaluation 
find about the effect of exposure to the campaign on key intermediate 
outcomes that were intended to lower youth drug use? (3) What did the 
evaluation find about the effect of exposure to the campaign on youth 
drug use? 

In addressing our objectives, a team of GAO social scientists reviewed 
and assessed materials related to Westat's phase III evaluation, 
applying generally accepted social science research standards, 
including such elements as when and how the sample data were collected, 
adjustments made to the sample to address nonresponse, how program 
effects were isolated (i.e., the use of statistical controls), and the 
appropriateness of outcome measures. The materials reviewed included 
interim and final evaluation reports, documentation and analyses 
provided by Westat to us in response to several sets of questions that 
we submitted about the details of its methodology, documentation 
pertaining to meetings of scientific panels that provided guidance on 
the evaluation, and documentation prepared by ONDCP about the design 
and implementation of the campaign. We conducted our work in accordance 
with generally accepted government auditing standards from October 2005 
through June 2006. 

Results in Brief: 

By collecting longitudinal data--i.e., multiple observations on the 
same persons over time--using generally accepted and appropriate 
sampling and analytic techniques, and establishing reliable methods for 
measuring campaign exposure, Westat was able to produce credible 
evidence to support its findings about the relationship between 
exposure to campaign advertisements and both drug use and intermediate 
outcomes. In implementing the study, Westat encountered problems that 
are common to large-scale longitudinal studies, and it addressed those 
using methods that are generally recognized as appropriate approaches 
for the study implementation challenges Westat faced. Challenges that 
Westat encountered were (1) lack of baseline data, which precluded 
Westat from comparing postprogram outcomes to preprogram conditions, 
and the redirection of the campaign; (2) sampling concerns, 
particularly ensuring the coverage of eligible households with youth in 
the targeted age range and ensuring that attrition over successive 
survey cycles did not result in insufficient sample size to detect 
campaign effects or in systematic bias within the sample; (3) 
establishing measures that would allow for both the sufficient 
detection of and the reliable measurement of exposure to the campaign 
on NSPY survey respondents; and (4) disentangling causal effects of 
exposure and drawing meaningful comparisons in the absence of ability 
to employ an experimental design where NSPY respondents would have been 
randomly assigned to various levels of exposure--the generally 
preferred approach for assessing program effects, where possible. Our 
examination of Westat's evaluation report and related documentation 
leads us to conclude that it addressed each of these challenges 
sufficiently to allow it to report credible findings about the effect 
of campaign exposure on drug use and intermediate variables believed to 
be precursors to drug use. Specifically, (1) several factors suggest 
that the lack of baseline data was not fatal to the evaluation's 
findings, and Westat was able to generate statistically significant 
findings related to the redirected campaign; (2) Westat found no 
evidence of bias in the NSPY estimates despite sample coverage losses, 
and it also maintained high follow-up response rates of sampled 
individuals to provide for robust longitudinal analysis; (3) the NSPY 
sample could be used to detect changes in outcomes that were on the 
order of magnitude of changes that ONDCP expected for the campaign, and 
Westat demonstrated that its measures of exposure were valid and could 
reliably predict outcomes, whether results of the associations between 
exposure and outcomes were favorable or unfavorable to the campaign; 
and (4) using sophisticated statistical methods, Westat matched 
respondents on their underlying propensity to be exposed to campaign 
advertisements and, by comparing differences in outcomes among groups 
with different levels of exposure resulting from its matching methods, 
isolated the effects of the campaign from other variables. (See 
appendix I for further details.) 

For intermediate outcome measures thought to influence the ultimate 
target of the campaign, youth drug use--for example, recall and 
identification of campaign messages, youth anti-drug attitudes, and 
parents' beliefs and behaviors--Westat found favorable effects for some 
measures and subgroups, as well as unfavorable effects and no 
significant effects for others. In general, both youth and parents' 
recall of specific campaign messages increased over the life of the 
campaign. In addition, NSPY trend data showed some increasing trends in 
anti-drug attitudes and beliefs as well as the proportion of youth who 
reported never intending to try marijuana. However, cross-sectional and 
longitudinal analysis provided no evidence that these trends resulted 
from campaign exposure. Westat's analysis also indicated that among 
current, non-drug-using youth, exposure to the campaign had unfavorable 
effects on their anti-drug norms and perceptions of other youths' use 
of marijuana--that is, greater exposure to the campaign was associated 
with weaker anti-drug norms and increases in the perceptions that 
others use marijuana. Data for parents in the NSPY on five intermediate 
measures show some favorable effects of campaign exposure on parents' 
behaviors and beliefs. However, for a major aim of the campaign, 
affecting parental behaviors regarding monitoring their children's 
whereabouts, activities, and friends, Westat found no evidence of a 
significant effect. Moreover, where the data showed favorable 
relationships between campaign exposure and parental beliefs and 
behaviors, Westat did not find that these effects on parents ultimately 
lead to corresponding changes in their children's beliefs and 
behaviors. 

Westat's evaluation found no significant favorable effects of campaign 
exposure on marijuana initiation among non-drug-using youth or 
cessation and declining use among prior marijuana users. Westat's NSPY 
data did show some declining trends in self-reported lifetime and past- 
month use of marijuana by youth over the period from 2002 to 2004 and 
declining trends in youth reports of offers to use marijuana. Declining 
drug use trends in the NSPY were consistent with trends in other 
national surveys of drug use over these years. However, Westat 
cautioned that because trends do not account for the relationship 
between campaign exposure and changes in self-reported drug use, trends 
alone should not be taken as definitive evidence that the campaign was 
responsible for the declines. ONDCP has also acknowledged the 
limitation of drug use trends for the purpose of demonstrating a causal 
link between campaign exposures and declines in drug use trends. 
Westat's analysis of the relationship between exposure to campaign 
advertisements and youth self-reported drug use in the NSPY data for 
the entire period covered by its evaluation--assessments that used 
statistical methods to adjust for individual differences and control 
for other factors that could explain changes in self-reported drug use-
-showed no significant effects of exposure to the campaign on 
initiation of marijuana by prior nonusing youth. Westat's analysis 
found significant unfavorable effects--that is, a relationship between 
campaign exposure and higher rates of initiation--during one round of 
NSPY data and for the whole period of the campaign among certain 
subgroups of the sample (e.g., 12 ˝-to 13-year-olds and girls). Westat 
found no effects of campaign exposure on rates of quitting or use by 
prior users of marijuana. 

In light of the fact that the phase III evaluation of the media 
campaign yielded no evidence of a positive outcome in relation to teen 
drug use and congressional conferees' indications of their intentions 
to rely on the Westat study, Congress should consider limiting 
appropriations for the National Youth Anti-Drug Media Campaign 
beginning in the fiscal 2007 budget year until ONDCP is able to provide 
credible evidence of the effectiveness of exposure to the campaign on 
youth drug use outcomes or provide other credible options for a media 
campaign approach. In this regard we believe that an independent 
evaluation of the new campaign should be considered as a means to help 
inform both ONDCP and Congressional decision making. 

We provided a draft of this report to the Director of ONDCP for 
comment. In response, ONDCP provided written comments (reproduced in 
appendix II), which stated that ONDCP was puzzled that we did not make 
recommendations to it about how to improve the campaign. However, the 
main purpose of our report was to assess Westat's evaluation rather 
than to comment on how to improve the media campaign. In so doing, we 
focused on Westat's methods. Our role was to inform Congress about the 
reliability of Westat's evaluation so that Congress could decide the 
extent to which it will continue to fund the campaign. 

ONDCP expressed a number of concerns about our assessment of Westat's 
evaluation and its implications concerning the effectiveness of the 
campaign. Most importantly, it stated that the Westat study is ill 
suited to assess impact and the study's findings are of limited 
relevance. Our extensive review of the Westat study does not support 
ONDCP's conclusion. Westat successfully addressed implementation 
challenges and used sophisticated analytic techniques to develop its 
findings. Another major issue ONDCP presents in its comments deals with 
the fact that the campaign has made major changes since the Westat data 
collection, rendering the study's findings irrelevant. Neither we nor 
ONDCP has factual data upon which to base an assessment of the 
effectiveness of the current campaign. However, other major efforts to 
substantially change the campaign during the time frame of the Westat 
data collection did not yield positive results. ONDCP raised a number 
of other issues that are generally related to the issues discussed 
above. These are addressed in the Agency Comments and Our Evaluation 
section of this report. 

Background: 

The National Youth Anti-Drug Media Campaign: 

As part of the Treasury and General Government Appropriation Act of 
1998,[Footnote 7] the Drug Free Media Campaign Act of 1998 required, 
among other things, the Office of National Drug Control Policy to 
conduct a national media campaign for the purpose of reducing and 
preventing drug abuse among young people in the United States.[Footnote 
8] The National Youth Anti-Drug Media Campaign may be the most visible 
federal effort devoted to preventing drug use among the nation's youth. 
It aims to educate and enable America's youth to reject illegal drugs; 
to prevent youth from initiating use of drugs, especially marijuana and 
inhalants; and to convince occasional users of these and other drugs to 
stop using drugs. Administered by ONDCP, and implemented in three 
phases, the campaign has as its centerpiece a paid advertising effort 
in which campaign funds were used to purchase media time and space for 
advertisements that delivered anti-drug messages to the campaign's 
target audiences--youth aged 9 to 18 and their parents and adult 
caregivers--through strategic placement of anti-drug advertisements on 
television and radio and in print media. 

The campaign's first two phases were pilot phases that had as their 
objectives developing advertising concepts, creating limited 
advertisements, testing public awareness of the advertisements in 12 
metropolitan areas, and eventually extending the pilot program 
nationwide. Phase III of the campaign, which began in mid-1999, 
continued the nationwide advertising campaign begun during phase II and 
integrated the advertising with outreach efforts. In addition to the 
advertising, the fully integrated phase III campaign included community 
outreach, work with the entertainment and media industries to encourage 
the accurate depiction of the consequences of drug use, outreach to 
faith-based organizations, and work with youth organizations. 

During phase III, ONDCP had overall responsibility for developing and 
implementing the campaign, and to do so, it enlisted the support of 
nonprofit organizations, trade associations, private businesses, and 
federal agencies. Appropriated media campaign funds were to be used to 
cover the costs of actually making the advertisements as well as the 
costs for planning of purchase of media time and space. The campaign 
also included public outreach and specialized communications efforts. 
The purpose of public outreach and communications was to extend the 
reach and influence of the campaign through nonadvertising forms of 
marketing communications. Examples of these nonadvertising forms of 
communication included submitting articles related to key campaign 
messages such as effective parenting or the effects of marijuana on 
teen health to newspapers and magazines; building partnerships and 
alliances, for example, coordinating positive activities for teens with 
local schools and community groups; creating Web sites and exploring 
alternative media approaches; and entertainment industry outreach. 

According to the campaign's communications strategy, youth aged 9 to 18 
were segmented into three school and age risk-level categories: late 
elementary school adolescents, aged 9 to 11; middle school children, 
aged 11 to 13; high school youth, ages 14 to 18. The campaign 
originally targeted youth aged 9 to 18 with a focus on middle school 
age adolescents (roughly 11-to 13-year-olds); its secondary focus was 
on high school-aged youth (approximately 14 to 18 years of age). In 
2001, the campaign shifted its creative focus to 11-to 14-year-olds in 
order to more effectively reach youth at the time they are most at risk 
for trying drugs. In 2002, the campaign altered its target age group to 
focus primarily on 14-to 16-year-olds. For all age groups, the 
communications strategy identified the primary focus of the campaign as 
at-risk nonusers and occasional users of drugs. For all groups, it was 
designed to give consideration to differences arising from gender, 
race, ethnicity, and regional and population density factors. 

From fiscal year 1998 through fiscal year 2004, Congress appropriated 
$1.225 billion to support the campaign (table 1). 

Table 1: Appropriations for the National Youth Anti-Drug Media 
Campaign, Fiscal Years 1998 through 2006: 

Fiscal year: 1998; 
Final appropriations (in millions of dollars): $195. 

Fiscal year: 1999; 
Final appropriations (in millions of dollars): $185. 

Fiscal year: 2000; 
Final appropriations (in millions of dollars): $185. 

Fiscal year: 2001; 
Final appropriations (in millions of dollars): $185. 

Fiscal year: 2002; 
Final appropriations (in millions of dollars): $180. 

Fiscal year: 2003; 
Final appropriations (in millions of dollars): $150. 

Fiscal year: 2004; 
Final appropriations (in millions of dollars): $145. 

Fiscal year: 2005; 
Final appropriations (in millions of dollars): $120. 

Fiscal year: 2006; 
Final appropriations (in millions of dollars): $100. 

Fiscal year: Total, 1998 through 2006; 
Final appropriations (in millions of dollars): $1,445. 

Fiscal year: Total, 1998 through 2005; 
Final appropriations (in millions of dollars): $1,345. 

Fiscal year: Total, 1998 through 2004; 
Final appropriations (in millions of dollars): $1,225. 

Sources: Appropriations Acts from various years. 

Note: Appropriated amounts are prerescission amounts. For example, 
rescissions were 0.38 percent in fiscal year 2000, 0.22 percent in 
fiscal year 2001, 0.65 percent in fiscal year 2003, and 0.59 percent in 
fiscal year 2004. 

[End of table] 

For fiscal year 2007, the President's budget requested $120 million for 
campaign activities. The 2007 request represents an increase of $21 
million above the fiscal year 2006 budget authority. The additional 
resources were requested to help to purchase additional media time and 
space to increase the reach and frequency of the campaign's messages. 

Planning and the Underlying Logic of the Campaign: 

According to ONDCP, its planning for the campaign's communications 
strategy included reviews of published studies on the etiology and 
prevention of adolescent drug use, drug prevention campaigns, other 
public health campaigns, and general consumer marketing campaigns 
targeting youth and their parents. ONDCP also supplemented its research 
evidence with an extensive expert consultation process that included 
input from over 200 experts in academia, civic and community 
organizations, government agencies, and the private sector. A campaign 
design expert panel that included experts in the fields of drug use and 
prevention, public health communication, advertising, market research, 
consumer marketing, and public policy met over a 4-day period during 
the fall of 1997 and played a key role in integrating diverse sources 
of information and guiding the development of the communications 
strategy for the campaign. 

The planning process resulted in a statement of ONDCP's communications 
strategy for the campaign, which described the premises of the 
campaign. Among these were the following: First, that the media can 
influence people in a variety of ways, such as informing and alerting 
them to important developments and shaping subsequent actions; 
satisfying leisure time needs, thereby influencing individuals' views 
and beliefs about the world; and stimulating interest in commercial 
goods and services, thereby influencing where and how people shop. 
Second, that media messages have more potential to reinforce rather 
than to alter existing attitudes and beliefs. Third, to the extent that 
youth attitudes, beliefs, and intentions toward drug use vary with 
their age, the potential of a media campaign to influence drug use may 
be directly related to the age of the youth. Fourth, the campaign had 
to be sustained over time and to have a significant media presence, and 
its central messages have to be repeated often and in a variety of 
ways. Citing research showing that attitudinal and behavioral change 
took time to occur, ONDCP reported that it expected to observe 
"improvements in anti-drug attitudes that would lead to decreases in 
youth drug use within three years" of the implementation of phase III 
of the campaign. Fifth, as parents and adult caregivers play a vital 
role in youth drug use behaviors, and by also targeting parents, the 
campaign would aim to affect the nature of their interaction with their 
children, thereby strengthening their children's capacity to resist 
using illicit drugs. 

The campaign focused on primary prevention--that is, preventing those 
who did not use drugs from starting to use drugs. According to ONDCP, a 
media campaign that focused on primary prevention targets the 
underlying causes of drug use and therefore has the greatest potential 
to reduce the scope of the problem over the long term. Further, a 
primary prevention campaign also has greater potential to affirm and 
reinforce anti-drug attitudes of nonusers than to persuade experienced 
users to change their behaviors, and a primary prevention campaign 
would also, over time, lessen the need for drug treatment services. 
With a focus on young, non-drug-using adolescents, an expectation 
underlying the campaign's potential success was that as these young, 
non-drug-using adolescents aged, the campaign's messages would 
intervene, retard the development of more pro-drug attitudes, and 
enable adolescents to continue to maintain their preexisting anti-drug 
attitudes. By maintaining these attitudes, or preventing the 
development of pro-drug sentiments, the campaign would affect drug use 
rates by lowering the rate at which youth initiated drug use, 
particularly the use of marijuana or inhalants. 

The campaign was designed to have a significant and sustained media 
presence. During planning, ONDCP acknowledged that the campaign would 
have to be sustained for a period of time sufficient to bring about a 
measurable change in the beliefs and behaviors of youth in the target 
audience. On the basis of the experiences of successful social 
marketing campaigns, ONDCP reported that it expected that changes in 
awareness or recall of the campaign would be detectable within a few 
months of the start of the campaign, that changes in perceptions and 
attitudes would be detectable within 1 to 2 years of the start of the 
campaign, and that changes in behavior would be detectable within 2 to 
3 years. 

Campaign Activities during Phase III: 

From mid-1999, the start of phase III, through June 2004, the end of 
the phase III evaluation, campaign activities included extensive media 
dissemination of campaign messages to a national audience of youth and 
parents; an interactive media component, which involved using content- 
based Web sites and Internet advertising; use of experienced 
individuals and organizations with expertise in marketing to teens, 
advertising and communications, behavior change, and drug prevention to 
inform the campaign strategy and implementation; use of multicultural 
initiatives that focused on sufficiently exposing campaign messages to 
African Americans, Asian Americans, Pacific Islanders, Hispanic 
Americans, American Indians, and Alaskan Natives; and the 
implementation of the integrated social marketing and public health 
communications campaign through the creation of partnerships with 
civic, professional, and community groups and outreach to media, 
entertainment, and sports industries. Through the partner 
organizations, the campaign attempted to strengthen local anti-drug 
efforts, and through outreach, it encouraged the news media to run 
articles that conveyed campaign messages. Youth and parent exposure to 
campaign messages could come from the direct, paid and donated 
advertising or from content delivered by news media and entertainment 
industries through the outreach efforts. Additional opportunities for 
exposure to anti-drug messages could be enhanced through personal 
involvement with organizations that became partners as a result of 
campaign outreach or by interaction with the campaign's Web site. 
Further, youth exposure to anti-drug messages could also occur through 
interactions with friends, peers, parents, or other adults that 
occurred directly from either campaign ads or outreach efforts. 

Campaign Themes and Messages: 

Campaign messages for both youth and their parents and caregivers were 
to focus on common transitions--such as the transition from elementary 
to secondary school--and common situations--such as the amount of time 
spent in settings without adult supervision--that were believed to 
heighten adolescents' vulnerability to drug use initiation. In 
addition, messages were to focus on altering mediating variables--such 
as beliefs and intentions--that were known to have a significant impact 
on adolescent drug use. Finally, campaign messages were designed to 
create a "brand identity" in the minds of target audience members and 
through brand identity position campaign messages as credible and 
important. Throughout phase III, themes such as parents as "The Anti- 
Drug" and the "My Anti-Drug" theme for youth were designed to promote 
identification and positive associations with the campaign's messages. 

While they evolved throughout the campaign, the central strategic 
messages or themes for youth focused on resistance skills and self- 
efficacy to refuse drugs, normative education and positive messages, 
negative consequences of drug use, and early intervention. Resistance 
skills and self-efficacy advertisements were designed to enhance the 
personal and social skills of youth that promote lifestyle choices and 
to help build youth's confidence that they could resist drugs. 
Normative education themes attempted to instill the beliefs that most 
young people do not use drugs or convey messages that "cool people 
don't use drugs," while positive message themes reinforced the idea of 
positive uses of time as alternatives to illicit drugs. Negative 
consequences themes aimed to enhance youth perceptions that drug use is 
likely to lead to a variety of negatively valued consequences, such as 
loss of parental approval, reduced performance in school, and negative 
social, aspirational, and health effects. Negative consequences themes 
were the primary focus of the Marijuana Initiative, which was 
introduced during 2002. An early intervention theme sought to motivate 
youth to intervene with friends who they perceived as having problems 
with drugs or alcohol and tried to convince youth of their ability to 
take action and to give them the tools and skills they needed to 
intervene. 

For parents, the campaign's themes included messages that every child, 
including their own, was at risk of doing drugs; that they can learn 
parenting skills to help them help their children avoid drugs; that 
they need to be aware of the harmful effects of drugs including 
marijuana and inhalants; and, as part of the Early Intervention 
Initiative, that it was important that they intervene at the earliest 
possible opportunity in their child's life if their child was using 
drugs or alcohol. 

Design of the Evaluation, Interim Evaluation Reports, and Redirection 
of the Campaign: 

ONDCP recognized the need for a separate evaluation of the campaign and 
for ongoing reporting of evaluation results. The need for a separate 
evaluation stemmed in part from the limitations of existing national 
surveys that monitor drug use, such as Monitoring the Future, which 
provides data on drug use by high school students, the National 
Household Survey on Drug Abuse,[Footnote 9] and the Youth Risk Behavior 
Survey, which addresses health risk behaviors including drug use. These 
recurring surveys provide very little information with which to 
evaluate the impact of the campaign, because they were not designed to 
evaluate it. As ONDCP has written, these surveys contain no questions 
about target audience exposure and response to the campaign, and as a 
result, any changes in attitudes, beliefs, and behaviors toward drug 
use could not be associated directly with the campaign. By comparison, 
ONDCP acknowledged that it was using the Westat evaluation to assess 
the extent to which changes in anti-drug attitudes and beliefs or drug- 
using behavior can be attributed to the campaign, as opposed to other 
socioeconomic factors. In addition, ONDCP indicated that for the 
campaign, data from Westat's evaluation would enable ONDCP to assess 
whether the campaign is working. 

The primary tool of the Westat evaluation was the National Survey of 
Parents and Youth. The NSPY is a longitudinal panel study of children 
and their parents' exposure and response to the campaign. The NSPY was 
designed to collect initial and follow-up data from nationally 
representative samples of youth aged 9 to 18 and from the parents of 
these youth. The sample was designed to represent youth living in homes 
in the United States and their parents. Data collection began in 
November 1999 and was conducted over four rounds--each of which was 
about 1 year apart from the next round--in nine waves of interviews. An 
interview wave refers to the fielding of a survey round to a specific 
subsample in the NSPY. An interview round refers to the completion of 
interviews with the entire sample. Data for each of the nine waves were 
collected using a laptop computer and a combination of computer- 
assisted interview technologies. To collect sensitive data, audio 
computer-assisted self-interview technology was used, allowing 
respondents to self-administer the questionnaire in total privacy. The 
final wave of data collection was completed in June 2004 (fig. 1). 
Eligible youth and parents were to be interviewed four times. 

Figure 1: Data Collection Rounds and Waves of the NSPY: 

[See PDF for image] 

Source: Adopted from Westat, 2005, Vol. 2: Appendices. 

[End of figure] 

The evaluation aimed to assess whether exposure to the campaign 
affected the self-reported knowledge, attitudes, beliefs, and drug use 
of youth. Because the campaign reached out to all youth nationwide, the 
evaluators could not assess its effects using experimental methods, in 
which some subjects are randomly assigned to the intervention and 
others are randomly assigned to control groups that were not exposed to 
the intervention. Westat's evaluation was designed to take into account 
the variation in self-reported exposure to the campaign messages and to 
assess how this variation in exposure was correlated with outcomes that 
the campaign intended to affect. To attribute changes in drug use 
attitudes and behaviors to the campaign, the evaluation was designed to 
assess exposure to the campaign and to compare differences in outcomes 
for groups of persons that were exposed to varying levels of the 
campaign's messages, and to use statistical controls to account for 
individual-level differences among survey respondents. 

Westat's evaluation assessed youth self-reported drug use and 
intermediate outcomes--such as youth and parent attitudes and beliefs 
toward drug use and parental involvement with their children--that were 
believed to influence youth drug use. The evaluation of phase III 
addressed issues related to (1) whether the campaign was reaching its 
target populations, (2) whether the desired outcomes moved in favorable 
or unfavorable directions, (3) whether the campaign was influencing 
changes in the desired outcomes, and (4) what could be learned from the 
overall evaluation to support ongoing decision making for the campaign. 
These issues led to the five major objectives for the evaluation: 

* to measure changes in drug-related knowledge, attitudes, beliefs, and 
behavior in youth and their parents; 

* to assess the relationship between changes in drug-related knowledge, 
attitudes, beliefs, and behavior and self-reported measures of media 
exposure, including the salience of the measures; 

* to assess the association between parents' drug-related knowledge, 
attitudes, beliefs, and behavior and those of their children; 

* to assess changes in the association between parents' drug-related 
knowledge, attitudes, beliefs, and behavior and those of their children 
that may be related to the campaign; and: 

* to compare groups of people with high exposure to other groups with 
low exposure. 

Westat submitted semiannual and special topic reports to NIDA, as the 
findings from these interim evaluation reports were to be used to 
support ongoing decision making for the campaign. Westat submitted the 
first semiannual report in November 2000. By December 2003, Westat had 
submitted six additional reports, four of which were labeled as 
semiannual reports, and the other two included a special report on 
historical trends in drug use and a 2003 report of findings.[Footnote 
10] Westat submitted its first draft of its final report to NIDA in 
February 2005. 

In addition to Westat's evaluation of the relationship between exposure 
and outcomes, Westat also prepared a report on the environmental 
context of the campaign.[Footnote 11] In May 2002, Westat reported 
findings from this qualitative study of views of representatives from 
major national organizations and state prevention coordinators about 
the messages conveyed by the campaign and the role of the campaign as 
an organizing partner in helping to bolster local substance abuse 
prevention efforts. According to Westat, representatives felt that the 
campaign's messages reinforced their own messages that encouraged youth 
to find healthy alternatives to drug use and to raise public awareness 
of the issue of illicit drugs among youth. Westat also reported that 
representatives were less enthusiastic about the role of the campaign 
as an organizational partner in helping with local substance abuse 
prevention efforts. 

In November 2002, Westat submitted its fifth semiannual report to NIDA. 
In it, Westat reported that it found little evidence that the campaign 
had direct, favorable effects on youth self-reported drug use between 
2000 and 2002. Specifically, Westat reported: 

"There is little evidence of direct favorable Campaign effects on 
youth. There is no statistically significant decline in marijuana use 
to date, and some evidence for an increase in use from 2000 to 2001. 
Nor are there improvements in beliefs and attitudes about marijuana use 
between 2000 and the first half of 2002. Contrarily, there are some 
unfavorable trends in youth anti-marijuana beliefs. Also there is no 
tendency for those reporting more exposure to Campaign messages to hold 
more desirable beliefs."[Footnote 12] 

Westat further reported that there were unfavorable delayed effects of 
campaign exposure on subsequent intentions to use marijuana and on 
other beliefs. By delayed effects, Westat referred to the relationship 
between exposure to the campaign measured in one survey round having an 
effect on intentions or beliefs outcomes at a subsequent survey round. 
For parents, Westat reported that the evidence was consistent with 
favorable campaign effects, as it found that there were favorable 
changes for three of five parents' belief and behavior outcome 
measures. However, Westat also reported that it found no evidence for 
favorable indirect effects on youth behavior as the result of their 
parents' exposure to the campaign. 

Congressional appropriators expressed concerns about the findings of 
Westat's fifth semiannual report. In the conference report for fiscal 
year 2003 omnibus appropriations, the conferees reported that they were 
"deeply disturbed by the lack of evidence that the National Youth Anti- 
Drug Media Campaign has had any appreciable impact on youth drug 
use."[Footnote 13] The conferees further acknowledged that while the 
evaluation conducted under NIDA's auspices showed "slight and sporadic 
impact on the attitudes of parents, it has had no significant impact on 
youth behavior." The conferees further acknowledged that while other 
surveys of youth drug use--such as Monitoring the Future, a survey of 
high school youth--showed recent declines in drug use, "the NIDA study 
was undertaken to measure the specific impact of the Media Campaign, 
not simply to gauge general trends," and the conferees stated that they 
"intend to rely on the scientifically rigorous NIDA study to gauge the 
ultimate impact of the campaign" and to reevaluate the use of taxpayer 
money to support the campaign if the campaign continued to fail to 
demonstrate its effectiveness. 

In 2002, the strategy for the campaign was redirected. In the spring, 
the target age group of the campaign became 14-to 16-year-olds--youth 
who have higher rates of marijuana initiation than younger youth--from 
its original targeting of 11-to 13-year-olds. The shift to teens in the 
14-to 16-year-old range aimed to allow the campaign to more effectively 
reach youth during the time at which they are most at risk for trying 
drugs. ONDCP also required more rigorous copy test procedures of all 
television advertisements before they were aired, and ONDCP increased 
its oversight in guiding the development and production of 
advertisements. In October 2002, ONDCP launched a new initiative called 
the Marijuana Initiative. This initiative contained more focused 
advertising to address youth marijuana use. In a hearing before the 
House Committee on Government Reform, ONDCP announced that it would 
reverse the ratio of campaign advertising expenditures directed to 
adults and youth, respectively. Previously, about 60 percent of 
expenditures were directed to adults and 40 percent toward youth. 
Finally, during February 2004, it expanded the campaign's 
communications goals to include the Early Intervention Initiative. This 
intervention was targeted toward both parents and teen friends, and 
ONDCP intended to use parental and peer pressure to stop drug and 
alcohol use among teens. 

Assessment of the Campaign by the Office of Management and Budget and 
ONDCP's Current Approach: 

To strengthen the linkages between resources and performance envisioned 
in the Government Performance and Results Act of 1993 (GPRA),[Footnote 
14] the Office of Management and Budget (OMB) developed the Program 
Assessment Rating Tool (PART) to bring performance information into the 
executive budget formulation process. PART is designed to determine the 
strengths and weaknesses of federal programs by drawing upon available 
program performance and evaluation data so that the federal government 
can achieve better results. The PART therefore looks at factors that 
affect and reflect program performance, including program purpose and 
design; performance measurement, evaluations, and strategic planning; 
program management; and program results. Because the PART includes a 
consistent series of analytical questions, it allows programs to show 
improvements over time and allows comparisons between similar programs. 

OMB's PART rating of the campaign addressed issues related to its 
purpose and design, strategic planning, program management, and program 
results and accountability. OMB indicated that the purpose was clear-- 
giving ONDCP a 100 percent score on this factor--and it rated the 
campaign's planning and management with scores of 67 percent and 70 
percent, respectively. In its assessment of ONDCP's strategic planning, 
OMB noted that in response to its 2002 PART review, ONDCP revised the 
campaign's logic model and significantly changed its long-term and 
annual performance measures. 

However, OMB's assessment rating for the campaign was "results not 
demonstrated." OMB indicated that its assessment of the campaign's 
progress toward both the long-term goals and annual performance goals 
will be reviewed against the results of the NIDA-managed evaluation. 
OMB noted that while there is no federal program closely comparable to 
the campaign, evaluations of other health behavior change efforts found 
short-term effects after exposure to media. While acknowledging that a 
final assessment of the effects of the campaign awaited the final 
report from the NIDA-managed evaluation, OMB also indicated that 
"outcome data from the evaluation suggest little or no direct positive 
effect on youth behavior and attitudes attributable to the campaign to 
date. Perhaps some positive effect on parental attitudes/behavior but 
that has not yet translated into an effect on youth." 

ONDCP has credited the campaign, along with a variety of collective 
prevention efforts, with contributing to "significant success in 
reducing teen drug use, as evidenced by the 19 percent decline from 
2001 to 2005." It has introduced a new youth brand approach to connect 
youth with aspiration themes. ONDCP also has indicated that while it 
awaits our formal assessment of the evaluation, that it will use 
existing national surveys to evaluate the campaign and suspend its 
request for proposals for a new evaluation contract. Specifically, 
ONDCP indicated that it would use the MTF survey to track improvements 
in perception of the risk of drug use--a predictor of lower drug use by 
youth--and it would use a special analysis of the PATS survey--the 
Partnership for a Drug-Free America's Attitude Tracking Survey--data on 
anti-drug messages. According to the 2005 data from MTF, there were no 
significant 1-year declines in marijuana use for youth in any grade 
levels, and while gradual declines in the upper grades continued, 
declines halted for youth in the 8th grade. Additionally, for 8th 
graders, perceived risk of marijuana use held steady, while for youth 
in 10th and 12th grade, there was an increase in perceived risk of 
marijuana use. 

Recent Research on the Effects of the Campaign in Local Settings: 

Two recently released studies have reported that exposure to the 
campaign was associated with changes in past-month marijuana use under 
certain conditions for certain groups of students exposed to the 
campaign. In one of the studies,[Footnote 15] 45 South Dakota high 
schools and their middle-school feeder(s) were randomly assigned to 
three groups: (1) a basic prevention curriculum, (2) a group given this 
curriculum with booster lessons, and (3) a control group. All schools 
were exposed to the campaign during the fall of 1999 and spring of 
2000. This permitted the researchers to test for a synergistic effect 
between exposure to the campaign's anti-drug messages and participation 
in the school-based drug prevention curriculum. The sample of about 
4,100 youth were asked how often they had seen anti-drug advertisements 
in recent months in five media outlets that were used by the campaign, 
and the researchers measured exposure to the campaign that indicated 
whether or not the adolescents reported seeing ads at least one to 
three times per week in any of the five media outlets. Consistent with 
Westat's fifth interim report, the evaluation of the South Dakota drug 
prevention curriculum found no direct effects of exposure to the 
campaign on its sample of adolescents' use of illegal drugs. However, 
the evaluation also found that marijuana use in the past month was 
significantly less likely among adolescents who received both the 
curriculum with booster lessons and weekly exposure to the campaign's 
messages. In other words, neither the enhanced curriculum nor the 
campaign alone had a substantial effect on marijuana use in the absence 
of the other. In addition, this evaluation's measure of exposure was 
based on weekly exposure, suggesting that the synergistic effect of the 
campaign observed in these South Dakota schools was based on the 
delivery of repeated messages. 

The second study used monthly random samples of 100 youth from the 
enrollment lists of 4th to 8th graders in the public schools in the 
spring of 1999 in two moderate-sized communities--Fayette County 
(Lexington), Kentucky, and Knox County (Knoxville), Tennessee--over 48 
months from April 1, 1999, through March 31, 2003.[Footnote 16] The 
study period included advertisements aired under the campaign's 
Marijuana Initiative. Students in the samples aged over time and were 
13 to 17 years of age at the beginning of the Marijuana Initiative. 
Youth in the samples were measured on marijuana use during the past 30 
days, as well as on their attitudes toward marijuana. Exposure to 
television and radio advertisements was measured by self-reported past- 
month exposure. The study found that among high-sensation-seeking 
youth--that is, youth who desire novel, complex, and intense sensations 
and experiences and who are willing to take social risks to obtain 
them--exposure to the first 6 months of the campaign's Marijuana 
Initiative led to reductions in marijuana use. The study's authors 
reasoned that the effects that they found for the Marijuana Initiative 
were consistent with an approach termed SENTAR (for sensation-seeking 
targeting), in which high-sensation-seeking youth are targeted with 
high sensation value messages to prevent risky behaviors. 

Westat's Evaluation Design, Use of Generally Accepted and Appropriate 
Sampling and Analytic Techniques, and Reliable Methods for Measuring 
Campaign Exposure Produced Credible Evidence to Support Its Findings: 

Westat was able to produce credible evidence to support its findings 
about the relationship between exposure to campaign advertisements and 
both drug use and intermediate outcomes by employing a longitudinal 
panel design--i.e., collecting multiple observations on the same 
persons over time--using generally accepted and appropriate sampling 
and analytic techniques and establishing reliable and sufficiently 
powerful measures of campaign exposure. Westat encountered various 
challenges and threats to validity that are commonly associated with 
large-scale longitudinal studies, including lack of an opportunity to 
use experimental methods, lack of baseline data, and changes in 
campaign focus that were not timed with data collection; issues with 
ensuring adequate sample coverage and controlling for sample attrition 
over time; establishing measures that were sufficient to detect and 
reliably measure campaign effects; and disentangling causal effects 
without being able to employ an experimental design where subjects 
would have been randomly assigned to different levels of exposure. Our 
review of Westat's evaluation report and associated documentation leads 
us to conclude that the design and methodology used in its evaluation 
responded appropriately to these challenges, resulting in credible 
findings. 

Although Elements of the Campaign Limited Choices of Evaluation Designs 
and Affected Data Collection, Westat's Design Was Rigorous and Provided 
a Means to Test for Campaign Effects: 

The nationwide scope of the campaign precluded Westat from using 
experimental methods or obtaining baseline data, and the timing of the 
introduction of some new campaign initiatives limited some of the data 
available to evaluate them. However, Westat's longitudinal panel survey 
design provided a framework for developing strong evidence of within- 
respondent changes in outcomes over time as a result of exposure to the 
campaign. The consensus of a scientific panel convened by NIDA in 
August 2002 to review the evaluation was that Westat's use of a 
national probability sample to study change arising from the campaign 
was preferable as the "gold standard" to a study based on other 
alternatives, such as in-depth community-based studies of the 
mechanisms of change and campaign effects. Additionally, the 
theoretical underpinnings of behavioral change through advertising, 
along with statistically significant outcomes in some but not all 
groups, suggest that the absence of baseline data and introduction of 
new campaign initiatives did not invalidate the evaluation's findings. 
Finally, despite the introduction of new campaign initiatives that were 
not timed with data collection cycles, Westat was able to assess change 
in the NSPY data and generate statistically significant findings using 
these data. 

Westat's longitudinal panel design was based on the premise that 
effects of exposure to the campaign on outcomes could be measured and 
detected within individuals over time, after controlling for various 
other factors that could have influenced outcomes. The design called 
for measuring the same respondents up to four times to assess how the 
natural variation in exposure to the campaign correlated with campaign 
outcomes. Westat's approach--an exposure (or dose)-response model--is 
based upon a premise that respondents' recall of advertisements 
(exposure or dose) is related to outcomes (response). In two recent 
studies of the effects of the campaign on specific groups of youth in 
local areas, an exposure-response approach has been shown to be an 
effective method for detecting effects of the campaign in reducing 
youth drug use in local settings. One of the studies reported a 
synergistic effect of exposure to the campaign and a classroom-based 
drug prevention curriculum among 9th grade students in 45 South Dakota 
high schools. The other study reported reductions in drug use during 
the period of the redirected campaign among high-sensation-seeking 
youth in schools in Knoxville, Tennessee, and Lexington, Kentucky. To 
assess the possibility of preexisting differences between groups of 
exposed youth and parents that might explain both the variation in 
exposure to the campaign and variation in outcomes, Westat included in 
the NSPY structured interview many questions on personal and family 
history, and it used the responses to control statistically for 
differences in attributes of respondents in order to attempt to isolate 
the relationship between exposure to the campaign and outcomes. 

The absence of baseline data--that is, precampaign data on outcomes-- 
was beyond Westat's control, as phase III of the campaign began before 
the first wave of data collection for the phase III evaluation began. 
The lag between the start of phase III of the campaign in mid-1999 and 
the completion of the evaluation's first round of data collection-- 
around mid-2001--leaves open the possibility that there were effects of 
the campaign that occurred very early on in the campaign, prior to when 
Westat began data collection. Several factors suggest that the absence 
of pre-phase III baseline data was not fatal to the evaluation's 
findings. First, if there were effects of the campaign that could not 
be detected because of the absence pre-phase III baseline data, those 
effects must have occurred very rapidly and then endured throughout the 
remainder of the campaign, from 1999 through 2004. However, rapid 
changes in youth drug use were not observed in MTF data; rather, the 
overall trend in MTF past year drug use was flat between 1998 and 1999. 
Second, rapidly occurring effects were not expected by ONDCP in 
designing the campaign. As we reported in 2000, and as ONDCP wrote in 
2001, ONDCP believed that it would take 2 to 3 years for changes in 
drug use to be evident as a result of the campaign.[Footnote 17] 

Another campaign design factor that affected Westat's evaluation was 
the implementation of new campaign initiatives, such as the Marijuana 
Initiative, which were implemented at times that officials at ONDCP 
considered to be important, and therefore they may not have coincided 
with planned data collection for the evaluation, nor should they 
necessarily have done so. For example, the Marijuana Initiative was 
implemented in October 2002, and the NSPY data available to evaluate 
outcomes during it were limited to three complete survey waves. For its 
longitudinal analysis of change during the Marijuana Initiative, Westat 
was limited to data from two survey waves. Despite these limitations, 
the evaluation produced data that enabled Westat to detect effects 
during the period of the Marijuana Initiative. 

Sample Coverage Issues Did Not Invalidate Westat's Assessment of the 
Effectiveness of Exposure to the Campaign on Intermediate and Drug Use 
Outcomes: 

During the enrollment phase of the NSPY, Westat experienced sample 
coverage problems, in that it enrolled--or rostered--a smaller 
percentage of households with youth in the targeted age range than 
would be expected based on comparable Current Population Survey (CPS) 
estimates--the data that Westat used to develop its expectations about 
the percentage of households having youth in the targeted age ranges. 
Coverage refers to the extent to which a sample is representative of 
the intended population on specified characteristics, and it is 
important because the omission of segments of the intended population 
from a sample--or undercoverage--can lead to biased results, in that 
omitted segments may differ in some important respect from those 
segments included. Westat estimated the extent of undercoverage in the 
NSPY to be about 30 percent as compared to the CPS estimates, and 
according to Westat and NIDA, the undercoverage arose during the stage 
of sampling in which Westat was developing rosters of households that 
were believed to contain youth in the target age range. At this stage, 
the survey rostering process required entry into the household, which 
may have led respondents in potentially eligible households to refuse 
to participate. 

Our review of Westat's documentation leads us to conclude that there 
was no evidence of biased results due to undercoverage and that the 
sample was sufficiently reliable both for the purposes of estimating 
changes over time in individual outcomes and for assessing the 
effectiveness of exposure to the campaign on outcomes. Westat's 
comparisons of the estimated population characteristics of the NSPY-- 
such as race and ethnicity of head of household and race and ethnicity 
of youth in households--with the estimated population characteristics 
from the CPS show that they are generally similar. That is, the 
distributions of characteristics of eligible households with youth 
included in the NSPY were broadly consistent with a variety of 
corresponding distributions from the 1999 CPS. These comparisons 
suggest that the NSPY estimated population by race and ethnicity was 
similar to that of CPS. Westat also used multivariate modeling 
techniques to develop weighting adjustments, and it developed weights 
to adjust its sample for nonresponse that were reasonably effective in 
reducing nonresponse bias. 

An additional test for bias in a sample is to compare estimates derived 
from it with estimates on the same variable derived from another 
sample. If the NSPY results were biased, then one would expect that 
estimates derived from it would differ from estimates derived from 
unbiased samples. For example, if eligible households refused to 
participate in the NSPY because they contained teens with drug issues 
and as a result avoided participation at a higher rate than did 
households containing teens without drug issues, then these higher 
refusal rates by households containing teens with drug issues would 
lead to NSPY estimates of the percentage of youth reporting that they 
used drugs that were lower than those obtained from other, comparable 
national surveys. According to data provided by NIDA officials and our 
review of Westat's final report, estimated self-reported drug use rates 
from the NSPY are comparable to estimates derived from other major 
surveys of drug use, such as the National Survey on Drug Use and 
Health. For example, in the NSPY, rates of past-month marijuana use 
among 12-to 18-year-olds were 7.2 percent in 2000, 8 percent in 2001, 
8.9 percent in 2002, and 7.9 percent in 2003. These rates were similar 
to those reported for 12-to 17-year-olds in the National Survey on Drug 
Use and Health (NSDUH) of 7.2 percent in 2000, 8 percent in 2001, 8.2 
percent in 2002, and 7.9 percent in 2003. If youth with known drug use 
problems consistently opted out of both the NSPY and the NSDUH--a 
hypothesis that is not testable with the available data--then the 
estimates from both the NSPY and the NSDUH of the true prevalence of 
youth drug use would be biased underestimates. 

Sample Attrition across NSPY Interview Rounds Was Sufficiently Low to 
Allow for Reliable Assessments of the Effect of Campaign Exposure on 
Outcomes: 

As the NSPY was a longitudinal survey--in which eligible sample 
respondents were re-interviewed up to three times after their 
enrollment interviews--attrition was a concern with which Westat had to 
contend. If comparatively large numbers of sample respondents were not 
retained across successive rounds of the survey, the capacity of the 
NSPY to provide data to assess changes in outcomes in response to 
exposure over time would be greatly diminished. Further, if attrition 
was specific to certain groups, then the NSPY estimates would also be 
biased. 

For the purpose of estimating within-respondent changes in outcomes in 
response to changes in exposure across sample periods--the main use of 
the NSPY data--Westat achieved follow-up longitudinal response rates of 
between 82 percent and 94 percent for waves 4 through 9, the follow-up 
waves to the first three enrollment waves. The longitudinal response 
rate consists of two elements: (1) the percentage of prior survey 
respondents that are tracked and for whom eligibility is determined and 
(2) the percentage of those eligible that actually complete an 
interview. Across the three follow-up survey rounds, Westat tracked and 
determined the eligibility to participate in a follow-up survey of 
between 92 percent and 96 percent of the youth and parents who 
completed a survey in the prior round. Of these, Westat obtained 
consent and completed extended interviews with between 94 percent and 
96 percent of youth and parents for whom eligibility for a follow-up 
survey had been determined. 

In our view, Westat's follow-up response rates resulted in a sample 
that was sufficient to provide reliable findings about the effects of 
exposure on outcomes. In addition, Westat's nonresponse adjustment 
methodology compensated for effects of differential response rates 
related to the percentage of persons in certain age groups, of certain 
races and ethnicities, of those that owned homes versus rented, those 
that were U.S. citizens versus noncitizens, and those with incomes 
below the poverty level. 

The NSPY Data Could Be Used to Detect Reasonably Small Effects, and 
Westat's Measurement of Exposure and Outcomes Were Valid and Could 
Detect Effects, if They Occurred: 

The NSPY sample could be used to detect changes in outcomes that were 
on the order of magnitude of changes expected by ONDCP for the 
campaign, and its measures of exposure were valid and reliably 
predicted outcomes. In early meetings on the design of the evaluation 
of the media campaign, ONDCP officials reported that it had a specific 
Performance Measures of Effectiveness system and that the campaign was 
embodied within the first goal of the National Drug Strategy, which was 
to "educate and enable America's youth to reject illegal drugs as well 
as the use of alcohol and tobacco." Under this goal, ONDCP's proposed 
targets for reducing the prevalence of past-month use of illicit drugs 
and alcohol among youth from a 1996 base year--by 2002, reduce this 
prevalence by 20 percent, and by 2007, reduce it by 50 percent. ONDCP 
officials identified other specific targets, again from the base year 
1996--by 2002, increase to 80 percent the proportion of youth who 
perceive that regular use of illicit drugs, alcohol, and tobacco is 
harmful; and by 2002, increase to 95 percent the proportion of youth 
who disapprove of illicit drug, alcohol, and tobacco use. To achieve a 
goal of 80 percent of 12th grade youth who perceive that regular use of 
marijuana is harmful would require increasing the 1996 baseline 
percentage of youth perceiving marijuana as harmful from 60 percent, as 
measured by MTF, or by about 3.3 percentage points per year from 1996 
to 2002. Westat's sample could be used to detect this amount of annual 
change in youth attitudes. 

In order to detect changes in outcomes due to exposure to the campaign, 
it also was necessary that Westat accurately measure and characterize 
exposure to the campaign. Westat provided evidence for the validity of 
its measures of self-reported exposure, and the evidence suggests that 
the measure of exposure was both valid and reliable. To measure 
exposure to the campaign for both youth and parents, NSPY interviewers 
asked respondents about their recall of anti-drug advertisements 
(general exposure) and their recognition of specific current or very 
recent television and radio advertisements (specific 
exposure).[Footnote 18] To facilitate measures of recall, respondents 
viewed television and radio advertisements on laptop computers. Youth 
and parents were only shown or listened only to advertisements targeted 
to their respective groups. In addition, both youth and parents were 
asked some general questions about their recall of advertisements seen 
or heard in various media, including television, radio, newspapers, 
magazines, movie theaters, billboards, and the Internet. 

Westat's assessments of the validity of its measure of exposure to 
campaign advertisements confirm that the NSPY data were able to measure 
exposure. First, Westat examined respondents' recall of campaign 
advertisements using "ringer" television advertisements-- 
advertisements that never had appeared. According to Westat's analysis 
of ringer advertisements, youth were more likely to recognize an 
advertisement as a campaign advertisement when presented with an actual 
campaign advertisement than a bogus one. For example, a far lower 
percentage of respondents (11 percent) claimed to have seen a ringer, 
or bogus, advertisement than the percentage who claimed to have seen 
the broadcast advertisements (45 percent), particularly the 
advertisements that were delivered with high frequency. The result held 
for youth and for parents. 

Second, comparing data on advertisement time purchases with self- 
reported exposure to these advertisements in the NSPY, Westat found a 
high correlation between advertising and exposure. Specifically, on the 
basis of analysis of individual advertisements' gross rating points 
(GRP)--a measure of the underlying reach and frequency of each 
advertisement--and self-reported exposures by respondents, Westat found 
a high correlation between GRPs purchased by the campaign and self-
reported exposure to advertisements among youth. The correlation for 
parents was somewhat smaller, but was also significant. Third, Westat 
also compared self-reported exposure with recall of the correct brand 
phrase and found a strong association between self-reported exposure 
and correct recognition of the brand phrase. This is further evidence 
for the validity of its measures of self-reported exposure. 

Westat measured a variety of outcomes for youth and parents and took 
steps to ensure that the measures were consistent with existing 
research. The youth questionnaires included numerous questions that 
were designed to measure exposure to the campaign advertisements and 
other anti-drug messages. The youth question areas included exposure 
propensity to media; current and past use of tobacco, alcohol, 
marijuana, inhalants, and Ecstasy; past discussions with and 
communication of anti-drug messages from parents and friends; 
expectations of others about respondent's drug use; knowledge and 
beliefs about the positive and negative consequences of drug use; 
exposure to campaign messages; family and peer factors; personal 
factors; and demographic information. Westat used separate 
questionnaires for youth of different ages; one questionnaire was used 
for children (aged 9 to 11) and another one was used for teens (aged 12 
to 18). 

Westat's Analytic Methods Aimed to Isolate Causal Effects of the 
Campaign and Did So Using Sophisticated Techniques That Enhanced the 
Strength of Its Findings: 

In it analysis, Westat used three types of evidence to draw inferences 
about the effects of the campaign: (1) trend data--data that describe 
increases or decreases in drug use and other outcomes over time; (2) 
cross-section analysis--measures of association between exposure to 
campaign messages and individual drug use beliefs, intentions, and 
behaviors, at the time data were collected; and (3) longitudinal 
analysis--measures of association, for youth and parents who were 
observed at two points in time, between exposure to campaign messages 
at the earlier time on outcomes at the later time.[Footnote 19] Westat 
indicated that trends over time, by themselves, could not be used to 
provide definitive support for campaign effects. Rather, the trends 
needed to be supported by measures of association. Westat also 
indicated that measures of association, whether cross-sectional or 
longitudinal, needed to control for variables that could influence 
outcomes independently of the campaign or otherwise confound the 
association between exposure and outcomes. Cross-section association 
between exposure and outcomes measured at the same time would provide 
stronger evidence of campaign effects than would trend data alone, 
provided that controls for other variables were introduced into the 
associational analyses. However, even if cross-section associations 
between exposure and outcomes hold after controlling for the effects of 
other variables, as Westat pointed out, there may remain an alternative 
explanation for cross-section associations: For example, an outcome-- 
like perceptions of others' use of drugs--may be the cause of exposure 
rather than an effect of it. Westat's longitudinal analysis attempts to 
address the ambiguities that exist with cross-sectional associations. 
With longitudinal data, if, after controlling for other confounding 
variables, exposure measured at an earlier time is associated with an 
outcome at a later time, the inference made is that the causal 
direction is from exposure to outcome, since an effect cannot precede a 
cause in time. 

As the campaign was implemented nationally and it was therefore not 
possible to assign youth and their parents randomly to treatment and 
control groups, a major threat to the validity of the conclusions from 
the evaluation is that the observed correlations between exposure to 
the campaign and self-reported attitudes and behaviors could reflect 
preexisting differences among individuals in their underlying 
susceptibility to campaign messages. The evaluation's associations 
between exposure to the campaign and self-reported initiation of 
marijuana use took into account statistically the individual 
differences in attributes among youth who were exposed to various 
levels of campaign messages, and they adjusted for the influence of 
other variables that could determine marijuana initiation--called 
confounder variables. As such, Westat's evaluation of the associations 
between campaign exposure and marijuana initiation have accounted for 
individual differences among youth and can be viewed as comparisons of 
outcomes for statistically similar individuals. Further, the 
statistical test Westat used in assessing the relationship between 
exposure and initiation did not rely upon assumptions of linearity 
between levels of exposure and initiation. Instead, it tests for an 
ordered relationship between exposure and an outcome such as marijuana 
use initiation. 

Westat used statistical methods to address the possibility that 
preexisting differences between individuals could have caused both 
reported levels of exposure and respondent outcomes, and its use of 
these methods contributed to the validity of its findings about the 
effects of the campaign on outcomes. If, independently of the campaign, 
individuals differed in their underlying tendencies to accept and 
recall campaign messages, and if the individuals who were more likely 
to recall advertisements also were those who were more likely to 
respond to advertisements, then, absent efforts to address this 
confounding factor, the findings about the evaluation would be 
questionable. This type of bias is often called a selection effect. If 
selection effects occurred in the campaign, then both exposure and 
reported changes in attitudes and behaviors could reflect underlying 
beliefs that were not affected by the campaign, despite the presence of 
statistical correlations between self-reported exposure and changes in 
attitudes and behaviors. 

To control for selection effects and the many factors that could have 
influenced both exposure and outcomes independently of, or in 
conjunction with, the campaign, Westat used propensity scoring methods. 
These methods limit the influence of preexisting differences among 
exposed groups by controlling for a wide range of possible confounding 
variables. Propensity score methods are used to create comparison 
groups that are similar on measured and potentially confounding 
variables but that differ on their levels of treatment. In the 
evaluation of the campaign, the comparison groups were similar on 
confounding variables but differed on their level of exposure to 
campaign messages. Propensity score methods replace a set of 
confounding variables with a single function of these variables, which 
is called the propensity score. In Westat's analysis, an individual's 
propensity score is considered to represent an individual's probability 
of being assigned to a particular level of exposure to the campaign, 
conditional upon the individual's values of the confounding variables. 
By including relevant, potentially confounding variables and matching 
individuals on their propensity scores, Westat was able to minimize 
bias due to selection effects. The comparison groups that Westat 
created by using propensity score methods can be considered as 
statistical analogues to randomly assigning individuals to different 
levels of exposure. After creating these groups, Westat then analyzed 
outcomes among the groups having different propensities to be exposed 
to campaign messages. 

Our assessment of Westat's methods leads us to conclude that Westat 
took reasonable steps to develop valid propensity models, and as a 
result of its models, its analysis identified the effects of the 
campaign, net of other factors included in its propensity score models. 
First, rather than simply compare individuals who were exposed to 
campaign messages with those who were not exposed, Westat estimated and 
compared groups of individuals with different levels of exposure, where 
the number of exposure groups was measured alternatively as a three-or 
four-level variable--e.g., low, medium, or high exposure.[Footnote 20] 
Second, for the results of propensity methods to be valid, it is 
important that the propensity scoring models include all relevant 
variables that could otherwise explain differences in both exposure and 
outcomes, as evaluators can adjust only for confounding variables that 
are observed and measured. If an important variable is omitted from the 
propensity model, the results of analyses may be affected. Westat's 
models included many relevant and potentially confounding variables. 
For example, in the youth models, the propensity score models included 
measures of demographic attributes, educational attainment and 
educational aspiration, family and parent background, parental 
consumption of television and other media, income and employment, 
reading habits, Internet usage, location of residence in an urban area, 
among other variables. Third, for propensity models to remove the 
effects of confounding variables from the association between exposure 
and response, it is necessary that the population means of the 
confounder variables not vary across exposure levels. If a confounder 
is successfully balanced, then it will have the same theoretical effect 
across all exposure levels. After estimating models, Westat also 
assessed and demonstrated the balance of variables in its propensity 
models. 

The Phase III Evaluation Provided Mixed Evidence of the Campaign's 
Effectiveness on Intermediate Outcomes, but It Found No Effect of the 
Campaign on Parental Monitoring of Youth: 

Westat reported mixed evidence about the effectiveness of the campaign 
on intermediate outcome measures--such as recall and identification of 
campaign messages, youth anti-drug attitudes, and parents' beliefs and 
behaviors--that were thought to be causal factors influencing youth 
drug use, the ultimate target of the campaign. Most parents and youth 
recalled exposure to campaign anti-drug messages, and for both groups, 
recall increased during the September 1999 to June 2004 period covered 
by the phase III evaluation. For current, non-drug-using youth--whose 
resistance to initiating marijuana use the campaign intended to affect-
-although NSPY data showed some favorable trends in anti-drug attitudes 
and beliefs and in the proportion of youth who said that they would 
definitely not try marijuana, there was no evidence that exposure to 
the campaign influenced these trends. Conversely, among current, non- 
drug-using youth, evidence suggested that exposure to the campaign had 
unfavorable effects on their anti-drug norms and perceptions of other 
youths' use of marijuana--that is, greater exposure to the campaign was 
associated with weaker anti-drug norms and increases in the perceptions 
that others use marijuana. On three of five parent belief and behavior 
outcome measures--including talking with children about drugs, doing 
fun activities with children, and beliefs about talking with children-
-the evidence pointed to a favorable campaign effect on parents. 
However, while there was mixed evidence on the effect of the campaign 
on parents' beliefs and attitudes about monitoring children's 
behaviors, there was no evidence to support a claim that the campaign 
actually affected parents' monitoring behaviors--an area of the 
campaign's focus for parents--and there was little evidence for 
favorable indirect effects on youth behavior or beliefs as the result 
of parental exposure to the campaign. 

Youth and Parents' Recall of Campaign Advertisements Increased over 
Time, Their Impressions of the Advertisements Were Favorable, and They 
Could Identify the Campaign Brand: 

According to Westat, the campaign purchased enough advertising time 
over the 58-month period from September 1999 to June 2004 to achieve an 
average exposure of 2.5 youth-targeted ads per week for youth and an 
average of 2.2 parent-targeted advertisements per week for parents. 
Westat's estimates include campaign advertisements intended for either 
all youth or all parents, but they do not include exposure of youth to 
parent advertisements or parents to youth advertisements, nor do they 
account for separate advertising targeted to specific race-or ethnicity-
defined audiences. 

Using exposure indexes, Westat measured trends in general and specific 
exposure to campaign advertisements. The general exposure index was 
based on questions that asked about exposure to anti-drug messages in 
recent months through a variety of channels, including movies, 
television, radio, and billboards, and was not limited to campaign 
advertisements.[Footnote 21] The specific exposure index was based on 
recall of specific advertisements broadcast during the 60 days prior to 
the respondent's interview, and was limited to advertisements that 
targeted the respondent. For example, for youth, only youth 
advertisements were sampled to measure specific exposure. Youth aged 
12˝ to 18 and their parents reported increasing levels of recall of 
specific but not general exposure to campaign advertisements over time. 
For both parents and youth, there was a sharp increase over time in the 
recall of specific exposure of television ads across the campaign. 
Westat speculated that the increase in specific recall may have arisen 
from better-placed, more memorable, or longer-aired advertisements 
rather than only to an overall increase in television advertisements. 
However, recall of all general anti-drug advertising was fairly stable 
over time, as there was no overall detectable change in reported 
general exposure over the course of the campaign. 

Beginning in 2001, when the evaluation started to measure brand phrase 
recall, and continuing through 2004, the evidence indicates that youth, 
in particular, exhibited increases in brand phrase recall. Advertising 
campaigns may use a brand phrase to provide a recognizable element, and 
to the extent that the brand is recognized and positively regarded, its 
familiarity may lead to a positive response to a new advertisement or 
increase the perception that each advertisement is part of a larger 
campaign. The campaign included both a parent and a youth brand. Brand 
messages may have involved a series of phrases or the portrayal of an 
activity or lifestyle as positive (e.g., participating in team sports) 
to set up the brand phrase of "The Anti-Drug." Westat reported that the 
evidence from the NSPY shows that the greater the exposure to media 
campaign advertising, the more likely respondents were to recall the 
brand phrase. In addition, the more that respondents recalled specific 
ads, the more likely they were to recognize the brand phrase, although 
over time even those with less exposure had learned the brand phrase. 

Overall, youth reported favorable impressions of the subset of campaign 
television advertisements that they were asked to evaluate, and their 
favorable impressions increased over time. Responses to the 
advertisements--whether they were attention getting, convincing, or 
said something important to the respondent--were positive among both 
youth and their parents. Parents' evaluations of the advertisements 
were generally more positive than those of youth, and parents' positive 
views also increased over time. 

In addition to distributing messages directly in media advertisements, 
the campaign aimed to reach its target audiences indirectly through 
other institutions and routes, such as community groups, in-school and 
out-of-school anti-drug education, and discussions among youth and 
parents, and youth and friends, concerning drug use and the drug 
advertisements. The NSPY data indicated that the campaign's messages 
were not accompanied by similar increases in exposure to messages from 
other sources. Both youth and parents reported receiving anti-drug 
messages from other sources, but they did not consistently report 
increases in exposure to messages from these sources. For example, from 
the 2000 to 2004 samples, the percentages of youth reporting receiving 
in-school drug education messages and attending out-of-school drug 
education both declined. 

Westat Found That the Campaign Generally Had No Effect on the Attitudes 
of Youth Not Using Marijuana toward Its Use but That Exposure to the 
Campaign Was Associated with Unfavorable Effects on Youth Perceptions 
of Others' Use of Marijuana: 

Westat generally found no significant effects of campaign exposure on 
the cognitive outcomes of adolescent nonusers of marijuana--i.e., 
development of anti-drug attitudes and beliefs. For current nonusers, 
the evaluation reported on four cognitive measures and a fifth measure 
of their perceptions of others' use of marijuana. Three of the four 
measures--attitudes and beliefs about the consequences of marijuana 
use; perceived social norms or pressures from parents, friends, and 
peers about infrequent or regular marijuana use; and perceived self- 
efficacy to avoid using marijuana, or their confidence to turn down use 
of marijuana under various circumstances--were premised to affect the 
fourth--youth intentions to use marijuana at all during the next year. 
The fifth outcome, perceptions of other youths' use of 
marijuana,[Footnote 22] was included to examine whether exposure to the 
campaign was leading to increased perception among youth that others 
use marijuana, and whether this perception, in turn, affected their own 
behaviors. 

Westat reported that the evidence from the analysis of trend data from 
2000 to 2004 for two of the youth cognitive measures--attitudes and 
beliefs about the consequences of marijuana use and intentions to use 
marijuana--showed significant increases in youth believing that 
marijuana use had negative consequences and significant increases in 
the percentage of youth that reported that they had no intention to use 
marijuana. However, evidence from both cross-section and longitudinal 
associations between exposure and these two cognitive outcomes did not 
substantiate that the favorable trends arose from exposure to the 
campaign. Specifically, the cross-sectional associations between both 
general and specific exposure to the campaign and intentions not to use 
marijuana show no significant favorable effects of exposure on this 
outcome. None of the cross-section associations between either general 
or specific exposure and intention to use marijuana are significant, 
and none of the longitudinal associations between specific exposure and 
intentions are significant. Two of the longitudinal associations 
between general exposure and intentions are significant, but the 
direction of the effect is unfavorable, in that greater exposure led to 
declines in intentions not to use marijuana. The evidence from the 
associational analyses between exposure and attitudes and beliefs about 
the consequences of marijuana use generally did not show an effect of 
the campaign. While there was one significant cross-section association 
between general exposure and attitudes and beliefs about consequences 
during the final two waves of survey data, there were no significant 
cross-section associations between specific exposure and attitudes and 
beliefs about consequences, nor were there any significant longitudinal 
associations with either general or specific exposure. 

The associational analysis also produced some evidence of unfavorable 
effects of exposure on social norms--i.e., social pressures from 
parents, peers, and other important persons about marijuana use. 
Westat's cross-section associations showed no significant effects of 
exposure on social norms, but its longitudinal associations showed that 
across all survey rounds, there was a significant relationship between 
specific exposure and weaker social norms. Westat's analysis of 
associations between exposure and perceptions of others' use of 
marijuana also produced significant results. Cross-section associations 
between specific exposure and perceptions of others' use were 
significant, as were longitudinal associations of this relationship. In 
other words, among youth who reportedly did not use marijuana at the 
time of their interview, there was a significant effect of specific 
exposure on the perception that others used marijuana, and the 
direction of the effect was unfavorable--that is, those reporting 
higher exposure to anti-drug ads were more likely to believe that their 
peers used marijuana regularly. A significant and unfavorable 
relationship between specific exposure and perceptions of others' use 
of marijuana was obtained for the data covering the entire period of 
the evaluation as well as for the period of the redirected campaign, 
from 2002 to 2004. 

The Evaluation Reported Favorable Effects of the Campaign on Three 
Parent Outcomes but Not on Parental Monitoring: 

A theme of the campaign was to encourage parents to engage with their 
children to protect them against the risk of drug use, and parent 
skills were a focus of parent advertising almost since the start of the 
campaign. The campaign encouraged parents to monitor their children's 
behavior by knowing where they were and with whom, and to make sure 
that they had adult supervision. It also encouraged parents to talk 
with their children about drugs and to a lesser degree to engage in fun 
activities with their children. The evaluation observed five outcomes 
for parents, and for four of the five found significant and favorable 
effects of exposure to the campaign. For three outcomes--parent-child 
conversations about drugs (talking behavior), parents' beliefs and 
attitudes about talking with their children about drugs (talking 
beliefs), and parents' engagement with their children in in-home and 
out-of-home activities (fun activities)--both cross-section and 
longitudinal associations between exposure and outcomes were generally 
significant and favorable to the campaign. For parents' beliefs and 
attitudes toward monitoring their children's behaviors, Westat reported 
favorable trend and cross-sectional associations but no significant 
overall longitudinal effects of either general or specific exposure on 
this outcome. For the fifth outcome, parent monitoring behaviors--that 
is, parents' knowing or having a pretty good idea about what their 
child was doing or planned to do--the evidence did not support a 
finding of an effect of the campaign. There were no significant 
favorable trends in parents' reports of monitoring behaviors, and there 
were no significant cross-section or longitudinal associations of 
either general or specific exposure on monitoring behaviors. 

No Evidence of Favorable Effects of the Campaign on Youth Outcomes 
through Campaign Effects on Parental Outcomes: 

Despite evidence of some favorable parental outcomes for the campaign, 
Westat found no significant evidence for the overall evaluation that 
these favorable parent outcomes affected youth attitudes and behaviors 
toward drug use. Specifically, for the entire period covered by the 
evaluation, Westat found no evidence of overall, indirect campaign 
effects on parents leading to changes in marijuana use, intentions to 
use marijuana, social norms, self-efficacy, or cognitions among youth 
who were not marijuana users. Westat found that there were some 
significant indirect effects of parental specific exposure on some 
youth outcomes for some subgroups. For example, parental specific 
exposure was favorably associated with intentions to use marijuana for 
14-to 18-year-olds and for boys, and it was also associated favorably 
with attitudes and beliefs about the consequences of marijuana use for 
Hispanics. Westat also found significant but unfavorable indirect 
effects of parents' general exposure on subgroups of youth in other 
youth outcomes. For example, parental general exposure was unfavorably 
associated with youth social norms for 14-to 16-year-olds and for 
girls. 

The Phase III Evaluation Found No Significant Effects of Exposure to 
the Campaign on Youth Drug Use Outcomes Other than Limited Unfavorable 
Effects on Marijuana Initiation: 

Westat reported that the NSPY data showed some declines in self- 
reported lifetime and past-month use of marijuana by youth over the 
period from 2002 to 2004, and these trends in NSPY were consistent with 
trends in other national surveys of drug use over these years. Westat 
also reported that the NSPY data showed declining trends in youth 
reports of offers to use marijuana. However, Westat cautioned that 
because trends do not account for the relationship between campaign 
exposure and changes in self-reported drug use, drug use trends alone 
should not be taken as definitive evidence that the campaign was 
responsible for the declines. On the basis of the analysis of the 
relationship between exposure to campaign advertisements and youth self-
reported drug use in the NSPY data--assessments that used statistical 
methods to adjust for individual differences and control for other 
factors that could explain changes in self-reported drug use-
-for the entire period covered by its evaluation, Westat found no 
significant[Footnote 23] effects of exposure to the campaign on 
initiation of marijuana by prior nonusing youth. The only significant 
effect indicated in Westat's analysis of the relationship between 
campaign exposure and self-reported drug use was an unfavorable effect 
of exposure on marijuana initiation--that is a relationship between 
campaign exposure and higher rates of initiation--for one round of NSPY 
data and similar unfavorable effects of campaign exposure on marijuana 
initiation among certain subgroups of the sample (e.g., 12˝-to 13-year- 
olds and girls). Westat found no effects of campaign exposure on rates 
of quitting or use by prior users of marijuana. 

Westat Tracked Trends in Marijuana Use from Several Sources and 
Reported That the Trend Data by Themselves Were Insufficient to 
Demonstrate Effects of Exposure to the Campaign: 

Westat tracked trends in self-reported use of marijuana by youth and 
trends in youth reports of offers to use marijuana for the period from 
2000 to the first half of 2004 to determine if there were significant 
declines. Westat also assessed these trend data for changes occurring 
since 2002, or during the period of the redirected campaign. Westat's 
trend analysis was designed to provide supportive but not definitive 
evidence for campaign effects. 

In its trend analysis, Westat compared trends in self-reported drug 
use--lifetime, past year, and past month--in the NSPY with trend data 
on self-reported drug use from three other nationally representative 
surveys of drug use--Monitoring the Future, the Youth Risk Behavior 
Surveillance System (YRBSS), and the National Survey on Drug Use and 
Health.[Footnote 24] Both MTF and YRBSS are school-based surveys, and 
NSDUH is a household survey that provides estimates of drug use by the 
civilian, noninstitutionalized population of the United States aged 12 
years and older. Methodological differences between the school-based 
surveys--MTF and YRBSS--and the household surveys--NSPY and NSDUH--have 
been shown to account for the some of the differences in estimates of 
marijuana use. 

According to Westat's analysis, the surveys of self-reported marijuana 
use show some similarities and differences in trends depending upon the 
measure, age group, or subperiod covered within the longer 2000 to 2004 
period. For example, the MTF data generally show declines in lifetime, 
past-year, and past-month self-reported drug use for 8th, 10th, and 
12th graders over the years from 2000 to 2004, although only some of 
the year-to-year differences in the MTF self-reported drug use data 
were statistically significant. Nonetheless, for the subperiod from 
2002 to 2004, MTF data show statistically significant declines in past- 
year and past-month use for 8th graders and past-year use for 10th 
graders, and the NSPY data also show statistically significant declines 
in past-month use from 2002 to 2004 for youth aged 12˝ to 18 years old 
and for 14-to 18-year-olds. On the other hand, the MTF data suggest a 
decline in past-year and past-month use by 10th graders from 2000 to 
2002, but the NSPY data suggest an increase in past-month marijuana use 
during this period.[Footnote 25] Further, the data from NSDUH for 2000 
and 2001 also show statistically significant increases in lifetime, 
past-year, and past-month marijuana use among youth aged 12 to 17, 
statistically significant increases in lifetime and past-year marijuana 
use for youth aged 16 to 17, and a statistically significant increase 
in past year use for youth aged 14 to 15. The pattern of increase in 
NSDUH data from 2000 to 2001 is consistent with the 2000 to 2002 
increases in past-month use in NSPY, but they differ from the MTF 
trends over this period. 

All four surveys generally show declines in marijuana use beginning in 
2002, but not all of the declines are statistically significant. Both 
MTF and NSPY show some statistically significant declines since 2002, 
and while NSDUH and YRBSS show declines, the declines were not 
statistically significant. These declines starting in 2002 coincide 
with the redirected campaign and the introduction of the Marijuana 
Initiative. 

Despite the concurrence of the trend data from all sources for the 2002 
to 2004 period, Westat concluded that the existence of declining trends 
in self-reported drug use by themselves do not provide definitive 
evidence that the campaign caused the declines because factors other 
than the campaign also could affect behavior. For example, changes in 
high-school completion rates among youth could affect drug use 
behaviors, as high school dropouts may have more involvement with drugs 
than youth who stay in school. Additionally, declines in self-reported 
drug use that began before the initiation of phase III of the campaign 
could not have been caused by the campaign. The declines reported in 
MTF began prior to the start of phase III of the campaign; therefore, 
factors other than the campaign had to have been responsible for the 
start of the decline occurring in these data. Further, ONDCP also has 
acknowledged the limitations of trends in the national surveys for 
determining whether changes in drug use were the result of the 
campaign. ONDCP's Office of Programs, Budget, Research and Evaluation 
wrote about the MTF, YRBSS, and NSDUH:[Footnote 26] 

"They provide policy makers with broad indicators of the success of 
policy…However, they will not be able to answer the critical question 
of whether these changes were the result of the Media Campaign. These 
surveys do not ask respondents about their exposure and reactions to 
the messages of the Media Campaign that can then be linked to their 
drug-related attitudes and behavior."[Footnote 27] 

Westat Reported That Trends in Marijuana Offers Declined over Time, but 
Factors Other than the Campaign Contributed to Changes in Offers: 

Westat assessed trends in youth reports of receiving offers of 
marijuana--whether anyone had ever offered youth marijuana and the 
frequency of offers within the past 30 days. Marijuana offers are 
closely related to marijuana use, and the campaign aired messages that 
encouraged resistance to offers of marijuana. Over the 2000 to 2004 
period, Westat found significant increases in the percentage of youth 
reporting that they had never received offers, and it also found 
significant decreases in the percentage of youth reporting that they 
had received offers in the prior month. Westat also found significant 
changes in offers over 2002 to 2004, during the period of the 
redirected campaign, and these changes were generally consistent with 
the trends for the overall 2000 to 2004 period. Further, on the basis 
of longitudinal analysis of the relationship between offers in one 
period and marijuana use in the subsequent period among youth who were 
nonusers in an initial survey round--an analysis that assesses whether 
offers precede use or are simply a correlate of it--Westat found that 
youth who reported having received a marijuana offer at one period were 
much more likely--between three and seven times more likely, depending 
upon age group--to have initiated marijuana use at a following period 
than nonusing youth who reported never having received such an offer. 
However, as Westat reported, while the findings on offers are 
favorable, they cannot be ascribed to the campaign because they may be 
caused by other factors, as the analysis of the relationship between 
offers and use did not take into account other factors that could 
affect use. 

On the Basis of Its Analysis of the Association between Exposure and 
Drug Use Outcomes, Westat Found No Evidence That Exposure to the 
Campaign Affected Initiation or Cessation of Marijuana Use: 

From its longitudinal analysis of associations between exposure and 
initiation of marijuana use, Westat found no evidence that increased 
exposure to the campaign reduced youth's initiation of marijuana use. 
Westat's longitudinal analysis assessed the effects of exposure at one 
survey wave on marijuana initiation at a subsequent survey wave, 
controlling for potential confounding variables that could affect the 
exposure initiation relationship. Westat assessed the effects of two 
types of exposure on initiation of marijuana use--general exposure and 
specific exposure. General exposure represents the sum of recalled 
exposure to anti-marijuana advertising in four types of sources of 
advertisements--television and radio, movies and videos, print media 
including newspapers and magazines, and outdoor media. Specific 
exposure represents the sum of recalled exposure to youth-targeted 
individual campaign television advertisements that had been aired in 
the 60 days prior to an interview. 

Westat found no significant effects of the level of general exposure on 
marijuana use initiation, either over the entire period of the campaign 
or between subperiods as defined by survey rounds.[Footnote 28] Westat 
also found no overall effects of levels of specific exposure on 
marijuana initiation during the entire period of the campaign, but it 
found one significant association between specific exposure and 
marijuana use initiation that occurred in the data from wave 7 and its 
wave 9 follow-up, or during the period of the Marijuana Initiative. 
Wave 7 was the first complete survey wave covering exposure to the 
Marijuana Initiative. The significant association from this analysis 
was that higher levels of specific exposure were associated with higher 
levels of initiation of marijuana use among previously nonusing youth. 

Westat also examined the longitudinal relationships between exposure 
and initiation for nine subgroups of youth (two sexes, three race/ 
ethnicity groups, two risk groups, and two nonoverlapping age groups). 
For several subgroups, it found significant associations between 
specific exposure and marijuana initiation. These associations were in 
a direction that was unfavorable to the campaign, in that greater 
specific exposure was associated with higher levels of initiation. The 
subgroups for which these unfavorable associations were most pronounced 
included 12˝-to 13-year-olds, girls, African Americans, and lower risk 
youth. 

On cessation and reduction of marijuana use, Westat assessed two 
outcomes among current marijuana users: the rate at which they quit 
using marijuana and their frequency of use. The frequency of use 
measure allowed for campaign effects to be observed if users did not 
quit but reduced their use of marijuana. Westat estimated that the quit 
rate--the percentage of prior-year users reporting that they no longer 
used marijuana--among prior-year users of marijuana was 24.8 percent. 
However, it found no statistically significant association between 
general exposure and quitting or between specific exposure and 
quitting. It also found that among adolescent marijuana users, the 
frequency of use--increase, decrease, or no change--was not affected by 
exposure to the campaign. 

Conclusions: 

A well-designed and executed multiyear study of the impact of the ONDCP 
anti-drug media campaign on teen initiation of drug use, or cessation 
of drug use, shows disappointing results for the campaign. The study 
provides no evidence that the campaign had a positive effect in 
relation to teen drug use, and shows some indications of a negative 
impact. Some intermediate outcomes, such as parents talking with 
children about drugs, and doing fun activities with their children, 
showed positive results in that the media campaign encouraged parents 
to adopt these behaviors. However, other intermediate outcomes, such as 
parents' monitoring of their children's behavior, were not shown to be 
affected by the campaign. Moreover, the evaluation did not provide 
evidence that intermediate outcomes that showed positive results 
translated into greater resistance to drugs among the teenage target 
population. 

Unfavorable preliminary findings from the evaluation were reported by 
Westat in 2002. Beginning in 2002, ONDCP took a number of steps that 
were intended to strengthen the power of the campaign to achieve 
positive results. These steps included more rigorous ad copy testing 
and a concentration on anti-marijuana messages. However, the post-2002 
results yielded no evidence of positive impacts and some evidence of 
negative and unintended consequences in relation to marijuana use. 
Specifically, exposure to advertisements during the redirected campaign 
was associated with higher rates of marijuana use initiation among 
youth who were prior nonusers of marijuana. 

Most parents and youth recalled exposure to the campaign messages and, 
further, they recognized the campaign brand. Thus, the failure of the 
campaign to show positive results cannot be attributed to a lack of 
recognition of the messages themselves. This raises concerns about the 
ability of messages such as these to be able to influence teen drug 
attitudes and behaviors. It raises questions concerning the 
understanding of the factors that are most salient to teens' decision 
making about drugs and how they can be used to foster anti-drug 
decisions. 

Westat's evaluation is centered on this particular configuration of a 
media campaign as it was presented from 1999 to 2004, and its results 
pertained to the campaign nationwide. It cannot be construed to mean 
that a media campaign that is configured differently from this one 
cannot work. Nor should its results be construed to mean that in some 
locations, for some groups of youth, the campaign did not have an 
effect on drug use. However, substantial effort and expertise were 
brought to the task of designing the advertisements from the outset, 
and the 2002 redirection of the campaign placed even greater emphasis 
on copy testing and enhanced ONDCP oversight. This casts some doubt on 
the notion that a better media campaign can lead to positive results. 

It is also important to note that two recent smaller studies in three 
locations have provided evidence of a limited effect of the campaign 
for some youth, and it is quite possible that additional analyses of 
the NSPY data using different methods or measures may find other 
effects of the campaign, at least for some adolescents, than have been 
produced by Westat's evaluation team. The data from the evaluation have 
only recently been made available to academic and other researchers, 
and while the analyses undertaken by Westat are, as we have noted 
elsewhere, appropriate and thorough, they are not exhaustive. 

It is heartening that surveys intended to measure teen drug use, such 
as Monitoring the Future, are showing declines in marijuana use in 
recent years. Indeed, NPSY also shows some evidence of a decline in 
drug use among teens. However, Monitoring the Future and other surveys 
of teens concerning drug use are not linked to exposure to the media 
campaign, and NPSY shows no relationship between anti-drug media 
campaign exposure and favorable drug outcomes for teens. This seems to 
indicate that other unidentified factors, other than the anti-drug 
media campaign, are affecting drug use decisions among teens. 

Although ONDCP has pointed to declines in teen drug use and credited 
the campaign along with other prevention efforts as contributing to 
significant success in reducing teen drug use, trend data derived from 
the Monitoring the Future survey that show declines in teen marijuana 
use from 2001 to 2005 do not explicitly take into account exposure to 
the campaign, and therefore, by themselves, cannot be used as evidence 
of effectiveness. ONDCP has indicated in the past, and we concur, that 
because these surveys cannot link their results with the media 
campaign, they do not measure campaign effectiveness. The evaluation of 
the media campaign reinforces the lack of linkage between the media 
campaign and teen drug use behavior. 

It is important to note that virtually all social science research is 
imperfect. Attempting to systematically observe and document human 
behavior in real-world settings is a daunting task given the extremely 
wide variation in both humans and settings. We believe that the 
evaluation of the ONDCP media campaign is credible in that it was well 
designed given the circumstance of the campaign, and appropriately 
executed. 

Matter for Congressional Consideration: 

In light of the fact that the phase III evaluation of the media 
campaign yielded no evidence of a positive outcome in relation to teen 
drug use and congressional conferees' indications of their intentions 
to rely on the Westat study, Congress should consider limiting 
appropriations for the National Youth Anti-Drug Media Campaign 
beginning in the fiscal 2007 budget year until ONDCP is able to provide 
credible evidence of the effectiveness of exposure to the campaign on 
youth drug use outcomes or provide other credible options for a media 
campaign approach. In this regard we believe that an independent 
evaluation of the new campaign should be considered as a means to help 
inform both ONDCP and Congressional decision making. 

Agency Comments and Our Evaluation: 

We provided a draft of this report to the Director of the Office of 
National Drug Control Policy for comment on July 31, 2006. ONDCP 
provided us with written technical comments on the report, which we 
incorporated where appropriate. In addition, ONDCP provided written 
comments about our report in which it raised a question about our 
matter for congressional consideration and outlined a number of 
concerns that it had with our report on Westat's findings. These 
written comments are reproduced in appendix II. In our evaluation of 
ONDCP's written comments, we address each of the other concerns in the 
order ONDCP presented them. 

Westat Evaluation's Role in Judging the Impact of the Advertising 
Campaign: 

ONDCP comments that Westat's evaluation is ill suited to judge the 
impact of an advertising campaign in part because Westat attempted to 
establish a causal relationship between exposure and outcomes, and 
this, ONDCP indicates, is something that major marketers rarely attempt 
because of its difficulty. ONDCP writes, "we take issue with the 
fundamental method pursued by Westat and GAO, and therefore, believe 
that the study's findings are deeply flawed." We find this response 
surprising for a number of reasons. First, ONDCP is on record as 
stating that the evaluation conducted by Westat would be the means to 
assess the impact of the campaign. Indeed, in February, 2001, in the 
ONDCP publication entitled Youth Drug Use and the National Youth Anti- 
Drug Media Campaign, ONDCP states: 

"ONDCP, on the other hand, is measuring the impact of the Media 
Campaign with a thorough, rigorous, and independent evaluation. The 
nationally representative evaluation is being conducted for ONDCP by 
the National Institute on Drug Abuse (NIDA).…The evaluation is a 4-year 
longitudinal study of parents' and their children's exposure and 
response to the Media Campaign.…ONDCP will be able to assess the extent 
to which changes in anti-drug attitudes and beliefs or drug using 
behavior can be attributed to the Media Campaign."[Footnote 29] 

ONDCP officials had opportunities during the evaluation to raise 
concerns about Westat's design and its efforts to establish a link 
between exposure to the campaign and outcomes, but we are not aware of 
their having done so. However, we are aware of ONDCP's participation in 
a NIDA-sponsored expert panel review of Westat's evaluation that was 
held in August 2002. Our review of the minutes of that meeting reveals 
that while an ONDCP official raised concerns about issues such as 
assessing the nonadvertising components of the campaign and the number 
of interim reports, ONDCP officials did not at that time raise concerns 
that the evaluation was fundamentally flawed. The consensus of the 
expert panel was that Westat's evaluation was "pretty impressive" given 
the challenges presented by the absence of baseline data and of an 
experimental design. Panel members also asserted that Westat's use of 
propensity score models to isolate the effects of the campaign was 
termed both "sensible" and "state-of-the-art." 

ONDCP further states that major advertisers evaluate the success of 
their campaigns by rigorously testing advertisements prior to airing 
and by developing correlations between messages and consumer attitudes 
and behavior. While we do not dispute whether this is a commonly used 
approach among major advertisers, we believe that in assessing the 
expenditure of public funds researchers should attempt, where feasible, 
to establish causal relationships or use research designs that attempt 
to isolate the effects of federally funded interventions. While we 
acknowledge that establishing causal relationships is difficult, we 
maintain that Westat used sophisticated and appropriate statistical 
methods that aimed to isolate the effects of recalled exposure to the 
campaign on youth drug use. Further, adopting a methodology that relies 
upon correlations between advertising messages and an outcome, such as 
reductions in youth drug use, without attempting to take into account 
many of the other factors that could affect drug use allows for too 
many post hoc explanations of findings. Westat's analysis included 
socioeconomic factors, parent characteristics, television viewing 
habits, risk of using drugs, and sensation-seeking tendencies to be 
able to determine whether exposure was related to drug use net of the 
influences of these factors. We conclude, on the basis of our 
assessment of Westat's methods, that exposure to campaign messages 
generally did not influence youth drug use net of these other 
influences. 

ONDCP notes that correlational findings have been used to assess anti- 
tobacco advertising campaign results. We have not reviewed the anti- 
tobacco campaign and cannot comment on its relationship to youth 
smoking prevalence. We notice, however, that in ONDCP's comments on 
"Consequences of Further Budget Cuts," it appears to contradict its 
statements about establishing causal relationships to determine the 
effect of advertising campaigns. ONDCP writes, "Previous studies have 
established a relationship between exposure to anti-tobacco messages 
and smoking rates among teens." ONDCP goes on to draw an analogy 
between anti-smoking messages and anti-drug messages to write, "We 
should expect similar results for illicit drug use if anti-drug 
messages decline." These statements emphasize very directly the same 
kind of causal relationships that ONDCP cites as not appropriate in its 
opening comments. 

We also note that ONDCP indicates in its comments that it has made 
multiple refinements to the media campaign on the basis of earlier 
findings from the Westat study. This seems to be inconsistent with a 
position of major concerns with the fundamental soundness of the study. 

Finally, the three research papers that ONDCP cites on page 2 of its 
comments on "Conflicting Evidence from Other Research" all use exposure-
response methodologies that are analogous to Westat's and all attempt 
to isolate the causal effects of exposure either to ONDCP's campaign or 
to other media campaigns. Thus, it would seem that ONDCP's comment that 
efforts to isolate causal effects of media campaigns are fundamentally 
flawed would also apply to these three studies. 

ONDCP Made Campaign Changes as a Result of Westat Interim Findings: 

ONDCP indicates that it has sought to improve the performance of the 
media campaign by using the results from the Westat study and other 
data. We are aware that ONDCP redirected the campaign in 2002 in 
response to Westat's interim findings that indicated some negative 
impacts of the campaign on youth marijuana use. However, the 2002 to 
2004 Westat study results also did not show positive outcomes. Westat's 
study is the only national evaluation of the campaign. Although 
Monitoring the Future provides context regarding general drug trends 
among youth, as ONDCP has stated: 

"These surveys [MTF, the National Household Survey on Drug Use, and the 
Youth Risk Behavior Survey] will permit the determination of whether 
drug use behavior and related attitudes and beliefs changed after the 
launching of Phase III of the Media Campaign in mid-1999. However, they 
will not be able to answer the critical question of whether these 
changes were the result of the Media Campaign. These surveys do not ask 
respondents about their exposure and reactions to the messages of the 
Media Campaign that can then be linked to their drug-related attitudes 
and behavior."[Footnote 30] 

More recently in late 2005, ONDCP launched a newly designed campaign. 
The impact of this campaign is not known and should be independently 
evaluated. 

Other Youth Drug Use Findings: 

ONDCP believes we did not provide adequate discussion of studies that 
report findings contrary to those of Westat. Our report mentions two of 
the three studies that ONDCP identifies--the Longshore and Palmgreen 
studies. Our report does not mention the third study, Slater, because 
it focused on a different anti-drug media campaign approach and not on 
the ONDCP media campaign. Overall, these studies' findings are not 
necessarily "contrary" to Westat's findings. Rather, they assess small 
slices of the youth population or particular circumstances (such as 
other programs that could reinforce an anti-drug message) and find some 
positive results. The Westat national findings do not preclude the 
findings of positive results for some subpopulations of youth. The 
Palmgreen study, for example found a positive effect for the media 
campaign on high-sensation-seeking youth, but did not find an effect on 
non-high-sensation-seeking youth in the two moderate size communities 
in which the study was conducted. The distribution of these youth in 
the nationwide population could be consistent with both studies being 
correct. Our objective was to assess the Westat study as a national 
evaluation of the impact of the national campaign. In the Slater study, 
after being trained in the use of campaign media materials, leaders in 
each of eight communities that received a media campaign were allowed 
to develop their own media strategies and were able to use whatever 
materials they chose or developed on their own. This approach 
emphasized the flexibility to adopt different media strategies deemed 
appropriate by individual communities and not the use of a single 
national strategy. 

ONDCP expressed concern that we had not discussed Westat's hypothesis 
concerning why the campaign might have contributed to youth 
experimentation with marijuana. We are unable to draw a conclusion 
about this hypothesis based on Westat's report, nor do we have 
additional information upon which to base an assessment. ONDCP also 
faults our report for not discussing other potential competing 
explanations for the substantial downturn in teen drug use and increase 
in anti-drug attitudes. Although this is beyond the objectives of this 
report, we note that multiple other indicators of youth responsibility 
also seem to be trending in a positive direction at the same time that 
MTF reports declines in youth drug use. For example, from 1991 through 
1999, the teen pregnancy rate declined by 27 percent and from 1991 
through 2002, the teen birth rate fell 30 percent. Similarly, the 
number of juvenile homicides declined by 44 percent from 1993 to 2002, 
and the juvenile violent crime arrest rate fell by more than 40 percent 
from 1994 to 2003. All of these trends--including declines in drug use-
-could be related to broader environmental, familial, or other 
influences. The coincidence of these trends with drug use trends 
indicates that factors other than the campaign could be responsible for 
the decline in drug use and points to the necessity of trying to 
isolate the effects of the campaign, rather than relying upon simple 
correlations. 

Steps Taken to Remedy Potential Problems: 

ONDCP states that it has taken extensive "due diligence" steps that are 
briefly acknowledged in our report, but that our report "fails to 
acknowledge the thoroughness of our actions to identify, assess, and 
attenuate any possible negative consequences of the campaign once 
Westat reported the possibility of such an effect." Apart from those 
actions described in Westat's evaluation reports, a full discussion of 
the steps that ONDCP took in response to Westat's interim evaluation 
reports that highlighted the possibility of unintended negative 
consequences of exposure to the campaign on youth initiation of 
marijuana was not salient to our assessing whether Westat took 
appropriate steps to address the evaluation implementation challenges 
that it faced. However, Westat's findings for the period from 2002 to 
2004 showed that the campaign also was not effective after ONDCP took 
these steps. 

ONDCP Cites Major Changes in Campaign: 

ONDCP states that the campaign is substantially different from what it 
was when the last data were collected by Westat more than 2 years ago. 
We are not in a position to comment on ONDCP's new campaign ("Above the 
Influence"), launched in November 2005, as these current efforts are 
beyond the scope of our report and outside the time frame of the Westat 
data collection. At this time, neither we nor ONDCP have empirical 
information with which to assess this revised campaign. However, 
Westat's evaluation showed that neither the campaign as initially 
implemented nor the redirected campaign implemented after 2002 was 
effective. Hence, although a new and improved campaign may be 
effective, Westat's findings raise concerns about whether any campaign 
can affect youth drug use, especially since the lack of effect does not 
seem to be related to recognition of campaign ads, but rather to 
subsequent impact on attitudes and behaviors. Finally, ONDCP cites the 
receipt of awards from both the advertising and communications industry 
for its newest campaign. While laudable, these awards are not evidence 
that the new campaign will change youth drug attitudes and behavior. 
Only an independent evaluation can assess the current campaign's 
effectiveness. 

ONDCP Offers an Alternative Explanation for Counterintuitive Results: 

ONDCP stated that there is growing research evidence showing that 
asking people a question about their future behavior influences the 
subsequent performance of the behavior in question. ONDCP then 
indicates that the use of a panel design for the Westat study with 
repeated interviews of youth concerning drug attitudes and behaviors 
might, itself, have resulted in increased perceptions that drug use is 
widely pervasive among youth. If, during the course of the Westat 
study, ONDCP and NIDA, who acted as monitor for the study, felt that 
the study itself--that repeated interviews of youth by Westat 
concerning the campaign and drug attitudes and behavior--was resulting 
in a negative effect, it would have been appropriate for them to 
discontinue the study to avoid potential harm to subjects. Although 
ONDCP raised this issue in its comments to us, neither ONDCP nor NIDA 
mentioned this issue in any of our previous meetings specific to this 
engagement. 

ONDCP Takes Issue with the Timing of Our Review: 

ONDCP said that the "long delay" in receiving our assessment of the 
Westat report has prevented it from making progress on the next round 
of evaluation. We note that Westat's draft final report was not made 
available to us until spring 2005 (not 2 years ago as seems to be 
indicated in ONDCP's comments). The volume of reports from the 4˝-year 
study, and the complexity of the review required a great deal of time 
from our most skilled social scientists and statisticians. Time was 
required to ensure that our review of the Westat study was both 
comprehensive and correct. 

Points Concerning Our Matter for Congressional Consideration: 

ONDCP said that our matter for congressional consideration--that 
Congress consider limiting appropriations until ONDCP is able to 
provide credible evidence of the effectiveness of exposure to the 
campaign on youth drug use outcomes--offers insufficient detail 
concerning how to demonstrate satisfactory evidence of progress and 
that it was puzzled by our lack of recommendations to ONDCP for 
improving the campaign. Our mandate was to assess Westat's evaluation 
and to draw conclusions about the reliability of its findings so that 
Congress could make decisions about funding for the campaign, and 
developing suggestions for improvements to the media campaign itself 
was beyond our scope. In so doing, we focused on Westat's methods and 
efforts to address challenges in implementing the evaluation. Our 
matter for congressional consideration was intended to allow ONDCP to 
explore a number of approaches to providing credible evidence of 
campaign effectiveness to Congress. Our report clearly indicates that 
one approach is the one applied in the Westat evaluation, which is the 
focus of this report, but we do not want to rule out other approaches. 
At the same time, we acknowledge that providing such evidence is not 
easy. 

ONDCP Posits Consequences of Further Budget Cuts: 

ONDCP states that further budget cuts to the campaign could have far- 
reaching and unfavorable consequences in youth drug use. Given that the 
Westat findings show that the campaign was not having a positive 
impact, we found no evidence that a reduction in campaign 
advertisements would have a negative impact. ONDCP cites the 2005 MTF 
as an indicator of media campaign effectiveness by indicating that the 
reduction in anti-drug messages has resulted in a flattening of 8th 
graders' perception of risk. Again, as ONDCP has indicated, the 
relationship cannot be assessed with MTF because it does not ask 
respondents about their exposure and reactions to the messages of the 
media campaign that can then be linked to their drug-related attitudes 
and behaviors. 

Failure to continue the media campaign's efforts, according to ONDCP, 
is "raising a white flag to those who favor drug legalization, with the 
expectation that youth drug use soon would begin to rise, reversing 
years of hard-earned positive news." In our view, on the other hand, 
continuation of programs that have been demonstrated not to work 
diverts scarce resources from programs that may be more effective. 

We are sending copies of this report to other interested congressional 
committees and the Director of the Office of National Drug Control 
Policy. We will make copies of the report available to others upon 
request. In addition, the report will be available at no charge on 
GAO's Web site at [Hyperlink, http://www.gao.gov]. 

If you or your staff have any questions about this report, please 
contact either Nancy Kingsbury at 202-512-2700 or by e-mail at 
KingsburyN@gao.gov or Laurie Ekstrand at 202-512-8777 or by e-mail at 
EkstrandL@gao.gov. Contact points from our Offices of Congressional 
Relations and Public Affairs may be found on the last page of this 
report. Key contributors are listed in appendix III. 

Sincerely, 

Signed by: 

Nancy Kingsbury, Managing Director: 
Applied Research and Methodology: 

Signed by: 

Laurie E. Ekstrand, Director: 
Homeland Security and Justice: 

[End of section] 

Appendix I: Westat's Methods for Addressing Evaluation Implementation 
Issues: 

This appendix provides additional details about how Westat's addressed 
evaluation implementation issues related to the coverage of the 
National Survey of Parents and Youth (NSPY), sample attrition, and its 
analytic methods. 

Coverage in the NSPY: 

The NSPY was a nationwide household survey of youth aged 9 to 18 and 
their parents. Westat used a dual-frame sampling frame--or list of the 
members of the population from which the sample was ultimately 
selected. One frame--the area frame--consisted of housing units that 
had been built by late 1991; the second frame--the building permit 
frame--consisted of building permits issued between January 1990 and 
December 1998 for new housing.[Footnote 31] Combined, these frames 
constituted an estimated 98 percent of dwelling units nationwide that 
existed by the end of 1998. 

A household had to meet two criteria in order to be eligible to be 
included in the NSPY sample: It had to (1) contain children within a 
specified age group and (2) be a housing unit that was built before 
April 1, 1990, was a mobile home, or was selected from a roster of 
building permits for new housing units issued between January 1990 and 
December 1998. To identify households that met these conditions, Westat 
drew a sample of dwelling units and from this sample it screened 
households to determine their eligibility for inclusion in the NSPY, 
that is, whether a household contained children in a specified age 
group, where the specified age groups were children aged 9 through 13, 
12 and 13, or 9 through 18. 

According to estimates provided by Westat, after completing enrollment 
in the NSPY--which occurred during waves 1 through 3--the NSPY sample 
covered more than an estimated 95 percent of occupied dwelling units 
(households) nationwide. From its sample of occupied dwelling units, 
Westat developed rosters of households that were believed to contain 
youth in the target age range. At this second stage of sample 
enumeration, Westat experienced a drop-off in the coverage of 
households that were believed to be eligible for inclusion in the 
sample. The number of eligible households enumerated in the NSPY was 30 
percent smaller than the number expected from the 1999 Current 
Population Survey (CPS) data. 

According to Westat, coverage losses in the NSPY could have occurred 
for several reasons: (1) because an interviewer may have decided to 
classify a household as an ineligible household rather than as a 
nonresponding household, (2) because the household respondent took cues 
from the screening questions to avoid selection into the sample by 
giving an incorrect answer, or (3) because the doorstep enumeration 
process was considered to be intrusive. Westat reported that it could 
not conclusively rule out the first explanation for coverage losses. 
However, it undertook sample validation procedures that examined 
whether ineligible households in the recruitment waves were 
misclassified, and it found none. Neither Westat nor the National 
Institute on Drug Abuse reported that undercoverage was primarily due 
to respondents avoiding selection into the sample by taking cues from 
the screening questions and giving incorrect answers as a way to avoid 
selection into the sample. Overall, Westat reported that the main 
reason for undercoverage was the rostering component of the survey, 
which required actual entry into the home, and led to "a great many 
respondents" asking the interviewer to come back at a later date, only 
to repeat the request when the interviewer reappeared. Westat inferred 
that this represented passive refusal to participate. Therefore, 
according to Westat, most of the coverage losses occurred during the 
doorstep screening process in which simple, focused screening questions 
about the composition of the household were used to identify households 
from which to sample eligible youth. 

NSPY and CPS Comparisons of Distributions on Analyzed Variables: 

In response to questions from us, Westat provided data that indicated 
that the coverage losses in the NSPY did not result in differences in 
the estimated distributions of population characteristics from the NSPY 
as compared with those estimated from the CPS data. In other words, the 
distributions of characteristics of eligible households with youth 
included in the NSPY were broadly consistent with a variety of 
corresponding distributions from the 1999 Current Population Survey. 

The comparisons of NSPY-estimated populations to CPS-estimated 
populations were based on weighted NPSY estimates, where the weights 
adjusted for nonresponse at the doorstep and household enumeration 
(roster) stages, and the weights also reflected the differential 
probabilities of retaining a household for the NSPY depending on the 
screener group to which it was applied. These weights were calculated 
prior to Westat's poststratification calibration techniques, which 
brought the estimated NSPY population totals into line with the 
estimated CPS population totals. Hence, if upon using the weights based 
only on the probability of selection and nonresponse adjustments, the 
population characteristics in the NSPY differed widely from those 
derived from the CPS, this would constitute evidence of potential bias 
in the NSPY sample due to undercoverage. 

Westat compared NSPY and CPS distributions for each of the three 
enrollment waves of the NSPY (waves 1 through 3) on several variables, 
including the race/ethnicity of the householder and the presence of 
males 28 years of age or older, the distribution of eligible households 
by the age of the youth in the household, the age and gender 
distributions of youth, and the age distributions of youth by race and 
ethnicity. Each of these comparisons involved discrete subgroups within 
the focused subpopulation of the NSPY. The largest differences between 
the NSPY and CPS estimates arose in the comparison of the distributions 
by race/ethnicity of household and the presence of a male 28 years of 
age or older in the household. Some of these differences could also 
arise from sampling variance, as both the NSPY and CPS estimates are 
based on samples that are subject to sampling errors. Although Westat 
did not provide sampling errors with the estimates that it provided to 
us, some of the differences in distributions could be apparent, as 
opposed to real, differences, in statistical terms. 

Undercoverage in the NSPY and Other Widely Known and Used Longitudinal 
Surveys: 

Coverage issues are not an uncommon problem with surveys that focus on 
relatively small subpopulations within a larger population, such as 
occurred with the NSPY's focus on youth aged 9 to 18. The NSPY's target 
population of households with youth aged 9 to 18 focused on a 
subpopulation that, according to 1999 CPS data, constituted about 25 
percent of the roughly 104 million households in the United States. 

The estimated extent of undercoverage of eligible youth in the NSPY was 
comparable to the extent of undercoverage in other well-known and 
widely used longitudinal surveys. Both the National Longitudinal Survey 
of Youth (NLSY)--sponsored by the Bureau of Labor Statistics--and the 
National Immunization Survey of Children (NIS)--sponsored by the 
National Immunization Program (NIP) and conducted jointly by the NIP 
and the National Center for Health Statistics of the Centers for 
Disease Control and Prevention--focus on specific subpopulations, and 
both experienced undercoverage that was comparable to that of the NSPY. 
The 1979 NLSY is a nationally representative sample of men and women 
born in the years 1957 to 1964 who were ages 14 to 22 when first 
interviewed in 1979. It had a coverage rate of 68 percent. The 1979 
NLSY has been widely used and cited to examine a wide variety of policy 
issues. As documented in the National Longitudinal Surveys' annotated 
bibliography, about 3,100 journal articles, working papers, monographs, 
and other research documents have been catalogued as having used the 
1979 NLSY data. The target population for the NIS is children between 
the ages of 19 and 35 months living in the United States at the time of 
the interview, and it has been conducted annually since 1994. The 
survey involves the selection of a quarterly probability sample of 
telephone numbers, and the coverage has been about 20 percent lower 
than estimated by two other benchmark surveys. Survey data are used 
primarily to monitor immunization coverage in the preschool population 
in the nation and to provide national, state, and selected urban area 
estimates of vaccination coverage rates for these children. 

Sample Attrition across NSPY Interview Rounds: 

In the NSPY, respondents initially recruited into the sample were to be 
tracked for three additional survey rounds that covered about a 3-year 
period following the recruitment round. By the final survey round of 
the NSPY, the cumulative response rate--the percentage of youth or 
parents in eligible households that completed all four interviews-- 
reached between 50 percent and 55 percent. These cumulative response 
rates after four survey rounds were determined largely by the response 
rates during the enrollment waves, as postenrollment, Westat was able 
to track, contact, determine eligibility for reinterview, and complete 
interviews for between 82 percent and 94 percent of previously 
interviewed respondents between two successive interview waves. The 
response rates achieved for the first three survey waves--the 
enrollment waves--were generally similar. Specifically, about 74 
percent to 75 percent of the dwelling units determined to be eligible 
for the survey in waves 1 through 3 completed the household enumeration 
(or rostering of youth). After obtaining consent to conduct interviews 
from parents and youth, interviewers completed extended interviews-- 
that is, completed the full NSPY questionnaire--with about 91 percent 
of the sampled youth in each of waves 1 through 3. Among sampled 
parents, about 88 percent gave consent and completed extended 
interviews in the enrollment waves. (See table 2.) 

Table 2: NSPY Survey Rounds and Response Rates, Sampled and Surveyed 
Youth: 

Rounds and stages of sampling:  Survey waves: Wave 1; Survey waves: 
Wave 2; Survey waves: Wave 3. 

Rounds and stages of sampling: Round 1: enrollment waves: 
* Percentage of sampled dwelling units for which eligibility was 
determined; 
Survey waves: Wave 1: 95.1%; 
Survey waves: Wave 2: 95.7%; 
Survey waves: Wave 3: 95.5%. 

Rounds and stages of sampling: Round 1: enrollment waves: 
* Percentage of eligible dwelling units completing household roster; 
Survey waves: Wave 1: 74.4%; 
Survey waves: Wave 2: 74.6%; 
Survey waves: Wave 3: 75.3%. 

Rounds and stages of sampling: Round 1: enrollment waves: 
* Percentage of youth completing interview; 
Survey waves: Wave 1: 90.3%; 
Survey waves: Wave 2: 91.9%; 
Survey waves: Wave 3: 91.2%. 

Rounds and stages of sampling: Round 1: enrollment waves: 
* Cumulative (overall) response rate, enrollment waves; 
Survey waves: Wave 1: 63.8%; 
Survey waves: Wave 2: 65.5%; 
Survey waves: Wave 3: 65.5%.  

Rounds and stages of sampling: Round 2: first follow-up: 
* Percentage of dwelling units (from prior wave) refielded for follow-
up; 
Survey waves: Wave 4: 94.2%; 
Survey waves: Wave 5: 92.9%. 

Rounds and stages of sampling: Round 2: first follow-up: 
* Percentage of refielded dwelling units for which eligibility was 
determined; 
Survey waves: Wave 4: 86.8%; 
Survey waves: Wave 5: 93.8%. 

Rounds and stages of sampling: Round 2: first follow-up: 
* Percentage of youth completing interview; 
Survey waves: Wave 4: 93.5%; 
Survey waves: Wave 5: 93.6%. 

Rounds and stages of sampling: Round 2: first follow-up: 
* Cumulative (overall) response rate; 
Survey waves: Wave 4: 54.1%; 
Survey waves: Wave 5: 58.4%. 

Rounds and stages of sampling: Round 2: first follow-up: 
* Follow-up (conditional) longitudinal response rate; 
Survey waves: Wave 4: 82.2%; 
Survey waves: Wave 5: 88.8%. 

Rounds and stages of sampling: Round 3: second follow-up: 
* Percentage of dwelling units (from prior wave) refielded for follow-
up; 
Survey waves: Wave 6: 85.1%; 
Survey waves: Wave 7: 89.8%. 

Rounds and stages of sampling: Round 3: second follow-up: 
* Percentage of refielded dwelling units for which eligibility was 
determined; 
Survey waves: Wave 6: 93.1%; 
Survey waves: Wave 7: 92.8%. 

Rounds and stages of sampling: Round 3: second follow-up: 
* Percentage of youth completing interview;
Survey waves: Wave 6: 94.7%; 
Survey waves: Wave 7: 93.8%. 

Rounds and stages of sampling: Round 3: second follow-up: 
* Cumulative (overall) response rate; 
Survey waves: Wave 6: 53.1%; 
Survey waves: Wave 7: 56.0%. 

Rounds and stages of sampling: Round 3: second follow-up: 
* Follow-up (conditional) longitudinal response rate; 
Survey waves: Wave 6: 93.4%; 
Survey waves: Wave 7: 91.6%. 

Rounds and stages of sampling: Round 4: third follow-up: 
* Percentage of dwelling units (from prior wave) refielded for follow-
up; 
Survey waves: Wave 8: 78.7%; 
Survey waves: Wave 9: 83.5%. 

Rounds and stages of sampling: Round 4: third follow-up: 
* Percentage of refielded dwelling units for which eligibility was 
determined; 
Survey waves: Wave 8: 95.9%; 
Survey waves: Wave 9: 94.8%. 

Rounds and stages of sampling: Round 4: third follow-up: 
* Percentage of youth completing interview; 
Survey waves: Wave 8: 94.0%; 
Survey waves: Wave 9: 94.3%. 

Rounds and stages of sampling: Round 4: third follow-up: 
* Cumulative (overall) response rate; 
Survey waves: Wave 8: 50.2%; 
Survey waves: Wave 9: 53.4%. 

Rounds and stages of sampling: Round 4: third follow-up: 
* Follow-up (conditional) longitudinal response rate; 
Survey waves: Wave 8: 92.4%; 
Survey waves: Wave 9: 93.4%. 

Source: Westat, June 2005. 

[End of table] 

Across the three follow-up rounds of the NSPY, Westat achieved between 
an 82 percent and a 94 percent longitudinal response rate. Follow-up 
required that respondents be tracked over time and across places, as 
persons enrolled in the sample could move, and their eligibility for a 
follow-up interview had to be determined. For example, youth who turned 
19 years of age between survey rounds would no longer be eligible for 
reinterview, as they were beyond the target age of the campaign. 
Efforts to track individuals prior to the second survey round included 
verifying address change information with the U.S. Postal Service and 
obtaining location information from a national database company. Westat 
obtained updated location information from these sources, and telephone 
interviewers placed calls to these households to verify the identity of 
respondents. According to Westat, a high proportion of the households 
that moved were contacted and respondents verified their new addresses. 
During the third and fourth survey rounds, Westat used procedures to 
track and verify addresses that were similar to those used to track 
respondents from the first to second survey rounds, although Westat 
modified these procedures as necessary. The key eligibility requirement 
for youth for a follow-up interview was the youth had to be 18 years of 
age or younger at the time of the interview. 

For the first follow-up round--waves 4 and 5--Westat located 
individuals and determined eligibility for 92 percent of the youth and 
92 percent of the parents who completed an initial interview during the 
first round of the survey--that is, in waves 1, 2, and 3, and of these 
youth who were still eligible, 94 percent completed an interview. Among 
parents from the first round who were tracked and determined to be 
eligible in the second round, 92 percent completed a second round 
interview. In the third and fourth survey rounds of the NSPY, between 
96 percent and 97 percent of the youth and parents who had completed 
prior round surveys were tracked and determined to be eligible, and of 
these, the youth response rates were 96 percent and the parent rates 
were 95 percent. 

Comparisons of Respondents and Nonrespondents across NSPY Survey Waves: 

Even with the relatively high follow-up response rates that Westat 
achieved, it is possible that respondents could differ from 
nonrespondents in follow-up rounds, and if so, the NSPY estimates of 
the effects of exposure on outcomes would be biased. Westat provided 
data that compared nonrespondents to the respondents across the three 
enrollment waves, indicating that with some differences, nonrespondents 
were generally similar to respondents with respect to characteristics 
that might affect survey outcomes. Nonrespondents were compared to 
respondents on gender, age at interview, whether both parents were in 
the household, the number of youth in the household, the type of 
household dwelling, and the type of area in which the household was 
located. For example, apart from the three differences below, 
nonrespondents and respondents were similar in characteristics across 
survey waves: In the three enrollment waves, nonrespondents were 
proportionately older youth than respondents; in waves 2 and 3, there 
were proportionately more youth living in cities among nonrespondents 
than respondents; and in wave 1, there were proportionately more youth 
in the building permit sample among nonrespondents than respondents. 

Differences in Sampling Methodologies between NSPY and MTF: 

Westat compared estimates of drug-use prevalence from the NSPY data 
with those obtained from other national surveys such as Monitoring the 
Future (MTF). While the NSPY estimates of marijuana use prevalence 
differ over some periods covered by the NSPY from those derived from 
the MTF survey of youth in school, differences between the two surveys' 
sampling frames and methodologies mean that direct comparisons between 
the two surveys must be made with caution and must take the 
methodological differences into account. Specifically, MTF showed a 
decline in marijuana use for some teenage groups during the 2000 to 
2002 period, while the NSPY showed the increases reported above. 
However, the difference in drug use rates reported from the two surveys 
could plausibly arise from differences in the sampling frames. The MTF 
sampling frame covers only youth who are in school and not those who 
drop out of school, who are truant on the survey day, or who are 17-and 
18-year-olds who have graduated from high school. To the extent that 
high school dropouts and truants have more involvement with drugs than 
those who stay in school, the MTF estimates of drug use may 
underrepresent drug use among all youth of high school age. By 
comparison, the NSPY household survey includes youth who are not 
enrolled in school in its sampling frame. To the extent that dropping 
out of high school is correlated with drug use, and given that dropouts 
are excluded from the MTF sampling frame, differences in drug use 
between MTF and NSPY could reflect the fact that youth enrolled in high 
school reported drug use at different rates from all youth in the 
general population covered by the NSPY, which would include dropouts 
who may be at higher risk of using drugs. 

The Capacity of the NSPY to Detect Reasonably Small Effects: 

One challenge in designing surveys to evaluate changes in outcomes as 
the result of an intervention lies in selecting a sample with 
sufficient power to detect differences between groups--including the 
same individuals at two points in time--or significant associations 
among variables, such as between levels of exposure to the campaign and 
outcomes. Sample size is a major factor determining a study's power to 
detect differences, and while larger sample sizes will generally allow 
researchers to detect smaller differences over time, as the size and 
power of a sample to detect changes increases, so too generally does 
its cost. 

In consultation with the National Institute on Drug Abuse (NIDA), 
Westat chose to compute power for analyses of annual change in a 
prevalence statistic--that is, change in the percentage of a population 
that reported an outcome. For purposes of its power analysis, Westat 
chose to assume different baseline prevalences for parents and for 
youth of all ages and to assume that the study should be able to detect 
reliably declines of specified sizes. For example, for youth of all 
ages, Westat assumed a baseline prevalence of 10 percent and determined 
the power of its sample for detecting a minimum downswing in an 
outcome--such as past-month drug use--of 2.3 percentage points over a 
year.[Footnote 32] The power of the sample to detect this difference 
was well within conventional power criteria.[Footnote 33] 

As reported above, the sizes of differences that Westat's sample could 
detect were consistent with the Office of National Drug Control 
Policy's (ONDCP) goals for the campaign. In early meetings on the 
design of the evaluation of the media campaign, ONDCP officials 
reported that ONDCP had a specific Performance Measures of 
Effectiveness (PME) system and that the campaign was embodied within 
the first goal of the National Drug Strategy, which was to "educate and 
enable America's youth to reject illegal drugs as well as the use of 
alcohol and tobacco." Under this goal, ONDCP's PME proposed targets for 
reducing the prevalence of past-month use of illicit drugs and alcohol 
among youth from a 1996 base year: by 2002, reduce this prevalence by 
20 percent, and by 2007, reduce it by 50 percent. ONDCP officials 
further identified specific targets for the media campaign, again with 
respect to a base year of 1996: by 2002, increase to 80 the percentage 
of youth who perceive that regular use of illicit drugs, alcohol, and 
tobacco is harmful; and by 2002, increase to 95 the percentage of youth 
who disapprove of illicit drug, alcohol, and tobacco use. To achieve a 
goal of 80 percent of youth who perceive that regular use of marijuana 
is harmful would require increasing the 1996 baseline percentage of 
youth perceiving marijuana as harmful from 60 percent, as measured by 
MTF, or by about 3.3 percentage points per year from 1996 to 2002. 
Westat's sample had sufficient power to detect this amount of annual 
change in youth attitudes. 

The power of the NSPY to detect changes in outcomes due to exposure to 
the campaign also presumes that it was possible to accurately measure 
and characterize exposure to the campaign by the reported number of 
advertisements recalled by respondents. While the general question of 
how exposure to advertisements affected respondents was beyond the 
scope of the evaluation, if by exposure is meant a recognition-based 
task--or encoded exposure--then the NSPY measures of exposure can be 
viewed as valid. According to communications researchers, often what is 
of interest to campaign planners and evaluators is whether the 
presentation of campaign content generates at least a memory trace in 
individuals. At this point, a potential audience member can be said to 
have engaged the campaign's presentation in a meaningful sense, and 
this is what is meant by encoded exposure. To measure exposure to the 
campaign for both youth and parents, NSPY interviewers asked 
respondents about their recall of specific current or very recent 
television and radio advertisements.[Footnote 34] 

There was variation in recall of advertisements by both youth and 
parent respondents, and this type of variation is needed in order to 
examine associations between levels of exposure and outcomes. For 
example, for the entire campaign, youth reported a median of 12 
exposures per month, and 76.7 percent reported 4 or more exposures per 
month. Comparatively few youth--about 6 percent--reported less than 1 
exposure per month. Youth recall of specific exposure also varied, as 
41.2 percent of youth reported 12 or more television exposures per 
month throughout the campaign while reporting a median of 4.4 exposures 
to television advertisements. Additionally, Westat's measures of 
exposure and outcomes have demonstrated sensitivity to detect favorable 
campaign effects among parents. 

Westat's test for associations between exposure and outcomes--the gamma 
coefficient--was an ordinal test statistic for whether two variables 
(e.g., exposure and marijuana use initiation) have a montonic, but not 
necessarily a linear, relationship. Therefore, were there nonlinear 
relationships, its test would have allowed for them. Finally, nonrandom 
measurement error in the measure of exposure is unlikely to have biased 
estimates of campaign effects, as if the nonrandom measurement error 
were constant, it would not affect measures of association, and if it 
was not constant, it would be addressed by Westat's statistical 
methods. 

Westat Methods to Measure Outcomes: 

Westat measured a variety of outcomes for youth and parents and took 
steps to ensure that the measures were consistent with existing 
research. The youth questionnaires included numerous questions that 
were designed to measure exposure to the campaign advertisements and 
other anti-drug messages. The youth question domains included exposure 
propensity to media; current and past use of tobacco, alcohol, 
marijuana, inhalants, and Ecstasy; past discussions with and 
communication of anti-drug messages from parents and friends; 
expectations of others about respondent's drug use; knowledge and 
beliefs about the positive and negative consequences of drug use; 
exposure to campaign messages; family and peer factors; personal 
factors; and demographic information. Westat used two separate 
questionnaires for youth of different ages; one questionnaire was used 
for children (aged 9 to 11) and another one was used for teens (aged 12 
to 18). 

The NSPY parent questionnaire also included numerous questions that 
were intended to measure parents' exposure to the campaign's messages 
and other anti-drug messages. The question domains for parents included 
media consumption; past discussions with child about drug attitudes and 
avoidance strategies; past child monitoring behaviors; self-efficacy of 
discussing drugs with child and monitoring of child's actions; belief 
that the child is at risk of drug use; belief that drug use has bad 
consequences; exposure to the campaign's advertising, including brand 
recognition; parent's own current and past use of tobacco, alcohol, and 
drugs; and demographic information. 

Westat followed generally accepted procedures in developing the survey 
instruments for the NSPY by using information from a prototype prepared 
by NIDA and using information from other surveys that addressed youth 
drug use and prevention. Prior to the phase III evaluation, and in 
preparation for the NSPY, NIDA convened an expert panel to assist in 
the development of the youth and parent questionnaires. The panel, 
which consisted of experts in adolescent drug use prevention and 
parenting behaviors, drafted NSPY survey questionnaires for children, 
teens, and parents, and NIDA shared these prototypes with Westat at the 
beginning of Westat's evaluation contract. In developing the final 
questionnaire for the NSPY, Westat created a questionnaire development 
team consisting of evaluation experts. In developing the final NSPY 
questionnaires, the Westat team reviewed NIDA's prototype and other 
surveys. 

Westat measured youth drug use by self-reported data on use. We have 
previously cautioned about limitations associated with self-reported 
data on youth drug use.[Footnote 35] Additionally, the National 
Research Council (NRC) of the National Academy of Sciences also has 
pointed out limitations associated with self-reported drug use in 
national surveys such as the National Survey of Drug Use and Health 
(NSDUH) and MTF.[Footnote 36] As NRC has pointed out, while self- 
reported data on drug use may have limitations for estimating the 
actual levels of use at a particular point in time, they may not suffer 
from these same limitations when they are used to assess changes in use 
over time, unless there is reason to believe that attitudes about drug 
use change in ways that affect respondents' willingness to honestly 
report drug use, or stigma. 

Specifically, if there is a stigma associated with self-reporting drug 
use, that stigma may affect the levels of use reported, as some have 
argued that the propensity of respondents to give valid responses may 
be affected by social pressures. In particular, the incentive to give 
false negative reports may increase over time if drug use becomes 
increasingly perceived as harmful or socially unacceptable. Using data 
from NSDUH and MTF, NRC showed an inverse relationship between the 
percentages of respondents who either disapproved of illegal drug 
consumption or perceived it to be harmful. Thus, as stigma increased, 
self-reported drug use decreased. As NRC cautioned, one could interpret 
this relationship as indicating that changes in stigma are associated 
with changes in invalid reporting, or as stigma increases, false 
negative reports increase, rather than necessarily indicating that as 
stigma increases, drug use decreases. 

The NRC analysis leads to two inferences: First, if social stigma 
remains constant over time, changes in the propensity to give valid 
responses would be unaffected and estimates of change in self-reported 
drug use would not be biased by social stigma. For the evaluation 
results, this would imply that its measures of changes in self-reported 
drug use would provide valid measures of changes in use, so long as 
factors other than stigma did not affect the propensity to self-report 
use. Second, if the social stigma associated with reporting drug use is 
inversely related to disapproval of illicit drug use or increased 
perceptions that it is harmful, then the estimates of self-reported 
drug use are likely to decrease as a result of the stigma. According to 
results from the evaluation, trends in youth attitudes and beliefs 
about illicit drugs changed significantly over the entire campaign in a 
direction that was favorable to the campaign. Specifically, the trends 
in youth attitudes and beliefs about illicit drug use meant that youth 
were more likely to believe, as the campaign went on, that use of 
illicit drugs was likely to have negative consequences. Alternatively, 
the social stigma associated with drug use increased over time. If the 
relationship between stigma and reporting that NRC found held and 
applied to the data in the evaluation of the campaign, this would imply 
that the increased stigma associated with drug use would lead to 
decreases in self-reports of drug use over time. 

Westat's Analytic Methods: 

To control for the many factors that could have influenced both 
exposure and outcomes independently of, or in conjunction with, the 
campaign, Westat used propensity scoring methods to match individuals 
based on numerous measured attributes and to create groups of 
individuals who differed on their underlying propensity to be exposed 
to different levels of campaign advertisements. A propensity score is a 
weighted sum of the individual effects of variables in a model that 
predicts the likelihood of exposure to campaign messages. Westat's 
propensity scoring methods resulted in the creation of groups of 
individuals who were statistically similar on exposure propensities. 
These groups can be considered as statistical analogues to randomly 
assigning individuals to different levels of exposure. After creating 
these groups, Westat then analyzed outcomes between the groups having 
different propensities to be exposed to campaign messages. 

Westat used ordinal logit models to estimate the chances of being 
exposed, where exposure was measured alternatively as a three-or four- 
level variable--e.g., low, medium, or high exposure.[Footnote 37] 
Westat used a myriad of variables to predict exposure levels in both 
the youth and parent models. For example, in the youth models, the 
propensity score models included measures of demographic attributes, 
educational attainment and educational aspiration, family and parent 
background, parent consumption of television and other media, income 
and employment, reading habits, Internet usage, location of residence 
in urban areas, among other variables. After estimating models, Westat 
also assessed the balance of variables in its propensity models. For 
propensity models to remove the effects of confounding variables from 
the association between exposure and response, it is necessary that the 
population means of the confounder variables not vary across exposure 
levels. If a confounder is successfully balanced, then it will have the 
same theoretical effect across all exposure levels. 

The net result of the propensity scoring models is to provide each 
individual with a score that reflects the individual's propensity to 
recall advertisements based upon a weighted sum of all of the variables 
in the model. Therefore, while two individuals may differ on the 
likelihood that a particular variable affects their chances of being 
exposed to messages or on their levels of a certain variable--such as 
age or education--they could be similar in their overall propensity to 
be exposed to campaign messages if the differential effects of any 
individual variables sum to the same total propensity. 

In order for the results of propensity methods to be valid, it is 
important that the propensity scoring models include all relevant 
variables that could otherwise explain differences in both exposure and 
outcomes. Propensity score models can adjust only for confounding 
variables that are observed and measured. In other words, they are 
built upon the assumption that all relevant variables are measured and 
controlled for. If an important variable is omitted from the propensity 
model, the results of analyses may be affected. Westat made reasonable 
attempts to identify and control for a variety of confounding 
variables, include them in its models, and reduce bias. 

[End of section] 

Appendix II: Comments from the Office of National Drug Control Policy: 

Executive Office Of The President Office Of National Drug Control 
Policy: 
Washington, D.C. 20503: 

August 10, 2006: 

Mr. David M. Walker: 
Comptroller General: 
Government Accountability Office: 
441 G Street, NW: 
Washington, DC 20548: 

Dear Mr. Walker: 

I am writing in response to your request for comments on the Government 
Accountability Office (GAO) report (GAO-06-818) entitled, "Contractor's 
National Evaluation Does Not Find That the Youth Anti- Drug Media 
Campaign Was Effective in Reducing Youth Drug Use." I appreciate the 
opportunity to respond to the findings and recommendation made in the 
report and to provide you with additional information on how the Office 
of National Drug Control Policy is dedicated to the task of reducing 
youth drug use through media outreach. 

I have a number of concerns with the Westat findings and your report's 
assessment of them. In brief. 1) Westat's evaluation is ill-suited to 
judge the impact of an ad campaign; 2) the findings are now more than 2 
years old and have limited relevance; 3) conflicting evidence from 
other research is given minimal attention; 4) the Campaign has 
undergone major changes - with encouraging results to date; 5) our "due 
diligence" efforts to address the potential for harm are not well 
characterized; and 6) your recommendation to Congress offers 
insufficient detail to demonstrate satisfactory evidence of progress. 
Finally, I have identified probable consequences of further cuts to the 
Campaign budget that might be made pursuant to your recommendation to 
Congress. 

Westat Evaluation is Ill-Suited to Judging Impact of an Advertising 
Campaign: 

Major advertisers, who spend billions of dollars annually on 
advertising in the U.S. and abroad, do not attempt to establish a 
causal relationship between advertising exposure and product sales, but 
evaluate the success of their advertising campaigns by rigorous testing 
of individual ads prior to air (which the Campaign has consistently 
done with increasing rigor), monitoring the performance of the ads once 
aired (which the Campaign has done) and by carefully developing 
correlations between various advertising messages, levels of media 
expenditure, and consumer attitudes and behavior. 

Establishing a causal relationship between exposure and outcomes is 
something major marketers rarely attempt because it is virtually 
impossible to do - particularly if both a pre-advertising baseline and 
/ or unexposed control groups are lacking. As a senior market 
researcher at a major advertiser has recently said, "Even when campaign 
design includes a media blackout region as a control (inappropriate for 
a public-service campaign), non-measured factors make it impossible to 
isolate the effect of a single advertisement or advertising campaign." 

This is one reason why the "truth" anti-tobacco advertising campaign, 
acclaimed as a successful initiative in view of the significant 
declines we've seen in teen smoking, did not claim to prove a causal 
relationship between campaign exposure and smoking outcomes, reporting 
instead that the campaign was associated with substantial declines in 
youth smoking. They relied instead on the correlations between campaign 
weight levels in various markets (they were able to vary advertising 
weight by market) and teen behavior in those markets as tracked by the 
University of Michigan's "Monitoring the Future" survey (see American 
Journal of Public Health, March 2005, "Evidence of a Dose-Response 
Relationship Between Antismoking Ads and Youth Smoking Prevalence"). 

Consequently, we take issue with the fundamental method pursued by 
Westat and GAO, and therefore, believe that the study's findings are 
deeply flawed. 

Findings are "Old News" 

Your report's findings do not come as a surprise to us, for each year 
as preliminary results have been made known to us, to Congress, and to 
the public through media coverage, we have studied the reports and have 
sought to improve the performance of the Campaign, using these and 
other data that we have available. We have responded when such 
findings, although not conclusive, were released to the public or to 
researchers, and have dealt with criticisms of the Campaign from 
adversaries, including those who advocate the legalization of drugs. 
And we have periodically needed to place these findings in context, 
especially because all major youth surveys report declining teen drug 
use, including Monitoring the Future, which documents a 19% decline in 
current illegal drug use among 8tH 10` and 12`H graders combined over 
the past four years, and a 21% decline in marijuana use from 1998 to 
2005 among these youth. 

Conflicting Evidence from Other Research: 

Although two recent studies that report findings contrary to Westat 
(Longshore and Palmgreen) are cited, no assessment of their importance 
is provided. GAO does not explain why such studies are not given more 
credence. For example, Longshore examined the possibility of 
synergistic effects between in-school drug education and the Campaign, 
and concludes that results ".showed that marijuana use in the past 
month was significantly less likely among adolescents who received both 
the ALERT Plus curriculum and weekly exposure to the Campaign's anti- 
drug media messages." This has significant implications nationally, and 
should not be dismissed. Palmgreen et al, whose paper has now been 
accepted for publication in the prestigious peer-reviewed American 
Journal of Public Health, concluded that the Campaign's marijuana 
initiative "reversed upward developmental trends in high sensation 
seeking 30-day marijuana use and significantly reduced positive 
marijuana attitudes and beliefs in this at-risk population." 

Another recent peer-reviewed journal article (Slater et al, "Combining 
in-school and community-based media efforts: reducing marijuana and 
alcohol uptake among younger adolescents," in Health Education Research 
- Theory and Practice, September 2005) which evaluated the results of 
an in-school media campaign in 16 cities (8 campaign cities and 8 
control communities) with a similar strategy and brand to our new Above 
the Influence campaign, has shown very positive results. The authors 
conclude that "substance use uptake for youth in treatment communities 
was half or less than that of control communities" and that 
"effectiveness did not depend on the presence of an in-school 
prevention curriculum." 

Your analysis thoughtfully acknowledges the serious challenges that 
Westat faced, due to a lack of a baseline and control group, the 
difficulties of retaining youth and parents in the study over the 
years, and the need to sort out other possible influences. These 
complexities add support to your expressed view that ".virtually all 
social science research is imperfect." 

Westat, in its assessment of why the Campaign might potentially have 
contributed to youth experimentation with marijuana, seems to rely on 
the argument that frequent exposure to the Campaign's anti-drug 
messages leads youth to conclude that most youth are, in fact, using 
marijuana, and in an effort to "fit in"with their peers, they decide to 
try marijuana. No theoretical basis for such an argument is provided in 
your report, nor is there any critique of the Campaign's underlying 
theory, the well-tested Theory of Reasoned Action. Therefore, this 
reported effect is wholly counterintuitive because, by Westat data, it 
is clear that youth are seeing the Campaign's anti-drug ads, seeing 
them frequently, recalling and assessing them evermore favorably, 
holding ever-stronger anti-drug attitudes, receiving fewer offers of 
marijuana, and overall using drugs (including marijuana) less and less. 
No competing explanation for the substantial downturn in teen drug use 
or increase in anti-drug attitudes is offered. 

Further, key studies, including both Monitoring the Future (MTF) and 
the Partnership Attitude Tracking Study (PATS --conducted by Roper for 
the Partnership for a Drug-Free America) report positive attitude 
changes. PATS data reveal that in the past few years more youth are 
likely to say most teens don't smoke marijuana, and those who say they 
have friends who smoked marijuana declined substantially. Further, 
according to MTF, teens are increasingly likely to disapprove of trying 
marijuana, and these higher rates of disapproval are associated with 
lower rates of current use, especially among 10`H graders, the core 
target audience of the Campaign. And in an analysis conducted of 
National Survey of Drug Use and Health (NSDUH) data, youth who reported 
having seen or heard media prevention messages in the past year were 
significantly less likely to report illicit drug use. And some states 
and communities are observing that as teen drug use is declining, 
student surveys report an increase in the number of youth who say their 
peers disapprove of drug use and attribute the good news in part to 
anti-drug media messages (Coalition for a Drug Free Greater 
Cincinnati). Your report makes no claims that other media campaigns, or 
in-school drug education (which youth say they are getting less and 
less of) or other influences are responsible for these substantial 
shifts in beliefs, attitudes, and behaviors. 

"Due Diligence" Steps Taken to Remedy Potential Problems: 

We have taken extensive "due diligence" steps which are briefly noted 
in your report. The report, however, fails to acknowledge the 
thoroughness of our actions to identify, assess, and attenuate any 
possible negative consequences of the campaign once Westat reported the 
possibility of such an effect. We have convened experts from mass 
communication, youth drug prevention, advertising, behavioral research, 
and related fields to explore what might be the theoretical basis for 
such potential effects. We have implemented recommendations to
strengthen qualitative and quantitative research methods to probe for 
such effects in subsequent advertising, and have made continual 
refinements in message strategy and have developed protocols for 
minimizing possible normative perceptions. We have increased the 
sensitivity to such a possibility among advertising planners of the 
Partnership for Drug-Free America - which under our direction provides 
most of the advertising for the campaign - as well as the pro bono 
advertising agencies who contribute their creative talents to the 
campaign. Further, we have strengthened our ability to detect any such 
effects through our monthly tracking studies which monitor youth 
responses to the media messages they are seeing and hearing. 

Major Changes in Campaign: 

The Campaign is now substantially different from that which was 
measured when the last wave of data was collected by Westat more than 
two years ago. The Campaign has changed direction to be even more 
relevant to today's youth, and after a full year of research, last 
November launched a new brand, "Above the Influence." Designed to 
encourage youth to aspire to their full potential by avoiding the 
negative influences in their lives - specifically drugs and peer 
influences to use drugs - this new brand already is showing positive 
results in brand awareness (exceeding within six months the awareness 
of the previous brand, the Anti-Drug), perception of the risk in using 
drugs, and anti-drug attitudes, as revealed by monthly tracking surveys 
that are typical of advertising industry "best practices" for measuring 
an advertising campaign. This new Campaign has the breadth to address 
not only marijuana but other emerging drug threats, including 
methamphetamine and non-medical use of prescription drugs and over-the- 
counter medicines. We are pleased to report that "Above the Influence" 
already has won significant advertising and communications industry 
awards, including the prestigious Media Week "Media Plan of the Year," 
for campaigns spending $25 million or more, the American Association of 
Advertising Agencies Jay Chiat Planning Award (the top honor for 
creative planning and strategy in its category), and two Webby Awards, 
including Best Youth Web Site of 2006, given by the international 
Academy of Digital Arts and Sciences. 

An Alternate Explanation for Counterintuitive Results: 

A possible explanation for these counterintuitive findings could be a 
function of what is theorized by Westat to occur with heavy exposure to 
anti-drug messages. There is growing evidence from research in 
psychology and consumer behavior that asking people a question about 
their future behavior influences the subsequent performance of that 
very behavior, known as the "mere measurement effect." The Westat 
evaluation, by conducting an extensive interview of youth (and their 
parents) in their homes, including showing them the ads being studied 
on as many as four separate occasions over the life of the evaluation, 
could stimulate interest in drugs where none previously existed, 
increase beliefs that drugs are important to youth and that more teens 
are using drugs, thus stimulating interest in and causing intent to use 
drugs and eventual drug experimentation. Impressionable youth, being 
inherently interested in pursuing behaviors that they believe would 
make them more grown-up, arguably would be more sensitive than adults 
to the effects of the evaluation interviews. One recent peer-reviewed 
journal article, for example, reported that ".when a question is asked 
about a socially non-normative health behavior (i.e. illegal drugs), 
instead of decreases in the behavior we see increased rates of the non-
normative behavior" (Williams, Fitzsimons and Block, "Simply asking 
questions about health behavior increases both healthy and unhealthy 
behavior," Social Influence, 2006). Such a problem would be less likely 
with a study design that used only cross-sectional surveys, but the 
frequency and intensity of youth interviews required by the Westat 
longitudinal design clearly would result in a greater impact of this 
type on survey respondents. 

Consequences of Delayed Report: 

The long delay in receiving the GAO assessment of the Westat report has 
had serious operational consequences, such as severely compromising our 
ability to make progress on the next round of evaluation. More than two 
years ago, ONDCP announced an RFP for the new outcome evaluation, but 
in subsequent discussions GAO strongly discouraged us from pursuing 
that course because of the difficulty of evaluating this Campaign. GAO 
even posed a probable recommendation that we should first undertake an 
evaluability assessment. That recommendation is now not put forward by 
GAO. As a result, our plans for evaluation when resumed after more than 
two years will not provide meaningful outcome evaluation for several 
years to come. Meanwhile, we continue to use monthly tracking surveys 
to monitor Campaign progress, and although such studies cannot give us 
long term outcome data, they do permit real-time tracking of 
performance and allow for effective decision-making. 

Your Recommendation: 

Finally, we are puzzled as to the lack of recommendations for 
improvement of the Campaign, given the GAO's extensive review of the 
Westat results. For example, after the GAO staff briefing more than a 
year ago, we anticipated specific recommendations on future Campaign 
evaluations, or on Campaign theory, design, or implementation. While we 
appreciate your understanding that the Campaign has rigorously applied 
enormous expertise to the Campaign and has made continual improvements, 
we are concerned with your conclusion that the result as judged by 
Westat ".raises questions concerning the understanding of the factors 
that are most salient to teens' decision making about drugs and how 
they can be used to foster anti-drug decisions." In addition, while 
your recommendation to Congress is clear, there are no criteria set 
forth for or even a cursory discussion of how one might demonstrate 
".credible evidence of the effectiveness of the campaign on drug use 
outcomes" especially given the widely-accepted understanding that a 
media campaign, in and of itself, should not be held uniquely 
responsible for reductions in teen drug use. 

Consequences of Further Budget Cuts: 

I want to make it clear that the consequences of further budget cuts to 
the campaign could have far-reaching and unfavorable consequences. 
Lessons learned from relevant youth health behaviors include recent 
evidence on tobacco use among teens which reveals that the downturn in 
media exposure to anti-tobacco messages already is resulting in 
decreased perception of risk, which more than 30 years of research 
shows is directly related to smoking rates (the same is true for 
illicit drugs, including marijuana). The 2005 MTF reported that all 
three grades surveyed showed a decline in weekly exposure to anti- 
tobacco messages, and the rate of decline of smoking is slowing and in 
fact has halted among 8`h graders, who have been the bellwethers of 
smoking trends among teens. Previous studies have established a 
relationship between exposure to anti-tobacco messages and smoking 
rates among teens. CDC has observed that smoking prevention media 
campaigns are effective in reducing youth smoking initiation and has 
expressed concern that less exposure to such messages may translate to 
a reversal of the long downward trends in youth tobacco use. We should 
expect similar results for illicit drug use if anti-drug messages 
decline, and, in fact, already the MTF in 2005 has detected among 8tH 
graders a flattening out in the previously increasing perception of 
risk, due to somewhat lower media exposures. 

Further, because mass media and popular culture convey a pervasive and 
disheartening array of pro-drug messages that reach the nation's youth, 
further cutting back the Campaign would have the unfortunate result of 
essentially abandoning our ability to counter those messages with 
clear, consistent, credible anti-drug messages, which only a national 
anti-drug media campaign can do. Teens today are exposed to media for 
well more than six hours per day. Half of teens live in households 
where there are no rules about TV exposure, and among the other half 
only one of five reports that those rules are actually enforced. Many 
of the most popular films among teens include scenes and references to 
drug use, seldom with any portrayal of social disapproval or negative 
consequences. Studies of websites accessible to any youth with computer 
access report ubiquitous - and highly detailed - pro-drug content, 
ranging from where to buy marijuana seeds, how to beat a drug test, and 
how to make methamphetamine. And anyone familiar with current trends in 
blogs and personal-space websites knows of the overwhelming pro-drug 
content, readily accessible to teens without their parents having any 
knowledge of such troubling exposures or risks to their children. 

Failure to continue the Campaign's efforts to counter such messaging is 
the equivalent of raising a white flag to those who favor drug 
legalization, with the expectation that youth drug use soon would begin 
to rise, reversing years of hard-earned positive news. Compounding the 
problem is the fact that the news media seldom cover the risks of drug 
use to teens, but often report on the use of drugs by celebrities and 
others in the public eye, as well as efforts to legalize drugs and 
promote so-called medical marijuana. 

In addition, further cuts to the Campaign likely would create a 
chilling effect on those media companies who have donated well over a 
billion dollars worth of media time and space to ensure youth receive 
anti-drug messages, as well as those ad agencies - more than 80 to date 
- who have contributed their creative talents towards reducing teen 
drug use. The Partnership for a Drug-Free America, whose efforts helped 
persuade the Congress of the need for this Campaign, has been 
singularly effective at recruiting private sector largesse to this 
important goal. Such resources are essential to the success of our 
efforts. 

Conclusion: 

As always, I remain interested in finding ways to improve the 
performance of the Campaign, as well as other efforts within the scope 
of the National Drug Control Strategy. I, too, once was skeptical of 
the ability of a media campaign to make a difference. Due to our 
considerable energies spent to re-focus and strengthen the Campaign, I 
have come to believe it is among the most important tools we have to 
reduce teen drug use. Much is at stake and we must work together to 
overcome the unwarranted cynicism over whether we can reduce drug use 
among America's teens. We can, we have, and we will continue to do so, 
using the best available means we have at our disposal. 

Respectfully, 

Signed by: 

John P. Walters: 
Director: 

[End of section] 

Appendix III: GAO Contacts and Staff Acknowledgments: 

GAO Contacts: 

Nancy Kingsbury, 202-512-2700: 
Laurie E. Ekstrand, 202-512-8777: 

Acknowledgments: 

In addition to the contacts named above, contributors to this report 
included David P. Alexander, Billy Commons, James Fields, Kathryn 
Godfrey, Mary Catherine Hult, Jean McSween, Karen V. O'Conor, Mark 
Ramage, William J. Sabol, Barry J. Seltser, and Douglas Sloane. 

FOOTNOTES 

[1] Pub. L. No. 105-61, 111 Stat. 1272. 

[2] See: GAO, Anti-Drug Media Campaign: ONDCP Met Most Mandates, but 
Evaluations of Impact Are Inconclusive, GAO/GGD/HEHS-00-153 
(Washington, D.C.: July 31, 2000). We also reported on ONDCP's use of 
consultants in the campaign in GAO, Anti-Drug Media Campaign: An Array 
of Services Was Provided, but Most Funds Were Committed to Buying Media 
Time and Space, GAO-05-175 (Washington, D.C.: Mar. 31, 2005). We also 
described the phase III evaluation in GAO, Program Evaluation: 
Strategies for Assessing How Information Dissemination Contributes to 
Agency Goals, GAO-02-923 (Washington, D.C.: Sept. 30, 2002). 

[3] Hereafter, we refer to the contractor as "Westat," and this 
implicitly includes Annenberg. In addition, a second subcontractor, the 
National Development and Research Institutes, Inc., provided expertise 
in developing drug use questions and assisted in preparing the first 
special topics report on trends in drug use. 

[4] Westat and Annenberg jointly submitted to NIDA all evaluation 
reports except for the final report, which was submitted by Westat 
only. 

[5] Senate Report No. 108-146, at 143 (2003). 

[6] See GAO-05-175 for our review of ONDCP's use of consultants in the 
campaign. 

[7] Pub. L. No. 105-61, 111 Stat. 1272. 

[8] Drug Free Media Campaign Act of 1998, 21 U.S.C. § 1801 et. seq. 

[9] The survey is now known as the National Survey on Drug Use and 
Health. 

[10] All of these reports were submitted jointly by Westat and 
Annenberg. 

[11] Westat, "Environmental Context of the National Youth Anti-Drug 
Media Campaign: Findings from In-Depth Discussions with Representatives 
of National Organizations and State Prevention Coordinators." Report 
delivered to National Institute on Drug Abuse, National Institutes of 
Health, Rockville, Maryland, May 2002. 

[12] Hornik, Robert, et al. Evaluation of the National Youth Anti-Drug 
Media Campaign: Fifth Semi-Annual Report of Findings, (Rockville, 
Maryland: Westat, November 2002), p. xi. 

[13] Conference Report No. 108-10, at 1345 (2003). 

[14] Pub. L. No. 103-62, 107 Stat. 285 (1993). 

[15] Longshore, Douglas, et al., "National Youth Anti-Drug Media 
Campaign and School-Based Drug Prevention: Evidence for a Synergistic 
Effect in ALERT Plus," Addictive Behaviors, Vol. 31, (2006) pp. 496- 
508. 

[16] Palmgreen, Philip, et al., "Effects of ONDCP's Marijuana 
Initiative Campaign on High Sensation-Seeking Youth." Paper presented 
to the American Public Health Association, Philadelphia, Pennsylvania, 
December 2005. 

[17] GAO/GGD/HEHS-00-153, (Washington, D.C.: July 31, 2000), p. 68. 

[18] Each respondent was presented ads that had been broadcast 
nationally in the 2 calendar months prior to the interview. 

[19] Westat also called its longitudinal analysis a "delayed effects" 
analysis. 

[20] Propensity score methods have been demonstrated to be robust 
against bias associated with the specification of incorrect functional 
forms--e.g., linear rather than quadratic--of variables. 

[21] According to Westat, the reference period for the general exposure 
index, is "in recent months," and this wording was chosen to maintain 
equivalence to the wording used in the Monitoring the Future surveys in 
its questions about anti-drug advertising. 

[22] This was measured as the "Percent perceiving few other kids 
regularly use marijuana." 

[23] In discussing Westat's findings, any references to significance 
refers to statistical significance. 

[24] This survey was formerly known as the National Household Survey on 
Drug Abuse. 

[25] Westat points out, however that the MTF decline in use among 10th 
graders between 2001 and 2002 was within the statistical confidence 
limits of NSPY. 

[26] At the time that ONDCP prepared this document, NSDUH was still 
known as the National Household Survey on Drug Abuse, or NHSDA. 

[27] Office of Programs, Budget, Research and Evaluation, Office of 
National Drug Control Policy, "Youth Drug Use and the National Youth 
Anti-Drug Media Campaign," February, 2001 (Washington, D.C.: Executive 
Office of the President), p. 16. 

[28] Westat's assessed the exposure-initiation relationship using data 
from survey rounds 1 and 2, survey rounds 2 and 3, and within survey 
round 4; it assessed the exposure-initiation relationship between waves 
6 and 8 and waves 7 and 9. 

[29] Office of Programs, Budget, Research and Evaluation, Office of 
National Drug Control Policy, "Youth Drug Use and the National Youth 
Anti-Drug Media Campaign," February, 2001 (Washington, D.C.: Executive 
Office of the President), p. 3. 

[30] Office of Programs, Budget, Research and Evaluation, Office of 
National Drug Control Policy, "Youth Drug Use and the National Youth 
Anti-Drug Media Campaign," February, 2001 (Washington, D.C.: Executive 
Office of the President), p. 16. 

[31] Housing units built after 1998 had no chance of selection in 
either sampling frame. Also, a housing unit had no chance of selection 
if it had been built during the 1990s in a jurisdiction where no permit 
was required. Finally, modular housing built during the 1990s was 
inadvertently omitted from the permit sample. Any biases resulting from 
excluding housing units built after 1999 are likely to be small, as 
they constituted a small fraction of all housing units in the NSPY 
sampling frame, and they were accounted for by Westat's 
poststratification adjustments. For example, housing units built after 
April 1999 accounted for an estimated 1.0 percent of all housing units 
in existence in the time period covered by the wave 1 sample. 

[32] The power to detect differences for upswings in prevalence would 
depend upon the baseline level. However, the power to detect an upswing 
from a baseline of 90 percent of youth would be exactly the same as 
that for detecting a downswing from a 10 percent baseline. 

[33] Specifically, the minimum detectable difference for wave-to-wave 
changes was at least 80 percent using a one-sided hypothesis test at 
the 0.05 level. 

[34] Each respondent was presented ads that had been broadcast 
nationally in the 2 calendar months prior to the interview. 

[35] GAO: Drug Use Measurement: Strengths, Limitations, and 
Recommendations for Improvement, GAO/PEMD-93-18 (Washington, D.C.: June 
25, 1993). 

[36] National Research Council, Informing America's Policy on Illegal 
Drugs: What We Don't Know Keeps Hurting Us. National Academy Press, 
Washington, D.C.: 2001. 

[37] Propensity score methods have been demonstrated to be robust 
against bias associated with the specification of incorrect functional 
forms--e.g., linear rather than quadratic--of variables. 

GAO's Mission: 

The Government Accountability Office, the investigative arm of 
Congress, exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability 
of the federal government for the American people. GAO examines the use 
of public funds; evaluates federal programs and policies; and provides 
analyses, recommendations, and other assistance to help Congress make 
informed oversight, policy, and funding decisions. GAO's commitment to 
good government is reflected in its core values of accountability, 
integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains 
abstracts and full-text files of current reports and testimony and an 
expanding archive of older products. The Web site features a search 
engine to help you locate documents using key words and phrases. You 
can print these documents in their entirety, including charts and other 
graphics. 

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as "Today's Reports," on its 
Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order 
GAO Products" heading. 

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to: 

U.S. Government Accountability Office 

441 G Street NW, Room LM 

Washington, D.C. 20548: 

To order by Phone: 

Voice: (202) 512-6000: 

TDD: (202) 512-2537: 

Fax: (202) 512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm 

E-mail: fraudnet@gao.gov 

Automated answering system: (800) 424-5454 or (202) 512-7470: 

Public Affairs: 

Jeff Nelligan, managing director, 

NelliganJ@gao.gov 

(202) 512-4800 

U.S. Government Accountability Office, 

441 G Street NW, Room 7149 

Washington, D.C. 20548: