ONDCP Media Campaign:
Contractor's National Evaluation Did Not Find that the Youth Anti-Drug Media Campaign Was Effective in Reducing Youth Drug Use
GAO-06-818, Aug 25, 2006
Between 1998 and 2004, Congress appropriated over $1.2 billion to the Office of National Drug Control Policy (ONDCP) for the National Youth Anti-Drug Media Campaign. The campaign aimed to prevent the initiation of or curtail the use of drugs among the nation's youth. In 2005, Westat, Inc., completed a multiyear national evaluation of the campaign. GAO has been mandated to review various aspects of the campaign, including Westat's evaluation which is the subject of this report. Applying generally accepted social science research standards, GAO assessed (1) how Westat provided credible support for its findings and Westat's findings about (2) attitudes, beliefs, and behaviors of youth and parents toward drug use and (3) youth self-reported drug use.
GAO's review of Westat's evaluation reports and associated documentation leads to the conclusion that the evaluation provides credible evidence that the campaign was not effective in reducing youth drug use, either during the entire period of the campaign or during the period from 2002 to 2004 when the campaign was redirected and focused on marijuana use. By collecting longitudinal data--i.e., multiple observations on the same persons over time--using generally accepted and appropriate sampling and analytic techniques, and establishing reliable methods for measuring campaign exposure, Westat was able to produce credible evidence to support its findings about the relationship between exposure to campaign advertisements and both drug use and intermediate outcomes. In particular, Westat was able to demonstrate that its sample was not biased despite sample coverage losses, maintained high follow-up response rates of sampled individuals to provide for robust longitudinal analysis, established measures of exposure that could detect changes in outcomes on the order of magnitude that ONDCP expected for the campaign and that could reliably measure outcomes, and used sophisticated statistical methods to isolate causal effects of the campaign. Westat's findings on the effects of exposure on intermediate outcomes--theorized precursors of drug use--were mixed. Specifically, although sampled youth and parents' recall of campaign advertisements increased over time, they had good impressions of the advertisements, and they could identify the specific campaign messages, exposure to the advertisements generally did not lead youth to disapprove of using drugs and may have promoted perceptions among exposed youth that others' drug use was normal. Parents' exposure to the campaign led to changes in beliefs about talking about drug use with their children and the extent to which they had these conversations with their children. However, exposure did not appear to lead to increased monitoring of youth. Moreover, the evaluation was unable to demonstrate that changes in parental attitudes led to changes in youth attitudes or behaviors toward drug use. Westat's evaluation indicates that exposure to the campaign did not prevent initiation of marijuana use and had no effect on curtailing current users' marijuana use, despite youth recall of and favorable assessments of advertisements. Although general trend data derived from the Monitoring the Future survey and the Westat study show declines in the percentage of youth reportedly using marijuana from 2002 to 2004, the trend data do not explicitly take into account exposure to the campaign, and therefore, by themselves, cannot be used as evidence of effectiveness. In Westat's evaluation of relationships between exposure and marijuana initiation the only significant finding was of small unfavorable effects of the campaign exposure on marijuana initiation during some periods of data collection and in some subgroups.
- Closed - implemented
- Closed - not implemented
Matter for Congressional Consideration
Matter: In light of the fact that the phase III evaluation of the media campaign yielded no evidence of a positive outcome in relation to teen drug use and congressional conferees' indications of their intentions to rely on the Westat study, Congress may wish to consider limiting appropriations for the National Youth Anti-Drug Media Campaign beginning in the fiscal 2007 budget year until ONDCP is able to provide credible evidence of the effectiveness of exposure to the campaign on youth drug use outcomes or provide other credible options for a media campaign approach. In this regard we believe that an independent evaluation of the new campaign should be considered as a means to help inform both ONDCP and Congressional decision making.
Status: Closed - Implemented
Comments: Between fiscal years 1998 and 2006, Congress appropriated over $1.4 billion to the Office of National Drug Control Policy (ONDCP) for the National Youth Anti-Drug Media Campaign. The campaign aimed to prevent the initiation of or curtail the use of drugs among the nation's youth. In 2005, Westat, Inc., completed a multiyear national evaluation of the campaign. GAO reviewed various aspects of the campaign, including Westat's evaluation. Based on this review of Westat's evaluation reports and associated documentation, we found that the phase III evaluation of the campaign yielded no evidence of a positive outcome in relation to teen drug use either during the entire period of the campaign or during the period from 2002 to 2004 when the campaign was redirected and focused on marijuana use. As a result of this finding and Congressional conferees' indications of their intentions to rely on the Westat study, we suggested Congress consider limiting appropriations for the campaign beginning in the fiscal year 2007 budget year until ONDCP was able to provide credible evidence of the effectiveness of exposure to the campaign on youth drug use outcomes. We also suggested to Congress that an independent evaluation of the new campaign be considered as a means to help inform both ONDCP and Congressional decision-making. With regard to the first suggestion, Congress subsequently appropriated $100 million dollars for the campaign for fiscal year 2007, a reduction of $20 million dollars from the requested $120 million dollars. Congress appropriated $60 million dollars for the campaign for fiscal year 2008, a reduction of $70 million dollars from the requested $130 million dollars. The net present value of these amounts is $90,926,000 dollars. Senate Appropriations staff indicated that the GAO report was one reason for these levels provided for the campaign. In addition, in the Senate report accompanying the fiscal year 2008 appropriation, the Committee noted that it continued to be concerned about the direction and efficacy of the campaign and that GAO had confirmed that the campaign had not been effective in reducing youth drug use. In a conversation with GAO, House Appropriations staff also expressed concern about the cost of the campaign in light of the evidence that it had not been effective in reducing drug use among youth. With regard to the second suggestion, in a report accompanying the fiscal year 2008 Consolidated Appropriations Act, the House Committee on Appropriations directed ONDCP to report to the Committee within 90 days of the enactment of the appropriations act with recommendations on the development of improved and meaningful measurements of the effectiveness of the media campaign, including measurements that would indicate how the campaign influences youth and parent behavior. In May 2008, ONDCP submitted a report to Congress, "Recommendations on the Development of Improved and Meaningful Measurements of the Effectiveness of the National Youth Anti-Drug Media Campaign." In this report, ONDCP proposed a 3-step approach for outcome evaluation of the campaign: (1) convene an expert group of evaluators to provide insights and input on a new scope of work for an evaluation contract, (2) strengthen the "Partnership Attitudes Tracking Survey" to permit improved associations between exposure to campaign messages and expected outcomes for both youth and parents, and (3) contract for an independent evaluation of the campaign, based on guidance from the expert panel and other insights to perform an outcome evaluation.