This is the accessible text file for GAO report number GAO-09-721R 
entitled 'Juvenile Justice: Technical Assistance and Better Defined 
Evaluation Plans Will Help to Improve Girls' Delinquency Programs' 
which was released on July 24, 2009. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

GAO-09-721R: 

United States Government Accountability Office: 
Washington, DC 20548: 

July 24, 2009: 

The Honorable Robert C. Scott:
Chairman:
Subcommittee on Crime, Terrorism, and Homeland Security: 
Committee on the Judiciary:
House of Representatives: 

Subject: Juvenile Justice: Technical Assistance and Better Defined 
Evaluation Plans Will Help to Improve Girls' Delinquency Programs: 

Dear Mr. Chairman: 

Girls' delinquency has attracted the attention of federal, state, and 
local policymakers for more than a decade as girls have increasingly 
become involved in the juvenile justice system. For example, from 1995 
through 2005, delinquency caseloads for girls in juvenile justice 
courts nationwide increased 15 percent while boys' caseloads decreased 
by 12 percent. Also, from 1995 through 2005, the number of girls' cases 
nationwide involving detention increased 49 percent compared to a 7 
percent increase for boys.[Footnote 1] More recently, in 2007, 29 
percent of juvenile arrests--about 641,000 arrests--involved girls, who 
accounted for 17 percent of juvenile violent crime arrests and 35 
percent of juvenile property crime arrests.[Footnote 2] Further, in a 
2007 survey of states conducted by the Federal Advisory Committee on 
Juvenile Justice, 22 states listed girls' delinquency as an issue 
affecting their states' juvenile justice systems.[Footnote 3] State 
justice officials responding to the survey noted that juvenile female 
offenses have increased sharply and also noted that juvenile female 
offenders generally had more serious and wide-ranging service needs 
than juvenile male offenders, including treatment for substance abuse 
and mental health conditions. 

The Office of Juvenile Justice and Delinquency Prevention (OJJDP) is 
the Department of Justice (DOJ) office charged with providing national 
leadership, coordination, and resources to prevent and respond to 
juvenile delinquency and victimization. OJJDP supports states and 
communities in their efforts to develop and implement effective 
programs to, among other things, prevent delinquency and intervene 
after a juvenile has offended. For example, from fiscal years 2007 
through 2009, Congress provided OJJDP almost $1.1 billion for grants to 
states, localities, and organizations for a variety of juvenile justice 
programs. In support of this mission, the office also funds research 
and program evaluations related to a variety of juvenile justice 
issues, including girls' delinquency. 

As programs have been developed at the state and local levels in recent 
years that specifically target preventing girls' delinquency or 
intervening after girls have become involved in the juvenile justice 
system, it is important that agencies providing grants and 
practitioners operating the programs have information about which of 
these programs are effective. In this way, agencies can help to ensure 
that limited federal, state, and local funds are well spent. In 
general, effectiveness is determined through program evaluations, which 
are systematic studies conducted to assess how well a program is 
working--that is, whether a program produced its intended effects. To 
help ensure that grant funds are being used effectively, you asked us 
to review OJJDP's efforts related to studying and promoting effective 
girls' delinquency programs. This report addresses the following 
questions: 

1. What efforts, if any, has OJJDP made to assess the effectiveness of 
girls' delinquency programs? 

2. To what extent are OJJDP's efforts to assess girls' delinquency 
programs consistent with generally accepted social science standards 
and the internal control standard to communicate with external 
stakeholders? 

3. What are the findings from OJJDP's efforts to assess the 
effectiveness of girls' delinquency programs, and how, if at all, does 
OJJDP plan to address the findings from these efforts? 

To identify OJJDP's efforts to assess the effectiveness of girls' 
delinquency programs, we analyzed relevant laws related to the office's 
role in supporting research and evaluations on delinquency programs. We 
also analyzed OJJDP budget data for fiscal years 2007 through 2009. We 
chose these years because they provide the most recent overview of the 
funding the office has had available to support its evaluation 
activities. We examined reports from research organizations and 
academic journal articles on girls' delinquency issues. In our review, 
we focused on OJJDP's efforts related to programs that are specifically 
designed for girls, not programs designed for both girls and boys. To 
identify OJJDP's efforts, we reviewed a list of its grants to fund 
studies of girls' delinquency programs from 1998 to 2008. We chose this 
time frame, the past 10 years from the start of our work, because it 
provided us with an overview of OJJDP's efforts related to assessing 
girls' delinquency programs. We also analyzed documentation about 
OJJDP's establishment of a study group on girls' delinquency issues, 
including the program announcement and cooperative agreement.[Footnote 
4] We interviewed OJJDP officials, including the research coordinator 
who managed the study group project, about the office's role in 
overseeing the group's research. We also interviewed the current and 
former principal investigators of the study group project regarding the 
formation of the group, its activities, and its methodologies. To 
gather information on OJJDP's efforts, we conducted interviews with 18 
girls' delinquency subject matter experts, that is, researchers and 
practitioners. We selected these experts based on their knowledge and 
experience with girls' delinquency issues, which we determined through 
our review of the literature and from suggestions of experts to 
interview from study group members and OJJDP.[Footnote 5] These 18 
experts included 11 of the 15 study group members and 7 experts who 
were not members of the group.[Footnote 6] While their comments cannot 
be generalized to all girls' delinquency experts, we nonetheless 
believe that their views gave us useful insights on issues related to 
girls' delinquency and OJJDP's efforts to assess girls' programs. 

To determine the extent to which OJJDP's efforts to assess girls' 
delinquency programs were consistent with generally accepted social 
science standards, we reviewed the criteria the study group used to 
assess studies of girls' delinquency programs and whether the group's 
application of those criteria was consistent with standards for 
evaluation research.[Footnote 7] To determine the extent to which these 
OJJDP efforts were consistent with the internal control standard to 
communicate with external stakeholders, we compared the office's 
efforts with criteria in Standards for Internal Control in the Federal 
Government, specifically that agency management should ensure that 
there are adequate means of obtaining information from and 
communicating with external stakeholders who may have a significant 
impact on the agency achieving its goals.[Footnote 8] We reviewed 
documentation about the composition of the study group and the criteria 
used to select the group members, such as their relevant fields of 
expertise, knowledge, and experience with girls' issues. We also 
examined the study group's external communications efforts, including 
its Web site, findings bulletins, conference presentations, academic 
journal articles, and published book. In addition, we interviewed OJJDP 
officials about these dissemination efforts, as well as 18 girls' 
delinquency experts regarding their views on the composition of the 
study group. 

To determine the findings from OJJDP's efforts to assess the 
effectiveness of girls' delinquency programs, and to assess how, if at 
all, OJJDP plans to address these findings, we analyzed documentation 
such as published bulletins and conference presentations about the 
study group's findings and recommendations related to program 
effectiveness. We also interviewed OJJDP officials knowledgeable about 
the office's planning efforts and the current and former study group 
principal investigators regarding the group's findings and 
recommendations. We compared OJJDP's stated plans with criteria in 
standard practices for program management.[Footnote 9] 

We conducted this performance audit from July 2008 through July 2009 in 
accordance with generally accepted government auditing standards. Those 
standards require that we plan and perform the audit to obtain 
sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe that 
the evidence obtained provides a reasonable basis for our findings and 
conclusions based on our audit objectives. 

Results in Brief: 

To assess the effectiveness of girls' delinquency programs, OJJDP 
established the Girls Study Group (Study Group). With an overall goal 
of developing research that communities need to make sound decisions 
about how best to prevent and reduce girls' delinquency, the Study 
Group was established in 2004 under a $2.6 million multiyear 
cooperative agreement with a research institute. OJJDP's objectives for 
the group, among others, included identifying effective or promising 
programs, program elements, and implementation principles (i.e., 
guidelines for developing programs) and developing program models to 
help inform communities of what works in preventing or reducing girls' 
delinquency; identifying gaps in girls' delinquency research and 
developing recommendations for future research; and disseminating 
findings to the girls' delinquency field about effective or promising 
programs. To meet OJJDP's objectives, among other things, the Study 
Group identified studies of delinquency programs that specifically 
targeted girls. The group then assessed the methodological quality of 
the studies using a set of criteria developed by DOJ's Office of 
Justice Programs (OJP) called What Works to determine whether the 
studies provided credible evidence that the programs were effective at 
preventing or responding to girls' delinquency.[Footnote 10] 

OJJDP's effort to assess girls' delinquency programs through the use of 
a study group and the group's methods for assessing studies were 
consistent with generally accepted social science research practices 
and standards, and OJJDP's efforts to involve practitioners in Study 
Group activities and disseminate findings were also consistent with the 
internal control standard to communicate with external stakeholders. 
[Footnote 11] 

* According to OJJDP officials--including the research coordinator--
they formed the Study Group rather than funding individual studies of 
programs because study groups provide a cost-effective method of 
gaining an overview of the available research in an issue area. As part 
of its work, the group collected, reviewed, and analyzed the 
methodological quality of research on girls' delinquency programs. The 
use of such a group, including its review, is an acceptable approach 
for systematically identifying and reviewing research conducted in a 
field of study. This review helped consolidate the research and provide 
information to OJJDP for determining evaluation priorities. Further, we 
reviewed the criteria the group used to assess the studies and found 
that they adhere to generally accepted social science standards for 
evaluation research. We also generally concurred with the group's 
assessments of the programs based on these criteria. According to the 
group's former principal investigator, the Study Group decided to use 
the What Works criteria to ensure that its assessment of program 
effectiveness would be based on highly rigorous evaluation standards, 
thus eliminating the potential that a program that may do harm would be 
endorsed by the group. However, 8 of the 18 experts we interviewed said 
that the criteria created an unrealistically high standard, which 
caused the group to overlook potentially promising programs. OJJDP 
officials stated that despite such concerns, they approved the group's 
use of the criteria because of the methodological rigor of the 
framework and their goal for the group to identify effective programs. 

* In accordance with the internal control standard to communicate with 
external stakeholders, OJJDP sought to ensure a range of stakeholder 
perspectives related to girls' delinquency by requiring that Study 
Group members possess knowledge and experience with girls' delinquency 
and demonstrate expertise in relevant social science disciplines. The 
initial Study Group, which was convened by the research institute and 
approved by OJJDP, included 12 academic researchers and 1 practitioner, 
a member with experience implementing girls' delinquency programs. 
Eleven of the 18 experts we interviewed stated that this composition 
was imbalanced in favor of academic researchers, six of whom said that 
the composition led the group to focus its efforts on researching 
theories of girls' delinquency rather than gathering and disseminating 
actionable information for practitioners.[Footnote 12] According to 
OJJDP officials, they acted to address this issue by adding a second 
practitioner as a member and involving two other practitioners in study 
group activities. OJJDP officials stated that they plan to more fully 
involve practitioners from the beginning when they organize study 
groups in the future and to include practitioners in the remaining 
activities of the Study Group, such as presenting findings at a 
national conference. Also, in accordance with the internal control 
standard, OJJDP and the Study Group have disseminated findings to the 
research community, practitioners in the girls' delinquency field, and 
the public through conference presentations, Web site postings, and 
published bulletins and plan to report on all of the group's activities 
by spring 2010. 

To address the Study Group findings that few girls' delinquency 
programs had been studied and that the available studies lacked 
conclusive evidence of program effectiveness, OJJDP plans to provide 
technical assistance to help programs be better prepared for 
evaluation; however, more fully developing plans for supporting 
evaluations could help OJJDP address its girls' delinquency goals. The 
Study Group found that the majority of the girls' delinquency programs 
it identified--44 of the 61--had not been studied by researchers. For 
the 17 programs that had been studied, the Study Group reported that 
none of the studies provided conclusive evidence with which to 
determine whether the programs were effective at preventing or reducing 
girls' delinquency. For example, according to the Study Group, 11 of 
the 17 studies lacked evidence of program effectiveness because, for 
instance, the studies involved research designs that could not 
demonstrate whether any positive outcomes, such as reduced delinquency, 
were due to program participation rather than other factors. Based on 
the results of this review, the Study Group reported that among other 
things, there is a need for additional, methodologically rigorous 
evaluations of girls' delinquency programs; training and technical 
assistance to help programs prepare for evaluations; and funding to 
support girls' delinquency programs found to be promising. According to 
OJJDP officials, in response to the Study Group's finding about the 
need to better prepare programs for evaluation, the office plans to 
work with the group and use the remaining funding from the effort-- 
approximately $300,000--to provide technical assistance workshop in 
October 2009. The workshop is intended to help approximately 10 girls' 
delinquency programs prepare for evaluation by providing information 
about how evaluations are designed and conducted, how to identify 
appropriate performance measures, and how to collect data that will be 
useful for program evaluators in assessing outcomes. In addition, OJJDP 
officials stated that as a result of the Study Group's findings along 
with feedback they received from members of the girls' delinquency 
field, OJJDP plans to issue a solicitation in early fiscal year 2010 
for researchers to apply for funding to conduct evaluations of two to 
five girls' delinquency programs. OJJDP has also reported that the 
Study Group's findings are to provide a foundation for moving ahead on 
a comprehensive program related to girls' delinquency. However, OJJDP 
has not developed a plan that is documented, is shared with key 
stakeholders, and includes specific funding requirements and 
commitments and time frames for meeting its girls' delinquency goals. 
Standard practices for program and project management state that 
specific desired outcomes or results should be conceptualized, defined, 
and documented in the planning process as part of a road map, along 
with the appropriate projects needed to achieve those results, 
supporting resources, and milestones.[Footnote 13] In addition, 
government internal control standards call for policies and procedures 
that establish adequate communication with stakeholders as essential 
for achieving desired program goals.[Footnote 14] According to OJJDP 
officials, they have not developed such a plan because the office is in 
transition and is in the process of initiating efforts to develop an 
officewide research plan, but they are taking steps to address their 
girls' delinquency goals, for example, through the workshop and planned 
evaluations. Developing such a plan would help OJJDP to demonstrate 
leadership to the girls' delinquency field by clearly articulating the 
actions it intends to take to meet its goals and would also help the 
office to ensure that the goals are met. 

To help ensure that OJJDP meets its goals to identify effective or 
promising girls' delinquency programs and supports the development of 
program models, we are recommending that the Administrator of OJJDP 
develop and document a plan that (1) articulates how the office intends 
to respond to the findings of the Study Group, (2) includes time frames 
and specific funding requirements and commitments, and (3) is shared 
with key stakeholders. In commenting on a draft of this report, OJP 
agreed with our recommendation and outlined efforts that OJJDP plans to 
undertake to respond to the findings of the Study Group, which we 
describe in the report. OJP comments are reprinted in the enclosure. 

Background: 

Over the past two decades girls have increasingly become involved in 
the juvenile justice system, and while the majority of juvenile arrests 
and cases involve boys, research has indicated that girls have more 
intensive treatment needs than boys. In 1980, 20 percent of all 
juvenile arrests were girls; by the mid-1990s about one quarter of 
these arrests were girls; and by 2007, girls accounted for 29 percent 
of all juvenile arrests. Additionally, while arrests for some violent 
crimes, such as assaults, have decreased for males, they have decreased 
less, or in some cases have increased, for females. For example, 
between 1998 and 2007 juvenile male arrests for simple assault declined 
4 percent, and female arrests increased 10 percent.[Footnote 15] 
Further, from 1985 through 2005, the estimated number of girls' 
delinquency cases involving detention increased by 92 percent, and 
those cases that involved probation increased by 88 percent. Research 
on girls has highlighted that delinquent girls have higher rates of 
mental health problems than delinquent boys, receive fewer special 
services, and are more likely to abandon treatment programs. For 
example, one study showed that detained girls have more symptoms of 
mental illness than would be predicted on the basis of gender or 
setting alone.[Footnote 16] Research has also shown that delinquent 
girls have higher mortality rates, dysfunctional and violent 
relationships, poor educational achievement, and less stable work 
histories than nondelinquent girls. Further, girls' delinquency has 
been linked to drug abuse, mental health problems and disorders, poorer 
physical health, and victimization by and violence toward partners in 
adulthood. 

In recent years, programs have been developed that specifically target 
preventing girls' delinquency and intervening once girls have become 
involved in the juvenile justice system. In general, prevention 
programs provide services and programming, such as substance abuse 
education, mentoring, and life skills education, to deter girls from 
becoming involved in criminal or other antisocial activities. 
Intervention programs provide services to girls once they have entered 
the juvenile justice system, for example, through programs that are 
alternatives to probation or that provide intensive services for girls 
who are on probation, to prevent them from returning to the system or 
entering the adult criminal justice system. These services could 
include visits by probation officers, individual case plans, substance 
abuse treatment and therapy, funds for emergency situations, life 
skills courses, teen pregnancy services, and therapy sessions. 

The Juvenile Justice and Delinquency Prevention Act (the Act) 
established OJJDP in 1974.[Footnote 17] As the only federal office 
charged exclusively with preventing and responding to juvenile 
delinquency and victimization and with helping states improve their 
juvenile justice systems, OJJDP supports its mission through a variety 
of activities, including: funding research and evaluation efforts, 
statistical studies, and demonstration programs; providing training and 
technical assistance; producing and distributing publications and other 
products containing information about juvenile justice topics; and 
administering a wide variety of grants to states, territories, 
localities, and public and private organizations through formula, 
block, and discretionary grant programs.[Footnote 18] Table 1 shows 
OJJDP's enacted appropriations for fiscal years 2007 through 2009. 

Table 1: Juvenile Justice Appropriations Fiscal Years 2007 through 2009 
(Dollars in thousands): 

Line item: Part A - Concentration of Federal Efforts[A]; 
Funding by fiscal year 2007: $703; 
Funding by fiscal year 2008: $658; 
Funding by fiscal year 2009: $0. 

Line item: Part B - State Formula Grants; 
Funding by fiscal year 2007: $78,976; 
Funding by fiscal year 2008: $74,260; 
Funding by fiscal year 2009: $75,000. 

Line item: Part D--Research, Evaluation, Technical Assistance and 
Training; 
Funding by fiscal year 2007: $0; 
Funding by fiscal year 2008: $0; 
Funding by fiscal year 2009: $0. 

Line item: Part E - Developing, Testing and Demonstrating Promising New 
Initiatives and Projects; 
Funding by fiscal year 2007: $104,670; 
Funding by fiscal year 2008: $93,835; 
Funding by fiscal year 2009: $82,000. 

Line item: Youth Mentoring Grants; 
Funding by fiscal year 2007: $9,872; 
Funding by fiscal year 2008: $70,000; 
Funding by fiscal year 2009: $80,000. 

Line item: Title V - Local Delinquency Prevention Incentive Grants; 
Funding by fiscal year 2007: $64,168; 
Funding by fiscal year 2008: $61,100; 
Funding by fiscal year 2009: $62,000. 

Line item: Project Childsafe[B]; 
Funding by fiscal year 2007: $987; 
Funding by fiscal year 2008: $0; 
Funding by fiscal year 2009: $0. 

Line item: Secure Our Schools; 
Funding by fiscal year 2007: $14,808; 
Funding by fiscal year 2008: $15,040; 
Funding by fiscal year 2009: $0. 

Line item: VOCA--Improving Investigation and Prosecution of Child Abuse 
Program; 
Funding by fiscal year 2007: $14,808; 
Funding by fiscal year 2008: $16,920; 
Funding by fiscal year 2009: $20,000. 

Line item: Juvenile Accountability Block Grant Program[C]; 
Funding by fiscal year 2007: $49,360; 
Funding by fiscal year 2008: $51,700; 
Funding by fiscal year 2009: $55,000. 

Line item: Total; 
Funding by fiscal year 2007: $338,352; 
Funding by fiscal year 2008: $383,513; 
Funding by fiscal year 2009: $374,000. 

Sources: Revised Continuing Appropriations Resolution, 2007, Pub. L. 
No. 110-5, 121 Stat. 8, 8-9 (including the across-the-board rescission 
of 1.28 percent provided in the continuing resolution); Consolidated 
Appropriations Act, 2008, Pub. L. No. 110-161, 121 Stat. 1844, 1911-12 
(2007); and Omnibus Appropriations Act, 2009, Pub. L. No. 111-8, 123 
Stat. 524, 581-82. 

[A] According to OJP's fiscal year 2010 congressional budget 
submission, the Concentration of Federal Efforts program promotes 
interagency cooperation and coordination among federal agencies with 
responsibilities in the area of juvenile justice, as authorized by Part 
A of the Act, as amended. 

[B] Project Childsafe is a nationwide program to promote safe firearms 
handling and storage practices through the distribution of safety 
education messages and free gun-locking devices. 

[C] Under the Juvenile Accountability Block Grant Program, OJJDP 
provides funds to states and units of local government for the purpose 
of strengthening the juvenile justice system. These funds can be used 
for 17 different purpose areas, including establishing programs to help 
the successful reentry of juvenile offenders from state and local 
custody in the community or for hiring or training programs for 
detention and corrections personnel. 

[End of table] 

OJJDP, through its various grant programs, has provided funding to 
states and organizations to support girls' delinquency programs, 
although it is not specifically required by the Act to fund such 
programs in particular. For example, to be eligible to receive formula 
grants, states are required to submit a plan to OJJDP for providing 
gender-specific services for juvenile delinquency prevention and 
treatment.[Footnote 19] However, the states generally have the 
authority to determine how formula and block grants are allocated and 
may use these funds to support a range of program areas, including 
programs specifically for delinquent girls. For example, for fiscal 
years 2007 and 2008, OJJDP reported that states used approximately $1.9 
million in Part B formula grant money for girls' delinquency programs, 
representing approximately 1 percent of such funding for those years. 
In addition, in fiscal year 2007, OJJDP reported awarding about $1.8 
million in discretionary grant awards to prevention and intervention 
programs addressing girls' delinquency.[Footnote 20] 

The Act requires the OJJDP Administrator to conduct and support 
evaluations and studies of the performance and results achieved by 
federal juvenile delinquency programs and activities, although the law 
does not specifically require OJJDP to fund evaluations of state or 
locally funded programs or those specifically focused on girls' 
delinquency.[Footnote 21] OJJDP has provided funding for evaluations 
using (1) funds appropriated for Part D of the Act--which allows the 
Administrator to conduct research and evaluation, information 
dissemination, and training and technical assistance,[Footnote 22] or 
(2) funds set aside from several of its appropriations accounts for use 
for research, evaluation, and statistics activities.[Footnote 23] 
Funding has not been appropriated to OJJDP for Part D since fiscal year 
2005 when it received $10 million, so OJJDP has allocated funding for 
research and evaluation of programs from fiscal years 2006 through 2008 
using approximately $40 million in funding from appropriation set 
asides.[Footnote 24] 

OJJDP has provided funding for several efforts designed to provide 
information about girls' delinquency programs to the juvenile justice 
field in the past decade. For example, in 1998, the office published an 
inventory of best practices that included a list of 16 promising girls' 
delinquency programs, which had been compiled by a research 
organization as part of a $1.1 million cooperative agreement to provide 
training and technical assistance to states and localities about girls' 
programs.[Footnote 25] The research organization identified these 16 
programs on the basis of programmatic criteria--such as whether the 
program used appropriate assessments to determine treatment plans; 
provided empowerment strategies, such as skill training and vocational 
training; or provided its staff with gender-specific training--rather 
than on whether the program's effectiveness had been studied by 
researchers. Further, this effort found that more research was needed 
to draw conclusions about the effectiveness of girls' delinquency 
program models. In addition, during this time OJJDP spent approximately 
$1.1 million to fund four studies of girls' delinquency issues. While 
these studies assessed issues related to girls' delinquency, they did 
not specifically assess the effectiveness of girls' delinquency 
programs. For example, in 2000, OJJDP funded one study of women in 
gangs, which found, among other things, that the optimum time for 
prevention and intervention was the middle teen years and that the 
optimum place for intervention was school before girls drop out. 
Another study compared three treatment models to determine which was 
most effective at reducing the number of institutional placements for 
adjudicated female offenders. The study found that girls with the most 
serious and frequent crises were more dissatisfied with social services 
or were denied access to such services. The study highlighted the 
importance of youth assistance programs to provide opportunities for 
girls to develop pro-social skills through family, school, and 
community connections. 

OJJDP Established the Girls Study Group to Assess the Effectiveness of 
Girls' Delinquency Programs: 

OJJDP initiated the Study Group to assess the effectiveness of girls' 
delinquency programs. In response to increases in girls' arrests 
through the 1990s and early 2000s and questions about the causes of 
these increases and how best to respond to the needs of girls entering 
the juvenile justice system, OJJDP issued a program announcement in 
2003 for a study group to focus on girls' delinquency issues.[Footnote 
26] While OJJDP had funded studies on girls' issues and a technical 
assistance effort to assist girls' delinquency programs in their 
operations, in forming the Study Group, OJJDP determined that a 
comprehensive, research-based foundation was needed to guide state and 
local policymakers and practitioners in their efforts to effectively 
prevent and reduce girls' delinquency. In its announcement for the 
Study Group, OJJDP highlighted the need for more information about 
female development and female-specific delinquency risk factors, as 
well as the effectiveness of girls' delinquency programs to ensure the 
best services and treatment. OJJDP sought applications from public and 
private organizations to convene a study group to address these issues 
and in 2004 awarded a 2-year cooperative agreement to Research Triangle 
Institute (RTI) to do so. OJJDP has since provided RTI with an 
extension through June 2010 to complete all of the Study Group's 
activities. The total funding awarded for the cooperative agreement was 
almost $2.6 million. 

OJJDP articulated five broad objectives for the Study Group in its 
September 2003 program announcement. Three of these objectives 
specifically related to assessing and promoting girls' delinquency 
programs: (1) identifying effective or promising programs, program 
elements, and implementation principles to help inform communities 
about what works in preventing or reducing girls' delinquency and to 
support the development of these program models; (2) identifying gaps 
in girls' delinquency research and developing recommendations for 
future research to fill these gaps; and (3) disseminating findings to 
the girls' delinquency field about effective or promising programs. The 
other two objectives included understanding the trends and consequences 
related to girls' delinquency and developing a comprehensive theory of 
girls' delinquency.[Footnote 27] 

To meet OJJDP's program assessment objectives, among other activities, 
the Study Group conducted a review of the literature on girls' 
delinquency that included over 1,000 documents in relevant research 
areas, such as criminological and feminist explanations for girls' 
delinquency, patterns of delinquency, and the justice system's response 
to girls' delinquency. To identify girls' delinquency programs, from 
June 2005 through October 2006, the Study Group analyzed the results of 
this literature search, conducted Web searches, reviewed juvenile 
justice 3-year plans from 2000 to 2004 for all 50 states, reviewed 
federal agency and private organization lists of delinquency programs, 
and solicited suggestions on its Web site.[Footnote 28] The Study Group 
initially set out to identify federally funded girls' delinquency 
programs but expanded its search to include state and locally funded 
programs after it found few federally funded programs. As a result, the 
Study Group identified 61 programs that specifically targeted 
preventing or responding to girls' delinquency. The group then 
determined which of these programs had been studied for program 
effectiveness by conducting Web searches for evaluation materials and 
published research, reviewing abstracts from academic journals, 
contacting program directors, and reviewing program Web sites. 

To identify effective programs, the Study Group reviewed the studies of 
girls' delinquency programs that it identified and classified them 
based on evidence of their effectiveness. To make this determination, 
the Study Group compared the studies' methodologies to criteria 
established in the OJP What Works classification framework, which 
defines six levels of evidence of effectiveness, which are effective, 
effective with reservation, promising, and ineffective, as well as 
inconclusive evidence and insufficient evidence, as described in table 
2.[Footnote 29] 

Table 2: Summary of What Works Criteria Used by the Girls Study Group 
to Assess Studies of Girls' Delinquency Programs: 

Level of effectiveness: Effective; 
Description: Effective programs have studies with a randomized 
controlled research design. These are designs that compare the outcomes 
for individuals that are randomly assigned either to the program or to 
a nonparticipating control group before the intervention in an effort 
to control for any systematic difference between the groups that could 
account for a difference in their outcomes. Effective programs also 
demonstrate a significant and sustained effect--that is, statistically 
significant positive outcomes that remain for at least 1 year after 
subjects stop participating in a program. The program should have been 
replicated at least once externally at another site to confirm results. 

Level of effectiveness: Effective with reservation; 
Description: These programs have studies with a randomized controlled 
research design that demonstrates a significant and sustained effect. A 
program should have at least one replication to confirm results. 
Reservations occur either because the program has only an internal 
replication at the same site or because it has an external replication 
with modest results. 

Level of effectiveness: Promising; 
Description: Promising programs have either studies with (1) a 
randomized controlled research design without a replication or (2) a 
quasi-experimental research design. These programs have significant and 
sustained effects. 

Level of effectiveness: Insufficient evidence; 
Description: These are studies of programs that have a quasi- 
experimental research design that lack sufficient methodological rigor, 
or have a pre-post test design that involves tests that analyze 
measures before and after individuals participated in the program. 

Level of effectiveness: Inconclusive evidence; 
Description: These studies of programs may have adequately rigorous 
research designs but not sustained effects, or they may have 
contradictory findings and not enough evidence demonstrating that the 
programs are either effective or ineffective. 

Level of effectiveness: Ineffective; 
Description: These are studies of programs that have an experimental or 
quasi-experimental research design that failed to demonstrate a 
significant effect in an initial study or in a replication. 

Source: GAO analysis of OJP What Works criteria. 

[End of table] 

According to the Study Group's principal investigator, as of May 2009, 
the group had finalized its program review findings and was in the 
process of finishing a bulletin on these findings before providing it 
to OJJDP for publication. As of June 2009, OJJDP has issued three 
bulletins on several of the group's activities. These bulletins have 
provided an overview of the Study Group's activities and the group's 
findings on its two objectives related to girls' delinquency risk 
factors and patterns of offending. According to OJJDP officials, the 
Study Group plans to issue a final report that summarizes all of its 
activities and findings to OJJDP by spring 2010. 

OJJDP Efforts to Assess Program Effectiveness Were Consistent with 
Social Science Practices and Standards, and OJJDP Has Taken Action to 
Enhance Its Communication about Study Group Activities and Findings 
with External Stakeholders: 

OJJDP's efforts to assess program effectiveness through the use of a 
study group as well as the group's efforts were consistent with 
generally accepted social science practices and standards, although 
experts we interviewed presented differing views on the criteria used 
to assess programs. OJJDP also took action to include external 
stakeholders in study group activities and is disseminating the group's 
findings consistent with standards for control in the federal 
government. 

The Use of a Study Group and the Group's Efforts Were Consistent with 
Generally Accepted Social Science Practices and Standards; However, 
Experts We Interviewed Presented Differing Views on the Criteria Used 
to Assess Programs: 

OJJDP's efforts to assess girls' delinquency programs, including its 
approach of using a study group and the group's methods of assessing 
studies, were consistent with generally accepted social science 
standards for evaluation research. According to OJJDP officials, 
including the research coordinator, they chose to form a study group 
rather than fund individual evaluations of programs because study 
groups are a cost-effective method of gaining an overview of the 
available research in an issue area. As part of its work, the group 
collected, reviewed, and analyzed the methodological quality of 
research on girls' delinquency programs. Such an approach of 
systematically identifying and reviewing research conducted in a field 
of study is an acceptable practice to consolidate the research in an 
area and provide information to enable program managers to determine 
where they might best commit future evaluation resources.[Footnote 30] 
Thirteen of the 18 girls' delinquency experts we interviewed (including 
11 Study Group members) stated that the Study Group's efforts were 
useful for providing an overview of girls' delinquency issues. However, 
6 experts (including 2 Study Group members) also noted that it would 
have been beneficial to the girls' delinquency field for the group to 
conduct evaluations to determine program outcomes or promising models 
rather than reviewing completed studies. OJJDP has funded individual 
studies of girls' delinquency programs in the past but, according to 
OJJDP officials, was seeking to use the Study Group's research to form 
a baseline of the available knowledge about girls' delinquency issues. 

The Study Group's effort to review the studies according to the What 
Works criteria was consistent with generally accepted social science 
standards. Specifically, we reviewed the OJP What Works criteria and 
found that they adhere to these standards for evaluation research. 
Using the What Works criteria, we also assessed the same studies for 
the 17 girls' delinquency programs that the Study Group had reviewed 
and generally concurred with the Study Group's ratings of the program 
studies. While the Study Group's use of the What Works criteria was in 
keeping with social science standards, experts we interviewed expressed 
differing views on the group's decision to use these criteria. 
According to the Study Group's former principal investigator, the group 
decided to use the What Works criteria in 2005 because the criteria 
ensured that the group's assessment of the effectiveness of programs in 
preventing or reducing girls' delinquency would be based on highly 
rigorous evaluation standards to identify effective programs--thus 
eliminating the potential that a program that may do harm would be 
endorsed by the group. Eight Study Group members we interviewed also 
stated that the Study Group's use of the criteria was appropriate 
because it ensured that the group would only disseminate information on 
programs determined to be effective based on a high level of evidence. 
However, 8 other experts, including three Study Group members, said 
that the criteria created an unrealistically high standard, which 
caused the Study Group to overlook potentially promising programs. 
[Footnote 31] Further, 9 of the 18 experts (including five Study Group 
members) we interviewed also noted that requiring a randomized 
controlled research design--a research design that compares the 
outcomes for individuals who are randomly assigned to either the 
program being studied or to a nonparticipating control group before the 
intervention--to demonstrate effectiveness, as the What Works criteria 
does, is a difficult standard to achieve because such a design is 
expensive, and programs may be reluctant to divert resources from 
programming to pay for evaluations. OJJDP officials stated that they 
understood the experts' concerns and the trade-offs in using a 
classification framework that requires a randomized controlled research 
design to demonstrate effectiveness; however, they approved the group's 
use of the criteria because it provided a rigorous framework for 
assessing program evaluations. We understand that studies can produce 
valid results using other research designs, such as studies using quasi-
experimental designs or studies comparing the outcome results for 
groups of girls that are statistically matched. We have also previously 
reported that randomized controlled research designs provide 
researchers with the best method for assessing a program's 
effectiveness--they isolate changes caused by the program from other 
factors--when doing so is feasible and ethical.[Footnote 32] 

OJJDP Has Taken Actions to Reach Out to External Stakeholders on Study 
Group Activities and Findings and Is Disseminating the Findings in 
Keeping with Internal Control Standards: 

OJJDP has taken action to reach out to external stakeholders to address 
concerns about the composition of the Study Group after its initial 
formation and, moving forward, plans to continue to incorporate program 
practitioners in its planned efforts. Standards for Internal Control in 
the Federal Government states that program managers should ensure that 
there are adequate means of obtaining information from and 
communicating with external stakeholders who may have a significant 
impact on the agency achieving its goals.[Footnote 33] Regarding 
gaining information from external stakeholders, OJJDP's program 
announcement for the Study Group sought to ensure a range of 
stakeholder perspectives related to girls' delinquency. The 
announcement required that the members of the Study Group possess 
knowledge of and experience with female development and delinquent 
girls and demonstrate expertise in a variety of relevant social science 
disciplines, such as criminology, sociology, and developmental 
psychology. In awarding the cooperative agreement to RTI through a peer 
review process, OJJDP approved the RTI proposal for the Study Group as 
responding to the requirements and expectations of the program 
announcement. Consistent with the fields of expertise cited in the 
program announcement, RTI convened a group of 13 members, including 12 
academic researchers from social science disciplines and one 
practitioner, a member directly involved in girls' delinquency 
programming.[Footnote 34] However, according to several of the experts 
we interviewed, this Study Group composition did not include sufficient 
representation and input from a key external stakeholders group--girls' 
delinquency program practitioners. For example, 11 of the 18 girls' 
delinquency experts we interviewed, including 5 study group members, 
said that the Study Group was imbalanced in favor of academic 
researchers, 6 of whom (including 2 study group members) said that the 
composition led the group to focus its efforts on researching theories 
of girls' delinquency rather than gathering and disseminating 
actionable information for practitioners.[Footnote 35] According to 
OJJDP officials we interviewed, they had received feedback from girls' 
delinquency stakeholders in 2006 on this issue. In response, according 
to OJJDP program managers, they acted to address the imbalance of the 
Study Group by adding a second practitioner as a member and involving 2 
other practitioners in group activities, such as presenting successful 
girls' delinquency program practices at conferences and reviewing the 
group's work products.[Footnote 36] OJJDP officials stated that as a 
lesson learned, they plan to more fully involve practitioners from the 
beginning when they organize study groups in the future. In addition, 
OJJDP officials noted that specific to the Study Group, they plan to 
continue to reach out to obtain information from and include 
practitioners in the remaining activities of the group, such as 
presenting findings at a national juvenile justice conference. 

OJJDP and the Study Group have disseminated the group's findings to the 
research community, practitioners in the girls' delinquency field, and 
the public in a variety of ways, and in doing so have made efforts to 
respond to stakeholder concerns. In its 2003 program announcement, in 
keeping with the internal control standard for communicating with 
stakeholders, OJJDP required that the Study Group disseminate its 
findings through publications and products that address the needs of 
various practitioner audiences in diverse fields, including juvenile 
justice, child welfare, mental health, and substance abuse prevention. 
Since 2004, Study Group principal investigators and group members have 
presented findings at 24 conferences and posted the presentation slides 
to the group's Web site.[Footnote 37] OJJDP has also published three 
bulletins on the Study Group's activities and findings. Six girls' 
delinquency experts we interviewed (including five Study Group members) 
stated that the information disseminated was generally helpful because 
it provided a useful overview of girls' delinquency trends and 
research. However, 10 of the 18 experts we interviewed (including three 
Study Group members) also noted that some of the group's dissemination 
efforts created confusion among practitioners because Study Group 
members presented findings that did not acknowledge factors that 
practitioners believed contribute to girls' delinquency, such as 
traumatic life experiences. According to OJJDP officials, in response 
to feedback they received from girls' delinquency stakeholders about 
such concerns, the office and the Study Group sponsored workshop 
sessions at a conference for juvenile justice practitioners where the 
group clarified its findings and sought practitioner input on subjects 
such as delinquency risk and protective factors and trends in girls' 
delinquency. According to OJJDP officials, the office and the Study 
Group plan to continue disseminating the group's findings by issuing 
four additional bulletins and by presenting the findings at a national 
conference on juvenile delinquency. 

In Response to Study Group Findings of No Evidence of Effective Girls' 
Delinquency Programs, OJJDP Plans Technical Assistance to Help Programs 
but Could Strengthen Its Plans for Supporting Evaluations: 

The OJJDP-sponsored Study Group found that no programs in its review 
had evidence of effectiveness and, among other things, that additional 
support for program evaluation is needed. To address these findings, 
OJJDP plans to provide technical assistance to help girls' delinquency 
programs so that they will be better prepared to be evaluated. However, 
by articulating time frames and specific funding requirements and 
commitments in its plans to support evaluations, OJJDP could better 
address its goals for preventing and reducing girls' delinquency. 

The Study Group Found No Evidence of Effective Girls' Delinquency 
Programs to Promote as Models and, among Other Things, That Evaluation 
Is Needed: 

In its review of girls' delinquency programs, the Study Group's 
findings showed that the majority of the programs it identified--44 of 
61--had not been studied by researchers, while 17 of the programs had 
been the subject of published studies. The Study Group determined that 
none of the 17 programs that had been studied had conclusive evidence 
of their effectiveness. Specifically, the Study Group found that the 
studies provided insufficient evidence of the effectiveness of 11 of 
these 17 programs. For example, our review of one study that the Study 
Group assessed as having insufficient evidence showed that the study 
had a quasi-experimental design but lacked methodological rigor in that 
the treatment and comparison groups had small sample sizes and did not 
appear to be well matched, and any statistical tests reported were only 
performed on treatment group participants.[Footnote 38] The Study Group 
found that for the remaining 6 programs, the studies provided 
inconclusive evidence of effectiveness. For example, our review of one 
study that the group assessed as having inconclusive evidence showed 
statistically significant results for the program; however, sustained 
effects were not indicated for at least a 1-year period beyond the end 
of the intervention. Further, it was unclear whether the study 
participants were representative of the population of girls that the 
program was designed to reach. As a result, there was not enough 
evidence to demonstrate that the program was either effective or 
ineffective for the intended population of delinquent girls. Among the 
other findings that the Study Group reported was that 7 of the 17 
programs it assessed were no longer in operation, primarily because the 
initial grants that supported their operations were not renewed. 

Based on its review of girls' delinquency programs, the Study Group 
reported several conclusions and recommendations. Among these 
conclusions and recommendations is the need for evaluations and support 
of girls' delinquency programs. In particular, the Study Group found 
that insufficient funding has been provided for evaluations of girls' 
delinquency programs, so definitive conclusions of what works for girls 
cannot be made. Further, the Study Group found that additional, 
methodologically rigorous evaluations of girls' delinquency programs 
are needed in order to identify effective and promising programs and 
models that could be replicated at the state and local levels. While 
the Study Group did not specifically quantify the funding needed to 
support these evaluations, it did note that federal sources for 
evaluation funding and partnerships with local colleges and 
universities are needed. The Study Group also concluded that programs 
need technical assistance to help them prepare for evaluations. Lastly, 
the group found that girls' delinquency programs that are based on 
evidence of promising techniques should be supported and expanded. In 
particular, the Study Group highlighted program sustainability as an 
issue, stating that funding needs to be provided to ensure that the 
most promising programs continue to operate after their initial funding 
period is over so that practitioners and policymakers can continue to 
implement them. 

OJJDP Plans to Provide Technical Assistance to Help Programs, but Could 
More Fully Develop Plans for Supporting Evaluations to Address Its 
Goals to Prevent and Reduce Girls' Delinquency: 

OJJDP has plans to provide technical assistance to girls' delinquency 
programs; however, its plans for supporting evaluations could be more 
fully developed to help OJJDP reach its goals for addressing girls' 
delinquency issues. OJJDP's goals for addressing girls' delinquency, as 
stated in the Study Group program announcement, are to identify 
effective and promising programs, program elements, and implementation 
principles and support the development of program models to prevent and 
reduce girls' delinquency. According to OJJDP officials, in response to 
the group's finding about the need to better prepare programs for 
evaluation, the office plans to work with the Study Group and using the 
remainder of its funding--approximately $300,000--provide a technical 
assistance workshop in October 2009 to help about 10 girls' delinquency 
programs prepare to be evaluated. In this workshop, OJJDP and the Study 
Group plan to provide information to programs about how evaluations are 
designed and conducted, how to identify appropriate performance 
measures, and how to collect data needed for program evaluators to 
assess outcomes. OJJDP officials stated that the programs are to be 
selected for participation through an application process and have to 
meet minimum criteria, including having experience working with girls 
and the capability to collect program outcome data. OJJDP officials 
noted that they intend to limit participation in the workshop to about 
10 programs to ensure that the programs that are selected receive 
technical assistance that is targeted to their specific needs. This 
assistance, according to OJJDP officials, will help ensure that when 
programs do undergo evaluations--whether funded by OJJDP, another 
federal agency, or an independent research organization--the 
evaluations will be more likely to lead to conclusive findings on 
program effectiveness. 

In addition to providing girls' delinquency programs with training and 
technical assistance, OJJDP officials also described their plan to fund 
evaluations of girls' delinquency programs. OJJDP officials stated that 
as a result of the Study Group's findings along with feedback they 
received from members of the girls' delinquency field, they recognized 
the need for evaluations of girls' delinquency programs. OJJDP 
officials stated that they recognized the need for evaluation in fiscal 
year 2007 but at the time lacked funding to issue a solicitation for 
such evaluations. Further, 14 of the 18 girls' delinquency experts that 
we interviewed (including nine Study Group members) emphasized the need 
for OJJDP leadership in supporting evaluations of girls' delinquency 
programs to identify effective programs. For example, one expert noted 
that since the Study Group found that few programs had been studied, 
OJJDP would be doing a disservice to the girls' delinquency field if it 
did not fund rigorous evaluations and help programs partner with 
research organizations. According to the OJJDP officials, the office's 
goal is to issue a solicitation in early fiscal year 2010 for 
researchers to apply for funding to conduct evaluations of two to five 
girls' delinquency programs. These evaluations, according to OJJDP 
officials, are to focus on girls' delinquency programs that have been 
in operation for a number of years and have data to support 
evaluations. The officials also stated that the planned solicitation 
would require researchers to conduct studies that involve either 
randomized controlled or quasi-experimental research designs. 

OJJDP officials stated that they expect to fund evaluations using the 
portion of appropriation accounts that has been available for research 
and evaluations, and noted that the number of evaluations to be 
allocated funding depends, in part, on the number of applications 
received, the total available funding, as well as other competing 
research needs and goals. While OJJDP has not yet received an 
appropriation for fiscal year 2010, OJJDP used approximately $12 
million in fiscal year 2007 and $14 million in fiscal year 2008 to 
support research and evaluations from accounts eligible to support 
research and evaluations of girls' delinquency programs. OJJDP 
officials stated that they used this funding because in recent years 
they have not received an appropriation for programs and activities 
authorized under Part D, which is specifically designated for research 
and evaluation, but if they were to receive a Part D appropriation they 
could increase the number of evaluations funded. While OJJDP officials 
verbally described the planned evaluations and funding, they did not 
provide us with written documentation of the planned solicitation 
because, as of June 2009, it was in draft and subject to change. 

OJJDP officials have described actions they plan to take to respond to 
the Study Group's findings, and OJJDP reported that these findings will 
provide a foundation for creating a comprehensive program of 
information dissemination, training, technical assistance, and 
programming to help prevent and reduce girls' delinquency. However, the 
office has not developed a plan that is documented, is shared with key 
stakeholders, and includes time frames and specific funding 
requirements and commitments for meeting its girls' delinquency goals. 
According to OJJDP officials, they have not developed such a plan 
because the office is in transition and is in the process of initiating 
efforts to develop an officewide research plan, but they are taking 
steps to address their girls' delinquency goals, for example, through 
the workshop and planned evaluations. Standard practices for program 
and project management state that specific desired outcomes or results 
should be conceptualized, defined, and documented in the planning 
process as part of a road map, along with the appropriate projects 
needed to achieve those results, supporting resources, and 
milestones.[Footnote 39] Standards for Internal Control in the Federal 
Government states that program managers should ensure that there are 
adequate means of obtaining information from and communicating with 
external stakeholders who may have a significant impact on the agency 
achieving its goals.[Footnote 40] We have also previously reported that 
critical to guiding evaluation and research efforts on a national level 
is a strategy that outlines a process for funding and conducting 
rigorous evaluations and research, identifies the resources needed to 
achieve it, and assigns accountability for accomplishing these actions. 
[Footnote 41] In that regard, developing a plan that provides a road 
map to meeting its goals would provide additional assurance that 
OJJDP's goals for identifying and promoting promising programs and 
program models would be met and communicated to state and local 
policymakers and practitioners responsible for implementing programs to 
prevent and reduce girls' delinquency. 

Conclusions: 

Preventing and responding to girls' delinquency have been a concern for 
federal, state, and local governments as well as private and nonprofit 
juvenile justice organizations for over a decade, and the most recent 
statistics show that girls' involvement in the juvenile justice system 
is not stabilizing or declining. While OJJDP has undertaken a 6-year, 
$2.6 million study group effort to learn about effective and promising 
girls' delinquency programs, the lack of rigorous studies of such 
programs meant that the group was unable to identify and promote 
effective programs and to develop program models to be supported at 
state and local levels. In response to these findings, OJJDP has taken 
steps to provide technical assistance to programs to help prepare them 
for evaluations and has described plans for funding evaluations of 
girls' delinquency programs. While these steps are consistent with 
OJJDP's stated goals, the office lacks a comprehensive documented plan 
that includes time frames and specific funding requirements and 
commitments for meeting its girls' delinquency goals that it can share 
with stakeholders. As the Study Group plans to conclude its efforts in 
spring 2010, OJJDP is planning to help ensure the development of 
effective girls' delinquency programs and program models by providing 
training and technical assistance to help these programs plan for 
future evaluations. Moreover, such action better positions OJJDP in 
ensuring that funding for such programs is directed to those that are 
effective in preventing girls' delinquency and intervening after girls 
have entered the juvenile justice system. As states are continuing to 
make determinations about how to allocate their formula and block 
grants, and OJJDP continues to provide funding to programs through some 
of its discretionary grant programs, information about promising or 
effective programs and program models could help guide these resource 
decisions. Developing a plan with time frames that clearly articulates 
the office's approach to its evaluation efforts, including available 
resources needed and committed toward implementing that plan, would 
help OJJDP ensure that its goals to support the development of 
effective programs are met, and sharing that plan with stakeholders 
would help demonstrate federal leadership to the girls' delinquency 
field. 

Recommendation for Executive Action: 

To help ensure that OJJDP meets its goals to identify effective or 
promising girls' delinquency programs and supports the development of 
program models, we recommend that the Administrator of OJJDP develop 
and document a plan that (1) articulates how the office intends to 
respond to the program findings of the Study Group, (2) includes time 
frames and specific funding requirements and commitments, and (3) is 
shared with key stakeholders. 

Agency Comments and Our Evaluation: 

We requested comments on a draft of this report from the Attorney 
General. On July 16, 2009, we received written comments from OJP, which 
are reprinted in the enclosure. 

OJP agreed with our recommendation and stated that OJJDP has always 
intended to respond to the findings of the Study Group. OJP described 
efforts planned in response to the findings of the Study Group, 
including a technical assistance workshop and evaluations, which we 
have discussed in our report. OJP also stated that subsequent refined 
plans and related funding commitments will be based on the outcome of 
these activities and noted that OJJDP, in accordance with the Act, will 
publish these program plans in the Federal Register for review and 
comment by key stakeholders as well as members of the public by 
December 2009. 

We recognize that OJJDP's planned activities represent a worthwhile 
step in responding to the findings from the Study Group effort, and are 
encouraged that OJJDP intends to publish a program plan, to include how 
it will address girls' delinquency issues. However, it is important to 
note that while OJJDP has been required to publish a program plan 
annually according to the Act, it has not done so since 2002.[Footnote 
42] Following through on its current pledge to issue such a plan by 
December of this year will help provide OJJDP with reasonable assurance 
that it has a well-thought-out approach to ensure that its goals for 
preventing and reducing girls' delinquency are met. We also continue to 
maintain that it will be important for this plan to include more than a 
list of activities in response to the Study Group's findings as OJJDP 
describes in commenting on this report. Specifically, the plan should 
serve as a road map for OJJDP's approach for responding to the Study 
Group's findings, establish overall time frames as well as those for 
each activity, specify funding requirements and associated commitments, 
and integrate the input of key stakeholders, such as girls' delinquency 
practitioners. Publishing and implementing such a plan would help OJJDP 
ensure that it meets the goal it articulated at the beginning of the 6- 
year Study Group effort--to identify effective and promising programs, 
program elements, and implementation principles and to support the 
development of program models to prevent and reduce girls' delinquency. 

We are sending copies of this report to interested congressional 
committees, the Attorney General, and other interested parties. In 
addition, this report will be available at no charge on GAO's Web site 
at [hyperlink, http://www.gao.gov]. 

If you or your staff have any questions concerning this report, please 
contact me at (202) 512-6510 or larencee@gao.gov. Contact points for 
our Offices of Congressional Relations and Public Affairs may be found 
on the last page of this report. Mary Catherine Hult, Assistant 
Director; David Alexander; Elizabeth Blair; Amy Brown; Kevin Copping; 
Katherine Davis; Dawn Locke; and Janet Temko made key contributions to 
this report. 

Sincerely yours, 

Signed by: 

Eileen Regen Larence:
Director, Homeland Security and Justice Issues: 

Enclosure: 

[End of section] 

Enclosure: Comments from the Department of Justice: 

U.S. Department of Justice: 
Office of Justice Programs: 
Office of Audit, Assessment, and Management: 
Washington, DC 20531: 

July 16, 2009: 

Ms. Eileen R. Larence: 
Director, Homeland Security and Justice Issues: 
Government Accountability Office: 
441 G Street, NW: 
Washington, DC 20548: 

Dear Ms. Larence: 

Thank you for the opportunity to comment on the draft Government 
Accountability Office (GAO) letter report entitled "Juvenile Justice: 
Technical Assistance and Better Defined Evaluation Plans Will Help to 
Improve Girls' Delinquency Programs" (GAO-09721R). The Office of 
Justice Programs agrees with the Recommendation for Executive Action, 
which is restated in bold text below and is followed by our response. 

To help ensure that OJJDP meets its goals to identify effective or 
promising girls' delinquency programs and supports the development of 
program models, we recommend that the Administrator of OJJDP develop 
and document a plan, that (1) articulates how the agency intends to 
respond to the program findings of the Girls Study Group, (2) includes 
time frames and specific funding requirements and commitments, and (3) 
is shared with key stakeholders. 

It has always been the intention of the Office of Juvenile Justice and 
Delinquency Prevention (OJJDP) to respond to the findings of the Girls' 
Study Group. As discussed with the GAO during the review, OJJDP has 
efforts underway, as well as planned initiatives to address the 
findings. These planned initiatives, and timeframes for implementation, 
are described below. Further planning and funding commitments will be 
based on the outcome of these activities. 

* Hands-On Evaluation Technical Assistance Workshop for Girls' 
Delinquency Programs: 

The workshop is scheduled for October 28-30, 2009, in Chapel Hill, NC. 
The goal of the workshop is to better equip programs to conduct 
rigorous evaluations of their interventions. Unlike general workshops, 
the Girls' Study Group Evaluation Technical Assistance Workshop will 
tailor instruction specifically to address the needs of participating 
programs. The faculty at the workshops will be highly skilled in 
evaluation methodology, program development, and strategies on how to 
partner with evaluation professionals. Each participant will leave the 
workshop with a customized concrete plan for `next steps' and upon 
request will receive an additional hour of technical assistance by 
phone following the workshop. 

Eligibility will be limited to programs that provide gender-responsive 
delinquency prevention or interventions for girls, and who have some 
level of evaluation experience. Organizations that have more than one 
distinct gender responsive program are eligible to submit more than one 
application. There will be a two-phase application process. The first 
phase will require the submission of general program information and a 
description of current evaluation history/experience. Program 
applications will be reviewed based on their program type and 
evaluation needs; and approximately 15 to 20 programs will be invited 
to proceed to the next application phase. Those applicants selected for 
the second phase will be asked to submit more detailed information on 
the evaluation needs of their programs and reports or findings based on 
previous evaluation work. The workshop organizers will use this 
information to determine which programs provide the best fit between 
evaluation needs and faculty expertise. Approximately 10 programs will 
ultimately be invited to participate in the workshop. Selection of 
participants for the workshop will be completed by the end of September 
2009. 

* Enhancement of OJJDP's Current Girls Delinquency Training and 
Technical Assistance Curriculum: 

Using a panel of experts (including staff from the Girls' Study Group, 
among others), OJJDP will update, enhance, and revise the existing 
Training and Technical Assistance Curriculum for Girls' Delinquency 
Programming. The targeted completion date is December 2009. 

* Release of the FY 2010 Evaluation of Girls' Delinquency Programs 
Solicitation: 

This solicitation will be released, pending availability of funds, for 
the purpose of encouraging partnerships between girls' delinquency 
programs and evaluators, and providing funding for experimental and 
quasi-experimental evaluations of girls' delinquency programs.
In response to the findings of the Girls' Study Group, OJJDP has 
planned the above described initiatives. Subsequent refined plans and 
related funding commitments will be based on the outcome of these 
initiatives. As mandated by the Juvenile Justice and Delinquency 
Prevention Act, the OJJDP will publish these program plans in the 
Federal Register for review and comment by key stakeholders as well as 
members of the public. OJJDP anticipates publishing the program plan in 
the Federal Register by December 2009. 

If you have any questions regarding this response, you or your staff 
may contact Maureen Henneberg, Director, Office of Audit, Assessment, 
and Management, on (202) 616-3282. 

Sincerely, 

Signed by: 

Laurie O. Robinson: 
Acting Assistant Attorney General: 

cc: 
Beth McGarry: 
Deputy Assistant Attorney for Operations and Management: 

Jeffrey Slowikowski: 
Acting Director: 
Office of Juvenile Justice and Delinquency Prevention: 

Maureen Henneberg: 
Director: 
Office of Audit, Assessment, and Management: 

LeToya A. Johnson: 
Audit Liaison: 
Office of Justice Programs: 

Richard P. Theis: 
Audit Liaison: 
Department of Justice: 

[End of section] 

Footnotes: 

[1] C. Puzzanchera and W. Kang, Juvenile Court Statistics Databook 
(2008), [hyperlink, http://ojjdp.ncjrs.gov/ojstatbb/jcsdb/] (accessed 
June 30, 2009). Most current data available. 

[2] C. Puzzanchera, Juvenile Arrests 2007, (2009) [hyperlink, 
www.ncjrs.gov/pdffiles1/ojjdp/225344.pdf] (accessed June 26, 2009). 

[3] The Federal Advisory Committee on Juvenile Justice is an advisory 
body established by the Juvenile Justice and Delinquency Prevention Act 
of 1974, as amended, to advise the President and Congress on state 
perspectives regarding the operation of the Office of Juvenile Justice 
and Delinquency Prevention and on federal legislation pertaining to 
juvenile justice and delinquency prevention, to advise the 
Administrator of the Office of Juvenile Justice and Delinquency 
Prevention, and to review federal policies regarding juvenile justice 
and delinquency prevention. 42 U.S.C. § 5633(f). The Federal Advisory 
Committee on Juvenile Justice comprises appointed representatives from 
each of the 50 states, the District of Columbia, and the 5 U.S. 
territories. 

[4] Cooperative agreements, rather than grant awards, can be used by 
federal agencies when substantial involvement is expected between the 
agency and the recipient when carrying out the activities described in 
the program announcement. 

[5] GAO defines an expert as a person who is recognized by others who 
work in the same subject matter area as having knowledge that is 
greater in scope or depth than that of most people working in that 
area. The expert's knowledge can come from education, experience, or 
both. We specifically identified researchers who focus on girls' 
delinquency issues and practitioners who operate programs that address 
girls' delinquency. 

[6] We contacted all 15 of the study group members. However, 1 member 
declined to be interviewed, and 3 study group members did not respond 
to requests for interviews. 

[7] For social science standards for evaluation research, see Donald T. 
Campbell and Julian Stanley, Experimental and Quasi-Experimental 
Designs for Research (Chicago: Rand McNally, 1963); William R. Shadish, 
Thomas D. Cook, and Donald T. Campbell, Experimental and Quasi- 
Experimental Designs for Generalized Causal Inference (Boston: Houghton 
Mifflin, 2002); Carol H. Weiss, Evaluation: Methods for Studying 
Programs and Policies, Second Edition (Englewood Cliffs, N.J.: Prentice-
Hall, Inc., 1998); and GAO, Designing Evaluations, [hyperlink, 
http://www.gao.gov/products/GAO/PEMD-10.1.4] (Washington, D.C.: May 
1991). 

[8] GAO, Standards for Internal Control in the Federal Government, 
[hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1] 
(Washington, D.C.: November 1999). 

[9] Program management standards we reviewed are reflected in the 
Project Management Institute's The Standard for Program Management © 
(2006). 

[10] The What Works criteria define six levels of effectiveness, 
including effective, promising, and ineffective, for use in assessing 
and classifying studies on the basis of their evidence of 
effectiveness. Additional details on these criteria are discussed later 
in this report. 

[11] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]. 

[12] The other seven experts did not express views regarding the 
balance of the study group's composition. 

[13] Project Management Institute, The Standard for Program Management. 

[14] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]. 

[15] The Study Group found that possible reasons for increased arrest 
rates for girls include changes in local law enforcement policies that 
lowered the threshold for reporting assaults or categorizing assaults 
as aggravated, reclassification of domestic dispute offenses as simple 
assaults that can result in arrest, and increased referrals to police 
resulting from schools' zero tolerance policies for violence. 

[16] Elizabeth Cauffman and others, "Gender Differences in Mental 
Health Symptoms among Delinquent and Community Youth," Youth Violence 
and Juvenile Justice, vol. 5, no. 3 (2007): 287-307. 

[17] 42 U.S.C. § 5611. 

[18] OJJDP allocates some formula and block grants to states on the 
basis of states' juvenile populations, while others may be awarded on 
the basis of a fixed level to all states. Discretionary grants are 
generally awarded through a competitive process to state and local 
governments as well as individual agencies and organizations. Under the 
Act, "state" means any of the United States, the District of Columbia, 
the Commonwealth of Puerto Rico, the Virgin Islands, Guam, American 
Samoa, and the Commonwealth of the Northern Mariana Islands. 42 U.S.C. 
§ 5603. 

[19] 42 U.S.C. § 5633(a)(7)(B). 

[20] OJJDP did not report awarding discretionary grants for girls' 
programs in fiscal year 2008, and as of June 2009, OJJDP had not 
awarded fiscal year 2009 discretionary grants. 

[21] 42 U.S.C. § 5614(b)(3). 

[22] 42 U.S.C. §§ 5661-62. 

[23] Appropriations statutes for fiscal years 2006 through 2008 
provided that OJJDP may use not more than 10 percent of each amount 
appropriated for research, evaluation, and statistics activities that 
benefit the programs or activities authorized, and not more than 2 
percent of each appropriated amount for training and technical 
assistance. See, e.g, Consolidated Appropriations Act, 2008, Pub. L. 
No. 110-161, 121 Stat. 1844, 1906-07 (2007). This provision applied to 
appropriation accounts under Juvenile Justice Programs, but did not 
apply to amounts appropriated for demonstration projects, as authorized 
by sections 261 and 262 of the Act, 42 U.S.C. §§ 5665-66. 

[24] The $40 million comprises set asides eligible to be used for 
research and evaluation of girls' delinquency programs. As of July 
2009, OJJDP has not determined how it would use its fiscal year 2009 
appropriation set asides. 

[25] Greene, Peters Associates, Guiding Principles for Promising Female 
Programming: An Inventory of Best Practices (Washington, D.C.: Office 
of Juvenile Justice and Delinquency Prevention, 1998). 

[26] In 2000, OJJDP issued program announcements for two separate 
girls' delinquency efforts--a study group and a girls' institute. The 
first effort, a girls study group, was awarded to a university in 2001. 
However, because it was unable to reach agreement on project management 
issues, the university terminated the agreement in 2002. In 2003, OJJDP 
reissued the program announcement for a girls study group and revised 
the announcement to clearly delineate the level of expected federal 
involvement, for example, by explicitly stating that OJJDP planned to 
review and approve all project consultants, plans, and products 
developed. The second effort, a national girls' institute, was intended 
to put the study group's findings into practice by, among other things, 
promoting programs for girls; providing training and technical 
assistance to the field on girls' delinquency issues; facilitating 
coordination among federal, state, and local organizations serving 
girls; and disseminating information about the research findings of the 
study group. According to OJJDP officials, the 2000 announcement never 
received funding, and OJJDP did not reissue it in later years because 
of funding constraints. Instead, when OJJDP reissued the program 
announcement in 2003 for a study group, it incorporated elements of the 
planned institute. For example, the 2003 study group solicitation 
included objectives for identifying and promoting programs for girls 
and for disseminating information to the practitioner field. 

[27] Specifically, the Study Group's objective to understand the trends 
and consequences of girls' delinquency involved increasing research- 
based knowledge about the risk and protective factors related to girls' 
delinquency and determining the patterns and consequences of juvenile 
justice decisions on female offenders. The objective on developing a 
comprehensive theory of girls' delinquency involved examining the 
extent to which theories developed primarily to explain boys' 
delinquency applied to girls, as well as exploring whether theories 
that had been developed for girls were useful in developing and testing 
new prevention and intervention strategies. 

[28] Under the Act, states are required to submit 3-year plans to OJJDP 
outlining their activities for investing in delinquency prevention and 
for coordinating services delivered to at-risk juveniles and their 
families, among other things. 42 U.S.C. § 5633. 

[29] A multiagency working group led by DOJ's OJP, which included the 
Department of Health and Human Services and the Department of 
Education, developed the classification framework and criteria from 
2004 to 2005 to support a planned What Works repository to assist 
communities in selecting and replicating evidence-based programs that 
was never implemented. Federal government efforts to develop 
repositories of evidence-based programs have continued under Find Youth 
Info, the Substance Abuse and Mental Health Services Administration's 
National Registry of Evidence-Based Programs and Practices, and the 
OJJDP Model Programs Guide. Even though the repository was never 
implemented, the criteria within its framework are still valid to use 
in assessing evidence of program effectiveness. 

[30] The approach used by OJJDP is similar to the evaluation synthesis 
methodology described in GAO, The Evaluation Synthesis, GAO/PEMD-10.1.2 
(Washington, D.C.: March 1992). This type of approach might also be 
termed systematic review. 

[31] Two experts we interviewed did not express a view on the group's 
approach to evaluating programs. 

[32] GAO, Juvenile Justice: OJJDP Reporting Requirements for 
Discretionary and Formula Grantees and Concerns About Evaluation 
Studies, [hyperlink, http://www.gao.gov/products/GAO-02-23] 
(Washington, D.C.: Oct. 30, 2001); Justice Outcome Evaluations: Design 
and Implementation of Studies Require More NIJ Attention, [hyperlink, 
http://www.gao.gov/products/GAO-03-1091] (Washington, D.C.: Sept. 24, 
2003); Adult Drug Courts: Evidence Indicates Recidivism Reductions and 
Mixed Results for Other Outcomes, [hyperlink, 
http://www.gao.gov/products/GAO-05-219] (Washington, D.C.: Feb. 28, 
2005); and Abstinence Education: Assessing the Accuracy and 
Effectiveness of Federally Funded Programs, [hyperlink, 
http://www.gao.gov/products/GAO-08-664T] (Washington, D.C.: Apr. 23, 
2008). 

[33] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]. 

[34] The Study Group members represented 11 of the 12 disciplines 
specified in the program announcement. 

[35] The other seven experts did not express views regarding the 
balance of the study group's composition. 

[36] In addition to the practitioner, OJJDP also added an expert in 
program evaluation as a group member after the group had begun its 
activities. 

[37] The Web site is located at [hyperlink, 
http://girlsstudygroup.rti.org/index.cfm?fuseaction=dsp_home]. 

[38] A quasi-experimental design is a controlled study where study 
participants are assigned in a nonrandom manner to a treatment group 
(individuals participating in the program being studied) or a 
comparison group (individuals closely resembling those in the treatment 
group on many demographic variables but not participating in the 
program). 

[39] Project Management Institute, The Standard for Program Management. 

[40] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]. 

[41] GAO, South Florida Ecosystem Restoration: A Strategic Plan and a 
Process to Resolve Conflicts Are Needed to Keep the Effort on Track, 
[hyperlink, http://www.gao.gov/products/GAO/T-RCED-99-170] (Washington, 
D.C.: Apr. 29, 1999); South Florida Ecosystem Restoration: Substantial 
Progress Made in Developing a Strategic Plan, but Actions Still Needed, 
[hyperlink, http://www.gao.gov/products/GAO-01-361] (Washington, D.C.: 
Mar. 27, 2001); and Great Lakes: A Coordinated Strategic Plan and 
Monitoring System Are Needed to Achieve Restoration Goals, [hyperlink, 
http://www.gao.gov/products/GAO-03-999T] (Washington, D.C.: July 16, 
2003). 

[42] 42 U.S.C. § 5614. 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "E-mail Updates." 

Order by Phone: 

The price of each GAO publication reflects GAO’s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO’s Web site, 
[hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional 
information. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: fraudnet@gao.gov: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Ralph Dawn, Managing Director, dawnr@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548: 

Public Affairs: 

Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: