This is the accessible text file for GAO report number GAO-10-133T 
entitled 'Juvenile Justice: Technical Assistance and Better Defined 
Evaluation Plans Will Help Girls' Delinquency Program' which was 
released on October 20, 2009. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Testimony: 

Before the Subcommittee on Crime, Terrorism, and Homeland Security, 
Committee on the Judiciary, House of Representatives: 

United States Government Accountability Office: 

GAO: 

For Release on Delivery Expected at 2:30 p.m. EDT: 

Tuesday, October 20, 2009: 

Juvenile Justice: 

Technical Assistance and Better Defined Evaluation Plans Will Help 
Girls' Delinquency Programs: 

Statement of Eileen R. Larence, Director: 

Homeland Security and Justice: 

[End of section] 

GAO-10-133T: 

Mr. Chairman and Members of the Subcommittee: 

I am pleased to be here today as you examine issues related to girls' 
delinquency--a topic that has attracted the attention of federal, 
state, and local policymakers for more than a decade as girls have 
increasingly become involved in the juvenile justice system. For 
example, from 1995 through 2005, delinquency caseloads for girls in 
juvenile justice courts nationwide increased 15 percent while boys' 
caseloads decreased by 12 percent.[Footnote 1] More recently, in 2007, 
29 percent of juvenile arrests--about 641,000 arrests--involved girls, 
who accounted for 17 percent of juvenile violent crime arrests and 35 
percent of juvenile property crime arrests.[Footnote 2] Further, 
research on girls has highlighted that delinquent girls have higher 
rates of mental health problems than delinquent boys, receive fewer 
special services, and are more likely to abandon treatment 
programs.[Footnote 3] 

The Office of Juvenile Justice and Delinquency Prevention (OJJDP) is 
the Department of Justice (DOJ) office charged with providing national 
leadership, coordination, and resources to prevent and respond to 
juvenile delinquency and victimization. OJJDP supports states and 
communities in their efforts to develop and implement effective 
programs to, among other things, prevent delinquency and intervene 
after a juvenile has offended. For example, from fiscal years 2007 
through 2009, Congress provided OJJDP almost $1.1 billion to use for 
grants to states, localities, and organizations for a variety of 
juvenile justice programs, including programs for girls. Also, in 
support of this mission, the office funds research and program 
evaluations related to a variety of juvenile justice issues. 

As programs have been developed at the state and local levels in recent 
years that specifically target preventing girls' delinquency or 
intervening after girls have become involved in the juvenile justice 
system, it is important that agencies providing grants and 
practitioners operating the programs have information about which of 
these programs are effective. In this way, agencies can help to ensure 
that limited federal, state, and local funds are well spent. In 
general, effectiveness is determined through program evaluations, which 
are systematic studies conducted to assess how well a program is 
working--that is, whether a program produced its intended effects. To 
help ensure that grant funds are being used effectively, you asked us 
to review OJJDP's efforts related to studying and promoting effective 
girls' delinquency programs. We issued a report on the results of that 
review on July 24, 2009.[Footnote 4] My statement today, as requested, 
highlights findings from that report and addresses (1) efforts OJJDP 
has made to assess the effectiveness of girls' delinquency programs, 
(2) the extent to which these efforts are consistent with generally 
accepted social science standards and federal standards to communicate 
with stakeholders, and (3) the findings from OJJDP's efforts and how 
the office plans to address the findings. 

My statement is based on our July report and selected updates made in 
October 2009.[Footnote 5] For our report, we reviewed documentation 
about OJJDP's establishment of a study group to assess the 
effectiveness of girl's delinquency programs, analyzed the groups' 
activities and findings, and interviewed OJJDP research and program 
officials and the current and former principal investigators of the 
study group. Specifically, we reviewed the criteria the study group 
used to assess studies of girls' delinquency programs and whether the 
group's application of those criteria was consistent with generally 
accepted social science standards for evaluation research.[Footnote 6] 
We also compared OJJDP's efforts with criteria in Standards for 
Internal Control in the Federal Government, specifically that agency 
management should ensure that there are adequate means of obtaining 
information from and communicating with external stakeholders who may 
have a significant impact on the agency achieving its goals, such as 
practitioners operating programs or researchers assessing 
programs.[Footnote 7] In addition, we conducted interviews with 18 
girls' delinquency subject matter experts, that is, researchers and 
practitioners, whom we selected on the basis of their knowledge and 
experience with girls' delinquency issues.[Footnote 8] While their 
comments cannot be generalized to all girls' delinquency experts, we 
nonetheless believe that their views gave us useful insights on issues 
related to girls' delinquency and OJJDP's efforts to assess girls' 
programs. Our work was performed in accordance with generally accepted 
government auditing standards. More detail about our scope and 
methodology is included in our July report.[Footnote 9] 

OJJDP Established the Girls Study Group to Assess the Effectiveness of 
Girls' Delinquency Programs: 

With an overall goal of developing research that communities need to 
make sound decisions about how best to prevent and reduce girls' 
delinquency, OJJDP established the Girls Study Group (Study Group) in 
2004 under a $2.6 million multiyear cooperative agreement with a 
research institute.[Footnote 10] OJJDP's objectives for the group, 
among others, included identifying effective or promising programs, 
program elements, and implementation principles (i.e., guidelines for 
developing programs). Objectives also included developing program 
models to help inform communities of what works in preventing or 
reducing girls' delinquency, identifying gaps in girls' delinquency 
research and developing recommendations for future research, and 
disseminating findings to the girls' delinquency field about effective 
or promising programs. To meet OJJDP's objectives, among other 
activities, the Study Group identified studies of delinquency programs 
that specifically targeted girls by reviewing over 1,000 documents in 
relevant research areas. These included criminological and feminist 
explanations for girls' delinquency, patterns of delinquency, and the 
justice system's response to girls' delinquency. As a result, the group 
identified 61 programs that specifically targeted preventing or 
responding to girls' delinquency. Then, the group assessed the 
methodological quality of the studies of the programs that had been 
evaluated using a set of criteria developed by DOJ's Office of Justice 
Programs (OJP) called What Works to determine whether the studies 
provided credible evidence that the programs were effective at 
preventing or responding to girls' delinquency.[Footnote 11] The 
results of the group's assessment are discussed in the following 
sections. 

OJJDP Efforts to Assess Program Effectiveness Were Consistent with 
Social Science Practices and Standards, and OJJDP Has Taken Action to 
Enhance Communication about the Study Group with External Stakeholders: 

OJJDP's effort to assess girls' delinquency programs through the use of 
a study group and the group's methods for assessing studies were 
consistent with generally accepted social science research practices 
and standards. In addition, OJJDP's efforts to involve practitioners in 
Study Group activities and disseminate findings were also consistent 
with the internal control standard to communicate with external 
stakeholders, such as practitioners operating programs.[Footnote 12] 

According to OJJDP research and program officials, they formed the 
Study Group rather than funding individual studies of programs because 
study groups provide a cost-effective method of gaining an overview of 
the available research in an issue area. As part of its work, the group 
collected, reviewed, and analyzed the methodological quality of 
research on girls' delinquency programs. The use of such a group, 
including its review, is an acceptable approach for systematically 
identifying and reviewing research conducted in a field of study. This 
review helped consolidate the research and provide information to OJJDP 
for determining evaluation priorities. Further, we reviewed the 
criteria the group used to assess the studies and found that they 
adhere to generally accepted social science standards for evaluation 
research. We also generally concurred with the group's assessments of 
the programs based on these criteria. According to the group's former 
principal investigator, the Study Group decided to use OJP's What Works 
criteria to ensure that its assessment of program effectiveness would 
be based on highly rigorous evaluation standards, thus eliminating the 
potential that a program that may do harm would be endorsed by the 
group. However, 8 of the 18 experts we interviewed said that the 
criteria created an unrealistically high standard, which caused the 
group to overlook potentially promising programs. OJJDP officials 
stated that despite such concerns, they approved the group's use of the 
criteria because of the methodological rigor of the framework and their 
goal for the group to identify effective programs. 

In accordance with the internal control standard to communicate with 
external stakeholders, OJJDP sought to ensure a range of stakeholder 
perspectives related to girls' delinquency by requiring that Study 
Group members possess knowledge and experience with girls' delinquency 
and demonstrate expertise in relevant social science disciplines. The 
initial Study Group, which was convened by the research institute and 
approved by OJJDP, included 12 academic researchers and 1 practitioner; 
someone with experience implementing girls' delinquency programs. 
However, 11 of the 18 experts we interviewed stated that this 
composition was imbalanced in favor of academic researchers. In 
addition, 6 of the 11 said that the composition led the group to focus 
its efforts on researching theories of girls' delinquency rather than 
gathering and disseminating actionable information for 
practitioners.[Footnote 13] According to OJJDP research and program 
officials, they acted to address this issue by adding a second 
practitioner as a member and involving two other practitioners in study 
group activities. OJJDP officials stated that they plan to more fully 
involve practitioners from the beginning when they organize study 
groups in the future and to include practitioners in the remaining 
activities of the Study Group, such as presenting successful girls' 
delinquency program practices at a national conference. Also, in 
accordance with the internal control standard, OJJDP and the Study 
Group have disseminated findings to the research community, 
practitioners in the girls' delinquency field, and the public through 
conference presentations, Web site postings, and published bulletins. 
The group plans to issue a final report on all of its activities by 
spring 2010. 

The Study Group Found No Evidence of Effective Girls' Delinquency 
Programs; in Response OJJDP Plans to Assist Programs in Preparing for 
Evaluations but Could Strengthen Its Plans for Supporting Such 
Evaluations: 

The Study Group found that few girls' delinquency programs had been 
studied and that the available studies lacked conclusive evidence of 
effective programs; as a result, OJJDP plans to provide technical 
assistance to help programs be better prepared for evaluations of their 
effectiveness. However, OJJDP could better address its girls' 
delinquency goals by more fully developing plans for supporting such 
evaluations. 

In its review, the Study Group found that the majority of the girls' 
delinquency programs it identified--44 of the 61--had not been studied 
by researchers. For the 17 programs that had been studied, the Study 
Group reported that none of the studies provided conclusive evidence 
with which to determine whether the programs were effective at 
preventing or reducing girls' delinquency. For example, according to 
the Study Group, the studies provided insufficient evidence of the 
effectiveness of 11 of the 17 programs because, for instance, the 
studies involved research designs that could not demonstrate whether 
any positive outcomes, such as reduced delinquency, were due to program 
participation rather than other factors. Based on the results of this 
review, the Study Group reported that among other things, there is a 
need for additional, methodologically rigorous evaluations of girls' 
delinquency programs; training and technical assistance to help 
programs prepare for evaluations; and funding to support girls' 
delinquency programs found to be promising. 

According to OJJDP officials, in response to the Study Group's finding 
about the need to better prepare programs for evaluation, the office 
plans to work with the group and use the remaining funding from the 
effort--approximately $300,000--to provide a technical assistance 
workshop by the end of October 2009. The workshop is intended to help 
approximately 10 girls' delinquency programs prepare for evaluation by 
providing information about how evaluations are designed and conducted 
and how to collect data that will be useful for program evaluators in 
assessing outcomes, among other things. In addition, OJJDP officials 
stated that as a result of the Study Group's findings, along with 
feedback they received from members of the girls' delinquency field, 
OJJDP plans to issue a solicitation in fiscal year 2010 for funding to 
support evaluations of girls' delinquency programs. 

OJJDP has also reported that the Study Group's findings are to provide 
a foundation for moving ahead on a comprehensive program related to 
girls' delinquency. However, OJJDP has not developed a plan that is 
documented, is shared with key stakeholders, and includes specific 
funding requirements and commitments and time frames for meeting its 
girls' delinquency goals. Standard practices for program and project 
management state that specific desired outcomes or results should be 
conceptualized, defined, and documented in the planning process as part 
of a road map, along with the appropriate projects needed to achieve 
those results, supporting resources, and milestones.[Footnote 14] In 
addition, government internal control standards call for policies and 
procedures that establish adequate communication with stakeholders as 
essential for achieving desired program goals.[Footnote 15] According 
to OJJDP officials, they have not developed a plan for meeting their 
girls' delinquency goals because the office is in transition and is in 
the process of developing a plan for its juvenile justice 
programs,[Footnote 16] but the office is taking steps to address its 
girls' delinquency goals, for example, through the technical assistance 
workshop. Developing a plan for girls' delinquency would help OJJDP to 
demonstrate leadership to the girls' delinquency field by clearly 
articulating the actions it intends to take to meet its goals and would 
also help the office to ensure that the goals are met. 

In our July report, we recommended that to help ensure that OJJDP meets 
its goals to identify effective or promising girls' delinquency 
programs and supports the development of program models, the 
Administrator of OJJDP develop and document a plan that (1) articulates 
how the office intends to respond to the findings of the Study Group, 
(2) includes time frames and specific funding requirements and 
commitments, and (3) is shared with key stakeholders. OJP agreed with 
our recommendation and outlined efforts that OJJDP plans to undertake 
in response to these findings. For example, OJJDP stated that it 
anticipates publishing its proposed juvenile justice program plan, 
which is to include how it plans to address girls' delinquency issues, 
in the Federal Register to solicit public feedback and comments, which 
will enable the office to publish a final plan in the Federal Register 
by the end of the year (December 31, 2009). 

Mr. Chairman, this concludes my statement. I would be pleased to 
respond to any questions that you or other Members of the Subcommittee 
may have. 

Contacts and Acknowledgements: 

For questions about this statement, please contact Eileen R. Larence at 
(202) 512-8777 or larencee@gao.gov. Contact points for our Offices of 
Congressional Relations and Public Affairs may be found on the last 
page of this statement. Individuals making key contributions to this 
statement include Mary Catherine Hult, Assistant Director; Kevin 
Copping; and Katherine Davis. Additionally, key contributors to our 
July 2009 report include David Alexander, Elizabeth Blair, and Janet 
Temko. 

[End of section] 

Footnotes:  

[1] C. Puzzanchera and W. Kang, Juvenile Court Statistics Databook 
(2008), [hyperlink, http://ojjdp.ncjrs.gov/ojstatbb/jcsdb/] (accessed 
Oct.15, 2009). This Web site provides the most current data available. 

[2] C. Puzzanchera, Juvenile Arrests 2007, (2009) 
www.ncjrs.gov/pdffiles1/ ojjdp/225344.pdf (accessed Oct.15, 2009). 

[3] Elizabeth Cauffman and others, "Gender Differences in Mental Health 
Symptoms among Delinquent and Community Youth," Youth Violence and 
Juvenile Justice, vol. 5, no. 3 (2007): 287-307. Elizabeth Caufmann 
"Understanding the Female Offender" The Future of Children, vol. 18, 
no. 2 (2008): 119-142. 

[4] GAO, Juvenile Justice: Technical Assistance and Better Defined 
Evaluation Plans Will Help to Improve Girls' Delinquency Programs, 
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-09-721R] (Washington, 
D.C.: July 24, 2009). 

[5] In a September 18, 2009, letter regarding the recommendation we 
made in our July report, DOJ clarified actions it was taking to address 
our recommendation, which we have included in this statement. 

[6] For social science standards for evaluation research, see Donald T. 
Campbell and Julian Stanley, Experimental and Quasi-Experimental 
Designs for Research (Chicago: Rand McNally, 1963); William R. Shadish, 
Thomas D. Cook, and Donald T. Campbell, Experimental and Quasi- 
Experimental Designs for Generalized Causal Inference (Boston: Houghton 
Mifflin, 2002); Carol H. Weiss, Evaluation: Methods for Studying 
Programs and Policies, Second Edition (Englewood Cliffs, N.J.: Prentice-
Hall, Inc., 1998); and GAO, Designing Evaluations, [hyperlink, 
http://www.gao.gov/cgi-bin/getrpt?GAO/PEMD-10.1.4] (Washington, D.C.: 
May 1991). 

[7] GAO, Standards for Internal Control in the Federal Government, 
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?/GAO/AIMD-00-21.3.1] 
(Washington, D.C.: November 1999). 

[8] GAO defines an expert as a person who is recognized by others who 
work in the same subject matter area as having knowledge that is 
greater in scope or depth than that of most people working in that 
area. The expert's knowledge can come from education, experience, or 
both. We identified researchers who focus on girls' delinquency issues 
and practitioners who operate programs that address girls' delinquency. 
Specifically, these 18 experts included 11 of the 15 study group 
members and 7 experts who were not members of the group. While we 
contacted all 15 of the study group members, 4 members either did not 
respond to requests for interviews or declined to be interviewed. 

[9] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-09-721R]. 

[10] Cooperative agreements, rather than grant awards, can be used by 
federal agencies when substantial involvement is expected between the 
agency and the recipient when carrying out the activities described in 
the program announcement. OJJDP extended the cooperative agreement with 
the research institute through June 2010 to complete all of the Study 
Group activities. 

[11] The What Works criteria define six levels of effectiveness, 
including effective, promising, and ineffective, for use in assessing 
and classifying studies on the basis of their evidence of 
effectiveness. The criteria for an effective program include a 
randomized controlled research design--a design that compares the 
outcomes for individuals who are randomly assigned to either the 
program being studied or to a nonparticipating control group before the 
intervention. While other research designs can produce valid results, 
we have previously reported that when it is feasible and ethical to do 
so, randomized controlled designs provide researchers with the best 
method for assessing a program's effectiveness because they isolate 
changes caused by the program from other factors. 

[12] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO/AIMD-00-21.3.1]. 

[13] The other seven experts did not express views regarding the 
balance of the study group's composition. 

[14] Project Management Institute, The Standard for Program Management. 

[15] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO/AIMD-00-21.3.1]. 

[16] OJJDP is required under the Juvenile Justice and Delinquency 
Prevention Act to publish an annual program plan that will, among other 
things, lay out goals and criteria for conducting research and 
evaluation for its juvenile justice programs. 42 U.S.C. § 5614(b)(5). 
This plan is required to be published annually in the Federal Register 
for public comment, and is to describe the activities the Administrator 
intends to carry out under Parts D and E. Under Part D, OJJDP is 
authorized to conduct research, evaluation, and technical assistance, 
among other things. 42 U.S.C. §§ 5661-62. Under Part E, OJJDP is 
authorized to make grants for developing, testing and demonstrating 
promising new initiatives and programs. 42 U.S.C. §§ 5665-66. 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "E-mail Updates." 

Order by Phone: 

The price of each GAO publication reflects GAO’s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black and
white. Pricing and ordering information is posted on GAO’s Web site, 
[hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional 
information. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: fraudnet@gao.gov: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Ralph Dawn, Managing Director, dawnr@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548: 

Public Affairs: 

Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: 

[End of section]