This is the accessible text file for GAO report number GAO-04-333 
entitled 'Child and Family Services Reviews: Better Use of Data and 
Improved Guidance Could Enhance HHS's Oversight of State Performance' 
which was released on April 20, 2004.

This text file was formatted by the U.S. General Accounting Office 
(GAO) to be accessible to users with visual impairments, as part of a 
longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov.

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately.

Report to Congressional Requesters:

United States General Accounting Office:

GAO:

April 2004:

Child and Family Services Reviews:

Better Use of Data and Improved Guidance Could Enhance HHS's Oversight 
of State Performance:

GAO-04-333:

GAO Highlights:

Highlights of GAO-04-333, a report to Congressional Requesters 

Why GAO Did This Study:

In 2001, the Department of Health and Human Services’ (HHS) 
Administration for Children and Families (ACF) implemented the Child 
and Family Services Reviews (CFSR) to increase states’ accountability. 
The CFSR uses states’ data profiles and statewide assessments, as well 
as interviews and an on-site case review, to measure state performance 
on 14 outcomes and systemic factors, including child well-being and 
the provision of caseworker training. The CFSR also requires progress 
on a program improvement plan (PIP); otherwise ACF may apply financial 
penalties. This report examines (1) ACF’s and the states’ experiences 
preparing for and conducting the statewide assessments and on-site 
reviews; (2) ACF’s and the states’ experiences developing, funding, 
and implementing items in PIPs; and (3) any additional efforts that 
ACF has taken beyond the CFSR to help ensure that all states meet 
federal goals related to children’s safety, permanency, and well-
being.

What GAO Found:

ACF and many state officials perceive the CFSR as a valuable process 
and a substantial undertaking, but some data enhancements could 
improve its reliability. ACF staff in 8 of the 10 regions considered 
the CFSR a helpful tool to improve outcomes for children. Further, 26 
of 36 states responding to a relevant question in our survey commented 
that they generally or completely agreed with the results of the final 
CFSR report, even though none of the 41 states with final CFSR reports 
released through 2003 has achieved substantial conformity on all 14 
outcomes and systemic factors—see figure below. Additionally, both ACF 
and the states have dedicated substantial financial and staff resources 
to the process. Nevertheless, several state officials and child 
welfare experts we interviewed questioned the accuracy of the data 
used in the review process. While ACF officials contend that 
stakeholder interviews and case reviews complement the data profiles, 
many state officials and experts reported that additional data from 
the statewide assessment could bolster the evaluation of state 
performance.

State Performance on the 14 CFSR Outcomes and Systemic Factors: 

[See PDF for image]

[End of figure]

Program improvement planning is under way, but uncertainties have 
affected the development, funding, and implementation of state PIPs. 
Officials from 3 of the 5 states we visited said ACF’s PIP-related 
instructions were unclear, and at least 9 of the 25 states reporting 
on PIP implementation in our survey said that insufficient funding and 
staff were among the greatest challenges. While ACF has provided some 
guidance, ACF and state officials remain uncertain about PIP 
monitoring efforts and how ACF will apply financial penalties if states
fail to achieve their stated PIP objectives.

Since 2001, ACF’s focus has been almost exclusively on the CFSRs and 
regional staff report limitations in providing assistance to states in 
helping them to meet key federal goals. While staff from half of ACF’s 
regions told us they would like to provide more targeted assistance to 
states, and state officials in all 5 of the states we visited said that 
ACF’s existing technical assistance efforts could be improved, ACF 
officials acknowledged that regional staff might still be adjusting to 
the new way ACF oversees child welfare programs.

What GAO Recommends:

GAO recommends that the Secretary of HHS ensure that ACF uses the best 
available data to measure state performance, clarify PIP guidance, and 
help regional offices better integrate their oversight activities. In 
commenting on this report, HHS generally agreed with GAO’s findings 
and said that it is committed to continually monitoring and improving 
the CFSR process.

www.gao.gov/cgi-bin/getrpt?GAO-04-333.

To view the full product, including the scope and methodology, click on 
the link above. For more information, contact Cornelia Ashby at (202) 
512-8403 or AshbyC@gao.gov.

[End of figure]

Contents:

Letter:

Results in Brief:

Background:

The CFSR Is a Valuable yet Substantial Undertaking, but Data 
Enhancements Could Improve Its Reliability:

Program Improvement Planning Under Way, but Uncertainties Challenge 
Plan Development, Implementation, and Monitoring:

ACF's Focus Rests Almost Exclusively on Implementing the CFSR:

Conclusions:

Recommendations for Executive Action:

Agency Comments:

Appendix I: Objectives, Scope, and Methodology:

Objectives:

Scope and Methodology:

Appendix II: List of Outcomes and Systemic Factors and Their 
Associated Items:

Outcome Factors:

Systemic Factors:

Appendix III: Dates on Which Site Visit States Reached CFSR Milestones:

Appendix IV: Comments from the Department of Health and Human Services:

Appendix V: GAO Contacts and Acknowledgments:

GAO Contacts:

Staff Acknowledgments:

Related GAO Products:

Tables:

Table 1: Items Most Frequently Assigned a Rating of Area Needing 
Improvement among the 41 States Reviewed in 2001, 2002, and 2003:

Table 2: Number of States Including Each of the PIP Strategy Categories 
Used in This Study:

Table 3: ACF Regulations and Guidance on Program Improvement Planning:

Figures:

Figure 1: Four Phases of the CFSR Process:

Figure 2: State Performance on the 14 CFSR Outcomes and Systemic 
Factors:

Figure 3: Variation in the Start of States' PIP Development:

Figure 4: Average Time Lapse between States' PIP Due Dates and ACF's 
PIP Approval:

Figure 5: Most Common Challenges Affecting States' PIP Implementation:

Abbreviations:

ACF: Administration for Children and Families:

AFCARS: Adoption and Foster Care Analysis and Reporting System:

ASFA: Adoption and Safe Families Act:

CFSR: Child and Family Services Review:

CIP: court improvement program:

FTE: full-time equivalent:

HHS: Department of Health and Human Services:

NCANDS: National Child Abuse and Neglect Data System:

PIP: program improvement plan:

SACWIS: Statewide Automated Child Welfare Information System:

United States General Accounting Office:

Washington, DC 20548:

April 20, 2004:

Congressional Requesters:

Recent media reports have called into question a number of states' 
abilities to protect the more than 800,000 children estimated to spend 
some time in foster care each year. To help states investigate reports 
of child abuse and neglect, provide placements to children outside 
their homes, and deliver services to help keep families together, the 
federal government provides approximately $7 billion dollars to states 
annually. In addition, the federal Department of Health and Human 
Services (HHS) monitors states' compliance with key federal goals, 
specified in part by the Adoption and Safe Families Act (ASFA) of 1997, 
to keep children safe and ensure their placement in stable and 
permanent homes.

In 2001, HHS--through its Administration for Children and Families' 
(ACF) Children's Bureau--began implementing a new federal review 
system, known as the Child and Family Services Reviews (CFSR), to hold 
states accountable for improving child welfare outcomes. Unlike prior 
federal reviews--which determined states' adherence to certain process 
measures--ACF designed the CFSRs as an outcome-oriented approach to 
assess children's safety; their timely placement in permanent homes; 
and their mental, physical, and educational well-being. ACF also 
developed a number of factors to assess states' performance in systemic 
areas, such as staff training and foster parent licensing.[Footnote 1] 
According to ACF, this CFSR process combines statewide assessments, 
which the states complete using a profile of their recent child welfare 
data; on-site reviews of child welfare case files; and interviews with 
stakeholders[Footnote 2] to provide both qualitative and quantitative 
information that helps describe and clarify complex child welfare 
issues. Further, states are required to develop and implement program 
improvement plans (PIP) to improve their child welfare practice and 
capacity when they are found to be deficient. Pursuant to CFSR 
regulations, ACF can withhold federal funds if states do not show 
adequate progress implementing their PIPs.

To better understand the CFSR process, you asked us to examine (1) 
ACF's and the states' experiences preparing for and conducting the 
statewide assessments and on-site reviews; (2) ACF's and the states' 
experiences developing, funding, and implementing items in their PIPs; 
and (3) additional efforts, if any, that ACF has taken beyond the CFSR 
to help ensure that all states meet federal goals of safety, 
permanency, and well-being for children.

To conduct our work, we surveyed all 50 states, the District of 
Columbia, and Puerto Rico regarding their interactions with ACF, the 
resources and staffing required to prepare for the review and develop 
and implement their PIPs, and any assistance ACF provided throughout 
the process. We achieved a 98 percent response rate from this survey, 
as Puerto Rico was the only non-respondent. To supplement this 
information, we also engaged in intensive post-survey follow-up phone 
calls with key states in each phase of the CFSR process. Further, we 
visited 5 states--California, Florida, New York, Oklahoma, and Wyoming-
-to obtain first-hand information on states' experiences. We selected 
these states for diversity in their location, size, program 
administration, performance on the CFSR, and the timing of their 
review. We also examined all 31 approved PIPs available as of January 
1, 2004, to categorize common strategies states developed to improve 
their performance. In addition, we conducted interviews with ACF's 
senior officials and regional staff from all 10 regions, ACF 
contractors, directors and staff from all 10 national resource centers, 
and key child welfare experts.[Footnote 3] We conducted our work 
between May 2003 and February 2004 in accordance with generally 
accepted government auditing standards. A more detailed discussion of 
our scope and methodology appears in appendix I.

Results in Brief:

ACF and many state officials perceive the CFSR as a valuable process--
highlighting many areas needing improvement--and a substantial 
undertaking, but some state officials and child welfare experts told us 
that data enhancements could improve its reliability. ACF staff in 8 of 
the 10 regions considered the CFSR a helpful tool to improve outcomes 
for children, and officials from 8 regions noted that the CFSRs were 
more intensive and comprehensive than other types of reviews they had 
conducted in the past. Further, 26 of the 36 states responding to a 
relevant question in our survey commented that they generally or 
completely agreed with the results of the final CFSR report, even 
though none of the 41 states with final CFSR reports released through 
2003 has achieved substantial conformity on all 14 outcomes and 
systemic factors. In addition, both ACF and the states have dedicated 
substantial financial and staff resources to the process. For example, 
ACF committed $6.6 million to hire contractors for review assistance, 
and based on survey responses from the 45 states that reported on staff 
resources, an average of 47 full-time staff equivalents per state 
participated in the statewide assessment phase. Nevertheless, several 
state officials and child welfare experts we interviewed questioned the 
accuracy of the data used to compile state profiles and establish the 
national standards. While ACF officials in the central office contend 
that stakeholder interviews and case reviews complement the data 
profiles, several state officials and experts reported that additional 
data from the statewide assessment could bolster the evaluation of 
state performance.

The program improvement planning process for the CFSR, which all states 
with final reports have been required to undertake, has been 
characterized by many uncertainties, such as those related to federal 
guidance and monitoring and the availability of state resources to 
implement the plans. Despite such uncertainties, states have included a 
variety of strategies in their PIPs to address weaknesses identified in 
the review. Such strategies include involving foster and birth parents 
in identifying needed services and developing tools to assess the risk 
of harm to children. To assist states with PIP development, ACF issued 
regulations and guidance to states for developing the plans, but state 
responses to this assistance have been mixed. For example, state 
officials attributed a lengthy PIP approval process, in part, to 
unclear regulations and insufficient guidance. Our analysis of the time 
lapse between the date states must submit a PIP and ACF's PIP approval 
date shows a range from 45 to 349 business days. Further, 6 of the 21 
states reporting on the PIP approval process said that ACF did not 
clearly describe the process to them, and officials in 3 of the 5 
states we visited said they were unclear as to ACF's expectations 
regarding the format and content of the plans. In response to a 
question on PIP implementation challenges, at least 9 of 25 states 
reported that insufficient funding, staff, and time, as well as high 
caseloads, were the most challenging factors. For example, one survey 
respondent reported that unless the state receives additional funding, 
it would need to lay off social workers and attorneys, making it 
difficult to implement some of its improvement strategies. ACF 
officials noted that they have clarified PIP expectations and increased 
technical assistance over time, but they acknowledge that they are 
still unsure how best to monitor state PIP implementation and have not 
yet determined when and how to apply the prescribed financial penalties 
for states' failures to improve outcomes.

Since 2001, ACF's focus has been almost exclusively on implementing the 
CFSRs, and regional staff report limitations in providing assistance to 
states in helping them to meet key federal goals. For example, ACF 
provides internal training for its staff and technical assistance to 
states, but all 18 courses provided to regional staff have dealt with 
the various phases of the review. Staff in 2 regional offices told us 
that they have not been able to counsel states on such topics as 
contracting out for child welfare services and maximizing the receipt 
of federal child welfare funds because they have lacked the applicable 
training. In addition, while ACF organizes biennial conferences for 
state and federal child welfare officials, staff from 5 regions told us 
that they wanted more substantive interaction with their ACF colleagues 
to increase their overall child welfare expertise. While regional 
officials conduct site visits to states, CFSR on-site reviews and PIP-
related assistance account for the majority of regions' time and travel 
budgets, and ACF staff from 5 regions said that more frequent visits 
with state personnel--outside of the CFSR process--would allow them to 
better understand states' programs and cultivate relationships with 
state officials. Further, state officials in all 5 of the states we 
visited said that ACF's technical assistance needed improvement, 
acknowledging, in some cases, that regional office staff were stretched 
thin by CFSR demands and, in other cases, that the assistance from 
resource center staff lacked focus or practicality. Although ACF 
officials told us the CFSR has become the agency's primary mechanism 
for monitoring states and facilitating program improvement, they 
acknowledged that regional staff might still be adjusting to the new 
way ACF oversees child welfare programs.

Given the emphasis that ACF has placed on the CFSRs--and the resources 
and time states have invested to prepare for the reviews and address 
their deficiencies through program improvement plans--we are 
recommending that the Secretary of HHS ensure that ACF uses the best 
available data to measure state performance. We are also recommending 
that the Secretary clarify PIP guidance and provide guidance to 
regional officials on how to better integrate their many oversight 
responsibilities. In commenting on a draft of this report, HHS 
generally agreed with our findings and recommendations. HHS 
acknowledged that the CFSR is a new process that continues to evolve, 
and also noted several steps it has taken to address the data quality 
concerns we raise in this report. We believe that our findings fully 
address these initial actions, as well as the substantial resources the 
agency has already dedicated to the review process. However, to improve 
its oversight of state performance, our recommendations are meant to 
encourage HHS to take additional actions to improve its use of data in 
conducting these reviews and enhance PIP-related guidance and regional 
officials' understanding of how to incorporate the CFSR process into 
their overall improvement and oversight efforts.

Background:

ACF's Children's Bureau administers and oversees federal funding to 
states for child welfare services under Titles IV-B and IV-E of the 
Social Security Act, and states and counties provide these child 
welfare services, either directly or indirectly through contracts with 
private agencies.[Footnote 4] Among other activities, ACF staff are 
responsible for developing appropriate policies and procedures for 
states to follow to obtain and use federal child welfare funds and 
conduct administrative reviews of states' case files to ensure that 
children served by the state meet statutory eligibility requirements.

In 2001, ACF launched a new outcome-oriented process, known as the 
Child and Family Services Reviews (CFSR), to improve its existing 
monitoring efforts, which had once been criticized for focusing 
exclusively on states' compliance with regulations rather than on their 
performance over a full range of child welfare services. Passage of the 
1997 Adoption and Safe Families Act (ASFA) helped spur the creation of 
the CFSR by emphasizing the outcomes of safety, permanency, and well-
being for children. Subsequently, ACF consulted with state officials, 
child welfare experts, and other interested parties, and conducted 
pilot CFSR reviews in 14 states. In January 2000, ACF released a notice 
of proposed rule making and published final CFSR regulations. In March 
2001, ACF conducted the first of its state reviews in Delaware. By 
March of 2004, ACF had completed an on-site review in all 50 states, 
the District of Columbia, and Puerto Rico. Although ACF plans to 
initiate a second round of reviews, an official start date for this 
process has not yet been determined.

As figure 1 indicates, the CFSR is a four-phase process that involves 
state staff as well as central and regional ACF officials. This process 
begins with a statewide assessment in the first phase.

Figure 1: Four Phases of the CFSR Process:

[See PDF for image]

[End of figure]

The assessment of state performance continues in the second phase, most 
commonly known as the on-site review, when ACF sends a team of 
reviewers to three sites in the state for one week to assess state 
performance. A list of all the outcomes and systemic factors and their 
associated items appears in appendix II. In assessing performance, ACF 
relies, in part, on its own data systems, known as the National Child 
Abuse and Neglect Data System (NCANDS) and the Adoption and Foster Care 
Analysis and Reporting System (AFCARS), which were designed prior to 
CFSR implementation to capture, report, and analyze the child welfare 
information collected by the states.[Footnote 5] Today, these systems 
provide the national data necessary for ACF to calculate national 
standards for key CFSR performance items with which all states' data 
will be compared.

After the on-site review, ACF prepares a final report for the state--
identifying areas needing improvement, as well as the outcomes and 
systemic factors for which the state was determined not to be in 
substantial conformity--and provides the state with an estimated 
financial penalty.[Footnote 6] As a result, the state must develop a 2-
year PIP with action steps to address its noted deficiencies and 
performance benchmarks to measure progress. Once ACF approves the PIP, 
states are required to submit quarterly progress reports, which ACF 
uses to monitor improvement. Pursuant to CFSR regulations, federal 
child welfare funds can be withheld if states do not show adequate 
progress as a result of PIP implementation, but these penalties are 
suspended during the 2-year implementation term. As of January 2004, no 
financial penalties had been applied, but according to data on the 41 
states for which final CFSR reports have been released through December 
2003, potential penalties range from $91,492 for North Dakota to 
$18,244,430 for California.

ACF staff in HHS's 10 regional offices provide technical assistance to 
states through all phases of the CFSR process, and they are also 
responsible for reviewing state planning documents required by Title 
IV-B, assisting with state data system reviews, and assessing states' 
use of IV-E funds. In addition to these efforts, ACF has established 
cooperative agreements with 10 national resource centers to help states 
implement federal legislation intended to ensure the safety, 
permanency, and well-being of children and families. ACF sets the 
resource centers' areas of focus, and although each center has a 
different area of expertise, such as organizational improvement or 
information technology, all of them conduct needs assessments, sponsor 
national conference calls with states, collaborate with other resource 
centers and agencies, and provide on-site training and technical 
assistance to states.[Footnote 7]

Members of the 108th Congress introduced a proposal to provide federal 
incentive payments directly to states that demonstrate significant 
improvements to their child welfare systems. At the time of 
publication, the House of Representatives was considering H.R. 1534, 
the Child Protective Services Improvement Act, which contains 
provisions to award grants to states with approved PIPs and additional 
bonuses to states that have made considerable progress in achieving 
their PIP goals for the previous year.

The CFSR Is a Valuable yet Substantial Undertaking, but Data 
Enhancements Could Improve Its Reliability:

ACF and many state officials perceive the CFSR as a valuable process--
highlighting many areas needing improvement--and a substantial 
undertaking, but some state officials and child welfare experts told us 
that data enhancements could improve its reliability. ACF staff in 8 of 
the 10 regions considered the CFSR a helpful tool to improve outcomes 
for children. Further, 26 of the 36 states responding to a relevant 
question in our survey commented that they generally or completely 
agreed with the results of the final CFSR report, even though none of 
the 41 states with final CFSR reports released through 2003 has 
achieved substantial conformity on all 14 outcomes and systemic 
factors. In addition, both ACF and the states have dedicated 
substantial financial and staff resources to the process. However, 
several state officials and child welfare experts we interviewed 
questioned the accuracy of the data used to compile state profiles and 
establish the national standards. While ACF officials in the central 
office contend that stakeholder interviews and case reviews compliment 
the data profiles, many state officials and experts reported that 
additional data from the statewide assessment could bolster the 
evaluation of state performance.

The CFSR Is a Valuable Process for ACF and the States:

ACF and state officials support the objectives of the review, 
especially in focusing on children's outcomes and strengthening 
relationships with stakeholders, and told us they perceive the process 
as valuable. ACF staff in 8 of the 10 regions considered the CFSR a 
helpful tool to improve outcomes for children. Also, ACF officials from 
8 regional offices noted that the CFSRs were more intensive and more 
comprehensive than the other types of reviews they had conducted in the 
past, creating a valuable tool for regional officials to monitor 
states' performance. In addition, state officials from every state we 
visited told us that the CFSR process helped to improve collaboration 
with community stakeholders. For example, a court official in New York 
said that the CFSR acted as a catalyst for improved relations between 
the state agency and the courts, and he believes that this has 
contributed to more timely child abuse and neglect hearings. 
Additionally, a state official in Florida said that the CFSR stimulated 
discussions among agency staff about measuring and improving outcomes, 
particularly the data needed to examine outcomes and the resources 
needed to improve them. Furthermore, state staff from 4 of the 5 states 
we visited told us the CFSR led to increased public and legislative 
attention to critical issues in child welfare. For example, caseworkers 
in Wyoming told us that without the CFSR they doubted whether their 
state agency's administration would have focused on needed reforms. 
They added that the agency used the CFSR findings to request 
legislative support for the hiring of additional caseworkers.

In addition to affirming the value associated with improved stakeholder 
relations, the ACF officials we talked to and many state officials 
reported that the process has been helpful in highlighting the outcomes 
and systemic factors, as well as other key performance items that need 
improvement. According to our survey, 26 of the 36 states that 
commented on the findings of the final CFSR report indicated that they 
generally or completely agreed with the findings, even though 
performance across the states was low in certain key outcomes and 
performance items. For example, not one of the 41 states with final 
reports released through 2003 was found to be in substantial conformity 
with either the outcome measure that assesses the permanency and 
stability of children's living situations or with the outcome measure 
that assesses whether states had enhanced families' capacity to provide 
for their children's needs. Moreover, across all 14 outcomes and 
systemic factors, state performance ranged from achieving substantial 
conformity on as few as 2 outcomes and systemic factors to as many as 
9.[Footnote 8] As figure 2 illustrates, the majority of states were 
determined to be in substantial conformity with half or fewer of the 14 
outcomes and systemic factors assessed.

Figure 2: State Performance on the 14 CFSR Outcomes and Systemic 
Factors:

[See PDF for image]

[End of figure]

States' performance on the outcomes related to safety, permanency, and 
well-being--as well as the systemic factors--is determined by their 
performance on an array of items, such as establishing permanency 
goals, ensuring worker visits with parents and children, and providing 
accessible services to families. The CFSR showed that many states need 
improvement in the same areas, and table 1 illustrates the 10 items 
most frequently rated as needing improvement across all 41 states 
reviewed through 2003.

Table 1: Items Most Frequently Assigned a Rating of Area Needing 
Improvement among the 41 States Reviewed in 2001, 2002, and 2003:

Item: Assessing the needs and services of child, parents, and foster 
parents; 
Number of states assigned a rating of area needing improvement: 40.

Item: Assessing mental health of child; 
Number of states assigned a rating of area needing improvement: 37.

Item: Establishing the most appropriate permanency goal for the child; 
Number of states assigned a rating of area needing improvement: 36.

Item: Demonstrating efforts to involve child and family in case-
planning activities; 
Number of states assigned a rating of area needing improvement: 36.

Item: Ensuring stability of foster care placements; 
Number of states assigned a rating of area needing improvement: 35.

Item: Achieving a child's goal of adoption; 
Number of states assigned a rating of area needing improvement: 35.

Item: Providing a process that ensures that each child has a written 
case plan to be developed jointly with the child's parent(s) that 
includes the required provisions; 
Number of states assigned a rating of area needing improvement: 35.

Item: Ensuring workers conduct face-to-face visits with parent(s); 
Number of states assigned a rating of area needing improvement: 34.

Item: Providing services that are accessible to families and children 
in all political jurisdictions covered in the state's Child and Family 
Services Plan; 
Number of states assigned a rating of area needing improvement: 31.

Item: Ensuring workers conduct face-to-face visits with child; 
Number of states assigned a rating of area needing improvement: 31. 

Source: GAO analysis of the 41 states' final CFSR reports released 
through December 31, 2003.

[End of table]

ACF and the States Report That Reviews Have Been a Substantial 
Undertaking:

Given the value that ACF and the states have assigned to the CFSR 
process, both have spent substantial financial resources and staff time 
to prepare for and implement the reviews. In fiscal years 2001-2003, 
when most reviews were scheduled, ACF budgeted an additional $300,000 
annually for CFSR-related travel. In fiscal year 2004, when fewer 
reviews were scheduled, ACF budgeted about $225,000. To further enhance 
its capacity to conduct the reviews, and to obtain additional 
logistical and technical assistance, ACF spent approximately $6.6 
million annually to hire contractors. Specifically, ACF has let three 
contracts to assist with CFSR-related activities, including training 
reviewers to conduct the on-site reviews, tracking final reports and 
PIP documents, and, as of 2002, writing the CFSR final reports. 
Additionally, ACF hired 22 new staff to build central and regional 
office capacity and dedicated 4 full-time staff and 2 state government 
staff temporarily on assignment with ACF to assist with the CFSR 
process. To build a core group of staff with CFSR expertise, ACF 
created the National Review Team, composed of central and regional 
office staff with additional training in and experience with the review 
process. In addition, to provide more technical assistance to the 
states, ACF reordered the priorities of the national resource centers 
to focus their efforts primarily on helping states with the review 
process.

Like ACF, states also spent financial resources on the review. While 
some states did not track CFSR expenses--such as staff salaries, 
training, or administrative costs--of the 25 states that reported such 
information in our survey, the median expense to date was $60,550, 
although states reported spending as little as $1,092 and as much as 
$1,000,000 on the CFSR process.[Footnote 9] For example, California 
officials we visited told us that they gave each of the state's three 
review sites $50,000 to cover the salary of one coordinator to manage 
the logistics of the on-site review. Although ACF officials told us 
that states can use Title IV-E funds to pay for some of their CFSR 
expenses, only one state official addressed the use of these funds in 
our survey, commenting that it was not until after the on-site review 
occurred that the state learned these funds could have been used to 
offset states' expenses.[Footnote 10] Furthermore, 18 of 48 states 
responding to a relevant question in our survey commented that 
insufficient funding was a challenge--to a great or very great extent-
-in preparing for the statewide assessment, and 11 of 40 states 
responded that insufficient funding was similarly challenging in 
preparing for the on-site review. Officials in other states reported 
that because available financial resources were insufficient, they 
obtained non-financial, in-kind donations to cover expenses associated 
with the on-site review. For example, a local site coordinator in 
Oklahoma informed us that while the state budgeted $7 per reviewer each 
day of the on-site review for food, refreshments, and other expenses, 
she needed to utilize a variety of resources to supplement the state's 
budget with donations of supplies and food from local companies and 
agency staff.

States reported that they also dedicated staff time to prepare for the 
statewide assessment and to conduct the on-site review, which sometimes 
had a negative impact on some staffs' regular duties. According to our 
survey, 45 states reported dedicating up to 200 full-time staff 
equivalents (FTEs), with an average of 47 FTEs, to the statewide 
assessment process.[Footnote 11] Similarly, 42 states responded that 
they dedicated between 3 and 130 FTEs, with an average of 45 FTEs, to 
the on-site review process. To prepare for their own reviews, officials 
in all 5 states we visited told us that they sent staff for a week to 
participate as reviewers in other states. Additionally, local site 
coordinators in 4 of the 5 states we visited reported that planning for 
the CFSR on-site review was a full-time job involving multiple staff 
over a period of months. An official in Florida also told us that the 
extensive preparation for the CFSR dominated the work of the quality 
assurance unit, which was about 20 people at that time. In addition to 
preparing for the review, staff in all 5 of the states we visited, who 
served as reviewers in their own states, reported that to meet their 
responsibility as case reviewers they worked more than 12-hour days 
during the week of the on-site review. For some caseworkers, dedicating 
time to the CFSR meant that they were unable or limited in their 
ability to manage their typical workload. For example, Wyoming 
caseworkers whose case files were selected for the on-site review told 
us that they needed to be available to answer reviewers' questions all 
day every day during the on-site review, which they said prevented them 
from conducting necessary child abuse investigations or home visits.

Child welfare-related stakeholders--such as judges, lawyers, and foster 
parents--also contributed time to the CFSR, but some states found it 
took additional staff resources to initiate and maintain stakeholder 
involvement over time. According to our survey, 46 states reported that 
an average of about 277 stakeholders were involved with the statewide 
assessment, and 42 states responded that an average of about 126 
stakeholders participated in the on-site review.[Footnote 12] However, 
state officials told us that it was difficult to recruit and maintain 
stakeholder involvement because long time lapses sometimes occurred 
between CFSR phases. For example, more than a year can elapse between 
completion of the statewide assessment and the initiation of PIP 
development--both key points at which stakeholder participation is 
critical. Nonetheless, all 5 of the states we visited tried to counter 
this obstacle by conducting phone and in-person interviews, holding 
focus groups, or giving presentations in local communities to inform 
and recruit stakeholders to be involved in the CFSR process. As a 
result, some stakeholders were involved throughout the process. For 
example, a tribal representative in Oklahoma assisted with the 
development of the statewide assessment, was interviewed during the on-
site review, and provided guidance on the needs of Indian children 
during PIP development.

States and Child Welfare Experts Report That Several Data Improvements 
Could Enhance CFSR Reliability:

State officials in all 5 states, as well as child welfare experts, 
reported on several data improvements that could enhance the 
reliability of CFSR findings. In particular, they highlighted 
inaccuracies with the AFCARS and NCANDS data that are used for 
establishing the national standards and creating the statewide data 
profiles, which are then used to determine if states are in substantial 
conformity. These concerns echoed the findings of a prior GAO study on 
the reliability of these data sources, which found that states are 
concerned that the national standards used in the CFSR are based on 
unreliable information and should not be used as a basis for comparison 
and potential financial penalty.[Footnote 13] Several of the state 
officials we visited and surveyed also questioned the reliability of 
data given the variation in states' data-reporting practice, which they 
believe may ultimately affect the validity of the measures and may 
place some states at a disadvantage. Furthermore, many states needed to 
resubmit their statewide data after finding errors in the data profiles 
ACF would have used to measure compliance with the national 
standards.[Footnote 14] According to our national survey, of the 37 
states that reported on resubmitting data for the statewide data 
profile, 23 needed to resubmit their statewide data at least once, with 
1 state needing to resubmit as many as five times to accurately reflect 
revised data. Four states reported in our survey that they did not 
resubmit their data profiles because they did not know they had this 
option or they did not have enough time to resubmit before the review.

In addition to expressing these data concerns, child welfare experts as 
well as officials in all of the states we visited commented that 
existing practices that benefit children might conflict with actions 
needed to attain the national standards. Specifically, one child 
welfare expert noted that if a state focused on preventing child abuse 
and neglect while the child was still living at home, and only removed 
the most at-risk children, the state may perform poorly on the CFSR 
reunification measure--assessing the timeliness of children's return to 
their homes from foster care arrangements--because the children they 
did remove would be the hardest to place or most difficult to reunite. 
Additionally, officials in New York said that they recently implemented 
an initiative to facilitate adoptions. Because these efforts focus on 
the backlog of children who have been in foster care for several years, 
New York officials predict that their performance on the national 
standard for adoption will be lower since many of the children in the 
initiatives have already been in foster care for more than 2 years.

These same state officials and experts also commented that they believe 
the on-site review case sample of 50 cases is too small to provide an 
accurate picture of statewide performance, although ACF officials 
stated that the case sampling is supplemented with additional 
information. Of the 40 states that commented in our survey on the 
adequacy of the case sample size for the on-site review, 17 states 
reported that 50 cases were very or generally inadequate to represent 
their caseload.[Footnote 15] Oklahoma officials we visited also 
commented that they felt the case sample size was too small, especially 
since they annually assess more than 800 of their own cases--using a 
procedure that models the federal CFSR--and obtain higher performance 
results than the state received on its CFSR. Furthermore, because not 
every case in the states' sample is applicable to each item measured in 
the on-site review, we found that sometimes as few as one or two cases 
were being used to evaluate states' performance on an item. 
Specifically, an ACF contractor said that several of the CFSR items 
measuring children's permanency and stability in their living 
conditions commonly have very few applicable cases. For example, 
Wyoming had only two on-site review cases applicable for the item 
measuring the length of time to achieve a permanency goal of adoption, 
but for one of these cases, reviewers determined that appropriate and 
timely efforts had not been taken to achieve finalized adoptions within 
24 months, resulting in the item being assigned a rating of area 
needing improvement.[Footnote 16] While ACF officials acknowledged the 
insufficiency of the sample size,[Footnote 17] they contend that the 
case sampling is augmented by stakeholder interviews for all items and 
applicable statewide data for the five CFSR items with corresponding 
national standards, therefore providing sufficient evidence for 
determining states' conformity.

All of the states we visited experienced discrepant findings between 
the aggregate data from the statewide assessment and the information 
obtained from the on-site review, which complicated ACF's determination 
of states' performance. Each of the 5 states we visited had at least 
one, and sometimes as many as three, discrepancies between its 
performance on the national standards and the results of the case 
review. We also found that in these 5 states, ACF had assigned an 
overall rating of area needing improvement for 10 of the 11 instances 
where discrepancies occurred. ACF officials acknowledged the challenge 
of resolving data discrepancies, noting that such complications can 
delay the release of the final report and increase or decrease the 
number of items that states must address in their PIPs. While states 
have the opportunity to resolve discrepancies by submitting additional 
information explaining the discrepancy or by requesting an additional 
case review, only one state to date has decided to pursue the 
additional case review.[Footnote 18] In addition, among the states we 
visited, for example, one state acknowledged that it has not opted to 
pursue the supplemental case review because doing so would place 
additional strain on its already limited resources.

Several state officials and experts told us that additional data from 
the statewide assessments--or other data sources compiled by the 
states--could bolster the evaluation of states' performance, but they 
found this information to be missing or insufficiently used in the 
final reports. According to our survey, of the 34 states that commented 
on the adequacy of the final report's inclusion of the statewide 
assessment, 10 states reported that too little emphasis was placed on 
the statewide assessment.[Footnote 19] Specifically, 1 state reported 
that the final report would have presented a more balanced picture of 
the state's child welfare system if the statewide assessment were used 
as a basis to compare and clarify the on-site review findings. Further, 
child welfare experts and state officials from California and New York-
-who are using alternative data sources to AFCARS and NCANDS, such as 
longitudinal data that track children's placements over time--told us 
that the inclusion of this more detailed information would provide a 
more accurate picture of states' performance nationwide. North Carolina 
officials also reported in our survey that they tried to submit 
additional longitudinal data--which they use internally to conduct 
statewide evaluations of performance--in their statewide assessment, 
but HHS would not accept the alternative data for use in evaluating the 
state's outcomes. An HHS official told us that alternative data are 
used only to assess state performance in situations where a state does 
not have NCANDS data, since states are not mandated to have these 
systems.

Given their concerns with the data used in the review process, state 
officials in 4 of the 5 states believed that the threshold for 
achieving substantial conformity was difficult to achieve. One state 
official we visited acknowledged that she believed child welfare 
agencies should be pursuing high standards to improve performance, but 
she questioned the level at which ACF established the thresholds. While 
an ACF official told us that different thresholds for the national 
standards had been considered, ACF policy makers ultimately concluded 
that a more rigorous threshold would be used. ACF officials recognize 
that they have set a high standard. However, they believe it is 
attainable and supportive of their overall approach to move states to 
the standard through continuous improvement.

In preparation for the next round of CFSRs, ACF officials have formed a 
Consultation Work Group of ACF staff, child welfare administrators, 
data experts, and researchers who will propose recommendations on the 
CFSR measures and processes. The group began its meetings prior to our 
publication, but no proposals were available.

Program Improvement Planning Under Way, but Uncertainties Challenge 
Plan Development, Implementation, and Monitoring:

Forty-one states are engaged in program improvement planning, but many 
uncertainties, such as those related to federal guidance and monitoring 
and the availability of state resources, have affected the development, 
implementation, and funding of the PIPs. State PIPs include strategies 
such as revising or developing policies, training caseworkers, and 
engaging stakeholders, and ACF has issued regulations and guidance to 
help states develop and implement their plans. Nevertheless, states 
reported uncertainty about how to develop their PIPs and commented on 
the challenges they faced during implementation. For example, officials 
from 2 of the states we visited told us that ACF had rejected their 
PIPs before final approval, even though these officials said that the 
plans were similar in the level of detail included in previously 
approved PIPs that regional officials had provided. Further, at least 9 
states responding to our survey indicated that insufficient time, 
funding, and staff, as well as high caseloads, were the greatest 
challenges to PIP implementation. As states progress in PIP 
implementation, some ACF officials expressed a need for more guidance 
on how to monitor state accomplishments, and both ACF and state 
officials were uncertain about how the estimated financial penalties 
would be applied if states fail to achieve the goals described in their 
plans.

State Plans Include a Variety of Strategies to Address Identified 
Weaknesses:

State plans include a variety of strategies to address weaknesses 
identified in the CFSR review process. However, because most states had 
not completed PIP implementation by the time of our analysis, the 
extent to which states have improved outcomes for children has not been 
determined.[Footnote 20] While state PIPs varied in their detail, 
design, and scope, according to our analysis of 31 available PIPs, 
these state plans have focused to some extent on revising or developing 
policies; reviewing and reporting on agency performance; improving 
information systems; and engaging stakeholders such as courts, 
advocates, foster parents, private providers, or sister agencies in the 
public sector.[Footnote 21] Table 2 shows the number of states that 
included each of the six categories and subcategories of strategies we 
developed for the purposes of this study, and appendix I details our 
methodology for this analysis.

Table 2: Number of States Including Each of the PIP Strategy Categories 
Used in This Study:

PIP strategy category: Policies and procedures; 
Description (number of states that included the strategy in their 
PIP): Review, modify, or develop/implement any policy, procedure, or 
case practice standard (31); 
Enhance foster home/parent licensing standards (7); 
Develop child and family assessment tools, such as protocols for risk/
safety determinations (28); 
Identify and adopt any promising practices (19).

PIP strategy category: Data collection and analysis; 
Description (number of states that included the strategy in their 
PIP): Review and report on agency performance through self-assessments 
or internal audits/review (31); 
Apply federal CFSR or similar process for internal statewide case 
reviews (16); 
Improve information and data collection systems (31).

PIP strategy category: Staff supports; 
Description (number of states that included the strategy in their 
PIP): Train and develop caseworkers (through dissemination and 
training on policy or through revisions to overall curriculum) (30); 
Assess and monitor staff responsibilities, skills, or performance 
(24); 
Recruit additional staff/retain staff (14); 
Lower caseloads (11); 
Increase caseworker pay (1).

PIP strategy category: Foster parent supports/services and resources 
for children and families; 
Description (number of states that included the strategy in their PIP): 
Train and develop foster families'/ providers' skills and capacities 
(27); 
Recruit and retain foster families (22); 
Increase involvement of foster or birth families in case (18); 
Expand service array for children and families (includes developing or 
enhancing transportation systems to transport siblings and parents for 
visits, creating one-stop centers for assistance, modifying visitation 
services, and providing any additional support services) (27); 
Engage stakeholders such as courts, advocates, foster homes, private 
providers, or sister agencies in public sector, e.g., mental health 
(can include consultation, training, or formal partnering to improve 
services or placements) (31); 
Create or improve monitoring of contracts with private providers to 
enhance service delivery (includes development of performance based or 
outcome-based contracts or other evaluations of provider performance) 
(25).

PIP strategy category: State legislative supports; 
Description (number of states that included the strategy in their 
PIP): State request for legislative action to support any of the above 
strategies (20).

PIP strategy category: Federal technical assistance; 
Description (number of states that included the strategy in their 
PIP): State request technical assistance from ACF or any resource 
center to support any of the above strategies (27). 

Source: GAO analysis.

[End of table]

Our analysis found that every state's PIP has included policy revisions 
or creation to improve programs and services. For example, to address 
unsatisfactory performance in the prevention of repeat maltreatment, 
California's PIP includes a policy change, pending legislative 
approval, granting more flexibility to counties in determining the 
length of time they spend with families to ensure child safety and 
improve family functioning before closing cases. Additionally, 30 of 
the plans included caseworker training; 20 of the plans included 
requests for state legislative action; and 27 of the plans included 
requests for federal technical assistance from ACF or the resource 
centers. For example, California planned to ask its legislature to 
allow the agency to consolidate standards for foster care and adoption 
home studies, believing this would facilitate the adoption of children 
in the state. In addition, New York worked with its legislature to 
secure additional funding to improve the accessibility of independent 
living and adoption services for children and families.

Our analysis also showed that many states approached PIP development by 
building on state initiatives in place prior to the on-site review. Of 
the 42 surveyed states reporting in our survey on this topic, 30 said 
that their state identified strategies for the PIP by examining ongoing 
state initiatives. For example, local officials in New York City and 
state officials in California told us that state reform efforts--borne 
in part from legal settlements--have become the foundation for the PIP. 
In New York, the Marisol case was one factor in prompting changes to 
the state's child welfare financing structure.[Footnote 22] 
Subsequently, the state legislature established a quality enhancement 
fund. Much of this money today supports strategies in New York's PIP, 
such as permanency mediation to support family involvement in case 
planning and new tools to better assess children's behavioral and 
mental health needs. California state officials also informed us that 
state reform efforts initiated by the governor prior to the CFSR, such 
as implementing a new system for receiving and investigating reports of 
abuse and neglect and developing more early intervention programs, 
became integral elements in the PIP.

Insufficient Guidance Hampered State Planning Efforts, but ACF Has 
Taken Steps to Clarify Expectations and Improve Technical Assistance:

ACF has provided states with regulations and guidance to facilitate PIP 
development, but some states believe the requirements have been 
unclear. Some of the requirements for program improvement planning are 
outlined in the following table.

Table 3: ACF Regulations and Guidance on Program Improvement Planning:

Stage: Development of the PIP; 
Requirements: States are required to develop a PIP to address all of 
the outcomes and systemic factors determined not to be in substantial 
conformity as a result of a CFSR. Although ACF does not require a 
specific format for the PIP, it does offer states a suggested format, 
called the PIP matrix, to use; 
Time frame: States must submit the PIP to ACF for approval within 90 
calendar days from the date that the state received written 
notification that it is not operating in substantial conformity.

Stage: Approval of the PIP; 
Requirements: States must include certain information in the PIP, 
including measurable goals, action steps required to correct each 
deficiency identified in the final report, anticipated dates of 
completion for each action step, amount of progress the state will make 
toward national standards, benchmarks to measure state progress 
implementing the PIP, and technical assistance needs; 
Time frame: No time frame is specified for ACF review and approval of 
the PIP. However, ACF guidance says that the regional office should 
give prompt attention to the PIP when it is submitted and take the 
minimum amount of time necessary to review it and respond to the 
state. If ACF notifies the state that the PIP is unacceptable, the 
state has 30 calendar days to resubmit the plan for approval.

Stage: Implementation and monitoring of the PIP; 
Requirements: ACF will monitor the state's progress in completing the 
provisions of the approved PIP by evaluating quarterly status reports 
and reviewing the state's Annual Progress and Services Report, which 
incorporates the state's progress in implementing the PIP; 
Time frame: States must submit status reports to ACF on a quarterly 
basis, and the PIP must be implemented within 2 years. 

Source: ACF regulations at 45 C.F.R. 1355.35 and guidance.

[End of table]

Some states in our survey indicated that the guidance ACF had provided 
did not clearly describe the steps required for PIP approval. In 
addition, some state officials believe that even ACF's more recent 
efforts to improve PIP guidance have also been insufficient. Of the 21 
states reporting on the PIP approval process in our survey, 6 states--
4 reviewed in 2001 and 2 reviewed in 2002--said that ACF did not 
clearly describe its approval process, and another 8 states reported 
that ACF described the approval process as clearly as it did unclearly. 
Further, several states commented in our survey that several steps in 
the approval process were unclear to them, such as how much detail and 
specificity the agency expects the plan to include, what type of 
feedback states could expect to receive, when states could expect to 
receive such feedback, and whether a specific format was required. 
Officials in the states we visited echoed survey respondents' concerns 
with officials from 3 of the 5 states informing us that ACF had given 
states different instructions regarding acceptable PIP format and 
content. For example, California and Florida officials told us that 
their program improvement plans had been rejected prior to final 
approval, even though they were based on examples of approved plans 
that regional officials had provided. In addition, California officials 
told us that they did not originally know how much detail the regional 
office expected in the PIP and believed that the level of detail the 
regional office staff ultimately required was too high. Although some 
steps may be duplicative, officials in California said that the version 
of their plan that the region accepted included 2,932 action steps--a 
number these officials believe is too high given their state's limited 
resources and the 2-year time frame to implement the PIP.

ACF officials have undertaken several steps to clarify their 
expectations for states and to improve technical assistance, but state 
responses to this assistance have been mixed. For example, in 2002, 2 
years after ACF released the CFSR regulations and a procedures manual, 
ACF offered states additional guidance and provided a matrix format to 
help state officials prepare their plans. ACF officials told us the 
agency is also helping states through a team approach to providing on-
site technical assistance. Under this approach, when ACF determines a 
state is slow in developing its PIP, the agency sends a team of staff 
from ACF and resource centers to the state to provide intensive on-site 
technical assistance. An official from West Virginia who had received 
this team assistance reported that ACF's support was very beneficial. 
Further, ACF has attempted to encourage state officials to start 
developing program improvement plans before the final report is 
released. To do so, the agency has provided training to state officials 
and stakeholders almost immediately after the completion of the on-site 
review. ACF has sent staff from the resource center for Organizational 
Improvement to provide such training. An official from Utah, however, 
reported that the resource center training on PIP development had been 
general, and she wished the resource center staff had better tailored 
its assistance and provided more examples of strategies other states 
are pursuing to improve.

Analysis of state survey responses indicates that starting to develop 
improvement plans early can make the 90-day time frame to prepare a PIP 
seem adequate. Of 9 states reporting that they started developing their 
PIP before or during the statewide assessment phase, 5 said that 90 
days was adequate. Nonetheless, 21 of 35 state survey respondents 
reported that the 90-day time frame was insufficient. For example, one 
respondent reported that 90 days is too short a time to perform tasks 
necessary for developing a program improvement plan, such as analyzing 
performance data, involving diverse groups of stakeholders in planning 
decisions, and obtaining the approval of state officials. Survey 
results indicate that increasing numbers of states are developing their 
PIPs early in the CFSR process, which may reflect ACF's emphasis on PIP 
development. The following figure shows that of the 18 states reviewed 
in 2001, only 2 started developing their PIPs before or during the 
statewide assessment phase. Among states reviewed in 2003, this share 
increased to 5 of 9.

Figure 3: Variation in the Start of States' PIP Development:

[See PDF for image]

[End of figure]

Evidence suggests that lengthy time frames for PIP approval have not 
necessarily delayed PIP implementation, and ACF has made efforts to 
reduce the time the agency takes to approve states' PIPs. For example, 
officials in 3 of the 5 states we visited told us they began 
implementing new action steps before ACF officially approved their 
plans because many of the actions in their PIPs were already under way. 
In addition, according to our survey, of the 28 states reporting on 
this topic, 24 reported that they had started implementing their PIP 
before ACF approved it. Further, our analysis shows that the length of 
time between the PIP due date, which statute sets at 90 days after the 
release of the final CFSR report, and final ACF PIP approval has ranged 
considerably--from 45 to 349 business days. For almost half of the 
plans, ACF's approval occurred 91 to 179 business days after the PIP 
was due. As shown in figure 4, however, our analysis indicates that ACF 
has recently reduced the time lapse by 46 business days between states' 
PIP due dates and ACF's PIP approval.

Figure 4: Average Time Lapse between States' PIP Due Dates and ACF's 
PIP Approval:

[See PDF for image]

Notes: Data for states reviewed in 2003 are not included because only 2 
states' PIPs had been approved by March 1, 2004.

We used the date that states are required to submit a PIP to calculate 
the time lapse. Therefore, the time lapse is calculated from day 90 
through the date ACF ultimately approved the plans. States may have 
submitted their plans prior to or after the 90-day time frame 
designated in regulations, but ACF's contractor--charged with tracking 
CFSR-related milestones--does not track the actual PIP submission 
dates.

[End of figure]

The shorter time lapse for PIP approval may be due, in part, to the 
ACF's emphasis on PIP development. According to one official, ACF has 
directed states to concentrate on submitting a plan that can be quickly 
approved. Another ACF official added that because of ACF's assistance 
with PIP development, states are now submitting higher-quality PIPs 
that require fewer revisions.

State and Federal Uncertainties Cloud PIP Implementation and 
Monitoring:

Program improvement planning has been ongoing, but uncertainties have 
made it difficult for states to implement their plans and ACF to 
monitor state performance. Such uncertainties include not knowing 
whether state resources are adequate to implement the plans and how 
best to monitor state reforms. In answering a survey question about PIP 
implementation challenges, a number of states identified insufficient 
funding, staff, and time--as well as high caseloads--as their greatest 
obstacles. Figure 5 depicts these results.

Figure 5: Most Common Challenges Affecting States' PIP Implementation:

[See PDF for image]

Note: This is based on responses from 25 states. The results reported 
in the figure are a sum of the states reporting that the issue was a 
challenge to PIP implementation to a very great extent, great extent, 
moderate extent, or some/little extent. States not included answered no 
extent, no basis to judge, or not applicable.

[End of figure]

In regards to funding, an official from Pennsylvania commented that 
because of the state's budget shortfall, no additional funds were 
available for the state to implement its improvement plan, so most 
counties must improve outcomes with little or no additional resources. 
A Massachusetts official reported that fiscal problems in his state 
likely would lead the state to lay off attorneys and caseworkers and to 
cut funding for family support programs. While state officials 
acknowledged that they do not have specific estimates of PIP 
implementation expenses because they have not tracked this information 
in their state financial systems, many states indicated that to cope 
with financial difficulties, they had to be creative and use resources 
more efficiently to fund PIP strategies. Of the 26 states responding to 
a question in our survey on PIP financing, 12 said that they were 
financing the PIP strategies by redistributing current funding, and 7 
said that they were using no-cost methods. In an example of the latter, 
Oklahoma officials reported pursuing in-kind donations from a greeting 
card company so that they could send thank-you notes to foster parents, 
believing this could increase foster parent retention and engagement.

States also reported that PIP implementation has been affected by staff 
workloads, but these comments were mixed. In Wyoming, for example, 
caseworkers told us that their high caseloads would prevent them from 
implementing many of the positive action steps included in their 
improvement plan. In contrast, Oklahoma caseworkers told us that the 
improvement plan priorities in their state--such as finding permanent 
homes for children--have helped them become more motivated, more 
organized, and more effective with time management. For example, one 
caseworker explained that she is quicker now at locating birth fathers 
who were previously uninvolved in the child's life because she uses the 
Internet to search for these fathers' names. She said this new way of 
exploring leads and information--a strategy that stemmed from PIP 
development--has been motivating and rewarding because it has decreased 
the time spent tracking down paternal relatives and increased the 
number of available placements for the child.

ACF officials expressed uncertainty about how best to monitor states' 
progress and apply estimated financial penalties when progress was slow 
or absent, and 3 of the 5 states we visited reported frustration with 
the limited guidance ACF had provided on the PIPs quarterly reporting 
process. For example, 4 regional offices told us that they did not have 
enough guidance on or experience with evaluating state quarterly 
reports. Some regional offices told us they require states to submit 
evidence of each PIP action step's completion, such as training 
curricula or revised policies, but one ACF official acknowledged that 
this is not yet standard procedure, although the agency is considering 
efforts to make the quarterly report submission procedures more 
uniform. Moreover, ACF staff from one region told us that because PIP 
monitoring varies by region, they were concerned about enforcing 
penalties. Finally, shortly before California's quarterly report was 
due, state officials told us they still did not know how much detail to 
provide, how to demonstrate whether they had completed certain 
activities, or what would happen if they did not reach the level of 
improvement specified in the plan.

Based on data from the states that have been reviewed to date, the 
estimated financial penalties range from a total of $91,492 for North 
Dakota to $18,244,430 for California, but the impact of these potential 
penalties remains unclear. While ACF staff from most regional offices 
told us that potential financial penalties are not the driving force 
behind state reform efforts, some contend that the estimated penalties 
affect how aggressively states pursue reform in their PIPs. For 
example, regional office staff noted that one state's separate 
strategic plan included more aggressive action steps than those in its 
PIP because the state did not want to be liable for penalties if it did 
not meet its benchmarks for improvement. State officials also had mixed 
responses as to how the financial penalties would affect PIP 
implementation. An official in Wyoming said that incurring the 
penalties was equivalent to shutting down social service operations in 
one local office for a month, while other officials in the same state 
thought it would cost more to implement PIP strategies than it would to 
incur financial penalties if benchmarks were unmet. Nevertheless, these 
officials also said that while penalties are a consideration, they have 
used the CFSR as an opportunity to provide better services. One 
official in another state agreed that it would cost more to implement 
the PIP than to face financial penalties, but this official was 
emphatic in the state's commitment to program improvement.

ACF's Focus Rests Almost Exclusively on Implementing the CFSR:

To implement the CFSRs, ACF has focused its activities almost entirely 
on the four phases of the review process. However, staff in several 
regions report limitations in providing assistance to states in helping 
them to meet key federal goals. Although regional staff conduct site 
visits to states for reasons beyond the CFSR process, conducting the 
CFSR on-site reviews and providing PIP-related assistance to states 
account for the majority of regions' time and travel budgets, according 
to ACF officials. Further, regional office staff said that more 
frequent visits with state personnel--visits outside of the CFSR 
process in particular--would allow them to better understand states' 
programs and cultivate relationships with state officials. In addition, 
state officials in all five of the states we visited said that ACF 
technical assistance needed improvement, acknowledging that in some 
cases regional office staff were stretched thin by CFSR demands and in 
other cases that assistance from resource center staff lacked focus. 
While ACF officials in the central office said that the CFSR has become 
the primary method for evaluating states' performance, they 
acknowledged that regional staff might still be adjusting to the new 
way ACF oversees child welfare programs. Further, they told us that ACF 
is currently reevaluating the entire structure of its training and 
technical assistance, in part to address these concerns.

ACF officials told us that the learning opportunities in the Children's 
Bureau are intentionally targeted at the CFSR, but staff in 3 regions 
told us that this training should cover a wider range of subjects--
including topics outside of the CFSR process--so that regional 
officials could better meet states' needs. All 18 of the courses that 
ACF has provided to its staff since 2001 have focused on topics such as 
writing final CFSR reports and using data for program improvement. ACF 
officials in the central office said that the course selection reflects 
both the agency's prioritization of the CFSR process and staff needs. 
To ascertain staff training needs, ACF surveyed regional staff in 
October 2002, and ACF officials told us they used the survey results in 
deciding which courses to offer. Our analysis of this survey, however, 
showed that it focused only on training topics directly related to the 
CFSR, so it might offer only limited information on whether regional 
officials wanted training on other topics. Specifically, the survey 
asked staff to check their top 5 training choices from among 11 CFSR-
related topics. While survey respondents were also given the 
opportunity to write in additional training tropics they desired, only 
2 of the survey's 27 respondents did so.[Footnote 23] One indicated a 
greater need for training on Indian child welfare issues, and another 
expressed a desire to learn more about the entire child welfare system. 
Although it is not possible to determine whether more respondents would 
have prioritized non-CFSR training areas had the survey been designed 
to elicit such information, our interviews with regional staff suggest 
that some of them wish to obtain additional non-CFSR training. For 
example, a staff member from one region told us she has not been 
adequately trained in child welfare and believed that her credibility 
was damaged when a state wanted advice that she could not provide on 
how to help older youth prepare to exit from foster care.

In addition to offering training, ACF organizes biennial conferences 
for state and federal child welfare officials. Nonetheless, staff from 
5 regions told us that they wanted more substantive interaction with 
their ACF colleagues, such as networking at conferences, to increase 
their overall child welfare expertise. Staff from 6 of the 10 regions 
told us that their participation in conferences is limited because of 
funding constraints.

Further, staff in all 10 regions provide ongoing assistance or ad hoc 
counseling to states, but staff from 6 regions told us they would like 
to conduct site visits with states more regularly to improve their 
relationships with state officials and provide more targeted 
assistance. For example, staff in most regions told us that they assist 
states predominantly by e-mail and telephone on topics such as 
interpretation of Title IV-E eligibility criteria. Additionally, staff 
in 7 regions said that they sometimes visit with states to participate 
in state planning meetings--part of the annual child and family 
services planning effort--or to give presentations at state conferences 
on topics such as court improvement. However, staff in 4 regions felt 
their travel funds were constrained and explained that they try to 
stretch their travel dollars by addressing states' non-CFSR needs, such 
as court improvements, during CFSR-related visits. While an ACF senior 
official from central office confirmed that CFSR-related travel 
constituted 60 percent of its 2002 child welfare-monitoring budget, 
this official added that CFSR spending represents an infusion of 
funding rather than a reprioritization of existing dollars and stated 
that regional administrators have discretion over how the funds are 
allocated within their regions. In addition, the same official stated 
that he knew of no instance in which a region requested more money for 
travel than it received.

Concerns from state officials in all 5 of the states we visited echoed 
those of regional office staff and confirmed the need for improvements 
to the overall training and technical assistance structure, while 
respondents' comments on our survey showed more mixed perceptions on 
the quality of assistance they received. For example, state officials 
in New York and Wyoming commented that ACF staff from their respective 
regional offices did not have sufficient time to spend with them on 
CFSR matters because regional staff were simultaneously occupied 
conducting reviews in other states. Further, an Oklahoma state official 
commented that assistance from one resource center was not as specific 
or helpful as desired. Specifically, when the state asked the resource 
center to provide a summary of other states' policies regarding the 
intake of abuse and neglect allegations, the resource center did not 
provide an analysis of sufficient depth for the state to explore 
possible reforms.

According to state survey respondents, however, satisfaction with the 
training and technical assistance provided by regional offices varied 
by CFSR phase. For example, among states reviewed in 2001, 2002, and 
2003, satisfaction was generally highest in the statewide assessment 
phase, but then dropped during on-site review and PIP development 
before rising again in the PIP implementation phase. Across all phases 
of the CFSR process, however, states reviewed in 2003 had much higher 
levels of satisfaction with regional office assistance than those 
states reviewed in 2001, suggesting improvements to regional office 
training and technical assistance as the process evolved. Further, 
based on survey data and our follow-up calls with selected states, 
satisfaction was also mixed in regard to resource center provided 
assistance. For example, among states reporting in our survey on the 
quality of assistance provided by the resource center for 
Organizational Improvement and the resource center for Information 
Technology--the two resource centers that provide specific support to 
states regarding data issues and PIP development--satisfaction was 
generally lower in every phase among states reviewed in 2003 than among 
states reviewed in 2001. The only exception to this was during the PIP 
development phase, for which states reviewed in 2003 reported higher 
levels of satisfaction with the resource center for Organizational 
Improvement than states reviewed in 2001, suggesting positive responses 
to the on-site training and technical assistance this resource center 
has recently been providing to aid states in their PIP planning 
efforts.[Footnote 24]

ACF officials told us the CFSR has become the agency's primary 
mechanism for monitoring states and facilitating program improvement, 
but they acknowledged that regional office staff might not have 
realized the full utility of the CFSR as a tool to integrate all 
existing training and technical assistance efforts. Further, according 
to ACF officials, meetings to discuss a new system of training and 
technical assistance are ongoing, though recommendations were not 
available at the time of publication. Levels of resource center 
funding, the scope and objectives of the resource centers' work, and 
the contractors who operate the resource centers are all subject to 
change before the current cooperative agreements expire at the close of 
fiscal year 2004.[Footnote 25]

Conclusions:

ACF and the states have devoted considerable resources to the CFSR 
process, but concerns remain regarding the validity of some data 
sources and the limited use of all available information to determine 
substantial conformity. Further, no state to date has passed the 
threshold for substantial conformity on all CFSR measures. The majority 
of states surveyed agreed that CFSR results are similar to their own 
evaluation of areas needing improvement. However, without using more 
reliable data--and in some cases, additional data from state self-
assessments--to determine substantial conformity, ACF may be over-or 
underestimating the extent to which states are actually meeting the 
needs of the children and families in their care. These over-or 
underestimates can, in turn, affect the scope and content of the PIPs 
that states must develop in response. As states face difficult budget 
decisions, accurate performance information could be critical to 
deciding how best to allocate resources. We previously reported on the 
reliability of state-reported child welfare data and recommended that 
HHS consider additional ways to enhance the guidance and assistance 
offered to states to help them overcome the key challenges in 
collecting and reporting child welfare data. In response to this 
recommendation, HHS said that ACF has provided extensive guidance on 
how states can improve the quality of their data and acknowledged that 
additional efforts were under way.

In addition, the PIP development, approval, and monitoring processes 
remain unclear to some, potentially reducing states' credibility with 
their stakeholders and straining the federal/state partnership. 
Similarly, regional officials are unclear as to how they can accomplish 
their various training and technical assistance responsibilities, 
including the CFSR. Without clear guidance on how to systematically 
prepare and monitor PIP-related documents, and how regional officials 
can integrate their many oversight responsibilities, ACF has left state 
officials unsure of how their progress over time will be judged and 
potentially complicated its own monitoring efforts.

Recommendations for Executive Action:

To ensure that ACF uses the best available data in measuring state 
performance, we recommend that the Secretary of HHS expand the use of 
additional data states may provide in their statewide assessments and 
consider alternative data sources when available, such as longitudinal 
data that track children's placements over time, before making final 
CFSR determinations.

In addition, to ensure that ACF regional offices and states fully 
understand the PIP development, approval, and monitoring processes, and 
that regional offices fully understand ACF's prioritization of the CFSR 
as the primary mechanism for child welfare oversight, we recommend that 
the Secretary of HHS take the following two actions:

* issue clarifying guidance on the PIP process and evaluate states' and 
regional offices' adherence to this instruction, and:

* provide guidance to regional offices explaining how to better 
integrate the many training and technical assistance activities for 
which they are responsible, such as participation in state planning 
meetings and the provision of counsel to states on various topics, with 
their new CFSR responsibilities.

Agency Comments:

We received comments on a draft of this report from HHS. These comments 
are reproduced in appendix IV. HHS also provided technical 
clarifications, which we incorporated where appropriate.

HHS generally agreed with our findings and noted that the CFSR process 
has already focused national attention on child welfare reform, but 
because the CFSR is the first review of its kind, HHS is engaged in 
continuous monitoring and improvement of the process. However, in its 
technical comments, HHS commented that while it acknowledges that the 
CFSR is its top priority, it disagreed with our statement that HHS's 
focus rests exclusively on implementing the CFSR, stating that the 
Administration for Children and Families (ACF) also conducts other 
oversight efforts, such as Title IV-E eligibility reviews and AFCARS 
assessments. While we acknowledged ACF's other oversight activities in 
the background section of the report, this report focuses primarily on 
the CFSR and we reflected the comments that ACF officials made 
throughout the course of our work that the CFSR was the primary tool 
for monitoring state performance and that it served as the umbrella for 
all monitoring activities undertaken by central and regional ACF staff.

HHS further noted in its technical comments that we were wrong to 
suggest that federal staff do not know how to monitor state PIPs or 
assess financial penalties. However, we do not report that ACF is 
unsure of how to monitor PIPs or how to assess financial penalties--
rather, we reported that ACF regional staff have not received 
sufficient guidance on how to best monitor PIPs and that ACF officials 
have not decided how or when to apply such penalties, even though two 
states to date have completed their initial PIP implementation 
timeframe and all states reviewed thus far are engaged in PIP 
development and implementation.

With regard to our first recommendation, HHS acknowledged that several 
steps are under way to address necessary data improvements and said 
that states have begun to submit more accurate information in their 
AFCARS and NCANDS profiles, with HHS's assistance. HHS also commented 
that we failed to properly emphasize the states' responsibility to 
improve overall data quality. We believe that our report, as well as 
our previous report on child welfare data and states' information 
systems, addresses HHS's activities and the steps many states have 
taken to enhance their CFSR data. Given that many states have developed 
independent data collection tools--and included findings from these 
instruments in their statewide assessments--our recommendation is meant 
to encourage HHS to work more closely with all states to supplement 
their AFCARS and NCANDS data in order to improve the determinations 
made about state performance.

In addition, HHS commented that our report emphasized the limitations 
of the 50 case sample size without focusing on the expenses and the 
increased state and federal staff time that would likely be associated 
with efforts to increase the sample size. We agree that additional 
expenses and staff time would likely be needed to increase the sample 
size and recommended that ACF use additional data--beyond the 
information collected from the 50 case reviews and ACFARS and NCANDS 
data--to develop a more accurate picture of state performance. This 
information could include the data that many states already collect on 
their performance, such as longitudinal information tracking children 
from their time of entry into the system.

In response to our second recommendation, HHS said that it has 
continued to provide technical assistance and training to states and 
regional offices, when appropriate. HHS noted that it is committed to 
continually assessing and addressing training and technical assistance 
needs. In this context, our recommendation was intended to encourage 
HHS to enhance existing training efforts and focus both on state and on 
regional officials' needs in understanding and incorporating the CFSR 
process into their overall improvement and oversight efforts.

We also provided a copy of our draft report to child welfare officials 
in the five states we visited--California, Florida, New York, Oklahoma, 
and Wyoming. We received comments from California, Florida, New York, 
and Oklahoma, all of which generally agreed with our findings and 
provided various technical comments, which we also incorporated where 
appropriate.

We are sending copies of this report to the Secretary of Health and 
Human Services, state child welfare directors, and other interested 
parties. We will make copies available to others on request. In 
addition, the report will be available at no charge on GAO's Web site 
at http://www.gao.gov. If you or your staff have any questions or wish 
to discuss this material further, please call me at (202) 512-8403 or 
Diana Pietrowiak at (202) 512-6239. Key contributors to this report are 
listed in appendix IV.

Signed by: 

Cornelia M. Ashby, 
Director, Education, Workforce, and Income Security Issues:

List of Congressional Requesters:

The Honorable Tom DeLay
Majority Leader House of Representatives:

The Honorable Charles E. Grassley 
Chairman, Committee on Finance 
United States Senate:

The Honorable George Miller 
Ranking Democratic Member 
Committee on Education and the Workforce 
House of Representatives:

The Honorable Charles B. Rangel 
Ranking Democratic Member 
Committee on Ways and Means 
House of Representatives:

The Honorable Fortney Pete Stark 
Ranking Democratic Member 
Subcommittee on Health 
Committee on Ways and Means 
House of Representatives:

The Honorable Benjamin L. Cardin 
Ranking Democratic Member 
Subcommittee on Human Resources 
Committee on Ways and Means 
House of Representatives:

[End of section]

Appendix I: Objectives, Scope, and Methodology:

Objectives:

The objectives of our study were to report on (1) ACF's and the states' 
experiences preparing for and conducting the statewide assessments and 
on-site reviews; (2) ACF's and the states' experiences developing, 
funding, and implementing items in the PIP; and (3) additional efforts 
that ACF has taken beyond the CFSR to ensure that all states meet 
federal goals of safety, permanency, and well-being.

Scope and Methodology:

To gather information about ACF's and the states' experiences with the 
CFSR and PIP process, we utilized multiple methodologies to solicit 
information from both ACF and the states, including (1) a Web-based 
survey to state child welfare agencies; (2) site visits to five states; 
(3) a content analysis of all 31 PIPs available as of January 1, 2004; 
(4), interviews with ACF officials in Washington and all regional 
offices, directors of all resource centers, and child welfare experts 
nationwide; and (5) a review of CFSR regulations and the available 
guidance offered to states. We conducted our work between May 2003 and 
February 2004 in accordance with generally accepted government auditing 
standards.

Survey:

To gather information about states' experiences with each phase of the 
CFSR and PIP process, we distributed a Web-based survey to all 50 
states, the District of Columbia, and Puerto Rico on July 30, 2003. We 
pretested the survey instrument with officials in the District of 
Columbia, Kentucky, and Maryland; and after extensive follow-up, we 
received survey responses from all 50 states and the District of 
Columbia for a 98 percent response rate.[Footnote 26] We did not 
independently verify the information obtained through the survey.

The survey asked a combination of questions that allowed for open-ended 
and close-ended responses. Because some states had not yet begun their 
statewide assessments and others had already submitted quarterly PIP 
progress reports at the time that our survey was released, the 
instrument was designed with skip patterns directing states to comment 
only on the CFSR stages that they had begun or completed to that point. 
Therefore, the number of survey respondents for each question varied 
depending on the number of states that had experienced that stage of 
the CFSR and PIP processes.

To supplement the survey and elaborate on survey responses, we selected 
10 states with which to conduct follow-up phone calls based on their 
answers to the survey's open-ended questions.[Footnote 27] These calls 
helped us obtain more specific examples about states' experiences 
preparing for the CFSR; developing, funding, and implementing a PIP; 
and working with ACF to improve their child welfare systems.

Because this was not a sample survey, there are no sampling errors. 
However, the practical difficulties of conducting any survey may 
introduce errors, commonly referred to as nonsampling errors. For 
example, difficulties in how a particular question is interpreted, in 
the sources of information that are available to respondents, or in how 
the data are entered into a database or were analyzed, can introduce 
unwanted variability into the survey results. We took steps in the 
development of the questionnaire, the data collection, and the data 
analysis to minimize these nonsampling errors. For example, social 
science survey specialists designed the questionnaire in collaboration 
with GAO staff with subject matter expertise. Then, the draft 
questionnaire was pretested with a number of state officials to ensure 
that the questions were relevant, clearly stated, and easy to 
comprehend. When the data were analyzed, a second, independent analyst 
checked all computer programs. Since this was a Web-based survey, 
respondents entered their answers directly into the electronic 
questionnaire. This eliminated the need to have the data keyed into a 
database thus removing an additional source of error.

Site Visits:

To gather more detailed information about the states' experiences with 
the CFSR and PIP process, we selected five states to visit--California, 
Florida, New York, Oklahoma, and Wyoming--based on the timing and 
results of each state's CFSR, as well as their differences in location, 
size of child welfare population, degree of privatization of services, 
size of tribal populations, and whether they had state or locally 
administered systems. In preparation for the visits and to understand 
the unique circumstances in each state, we obtained and reviewed 
relevant literature from each of the five states, such as the statewide 
assessment, the CFSR final report, and any available PIPs or quarterly 
reports. Additionally, we reviewed relevant past or current litigation 
that may affect the states' delivery of services as identified by the 
National Center for Youth Law's Litigation Docket (2002).

During our visits to each state, we talked with officials from the 
state child welfare agency along with officials and staff from at least 
one local agency office that was selected for the CFSR on-site review. 
Specifically, in each state we spoke with state and local officials 
responsible for guiding the states' efforts throughout the review 
process; CFSR on-site reviewers; and stakeholders, including judges, 
child advocates, private providers, foster parents, and child welfare 
staff. Some detailed information regarding key CFSR milestones among 
the five states we visited is included in appendix III.

Content Analysis of Available PIPs:

To learn about states' improvement strategies, we conducted a content 
analysis of the 31 available PIPs that ACF had approved by January 1, 
2004. For each of these PIPs, we classified the state's action steps as 
relating to one or more of the following: policies and procedures, data 
collection and analysis, staff supports, foster parent supports or 
services and resources for children and families, state legislative 
supports, and federal technical assistance. Table 2 in the report 
summarizes how we classified the PIP strategies and indicates the 
number of states including each strategy in its PIP.

Interviews:

To gather information about ACF's experience with the CFSR and PIP 
process, we interviewed ACF officials in Washington, D.C., who are 
involved in the CFSR process and ACF staff in all 10 of the regional 
offices; directors of each resource center; and ACF contractors working 
on CFSR-related activities. Further, we observed the final debriefing 
sessions for three states--South Carolina, Virginia, and Washington--
during the weeks of their respective on-site reviews.

In addition to our interviews with ACF officials, we also interviewed 
10 prominent child welfare experts and researchers, such as those 
affiliated with the Chapin Hall Center for Children, the Child Welfare 
League of America, the National Coalition on Child Protection Reform, 
and the University of California at Berkeley, to learn additional 
information about the states' experiences with the CFSR process, 
including information about states' concerns with the reliability of 
CFSR data, states' involvement of tribes as stakeholders, and the 
media's coverage of egregious child welfare cases.

Review of ACF Guidance to States:

To gather information about CFSR regulations and the available training 
and technical assistance offered to states, we reviewed ACF's 
regulations, its policy memorandums, and the CFSR manual it makes 
available to states. In addition, we obtained and reviewed a list of 
all of the resource centers' training and technical assistance 
activities provided to the five states we visited during our site 
visits.

[End of section]

Appendix II: List of Outcomes and Systemic Factors and Their Associated 
Items:

(Items with an ampersand have associated national standards.):

Outcome factors:

Child Safety:

Outcome 1: Children are, first and foremost, protected from abuse and 
neglect.

Item 1: Timeliness of initiating investigations on reports of child 
maltreatment:

Item 2: Repeat maltreatment:

* Recurrence of maltreatment &:

* Incidence of child abuse and/or neglect in foster care &:

Outcome 2: Children are safely maintained in their own homes whenever 
possible and appropriate.

Item 3: Services to family to protect child(ren) in home and prevent 
removal:

Item 4: Risk of harm to child(ren):

Permanency for Children:

Outcome 3: Children have permanency and stability in their living 
conditions.

Item 5: Foster care re-entries &:

Item 6: Stability of foster care placement &:

Item 7: Permanency goal for child:

Item 8: Reunification, guardianship, or permanent placement with 
relatives:

* Length of time to achieve permanency goal of reunification &:

Item 9: Adoption:

* Length of time to achieve permanency goal of adoption &:

Item 10: Permanency goal of other planned permanent living arrangement:

Outcome 4: The continuity of family relationships and connections is 
preserved for children.

Item 11: Proximity of foster care placement:

Item 12: Placement with siblings:

Item 13: Visiting with parents and siblings in foster care:

Item 14: Preserving connections:

Item 15: Relative placement:

Item 16: Relationship of child in care with parents:

Child and Family Well-being:

Outcome 5: Families have enhanced capacity to provide for their 
children's needs.

Item 17: Needs and services of child, parents, foster parents:

Item 18: Child and family involvement in case planning:

Item 19: Worker visits with child:

Item 20: Work visits with parent(s):

Outcome 6: Children receive appropriate services to meet their 
educational needs.

Item 21: Education needs of the child:

Outcome 7: Children receive adequate services to meet their physical 
and mental health needs.

Item 22: Physical health of child:

Item 23: Mental health of child.

Systemic Factors:

Systemic factor 1: Statewide information system:

Item 24: State is operating a statewide information system that, at a 
minimum, can identify the status, demographic characteristics, 
location, and goals for the placement of every child who is (or within 
the immediately preceding 12 months has been) in foster care.

Systemic factor 2: Case review system:

Item 25: Provides a process that ensures that each child has a written 
case plan to be developed jointly with the child's parent(s) that 
includes the required provisions:

Item 26: Provides a process for the periodic review of the status of 
each child, no less frequently than once every 6 months, either by a 
court or by administrative review.

Item 27: Provides a process that ensures that each child in foster care 
under the supervision of the state had a permanency hearing in a 
qualified court or administrative body no later than 12 months from the 
date the child entered foster care and no less frequently than every 12 
months thereafter.

Item 28: Provides a process for termination of parental rights 
proceedings in accordance with the provisions of the Adoption and Safe 
Families Act.

Item 29: Provides a process for foster parents, pre-adoptive parents, 
and relative caregivers of children in foster care to be notified of, 
and have an opportunity to be heard in, any review or hearing held with 
respect to the child.

Systemic factor 3: Quality assurance system:

Item 30: The state has developed and implemented standards to ensure 
that children in foster care are provided quality services that protect 
the safety and health of children.

Item 31: The state is operating an identifiable quality assurance 
system that is in place in the jurisdictions where the services 
included in the Child and Family Services Plan are provided, evaluates 
the quality of services, identifies strengths and needs of the service 
delivery system, provides relevant reports, and evaluates program 
improvement measures implemented.

Systemic factor 4: Training:

Item 32: The state is operating a staff development and training 
program that supports the goals and objectives in the Child and Family 
Services Plan, addresses services provided under Titles IV-B and IV-E, 
and provides initial training for all staff who deliver these services.

Item 33: The state provides for ongoing training for staff that 
addresses the skills and knowledge base needed to carry out their 
duties with regard to the services included in the Child and Family 
Services Plan.

Item 34: The state provides training for current or prospective foster 
parents, adoptive parents, and staff of state-licensed or approved 
facilities that care for children receiving foster care or adoption 
assistance under Title IV-E that addresses the skills and knowledge 
base needed to carry out their duties with regard to foster and adopted 
children.

Systemic factor 5: Service array:

Item 35: The state has in place an array of services that assess the 
strengths and needs of children and families and determine other 
service needs, address the needs of families in addition to individual 
children in order to create a safe home environment, enable children to 
remain safely with their parents when reasonable, and help children in 
foster and adoptive placements achieve permanency.

Item 36: The services in item 35 are accessible to families and 
children in all political jurisdictions covered in the State's Child 
and Family Services Plan.

Item 37: The services in item 35 can be individualized to meet the 
unique needs of children and families served by the agency.

Systemic factor 6: Agency responsiveness to the community:

Item 38: In implementing the provisions of the Child and Family 
Services Plan, the state engages in ongoing consultation with tribal 
representatives, consumers, services providers, foster care providers, 
the juvenile court, and other public and private child-and family-
serving agencies and includes the major concerns of these 
representatives in the goals and objectives of the Child and Family 
Services Plan.

Item 39: The agency develops, in consultation with these 
representatives, annual reports of progress and services delivered 
pursuant to the Child and Family Services Plan.

Item 40: The state's services under the Child and Family Services Plan 
are coordinated with services or benefits of other federal or federally 
assisted programs serving the same population.

Systemic factor 7: Foster and adoptive parent licensing, recruitment, 
and retention:

Item 41: The state has implemented standards for foster family homes 
and child care institutions that are reasonably in accord with 
recommended national standards.

Item 42: The standards are applied to all licensed or approved foster 
family homes or child care institutions receiving Title IV-E or IV-B 
funds.

Item 43: The state complies with federal requirements for criminal 
background clearances as related to licensing or approving foster care 
and adoptive placements and has in place a case-planning process that 
includes provisions for addressing the safety of foster care and 
adoptive placements for children.

Item 44: The state has in place a process for ensuring the diligent 
recruitment of potential foster and adoptive families that reflect the 
ethnic and racial diversity of children in the state for whom foster 
and adoptive homes are needed.

Item 45: The state has in place a process for the effective use of 
cross-jurisdictional resources to facilitate timely adoptive or 
permanent placements for waiting children.

[End of section]

Appendix III: Dates on Which Site Visit States Reached CFSR Milestones:

Date the state submitted its statewide assessment; 
California: 7/02; 
Florida: 5/01; 
New York: 4/01; 
Oklahoma: 1/02; 
Wyoming: 6/02.

Date ACF began the on-site review; 
California: 9/23/02; 
Florida: 8/6/ 01; 
New York: 6/18/01; 
Oklahoma: 3/18/02; 
Wyoming: 7/8/02.

Date ACF released the final CFSR report; 
California: 1/10/03; 
Florida: 4/23/02; 
New York: 1/9/02; 
Oklahoma: 7/1/02; 
Wyoming: 4/1/03.

Time lapse in release of final report; 
California: 75 days; 
Florida: 182 days; 
New York: 143 days; 
Oklahoma: 71 days; 
Wyoming: 187 days.

Date the state's PIP was due; 
California: 4/10/03; 
Florida: 7/22/02; 
New York: 4/9/02; 
Oklahoma: 9/29/02; 
Wyoming: 6/30/03.

Date ACF approved the PIP; 
California: 6/24/03; 
Florida: 4/30/03; 
New York: 4/14/03; 
Oklahoma: 1/22/03; 
Wyoming: 1/1/04.

Time lapse in PIP approval; 
California: 54 days; 
Florida: 203 days; 
New York: 265 days; 
Oklahoma: 83 days; 
Wyoming: 134 days.

Date the state submitted its first quarterly report; 
California: 10/29/ 03; 
Florida: 7/30/03; 
New York: 8/11/03; 
Oklahoma: 5/15/03; 
Wyoming: 4/ 30/04.

Total number of outcomes and systemic factors not in substantial 
conformity[A]; 
California: 12; 
Florida: 8; 
New York: 8; 
Oklahoma: 9; 
Wyoming: 10.

Estimated financial penalties; 
California: $18,244,430; 
Florida: $3,639,552; 
New York: $2,284,407; 
Oklahoma: $727,300; 
Wyoming: $137,369. 

Source: GAO analysis of ACF data.

[A] The total number of outcomes and systemic factors evaluated in the 
CFSR is 14.

[End of table]

[End of section]

Appendix IV: Comments from the Department of Health and Human Services:

DEPARTMENT OF HEALTH & HUMAN SERVICES	
Office of Inspector General:

Washington, D.C. 20201:

MAR 29 2004:

Ms. Cornelia M. Ashby:

Director, Education, Workforce, and Income Security Issues:

United States General Accounting Office 
Washington, D.C. 20548:

Dear Ms. Ashby:

Enclosed are the Department's comments on your draft report entitled, 
"Child and Family Services Reviews: Better Use of Data and Improved 
Guidance Could Enhance HHS's Oversight of State Performance." The 
comments represent the tentative position of the Department and are 
subject to reevaluation when the final version of this report is 
received.

The Department provided several technical comments directly to your 
staff.

The Department appreciates the opportunity to comment on this draft 
report before its publication.

Sincerely,

Dara Corrigan:

Acting Principal Deputy Inspector General:

Enclosure:

The Office of Inspector General (OIG) is transmitting the Department's 
response to this draft report in our capacity as the Department's 
designated focal point and coordinator for General Accounting Office 
reports. OIG has not conducted an independent assessment of these 
comments and therefore expresses no opinion on them.

Comments of the Department of Health and Human Services on the General 
Accounting Office's Draft Report: "Child and Family Services Reviews: 
Better Use of Data and Improved Guidance Could Enhance HHS's Oversight 
of State Performance" (GAO-04-333):

The Department of Health and Human Services (HHS) appreciates the 
opportunity to comment on the above-referenced GAO draft report.

General Comments:

The Child and Family Service Review (CFSR) process has already had a 
significant impact on focusing unprecedented national attention on 
child welfare practices and has the potential to be an agent for real 
improvement both in child welfare services and outcomes. HHS would like 
to acknowledge that because the CFSR is the first review of its kind, 
continuous monitoring and improvements to the process itself would be 
needed as we become more experienced in assessing State programs in 
this manner. Just as States are engaging in this new process of 
continuous assessment and improvement, HHS is as well. Therefore, as 
issues and concerns arise in this process, we are addressing them 
through ongoing training and other avenues. The report confirms that 
States feel the same way about the process. The States also indicated 
to GAO that the results of the CFSR are typically consistent with what 
they know to be the strengths and deficiencies of their child welfare 
systems.

We also acknowledge that concerns regarding the usefulness of Adoption 
and Foster Care Analysis and Reporting System (AFCARS) data noted in 
GAO's report are consistent with those reported in the OIG inspection 
entitled, "AFCARS: Challenges and Limitations" (OEI-07-01-00660). 
Further, based on a series of inspections focused on "Children's Use of 
Health Care Services While in Foster Care," the OIG supports the use of 
Medicaid claims data in determining State performance in meeting 
outcome measures focused on the health and well being of children.

Recommendation 1:

To ensure that HHS uses the best available data in measuring State 
performance, the Secretary of HHS should expand the use of additional 
data States may provide in their Statewide assessments and consider 
alternative data sources, when available, such as longitudinal data 
that track children's placements over time, before making final CFSR 
determinations.

Comments:

For the most part, HHS is aware already of most of the CFSR-related 
issues that are presented in this GAO report. Specifically, the report 
addresses areas that HHS is currently working to improve, i.e. the 
Program Improvement Plan (PIP) development and approval process, the 
quality of technical assistance provided by both regional staff and the 
resource centers, the need for guidance on how States can improve data 
reliability, etc.

HHS has engaged an independent contractor to assess and make 
recommendations for improving the CFSR.	In completing its task, the 
contractor has convened a Consultation Work Group to provide input into 
the process. The establishment of this Work Group is responsive to 
requests received from representatives of States that have undergone a 
CFSR. The State officials at a national meeting in January 2003 first 
made the suggestion.

The role of the Work Group is to provide feedback, new ideas and 
insights that will assist the contractor to make recommendations to the 
Children's Bureau (Bureau) regarding possible changes or improvements 
to the CFSR process, as well as possible adjustments to the outcomes 
and measurements of outcomes used in the reviews.

On the issue of data sources and reliability, we specifically chose to 
use the National Child Abuse and Neglect Data Reporting System (NCANDS) 
and the Adoption and Foster Care Analysis and Reporting System (AFCARS) 
as the data sources for the review because these are the national 
systems for data reporting in child welfare. Although data reliability 
is a shared issue, the GAO report does not address the fact that States 
must take some ownership for problems with data quality, since it is 
the data they submit that ultimately is used for the national standards 
and for the on-site portion of the CFSR, especially the AFCARS data 
requirements. Nonetheless, we recognize the ongoing need to address 
data quality concerns. The Bureau and the ACF regional offices have 
made concerted efforts to work with States to help improve data 
reporting and reliability in recent years. It should be noted that the 
quality of the data being submitted by States has dramatically 
increased since FY 1998, with the most substantial improvements having 
occurred since FY 2000.

HHS provides funding for a national resource center to work with States 
around data reporting issues and conducts on-site AFCARS assessments in 
each State. Technical assistance on the reporting of NCANDS data is 
also available. HHS continues to work extensively with States to point 
out potential data inaccuracies and provide consultation and technical 
assistance to help them correct data issues. HHS also provides States 
with the opportunity to submit corrected data so that we have an 
accurate picture.

In addition to efforts to strengthen current AFCARS reporting, for the 
last year and a half, an internal work group has convened to look at 
the current AFCARS system. The current project will be working to 
revise and update the AFCARS system to reflect emerging needs. This 
should go a long way in improving the quality of the data submitted by 
States.

HHS suggests that the report would be improved if the tradeoffs 
inherent in addressing some of the important issues raised were 
recognized more explicitly. For instance, the relatively small sample 
size for the on-site reviews is discussed on page 15 and the burden of 
the on-site reviews is discussed on page 13. Yet, nowhere is it 
explicitly recognized that a larger sample size would increase the 
already substantial burden of the on-site reviews or that some staff's 
regular duties were negatively impacted, preventing caseworkers from 
conducting necessary child abuse investigations or home visits. Whether 
it is worth imposing this additional burden in order to:

avoid discrepancies between Statewide assessments and on-site reviews 
is a judgment call HHS made in designing the review process. That only 
one State has chosen to examine additional cases to resolve such a 
discrepancy would seem to indicate that the current balance is not 
unreasonable (see page 17). Also, while the GAO report criticism that 
the current sample size is too small from which to generalize, it fails 
to identify for the reader how large a sample size would be required to 
be sufficiently representative. Without knowing how much larger the 
sample would need to be to address the identified weaknesses, the 
reader cannot directly weigh the tradeoffs.

Recommendation 2:

To ensure that HHS regional offices and States fully understand the PIP 
development, approval, and monitoring processes, and that regional 
offices fully understand HHS's prioritization of the CFSR as the 
primary mechanism for child welfare oversight, we recommend that the 
Secretary of HHS take the following two actions:

* issue clarifying guidance on the PIP process and evaluate States' and 
regional offices' adherence to this instruction and:

provide guidance to regional offices explaining how to better integrate 
the many training and technical assistance activities for which they 
are responsible, such as participation in State planning meetings and 
the provision of counsel to States on various topics, with their new 
CFSR responsibilities.

Comments:

HHS has continued to provide technical assistance and training to 
States and regional offices, when appropriate. These efforts include, 
but are not limited to, an annual meeting with States that completed 
the CFSR in the previous year and ongoing communication with States, as 
well as the regional offices. National conferences focusing on this 
topic have been held each year. Regional offices have also been 
provided technical assistance on AFCARS reporting at Regional Child 
Welfare Administrators' meetings. HHS has provided an unprecedented 18 
courses on various topics related to the CFSR. HHS is committed to 
continue assessing and addressing training and technical assistance 
needs.

[End of section]

Appendix V: GAO Contacts and Acknowledgments:

GAO Contacts:

Diana Pietrowiak, (202) 512-6239 Joy Gambino, (202) 512-2838:

Staff Acknowledgments:

In addition to those named above, Elizabeth Caplick and Catherine Roark 
made key contributions to this report. Amy Buck, Karen Burke, Jason 
Kelly, Stuart Kaufman, Luann Moy, and Jerome Sandau also provided key 
technical assistance.

[End of section]

Related GAO Products:

Child Welfare: Improved Federal Oversight Could Assist States in 
Overcoming Key Challenges. GAO-04-418T. Washington, D.C.: January 28, 
2004.

D.C. Family Court: Progress Has Been Made in Implementing Its 
Transition. GAO-04-234. Washington, D.C.: January 6, 2004.

Child Welfare: States Face Challenges in Developing Information Systems 
and Reporting Reliable Child Welfare Data. GAO-04-267T. Washington, 
D.C.: November 19, 2003.

Child Welfare: Enhanced Federal Oversight of Title IV-B Could Provide 
States Additional Information to Improve Services. GAO-03-956. 
Washington, D.C.: September 12, 2003.

Child Welfare: Most States Are Developing Statewide Information 
Systems, but the Reliability of Child Welfare Data Could be Improved. 
GAO-03-809. Washington, D.C.: July 31, 2003.

D.C. Child and Family Services: Key Issues Affecting the Management of 
Its Foster Care Cases. GAO-03-758T. Washington, D.C.: May 16, 2003.

Child Welfare and Juvenile Justice: Federal Agencies Could Play a 
Stronger Role in Helping States Reduce the Number of Children Placed 
Solely to Obtain Mental Health Services. GAO-03-397. Washington, D.C.: 
April 21, 2003.

Foster Care: States Focusing on Finding Permanent Homes for Children, 
but Long-Standing Barriers Remain. GAO-03-626T. Washington, D.C.: April 
8, 2003.

Child Welfare: HHS Could Play a Greater Role in Helping Child Welfare 
Agencies Recruit and Retain Staff. GAO-03-357. Washington, D.C.: March 
31, 2003.

Foster Care: Recent Legislation Helps States Focus on Finding Permanent 
Homes for Children, but Long-Standing Barriers Remain. GAO-02-585. 
Washington, D.C.: June 28, 2002.

District of Columbia Child Welfare: Long-Term Challenges to Ensuring 
Children's Well-Being. GAO-01-191. Washington, D.C.: December 29, 2000.

Child Welfare: New Financing and Service Strategies Hold Promise, but 
Effects Unknown. GAO/T-HEHS-00-158. Washington, D.C.: July 20, 2000.

Foster Care: States' Early Experiences Implementing the Adoption and 
Safe Families Act. GAO/HEHS-00-1. Washington, D.C.: December 22, 1999.

Foster Care: HHS Could Better Facilitate the Interjurisdictional 
Adoption Process. GAO/HEHS-00-12. Washington, D.C.: November 19, 1999.

Foster Care: Effectiveness of Independent Living Services Unknown. GAO/
HEHS-00-13. Washington, D.C.: November 10, 1999.

Foster Care: Kinship Care Quality and Permanency Issues. GAO/HEHS-99-
32. Washington, D.C.: May 6, 1999.

Juvenile Courts: Reforms Aim to Better Serve Maltreated Children. GAO/
HEHS-99-13. Washington, D.C.: January 11, 1999.

Child Welfare: Early Experiences Implementing a Managed Care Approach. 
GAO/HEHS-99-8. Washington, D.C.: October 21, 1998.

Foster Care: Agencies Face Challenges Securing Stable Homes for 
Children of Substance Abusers. GAO/HEHS-98-182. Washington, D.C.: 
September 30, 1998.

FOOTNOTES

[1] The CFSR measures state performance on 45 performance items, which 
correspond to 7 outcomes and 7 systemic factors. The outcomes relate to 
children's safety, permanency, and well-being, and the systemic factors 
address state agency management and responsiveness to the community. 
Six national standards, as reported in the Adoption and Foster Care 
Analysis and Reporting System (AFCARS) and the National Child Abuse and 
Neglect Data System (NCANDS), apply to 5 of the 45 items. Three of 
these standards are based on the 75th percentile of all states' 
performance--adoption; stability of foster care placements; and length 
of time to achieve reunification, guardianship, or permanent placement 
with relatives--because a higher incidence is desirable. However, the 
remaining three standards--recurrence of maltreatment, incidence of 
child abuse/neglect in foster care, and foster care re-entries--are 
based on the 25th percentile of state performance, because lower 
incidence is a desired outcome for these measures.



[2] The term stakeholder refers to two groups: (1) agency stakeholders, 
such as judges or advocates, whose responsibilities are closely related 
to the work of the child welfare agency and who can comment on the 
agency's overall performance on outcomes and systemic factors, and (2) 
case-specific stakeholders, such as parents, caseworkers, children, or 
others who are interviewed to provide first-hand information that 
supplements reviewers' assessment of paper or electronic case files.

[3] Each of the 10 resource centers operates under a 5-year cooperative 
agreement with HHS to help states implement federal child welfare 
legislation and provide training and technical assistance to states. 

[4] Title IV-B of the Social Security Act, consisting of two subparts, 
is the primary source of federal funding for services to help families 
address problems that lead to child abuse and neglect and to prevent 
the unnecessary separation of children from their families. Funding 
under Title IV-E of the Social Security Act is used primarily to pay 
for the room and board of children in foster care.

[5] States began voluntarily reporting to NCANDS in 1990, and in 1995 
started reporting to AFCARS on the demographic characteristics of 
adoptive and foster children and their parents, as well as foster 
children's type of placement and permanency goals. We recently issued a 
report on states' child welfare information systems and the reliability 
of child welfare data. See Child Welfare: Most States Are Developing 
Statewide Information Systems, but the Reliability of Child Welfare 
Data Could Be Improved, GAO-03-809 (Washington, D.C.: July 31, 2003).

[6] States achieve substantial conformity on outcomes when at least 90 
percent of applicable cases are substantially achieved; stakeholder 
interviews confirm that state plan and other program requirements are 
in place and functioning as described in the applicable regulations or 
statute; and performance on items with national standards meets the 
applicable threshold. The formula for calculating penalties is based in 
part on each state's allocation of federal child welfare funds from 
Titles IV-B and IV-E and the number of outcomes and systemic factors 
for which substantial conformity has not been achieved. 

[7] Resource centers may provide unlimited assistance to states by 
phone or e-mail, but on-site training and technical assistance are 
restricted to 10 days each year. States must first obtain regional 
office approval before on-site training and technical assistance can 
occur. ACF officials in both the regions and in headquarters indicated 
that they have been flexible with extensions beyond the 10-day cap. 

[8] California and Puerto Rico were determined to be in substantial 
conformity on 2 outcomes and systemic factors, while North Dakota 
achieved substantial conformity on 9. 

[9] These values are state-reported and reflect officials' estimates of 
costs associated with all CFSR-related activities except those incurred 
during PIP implementation. In reporting on their expenses, states were 
instructed to include the value of training, travel, infrastructure, 
technology, food, administrative supplies, and any other expenses 
associated with the CFSR process. States were also asked to provide 
supporting documentation for this particular question, but most states 
were unable to provide documentation. Many states reported that they 
did not track CFSR-related expenses. The 25 states that did provide 
estimates were in different phases of the CFSR.

[10] ACF provided preliminary data, based on an inquiry of its regional 
offices, showing that 25 states have charged some CFSR-related costs to 
Title IV-E.

[11] The number of FTEs participating in each phase of the CFSR is 
state-reported. While states were not given specific instructions for 
how to calculate FTEs, they were asked to report only on the phases of 
the CFSR that they had started or completed. Therefore, states' 
responses varied depending on the phase of the CFSR process they were 
in and the methods they used to calculate FTEs. 

[12] These values are state-reported and represent states' estimates 
regarding the number of stakeholders that participated in each phase of 
the CFSR process. In answering this question, states were provided with 
a list of stakeholders, such as advocates and court officials, to guide 
their estimates, but they were not asked to indicate the duration of 
stakeholder involvement during each phase. 

[13] See GAO-03-809. 

[14] ACF provides states with their statewide data about 6 months prior 
to the on-site review, during which time states are allowed to make 
corrections to the data and resubmit the updated data so it can be used 
when determining state conformity with CFSR measures. 

[15] According to our calculations--which assumed that the attribute of 
interest occurred in about 50 percent of the cases--a sample size of 50 
would produce percentage estimates with a 95 percent margin of error of 
approximately plus or minus 14 percentage points. This level of 
variability is a limitation when attempting to interpret estimates 
based on this sample size. 

[16] Because one of the two cases applicable to the adoption measure 
was assigned a rating of area needing improvement, 50 percent of the 
cases for this item were assigned a rating of area needing improvement. 
As a result, the item was given an overall rating of area needing 
improvement since both cases would have needed to be assigned a rating 
of strength for this item to meet the 85 percent threshold necessary to 
assign an overall rating of strength.

[17] An ACF statistician also confirmed that the CFSR sample is too 
small to generalize to the states' populations and that the three 
sites, from which cases are selected, also are not representative. 

[18] Virginia requested an additional case review to resolve a 
discrepancy between the statewide data and on-site review findings for 
the item measuring the state's performance on foster care re-entries. 
According to an ACF regional official, the state met the national 
standard for this item but the case review findings showed the state 
did not meet the threshold for this measure. At the time of 
publication, ACF and the state were still finalizing plans to conduct 
the additional case review, and until the review is completed, the 
state cannot receive its final report.

[19] The remaining 24 respondents reported that the emphasis was "just 
about right."

[20] At the time of our analysis, only Delaware and North Carolina had 
completed the 2-year term of their PIPs, and to date, ACF is still 
analyzing the states' progress and has not determined if there has been 
overall improvement or if it will apply financial penalties. 

[21] Although 41 states were developing or implementing PIPs at the 
time of publication, we reviewed the 31 available PIPs that ACF had 
approved as of January 1, 2004.

[22] The plaintiffs in this class action lawsuit filed in 1995 were 11 
children, all of whom were abused, sometimes severely. The plaintiffs 
sought to restructure New York City's child welfare system and charged 
that the city mishandled their cases and deprived them of their rights 
under the state and federal constitutions as well as various state and 
federal laws. Marisol A. v. Giuliani, 929 F. Supp. 662, 669 (S.D.N.Y. 
1996). As part of its settlement with the parties, the state agreed to 
monitor the city's staffing requirements and conduct audits of the 
city's treatment of children in nine specified areas. Marisol A. v. 
Giuliani, 185 F.R.D. 152, 159-160 (S.D.N.Y. 1999). 

[23] ACF could not provide a response rate.

[24] According to interviews with ACF regional staff, only one region 
is tracking states' satisfaction with the resource centers 
systematically. While some regions conduct ad hoc conversations with 
state officials after a resource center has provided assistance, ACF 
does not have a formal mechanism for collecting or analyzing states' 
views on the quality of assistance provided.

[25] Funding levels for each resource center have varied over their 5-
year cooperative agreements, but our analysis of 1999-2003 data shows 
that since the implementation of the CFSRs, the funding for the 
resource center for Information Technology and the resource center for 
Organizational Improvement--the two resource centers that provide 
specific support to states regarding data issues and PIP development--
has consistently and substantially increased. 

[26] Puerto Rico did not respond to our survey. 

[27] The 10 states participating in our phone follow-up survey were 
Arkansas, Iowa, Kansas, Mississippi, North Dakota, New Jersey, 
Pennsylvania, Rhode Island, Utah, and West Virginia.

GAO's Mission:

The General Accounting Office, the investigative arm of Congress, 
exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability 
of the federal government for the American people. GAO examines the use 
of public funds; evaluates federal programs and policies; and provides 
analyses, recommendations, and other assistance to help Congress make 
informed oversight, policy, and funding decisions. GAO's commitment to 
good government is reflected in its core values of accountability, 
integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony:

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains 
abstracts and full-text files of current reports and testimony and an 
expanding archive of older products. The Web site features a search 
engine to help you locate documents using key words and phrases. You 
can print these documents in their entirety, including charts and other 
graphics.

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as "Today's Reports," on its 
Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order 
GAO Products" heading.

Order by Mail or Phone:

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to:

U.S. General Accounting Office

441 G Street NW,

Room LM Washington,

D.C. 20548:

To order by Phone: 	

	Voice: (202) 512-6000:

	TDD: (202) 512-2537:

	Fax: (202) 512-6061:

To Report Fraud, Waste, and Abuse in Federal Programs:

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470:

Public Affairs:

Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S.

General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C.

20548: