Skip to main content

Head Start: Further Development Could Allow Results of New Test to Be Used for Decision Making

GAO-05-343 Published: May 17, 2005. Publicly Released: May 17, 2005.
Jump To:
Skip to Highlights

Highlights

In September 2003, the Head Start Bureau, in the Department of Health and Human Services (HHS) Administration for Children and Families (ACF), implemented the National Reporting System (NRS), the first nationwide skills test of over 400,000 4- and 5-year-old children. The NRS is intended to provide information on how well Head Start grantees are helping children progress. Given the importance of the NRS, this report examines: what information the NRS is designed to provide; how the Head Start Bureau has responded to concerns raised by grantees and experts during the first year of implementation; and whether the NRS provides the Head Start Bureau with quality information.

Recommendations

Recommendations for Executive Action

Agency Affected Recommendation Status
Department of Health and Human Services To help ensure that the NRS successfully and efficiently achieves its purposes, the HHS Assistant Secretary for ACF should take steps to better monitor some aspects of NRS implementation and examine means of improving its efficiency, including monitoring the effects of the NRS on local Head Start instructional practices.
Closed – Implemented
As part of the National Reporting System (NRS) Quality Assurance effort, an independent contractor is collecting information from a national sample of 36 Head Start programs on how well they are implementing the NRS and what changes they seem to be making in curriculum and instructional practices in response to NRS requirements. Observed changes in instructional activities are documented in the periodic reports of the Quality Assurance contractor, which are carefully reviewed by Administration for Children and Families staff. Feedback on how programs can use the NRS to stimulate constructive changes has been and will be provided to local programs through the memoranda that accompany each report on NRS results and in nationwide satellite television broadcasts that focus on NRS data collection.
Department of Health and Human Services To help ensure that the NRS successfully and efficiently achieves its purposes, the HHS Assistant Secretary for ACF should take steps to better monitor some aspects of NRS implementation and examine means of improving its efficiency, including improving the management and accuracy of its data on the number of children eligible for and participating in the NRS.
Closed – Implemented
The Office of Head Start has requested that a National Reporting System contractor continue tracking major discrepancies between the number of children actually assessed and the number presumably eligible for assessment per the Program Information Report (PIR) and other sources of data.
Department of Health and Human Services To help ensure that the NRS successfully and efficiently achieves its purposes, the HHS Assistant Secretary for ACF should take steps to better monitor some aspects of NRS implementation and examine means of improving its efficiency, including working with the Technical Work Group to determine the feasibility of sampling options for administering the NRS, including documentation of their costs and benefits.
Closed – Implemented
The HHS contractor that developed the NRS prepared information regarding the costs and benefits of sampling options for review by HHS' Advisory Committee on Head Start Accountability and Educational Performance Measures. The Advisory Committee completed its work in February 2007, including a recommendation that HHS integrate the National Reporting System (NRS) into local assessment approaches. Congress has subsequently passed bills reauthorizing Head Start (H.R. 1429 and S. 556) that would suspend the NRS as currently implemented and ask the National Academy of Sciences to form a committee addressing developmental outcomes and assessments for young children.
Department of Health and Human Services The Assistant Secretary for ACF should reduce uncertainty about the appropriate uses of the NRS by determining how the NRS data will be used for the purposes of accountability and targeting training and technical assistance, and clearly communicate this information to grantees.
Closed – Implemented
In February 2007, the Secretary's Advisory Committee on Head Start Accountability and Performance Measures completed its work and recommended, in part, that HHS make clear to Head Start teachers the link between NRS results and technical assistance goals and that HHS determine appropriate gains and outcome levels so programs could meet their intended goals. HHS expressed the intention to consider these recommendations as it continued to improve the utility of the National Reporting System. Separately, P.L. 109-149 set aside funds for establishing a panel of independent experts and directed the Office of Head Start to sponsor a study by the National Academy of Sciences (NAS) to review and provide guidance on appropriate outcomes and assessments for young children. In August 2008, NAS issued its guidance in "Early Childhood Assessment: Why, What, and How?"
Department of Health and Human Services The Assistant Secretary for ACF should reduce uncertainty about the appropriate uses of the NRS by using the first year of NRS results to conduct further study to ensure that the results are reliable and valid for both the English and Spanish versions and that the results are appropriate for the intended purposes.
Closed – Implemented
An extensive series of analyses was conducted with the first year fall and spring data from the national implementation of the National Reporting System (NRS), as well as parallel data from the second year of national implementation. Data from the first year of implementation of the NRS suggest that the NRS measures have strong internal consistency reliability. Further, data from the original field test of the NRS suggest that these data also have strong inter-rater reliability and strong test-retest reliability. Taken together, these findings suggest that the NRS instruments are reliable. Similarly high reliability has been found for both the English and Spanish language versions of the NRS. These findings are currently being compiled in annual technical reports for the NRS that will be made available to the public upon completion. Further, a report on the predictive validity of the NRS, which is examined using FACES data, is currently being prepared. Preliminary results suggest that the NRS has strong predictive validity. Finally, a quality assurance study, which has been conducted every year since the implementation of the NRS, has shown high levels of fidelity in the implementation of the NRS direct child assessment. The findings of this report have been made available to grantees through the Computer-Based Reporting System.
Department of Health and Human Services The Assistant Secretary for ACF should reduce uncertainty about the appropriate uses of the NRS by compiling detailed technical information on the NRS, including appropriate uses, in a single, well-organized document and make this information publicly available.
Closed – Implemented
By the end of FY06, the contractor had prepared a technical report for the government's review that detailed both the process used to create the NRS, as well as findings from the NRS field test. As of August 2007, HHS officials indicated that the technical manual prepared by HHS' contractor was still in clearance. In 2008, HHS reported that the manual is publicly available upon request. Separately, P.L. 109-149 set aside funds for establishing a panel of independent experts and directed the Office of Head Start to sponsor a study by the National Academy of Sciences (NAS) to review and provide guidance on appropriate outcomes and assessments for young children. In August 2008, NAS issued its guidance in "Early Childhood Assessment: Why, What, and How?"

Full Report

Office of Public Affairs

Topics

Aid for educationData collectionData integrityEducation program evaluationEducational standardsEducational testingFederal aid programsPreschool educationReporting requirementsSurveys