Education for Disadvantaged Children:
Research Purpose and Design Features Affect Conclusions Drawn From Key Studies
HEHS-00-168, Aug 31, 2000
Pursuant to a congressional request GAO compared the Department of Education's Longitudinal Evaluation of School Change and Performance Plan (LESCP) to its previous assessment program, the Prospect Study, focusing on: (1) the purpose of each study and how these purposes relate to the needs of policymakers and educators; (2) the process used to design and implement the studies, including obtaining feedback from review panels and releasing the results to Congress and the public ; and (3) the studies' strengths and limitations in light of their purposes and determine the effect these strengths and limitations have on the conclusions that can be drawn from the data.
GAO noted that: (1) while Education staff, contractors, and members of advisory panels all agreed that the purpose of the Prospects study was to assess the effectiveness of Title I of the Elementary and Secondary Education Act, considerably less agreement existed on the precise purpose of the LESCP study; (2) these differences centered primarily around how much emphasis the study woul dplace on the Title I program; (3) the lack of agreement about LESCP's purpose created unclear expectations for the study, making it difficult to predict the degree to which the final report will meet the needs of Congress, other policymakers, and educators; (4) the LESCP and Prospects studies used similar processes for contracting out data collection and analysis, obtaining comments from review panels, and releasing the results; (5) both studies were conducted by outside research organizations under contracts with Educations; (6) for both Prospects and LESCP, Education and the contractors obtained advice from two panels of experts--a congressionally mandated review panel that advised Education on policy issues and a seperate technical panel that provided feedback specific to the individual study; (7) the panels that advised the Prospects study had proportionally greater representation from educators, while LESCP's panels had proportionally more researchers; (8) for both the LESCO and Prospect studies, the 3 to 4 year longitudinal data collection and the complexity of the implementation process posed challenges for providing study results in time to meet Congress' 5-year reauthorization schedule; (9) to provide information for reauthorization, Education issued interim reports to Congress in both cases; (10) for both Prospects and LESCP, major design features influenced the study's ability to address its overall puspose; (11) the design of each study has both strengths and potential limitation; (12) the LESCP study is not yet complete, but the depth of information provided by its longitudinal focus is likely to be a key strength; (13) however, LESCP, which uses a smaller, nonrepresentative sample, suffers from design limitations that will restrict its ability to fully satisfy any of the three potential purposes envisioned by Education, contractors, and panel members; and (14) because several design limitations are directly related to measuring the effects of Title I and standards-based reform, they will likey have a major effect on the conclusions that could be drawn in these areas of education policy.