Skip to Highlights
Highlights

Pursuant to a congressional request, GAO reviewed the approach the National Assessment Governing Board (NAGB) used to establish standards for student performance on the National Assessment of Educational Progress (NAEP) in mathematics, focusing on: (1) its strengths and weaknesses; (2) its suitability and that of alternative approaches for use with NAEP; and (3) NAGB capability to provide technically sound policy guidance to NAEP.

Skip to Recommendations

Recommendations

Matter for Congressional Consideration

Matter Status Comments
Congress should specify what it intends in directing NAGB to identify appropriate achievement goals: whether it envisions the establishment of overall performance standards, the establishment of content-based performance standards, or simply better alignment of test coverage with content mastery standards. Given that legislation to establish a mechanism for adopting national content standards is currently under consideration, Congress may also wish to express specific guidance with respect to activities to align NAEP to content standards before such a mechanism is in place.
Closed - Implemented
P.L. 103-382 replaces the term "achievement goals" with "performance levels" and sets forth criteria for determining whether such levels are technically sound.
Congress should clarify the division of responsibilities between NAGB and NCES, with a view toward concentrating NAGB efforts on the functions for which its broad representation is an asset and toward distinguishing functions NAGB itself is to implement from matters on which it is to give policy direction or advice to the Commissioner. While NAGB, as it is currently constituted, can appropriately advise the Commissioner from a constituency perspective regarding functions that are technical (such as the method and design of the assessment), it does not have the technical resources to carry out these functions and should be relieved of the apparent responsibility. When Congress has more clearly determined what NAGB's functions should be, it should review NAGB's membership and determine the number of technically trained members needed.
Closed - Implemented
P.L. 103-382 rephreases NAGB's responsibilities somewhat and directs it to consult with the technical experts, including the Advisory Council on Education Statistics, when dealing with technical issues. It also increases the representation of technical expertise on NAGB. These actions should lead to more effective coordination between NAGB and NCES.

Recommendations for Executive Action

Agency Affected Recommendation Status
National Assessment Governing Board In light of the many problems GAO found with the NAGB approach, NAGB should withdraw its direction to the National Center for Education Statistics (NCES) that the 1992 NAEP results be published primarily in terms of levels. The conventional approach to score interpretation should be retained until an alternative has been shown to be sound.
Closed - Implemented
NAGB has not followed this recommendation. However, methods and reporting practicies were changed in order to guard against misleading interpretations of the levels.
National Center for Education Statistics The Chairman of NAGB and the Commissioner of Education Statistics should develop a joint plan and schedule for a review of the NAGB achievement levels approach (its definitions of achievement, score selection procedures, and score interpretation), taking into account evaluations that are currently under way and providing for additional activities as needed. The plan should begin with a review of existing critiques of the approach and should include, at an early stage, a determination by the Commissioner whether: (1) the NAGB approach will necessarily produce invalid interpretation of NAEP scores and should not be pursued; or (2) the approach is sufficiently promising that a specific plan for preparing for NCES prepublication review should be designed and implemented. If option 1 is selected, the case is closed. If the decision is to proceed, NAGB should develop evidence that the levels results are valid and reliable and that the interpretations suggested for them are supported. NCES should make clear what evidence will be required.
Closed - Implemented
A conference on alternative methods was held in October 1994 and methods have been amended.
National Assessment Governing Board The Chairman of NAGB and the Commissioner of Education Statistics should develop a joint plan and schedule for a review of the NAGB achievement levels approach (its definitions of achievement, score selection procedures, and score interpretation), taking into account evaluations that are currently under way and providing for additional activities as needed. The plan should begin with a review of existing critiques of the approach and should include, at an early stage, a determination by the Commissioner whether: (1) the NAGB approach will necessarily produce invalid interpretation of NAEP scores and should not be pursued; or (2) the approach is sufficiently promising that a specific plan for preparing for NCES prepublication review should be designed and implemented. If option 1 is selected, the case is closed. If the decision is to proceed, NAGB should develop evidence that the levels results are valid and reliable and that the interpretations suggested for them are supported. NCES should make clear what evidence will be required.
Closed - Implemented
A conference to examine standard-setting methods was held in October 1994. Methods have been amended.
National Assessment Governing Board In view of the conceptual and technical flaws inherent in the NAGB achievement levels approach and of the many questions that need to be resolved before an alternative standard-setting method can be selected, NAGB should withdraw its policy of applying the 1990 achievement levels approach to future NAEP tests and join with NCES in exploring alternatives for setting both content-based and overall performance standards with respect to NAEP. This inquiry should examine issues of purpose, technical feasibility, cost, fairness, credibility, and usefulness.
Closed - Implemented
NAGB did not withdraw the policy, but has changed its approach.
National Assessment Governing Board To ensure that it does not formulate and adopt technically unsound policies or approve technically flowed results, NAGB should obtain NCES review of the technical strengths and weaknesses of proposed policies that implement NAGB's statutory responsibilities, prior to final decision on such policies.
Closed - Implemented
NAGB has instituted a formal procedure to obtain NCES review. This should help ensure that its policies are technically sound.
National Assessment Governing Board To ensure that it does not formulate and adopt technically unsound policies or approve technically flawed results, NAGB should analyze the probable effect of proposed policies (such as the achievement level policy) on NAEP ability to present achievement fairly and accurately and to support trend reporting that is both valid and reliable.
Closed - Implemented
NAGB's response stated that policies already in place provide for adequate monitoring. It proposed no additional action.
National Assessment Governing Board To ensure that it does not formulate and adopt technically unsound policies or approve technically flawed results, NAGB should pilot test and thoroughly evaluate any new design or analysis procedure before it is fully implemented and results are reported.
Closed - Implemented
1992 and 1994 procedures were pilot tested.
National Assessment Governing Board To ensure that it does not formulate and adopt technically unsound policies or approve technically flawed results, NAGB should adopt standards of technical quality (to be applied internally) for publications issued under its own authority and also secure competent external technical review of such publications prior to authorizing their release.
Closed - Implemented
In response to GAO's report, NAGB stated that it had no plan to issue publications that would require the adoption of technical standards but that if such plans arose, it would apply standards and reviews.
National Assessment Governing Board The Chairman of NAGB should review actions taken with respect to its statutory responsibility in the past 2 years, identify those whose technical consequences have not been sufficiently examined, and secure technical review as necessary to ensure that these actions will not generate unanticipated technical difficulties in the future.
Closed - Implemented
The review has been undertaken.
National Assessment Governing Board The Chairman of NAGB should review each proposed policy to ensure that NAGB prescribes policy ends, not technical details of implementation.
Closed - Implemented
NAGB stated that is policy will remain in force but does not propose specific actions.
National Assessment Governing Board With respect to NAGB membership, NAGB should nominate for the two testing and measurement positions only persons with relevant professional qualifications who are trained and experienced in the design and analysis of large-scale educational tests. To further add technical expertise within its currently mandated membership structure, NAGB should also ensure that two or more of its elected officials, educators, and representatives of the general public have significant technical knowledge and experience.
Closed - Implemented
NAGB reports that new appointees have been appropriately qualified.

Full Report