Educational Achievement Standards: NAGB's Approach Yields Misleading Interpretations

Highlights

Pursuant to a congressional request, GAO reviewed the approach the National Assessment Governing Board (NAGB) used to establish standards for student performance on the National Assessment of Educational Progress (NAEP) in mathematics, focusing on: (1) its strengths and weaknesses; (2) its suitability and that of alternative approaches for use with NAEP; and (3) NAGB capability to provide technically sound policy guidance to NAEP.

Recommendations

Matter for Congressional Consideration

Matter	Status	Comments
Congress should specify what it intends in directing NAGB to identify appropriate achievement goals: whether it envisions the establishment of overall performance standards, the establishment of content-based performance standards, or simply better alignment of test coverage with content mastery standards. Given that legislation to establish a mechanism for adopting national content standards is currently under consideration, Congress may also wish to express specific guidance with respect to activities to align NAEP to content standards before such a mechanism is in place.	Closed – Implemented	P.L. 103-382 replaces the term "achievement goals" with "performance levels" and sets forth criteria for determining whether such levels are technically sound.
Congress should clarify the division of responsibilities between NAGB and NCES, with a view toward concentrating NAGB efforts on the functions for which its broad representation is an asset and toward distinguishing functions NAGB itself is to implement from matters on which it is to give policy direction or advice to the Commissioner. While NAGB, as it is currently constituted, can appropriately advise the Commissioner from a constituency perspective regarding functions that are technical (such as the method and design of the assessment), it does not have the technical resources to carry out these functions and should be relieved of the apparent responsibility. When Congress has more clearly determined what NAGB's functions should be, it should review NAGB's membership and determine the number of technically trained members needed.	Closed – Implemented	P.L. 103-382 rephreases NAGB's responsibilities somewhat and directs it to consult with the technical experts, including the Advisory Council on Education Statistics, when dealing with technical issues. It also increases the representation of technical expertise on NAGB. These actions should lead to more effective coordination between NAGB and NCES.

Recommendations for Executive Action

Agency Affected Sort descending	Recommendation	Status
National Assessment Governing Board	In light of the many problems GAO found with the NAGB approach, NAGB should withdraw its direction to the National Center for Education Statistics (NCES) that the 1992 NAEP results be published primarily in terms of levels. The conventional approach to score interpretation should be retained until an alternative has been shown to be sound.	Closed – Implemented NAGB has not followed this recommendation. However, methods and reporting practicies were changed in order to guard against misleading interpretations of the levels.
National Assessment Governing Board	The Chairman of NAGB and the Commissioner of Education Statistics should develop a joint plan and schedule for a review of the NAGB achievement levels approach (its definitions of achievement, score selection procedures, and score interpretation), taking into account evaluations that are currently under way and providing for additional activities as needed. The plan should begin with a review of existing critiques of the approach and should include, at an early stage, a determination by the Commissioner whether: (1) the NAGB approach will necessarily produce invalid interpretation of NAEP scores and should not be pursued; or (2) the approach is sufficiently promising that a specific plan for preparing for NCES prepublication review should be designed and implemented. If option 1 is selected, the case is closed. If the decision is to proceed, NAGB should develop evidence that the levels results are valid and reliable and that the interpretations suggested for them are supported. NCES should make clear what evidence will be required.	Closed – Implemented A conference to examine standard-setting methods was held in October 1994. Methods have been amended.
National Assessment Governing Board	In view of the conceptual and technical flaws inherent in the NAGB achievement levels approach and of the many questions that need to be resolved before an alternative standard-setting method can be selected, NAGB should withdraw its policy of applying the 1990 achievement levels approach to future NAEP tests and join with NCES in exploring alternatives for setting both content-based and overall performance standards with respect to NAEP. This inquiry should examine issues of purpose, technical feasibility, cost, fairness, credibility, and usefulness.	Closed – Implemented NAGB did not withdraw the policy, but has changed its approach.
National Assessment Governing Board	To ensure that it does not formulate and adopt technically unsound policies or approve technically flowed results, NAGB should obtain NCES review of the technical strengths and weaknesses of proposed policies that implement NAGB's statutory responsibilities, prior to final decision on such policies.	Closed – Implemented NAGB has instituted a formal procedure to obtain NCES review. This should help ensure that its policies are technically sound.
National Assessment Governing Board	To ensure that it does not formulate and adopt technically unsound policies or approve technically flawed results, NAGB should analyze the probable effect of proposed policies (such as the achievement level policy) on NAEP ability to present achievement fairly and accurately and to support trend reporting that is both valid and reliable.	Closed – Implemented NAGB's response stated that policies already in place provide for adequate monitoring. It proposed no additional action.
National Assessment Governing Board	To ensure that it does not formulate and adopt technically unsound policies or approve technically flawed results, NAGB should pilot test and thoroughly evaluate any new design or analysis procedure before it is fully implemented and results are reported.	Closed – Implemented 1992 and 1994 procedures were pilot tested.
National Assessment Governing Board	To ensure that it does not formulate and adopt technically unsound policies or approve technically flawed results, NAGB should adopt standards of technical quality (to be applied internally) for publications issued under its own authority and also secure competent external technical review of such publications prior to authorizing their release.	Closed – Implemented In response to GAO's report, NAGB stated that it had no plan to issue publications that would require the adoption of technical standards but that if such plans arose, it would apply standards and reviews.
National Assessment Governing Board	The Chairman of NAGB should review actions taken with respect to its statutory responsibility in the past 2 years, identify those whose technical consequences have not been sufficiently examined, and secure technical review as necessary to ensure that these actions will not generate unanticipated technical difficulties in the future.	Closed – Implemented The review has been undertaken.
National Assessment Governing Board	The Chairman of NAGB should review each proposed policy to ensure that NAGB prescribes policy ends, not technical details of implementation.	Closed – Implemented NAGB stated that is policy will remain in force but does not propose specific actions.
National Assessment Governing Board	With respect to NAGB membership, NAGB should nominate for the two testing and measurement positions only persons with relevant professional qualifications who are trained and experienced in the design and analysis of large-scale educational tests. To further add technical expertise within its currently mandated membership structure, NAGB should also ensure that two or more of its elected officials, educators, and representatives of the general public have significant technical knowledge and experience.	Closed – Implemented NAGB reports that new appointees have been appropriately qualified.
National Center for Education Statistics	The Chairman of NAGB and the Commissioner of Education Statistics should develop a joint plan and schedule for a review of the NAGB achievement levels approach (its definitions of achievement, score selection procedures, and score interpretation), taking into account evaluations that are currently under way and providing for additional activities as needed. The plan should begin with a review of existing critiques of the approach and should include, at an early stage, a determination by the Commissioner whether: (1) the NAGB approach will necessarily produce invalid interpretation of NAEP scores and should not be pursued; or (2) the approach is sufficiently promising that a specific plan for preparing for NCES prepublication review should be designed and implemented. If option 1 is selected, the case is closed. If the decision is to proceed, NAGB should develop evidence that the levels results are valid and reliable and that the interpretations suggested for them are supported. NCES should make clear what evidence will be required.	Closed – Implemented A conference on alternative methods was held in October 1994 and methods have been amended.

See All 11 Recommendations

Full Report

Full Report (120 pages)

Office of Public Affairs

Chuck Young

Managing Director

youngc1@gao.gov

(202) 512-4800

Topics

Education

Academic achievementData collectionEducation program evaluationEducational researchEducational standardsEducational testingElementary school studentsMathematicsSecondary school studentsStatistical data