Educational Achievement Standards:

NAGB's Approach Yields Misleading Interpretations

PEMD-93-12: Published: Jun 23, 1993. Publicly Released: Jun 23, 1993.

Additional Materials:

Contact:

Robert L. York
(202) 512-5885
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

Pursuant to a congressional request, GAO reviewed the approach the National Assessment Governing Board (NAGB) used to establish standards for student performance on the National Assessment of Educational Progress (NAEP) in mathematics, focusing on: (1) its strengths and weaknesses; (2) its suitability and that of alternative approaches for use with NAEP; and (3) NAGB capability to provide technically sound policy guidance to NAEP.

GAO found that: (1) NAGB based its approach on a well-known standard-setting method but modified the method in untested ways; (2) the 1990 standard-setting approach NAGB used was unusual because achievement levels were to reflect mastery of different material, not differences in overall performance, panelists did not use consensus-based standards, and panelists were not assisted in judging students that met expectations for lower levels; (3) the NAEP scale can be used to express standards for overall performance on grade-level materials; (4) because of the way NAEP is designed, the current NAEP scale is not a good way to measure students' knowledge in specific areas; (5) if the measurement of knowledge of course content is desired, new tests will need to be developed; (6) NAGB designed and implemented its achievement levels measurement approach without adequate technical information; and (7) NAGB knowledge, resources, and procedures do not provide reasonable assurance that work done at its direction will be technically sound.

Matters for Congressional Consideration

  1. Status: Closed - Implemented

    Comments: P.L. 103-382 replaces the term "achievement goals" with "performance levels" and sets forth criteria for determining whether such levels are technically sound.

    Matter: Congress should specify what it intends in directing NAGB to identify appropriate achievement goals: whether it envisions the establishment of overall performance standards, the establishment of content-based performance standards, or simply better alignment of test coverage with content mastery standards. Given that legislation to establish a mechanism for adopting national content standards is currently under consideration, Congress may also wish to express specific guidance with respect to activities to align NAEP to content standards before such a mechanism is in place.

  2. Status: Closed - Implemented

    Comments: P.L. 103-382 rephreases NAGB's responsibilities somewhat and directs it to consult with the technical experts, including the Advisory Council on Education Statistics, when dealing with technical issues. It also increases the representation of technical expertise on NAGB. These actions should lead to more effective coordination between NAGB and NCES.

    Matter: Congress should clarify the division of responsibilities between NAGB and NCES, with a view toward concentrating NAGB efforts on the functions for which its broad representation is an asset and toward distinguishing functions NAGB itself is to implement from matters on which it is to give policy direction or advice to the Commissioner. While NAGB, as it is currently constituted, can appropriately advise the Commissioner from a constituency perspective regarding functions that are technical (such as the method and design of the assessment), it does not have the technical resources to carry out these functions and should be relieved of the apparent responsibility. When Congress has more clearly determined what NAGB's functions should be, it should review NAGB's membership and determine the number of technically trained members needed.

Recommendations for Executive Action

  1. Status: Closed - Implemented

    Comments: NAGB has not followed this recommendation. However, methods and reporting practicies were changed in order to guard against misleading interpretations of the levels.

    Recommendation: In light of the many problems GAO found with the NAGB approach, NAGB should withdraw its direction to the National Center for Education Statistics (NCES) that the 1992 NAEP results be published primarily in terms of levels. The conventional approach to score interpretation should be retained until an alternative has been shown to be sound.

    Agency Affected: National Assessment Governing Board

  2. Status: Closed - Implemented

    Comments: A conference on alternative methods was held in October 1994 and methods have been amended.

    Recommendation: The Chairman of NAGB and the Commissioner of Education Statistics should develop a joint plan and schedule for a review of the NAGB achievement levels approach (its definitions of achievement, score selection procedures, and score interpretation), taking into account evaluations that are currently under way and providing for additional activities as needed. The plan should begin with a review of existing critiques of the approach and should include, at an early stage, a determination by the Commissioner whether: (1) the NAGB approach will necessarily produce invalid interpretation of NAEP scores and should not be pursued; or (2) the approach is sufficiently promising that a specific plan for preparing for NCES prepublication review should be designed and implemented. If option 1 is selected, the case is closed. If the decision is to proceed, NAGB should develop evidence that the levels results are valid and reliable and that the interpretations suggested for them are supported. NCES should make clear what evidence will be required.

    Agency Affected: Department of Education: National Center for Education Statistics

  3. Status: Closed - Implemented

    Comments: A conference to examine standard-setting methods was held in October 1994. Methods have been amended.

    Recommendation: The Chairman of NAGB and the Commissioner of Education Statistics should develop a joint plan and schedule for a review of the NAGB achievement levels approach (its definitions of achievement, score selection procedures, and score interpretation), taking into account evaluations that are currently under way and providing for additional activities as needed. The plan should begin with a review of existing critiques of the approach and should include, at an early stage, a determination by the Commissioner whether: (1) the NAGB approach will necessarily produce invalid interpretation of NAEP scores and should not be pursued; or (2) the approach is sufficiently promising that a specific plan for preparing for NCES prepublication review should be designed and implemented. If option 1 is selected, the case is closed. If the decision is to proceed, NAGB should develop evidence that the levels results are valid and reliable and that the interpretations suggested for them are supported. NCES should make clear what evidence will be required.

    Agency Affected: National Assessment Governing Board

  4. Status: Closed - Implemented

    Comments: NAGB did not withdraw the policy, but has changed its approach.

    Recommendation: In view of the conceptual and technical flaws inherent in the NAGB achievement levels approach and of the many questions that need to be resolved before an alternative standard-setting method can be selected, NAGB should withdraw its policy of applying the 1990 achievement levels approach to future NAEP tests and join with NCES in exploring alternatives for setting both content-based and overall performance standards with respect to NAEP. This inquiry should examine issues of purpose, technical feasibility, cost, fairness, credibility, and usefulness.

    Agency Affected: National Assessment Governing Board

  5. Status: Closed - Implemented

    Comments: NAGB has instituted a formal procedure to obtain NCES review. This should help ensure that its policies are technically sound.

    Recommendation: To ensure that it does not formulate and adopt technically unsound policies or approve technically flowed results, NAGB should obtain NCES review of the technical strengths and weaknesses of proposed policies that implement NAGB's statutory responsibilities, prior to final decision on such policies.

    Agency Affected: National Assessment Governing Board

  6. Status: Closed - Implemented

    Comments: NAGB's response stated that policies already in place provide for adequate monitoring. It proposed no additional action.

    Recommendation: To ensure that it does not formulate and adopt technically unsound policies or approve technically flawed results, NAGB should analyze the probable effect of proposed policies (such as the achievement level policy) on NAEP ability to present achievement fairly and accurately and to support trend reporting that is both valid and reliable.

    Agency Affected: National Assessment Governing Board

  7. Status: Closed - Implemented

    Comments: 1992 and 1994 procedures were pilot tested.

    Recommendation: To ensure that it does not formulate and adopt technically unsound policies or approve technically flawed results, NAGB should pilot test and thoroughly evaluate any new design or analysis procedure before it is fully implemented and results are reported.

    Agency Affected: National Assessment Governing Board

  8. Status: Closed - Implemented

    Comments: In response to GAO's report, NAGB stated that it had no plan to issue publications that would require the adoption of technical standards but that if such plans arose, it would apply standards and reviews.

    Recommendation: To ensure that it does not formulate and adopt technically unsound policies or approve technically flawed results, NAGB should adopt standards of technical quality (to be applied internally) for publications issued under its own authority and also secure competent external technical review of such publications prior to authorizing their release.

    Agency Affected: National Assessment Governing Board

  9. Status: Closed - Implemented

    Comments: The review has been undertaken.

    Recommendation: The Chairman of NAGB should review actions taken with respect to its statutory responsibility in the past 2 years, identify those whose technical consequences have not been sufficiently examined, and secure technical review as necessary to ensure that these actions will not generate unanticipated technical difficulties in the future.

    Agency Affected: National Assessment Governing Board

  10. Status: Closed - Implemented

    Comments: NAGB stated that is policy will remain in force but does not propose specific actions.

    Recommendation: The Chairman of NAGB should review each proposed policy to ensure that NAGB prescribes policy ends, not technical details of implementation.

    Agency Affected: National Assessment Governing Board

  11. Status: Closed - Implemented

    Comments: NAGB reports that new appointees have been appropriately qualified.

    Recommendation: With respect to NAGB membership, NAGB should nominate for the two testing and measurement positions only persons with relevant professional qualifications who are trained and experienced in the design and analysis of large-scale educational tests. To further add technical expertise within its currently mandated membership structure, NAGB should also ensure that two or more of its elected officials, educators, and representatives of the general public have significant technical knowledge and experience.

    Agency Affected: National Assessment Governing Board

 

Explore the full database of GAO's Open Recommendations »

Sep 12, 2016

Aug 8, 2016

Jun 15, 2016

May 19, 2016

May 17, 2016

Apr 11, 2016

Mar 16, 2016

Feb 8, 2016

Dec 14, 2015

Nov 30, 2015

Looking for more? Browse all our products here