Elections:

Federal Programs for Accrediting Laboratories That Test Voting Systems Need to Be Better Defined and Implemented

GAO-08-770: Published: Sep 9, 2008. Publicly Released: Sep 9, 2008.

Additional Materials:

Contact:

Randolph C. Hite
(202) 512-6256
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

The 2002 Help America Vote Act (HAVA) created the Election Assistance Commission (EAC) and assigned both it and the National Institute of Standards and Technology (NIST) responsibilities for accrediting laboratories that test voting systems. NIST assesses a laboratory's technical qualifications and makes recommendations to EAC, which makes a final accreditation decision. In view of the continuing concerns about voting systems and the important roles that NIST and EAC play in accrediting the laboratories that test these systems, GAO was asked to determine whether each organization has defined an effective approach for accrediting laboratories that test voting systems and whether each is following its defined approach. To accomplish this, GAO compared NIST and EAC policies, guidelines, and procedures against applicable legislation and guidance, and reviewed both agencies' efforts to implement them.

NIST has largely defined and implemented an approach for accrediting voting system testing laboratories that incorporates many aspects of an effective program. In particular, its approach addresses relevant HAVA requirements and reflects relevant laboratory accreditation guidance, including standards accepted by the international standards community. However, NIST's defined approach does not, for example, cite explicit qualifications for the persons who conduct accreditation technical assessments, as called for in federal accreditation program guidance. Instead, NIST officials said that they rely on individuals who have prior experience in reviewing such laboratories. Further, even though the EAC requires that laboratory accreditation be based on demonstrated capabilities to test against the latest voting system standards, NIST's defined approach has not always cited these current standards. As a result, two of the four laboratories accredited to date were assessed using assessment tools that were not linked to the latest standards. Moreover, available documentation for the four laboratory assessments was not sufficient to determine how the checklists were applied and how decisions were reached. According to NIST officials, the four laboratories were consistently assessed. Moreover, they said that they intend to evolve NIST's accreditation approach to, for example, clearly provide for sufficient documentation of how accreditation reviews are conducted and decisions are reached. However, they had yet to develop specific plans for accomplishing this. EAC recently developed a draft laboratory accreditation program manual, but this draft manual does not adequately define all aspects of an effective approach, and it was not used in the four laboratory accreditations performed to date. Specifically, while this draft manual addresses relevant HAVA requirements, such as the requirement for the commissioners to vote on the accreditation of any laboratory that NIST recommends for accreditation, it does not include a methodology governing how laboratories are to be evaluated or criteria for granting accreditation. Because the manual was not approved at the time EAC accredited four laboratories, these accreditations were governed by a more broadly defined accreditation review process that was described in correspondence sent to each laboratory and a related document receipt checklist. As a result, these accreditations were based on review steps that were not sufficiently defined to permit them to be executed in a repeatable manner. According to EAC officials, including the official who conducted the accreditation reviews for the four laboratories, using the same person to conduct the reviews ensured that the steps performed on the first laboratory were repeated on the other three. However, given that both the steps and the results were not documented, GAO could not verify this. EAC officials stated that they intend to evolve the program manual over time and apply it to future accreditations and reaccreditations. However, they did not have specific plans for accomplishing this. Further, although EAC very recently approved an initial version of its program manual, this did not occur until after EAC provided comments, and GAO had finalized, this report.

Status Legend:

More Info
  • Review Pending-GAO has not yet assessed implementation status.
  • Open-Actions to satisfy the intent of the recommendation have not been taken or are being planned, or actions that partially satisfy the intent of the recommendation have been taken.
  • Closed-implemented-Actions that satisfy the intent of the recommendation have been taken.
  • Closed-not implemented-While the intent of the recommendation has not been satisfied, time or circumstances have rendered the recommendation invalid.
    • Review Pending
    • Open
    • Closed - implemented
    • Closed - not implemented

    Recommendations for Executive Action

    Recommendation: To help NIST in evolving its Voting System Testing Laboratory (VSTL) accreditation program, the Director of NIST should ensure that the accreditation program manager develops and executes plans that specify tasks, milestones, resources, and performance measures that provide for establishing and implementing transparent requirements for the technical qualifications and training of accreditation assessors.

    Agency Affected: Department of Commerce: National Institute of Standards and Technology

    Status: Closed - Implemented

    Comments: NIST has established and implemented requirements for the technical qualifications and training of accreditation Lead and Technical Assessors. Standardized selection criteria for the positions of Lead Assessor and Technical Assessor are captured in worksheets for each of the two positions. Each worksheet identifies qualification factors (such as education, work experience, and past assessment criteria), minimum requirements for each factor, how each factor was evaluated (e.g. documentation, interview, peer reference, etc.), and whether the support provided for each factor was acceptable. Each type of worksheet also allows for program-specific criteria. For example, the VSTL Technical Assessor criteria requires 4 years of testing voting systems, including at least 2 years of testing to Election Assistance Commission-approved standards and guidelines. With respect to assessor training, NIST has established standardized training requirements and documented these in an Assessor Training Checklist. This checklist specifies training topics, associated training resources, which positions are required to receive the training, and the method of training delivery. The checklist also supports implementation of the training requirements by a mandatory record of the date training was completed, and certification by the program manager on completion of training. In September 2009, NIST held accreditation assessor training on VSTL-related materials and completed a documented checklist for each of the two individuals acting as assessors for the program.

    Recommendation: To help NIST in evolving its VSTL accreditation program, the Director of NIST should ensure that the accreditation program manager develops and executes plans that specify tasks, milestones, resources, and performance measures that provide for ensuring that each laboratory accreditation review is fully and consistently documented in accordance with NIST program requirements.

    Agency Affected: Department of Commerce: National Institute of Standards and Technology

    Status: Closed - Implemented

    Comments: NIST is now using a standardized checklist to ensure that its laboratory accreditation reviews are fully and consistently documented. The NIST Handbook 150-22 checklist for its VTSL accreditation review program is to be used in conjunction with any review of the general accreditation criteria prescribed in the NIST Handbook 150-22, NVLAP Voting System Testing. The checklist provides a structured method for evaluating criteria found in the pertinent sections of Handbook 150-22. The 2007 version of this checklist was used in the re-assessment of one previously accredited laboratory in 2007 and two previously accredited laboratories 2008. A 2008 version of the checklist was used in 2009 to re-assess a fourth laboratory, also previously accredited. The laboratory assessed in 2007 no longer participates in the VSTL program. According to the VSTL program manager, the two laboratories assessed in 2008 are to be assessed again by the end of 2010. The laboratory assessed in 2009 is due for re-assessment during fiscal year 2011.

    Recommendation: To help EAC in evolving its VSTL accreditation program, the Chair of the EAC should ensure that the EAC Executive Director develops and executes plans that specify tasks, milestones, resources, and performance measures that provide for establishing and implementing practices for the VSTL accreditation program consistent with accreditation program management guidance published by NIST and GAO, including (1) documentation of specific accreditation steps and criteria to guide assessors in conducting each laboratory review; (2) transparent requirements for the qualifications of accreditation reviewers; (3) requirements for the adequate maintenance of records related to the VSTL accreditation program; and (4) requirements for determining laboratory financial stability.

    Agency Affected: Election Assistance Commission

    Status: Closed - Implemented

    Comments: EAC has taken several steps to implement practices consistent with accreditation program management guidance. First, EAC has developed a standardized checklist that is used in the biannual policy and procedures review of accredited laboratories. The checklist provides for detailed steps and criteria to be used in conducting an accreditation review, and it has thus far been used in the review of two laboratories, and according to the EAC Director of Voting System Testing and Certification, the checklist will be used in two other laboratory reviews that are scheduled for late 2011. Second, EAC has identified formal training requirements for its laboratory accreditation personnel, and it plans to send two key accreditation review personnel to this training in October 2010 and two other accreditation staff members to the same training in the spring of 2011. According to the testing and certification director, new accreditation staff will also be required to attend the training in an effort to ensure that the qualification of all accreditation reviewers. Third, the EAC Voting System Test Laboratory Accreditation Program Manual now specifies EAC laboratory retention requirements for records associated with the accreditation activities. According to the manual, the EAC is to retain all accreditation-related records and laboratories are to retain accreditation records related to the testing of a voting system for a minimum of five years. Laboratory compliance with these record-retention requirements is assessed through the EAC's assessment checklist. Finally, the EAC's accreditation program manual now includes requirements for a laboratory to demonstrate financial stability by producing an audited financial statement showing assets that are greater than liabilities.

    Apr 10, 2014

    Apr 8, 2014

    Apr 3, 2014

    Mar 31, 2014

    Mar 26, 2014

    Mar 25, 2014

    Looking for more? Browse all our products here