DHS Science and Technology:

Additional Steps Needed to Ensure Test and Evaluation Requirements Are Met

GAO-11-596: Published: Jun 15, 2011. Publicly Released: Jul 14, 2011.

Additional Materials:

Contact:

David C. Maurer
(202) 512-3000
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

In recent years, GAO has reported on challenges the Department of Homeland Security (DHS) has faced in effectively managing major acquisitions, including programs which were deployed before appropriate testing and evaluation (T&E) was completed. In 2009 and 2010 respectively, DHS issued new T&E and acquisition directives to address these challenges. Under these directives, DHS Science and Technology Directorate's (S&T) Test & Evaluation and Standards Office (TES) is responsible for overseeing T&E of DHS major acquisition programs--that is, those with over $300 million in life-cycle costs--to ensure that T&E and certain acquisitions requirements are met. GAO was asked to identify (1) the extent to which TES oversees T&E of major acquisitions; and (2) what challenges, if any, TES officials report facing in overseeing T&E across DHS components. GAO reviewed DHS directives and test plans, interviewed DHS officials, and reviewed T&E documentation from a sample of 11 major acquisition programs from each of 11 different DHS components. The results of the sample cannot be generalized to all DHS programs, but provided insights.

TES met some of its oversight requirements for T&E of acquisition programs GAO reviewed, but additional steps are needed to ensure that all requirements are met. Specifically, since DHS issued the T&E directive in May 2009, TES has reviewed or approved T&E documents and plans for programs undergoing testing, and conducted independent assessments for the programs that completed operational testing during this time period. TES officials told GAO that they also provided input and reviewed other T&E documentation, such as components' documents describing the programs' performance requirements, as required by the T&E directive. DHS senior level officials considered TES's T&E assessments and input in deciding whether programs were ready to proceed to the next acquisition phase. However, TES did not consistently document its review and approval of components' test agents--a government entity or independent contractor carrying out independent operational testing for a major acquisition--or document its review of other component acquisition documents, such as those establishing programs' operational requirements, as required by the T&E directive. For example, 8 of the 11 acquisition programs GAO reviewed had hired test agents, but documentation of TES approval of these agents existed for only 3 of these 8 programs. Approving test agents is important to ensure that they are independent of the program and that they meet requirements of the T&E directive. TES officials agreed that they did not have a mechanism in place requiring a consistent method for documenting their review or approval and the extent to which the review or approval criteria were met. Without mechanisms in place for documenting its review or approval of acquisition documents and T&E requirements, such as approving test agents, it is difficult for DHS or a third party to review and validate TES's decision-making process and ensure that it is overseeing components' T&E efforts in accordance with acquisition and T&E directives and internal control standards for government entities. TES and DHS component officials stated that they face challenges in overseeing T&E across DHS components which fell into 4 categories: (1) ensuring that a program's operational requirements--the key performance requirements that must be met for a program to achieve its intended goals-- can be effectively tested; (2) working with DHS component program staff who have limited T&E expertise and experience; (3) using existing T&E directives and guidance to oversee complex information technology acquisitions; and (4) ensuring that components allow sufficient time for T&E while remaining within program cost and schedule estimates. Both TES and DHS, more broadly, have begun initiatives to address some of these challenges, such as establishing a T&E council to disseminate best practices to component program managers, and developing specific guidance for testing and evaluating information technology acquisitions. In addition, S&T has reorganized to assist components in developing requirements that can be tested, among other things. However, since these efforts have only recently been initiated to address these DHS-wide challenges, it is too soon to determine their effectiveness. GAO recommends, among other things, that S&T develop mechanisms for TES to document its review or approval of component acquisition documentation and T&E requirements, such as approving operational test agents. DHS agreed with GAO's recommendations.

Status Legend:

More Info
  • Review Pending-GAO has not yet assessed implementation status.
  • Open-Actions to satisfy the intent of the recommendation have not been taken or are being planned, or actions that partially satisfy the intent of the recommendation have been taken.
  • Closed-implemented-Actions that satisfy the intent of the recommendation have been taken.
  • Closed-not implemented-While the intent of the recommendation has not been satisfied, time or circumstances have rendered the recommendation invalid.
    • Review Pending
    • Open
    • Closed - implemented
    • Closed - not implemented

    Recommendations for Executive Action

    Recommendation: To better ensure that testing and evaluation requirements are met, the Secretary of Homeland Security should direct the Under Secretary for Science & Technology to develop a mechanism to ensure that TES documents its approval of operational test agents and the extent that the test agents meet the requirements in the T&E directive, and criteria that TES use in reviewing these test agents for major acquisition programs.

    Agency Affected: Department of Homeland Security: Directorate of Science and Technology

    Status: Closed - Implemented

    Comments: We found that the Department of Homeland Security's (DHS) Science and Technology Directorate (S&T) Test & Evaluation and Standards Office (TES) did not consistently document its review and approval of DHS components' test agents--government entities or independent contractors carrying out independent operational testing for major acquisitions--which would help ensure that the test agents are independent of the program and that they meet the requirements in DHS's Test and Evaluation (T&E) directive. TES officials agreed that they did not have a mechanism in place requiring a consistent method for documenting their approval and the extent to which approval criteria were met. As a result, we recommended that S&T develop a mechanism to ensure that TES documents its approval of test agents and the extent that the test agents meet the requirements in the T&E directive, and criteria that TES use in reviewing these test agents for major acquisition programs. In August 2011, the TES Director issued a T&E procedure which requires TES test area managers to use a standard checklist with specific criteria to be used during the test agent approval process and retain supporting documentation. TES provided a spreadsheet that is used to track its review and approval of test agents and examples of completed checklists and supporting documentation, such as approval memoranda. The new procedure and evidence provided of its implementation is consistent with our recommendation and the recommendation is closed as implemented.

    Recommendation: To better ensure that testing and evaluation requirements are met, the Secretary of Homeland Security should direct the Under Secretary for Science & Technology to develop a mechanism to ensure that TES documents its required review of component acquisition documents, including the mission need statements, concept of operations, operational requirements documents, developmental test reports, test plans, and other documentation required by the T&E directive, the extent that these documents meet the requirements in the T&E directive, and criteria that TES uses in reviewing these documents.

    Agency Affected: Department of Homeland Security: Directorate of Science and Technology

    Status: Closed - Implemented

    Comments: We found that the Test & Evaluation and Standards Office (TES) did not consistently document its review of Department of Homeland Security (DHS) components' acquisition documents, including mission need statements, operational requirements documents, concept of operations, and developmental test reports, as required by the Test and Evaluation (T&E) directive. During our review, TES officials stated that they do not have a mechanism to document or track documents that they did review, what criteria they used when reviewing the documents, and the extent to which the documents reviewed met those criteria, and the officials recognized that such a mechanism would be beneficial. As a result, we recommended that S&T develop a mechanism to ensure that TES documents its required review of component acquisition documents, the extent that these documents meet the requirements in the T&E directive, and criteria that TES uses in reviewing these documents. In April 2011, the TES Director issued a procedure for TES test area managers to follow when reviewing acquisition documents, such as acquisition program baselines, operational requirements documents, and concept of operations. TES provided a spreadsheet that is used to track its review and examples of TES's review of acquisition program documentation. The procedure and evidence provided of its implementation is consistent with our recommendation.

    Recommendation: To ensure that the Acquisition Review Board (ARB) is provided with an independent assessment of the operational test results of the Advanced Spectroscopic Portal program (ASP) to help determine whether the program should be approved for purchase and implementation, the Secretary of Homeland Security should arrange for an independent assessment, as required by the T&E directive, of ASP's operational test results, to include an assessment of the adequacy of the operational test and a concurrence or nonconcurrence on the operational test agent's evaluation of operational suitability and operational effectiveness.

    Agency Affected: Department of Homeland Security: Directorate of Science and Technology

    Status: Closed - Not Implemented

    Comments: We found that one of the programs we reviewed, the Advanced Spectroscopic Portal program (ASP), would not have an independent assessment of its operational test results, a key Test and Evaluation (T&E) oversight requirement in the T&E directive, due to a Department of Homeland Security (DHS) decision made before the T&E directive became effective. Test and Evaluation and Standards office (TES) officials said that in this one-time situation, the ASP program would not have an independent assessment of ASP test results, such as approving the operational test plan or writing a letter of assessment on the final results of operational testing, because TES was the test agent for ASP and thus, not in a position to independently assess the results of testing that they conducted. We recommended that DHS arrange for an independent assessment, as required by the T&E directive, of ASP's operational test results, to include an assessment of the adequacy of the operational test and a concurrence or nonconcurrence on the operational test agent's evaluation of operational suitability and operational effectiveness. In June 2011, DHS announced its cancellation of the ASP program, so this recommendation is no longer applicable and is closed.

    Jul 23, 2014

    Jun 9, 2014

    May 20, 2014

    • science icon, source: National Cancer Institute

      Nanomanufacturing and U.S. Competitiveness:

      Challenges and Opportunities
      GAO-14-618T: Published: May 20, 2014. Publicly Released: May 20, 2014.

    Feb 7, 2014

    Dec 20, 2013

    Nov 4, 2013

    • science icon, source: National Cancer Institute

      Small Business Innovation Research:

      Data Rights Protections
      GAO-14-116R: Published: Nov 4, 2013. Publicly Released: Nov 4, 2013.

    Jul 19, 2013

    Apr 10, 2013

    Feb 15, 2013

    Jun 20, 2012

    Looking for more? Browse all our products here