Fast Facts

The Department of Homeland Security invests billions of dollars each year in new technologies such as facial recognition software, X-ray equipment, surveillance aircraft, and U.S. Coast Guard ships.

We looked at how DHS tests and evaluates technologies to ensure they work before deploying them. The agency has strengthened its policies, guidance, and training to encourage testing and evaluation earlier in the acquisition process.

We recommended that DHS assess whether it has enough workers with the right skills to adequately oversee test and evaluation activities.

DHS tests and evaluates new technologies like facial recognition software to verify the identities of travelers leaving the country.

Woman looking into a facial recognition screen at an airport

Woman looking into a facial recognition screen at an airport

Skip to Highlights
Highlights

What GAO Found

Since August 2010, the Department of Homeland Security’s (DHS) Office of Test and Evaluation (OTE) has assessed major acquisition programs’ test results and DHS leadership has used OTE’s assessments to make informed acquisition decisions. Programs generally received approval to progress through the acquisition life cycle, but DHS placed conditions on its approvals in more than half the cases GAO reviewed.

Since May 2017, OTE has updated its policies and released new guidance that met nearly all of the key test and evaluation (T&E) practices GAO identified as contributing to successful acquisition outcomes. For example, OTE’s policy directs program managers to designate a T&E manager who is required to be certified to the highest level in the T&E career field—level III. This met the key practice that programs establish an appropriately trained test team. However, OTE’s guidance partially met the key practice to demonstrate that subsystems work together prior to finalizing a system’s design. Specifically, the guidance instructs programs to conduct integration testing, but not until after the design is finalized. GAO’s past work has shown that changes after finalizing design can increase costs or delay schedules.

DHS faces challenges with its T&E workforce to effectively provide oversight. As shown in the figure, GAO determined that most programs do not have a level III certified T&E manager.

Status of Certified Test and Evaluation (T&E) Manager Designations Reported by DHS Major Acquisition Programs GAO Reviewed, as of March 2019

Figure described in preceding paragraph. For additional information about this figure, refer to contacts listed at http://www.gao.gov/products/GAO-20-20

OTE also compiles data to monitor programs and the status of T&E managers, but GAO found that OTE’s data were unreliable. Specifically, OTE had inaccurate data for about half of the programs reviewed. Establishing a process for collecting and maintaining reliable data can improve OTE’s ability to accurately track programs’ T&E managers. Further, DHS has expanded OTE’s responsibilities for T&E oversight in recent years. However, OTE officials said executing these responsibilities has been difficult because DHS has not authorized changes to its federal workforce since 2014. These officials added that they have had to prioritize their oversight efforts to programs actively engaged in testing and, as such, are unable to assist programs that are early in the acquisition life cycle. By assessing OTE’s workforce, DHS can take steps to ascertain the extent to which OTE has the number of staff with the necessary skills to fulfill the full scope of its oversight responsibilities.

Why GAO Did This Study

DHS invests several billion dollars in major acquisition programs each year to support its many missions. Conducting T&E of program capabilities is a critical aspect of DHS’s acquisition process to ensure systems work as intended before being delivered to end users, such as Border Patrol agents.

GAO was asked to review DHS’s T&E activities for major acquisition programs. This report examines, among other objectives, the extent to which DHS has (1) assessed programs’ test results and used this information to make acquisition decisions; (2) policies and guidance that reflect key T&E practices; and (3) a workforce to effectively oversee programs’ T&E activities.

GAO reviewed OTE’s assessments of program test results and acquisition decision memorandums. GAO assessed DHS’s policies and guidance against key T&E practices developed by GAO for the purpose of this report. GAO also reviewed data on the T&E workforce that provides oversight at the program and headquarters levels, as well as met with OTE and other relevant DHS officials.

Skip to Recommendations

Recommendations

GAO is making five recommendations, including that DHS revise its guidance to fully meet GAO’s key T&E practice to demonstrate that subsystems work together prior to finalizing a system’s design; establish a process for collecting and maintaining reliable data on programs’ T&E managers; and assess OTE’s workforce. DHS concurred with GAO’s recommendations.

Recommendations for Executive Action

Agency Affected Recommendation Status
Department of Homeland Security 1. The Secretary for Homeland Security should direct the Director of OTE to revise T&E policy or guidance as necessary to fully meet the key practice for programs to test that components and subsystems work together as a system in a controlled setting before finalizing a system's design. (Recommendation 1)
Closed - Implemented
DHS concurred with our recommendation and, in October 2020, revised its T&E policy to specify that programs should complete integration testing to demonstrate that components and subsystems work together before finalizing a system design.
Department of Homeland Security 2. The Secretary for Homeland Security should direct the Director of OTE, in coordination with OCPO, to develop an assessment process-including establishing performance measures-to help ensure T&E training achieves desired results. (Recommendation 2)
Open
In providing comments on this report, DHS concurred with our recommendation and stated that it planned to assess the knowledge and skill requirements for the T&E workforce and establish performance goals for the training. DHS Test and Evaluation Division (TED), in coordination with OCPO, also plan to develop strategies to address any deficiencies with the current training that do not meet the identified requirements. In April 2020, TED officials said that they developed a new survey process to obtain recurring feedback from participants on the training's impact on their ability to perform T&E duties as assigned over time to inform the annual review of the T&E curriculum. However, this effort is still in a piloting stage so the extent to which this information is used to assess the training is still unknown at this time. As of July 2020, TED was still in the process of executing these efforts.
Department of Homeland Security 3. The Secretary for Homeland Security should direct the Director of OTE to revise T&E policy or guidance as necessary to specify when in the acquisition life cycle a major acquisition program manager should designate a level III certified T&E manager. (Recommendation 3)
Closed - Implemented
DHS concurred with our recommendation and, in October 2020, revised its T&E policy to specify that programs should designate a certified test and evaluation manager by acquisition decision event 2A.
Department of Homeland Security 4. The Secretary for Homeland Security should direct the Director of OTE to establish an internal control process to ensure that data collected and maintained on major acquisition programs' T&E managers are reliable. (Recommendation 4)
Open
In providing comments on this report, DHS concurred with our recommendation and stated that it planned to establish an internal control process to reliably collect and maintain data on acquisition programs' assigned test and evaluation managers. In April 2020, DHS Test and Evaluation Division (TED) reported taking steps to ensure the validity of this data including establishing points of contacts within each component to cross-check collected information for accuracy and having the Director review collected data on a quarterly basis beginning in third quarter fiscal year 2020. As of July 2020, TED was still in the process of improving its internal collection process, but had not completed these efforts.
Department of Homeland Security 5. The Secretary for Homeland Security should direct the Under Secretary for Science and Technology to assess OTE's workforce to ascertain the extent to which it has the appropriate number of staff with the necessary skills to fulfill its responsibilities. (Recommendation 5)
Open
In providing comments on this report, DHS concurred with our recommendation and stated that it planned to assess the Test and Evaluation Division's (TED) workforce by reviewing current staffing levels and vacancies against the division's roles and responsibilities. The Senior Official Performing the Duties of the Under Secretary for Science and Technology plans to use the results of this review to inform strategic hiring in future years, if needed. In February 2020, DHS released its fiscal year 2020 strategic guidance memorandum for the Science and Technology (S&T) Directorate which included a statement pertaining to resourcing S&T's test and evaluation capabilities. However, as of July 2020, S&T had not yet conducted its review of TED's workforce.

Full Report

GAO Contacts