Arizona Border Surveillance Technology Plan:

Additional Actions Needed to Strengthen Management and Assess Effectiveness

GAO-14-368: Published: Mar 3, 2014. Publicly Released: Mar 12, 2014.

Additional Materials:

Contact:

Rebecca Gambler
(202) 512-8777
gamblerr@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

What GAO Found

The Department of Homeland Security's (DHS) U.S. Customs and Border Protection's (CBP) schedules and Life-cycle Cost Estimates for the Arizona Border Surveillance Technology Plan (the Plan) reflect some, but not all, best practices. Scheduling best practices are summarized into four characteristics of reliable schedules—comprehensive, well constructed, credible, and controlled (i.e., schedules are periodically updated and progress is monitored). GAO assessed CBP's schedules as of March 2013 for the three highest-cost programs that represent 97 percent of the Plan's estimated cost. GAO found that schedules for two of the programs at least partially met each characteristic (i.e., satisfied about half of the criterion), and the schedule for the other program at least minimally met each characteristic (i.e., satisfied a small portion of the criterion), as shown in the table below. For example, the schedule for one of the Plan's programs partially met the characteristic of being credible in that CBP had performed a schedule risk analysis for the program, but the risk analysis was not based on any connection between risks and specific activities. For another program, the schedule minimally met the characteristic of being controlled in that it did not have valid baseline dates for activities or milestones by which CBP could track progress.

Summary of GAO's Schedule Assessments for the Three Highest-Cost Programs under the Arizona Border Surveillance Technology Plan

Schedule characteristic

Program 1

Program 2

Program 3

Comprehensive

Partially met

Partially met

Partially met

Well constructed

Substantially met

Partially met

Partially met

Credible

Partially met

Partially met

Minimally met

Controlled

Partially met

Partially met

Minimally met

Source: GAO analysis of CBP data.

Note: Not met—CBP provided no evidence that satisfies any of the criterion. Minimally met—CBP provided evidence that satisfies a small portion of the criterion. Partially met—CBP provided evidence that satisfies about half of the criterion. Substantially met—CBP provided evidence that satisfies a large portion of the criterion. Met—CBP provided complete evidence that satisfies the entire criterion.

Further, CBP has not developed an Integrated Master Schedule for the Plan in accordance with best practices. Rather, CBP has used the separate schedules for each program to manage implementation of the Plan, as CBP officials stated that the Plan contains individual acquisition programs rather than integrated programs. However, collectively these programs are intended to provide CBP with a combination of surveillance capabilities to be used along the Arizona border with Mexico, and resources are shared among the programs. According to scheduling best practices, an Integrated Master Schedule is a critical management tool for complex systems that involve a number of different projects, such as the Plan, to allow managers to monitor all work activities, how long activities will take, and how the activities are related to one another. Developing and maintaining an Integrated Master Schedule for the Plan could help provide CBP a comprehensive view of the Plan and help CBP better understand how schedule changes in each individual program could affect implementation of the overall Plan.

Moreover, cost-estimating best practices are summarized into four characteristics—well documented, comprehensive, accurate, and credible. GAO's analysis of CBP's estimate for the Plan and estimates completed at the time of GAO's review for the two highest-cost programs showed that these estimates at least partially met three of these characteristics: well documented, comprehensive, and accurate. In terms of being credible, these estimates had not been verified with independent cost estimates in accordance with best practices. Ensuring that scheduling best practices are applied to the three programs' schedules and verifying Life-cycle Cost Estimates with independent estimates could help better ensure the reliability of the schedules and estimates.

CBP did not fully follow key aspects of DHS's acquisition management guidance for the Plan's three highest-cost programs. For example, CBP plans to conduct limited testing of the highest-cost program—the Integrated Fixed Tower (IFT: towers with cameras and radars)—to determine its mission contributions, but not its effectiveness and suitability for the various environmental conditions, such as weather, in which it will be deployed. This testing, as outlined in CBP's test plan, is not consistent with DHS's guidance, which states that testing should occur to determine effectiveness and suitability in the environmental conditions in which a system will be used. Revising the test plan to more fully test the program in the conditions in which it will be used could help provide CBP with more complete information on how the towers will operate once they are fully deployed.

CBP has identified mission benefits for technologies under the Plan, but has not yet developed performance metrics. CBP has identified such mission benefits as improved situational awareness and agent safety. Further, a DHS database enables CBP to collect data on asset assists, defined as instances in which a technology, such as a camera, or other asset, such as a canine team, contributed to an apprehension or seizure, that in combination with other relevant performance metrics or indicators, could be used to better determine the contributions of CBP's surveillance technologies and inform resource allocation decisions. However, CBP is not capturing complete data on asset assists, as Border Patrol agents are not required to record and track such data. For example, from fiscal year 2010 through June 2013, Border Patrol did not record whether an asset assist contributed to an apprehension event for 69 percent of such events in the Tucson sector. Requiring the reporting and tracking of asset assist data could help CBP determine the extent to which its surveillance technologies are contributing to CBP's border security efforts.

This is a public version of a For Official Use Only—Law Enforcement Sensitive report that GAO issued in February 2014. Information DHS deemed as For Official Use Only—Law Enforcement Sensitive has been redacted. 

Why GAO Did This Study

In recent years, nearly half of all annual apprehensions of illegal entrants along the southwest border have occurred along the Arizona border. Under the Secure Border Initiative Network (SBI net ), CBP deployed surveillance systems along 53 of the 387 miles of the Arizona border with Mexico. After DHS canceled further SBI net procurements, CBP developed the Plan, which includes a mix of radars, sensors, and cameras to help provide security for the remainder of Arizona's border. GAO was asked to review the status of DHS's efforts to implement the Plan. This report addresses the extent to which CBP (1) developed schedules and Life-cycle Cost Estimates for the Plan in accordance with best practices, (2) followed aspects of DHS's acquisition management guidance in managing the Plan's programs, and (3) identified mission benefits and developed performance metrics for surveillance technologies to be deployed under the Plan. GAO reviewed schedule, cost, and acquisition documents and analyzed fiscal year 2010 through June 2013 data on apprehensions and seizures.

What GAO Recommends

GAO recommends that CBP, among other things, apply scheduling best practices, develop an integrated schedule, verify Life-cycle Cost Estimates, revise the IFT test plan, and require tracking of asset assist data. DHS concurred with four of six GAO recommendations. It did not concur with the need for an integrated schedule or a revised IFT test plan. As discussed in this report, GAO continues to believe in the need for a schedule and a revised test plan.

For more information, contact Rebecca Gambler at (202) 512-8777 or gamblerr@gao.gov.

Recommendations for Executive Action

  1. Status: Closed - Implemented

    Priority recommendation

    Comments: In March 2014, U.S. Customs and Border Protection (CBP) concurred with our recommendation and in response, stated it planned to ensure that scheduling best practices are applied where practical when updating the three program schedules. As we reported, we found that CBP schedules for the three highest-cost programs under the plan- the Integrated Fixed Towers (IFT), Remote Video Surveillance System (RVSS) and Mobile Surveillance Capability (MSC)- reflected some, but not all, best practices. The IFT and RVSS had at least partially met each characteristic (i.e. satisfied about half of the criterion), and the MSC partially met two characteristics and minimally met two (i.e., satisfied a small portion of the criterion). Scheduling best practices are summarized into four characteristics of reliable schedules: comprehensive, well-constructed, credible, and controlled (i.e., schedules are periodically updated and progress is monitored). We recommended that when updating the IFT, RVSS, and MSC program schedules that CBP ensure that scheduling best practices were applied to the three programs' schedules. In response, CBP provided us with updated schedules for its three programs that CBP had completed as of January 2017, and based on our assessment, CBP applied scheduling best practices. Specifically, the updated IFT, RVSS, and MSC program schedules substantially or fully met the criterion for three of the four characteristics of reliable schedules (i.e., satisfied a large portion of or the entire criterion), and for the other characteristic, each program partially met the criterion (i.e., satisfied about half of the criterion). As a result, CBP has improved the quality of the programs' schedules, which will help CBP ensure the reliability of its schedules for its border surveillance technology programs. GAO is closing this recommendation as implemented.

    Recommendation: To improve the acquisition management of the Plan and the reliability of its cost estimates and schedules, assess the effectiveness of deployed technologies, and better inform CBP's deployment decisions, when updating the schedules for the IFT, Remote Video Surveillance System (RVSS), and Mobile Surveillance Capability programs, the Commissioner of CBP should ensure that scheduling best practices, as outlined in our schedule assessment guide, are applied to the three programs' schedules.

    Agency Affected: Department of Homeland Security: United States Customs and Border Protection

  2. Status: Closed - Not Implemented

    Comments: In March 2014, CBP did not concur with this recommendation and maintained that an integrated master schedule for the Arizona Border Surveillance Technology Plan (Plan) in one file undermines the implementation strategy approved by the Department of Homeland Security (DHS) for the individual programs making up the Plan, and that the implementation of this recommendation would essentially create a large, aggregated program, and effectively create an aggregated "system of systems". DHS further stated that a key element of the Plan has been the disaggregation of technology procurements. However, as we noted in the report, collectively these programs are intended to provide CBP with a combination of surveillance capabilities to be used along the Arizona border with Mexico. Moreover, while the programs themselves may be independent of one another, the Plan's resources are being shared among the programs. Developing and maintaining an integrated master schedule for the Plan could have allowed CBP insight into allocation of resources for all programs as opposed to attempting to resolve any resource constraints for each program individually. However, in 2014 CBP developed a separate and broader plan that incorporated the Arizona Border Surveillance Technology Plan and extended land-based surveillance technology deployments to the remainder of the southwest border - the Southwest Border Technology Plan. Given the passage of time and CBP's expanded scope of efforts, GAO is closing this recommendation as not implemented.

    Recommendation: To improve the acquisition management of the Plan and the reliability of its cost estimates and schedules, assess the effectiveness of deployed technologies, and better inform CBP's deployment decisions, the Commissioner of CBP should develop and maintain an Integrated Master Schedule for the Plan that is consistent with scheduling best practices.

    Agency Affected: Department of Homeland Security: United States Customs and Border Protection

  3. Status: Closed - Implemented

    Comments: In March 2014, DHS concurred with this recommendation. In May 2016, CBP officials stated that the Department of Homeland Security's Cost Analysis Division (CAD) had started piloting an independent cost estimate for the Remote Video Surveillance System program. CBP selected this program for the pilot because these technologies would be deployed along the border, expanding the program beyond Arizona, and thus would benefit most from having an independent cost estimate. CAD approved its independent cost estimate for the program in November 2016, and according to CBP officials, CBP finalized the life-cycle cost estimate and reconciled it with the independent cost estimate in March 2017. CBP reported that the component acquisition executive approved the reconciliation estimate in September 2017. In November 2017, GAO reviewed the approved life-cycle cost estimate and independent cost estimate and found that they met the intent of GAO's recommendations. That same month CBP officials confirmed that the agency does not plan to conduct an independent cost estimate for the Integrated Fixed Tower program, noting that there are no plans to expand the program and that they believe it would not provide any value as CBP is in the deployment stage of the program. GAO concurs given there are no current plans to expand the program. In November 2017, CBP officials reported that they plan to continue to use best practices to guide their cost estimating methods for future technology investments. CBP's efforts have helped to position the agency to better ensure the reliability of its cost estimates for border security technology investments, and as a result this recommendation is closed as implemented.

    Recommendation: To improve the acquisition management of the Plan and the reliability of its cost estimates and schedules, assess the effectiveness of deployed technologies, and better inform CBP's deployment decisions, when updating Life-cycle Cost Estimates for the IFT and RVSS programs, the Commissioner of CBP should verify the Life-cycle Cost Estimates with independent cost estimates and reconcile any differences.

    Agency Affected: Department of Homeland Security: United States Customs and Border Protection

  4. Status: Closed - Not Implemented

    Priority recommendation

    Comments: In March 2014, DHS did not concur with this recommendation and stated that the Test and Evaluation Master Plan includes tailored testing and user assessments that will provide much, if not all, of the insight contemplated by the intent of the recommendation. We continue to believe that DHS should have revised the Test and Evaluation Master Plan to more fully test the Integrated Fixed Tower (IFT) program, before beginning full production, in the various environmental conditions in which the IFT will be used. This would have been in line with DHS acquisition guidance, which underscores that for commercial-off-the-shelf systems, such as the IFT program, operational test and evaluation should occur in the environmental conditions in which a system will be used before a full production decision for the system is made and the system is subsequently deployed. However, according to CBP officials, DHS conducted limited user testing for the IFT system during October and November 2015 at just one site, despite CBP's intention to deploy IFTs to dozens of locations in southern Arizona, which can include different terrain and differences in climate throughout the year. As we noted in the report, conducting limited user testing in one area could limit the information available to CBP on how the IFTs may perform in other conditions and locations along the Arizona border. In addition, DHS guidance states that the primary purpose of test and evaluation is to provide timely and accurate information to managers, decision makers, and other stakeholders to support research, development, and acquisition in a manner that reduces programmatic financial, schedule, and performance risks. However, DHS approved the IFT program for production when testing for the IFTs had not yet begun. As noted in our 2014 report, revising the Test and Evaluation Master Plan to include more robust testing to determine operational effectiveness and suitability could have better positioned CBP to evaluate IFT capabilities before moving to full production for the system. In November 2017, CBP stated that they continue to non-concur with this recommendation and will not revise the Test and Evaluation Master Plan. As a result and given the passage of time, this recommendation is closed as not implemented.

    Recommendation: To improve the acquisition management of the Plan and the reliability of its cost estimates and schedules, assess the effectiveness of deployed technologies, and better inform CBP's deployment decisions, the Commissioner of CBP should revise the IFT Test and Evaluation Master Plan to more fully test the IFT program, before beginning full production, in the various environmental conditions in which IFTs will be used to determine operational effectiveness and suitability, in accordance with DHS acquisition guidance.

    Agency Affected: Department of Homeland Security: United States Customs and Border Protection

  5. Status: Closed - Implemented

    Comments: On June 6, 2014, in response to GAO's recommendation, the Border Patrol issued guidance informing agents that the asset assist data field within the e3 Processing database is now a mandatory data field. In February 2015, Border Patrol officials confirmed that agents are required to enter any assisting surveillance technology of other equipment within the e3 Processing database before proceeding. These actions meet the intent of our recommendation.

    Recommendation: To improve the acquisition management of the Plan and the reliability of its cost estimates and schedules, assess the effectiveness of deployed technologies, and better inform CBP's deployment decisions, the Commissioner of CBP should require data on asset assists to be recorded and tracked within the Enforcement Integrated Database, which contains data on apprehensions and seizures.

    Agency Affected: Department of Homeland Security: United States Customs and Border Protection

  6. Status: Open

    Priority recommendation

    Comments: In February 2015, the Border Patrol was planning to address this recommendation using the Capability Gap Analysis Process (CGAP) developed by the Johns Hopkins University Applied Physics Lab, specifically for the Border Patrol, to examine the effects of technology and other assets, such as infrastructure. In September 2016, Border Patrol provided a case study that assessed CGAP data with technology assist data and other measures. While this was a start to developing performance measures, the case study was limited to one location along the border and the analysis limited to select technologies. In May 2017, Border Patrol officials demonstrated a new system that was intended to allow for more comprehensive analysis of the contributions of surveillance technologies to Border Patrol's mission. In June 2018, Border Patrol officials stated they planned to issue a report--to executive management on a quarterly basis--that would identify border security technologies that assisted with Border Patrol apprehensions. Border Patrol distributed such a report to its leadership in July 2018 that included, among other things, the proportion of apprehensions in which certain surveillance technologies were utilized during the first three quarters of fiscal year 2018. However, Border Patrol officials stated that some of the data in the report were not reliable due to data entry errors in one of Border Patrol's data systems. Officials told GAO they have plans for training, system integration, and additional steps intended to improve data entry and reporting consistency by the first quarter of fiscal year 2020. In December 2018, Border Patrol officials stated they had not prepared the report for the fourth quarter of fiscal year 2018, due to organizational changes and limited staff with the technical skills needed to prepare the report. The officials stated they planned to meet with Border Patrol managers in early 2019 to discuss additional actions to ensure they have access to and are using performance information about existing technologies as they make decisions about future resource investments in additional or new technologies. GAO continues to believe that developing and applying performance metrics for assessing implementation of border technologies would help Border Patrol better determine if or when mission benefits have been realized.

    Recommendation: To improve the acquisition management of the Plan and the reliability of its cost estimates and schedules, assess the effectiveness of deployed technologies, and better inform CBP's deployment decisions, once data on asset assists are required to be recorded and tracked, the Commissioner of CBP should analyze available data on apprehensions and seizures and technological assists, in combination with other relevant performance metrics or indicators, as appropriate, to determine the contribution of surveillance technologies to CBP's border security efforts.

    Agency Affected: Department of Homeland Security: United States Customs and Border Protection

 

Explore the full database of GAO's Open Recommendations »

May 13, 2019

May 9, 2019

May 1, 2019

Apr 26, 2019

Apr 24, 2019

Apr 18, 2019

Apr 4, 2019

Apr 3, 2019

Looking for more? Browse all our products here