Information Technology:

Customs Has Made Progress on Automated Commercial Environment System, but It Faces Long-Standing Management Challenges and New Risks

GAO-06-580: Published: May 31, 2006. Publicly Released: May 31, 2006.

Additional Materials:

Contact:

Randolph C. Hite
(202) 512-6256
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

The Department of Homeland Security (DHS) is conducting a multiyear, multibillion-dollar acquisition of a new trade processing system, planned to support the movement of legitimate imports and exports and strengthen border security. By congressional mandate, plans for expenditure of appropriated funds on this system, the Automated Commercial Environment (ACE), must meet certain conditions, including GAO review. This study addresses whether the fiscal year 2006 plan satisfies these conditions; it also describes the status of DHS's efforts to implement prior GAO recommendations for improving ACE management, and provides observations about the plan and DHS's management of the program.

The fiscal year 2006 ACE expenditure plan, including related program documentation and program officials' statements, either satisfied or partially satisfied the legislative conditions imposed by the Congress; however, more can be done to better address several aspects of these conditions. In addition, DHS has addressed some recommendations that GAO has previously made, but progress has been slow in addressing several recommendations aimed at strengthening ACE management. For example, DHS has more to do to implement the recommendation that it establish an ACE accountability framework that, among other things, ensures that expenditure plans report progress against commitments made in prior plans. Implementing a performance and accountability framework is important for ensuring that promised capabilities and benefits are delivered on time and within budget. In addition, describing progress against past commitments is essential to permit meaningful congressional oversight. Among GAO's observations about the ACE program and its management are several related to the need to effectively set and use performance goals and measures. Although the program set performance goals, these targets were not always realistic. For example, in fiscal year 2005, the program set a target that 11 percent of all Customs and Border Protection (CBP) employees would use ACE. However, this target does not reflect the fact that many CBP employees will never need to use the system. Additionally, the program has established 6 program goals, 11 business results, 23 benefits, and 17 performance measures, but the relationships among these are not fully defined or adequately aligned with each other. For example, not every goal has defined benefits, and not every benefit has an associated performance measure. Without realistic ACE performance measures and targets that are aligned with the overall program goals and desired results, DHS will be challenged in its efforts to establish an accountability framework for ACE that will help to ensure that the program delivers its expected benefits. In addition, DHS plans to develop several increments, referred to as "releases," concurrently; in the past, such concurrency has led to cost overruns and schedule delays because releases contended for the same resources, and resources that were to be used on later releases were diverted to earlier ones. However, because of DHS's belief that such concurrent development will allow it to deliver ACE functionality sooner, it is reintroducing the same problems that resulted in past shortfalls.

Recommendations for Executive Action

  1. Status: Closed - Implemented

    Comments: Regarding the ACE Privacy Impact Assessment (PIA), our review of DHS's 2007 ACE Expenditure Plan (GAO 08-46, October 2007) reported that the ACE PIA did not cover recently completed screening releases S1 and S2. Program officials told us that S1 and S2 are covered by Automated Targeting System (ATS) PIA. However, our analysis of the ATS PIA showed that it addressed screening and targeting functions but did not specifically address releases S1 and S2. DHS subsequently implemented this recommendation by completing and documenting a PIA for releases S1 and S2. Regarding enterprise architecture alignment, our 2007 review also reported that DHS had not sufficiently documented the criteria and methodology used to determine how ACE aligned with the DHS Enterprise Architecture (EA), and did not address other relevant aspects of program alignment to an EA, such as data alignment. DHS subsequently implemented this recommendation by documenting ACE EA alignment criteria, methodology, and data architecture alignment and reported to the Congress in 2008 that the ACE Data Reference Model(DRM) was aligned to the CBP and DHS Data Reference Models (DRM).

    Recommendation: To assist CBP in managing ACE--and increasing the chances that it will deliver required capabilities on time and within budget, demonstrating promised mission benefits and results--the Secretary of Homeland Security should direct the appropriate departmental officials to fully address those legislative conditions associated with having an approved privacy impact assessment and ensuring architectural alignment.

    Agency Affected: Department of Homeland Security

  2. Status: Closed - Implemented

    Comments: Our review of DHS's 2007 ACE Expenditure Plan (GAO 08-46, October 2007) reported that CBP had implemented our 2006 recommendation to measure ACE performance and results by developing a range of ACE performance measures, such as user satisfaction; had taken steps to align the measures with program, CBP, and DHS strategic goals and outcomes; and planned to refine the measures and include them in the ACE accountability framework. We also reported that CBP had implemented our recommendation to employ effective independent validation and verification (IV&V) practices by certifying that an IV&V agent was under contract; developing an IV&V Implementation Management Plan that required an IV&V program consistent with the industry standard; providing a set of objectives, guidelines, and expectations for IV&V activities, including periodic independent reports on status, observations, recommendations and activities; and addressing satisfaction of quality standards for ACE products and user needs with the program. Program officials told us that these IV&V improvements allowed early identification and correction of program process and product weakness.

    Recommendation: The Secretary of Homeland Security, through CBP's Acting Commissioner, should direct the Assistant Commissioner for Information and Technology to fully address those legislative conditions associated with measuring ACE performance and results and employing effective independent verification and validation practices.

    Agency Affected: Department of Homeland Security: Directorate of Border and Transportation Security: Bureau of Customs and Border Protection: Office of Information and Technology

  3. Status: Closed - Implemented

    Comments: Our review of DHS's FY 2007 ACE Expenditure Plan (GAO-08-46, October 2007) reported that DHS had submitted quarterly reports from November 2002 through the second quarter of fiscal year 2007 to the House and the Senate Appropriations Subcommittees documenting CBP's efforts to address open GAO recommendations for ACE. We also reported that by taking these actions, CBP had demonstrated that full implementation of this recommendation.

    Recommendation: The Secretary of Homeland Security, through CBP's Acting Commissioner, should direct the Assistant Commissioner for Information and Technology to accurately report to the appropriations committees on CBP's progress in implementing our prior recommendations.

    Agency Affected: Department of Homeland Security: Directorate of Border and Transportation Security: Bureau of Customs and Border Protection: Office of Information and Technology

  4. Status: Closed - Not Implemented

    Comments: The Bureau of Customs and Border Protection (CBP) included a summary of its information technology (IT) human capital strategy and its accountability framework for the Automated Commercial Environment (ACE) in its fiscal year 2006 second quarter report (March 2006). However, our review of DHS's 2007 ACE Expenditure Plan (GAO 08-46, October 2007) determined that CBP's June 30, 2006, report to the Congress described ACE human capital goals, but did not meet basic tenets of strategic human capital management. For instance, the goals were not developed using a gap analysis that assessed existing staffing and skills against expected staffing needs. At that time, DHS officials reported that a new human capital strategy and implementation plan was in development. In November 2008, CBP assessed its ACE staffing by job function and labor category and determined that the gap between available and needed staff was within tolerable limits, and therefore concluded that no strategic changes in IT human capital were needed. However, CBP's assessment did not address future human capital needs due to changes in technology (such as training and hiring) or projected attrition (such as retirements and employee turnover), and thus did not address key aspects of an information technology human capital strategy. Furthermore, the ACE quarterly reports to the Congress since June 2006 have not reported on IT skill gaps and or plans to address the gaps. As a result, CBP has not yet developed a credible IT human capital strategy for ACE or provided the appropriations committees with updates on ACE IT human capital status and plans.

    Recommendation: The Secretary of Homeland Security, through CBP's Acting Commissioner, should direct the Assistant Commissioner for Information and Technology to include in the June 30, 2006, quarterly update report to the appropriations committees a strategy for managing ACE human capital needs and the ACE framework for managing performance and ensuring accountability.

    Agency Affected: Department of Homeland Security: Directorate of Border and Transportation Security: Bureau of Customs and Border Protection: Office of Information and Technology

  5. Status: Closed - Implemented

    Comments: Our review of DHS's 2007 ACE Expenditure Plan (GAO-08-46, October 2007) reported that CBP had completed implementation of this recommendation by amending the system life cycle gate review process for Office of Information and Technology projects, including ACE, to include milestone readiness decisions based on an assessment and acceptance of risks (including the risks related to unresolved defects). We further reported that the ACE Risk and Issue Management Process guidance -- which provides for the systematic identification, analysis, prioritization, planning, execution, evaluation, and documentation of program risks and issues and requires that any risks associated with going forward should be identified as part of each life cycle gate review -- was applied to two major gate reviews for ACE in 2006: Release A1 and Release M1. CBP plans to continue this process for future gate reviews. Quarterly reports to Congress on ACE stated that risks related to gate review decisions and the associated impacts were entered into a database to ensure visibility and mitigation and were included in the package submitted to the DHS CIO for review and certification to pass a given milestone. By taking these actions, CBP has demonstrated implementation of the recommendation.

    Recommendation: The Secretary of Homeland Security, through CBP's Acting Commissioner, should direct the Assistant Commissioner for Information and Technology to document key milestone decisions in a way that reflects the risks associated with proceeding with unresolved severe defects and provides for mitigating these risks.

    Agency Affected: Department of Homeland Security: Directorate of Border and Transportation Security: Bureau of Customs and Border Protection: Office of Information and Technology

  6. Status: Closed - Implemented

    Comments: Our review of DHS's FY 2007 ACE Expenditure Plan (GAO-08-46, October 2007) reported that the ACE program office had reduced overlap and concurrence in ACE releases by, for example decoupling and reducing dependencies among releases, reducing contention for hardware, dividing releases into drops for more flexible scheduling, and identifying overlap risk in a database. However, we reported that vague and incomplete mitigation strategies made it difficult to determine their effectiveness in reducing concurrency risks among releases and that challenges remained in managing scheduling slips. Since our 2007 report, DHS has taken two additional actions to address this recommendation. First it has stopped work on ACE drops M1 and M2, thus reducing concurrence with these drops. The Cargo Systems Program Office reported that the two remaining active projects, Entry Summary, Accounts, and Revenue (ESAR A2.3.1a) and Cargo Control and Release (CCR M1), do not contend for resources and that future work will be scheduled to avoid resource contention. Secondly, an Environments Working Group (EWG) convenes weekly to coordinate resources among all project teams and mitigate contention in the ACE environment using a chart of projects and resource requirements. By taking these actions, the agency has demonstrated that it has implemented the recommendation by minimizing the overlap among ACE releases -- thereby reducing resource contention -- and by establishing a mechanism to monitor and manage the residual risks of resource contention.

    Recommendation: The Secretary of Homeland Security, through CBP's Acting Commissioner, should direct the Assistant Commissioner for Information and Technology to minimize the degree of overlap and concurrence across ongoing and future ACE releases, and capture and mitigate the associated risks of any residual concurrence.

    Agency Affected: Department of Homeland Security: Directorate of Border and Transportation Security: Bureau of Customs and Border Protection: Office of Information and Technology

  7. Status: Closed - Implemented

    Comments: Our review of DHS's 2007 ACE Expenditure Plan (GAO-08-46, October 2007) reported this recommendation as implemented because CBP had established baselines to implement EVM for Release 5/Drops A1 and A2, Screening 3, and Release 6/Drop M1 and included EVM data in its accountability framework. By establishing this framework, DHS has demonstrated its commitment to using EVM in future releases and is positioned to execute EVM throughout the life of ACE.

    Recommendation: The Secretary of Homeland Security, through CBP's Acting Commissioner, should direct the Assistant Commissioner for Information and Technology to use earned value management in the development of all existing and future releases.

    Agency Affected: Department of Homeland Security: Directorate of Border and Transportation Security: Bureau of Customs and Border Protection: Office of Information and Technology

  8. Status: Closed - Implemented

    Comments: Our review of DHS's FY 2007 ACE Expenditure Plan (GAO-08-46, October 2007) reported that CBP had implemented this recommendation by establishing a) performance measures tied to program objectives and performance goals; b) processes for collecting, analyzing, and integrating performance data for each measure; and c) a life cycle process and database tool to manage these performance measures. These performance measures address user satisfaction, efficiency, and productivity, among other things, and are integral to the ACE accountability framework, which was approved by the CBP Commissioner and DHS CIO.

    Recommendation: The Secretary of Homeland Security, through CBP's Acting Commissioner, should direct the Assistant Commissioner for Information and Technology to develop the range of realistic ACE performance measures and targets needed to support an outcome-based, results-oriented accountability framework, including user satisfaction with ACE.

    Agency Affected: Department of Homeland Security: Directorate of Border and Transportation Security: Bureau of Customs and Border Protection: Office of Information and Technology

  9. Status: Closed - Implemented

    Comments: Our review of DHS's 2007 ACE Expenditure Plan (GAO-08-46, October 2007) reported that CBP had implemented this recommendation by developing ACE program measures tied to program goals and an ACE Performance Reference Model that aligns CBP strategic goals and objectives with performance measures for ACE releases. For example, CBP aligned the DHS strategy of preventing terrorists and terrorist weapons from entering the U. S. through improved information and targeting with the objective of using advanced passenger and cargo information from various ACE and non-ACE sources to pre-screen, target, and identify potential terrorists, terrorist shipments, and related activities. These were linked to CBP's strategy for increased use of targeting, desired results for the number of security-focused selections generated by ACE, and performance measures for capturing the actual selection numbers. Further, this model links the measures to specific ACE releases. This comprehensive alignment across DHS, CBP, and ACE provides the structure to ensure that ACE meets department and agency mission needs.

    Recommendation: The Secretary of Homeland Security, through CBP's Acting Commissioner, should direct the Assistant Commissioner for Information and Technology to explicitly align ACE program goals, benefits, desired business outcomes, and performance measures.

    Agency Affected: Department of Homeland Security: Directorate of Border and Transportation Security: Bureau of Customs and Border Protection: Office of Information and Technology

 

Explore the full database of GAO's Open Recommendations »

Sep 25, 2014

Sep 23, 2014

Jun 10, 2014

May 22, 2014

May 12, 2014

May 8, 2014

May 7, 2014

Apr 2, 2014

Looking for more? Browse all our products here