Air Traffic Control:

FAA Reports Progress in System Acquisitions, but Changes in Performance Measurement Could Improve Usefulness of Information

GAO-08-42: Published: Dec 18, 2007. Publicly Released: Dec 18, 2007.

Additional Materials:

Contact:

Gerald L. Dillingham
(202) 512-3000
contact@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

Acquiring new systems on budget and on schedule is critically important in transitioning to the Next Generation Air Transportation System (NextGen). However, air traffic control modernization has been on GAO's high-risk list since 1995, in part due to acquisitions exceeding budget and schedule targets. The Federal Aviation Administration's (FAA) Air Traffic Organization (ATO) has responsibility for managing air traffic control acquisitions. GAO was asked to examine (1) ATO's goals, performance measures, and reporting for systems acquisitions; (2) the validity of ATO's performance measures; and (3) the implications of using ATO's performance measures to assess progress in transitioning to NextGen. To address these issues, GAO compared ATO's measures with attributes of successful performance measures, interviewed agency officials, and sought perspectives of aviation experts.

To be consistent with federal guidance and with targets set in the Department of Transportation's strategic plan, ATO established annual acquisition goals and performance measures that call for a high percentage of its major acquisitions to be within 10 percent of budget and on schedule. ATO identifies major acquisitions and reports performance against its goals using its most recently approved budget and schedule estimates. To measure on-budget performance, ATO calculates budget increases over an 8-month period--between January and August of each year. To measure on-schedule performance, ATO selects a minimum of two annual milestones from its major acquisitions and calculates the percentage of milestones that are on schedule. Because ATO's acquisition performance measures lack objectivity, reliability, coverage of core activities, and clarity, and focus only on the preceding year, they may not provide a valid assessment of performance over time. On the positive side, the measures are aligned with FAA's strategic objectives, are measurable, have no overlap, and address governmentwide priorities. However, the performance measures lack objectivity because ATO has no objective criteria for designating which programs are "major" and should be selected for performance reporting. This makes it possible for subjective considerations to dominate the outcome and leaves the performance measures vulnerable to bias in the selection of programs for reporting. The lack of objective criteria for designating major programs also impairs the reliability of the measures (the ability of the measures to produce the same results each time they are applied under similar conditions) and undermines assurance that ATO managers include all core program activities in performance reporting each year. The performance measures also lack clarity in that they do not indicate that ATO measures the performance of many acquisitions against the most recently approved budget and schedule estimates rather than the original estimates. ATO's acquisition performance measurement and reporting could mask budget increases and schedule delays that could have a negative effect on the transition to NextGen. Although ATO reported performance that exceeded its goals for fiscal years 2004 through 2006 and showed nearly steady improvement, when measured against original baselines, acquisition performance improved but was lower than reported. Going forward, the absence of original budget and schedule information on ATO's acquisitions could give the impression that the transition to NextGen is progressing more smoothly than might actually be the case. It will be important for ATO and Congress to recognize budget increases and schedule delays so that the capacity, efficiency, and safety benefits of NextGen can be realized in a cost-efficient and timely fashion.

Status Legend:

More Info
  • Review Pending-GAO has not yet assessed implementation status.
  • Open-Actions to satisfy the intent of the recommendation have not been taken or are being planned, or actions that partially satisfy the intent of the recommendation have been taken.
  • Closed-implemented-Actions that satisfy the intent of the recommendation have been taken.
  • Closed-not implemented-While the intent of the recommendation has not been satisfied, time or circumstances have rendered the recommendation invalid.
    • Review Pending
    • Open
    • Closed - implemented
    • Closed - not implemented

    Recommendations for Executive Action

    Recommendation: Because of the importance of ensuring that key administration and congressional decision makers and stakeholders have complete information on the budget and schedule performance of FAA's critical ATC acquisition programs--both for the most recent fiscal year and since their inception--the Secretary of Transportation should direct the FAA Administrator to identify or establish a vehicle for regularly reporting to Congress and the public on ATO's overall, long-term performance in acquiring ATC systems by providing original budget and schedule baselines for each rebaselined program and the reasons for the rebaselining. If this information is not added to FAA's annual Performance and Accountability Report, then the Performance and Accountability Report should reference where this information can be found.

    Agency Affected: Department of Transportation

    Status: Closed - Implemented

    Comments: In December, 2007, we reported that FAA's acquisition performance measure focused on performance during the preceding year, rather than from the initiation of the acquisition--a period that often spans many years. FAA would report that it met its performance goal if a defined percentage of its major acquisitions for air traffic control modernization remained within budget and on schedule during the fiscal year. However, the budget and schedule baselines, as stated at the start of a fiscal year, often reflect earlier increases over the acquisition's original budget estimate, or reflect decisions in prior years to delay schedules beyond the original schedule estimates. Comparing an acquisition's current budget and schedule baselines against its original baselines would, in many cases, show more acquisitions being over budget or behind schedule than FAA was reporting. We reported that FAA's performance reporting could give the impression to Congress and the American people that FAA's acquisitions performance was better than was actually the case. We recommended that FAA establish a vehicle for regularly reporting to Congress and the public on its overall, long-term systems acquisitions performance by providing original budget and schedule baselines for each rebaselined program, and the reasons for the rebaselining. In response, FAA established a new appendix to Capital Investment Plan, which is reported annually to Congress and made available to the public over the internet. FAA first published the new appendix, titled "Response to GAO Report," in its fiscal year 2010 - 2014 Capital Investment Plan, but the appendix did not contain sufficient information to address our recommendation. FAA included additional baseline information in its fiscal year 2011 - 2015 Capital Investment Plan, including each acquisition's original and current budget and schedule as well as the reasons for changes, as we recommended. For example, the appendix shows that FAA had extended the completion date for its Integrated Terminal Weather System (ITWS) from the original completion date of July 2003 to June 2010 and increased its budget from $276.1 million to $282.1 million. The appendix explained that the rebaselining was necessary due to, among other things, added requirements. By presenting the most accurate and complete assessment possible when reporting its performance in acquiring air traffic control systems, FAA will better facilitate congressional understanding and oversight of FAA's progress in implementing NextGen.

    Recommendation: Because of the importance of ensuring that key administration and congressional decision makers and stakeholders have complete information on the budget and schedule performance of FAA's critical ATC acquisition programs--both for the most recent fiscal year and since their inception--the Secretary of Transportation should direct the FAA Administrator to improve the clarity of ATO's annual acquisition performance measurement process by disclosing in its Performance and Accountability Reports that the measurement for on-budget performance covers 8 months and is measured against the most recently approved budget baselines. Similarly, improve the wording of the target and reporting for on-schedule acquisitions to disclose that this measures 1 year of performance against selected program milestones.

    Agency Affected: Department of Transportation

    Status: Closed - Implemented

    Comments: In December 2007, we reported that FAA's acquisition performance measure focused on performance during the preceding year, rather than from the initiation of the acquisition--a period that often spans many years. FAA would report that it met its performance goal if a defined percentage of its major acquisitions remained within budget and on schedule during the fiscal year. However, the budget and schedule baselines, as stated at the start of a fiscal year, often reflect decisions in prior years to increase acquisitions' budgets or to delay schedules beyond the original budget and schedule estimates. Comparing an acquisition's current budget and schedule baselines against its original baselines would, in many cases, show more acquisitions being over budget or behind schedule than FAA was reporting. Consequently, FAA's performance reporting could give the impression to Congress and the American people that FAA's acquisitions performance was better than was actually the case. We recommended that FAA improve the clarity of its annual acquisition performance measurement process by disclosing in its Performance and Accountability Reports that (1) the measurement for on-budget performance is measured against the most recently approved budget baselines, and (2) the target and reporting for on-schedule acquisitions measures 1 year of performance against selected program milestones. In FAA's 2009 Performance and Accountability Report, the "Critical Acquisitions on Budget" performance measurement is "By FY 2009, 90% of Major System Investments are within 10% variance of current baseline total budget estimate at completion." The "Critical Acquisitions on Schedule" performance measurement is "In FY 2009, 90% of Major System Investments selected annual milestones are achieved." By including "current baseline" in the on-budget performance measure and "annual" in the on-schedule performance measure, FAA has improved the clarity of its performance reporting.

    Recommendation: Because of the importance of ensuring that key administration and congressional decision makers and stakeholders have complete information on the budget and schedule performance of FAA's critical air traffic control (ATC) acquisition programs--both for the most recent fiscal year and since their inception--the Secretary of Transportation should direct the FAA Administrator to improve the objectivity, reliability, and inclusion of core programs in ATO's acquisition performance measures by establishing written, objective criteria and guidance for managers to use in determining which programs are major--and thus selected for performance reporting--and in selecting schedule milestones.

    Agency Affected: Department of Transportation

    Status: Closed - Implemented

    Comments: In our December 2007 report we stressed the importance of ensuring that key administration and congressional decision makers and stakeholders have complete information on the budget and schedule performance of FAA's critical Air Traffic Control acquisition programs. We recommended in that FAA improve the objectivity, reliability, and inclusion of core programs in ATO's acquisition performance measures by establishing written, objective criteria and guidance for managers to use in determining which programs are major and thus selected for performance reporting and in selecting schedule milestones. Since 2009, FAA has defined "major programs" as new investments with acquisition category levels of 1, 2, or 3 (there are 5 levels in total). Each of the three category levels contains programs that require special management attention because of their importance to the agency's mission as defined in Office of Management and Budget Circular A-11, Part 7. Programs in the three categories include those with high development, operating or maintenance costs; high risk; high return; or a significant role in the administration of FAA programs, finances, property, or other resources. FAA reports on the performance of the major programs contained in acquisition categories' 1 through 3 in Appendix D of its Capital Investment Plan which it submits to Congress. With this action, FAA can provide assurance that programs important to the transition to NextGen are on time and within budget.

    Recommendation: Because of the importance of ensuring that key administration and congressional decision makers and stakeholders have complete information on the budget and schedule performance of FAA's critical ATC acquisition programs--both for the most recent fiscal year and since their inception--the Secretary of Transportation should direct the FAA Administrator to improve the usefulness of ATO's acquisition performance reporting by including information (in the Performance and Accountability Report or elsewhere) on the potential effects that any budget or schedule slippages could have on the overall transition to NextGen. This also could include information concerning any mitigation plans ATO has developed to lessen the effects of program slippages on the implementation of NextGen systems.

    Agency Affected: Department of Transportation

    Status: Closed - Implemented

    Comments: In our December 2007 report we stressed the importance of ensuring that key administration and congressional decision makers and stakeholders have complete information on the budget and schedule performance of FAA's critical Air Traffic Control acquisition programs. We recommended that FAA improve the usefulness of ATO's acquisition performance reporting by including information on the potential effects that any budget or schedule slippages could have on the overall transition to NextGen. In our February 2012 report on FAA's air traffic control modernization acquisitions (GAO-12-223) we stated that FAA had begun developing an integrated master schedule for the entire NextGen initiative that would, in part, capture related NextGen program schedules, governance activities, and other performance and financial data to provide real-time monitoring of the overall NextGen effort. We further noted that having a reliable integrated master schedule would enable FAA to determine how delays in one program impact other programs and the overall NextGen implementation timeline. According to FAA, by the end of 2012, it will update its NextGen Segment Implementation Plan to show dependencies through 2018 and this will serve as the Integrated Master Schedule. With this action, FAA will have more assurance that the NextGen in on track

    Jul 31, 2014

    Jul 23, 2014

    Jun 25, 2014

    Jun 24, 2014

    Jun 18, 2014

    Jun 11, 2014

    May 30, 2014

    Looking for more? Browse all our products here