This is the accessible text file for GAO report number GAO-05-23 
entitled 'Air Traffic Control: FAA's Acquisition Management Has 
Improved, but Policies and Oversight Need Strengthening to Help Ensure 
Results' which was released on December 13, 2004.

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov.

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately.

Report to the Chairman and Ranking Minority Member, Committee on 
Government Reform, House of Representatives: 

November 2004: 

AIR TRAFFIC CONTROL: 

FAA's Acquisition Management Has Improved, but Policies and Oversight 
Need Strengthening to Help Ensure Results: 

GAO-05-23: 

GAO Highlights: 

Highlights of GAO-05-23, a report to the Chairman and Ranking Minority 
Member, Committee on Government Reform, House of Representatives: 

Why GAO Did This Study: 

The Federal Aviation Administration’s (FAA) multibillion-dollar effort 
to modernize the nation’s air traffic control (ATC) system has resulted 
in cost, schedule, and performance shortfalls for over two decades and 
has been on GAO’s list of high-risk federal programs since 1995. 
According to FAA, performance shortfalls were due, in part, to 
restrictions imposed by federal acquisition and personnel regulations. 
In response, Congress granted FAA exemptions in 1995 and directed it to 
develop a new acquisition management system. 

In this report, GAO compared FAA’s AMS with (1) the FAR and (2) 
commercial best practices for major acquisitions, and (3) examined 
FAA’s implementation of AMS and its progress in resolving problems with 
major acquisitions.

What GAO Found: 

FAA’s Acquisition Management System (AMS) is broader and less 
prescriptive than the Federal Acquisition Regulation (FAR), but both 
afford managers flexibility. AMS establishes an acquisition life-cycle 
management system, including both a contracting and program management 
system, whereas the FAR is primarily a contracting system. In addition, 
AMS takes the form of guidance—it is not regulatory, while the FAR is a 
set of published regulations—a legal foundation that has the force and 
effect of law that most federal agencies are required to follow.

[See PDF for image]

[End of figure]

AMS provides some discipline for acquiring major ATC systems; however, 
it does not ensure a knowledge-based approach to acquisition found in 
the best commercial practices for managing commercial and DOD product 
developments that we have identified in numerous past reports. Best 
practices call for (1) use of explicit written criteria to attain 
specific knowledge at key decision points and (2) use of this knowledge 
by executives at the corporate level to determine whether a product is 
ready to move forward. Attainment and use of such knowledge by 
executives helps to avoid cost, schedule, and performance shortfalls 
that can occur if they commit to a system design prematurely. While AMS 
has some good features, including calling for key decision points, it 
falls short of best practices. 

GAO’s review of seven major ATC systems and analysis of FAA’s 
performance in acquiring major systems found that AMS has not resolved 
longstanding problems it experienced prior to its implementation of 
AMS—including developing requirements and managing software—and is just 
beginning to focus on how these acquisitions will improve the 
efficiency of ATC operations. combined found that long-standing 
problems While FAA has made progress by providing guidance for avoiding 
past weaknesses, it has not applied these improvements consistently. 
According to FAA officials, reorganization under and improved oversight 
by FAA’s new performance-based Air Traffic Organization should help 
ensure greater consistency and an increased focus on results. Past GAO 
reports have demonstrated that the success of an acquisition process 
depends on good management, whether it be under AMS or the FAR.

What GAO Recommends: 

GAO recommends that the Secretary of Transportation advise FAA to, 
among other things, (1) improve its development of requirements and 
management of software and (2) more closely align AMS with commercial 
best practices. 

In commenting on a draft of this report, FAA generally agreed with the 
report’s contents and said that our recommendations would be helpful to 
them as they continue to refine AMS. They also provided us with 
technical comments, which we have incorporated as appropriate. 

www.gao.gov/cgi-bin/getrpt?GAO-05-23.

To view the full product, including the scope and methodology, click on 
the link above. For more information, contact JayEtta Z. Hecker 
512-2834 or heckerj@gao.gov.

[End of section]

Contents: 

Letter: 

Results in Brief: 

Background: 

AMS Is Broader and Less Prescriptive Than the FAR: 

AMS Provides Some Discipline but Does Not Ensure a Knowledge-Based 
Approach to Acquisition: 

As Implemented, AMS Has Not Resolved Long-standing Acquisition 
Problems, but FAA Is Beginning to Focus More on Results: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments: 

Appendixes: 

Appendix I: FAA Has Begun Analyzing Spending Trends to Take a More 
Strategic Approach to Procurement: 

Appendix II: Objectives, Scope, and Methodology: 

Appendix III: Comparison of the Scope and Flexibility of FAA's 
Acquisition Management System and the Federal Acquisition Regulation 
Process: 

Background: 

AMS Defines an Investment/Life-Cycle Project Management System: 

Only a Portion of AMS Deals Directly with the Procurement Process: 

AMS States a Nonregulatory FAA Policy: 

AMS Chapter 3 Parallels a Subset of the FAR: 

AMS Includes a Less Rigorous Competition Requirement Than Does the FAR: 

The FAR Gives Procurement Professionals Tighter Control over 
Procurement Decisions: 

Although FAA Project Managers View AMS as More Efficient and Flexible 
Than the FAR, Some Procurement Officials We Interviewed Do Not Agree: 

Appendix IV: How FAA's Acquisition Policy Adapted Key Recommendations 
Made by GAO and DOT (1996-2003): 

FAA Refined AMS in Response to Recommendations: 

Appendix V: Status of the Seven ATC Modernization Acquisitions That GAO 
Reviewed: 

Appendix VI: Key Contributors: 

GAO Contacts: 

Staff Acknowledgments: 

Tables Tables: 

Table 1: Structure of Best Practices Model for Major Product 
Developments: 

Table 2: Structure of AMS: 

Table 3: Knowledge-Based Approach Called for in Our Best Practices 
Model: 

Table 4: AMS's Decision Points, Information Sources, and Oversight 
Reviews: 

Table 5: Description and Status of Seven Selected ATC Acquisitions: 

Table 6: Internal and External Reviews of FAA's Use of AMS for 
Acquiring Major ATC Systems: 

Table 7: Program Costs for the Seven Systems We Reviewed: 

Table 8: Comparison of AMS and the FAR: 

Table 9: Key Recommendations Made to Improve FAA's Acquisition 
Processes: 

Figures: 

Figure 1: Scope of AMS and the FAR: 

Figure 2: Review Process under Our Best Practices Model and under AMS: 

Figure 3: Key Milestones for Selected ATC Acquisitions Initiated before 
and after AMS: 

Figure 4: Our Analysis of FAA's Progress as of 2003 in Implementing Key 
Elements of Training for Its Acquisition Workforce: 

Abbreviations: 

AMS: Acquisition Management System: 

ASR-11: Airport Surveillance Radar, Model 11: 

ATC: air traffic control: 

ATO: Air Traffic Organization: 

ATOP: Advanced Technologies and Oceanic Procedures: 

BOE: Cost Basis of Estimate: 

CFO: Chief Financial Officer: 

CIO: Chief Information Officer: 

COO: Chief Operating Officer: 

CMMI: CMMI Capability Maturity Model Integration: 

COTS: Commercial-off-the-shelf: 

DOD: Department of Defense: 

DOT: Department of Transportation: 

DOTIG: Department of Transportation Office of Inspector General: 

ERAM: En Route Automation Modernization: 

F&E: Facilities and Equipment: 

FAA: Federal Aviation Administration: 

FAR: Federal Acquisition Regulation: 

FMFIA: Federal Managers' Financial Integrity Act of 1982: 

iCMM: integrated Capability Maturity Model: 

ISC: Initial System Configuration: 

IT: Information Technology: 

ITIM: Information Technology Investment Management framework: 

ITWS: Integrated Terminal Weather System: 

JRC: Joint Resources Council: 

LAAS: Local Area Augmentation System: 

NEXCOM: Next Generation Air/Ground Communications System: 

NDI: Non-developmental Item: 

OMB: Office of Management and Budget: 

RE&D: Research, Engineering and Development: 

SEI: Software Engineering Institute: 

SFFAS 4: Statements of Federal Financial Accounting Standards no. 4: 

STARS: Standard Terminal Automation Replacement System: 

WAAS: Wide Area Augmentation System: 

Letter November 12, 2004: 

The Honorable Tom Davis:
Chairman: 
The Honorable Henry A. Waxman: 
Ranking Minority Member: 
Committee on Government Reform: 
House of Representatives: 

In late 1981, the Federal Aviation Administration (FAA) began a 
modernization program to replace and upgrade the National Airspace 
System's (NAS) equipment and facilities to meet the expected increase 
in traffic volume, enhance the margin of safety, and increase the 
efficiency of the air traffic control (ATC) system--the principal 
component of the NAS. Historically, the modernization program has 
experienced cost overruns, schedule delays, and performance shortfalls 
of large proportions and has been on our list of high-risk programs 
since 1995. To date, FAA has spent $41 billion and expects to spend an 
additional $7.6 billion through fiscal year 2007 to, among other 
things, finalize key modernization projects designed to replace radar, 
navigation, communications, and information-processing 
systems.[Footnote 1]

According to FAA, the performance shortfalls in its modernization 
program were due, in part, to restrictions imposed by federal 
acquisition and personnel requirements. In response, Congress passed 
legislation in 1995 that granted FAA unique acquisition and personnel 
exemptions, or flexibilities, and directed FAA to develop a new 
acquisition management policy. FAA issued its new acquisition 
management policy, called the Acquisition Management System (AMS), in 
1996 and began using the new system instead of the Federal Acquisition 
Regulation (FAR). To further address long-standing weaknesses in the 
ATC modernization program, the President and Congress in 2000 directed 
FAA to reorganize and establish a new organization. FAA has just begun 
to do so.

Now that FAA has had several years to implement the earlier procurement 
flexibilities, as well as some time to reorganize, some results of its 
acquisition reform should be discernable. Moreover, FAA's experiences 
in exercising its acquisition flexibilities could provide valuable 
information to Congress in overseeing the use of these flexibilities.

You asked us to review the steps that FAA has taken to reform its 
acquisition of major ATC systems and the impact of the reforms on FAA's 
acquisition outcomes. Specifically, you asked us to (1) compare the 
scope and flexibility of AMS and the FAR, (2) compare AMS with 
commercial best practices for major acquisitions, and (3) examine FAA's 
implementation of AMS and progress in addressing long-standing problems 
with major acquisitions. In addition, you asked us to review FAA's 
general procurement of goods and services; we cover this topic in 
appendix I.

To address the first objective, we compared the topics addressed by, 
and the implementation options afforded to contracting and procurement 
officials under AMS and the FAR. To address the second objective, we 
used a model of best practices that we derived from our body of work on 
how leading private firms manage costly and complex product 
developments and how the Department of Defense (DOD) manages major 
weapon systems acquisitions.[Footnote 2] We used this model to assess 
the extent to which FAA's acquisition management policy mirrors the 
acquisition policies of high-performing organizations in the public and 
private sectors. This model consists of four phases: (1) concept and 
technology development; (2) product development, which includes both 
integration and demonstration activities; (3) production; and (4) 
operations and support. In between these four phases are three key 
knowledge decision points at which commercial firms and the government 
must have sufficient knowledge to make large investment decisions. To 
address the third objective, we selected the seven ATC systems with the 
largest budgets to explore the results of FAA's implementation of its 
acquisition management policy and procedures and to determine how FAA 
has addressed issues found to have contributed to cost, schedule, or 
performance problems. In selecting these seven systems, we ensured that 
some were initiated before and some after April 1996, when FAA 
implemented AMS. While the results of these analyses are not 
generalizable to all of FAA's major ATC acquisitions, they indicate the 
extent to which the agency has made progress in addressing long-
standing problems we have identified. To further assess both the 
implementation and the impact of FAA's acquisition reforms, we reviewed 
our work on FAA's major ATC acquisition efforts since 1996 as well as 
the work of the Department of Transportation's Office of Inspector 
General (DOTIG), FAA, and others. We also reviewed the actions that FAA 
has taken to refine AMS in response to internal and external reviews. 
Finally, to review FAA's procurement of goods and services across the 
agency, we used a commercial best-practices model for taking a more 
strategic approach to procurement, along with interviews with key 
agency officials, to determine whether FAA has begun to analyze 
spending trends to identify opportunities to leverage its buying power. 
We conducted our work from December 2003 through November 2004 in 
accordance with generally accepted government auditing standards. See 
appendix II for additional information on our objectives, scope, and 
methodology.

Results in Brief: 

AMS consists of broad guidance for acquisition life-cycle management--
from defining the requirements for a system through fielding 
(deploying) and decommissioning it (removing it from service). This 
broad guidance contrasts with the rather more detailed and prescriptive 
contract-formation and contract-administration requirements contained 
in the FAR. AMS is broader in scope because it addresses, among other 
areas of life-cycle management, both contract and program management, 
providing both policies and procedures for contracting and a toolset of 
recommended practices for managing individual acquisition projects over 
their life cycles. By contrast, the FAR focuses in far greater detail 
on contracting policies and procedures. FAA managers believe they have 
greater flexibility in interpreting and applying AMS than they would 
have under the FAR, in part because, in areas addressed by both, AMS is 
less directive than the FAR. For example, although AMS states a 
"preference" for competition, FAA personnel may use single-source 
contracting when necessary to fulfill FAA's mission. By contrast, other 
federal agency contracting officials operating under the FAR are 
generally required to seek "full and open competition"--a more rigorous 
standard. These other agency officials can generally use sole-source or 
limited-competition contracting only after higher-level agency 
procurement officials have approved a written justification. In 
addition, FAA contracting personnel operate as part of acquisition 
teams that are responsible to program managers; under the FAR, 
contracting decisions are made by contracting personnel who are 
responsible only to contracting officials. Nonetheless, the FAR also 
affords flexibility because it encourages innovation and addresses a 
wide selection of contracting methods; therefore, procurement officials 
can choose the approach that they consider most appropriate to their 
procurement. According to some current and former FAA procurement 
officials with experience in using both the FAR and AMS, the FAR may 
appear inflexible and cumbersome to inexperienced managers, but those 
who are familiar with it can navigate it effectively.

AMS provides some discipline through its various phases, activities, 
and decision points for acquiring major ATC systems; however, it does 
not ensure the use of a knowledge-based approach found in the best 
practices for managing commercial product developments and DOD 
acquisitions[Footnote 3] that we have identified in numerous past 
reports. Commercial best practices call for specific knowledge to be 
captured and used by corporate-level decision-makers to determine 
whether a product has reached a level of development (product maturity) 
sufficient to demonstrate its readiness to move forward in the 
acquisition process. The capture of such knowledge and its use by 
executives helps to avoid cost overruns, schedule slips, and 
performance shortfalls that can occur if decision-makers commit to a 
system design before acquiring critical technology, design, or 
manufacturing knowledge. AMS has some good features, which indicate a 
process that has some elements of discipline. For example, like the 
best practices model, AMS identifies critical junctures that it terms 
"decision points," the first three of which call for the preparation of 
detailed technical and programmatic information that FAA's corporate 
executive-level body, the Joint Resources Council,[Footnote 4] can use 
to assess whether or not FAA should initiate an acquisition program. 
However, AMS departs from recognized best practices primarily by (1) 
not requiring the attainment of specific knowledge satisfying explicit 
written criteria for decision-makers to use at each key decision point 
and (2) not requiring corporate executive-level oversight at all key 
decisions. For example, AMS allows the Joint Resources Council to 
delegate two key decisions--the decision to begin production and the 
decision to place a system in service. FAA maintains that this approach 
gives program managers flexibility, expedites decision-making, and 
allows those executives with the most knowledge about a major 
acquisition to make key decisions about its continued development. 
FAA's reliance on delegation assumes that managers will inform their 
superiors if they are unable to meet the performance schedules and 
system requirements approved by the Joint Resources Council. However, 
best practices call for more than this, including the use of measurable 
criteria at key points in the acquisition process to ensure that 
specific knowledge has been captured and the independent review of this 
knowledge by corporate executive-level decision-makers before the 
acquisition moves forward in its development. These criteria and 
reviews are particularly important for acquisitions that require a 
large funding commitment, such as those that include the production of 
multiple costly units (e.g., radars and controller workstations). In 
addition, oversight at the corporate-executive or agencywide level is 
needed to ensure consideration of an acquisition's likely impact on 
other agency projects or operations. These departures from best 
practices put FAA's major ATC acquisitions at risk of cost, schedule, 
and performance shortfalls. We are making recommendations to the 
Secretary of Transportation to align AMS more closely with commercial 
best practices.

According to our review of seven major ATC systems and analysis of 
FAA's performance in acquiring major systems, AMS has not resolved 
management problems that FAA experienced before it implemented AMS, but 
the agency is beginning to focus more on the expected results of its 
major acquisitions. (See table 5.) Specifically, our review found that 
AMS did not call for requirements that were specific enough to minimize 
the development of further requirements (requirements growth) or 
unplanned work in five of these systems. This lack of specificity 
resulted in the inadequate development or definition of requirements, 
requirements growth, unplanned work, or a reduction in performance for 
five of these systems. In addition, for three of these systems, FAA 
underestimated the difficulty of modifying available software to 
fulfill its mission needs. Consequently, FAA encountered unexpected 
software development needs, higher costs, and schedule delays. Because 
AMS guidance was not sufficient to account for the risks associated 
with modifying available software, the two systems we reviewed that 
were initiated after AMS's implementation--though currently meeting 
cost and schedule milestones--are nevertheless showing symptoms of 
FAA's past problems with developing requirements and managing software. 
It is too soon to tell if these two systems will remain within their 
cost, schedule, and performance parameters. In addition, our work on 
FAA's major acquisitions, along with that of the DOTIG and others, has 
shown that many of the problems FAA experienced in acquiring major 
systems before 1996 persist under AMS and that effective acquisition 
management, rather than the use of a specific contracting process 
(e.g., the FAR or AMS) is the key to successful acquisitions. To its 
credit, FAA is beginning to focus more on results, largely through its 
new Air Traffic Organization, which has been charged with taking a more 
performance-based approach to managing the agency's major acquisitions. 
This approach includes implementing a training framework for FAA's 
acquisition workforce. While FAA has taken some steps to develop an 
evaluation program with criteria for measuring the extent to which this 
framework is achieving organizational goals by improving the knowledge 
base of FAA's acquisition workforce, at the time of our audit FAA had 
no plans to conduct a comprehensive evaluation. We are making 
recommendations to the Secretary of Transportation to improve FAA's 
development of requirements and management of complex software, and to 
comprehensively evaluate FAA's implementation of the training framework 
to ensure that it is having the intended effect of improving the 
knowledge base of FAA's acquisition workforce. In commenting on a draft 
of this report, FAA said that it generally agreed with the report's 
contents and said that our recommendations would be helpful to them as 
they continue to refine AMS.

Background: 

Maintaining that federal procurement requirements contributed to some 
of its cost, schedule, and performance problems in the 1980s and early 
1990s, FAA sought a statutory exemption from the federal acquisition 
system,[Footnote 5] including the FAR, and those parts of title 5 of 
the United States Code, parts II and III, that govern federal civilian 
personnel management. According to FAA, exemptions from these 
requirements would enable it to streamline its acquisition processes, 
be more responsive to the airline industry's needs, and increase the 
efficiency of ATC operations while maintaining safety. Congress enacted 
legislation in November 1995 that exempted FAA from key federal 
procurement statutes and the FAR, and directed FAA to develop a new 
acquisition management system. In response to these legislative 
initiatives, FAA implemented a new, streamlined acquisition process--
the Acquisition Management System (AMS)--on April 1, 1996.

We developed a knowledge-based model of commercial best practices based 
on our findings about how leading private firms manage costly and 
complex acquisitions effectively--that is, within cost, schedule, and 
performance targets. The use of this knowledge-based model has been 
found to reduce the risks associated with developing products and 
increase the likelihood of successful outcomes. The model divides the 
product development cycle into four phases and related activities. 
Table 1 presents these phases and activities and explains what takes 
place during each.

Table 1: Structure of Best Practices Model for Major Product 
Developments: 

Phase/Activity: 1.Concept and technology development; 
What occurs during this phase or activity: Leading companies work to 
understand their mission needs and confirm that the technologies to be 
used are mature; that is, the technologies needed to meet essential 
product requirements have been demonstrated to work in their intended 
environment.

Phase/Activity: 2. Product development: * Integration; 
What occurs during this phase or activity: Components and subsystems 
are integrated into the product to stabilize the overall system design 
and show that the design can meet the product requirements.

Phase/Activity: 2. Product development: * Demonstration; 
What occurs during this phase or activity: Tests show that the product 
will work as required and can be manufactured within targets.

Phase/Activity: 3. Production; 
What occurs during this phase or activity: Operational test articles 
are built.

Phase/Activity: 4. Operations and support; 
What occurs during this phase or activity: Our best practices model 
does not explicitly cover operations and support activities; however, 
this phase focuses on maintenance of the system through its retirement.

Source: GAO.

[End of table]

AMS provides guidance for selecting and overseeing investments over 
their life cycle. Like our best practices model, it is divided into 
phases and activities, although the divisions sometimes occur at 
different points. Table 2 summarizes AMS's phases and activities.

Table 2: Structure of AMS: 

Phase/Activity: Needs and solution identification: * Mission analysis; 
What occurs during this phase or activity: FAA identifies a capability 
shortfall and determines that it needs an investment to better carry 
out its mission. Recently, FAA began analyzing its mission needs within 
the context of its overall goals for the National Airspace System.

Phase/Activity: Needs and solution identification: * Investment 
analysis; 
What occurs during this phase or activity: FAA, using an investment 
analysis team, evaluates alternatives, selects practical and affordable 
solutions, and develops a baseline of cost, schedule, and performance 
requirements. This document is called the acquisition program baseline.

Phase/Activity: Solution implementation: * System integration; 
What occurs during this phase or activity: Both hardware and software 
components and subsystems are integrated into a product. Also, intra-
and intersystem compatibility are tested and analyzed.

Phase/Activity: Solution implementation: * System demonstration; 
What occurs during this phase or activity: Tests show that the product 
can work as required and be manufactured within targets.

Phase/Activity: Solution implementation: * System production; 
What occurs during this phase or activity: All activities are carried 
out to produce needed quantities. Each end item is tested before it 
leaves the factory to verify that it conforms to specifications and is 
free from manufacturing defects.

Phase/Activity: In-service management; 
What occurs during this phase or activity: All required activities are 
carried out, including directly operating, providing maintenance 
functions (both scheduled and unscheduled), and furnishing technical 
and logistics support for the maintenance of FAA systems, subsystems, 
services, or equipment.

Source: FAA. 

[End of table]

To implement the new, performance-based organization for managing ATC 
modernization and operations, as the President and Congress directed in 
2000, FAA appointed a chief operating officer in August 2003 and 
formally established the Air Traffic Organization (ATO) in February 
2004. ATO, under the direction of a six-member executive council, is 
now responsible for further implementing acquisition reforms for major 
ATC systems.

AMS Is Broader and Less Prescriptive Than the FAR: 

AMS establishes an acquisition life-cycle management system that 
encompasses both contracting and program management, whereas the FAR is 
primarily a contracting system that focuses on contract formation and 
contract administration. As a result, AMS is broader in scope than the 
FAR. See figure 1. In addition, AMS takes the form of guidance. This 
guidance is expressed in documentation of FAA policy, handbooks, 
templates, flowcharts, forms, and standard contract language. It is not 
regulatory. By contrast, the FAR is a set of published regulations--a 
legal foundation that has the force and effect of law for the federal 
agencies that are required to follow it.[Footnote 6] Furthermore, the 
FAR is more detailed and prescriptive in establishing contracting 
requirements and can require more administrative involvement. This 
fundamental difference between AMS and the FAR may suggest to some that 
AMS is more flexible. FAA personnel can choose how to apply AMS's 
provisions to a major acquisition. Nonetheless, procurement officials 
under the FAR also have flexibility because the FAR encourages 
innovation consistent with its direction (and other applicable legal 
requirements), provides a wide selection of contracting solutions, and 
permits contracting officials to choose the methods that they consider 
most suitable for a given situation.

Figure 1: Scope of AMS and the FAR: 

[See PDF for image] 

Note: AMS provides policy for the four phases of life-cycle management, 
as well as 14 functional areas, (e.g., test and evaluation, human 
factors, procurement, real estate, security, and systems engineering).

[A] The NAS in-service decision is a key program milestone that 
authorizes the deployment of a system into the National Airspace System 
after thoroughly testing the system to verify its operational 
readiness.

[End of figure] 

AMS Addresses Both Procurement and Project Management, Whereas the FAR 
Focuses Primarily and in Far Greater Detail on Procurement: 

AMS comprises six policy sections and five appendixes.[Footnote 7] The 
procurement policy section of AMS covers a range of topics, including 
contract funding and administration, contracting with small and 
disadvantaged businesses, and compliance with labor laws. According to 
this section, competition is FAA's preferred method of contracting, but 
single-source contracting is permitted when appropriate to fulfill the 
agency's mission. This policy section also describes the procurement of 
commercially available or nondevelopmental items.

Other sections of AMS cover project management tools that the FAR does 
not address, such as investment analysis, configuration 
management,[Footnote 8] and integrated logistics support.[Footnote 9] 
AMS also addresses areas that fall outside project management and 
procurement, including real property management--an area that becomes 
important when FAA must lease or purchase real property so that it can 
install ATC systems such as radars or antennas on property that it does 
not currently own. FAA's policy directs FAA staff to "conduct this 
business in a fair and equitable manner following best practices." 

Although the FAR includes requirements that address procurement 
planning[Footnote 10] and major systems acquisition,[Footnote 11] it 
does so only in the context of government procurement policy and 
procedure. Agencies subject to the FAR find the broader program 
planning requirements, which appear in AMS but not in the FAR, in 
documents such as the Office of Management and Budget's Circular A-109 
and in their own planning guidance. For example, DOD has issued a 
series of directives and instructions on this subject.[Footnote 12]

The contracting procedures set forth in section 3 of AMS do not 
prescribe detailed contracting procedures for various categories of 
procurements, as do those detailed under the FAR. Instead, AMS provides 
two basic contracting models for obtaining products and services 
through FAA's contracting process. The first model is called "Complex 
and Noncommercial Source Selection" and is used for complex, large-
dollar, developmental, noncommercial items and services. This is the 
model that typically would be used for investments approved by the 
Joint Resources Council. The second model is called the "Commercial and 
Simplified Purchase Method" and is typically used for commercial items 
that are less complex and less costly. Procurements of such products or 
services may be routine in nature and are generally purchased on a 
fixed-price basis. Generally, source selection under AMS follows a 
screening process, with the awardee being selected on a "best value" 
basis from among those who remain in consideration when the selection 
is made.

AMS Provides Broad Guidance While the FAR Establishes Detailed 
Requirements, but Managers Have Flexibility under Both: 

AMS sets out a nonregulatory FAA policy that is binding on FAA 
personnel as FAA employees. AMS also sets out other guidelines that FAA 
states should be followed unless there is a rational basis for doing 
otherwise. AMS is subject to such internal controls and enforcement as 
the Administrator decides and to general overarching legal 
requirements, such as the Government Performance and Results Act of 
1993 (GPRA).[Footnote 13]FAA has also deemed certain acquisition laws 
applicable to its procurements (sometimes with modifications), such as 
the Service Contract Act.[Footnote 14] There is also a legal 
requirement, created by the 1995 legislation exempting FAA from the 
FAR, that small and socially or economically disadvantaged firms be 
given all reasonable opportunities to receive contract awards. FAA has 
adopted a dispute resolution process with some legal 
underpinnings.[Footnote 15] Otherwise, as the preface to AMS states, 
"nothing in this document creates or conveys any substantive [legal] 
rights." In short, although FAA is subject to the general legal 
requirement that government decisions cannot be arbitrary or 
capricious, AMS does not establish regulatory requirements for the 
conduct of procurements and does not create or convey substantive legal 
rights.

In contrast to AMS, the FAR is a set of published regulatory 
requirements. It has the force and effect of law, and agencies that are 
subject to it are bound to follow it. The FAR's requirements provide 
for a range of procurement strategies and approaches. In addition to 
negotiated procurement methods, it allows two-step sealed-bid and two-
phase design-build methods,[Footnote 16] among others. It includes 
streamlined procedures for soliciting and evaluating offers to furnish 
commercial items, as well as permits the use of simplified acquisition 
procedures in a broad range of procurements. Furthermore, the FAR 
supports a diverse selection of available contract types, product-
testing tools, and other tools that an agency's contracting personnel 
may select when conducting an acquisition to meet the agency's needs.

Although contracting personnel in agencies subject to the FAR are 
required to comply with it, they enjoy broad discretion in their 
management of procurements. For example, the FAR allows wide latitude 
in drafting requirements statements, from performance-based statements 
of work to design specifications as necessary. It allows broad 
discretion in framing solicitations and in conducting procurements, 
including scoring proposals, determining how negotiations will be 
conducted, eliminating firms whose proposals are not competitive, and 
selecting the awardees whose proposals afford the government the best 
value when evaluated against the selection criteria established in the 
solicitations.

Because AMS consists of broad guidance while the FAR comprises detailed 
and prescriptive regulatory requirements, FAA managers view AMS as 
giving them more flexibility than they would have under the FAR, 
particularly in two areas--competition and oversight. Whereas the FAR 
generally requires full and open competition, AMS calls for providing 
"reasonable access to" competition to firms interested in obtaining 
contracts--a less rigorous standard than full and open competition. AMS 
further states that the "preferred" method of selecting sources is to 
compete requirements among two or more sources. By contrast, full and 
open competition requires that all responsible sources be permitted to 
compete.[Footnote 17] Under AMS, there is no policy that firms that 
want to participate actually get a chance to do so. Rather, FAA told us 
that its system is beneficial because the agency can use screening 
requests to preselect competing firms, eliminating those firms that FAA 
believes are not likely to receive an award. The following example 
illustrates the differences between AMS and the FAR in their respective 
requirements on exceptions to competition. FAA may contract with a 
single source when this approach is determined to be in the best 
interest of FAA.[Footnote 18] The FAR, however, allows exceptions to 
full and open competition only for certain specified conditions (such 
as unusual and compelling urgency or the availability of only one 
source). The FAR describes in detail the circumstances of these 
conditions and the requirements for using them as justification for not 
providing for full and open competition. The FAR also requires the 
contracting officer to prepare a justification document that must 
generally be approved by higher-level agency procurement officials (up 
to the agency's senior procurement executive) depending on the 
estimated dollar value of the procurement. The content of this 
justification is prescribed by the FAR. When not providing for full and 
open competition, the contracting officer is required under the FAR to 
solicit offers from as many potential sources as is practicable under 
the circumstances. The FAR prohibits contracting if the justification 
for less than full and open competition results from a lack of advanced 
planning. For a more detailed comparison of AMS and the FAR, see 
appendix III.

Although some of the FAA personnel we interviewed see AMS as more 
efficient and flexible than the FAR, other current and former FAA 
procurement officials we interviewed who have experience using both the 
FAR and AMS did not agree that AMS is more flexible than the FAR. 
According to these officials, the FAR may appear inflexible and 
cumbersome to persons who lack experience with it, but those who are 
familiar with it are able to navigate its complexities effectively. The 
FAR requires full and open competition, but as experienced procurement 
personnel know, the system does not break down when emergencies 
necessitate quick and decisive action. For example, we recently 
reported that agencies generally complied with applicable FAR 
requirements in awarding new contracts for work in Iraq using other 
than full and open competition.[Footnote 19]In some circumstances, the 
government's legitimate need for prompt action was sufficient to 
justify selecting a contractor on an expedited basis from among the 
firms that appeared able to meet the government's emergency need. In 
other cases, the agencies reasonably determined that only one source 
could meet their requirements.

AMS Provides Some Discipline but Does Not Ensure a Knowledge-Based 
Approach to Acquisition: 

AMS provides some discipline through its various phases, activities, 
and decision points for acquiring major ATC systems; however, it does 
not ensure the use of a knowledge-based approach found in the best 
practices for managing commercial product developments and DOD 
acquisitions that we have identified in numerous past reports.[Footnote 
20]Commercial best practices call for specific knowledge to be captured 
and used by corporate-level decision-makers to determine whether a 
product has reached a level of development (product maturity) 
sufficient to demonstrate its readiness to move forward in the 
acquisition process. The capture of such knowledge and its use by 
executives helps to avoid cost overruns, schedule slips, and 
performance shortfalls that can occur if decision-makers commit to a 
system design before acquiring critical technology, design, or 
manufacturing knowledge. The absence of these key best practices under 
AMS puts FAA's major ATC acquisitions at risk of cost overruns, 
schedule slips, and performance shortfalls.

Best Practices for Managing Acquisitions Call for a Knowledge-Based 
Approach, Including Criteria for Knowledge Needed and Oversight at the 
Corporate Executive Level: 

Commercial best practices call for managing acquisitions using a 
knowledge-based approach, including (1) using established criteria to 
attain specific knowledge at three critical junctures in the 
acquisition cycle, which we call knowledge points, and (2) requiring 
oversight at the corporate executive level for each of these knowledge 
points. For example, at each knowledge point, successful product 
developers apply specific indicators, or criteria, to determine whether 
they have attained the knowledge they need to move to the next phase or 
activity in the acquisition process. Such developers also conduct 
corporate executive-level reviews to ensure that they obtain the 
insights and perspectives of stakeholders throughout their 
organization. If the knowledge attained does not meet the criteria for 
advancement or if the executive reviewers determine that further 
development is inconsistent with their priorities, the acquisition does 
not move forward. Table 3 summarizes the knowledge points, criteria, 
oversight reviews, and timing of oversight reviews included in our 
model of best practices for major acquisitions.

Table 3: Knowledge-Based Approach Called for in Our Best Practices 
Model: 

Knowledge point: 1. Resources and needs matched; 
Criteria: 
* Match customers' needs with available resources--technology, design, 
time, and funding; 
* Demonstrate that technologies needed to meet essential product 
requirements can work in intended environment; 
* Complete a preliminary product design using systems engineering to 
balance customers' desires and available resources; 
Oversight review: Executive-level review required to initiate the 
program; 
Timing of oversight review: Knowledge point 1 should precede the 
commitment to begin product development.

Knowledge point: 2. Product design stable; 
Criteria: 
* Complete 90 percent of design drawings by critical design review; 
* Obtain stakeholders' concurrence that drawings are complete and 
producible; 
* Review subsystem and system designs; 
* Demonstrate with prototype that design meets users' requirements; 
* Identify critical manufacturing processes; 
Oversight review: Executive-level review required to move to 
demonstration; 
Timing of oversight review: Knowledge point 2 should precede the 
commitment to build prototypes to demonstrate the design.

Knowledge point: 3. Production processes mature; 
Criteria: 
* Demonstrate manufacturing processes; 
* Build and test production prototypes; 
* Test production-representative prototypes to achieve reliability 
goals; 
* Test production-representative prototypes to demonstrate product 
performance in operational environment; 
* Collect statistical process control data; 
Oversight review: Executive-level review required to move to 
production; 
Timing of oversight review: Knowledge point 3 should precede the 
commitment to begin production. 

Source: GAO.

[End of table]

Experience with commercial best practices has shown that to the extent 
that the level of knowledge called for at each knowledge point is not 
attained, organizations take on risks in the form of unknowns that will 
persist into the later stages of development, where they will take more 
time and money to resolve if they become problems. Such problems lead 
to cost increases and schedule delays.

AMS Has Some Good Features but Does Not Ensure That High Levels of 
Knowledge Are Attained Before Major Commitments Are Made: 

AMS has some good features, including phases and key decision points 
indicative of an acquisition process that has some elements of 
discipline; however, AMS does not ensure that high levels of knowledge 
are attained and that corporate executive-level reviews occur before 
major commitments of agency resources are made. For example, like the 
best practices model, AMS identifies critical junctures, which it terms 
"decision points." Three of these decision points occur during the 
initial acquisition phase (mission need, initial investment, and the 
final investment decision). A fourth decision point occurs before 
production, and a fifth decision point occurs before the start of the 
final acquisition phase (in-service management). AMS also calls for 
detailed technical and programmatic information that decision-makers 
can use at the first three decision points to assess whether or not FAA 
should initiate an acquisition program. This information includes a 
final requirements document, a final acquisition program baseline, a 
final investment analysis report, an acquisition strategy paper, and an 
integrated program plan. Finally, AMS, like our best practices model, 
calls for senior executives to review the information and determine 
whether the acquisition is ready to move forward. The FAA executives 
who make the decisions at these points include associate and assistant 
administrators, acquisition executives, the chief financial officer, 
the chief information officer, and legal counsel; they form the Joint 
Resources Council (JRC), FAA's senior decision-making body for major 
ATC acquisitions. Table 4 summarizes this information.

Table 4: AMS's Decision Points, Information Sources, and Oversight 
Reviews: 

Decision point by phase/activity:
Phase: Needs and solution identification; 
* Activity: Mission analysis; 
Decision point: Mission need decision; 
Information sources and oversight reviews: Information sources: Input 
from users in the field and mission need statement; 
Oversight review: JRC review called for to move from mission analysis 
to investment analysis.

Decision point by phase/activity: 
Phase: Needs and solution identification; 
* Activity: Investment analysis; 
Decision Point: Initial investment decision; 
Information sources and oversight reviews: Information sources: Initial 
investment analysis report, initial life-cycle program baseline for the 
most viable alternative, updated initial requirements document and 
action plan for final investment analysis; 
Oversight review: JRC review called for to select a preferred solution.

Decision point: Final investment decision; 
Information sources and oversight reviews: Information sources: Final 
requirements document, final acquisition program baseline, final 
investment analysis report, acquisition strategy paper, integrated 
program plan; 
Oversight review: JRC review called for to move from investment 
analysis to solution implementation.

Decision point by phase/activity: 
Phase: Solution implementation; 
* Activity: System integration; 
* Activity: System demonstration; 
Decision point: Production decision; 
Information sources and oversight reviews: Information sources: 
Determined by JRC; 
Oversight review: JRC may retain or delegate decision making authority.

Decision point by phase/activity: 
* Activity: System production; 
Decision point: In-service decision; 
Information sources and oversight reviews: Information Sources: 
Determined by JRC; 
Oversight Review: JRC review called for to move from solution 
implementation to in-service management; however, the JRC may retain or 
delegate decision making authority.

Decision point by phase/activity: 
Phase: In-service management; 
 
Source: GAO analysis of FAA data.

Note: In this report, we place FAA's "mission analysis" and "investment 
analyses" activities in the "Needs and Solution Identification" phase 
to facilitate comparison with the "concept and technology development" 
phase in our best practices model. Similarly, we place "system 
integration" and "system demonstration" in the solution implementation 
phase for comparative purposes.

[End of table]

AMS departs from the best practices model in two key ways--it does not 
call for high levels of knowledge to be attained at three critical 
junctures (knowledge points), and does not call for corporate 
executive-level oversight at one of five junctures. Specifically, AMS 
does not establish explicit, written criteria for (1) the information 
needed to determine technology maturity at solution implementation, (2) 
releasable drawings at critical design review and production process 
controls at production. Our best practices model calls for attaining 
specific knowledge and setting out criteria for what information should 
be available to help organizations minimize risks in the form of 
unknowns. Risks associated with such unknowns can persist into the 
later stages of development, where they can take more time and money to 
resolve if they become problems, potentially leading to cost increases 
and schedule delays.

In addition, AMS does not provide for corporate executive-level 
oversight reviews at two of the three key junctures where our best 
practices model calls for such reviews. Although AMS calls for three 
Joint Resources Council reviews during the initial acquisition phase--
while our model calls for a single corporate executive-level review--
AMS allows the council to delegate its oversight responsibility later 
in the acquisition process to the program managers within the service 
organization responsible for an acquisition. By contrast, our model 
calls for two corporate executive-level reviews later in the 
acquisition process.

According to FAA, its approach gives program managers flexibility, 
expedites decision-making, and allows the executives with the most 
knowledge about a major acquisition to make key decisions about its 
continued development. FAA's reliance on this approach assumes that the 
program managers will inform higher-level managers if they are unable 
to meet the performance schedules and systems requirements approved by 
the Joint Resources Council. However, although program managers may 
have the most knowledge about their particular acquisition, they may 
not have the agencywide perspective of the Joint Resources Council 
members. Having an agencywide perspective, including a broad 
understanding of an acquisition's potential impact on other agency 
projects and operations, is especially critical when an acquisition 
includes the production of multiple units and requires a substantial 
commitment of agency resources, as do FAA's primarily multimillion-
dollar acquisitions, such as controller workstations and radars.

Because decisions about moving a major acquisition forward require both 
a program manager's specific knowledge of the acquisition itself and a 
senior executive's understanding of the acquisition's potential impact 
on other agency projects and operations, our best practices model calls 
for both measurable criteria at key points in the acquisition process 
to ensure that specific knowledge has been captured and corporate 
executive-level reviews to ensure that senior decision-makers have the 
opportunity to independently consider this knowledge. Without higher-
level reviews such as our best practices model recommends and the Joint 
Resources Council could provide later as well as early in the 
acquisition process, FAA cannot ensure that it has fully considered the 
impact of advancing an acquisition on other agency projects and 
operations. This opportunity for full consideration is a central 
advantage of managing acquisitions as a portfolio, as we concluded in 
our August 2004 report on FAA's information technology investment 
management process.[Footnote 21]

Figure 2 contrasts FAA's process for reviewing an acquisition's 
progress under AMS with the process that we found leads to successful 
commercial acquisitions.

Figure 2: Review Process under Our Best Practices Model and under AMS: 

[See PDF for image] 

[A] To facilitate the comparison of AMS with out best practices model 
in this report, we have done the following: (1) placed FAA's "Mission 
Analysis" and "Investment Analyses" activities in the "Needs and 
Solution Identification" phase to make it comparable with the "concept 
and technology development" phase in our best practices model; (2) 
depicted only the final investment decision point, recognizing that the 
investment analysis phase includes an initial investment decision; and 
(3) placed "system integration" and "system demonstration" in the 
solution implementation phase.

[B] AMS does not explicitly call for a design review decision point, 
which would fall between system integration and system demonstration.

[C] The in-service decision is a key program milestone. It authorizes 
the deployment of a system into the National Airspace System. At times, 
the JRC delegates its decision authority for the production and in-
service decisions to service organizations.

[End of figure] 

To its credit, FAA continues to improve its AMS process. For example, 
the agency is currently modifying its mission needs activity to make 
the selection of major ATC acquisitions more consistent with the 
overall goals of modernizing the National Airspace System. In addition, 
the Air Traffic Organization has established an executive council to 
review major acquisitions before they are sent to the Joint Resources 
Council. This review is designed to screen acquisitions to determine 
which ones are important enough to warrant higher-level review by the 
Council. Finally, FAA is currently revising AMS to bring it in line 
with the Office of Management and Budget's guidance. Specifically, the 
agency is incorporating OMB Exhibit 300, which provides the investment 
justifications and management plans required for major ATC 
acquisitions.

As Implemented, AMS Has Not Resolved Long-standing Acquisition 
Problems, but FAA Is Beginning to Focus More on Results: 

According to our review of seven major ATC systems and analysis of 
FAA's performance in acquiring major systems, AMS has not resolved the 
long-standing problems that FAA experienced before implementing AMS, 
but the agency is beginning to focus more on the expected results of 
its major acquisitions. (See table 5.) Specifically, our review found 
that AMS guidance did not call for requirements that were specific 
enough to minimize requirements growth or unplanned work for five of 
these systems. This lack of specificity resulted in the inadequate 
development or definition of requirements, growth in requirements, 
unplanned work, or a reduction in performance for five of these 
systems. In addition, for three of these systems, FAA underestimated 
the difficulty of modifying available software to fulfill its mission 
needs. Because AMS guidance was not sufficient to account for the risks 
associated with modifying available software, FAA encountered 
unexpected software development needs, higher costs, and schedule 
delays. The two systems we reviewed that were initiated after AMS was 
implemented are currently meeting cost and schedule milestones; 
however, both systems are showing symptoms of FAA's past problems with 
developing requirements and managing software, and it is too soon to 
tell if these programs will remain within their cost, schedule, and 
performance parameters. In addition, our work on FAA's major 
acquisitions, along with that of the DOTIG and others has shown that 
the problems FAA experienced before 1996 in acquiring major systems 
persist under AMS and that effective acquisition management, rather 
than the use of a specific contracting process (e.g., the FAR or AMS) 
is key to successful acquisitions. To its credit, FAA is beginning to 
focus more on results, largely through its new Air Traffic 
Organization, which has been charged with taking a more performance-
based approach to managing the agency's acquisitions.

Table 5: Description and Status of Seven Selected ATC Acquisitions: 

Dollars in millions; 

Project and description: STARS--new controller and maintenance 
workstations to replace the legacy system at terminal air traffic 
control facilities[A]; 
Original cost: $940.0; 
Current cost: $1,460.0; 
Original schedule: 1998; 
Current schedule: 2003; 
Acquisition issues and status: STARS is a joint FAA and DoD program. 
STARS delays and cost increases resulted from poor requirements 
definition and schedule estimates. STARS is fully operational at 25 FAA 
terminal radar facilities and 17 DoD facilities. Only 50 of the planned 
172 systems are being deployed. STARS had difficulties in achieving 
many human factor requirements for improving system efficiency and 
safety.

Project and description: ASR-11--digital radar for terminal 
environments; 
Original cost: $743; 
Current cost: $891.7; 
Original schedule: 1997; 
Current schedule: 2013; 
Acquisition issues and status: ASR-11 was approved for its in-service 
decision in September 2003 and is being deployed at 108 sites. These 
systems are being deployed at a slower pace than originally planned 
because of budget cuts and deferrals.

Project and description: ITWS--computer processors and displays to 
automate weather data near the airport; 
Original cost: $276.1; 
Current cost: $288.3; 
Original schedule: September 2001; 
Current schedule: 2002; 
Acquisition issues and status: Currently, six ITWS systems are 
operational. In May 2004, the ATO Executive Council rebaselined the 
program to include a new weather-forecasting capability into the 
production baseline. FAA proposes to defer 12 of the 34 systems it 
planned to procure.

Project and description: LAAS--a precision approach and landing system 
that augments the Global Positioning System; 
Original cost: $530.1; 
Current cost: $696.1; 
Original schedule: 2002; 
Current schedule: Deferred at least until 2009; 
Acquisition issues and status: LAAS has been adversely affected by poor 
requirements development, a lack of understanding of its technical 
complexity, incomplete software development, and an unrealistic 
development schedule. Unresolved radio interference precludes the safe 
operation of LAAS. As a result, FAA has delayed national deployment to 
continue further research on this issue.

Project and description: NEXCOM--digital radios to improve air traffic 
communications; 
Original cost: $318.4; 
Current cost: $318.4; 
October 2002; 
Current schedule: 2004; 
Acquisition issues and status: NEXCOM program delays were due to 
misunderstanding of a program requirement and testing procedures. 
NEXCOM was recently approved for its in-service decision in July 2004.

Project and description: ATOP--new workstations and processing 
capability to control ocean air traffic; 
Original cost: $548.2; 
Current cost: $548.2; 
Original schedule: June 2004; 
Current schedule: 2004; 
Acquisition issues and status: ATOP achieved its acquisition program 
baseline objectives; however, this baseline does not reflect program 
delays and cost increases resulting from poor requirements development, 
unrealistic schedule estimates, and inadequate evaluation of software 
complexity.

Project and description: ERAM--upgrades the existing en route system 
with improved hardware and software; 
Original cost: $3,649.0; 
Current cost: $3,649.0; 
Original schedule: December 2009; 
Current schedule: December 2009; 
Acquisition issues and status: To date, ERAM has not breached any cost 
and schedule parameters. However, it remains a high-risk program 
because of the large amount of software that must be developed. The 
ERAM contractor is experiencing software engineering difficulties as a 
result of lower-than-expected productivity and software code growth. 

Source: GAO analysis of FAA data.

[A] Terminal air traffic control facilities, known as Terminal Radar 
Approach Control (TRACON) facilities, direct aircraft in the airspace 
that extends from the point where the tower's control ends to about 50 
nautical miles from the airport. A TRACON can be located at or outside 
an airport.

[End of table]

Our Reviews of Seven Major Systems Show That Problems with Requirements 
and Software Management Persist under AMS: 

Our reviews of seven of FAA's costliest ATC system acquisitions found 
that the problems FAA experienced with requirements and software 
management and their related impact on cost, schedule, and performance 
goals persist today under AMS.[Footnote 22]Figure 3 identifies these 
seven acquisitions and their milestones, which are expressed in terms 
of AMS decisions even when the acquisitions were initiated before AMS 
was implemented. (See app. V for a description and the status of each 
of these projects.) Specifically, for 6 of these 7 major ATC 
acquisitions, FAA did not consistently (1) clearly define system 
requirements at the investment decision point or (2) adequately assess 
software complexity. Moreover, as FAA has acknowledged, it has never 
managed its major acquisitions by focusing on how each would improve 
the efficiency of ATC operations while maintaining or improving safety. 
Although FAA has made progress in improving its acquisition of major 
ATC systems--by, for example, improving the maturity of its processes 
for acquiring software, using a "build a little, test a little" 
approach to acquisitions as it did for Free Flight Phase 1,[Footnote 
23] and restructuring its organization to minimize stovepipes--long-
standing problems persist in these areas. In addition, the two systems 
we reviewed that were initiated after AMS's implementation are 
currently operating within cost and schedule goals; however, they are 
showing symptoms of past problems with developing requirements and 
managing software complexity. Moreover, our work for more than two 
decades--before and after AMS's implementation--has cited these types 
of weaknesses as central reasons for the agency's long history of cost, 
schedule, and performance shortfalls. This work has also found that the 
effectiveness of an agency's acquisition management has had a greater 
impact on the success of its major acquisitions than the contracting 
process used (e.g., the FAR or AMS).

Figure 3: Key Milestones for Selected ATC Acquisitions Initiated before 
and after AMS: 

[See PDF for image] 

[End of figure] 

Inadequate Development or Definition of Requirements Led to 
Requirements Growth or Unplanned Work for Five Acquisitions: 

For five of the seven acquisitions we reviewed, AMS guidance did not 
call for requirements that were specific enough to minimize 
requirements growth or unplanned work. For four of these five 
acquisitions--STARS, LAAS, NEXCOM, and ATOP--incomplete and poorly 
defined requirements in the final requirements documents, used at the 
investment decision point to assess an acquisition's readiness to enter 
the development phase, led to requirements growth, unplanned 
development work, or a reduction in system performance.[Footnote 24]For 
the fifth acquisition--ASR-11--FAA misjudged the extent to which the 
high-level requirements that were used to support the commercial-off-
the-shelf/nondevelopmental item (COTS/NDI) procurement by the 
Department of Defense could result in a product capable of meeting 
FAA's mission or user needs. As a result, unplanned software changes 
were required.

* FAA's cost estimate for STARS has grown from its original estimate of 
$0.94 billion in 1996 to $1.46 billion in 2004 and will deploy only 50 
of the 172 STARS initially planned. Much of the cost growth has been 
due to FAA requirements creep. As a result, the STARS program has 
experienced delays of more than five years from its original plan, in 
part due to added requirements to the commercial-off-the-shelf Initial 
System Configuration (ISC). However, the STARS ISC was satisfactory for 
use by the Department of Defense as deployed.

* A final requirements document was approved, and the development of 
LAAS was scheduled to begin in 1999. However, poorly established 
requirements resulted in the addition of 113 new requirements to the 
initial specification, entailing significant software and hardware 
changes. Furthermore, LAAS may not achieve its promised capabilities 
because FAA has been unable to develop technologies necessary to warn 
pilots of a disruption in the LAAS signal. Until this technology is 
developed, LAAS cannot be operated safely. As a result, FAA recently 
cut the fiscal year 2005 funding for LAAS, and the program will revert 
to a research and development effort.

* FAA developed a final requirements document for the NEXCOM system, 
but the requirements lacked the specificity needed to assess the 
development risk. According to a NEXCOM contractor program official, 
this led to miscommunication about the program requirement relating to 
signal interference. This official stated that they misunderstood this 
requirement and had not planned on the additional development work for 
the NDI solution to meet such program objectives and delayed the 
program 21 months. Another program requirement involved the NEXCOM 
radios meeting or exceeding the operational coverage area of the 
existing voice system. The existing radios had power output levels of 
50 watts but the NEXCOM contractor could only achieve 34 watts of power 
to meet the coverage requirement. A program official stated that the 
contractor and FAA had not agreed on the testing procedures to assess 
the power levels. This posed an "unacceptable consequence" and, as a 
result, FAA performed additional testing or flight checks of the 
reduced radio performance (50 watts versus 34 watts) and determined 
that the performance reduction should not affect NEXCOM's mission or 
its coverage requirement.

* FAA did not follow the AMS guidelines that call for completing a 
final requirements document before proceeding to the development phase 
for ATOP. The Joint Resources Council approved a delay in developing 
the final requirements until after contract award. This decision 
resulted in schedule delays and additional unplanned software 
development. The ATOP program office asserted that the requirements 
remained very stable and that the program is within cost and schedule 
objectives established by the Council. However, FAA's internal 
documents revealed that the requirements were not adequately defined. 
For example, the ATOP Investment Analysis Study reported to the Joint 
Resources Council prior to contract award that the lack of more 
detailed ATOP requirements at this stage of acquisition added risk and 
was of concern to the investment analysis team. Under AMS, this team is 
responsible for, among other things, conducting risk analyses for the 
various acquisitions. Furthermore, an ATOP Assessment Team conducted a 
study in March 2003 and determined that at the ATOP contract award, 
"requirements were written at a high level and not mutually understood 
by FAA and the contractor." However, FAA management allowed the ATOP 
program to proceed to solution implementation without the final 
requirements document and, according to the contractor, this resulted 
in schedule delays and growth in the amount of software needing 
development.

* The high-level requirements for ASR-11, jointly generated by FAA and 
the Department of Defense, to support a COTS/NDI acquisition, resulted 
in a product that did not initially meet the FAA mission or user needs. 
The software changes that were required to meet FAA's target detection 
needs, as well as significant hardware design changes, parts 
obsolescence, and production issues, added approximately two years to 
system qualification and acceptance.

FAA Underestimated Software Complexity for Three Systems: 

For three of the seven major ATC acquisitions we reviewed--ITWS, LAAS, 
and ATOP--FAA's AMS guidance was not sufficient to address the risks 
associated with modifying available software[Footnote 25] to fulfill 
FAA's mission needs. In all three cases, FAA officials underestimated 
the difficulty of modifying available software. Our work has shown that 
underestimates are likely to result in unexpected software development, 
higher costs, and schedule delays.

* ITWS experienced delays from the beginning because of the complexity 
of its software development. Although the program appeared to be 
progressing according to its baseline, immediately after the critical 
design review in September 1998, the contractor revealed that it had 
exceeded the target cost by $4 million. In addition, the contractor 
claimed that the program did not recognize that the computer processor 
originally planned for the program was becoming outdated, that the 
manufacturer planned to discontinue its production because the market 
was demanding a processor with greater processing and storage 
capability, and that as a result, the original computer processor would 
not be available to the program. Consequently, ITWS experienced cost 
increases, schedule delays, and performance shortfalls. According to 
the contractor and the original acquisition plan, all systems were 
scheduled for delivery by December 2001, but that date has now 
stretched to after 2009.

* LAAS's technology maturity was not adequately assessed, and further 
development was needed. Specifically, the potential for radio 
interference through the atmosphere was not understood and could limit 
LAAS's operations. FAA has now placed all LAAS activities in research 
and development. FAA did not adequately assess LAAS's software 
development. At the time of the contract award, the contractor and FAA 
estimated that 80 percent of the software that LAAS required had been 
developed. FAA later determined that only 20 percent had been 
developed. FAA and the contractor attribute this discrepancy to a lack 
of communication on the steps necessary to satisfy the program's 
requirements. FAA agrees that it should have conducted a software audit 
and a software capabilities assessment, but pressures to keep LAAS on 
schedule resulted in an inadequate assessment.

* The ATOP contractor underestimated by about half the extent to which 
legacy nondevelopmental item software, which is the core of the ATOP 
system, met the program's 1,036 requirements. As a result, a 
significant amount of unanticipated new software code development and 
other modifications were required.[Footnote 26]

ATC Systems Have Required Multiple Rebaselining Decisions to Address 
Delays and Cost Growth: 

As figure 3 illustrates, FAA initiated at least one rebaselining 
decision for three of the five acquisitions that were begun before AMS 
was implemented and were later transitioned to AMS. These rebaselining 
decisions responded to delays and cost growth--problems that arise when 
requirements are not stable, a program's design is not fixed, or 
software code growth is not controlled. For example, FAA rebaselined 
STARS two times--first in 1999 and again in 2002. Similarly, 2 years 
after the investment decision for ITWS, FAA rebaselined the program 
twice, in 1997 and again in 2001. Given the frequency of these past 
rebaselining decisions for major ATC systems and the number of years 
that elapsed before or between the rebaselining decisions (3 to 4 
years), it is too soon to tell whether the two systems that were 
initiated under AMS--ATOP and ERAM--will require similar rebaselinings 
and ultimately meet their cost, schedule, and performance goals. 
Although both programs are currently operating within their cost and 
schedule goals and have not yet been rebaselined, FAA has had problems 
with managing its major acquisitions in the past and is currently 
having difficulties developing requirements and managing software 
complexity. Furthermore, as we reported in May 2004, FAA's budget 
increased from $9 billion in 1998 to $14 billion in 2004 but will be 
constrained for the foreseeable future. In such a constrained budget 
environment, cost growth and schedule problems can have serious 
negative consequences for ongoing modernization efforts--postponed 
benefits, costly interim systems, delays in funding other systems, or 
reductions in the number of units purchased.

Internal and External Reviews Have Found That FAA Has Made Some 
Progress but Continues to Experience Problems in Acquiring Major ATC 
System under AMS: 

Reviews of FAA's acquisition process, conducted by FAA, GAO, the DOTIG, 
and others have shown that FAA has improved its management of major ATC 
acquisitions in recent years but continues to experience cost overruns, 
schedule slips, and performance shortfalls under AMS. Table 7 
summarizes the results of 22 internal and external reviews of FAA's 
major ATC acquisitions. According to these reviews, issued from 1997 
through 2004, the same problems have persisted over many years, despite 
various initiatives to address them, and FAA needs to strengthen its 
management controls. For example, a key FAA review of eight major ATC 
acquisitions, published in 1999, 3 years after AMS was implemented, 
found that these acquisitions, though on track to meet their 
performance goals, were not meeting their cost and schedule baselines. 
FAA attributed these cost and schedule issues to new or poorly 
understood requirements, underestimates of the acquisitions' technical 
complexity, and funding shortfalls.

In addition, our reviews of major FAA acquisitions--initiated before 
and after AMS was implemented--have found for more than two decades 
that FAA's failure to meet schedule, cost, and performance baselines 
for major ATC acquisitions has been due to shortfalls in planning, weak 
management controls, and a lack of systematic processes for acquiring 
new systems, including inadequate requirements management, cost-
accounting data, and estimates of technical difficulty. As we reported 
in August 2004, judged against the criteria of GAO's framework for 
information technology (IT) investment management, which measures the 
maturity of an organization's investment management processes, FAA has 
established about 80 percent of the basic selection and control 
practices that it needs to manage its mission-critical investments for 
the National Airspace System.[Footnote 27] For example, FAA's business 
units actively monitor projects throughout their life cycles.[Footnote 
28]However, the agency's senior IT investment board does not regularly 
review investments that are in the "in-service management," or 
operational phase, and this creates a weakness in FAA's ability to 
oversee more than $1 billion of its IT investments. In addition, the 
agency has not yet established the practices that would enable it to 
effectively manage its annual IT budget of about $2.5 billion, and 
agency executives lack assurance that they are selecting and managing 
the mix of investments that best meets the agency's needs and 
priorities. DOT has responded to our recommendations to FAA to 
strengthen its IT investment management capability.

Moreover, other reviews, such as those by Booz-Allen & Hamilton and 
MITRE, have identified other shortfalls, which reflect a lack of proper 
management controls and planning. For example, in 1997, Booz-Allen & 
Hamilton found, among other things, that FAA had not clearly defined 
organizational roles and responsibilities within the various phases of 
AMS and that greater guidance and training under AMS were warranted. In 
1999, Booz-Allen & Hamilton reported that FAA had not demonstrated 
improvement in adhering to planned costs and schedules under AMS and 
that the agency needed to better manage its development of requirements 
and address persistent funding shortfalls. Moreover, in 2001, a MITRE 
report on selected major acquisitions found inadequate management 
controls and deficiencies in both contractors' performance and in FAA's 
measurement of acquisition performance. See table 7 for a chronological 
listing of the reviews.

Table 6: Internal and External Reviews of FAA's Use of AMS for 
Acquiring Major ATC Systems: 

Review: GAO, Air Traffic Control: Improved Cost Information Needed to 
Make Billion-Dollar Modernization Investment Decisions, GAO/AIMD-97-
20, (Washington, D.C.: Jan. 22, 1997); 
Selected findings: FAA's cost- estimation practices do not satisfy 
recognized estimating requisites, increasing the likelihood of poor 
acquisition selection decisions; 
Contributing factors: FAA's cost-accounting practices do not provide 
for the proper accumulation of actual project costs.

Review: GAO, Air Traffic Control: Complete and Enforced Architecture 
Needed for FAA Systems Modernization, GAO/AIMD-97-30, (Washington, 
D.C.: Feb. 3, 1997); 
Selected findings: Incompatibilities exist between current and planned 
ATC acquisitions, resulting in high costs and reduced performance; 
Contributing factors: FAA lacks a complete systems architecture or 
overall "blueprint" to guide and constrain the development and 
maintenance of ATC acquisitions.

Review: GAO, Air Traffic Control: Immature Software Acquisition 
Processes Increase FAA System Acquisition Risks, GAO/AIMD-97-47, 
(Washington, D.C.: Mar. 21, 1997); 
Selected findings: Planned acquisitions frequently are not delivered on 
time and within budget; 
Contributing factors: Weaknesses in some key process areas, such as 
planning, requirements development, and management, limit FAA's ability 
to consistently acquire software-intensive ATC systems on time and 
within budget.

Review: FAA, Evaluation of FAA Acquisition Reform--The First Year: 
April 1996 - March 1997, (Washington, D.C.: May 1997); 
Selected findings: AMS addresses 15 of the 17 problems facing 
acquisitions; 
Contributing factors: Inadequate management has not enabled FAA to meet 
its goals of reducing acquisition deployment time by 50 percent and 
cost by 20 percent.

Review: FAA, Evaluation of FAA Acquisition Reform--The First Two Years: 
April 1996 - March 1998, Report #1998-02, (Washington, D.C.: May 29, 
1998); 
Selected findings: Further improvements are necessary if acquisition 
reform is going to allow FAA to meet its cost and schedule goals; 
Contributing factors: Procedural weaknesses limit FAA's ability to 
achieve cost and schedule goals.

Review: GAO, Air Traffic Control: Observations on FAA's Air Traffic 
Control Modernization Program, GAO/T-RCED/AIMD-99-137, (Washington, 
D.C.: Mar. 25, 1999); 
Selected findings: From the inception of its modernization efforts, FAA 
has not consistently followed a disciplined management approach for new 
acquisitions; 
Contributing factors: Weaknesses persist in key areas, such as how FAA 
monitors the status of its acquisitions throughout their life cycles.

Review: GAO, Air Traffic Control: FAA's Modernization Investment 
Management Approach Could Be Strengthened, GAO/RCED/AIMD-99-88, 
(Washington, D.C.: Apr. 30, 1999); 
Selected findings: AMS contained weaknesses in the selection of 
acquisitions and in the review of acquisitions' performance during the 
postimplementation phase; 
Contributing factors: FAA lacked adequate cost data for making 
selection decisions; adequate management controls, and a defined, 
documented process for conducting reviews during the in-service 
management phase.

Review: FAA, Evaluation of FAA Acquisition Reform--The First Three 
Years: April 1996 - March 1999, Report #1999-04,; (Washington, D.C.: 
May 28, 1999); 
Selected findings: FAA's cost and schedule plans were not on track, but 
performance plans were met; 
Contributing factors: Requirements changed or were misunderstood; 
technical difficulties were underestimated; and funding fell short.

Review: Booz-Allen & Hamilton, Independent Assessment of the Federal 
Aviation Administration's Acquisition Management System, (McLean, VA: 
July 6, 1999); 
Selected findings: FAA has yet to implement a seamless life-cycle 
approach to acquisitions management; 
Contributing factors: AMS is not being consistently implemented across 
all life-cycle phases.

Review: GAO, National Airspace System: Persistent Problems in FAA's New 
Navigation System Highlight Need for Periodic Reevaluation, GAO/RCED/
AIMD-00-130, (Washington, D.C.: June 12, 2000); 
Selected findings: FAA experienced delays and cost increases in 
developing its global positioning navigation system; as a result, it is 
unclear whether the benefits of the system will outweigh the cost; 
Contributing factors: FAA lacks a comprehensive plan with checkpoints 
for reviewing the contractor's approach to meeting the system's 
performance requirements.

Review: GAO, National Airspace System: Problems Plaguing the Wide Area 
Augmentation System and FAA's Actions to Address Them, GAO/T-RCED-00-
229, (Washington, D.C.: June 29, 2000); 
Selected findings: FAA experienced cost and schedule problems in 
developing this navigational system because of unplanned software 
development needs and a requirement to warn pilots of any system 
failure that would provide misleading information; 
Contributing factors: FAA underestimated the complexity of developing 
the acquisition.

Review: GAO, National Airspace System: Free Flight Tools Show Promise, 
but Implementation Challenges Remain, GAO-01-932, (Washington, D.C.: 
Aug. 31, 2001); 
Selected findings: Three acquisitions that are components of FAA's 
planned new approach for air traffic management have uncertain 
potential benefits and may not be worth FAA's investment; 
Contributing factors: FAA needs better data collection and analysis 
processes to ensure that benefits are realized.

Review: GAO, National Airspace System: Better Cost Data Could Improve 
FAA's Management of the Standard Terminal Automation Replacement 
System, GAO-03-343, (Washington, D.C.: Jan. 31, 2003); 
Selected findings: The reliability of the life-cycle cost estimate for 
STARS is uncertain because cost data obtained from the contractor do 
not reflect the current status of the contract; 
Contributing factors: The development cost estimate is based on the 
contractor's projections, which FAA has not yet independently analyzed, 
as called for under AMS.

Review: GAO, National Airspace System: Current Efforts and Proposed 
Changes to Improve Performance of FAA's Air Traffic Control System, 
GAO-03-542, (Washington, D.C.: May 30, 2003); 
Selected findings: FAA was unable to hire a chief operating officer to 
head the ATO; 
Contributing factors: Uncertainties about the position's 
responsibilities, reporting relationships, and performance measurement 
criteria hampered the hiring.

Review: DOT/OIG, Status of FAA's Major Acquisitions, AV-2003-045, 
(Washington, D.C.: June 26, 2003); 
Selected findings: Cost growth, schedule delays, and performance 
problems continue with FAA's major acquisitions; 
Contributing factors: Cost and schedule baselines are not reliable, and 
decisions are being made with unclear data.

Review: GAO, Air Traffic Control: FAA's Modernization Efforts--Past, 
Present, and Future, GAO-04-227T, (Washington, D.C.: Oct. 30, 2003); 
Selected findings: Systemic management issues, including inadequate 
management controls and human capital issues, have contributed to major 
ATC acquisitions' persistent cost overruns, schedule delays, and 
performance shortfalls; 
Contributing factors: FAA lacked the information technology and 
financial management systems that would have helped it reliably 
determine the acquisitions' technical requirements and estimate and 
control their costs and schedules; and the agency's organizational 
culture discouraged collaboration among technical experts and users.

Review: GAO, Information Technology: FAA Has Many Investment Management 
Capabilities in Place, but More Oversight of Operational Systems Is 
Needed, GAO-04-822, (Washington, D.C.: Aug. 20, 2004); 
Selected findings: Although weaknesses remain, FAA has established 
about 80 percent of the basic practices needed to manage its mission-
critical acquisitions so that it can be assured that it is selecting 
and managing the mix of investments that best meets its needs and 
priorities; 
Contributing factors: Remaining weaknesses include inadequate 
management controls and the lack of a defined, documented process for 
conducting reviews during the in-service management phase.

Review: GAO, Air Traffic Control: System Management Capabilities 
Improved, but More Can Be Done to Institutionalize Improvements, GAO-
04-901, (Washington, D.C.: Aug. 20, 2004); 
Selected findings: FAA made progress in improving its system management 
capabilities, but can do more to institutionalize process improvement 
initiatives; 
Contributing factors: Process improvement efforts have not been 
institutionalized. 

Source: GAO analysis.

[End of table]

FAA's ATO Is Taking Steps to Improve Major ATC Acquisitions: 

FAA's recent reorganization, which brought ATC acquisitions and 
operations together in the ATO,[Footnote 29] is expected to help the 
agency address many of the concerns we have identified for more than 
two decades, including those identified in this report. For example, 
the ATO is continuing to develop and refine specific guidance for 
critical areas, such as requirements management, software development, 
and cost estimation. In addition, as the overseer of both ATC 
acquisitions and operations, the ATO is in a position to facilitate 
more effective management of major ATC acquisitions than has occurred 
in the past. The ATO is attempting, for example, to link acquisition 
decisions directly with expected improvements in operational efficiency 
without compromising safety. This is important, given that FAA has 
spent about $2.5 billion on ATC modernization per year since 1996 while 
operating costs have continued to rise--from $4.6 billion to $7.5 
billion over the past decade. FAA had not completed its reorganization 
or implemented all of its initiatives at the time of our audit.

Improvements to Requirements Development: 

With the establishment of the ATO, FAA consolidated requirements 
development from two organizations (the organization sponsoring an 
acquisition and the former agencywide acquisition organization) into a 
single new organization--the Air Traffic System Requirements 
Service.[Footnote 30] In addition, the ATO developed guidance to better 
manage requirements during the middle phase of AMS (solution 
implementation). According to FAA officials, some more complex 
development efforts may need to develop systems requirements and a more 
detailed requirements document than AMS currently calls for in the 
final requirements document. More important, in January 2003, FAA 
issued guidance on requirements management, Roles in Requirements 
Management During Solution Implementation Phase, which provides for 
integrated requirements teams that maintain responsibility for 
requirements management throughout an acquisition's life cycle. 
According to this guidance, when the final requirements document is 
accepted by the Joint Resources Council at the investment decision 
point, a requirements baseline is established and any proposed changes 
to the requirements must be assessed for their impact on the program 
and shown to be operationally suitable, affordable, executable, and 
justifiable. An FAA official on an integrated requirements team stated 
that any changes that may affect an acquisition's cost and the schedule 
require approval by the Executive Committee. The FAA official also 
stated that this guidance has already helped to stabilize NEXCOM's 
requirements during the solution implementation phase. Other FAA 
officials representing the Joint Resources Council acknowledged that 
the guidance should ensure greater control over program requirements 
growth, but said that not all program offices have consistently applied 
it.

Improvements to Managing Software and System Acquisition and 
Development: 

To better manage software programs for ATC modernization acquisitions, 
FAA established a centralized process improvement office that reports 
to the Chief Information Officer (CIO).[Footnote 31] This office 
developed an FAA integrated capability maturity model (i-CMM), a 
software development and management model that is similar to a model 
developed by Carnegie Mellon University called the Capability Maturity 
Model Integration (CMMI®), which is used to appraise the maturity of an 
organization's processes for acquiring software. However, FAA's i-CMM 
goes beyond Carnegie Mellon's model to reflect international standards. 
The CMMI® appraisal methodology calls for assessing process areas--such 
as project planning, requirements management, and quality assurance--by 
determining whether key practices are implemented and overarching goals 
are satisfied. Both the i-CMM model and CMMI® appraisal methodologies 
provide a logical framework for measuring and improving key processes 
needed for achieving quality software and systems.

However, as we reported in August 2004,[Footnote 32] FAA projects are 
not required to use the capability maturity model for process 
improvement, and individual projects that use the i-CMM model are 
allowed to choose which process areas they seek to improve and to 
determine when they are ready for an appraisal of their progress. To 
date, fewer than half of FAA's major ATC projects have used this model. 
The recurring weaknesses we identified in our project-specific 
evaluations are due in part to the flexibility these projects were 
given in deciding whether and how to adopt this process improvement 
initiative. Furthermore, after combining its ATC organizations into a 
single performance-based organization (the ATO), FAA is reconsidering 
prior policies, and it is not yet clear whether process improvement 
will remain a priority. Without a strong senior-level commitment to 
process improvement and a consistent, institutionalized approach to 
implementing and evaluating it, FAA cannot ensure that key projects 
will continue to improve systems acquisition and development 
capabilities. As a result, FAA will continue to risk the project 
management problems--including cost overruns, schedule delays, and 
performance shortfalls--that have plagued past acquisitions. To address 
these shortcomings, we recommended that the Secretary of Transportation 
address specific weaknesses and institutionalize FAA's process 
improvement initiatives by establishing a policy and plans for 
implementing and overseeing process improvement initiatives.

Improvements to Estimating Costs: 

FAA has taken steps to improve its cost estimation for major ATC 
projects by issuing guidance on how to develop and use pricing under 
AMS. For example, AMS policy calls for audit trails to record and 
explain the values that are used as inputs to cost models. In addition, 
it calls for agency officials, when reporting to executive oversight 
agencies and Congress, to disclose the level of uncertainty and 
imprecision that are inherent in cost estimates for major ATC systems. 
According to AMS policy, estimators record the procedures, ground rules 
and assumptions, data, environment, and events that underlie their 
development or update of a cost estimate. This information supports the 
credibility of the cost estimate, aids in the analysis of changes in 
program costs, enables reviewers to assess the cost estimate 
effectively, and contributes to the population of FAA databases that 
can be used for estimating the cost of future programs. Finally, 
despite a delay of many years, FAA officials told us that they are in 
the final stages of completing the agency's cost-accounting system and 
plan to have it in place across the agency by the end of this calendar 
year, which will bring FAA into compliance with the Federal Managers' 
Financial Integrity Act of 1982. This measure will help reduce the 
likelihood of cost overruns or improper payments for unallowable costs 
and provide decision-makers with critical information. As we have 
reported in the past,[Footnote 33] a cost-accounting system is critical 
to managing major ATC acquisitions, because without it, FAA lacks the 
information it needs to reliably estimate operating costs over an 
acquisition's life cycle.

Other Improvement Efforts: 

In May 2004, the FAA Administrator testified to Congress that, to date, 
in attempting to improve the efficiency of ATC operations while 
maintaining safety, FAA had not managed its major ATC acquisitions to 
be aware of their cost implications for its operations. The 
Administrator said, however, that the agency was taking its first steps 
to fundamentally change how it makes acquisition decisions by adopting 
a more results-oriented approach. Under this approach, the agency plans 
to link its decisions to fund major acquisitions directly with their 
expected contribution to improving operational efficiency and 
controlling escalating operating costs. Whereas, in the past, FAA 
measured results in terms of its progress in completing and deploying a 
major ATC system, it was now going to focus on how a given system 
improved operational efficiency. Such an approach holds promise for 
helping FAA more effectively manage its portfolio of major ATC 
acquisitions by providing a sound basis for choosing among competing 
priorities. However, because FAA has only recently begun to incorporate 
this type of analysis of acquisitions' costs and operational efficiency 
into its decision-making and management processes, it is still too 
early to assess the results.

In addition, to its credit, FAA has created a training framework for 
its acquisition workforce, which we found mirrors human capital best 
practices that we have identified. In January 2003, we reported on 
FAA's efforts to define and train its workforce to meet the 
requirements of the Clinger-Cohen Act of 1996.[Footnote 34] This act 
required FAA and other civilian agencies to establish education, 
training, and experience requirements for their acquisition workforce. 
Our work on public and private best practices has identified six 
elements of training as critical to acquisition. These elements include 
(1) prioritizing the acquisition initiatives most important to the 
agency, (2) securing top-level commitment and resources, (3) 
identifying those who need training on specific initiatives, (4) 
tailoring training to meet the needs of the workforce, (5) tracking 
training to ensure it reaches the right people, and (6) measuring the 
effectiveness of training. These six elements are crucial for 
successfully implementing acquisition initiatives and reforms. 
Agencies that do not focus their attention on these critical elements 
risk having an acquisition workforce that is ill equipped to implement 
new processes. The probability of success is higher if training is 
well planned rather than left to chance. In 2003, we found that FAA's 
model for training its acquisition workforce largely mirrored public 
and private-sector best practices and that the agency had highly 
developed processes for four of these six elements. See figure 4.

Figure 4: Our Analysis of FAA's Progress as of 2003 in Implementing Key 
Elements of Training for Its Acquisition Workforce: 

[See PDF for image]

[End of figure]

Since 2003, FAA has taken some steps to measure the effectiveness of 
its training. For example, the agency collects and reviews 
participants' assessments of the knowledge they have gained, the extent 
that learning objectives were achieved and the applicability and 
usefulness of the training. In addition, members of FAA's Intellectual 
Capital Investment Plan Council[Footnote 35] have attempted to make 
qualitative judgments about the impact of the training on the 
effectiveness or efficiency of their organizations. However, FAA is 
still developing an evaluation program with metrics to measure the 
extent to which organizational goals are achieved when individual 
training objectives are met. Industry and government experts believe 
training and human capital investments are prerequisites for 
successfully introducing and implementing effective acquisition best 
practices. FAA's acquisition workforce plays a critical role in 
addressing long-standing weaknesses that we and others have identified 
with FAA's acquisition of major ATC systems. Given the importance of 
training for acquisition workforces, it will be important for the ATO 
to put mechanisms in place to comprehensively evaluate the 
effectiveness of the training it provides to improve the knowledge base 
of FAA's acquisition workforce.

To improve its investment management decision-making and oversight of 
major ATC acquisitions, the ATO also initiated the following 
procedures: 

* Integrate AMS and the Office of Management and Budget's Capital 
Planning and Investment Control Process[Footnote 36]to develop a 
process for analyzing, tracking, and evaluating the risks and results 
of all major capital investments made by FAA.

* Conduct Executive Council reviews of project breaches of 5 percent in 
cost, schedule, and performance to better manage cost growth;

* Issue monthly variance reports to upper management to keep them 
apprised of cost and schedule trends.

* Monitor progress in meeting the goals identified in FAA's Flight 
Plan, the agency's blueprint for action through 2008. The Executive 
Council tracks this progress monthly and reports to the Administrator, 
using a color-coded system to keep her apprised of how well FAA is 
meeting its goals. Green denotes that a goal will be met, yellow 
denotes that some of the activities leading to a main goal may be in 
jeopardy but the overall goal can be achieved, and red denotes serious 
concerns about reaching a goal without major intervention. A formal 
progress report is issued quarterly and made publicly available on the 
agency's Web site; and: 

* Increase the use of cost monitoring or earned value management 
systems[Footnote 37] to improve oversight of programs.

Despite FAA's current and planned efforts to improve its acquisition of 
major ATC systems under the ATO, given the newness of these efforts and 
the agency's poor track record in this area for more than two decades, 
it is critical for FAA to (1) modify AMS to more fully reflect the best 
practices followed by high-performing acquisition organizations, (2) 
follow through on planned improvement initiatives, and (3) adopt a 
continuous improvement approach to acquiring new ATC systems.

Conclusions: 

In the early 1990s, FAA contended that it needed relief from the FAR to 
remedy long-standing problems with cost, schedule, and performance 
shortfalls in its major ATC acquisitions; however, our work for more 
than two decades in this area has found that acquiring major ATC 
systems successfully depends more on managing an acquisition process 
well than on using a specific contracting process (e.g., the FAR or 
AMS). While our recent work has shown some improvement in FAA's 
management of major ATC system acquisitions, some key problems that 
existed before 1996 persist under AMS--including difficulty with 
clearly defining system requirements at the investment milestone and 
adequately assessing complex software requirements. These problems 
continue to make these acquisitions vulnerable to cost, schedule, and 
performance shortfalls. Without further measures to improve the 
development and management of requirements and to better estimate the 
complexity of the software development needed for major ATC systems, 
such shortfalls are likely to persist.

Although AMS provides some discipline for acquiring major ATC systems 
through its various phases, activities, and decision points, it does 
not require that (1) specific knowledge be attained using explicit 
written criteria and (2) corporate executive-level oversight be 
provided to determine--independently from the program offices--whether 
a system has reached a level of development (product maturity) 
sufficient to move forward in the acquisition process. Commercial best 
practices call for such knowledge-based decision-making at the 
corporate executive-level to help ensure that acquisitions are not 
moved into the development phase prematurely, to obtain greater 
predictability in ATC system program costs and schedules, to improve 
the quality of the ATC systems that are deployed, and to deliver new 
capability to the National Airspace System faster. A knowledge-based 
approach is also important because it provides assurance that agency 
decision-makers have critical information about an acquisition's 
ability to meet a mission need and FAA's readiness to move forward in 
the acquisition process before making large commitments of agency 
resources. Absent such an approach, FAA lacks assurance that it has 
obtained the critical technological, design, or manufacturing knowledge 
that best practices call for to avoid cost overruns, schedule slips, 
and performance shortfalls. As a result, FAA is not doing all that it 
can to systematically address persistent shortcomings in its management 
of major ATC acquisitions. Moreover, although FAA has established a 
framework for training its acquisition workforce under the ATO, it has 
not yet developed comprehensive performance criteria to evaluate how 
effectively it has implemented this framework. As a result, the agency 
lacks assurance that its use of this framework is having the intended 
effect of improving the knowledge base of this workforce.

Recommendations for Executive Action: 

We are making five recommendations to the Secretary of Transportation. 
To reduce the risk of persistent cost and schedule shortfalls in major 
ATC system acquisition programs, to improve the quality of the ATC 
systems that are deployed, and to deliver new capability to the 
National Airspace System faster, we recommend that the Secretary of 
Transportation advise the FAA Administrator to take the four following 
actions: 

* Modify AMS to specify that requirements be more clearly defined for 
major ATC systems, including providing more detailed guidance on 
setting clear, objective, and measurable requirements that reflect 
customers' needs, before making large investments of agency resources.

* Establish a strategy for identifying and measuring all additional 
development needed for complex software (e.g., commercial-off-the-
shelf or nondevelopmental items) used for major ATC systems.

* Develop explicit written criteria for the key decision points called 
for under best practices, including the capture of specific design and 
manufacturing knowledge.

* Require corporate executive-level decisions at these key decision 
points (before an acquisition moves from integration to demonstration 
and, again, before it moves to production).

In addition, to assure FAA that the training framework it has adopted 
for the ATO's acquisition workforce is improving the knowledge base of 
this workforce as intended, we recommend that the Secretary advise the 
Administrator to develop performance criteria to comprehensively 
evaluate the framework's effectiveness.

Agency Comments: 

We provided copies of a draft of this report to DOT for review and 
comment and met with Department and FAA officials, including the ATO's 
Vice President for Acquisition and Business Services, to obtain their 
comments. FAA officials told us that they have made great strides in 
improving their acquisition of major ATC systems under AMS; however, 
they recognize that there is room for improvement and are firmly 
committed to implementing best practices for acquisitions. These 
officials generally agreed with the report's findings and conclusions 
and said that our recommendations would be useful to them as they 
continue to refine their acquisition management system, including 
training their acquisition workforce. The agency provided us with oral 
comments, primarily technical clarifications, which we have 
incorporated as appropriate.

As agreed with your office, unless you publicly announce the contents 
of this report earlier, we plan no further distribution until 30 days 
from the report date. At that time, we will send copies of this report 
to interested congressional committees, the Secretary of 
Transportation, and the Administrator, FAA. We will also make copies 
available to others upon request. In addition, the report will be 
available at no charge on the GAO Web site at 
[Hyperlink, http://www.gao.gov].

Please call me at (202) 512-2834 if you or your staff have any 
questions concerning this report. Key contributors to this report are 
listed in appendix VI.

Signed by: 

JayEtta Z. Hecker: 
Director, Physical Infrastructure Team: 

[End of section]

Appendixes: 

Appendix I: FAA Has Begun Analyzing Spending Trends to Take a More 
Strategic Approach to Procurement: 

Our review of the Federal Aviation Administration's (FAA) general 
procurement of goods and services focused on the Air Traffic 
Organization (ATO) and its predecessor offices. According to FAA 
officials, the ATO has recently begun to consider ways to better 
leverage its buying power by taking a more strategic approach to 
procurement. While FAA uses the Acquisition Management System (AMS) for 
all FAA acquisitions, including the procurement of such goods and 
services as office supplies, computers, telephone services, and 
engineering and technical support services, these procurement 
activities take place in a decentralized environment of independent, 
transaction-oriented buying processes. Each FAA unit determines its 
need for goods and services and procures them as necessary, leaving 
headquarters with limited oversight of the agency's total procurement 
spending. For example, in 2003, FAA units carried out over 346,000 
procurement actions for goods and services and purchase 
cardholders[Footnote 38] made an additional 335,000 transactions. This 
fragmented environment does not permit the agency to leverage its 
buying power through lower-cost, consolidated contracts, at the local, 
regional, or national level and to rationalize the number of suppliers 
best suited to meet the agency's needs. At the same time, as part of a 
strategic procurement effort, FAA can use spend analysis to monitor 
trends in small and disadvantaged business participation so that it can 
balance the goals of lower-cost contract consolidation and promoting 
small business contracting opportunities.

Spend analysis, a tool used in a strategic approach to procurement, 
provides knowledge about how much is being spent for what goods and 
services, who the buyers are, who the suppliers are, and where the 
opportunities are to leverage buying power. Our past work[Footnote 39] 
shows that private companies are using spend analysis as a foundation 
for employing a strategic approach to procurement. The analysis 
identifies where numerous suppliers are providing similar goods and 
services--often at varying prices--and where purchasing costs can be 
reduced and performance improved by better leveraging buying power and 
reducing the number of suppliers to meet the company's needs. Our 
research on commercial best practices has found that spend analysis is 
an important driver of strategic planning and execution. As part of an 
overall strategic procurement effort, companies use spend analysis to 
(1) define the magnitude and the characteristics of their spending, (2) 
understand their internal clients and supply chain, (3) create lower-
cost consolidated contracts, and (4) monitor spending with small and 
disadvantaged businesses to achieve socioeconomic procurement goals.

We previously reported[Footnote 40]that six agencies, including DOT, 
did not take advantage of opportunities to obtain more favorable prices 
on purchase card buys with frequently used vendors--vendors where an 
agency spends more than $1 million annually. In these six agencies, 
which accounted for over 85 percent of federal government purchase card 
spending, frequently used vendors accounted for purchases totaling 
nearly $3 billion in 2002. We recommended several actions--including 
conducting spend analysis using available data and gathering additional 
information where feasible--that could ultimately help these agencies 
achieve $300 million annually in potential savings.

In fiscal year 2003, FAA procured nearly $4 billion in goods and 
services and spent an additional $132 million using purchase cards. 
According to senior FAA officials, the agency has just begun to 
implement a strategic approach to general procurements. Other federal 
agencies are beginning to use strategic tools such as spend analysis to 
improve their spending for goods and services, and some have 
initiatives under way to obtain more favorable prices on purchase card 
buys. According to a senior FAA acquisition official, FAA has to 
balance the need of its units to independently make purchases that 
pertain solely to unit requirements with the agency's need to aggregate 
purchases of goods and services that are used by more than one unit. 
FAA has hired a consultant to help begin the use of spend analysis. 
This effort could reduce the agencywide costs for mobile wireless 
services by 40 percent--an effort expected to save the agency $8 to $10 
million annually. FAA intends to expand its use of spend analysis to 
target other procurement category savings opportunities, including 
information technologies, training, facilities, and professional 
services, as its accounting systems improve.

FAA has taken some preliminary steps to set up a spend analysis 
program; however, progress has been challenging for FAA because of 
deficiencies in its accounting systems. For example, because the 
agency's accounting system did not identify all of the mobile wireless 
services for which it was being billed, the contractor implementing the 
spend analysis had to obtain this information from the wireless 
providers. FAA will need to expedite its efforts in this area to fully 
realize potential savings. Our prior research has shown that setting up 
a spend analysis program can be challenging. Companies have had 
problems accumulating sufficient data from internal financial systems 
that do not capture information on all of what a company buys or is 
using in different, unconnected parts of the company. Despite these 
challenges, companies that have developed formal, centralized spend 
analysis programs have been able to track their costs and identify 
areas for strategic sourcing and savings opportunities.

In our recent report on spend analysis,[Footnote 41] we found that DOT, 
at the time of our review, had not yet begun to collect the data needed 
for a strategic approach to procurement; however, the department is 
engaged in ongoing efforts to improve procurements, and its top 
leadership is committed to using spend analysis to change the way goods 
and services are purchased. One obstacle to using spend analysis that 
the department cited during our review was a lack of comprehensive and 
reliable spending data. However, since we completed our review, the 
department reports stepping up efforts to use currently available data 
and evaluate business intelligence software to overcome those 
obstacles. In commenting on our report, Transportation's senior 
procurement executive told us that the department is expanding its 
spend analysis efforts. For example, his office recently reviewed 
purchase card spending data to identify volume discount opportunities 
and is now using the results to negotiate new discount agreements with 
several office product vendors. In addition, he told us that to 
facilitate future agencywide purchase card spend analyses, DOT awarded 
a task order in June 2004 to one bank card company that will provide 
purchase-card audit software and enhanced data-mining capabilities. He 
also indicated that the department's leadership supports fiscal year 
2005 funding to enhance spend analysis capabilities and that software 
options for the new agencywide spend analysis system are now being 
evaluated as part of an ongoing financial and procurement review.

[End of section]

Appendix II: Objectives, Scope, and Methodology: 

To compare FAA's Acquisition Management System (AMS) with the Federal 
Acquisition Regulation (FAR), we reviewed AMS and changes in it over 
time. We also compared FAA's acquisition authority under the FAR and 
under AMS. In addition, we identified relevant recommendations from 
reports that we, the Department of Transportation's Inspector General 
(DOTIG), and others have issued to determine which recommendations have 
been implemented, rejected, or left open, and to evaluate how those 
recommendations have modified FAA's acquisition policies and practices. 
We also collected and summarized published reports and analyzed 
available life-cycle management data on the current status of major and 
nonmajor acquisitions being carried out under AMS.

To determine the ways in which FAA's acquisition policies compare with 
our best practices model, we used information from several of our 
products that examine how commercial best practices can improve 
outcomes for acquisition programs. This model consists of four phases: 
(1) concept and technology development; (2) product development, which 
includes both integration and demonstration activities; (3) production; 
and (4) operations and support. In between these four phases are three 
key knowledge points at which commercial firms must have sufficient 
knowledge to make large investment decisions. We also reviewed and 
analyzed AMS, accessible at h [Hyperlink, http://fast.faa.gov] ttp: //
fast.faa.gov. Furthermore, to clarify the content of FAA's acquisition 
process, we met with various FAA vice-presidents and officials from 
FAA's Acquisition Planning and Policy Division. Next, we compared and 
contrasted FAA's acquisition policies with the best practices for 
commercial acquisitions identified in our past reports. Our analysis 
focused on whether FAA's policies contained the measurable criteria and 
management controls necessary to achieve FAA's intent of minimizing 
cost, schedule, and performance risks. We also interviewed current and 
former FAA procurement officials that have experience using both the 
FAR and AMS.

To determine if FAA has effectively implemented its new acquisition 
authority and improved its acquisition outcomes, we reviewed seven of 
FAA's most expensive major ATC acquisitions, including the Airport 
Surveillance Radar 11 (ASR-11), Standard Terminal Automation 
Replacement System (STARS), Integrated Terminal Weather System (ITWS), 
Local Area Augmentation System (LAAS), Next Generation Air/Ground 
Communications System (NEXCOM), Advanced Technologies and Oceanic 
Procedures (ATOP), and En Route Automation Modernization (ERAM). See 
table 7 for specific program costs.

Table 7: Program Costs for the Seven Systems We Reviewed: 

Dollars in millions.

Program: STARS; 
Total program cost as of 9/30/04: $1,460.0.

Program: ASR-11; 
Total program cost as of 9/30/04: $891.7.

Program: ITWS; 
Total program cost as of 9/30/04: $288.3.

Program: LAAS; 
Total program cost as of 9/30/04: $696.1.

Program: NEXCOM; 
Total program cost as of 9/30/04: $318.4.

Program: ATOP; 
Total program cost as of 9/30/04: $548.2.

Program: ERAM; 
Total program cost as of 9/30/04: $2,154.6.

Total; 
Total program cost as of 9/30/04: $6,357.3.

Source: GAO analysis.

Note: These amounts are for facilities and equipment only (not 
operations and maintenance).

[End of table]

We also selected these seven acquisitions because we considered them to 
fall into two basic categories-pre-AMS and post-AMS. Five of the 
acquisitions were initiated before AMS was implemented in April 1996 
and were transitioned into AMS at various times before their 
completion. The two remaining acquisitions--ATOP and ERAM--were 
initiated and have remained completely under AMS. We then reviewed 
program documents and reports and interviewed program and agency 
officials responsible for developing these acquisitions, as well as 
other acquisitions experts in the private sector. For some 
acquisitions, we discussed programmatic issues with representatives of 
the primary contractor for the specific acquisition to obtain 
information on the practices and procedures used for the acquisition. 
In addition, we interviewed some current and former FAA procurement 
officials with experience using both the FAR and AMS to obtain their 
views on the use of each contracting process and how the two compare. 
Furthermore, to see how FAA has progressed in addressing problems with 
its acquisitions, we reviewed our work on acquisitions over the last 20 
years, as well as reports by the DOTIG, FAA, Booz-Allen & Hamilton, and 
MITRE. Because the data in this report on cost, schedule and 
performance are used as background information or to otherwise provide 
a description of acquisitions, we did not assess their reliability.

The effect of the current budget process on FAA's ability to 
successfully modernize the National Airspace System, including 
acquiring major ATC systems is not within the scope of this review.

[End of section]

Appendix III: Comparison of the Scope and Flexibility of FAA's 
Acquisition Management System and the Federal Acquisition Regulation 
Process: 

Background: 

FAA's business processes, including its acquisition of major systems, 
differ significantly from the business processes followed by most other 
federal agencies. FAA relies on its Acquisition Management System 
(AMS), which establishes FAA internal acquisition policy. AMS resulted 
from the adoption of language in the Department of Transportation and 
Related Agencies Appropriations Act,[Footnote 42] which directed the 
FAA Administrator to develop and implement an acquisition management 
system for FAA. The adoption of this language (section 348) followed 
FAA's assertions that the requirement that it conduct procurements in 
accordance with the Federal Acquisition Regulation (FAR) was at least a 
contributing factor in its repeated failure to complete air traffic 
control (ATC) and other modernization programs on schedule. The 
Administrator was directed to put in place a system that would address 
the "unique needs of the agency" that FAA contended prevented its 
acquisitions from being timely and cost-effective.

Section 348 distinguished FAA from other federal agencies by removing 
FAA from the federal acquisition system. Under section 348, FAA was no 
longer subject to title III of the Federal Property and Administrative 
Services Act of 1949,[Footnote 43] which among other things requires 
that the government procure supplies and services competitively. It 
removed FAA as an agency subject to the Office of Federal Procurement 
Policy Act[Footnote 44] and eliminated the requirement that FAA comply 
with the FAR. While mandating that FAA conduct its acquisitions so that 
"all reasonable opportunities to be awarded contracts shall be provided 
to small business concerns and small business concerns owned and 
controlled by socially and economically disadvantaged individuals," 
section 348 eliminated the requirement that FAA comply with the Small 
Business Act.[Footnote 45] Furthermore, it made the procurement protest 
system of the U.S. Government Accountability Office inapplicable to 
FAA, although disappointed offerors can still file protests with FAA's 
Office of Dispute Resolution for Acquisition.[Footnote 46]

AMS Defines an Investment/Life-Cycle Project Management System: 

Much of AMS guidance concerns project, financial, and property life-
cycle management issues. In fact, FAA's policy describes AMS as 
applying to all investment programs regardless of cost or the 
appropriation funding them. It recognizes that a single investment 
program may span multiple procurements and projects. It applies, 
according to its terms, to the activities associated with needs 
analysis, determination of requirements, analysis of investment 
alternatives, establishment of investment programs, allocation and 
expenditure of resources, procurement and deployment of needed products 
and services, in-service management of fielded capability, and eventual 
disposal of obsolete products.

AMS focuses on the following key program milestones: 

* Mission Analysis--encompasses those key corporate and service-level 
processes that define, coordinate, and integrate the work of service 
organizations,[Footnote 47] thereby providing strategic direction to 
keep FAA responsive to the service needs of its customers. Mission 
analysis is used to update a mission need statement, which in turn may 
identify capability shortfalls or technological opportunities, that is, 
unmet needs. Unmet needs are presented to the Joint Resources Council 
(JRC) for a mission need decision. To be approved, the unmet need 
should be supported by the updated mission need statement and the 
initial requirements document, including a concept of use, and the 
initial investment plan.

* Investment Analysis--builds on the results of the mission need 
decision by developing detailed plans and final requirements for each 
proposed investment program and by defining an acquisition program 
baseline that establishes cost, schedule, performance, benefit, and 
risk-management boundaries for the program. AMS calls for planning the 
entire solution--an effort that may use market survey data but is based 
in large measure on FAA's assumptions and data. The service 
organization produces a final implementation and life-cycle support 
strategy. A detailed program plan and an acquisition program baseline 
are also produced. The results are presented to the JRC for a "final 
investment decision."[Footnote 48]

* Solution Implementation--encompasses acquiring, accepting, 
deploying, installing and preparing for the operational use of an 
approved investment. Approval of the investment carries with it 
authorization for the service organization to conduct all acquisitions 
needed to execute the investment decision, subject to any constraints 
established in the final investment decision.

* In-Service Decision--is an FAA system qualification milestone, which 
is achieved when an otherwise operational investment is satisfactorily 
tested to demonstrate its operational effectiveness and suitability 
before it is placed in service in the National Airspace System. The JRC 
designates the decision maker.

* In-Service Management--covers activities throughout a system's life 
cycle, starting at the time that an investment becomes operational. In-
service product improvements may eliminate latent defects, fix systemic 
problems, and enhance the utility of the investment. These changes may 
be made within the approved acquisition program baseline without 
corporate-level approval. In-service management also includes 
planning, programming, and developing supporting budget input; 
monitoring and assessing performance, cost of ownership, and support 
trends; and planning for service-life investment decisions.

* Service Life Extension--seeks a new investment decision by the JRC 
when a current capability is unable to satisfy demand or when another 
solution may be more effective. The JRC can decide to revalidate the 
mission need satisfied by the solution by upgrading or refurbishing 
fielded capability or by replacing that capability with another 
equivalent or new superior solution. The JRC may also decide that the 
capability should be retired.

Only a Portion of AMS Deals Directly with the Procurement Process: 

Although the FAR includes requirements addressing procurement planning 
and major system acquisition, AMS as just outlined differs 
significantly from the FAR in its focus and scope. The FAR addresses 
planning[Footnote 49] and major system acquisition[Footnote 50] in the 
context of government procurement policy and procedure. Agencies other 
than FAA find the broader program planning and management issues 
addressed in AMS outside of the FAR, in documents such as the Office of 
Management and Budget's (OMB) Circular A-109, in their own planning 
guidance, such as the Department of Defense's (DOD) 5000 
series,[Footnote 51]and in established knowledge-based best practices. 
As indicated earlier, much of AMS focuses on just such issues. Only AMS 
section 3 addresses procurement policy and procedure.[Footnote 52]

AMS States a Nonregulatory FAA Policy: 

A further significant foundational difference between AMS and the FAR 
is that AMS sets out a nonregulatory FAA policy, whereas the FAR was 
adopted and is maintained as a set of published governmentwide 
regulatory requirements, which form a legal basis for federal agencies' 
contract decision-making. AMS is binding on FAA personnel as FAA 
employees and establishes other guidelines that FAA states should be 
followed unless there is a rational basis for doing otherwise. AMS is 
subject to such internal controls as the Administrator chooses to 
enforce and general overarching legal requirements, such as the 
Government Performance and Results Act of 1993 (GPRA).[Footnote 53] 
There is a legal requirement, created by section 348, that small and 
socially or economically disadvantaged firms be given all reasonable 
opportunities to receive contract awards. FAA in its Office of Dispute 
Resolution for Acquisition has adopted a dispute resolution process 
with some legal underpinnings.[Footnote 54] Otherwise, as the preface 
to AMS states, "nothing in this document creates or conveys any 
substantive [legal] rights." In short, FAA has assumed no legal 
obligation to follow AMS other than to ensure that its actions are not 
arbitrary and capricious or contrary to law. By contrast, the FAR has 
the force and effect of law, and agencies that are subject to the FAR 
are bound to follow it.

AMS Chapter 3 Parallels a Subset of the FAR: 

When FAA personnel apply the procurement methodology in AMS chapter 3, 
they are applying guidance that closely parallels some of the 
procedures set out in the FAR. The AMS Chapter 3 acquisition process 
parallels a subset of the varied selection of procurement methods 
available under the FAR, requiring that all competitive FAA contracts 
be negotiated with the awardee being selected on a "best value" basis. 
The FAR also provides a much more detailed set of information and 
guidance than does AMS. A comparison of high-level differences and 
similarities between AMS and the FAR is presented in table 8.

Table 8: Comparison of AMS and the FAR: 

Best value source selection; 
AMS: Yes, following screening; 
FAR: Yes, although other methods are also available for use when 
appropriate.

Public announcement of requirement; 
AMS: Public announcement through Internet or other means when value of 
contract is anticipated to exceed $100,000; 
FAR: Yes, for proposed contract actions expected to exceed $25,000.

Competition; 
AMS: FAA's policy is to provide reasonable access to competition for 
firms interested in obtaining contracts. In selecting sources, the 
preferred method of procurement is to compete requirements among two or 
more sources; 
FAR: Full and open competition--all responsible sources are permitted 
to compete.

Sole-source procurement; 
AMS: Yes, when deemed to be in FAA's "best interest" as determined by 
the service organization on the basis of "adequate objective supporting 
data."; 
FAR: Yes, full and open competition need not be obtained under certain 
specified conditions based upon a written justification from the 
contracting officer that is approved at an appropriate level of 
authority.

Prequalification; 
AMS: Yes, qualification information screens for those vendors that meet 
FAA's stated minimum capabilities or requirements for providing a given 
product or service; 
FAR: Yes, for products or manufacturers when justified in writing and 
conducted in a manner that meets requirements justifying the use of 
qualifications requirements.

Basic methodology in negotiated procurement; 
AMS: FAA issues one or more "screening requests," which may include
requests for binding offers from competing firms; 
FAR: Agency issues a solicitation, usually a request for proposals.

Methodology for negotiation; 
AMS: FAA encourages one-on-one communications throughout the process 
provided that no offeror is given an "unfair advantage"; 
FAR: Clarification and discussions are permitted; 
one offeror cannot be favored over another.

Evaluation and award selection; 
AMS: Selection is based on evaluation in accord with criteria 
identified in the screening request. The selection decision is a 
judgmental decision made by the source selection official; 
FAR: Selection is based on evaluation in accord with criteria 
identified in the request for proposals. The selection decision is a 
judgmental decision made by the source selection official.

Use of simplified acquisition methods; 
AMS: Commercial and simplified purchases are used for commercial items 
or for products or services that have been sold at established catalog 
or market prices and are generally purchased on a fixed-price basis; 
FAR: Generally required for purchases up to $100,000, for noncommercial 
items, or on a test basis, up to $5,000,000 for commercial items 
competition is to be obtained to the maximum practicable extent.

Use of credit card purchases; 
AMS: Permitted; 
FAR: Permitted.

Procurement methodology; 
AMS: AMS does not include the level of detail found in the FAR. It does 
not prescribe many of the procurement methods and techniques permitted 
under the FAR, but encourages use of "any method of procurement deemed 
appropriate."; 
FAR: Provides a broad selection of procurement methods and techniques 
suitable for use in most circumstances.

Responsibility; 
AMS: Awards to responsible offerors only; 
FAR: Awards to responsible offerors only. 

Source: GAO analysis.

[End of table]

AMS Includes a Less Rigorous Competition Requirement Than Does the FAR: 

As table 8 indicates, AMS incorporates a less rigorous competition 
standard than the FAR imposes on the rest of the government. AMS states 
that it is FAA's policy to provide reasonable access to competition for 
firms interested in obtaining contracts. According to AMS, in selecting 
sources, the preferred method of procurement is to compete requirements 
among two or more sources. However, there is no requirement to ensure 
that firms that want to participate actually get a chance to do so. 
Instead FAA may limit competition for further consideration in its 
screening process to firms with known capabilities or past performance.

The FAR Gives Procurement Professionals Tighter Control over 
Procurement Decisions: 

AMS states that authority is delegated to appropriate levels. Once the 
final investment decision is made, and subject only to any constraints 
imposed by that decision, the service-level organization is responsible 
for conducting required acquisitions. Contracting personnel as well as 
other specialists are then assigned to teams that are responsible to a 
program manager within the service-level organization. FAA states that 
this approach increases the pace of doing business. By comparison, the 
FAR gives contracting professionals clear control over contracting 
decisions by requiring that procurement decisions be made by 
procurement professionals--typically contracting officers or their 
superiors.

Although FAA Project Managers View AMS as More Efficient and Flexible 
Than the FAR, Some Procurement Officials We Interviewed Do Not Agree: 

As part of our work, we interviewed project management personnel within 
FAA as well as current and former FAA procurement officials that have 
experience using both the FAR and AMS. Generally, FAA personnel see AMS 
as more efficient and flexible than the FAR, although 9 years after 
AMS's adoption, many FAA officials have only limited knowledge of and 
experience with the FAR. The FAA project managers we interviewed see 
AMS as more efficient and flexible than the FAR,[Footnote 55] but some 
procurement officials with experience in applying both AMS and the FAR 
did not agree with the view that the FAR was unduly rigid. According to 
these officials, the FAR may appear inflexible and cumbersome to 
persons who are inexperienced with it, but those who are familiar with 
it are able to navigate its complexities effectively. For example, even 
though the FAR generally requires full and open competition--a process 
that can take time to give all interested firms an opportunity to 
participate--contracting officers may be able to expedite the 
procurement process by using authorized streamlined procedures or, if 
circumstances warrant, by justifying sole-source or limited 
competition.

[End of section]

Appendix IV: How FAA's Acquisition Policy Adapted Key Recommendations 
Made by GAO and DOT (1996-2003): 

FAA Refined AMS in Response to Recommendations: 

Since FAA developed and implemented AMS in 1996, GAO and the DOTIG have 
made recommendations to improve FAA's acquisition processes. FAA has 
adopted many of these recommendations and incorporated them into AMS 
(see table 9). These implemented recommendations address four main 
themes: 

* Developing a strategy for culture change that relies on successfully 
integrating the various elements of acquisition, including specific 
responsibilities and performance measures for all stakeholders, and 
providing the incentives needed to promote the desired changes.

* Establishing an effective management structure for developing, 
maintaining, and enforcing the ATC systems architecture to provide an 
overall plan for the National Airspace System (NAS). This management 
structure should assign the responsibility and accountability to 
develop, maintain, and enforce a complete and unified ATC system by 
ensuring that every project conforms to the overall plan.

* Improving cost and schedule tracking to provide data for estimating 
the costs and schedules of programs. To estimate the costs and time 
needed for projects, a historical database that includes cost and 
schedule estimates, revisions, reasons for revisions, actual cost and 
schedule information, and relevant contextual information is needed.

* Improving the management of modernization projects, including the use 
of project reviews, milestones, and baselines, and cost-accounting 
information to ensure that programs can be adjusted as needed.

The reports identified in table 10 provide recommendations to address 
problems we and the DOTIG have identified under these four themes.

Table 9: Key Recommendations Made to Improve FAA's Acquisition 
Processes: 

Key recommendation: Aviation Acquisition: A Comprehensive Strategy Is 
Needed for Cultural Change at FAA; August 22, 1996, (GAO/RCED-96-159); 
FAA should develop a comprehensive strategy for cultural change. This 
strategy should include specific responsibilities and performance 
measures for all stakeholders throughout FAA and provide the incentives 
needed to promote the desired behaviors and to achieve agencywide 
cultural change;

Evidence of policy change: FAA issued an organizational culture 
framework in 1997 and is working to implement it;

Rationale for change: Over the past 15 years, FAA's ATC modernization 
projects have experienced substantial cost overruns, lengthy schedule 
delays, and significant performance shortfalls. We found that FAA's 
organizational culture has been an underlying cause of the agency's 
acquisition problems. Its acquisitions were impaired because employees 
acted in ways that did not reflect a strong commitment to mission 
focus, accountability, coordination, and adaptability.

Key recommendation: Air Traffic Control: Improved Cost Information 
Needed to Make Billion Dollar Modernization Investment Decisions; 
January 22, 1997, (GAO/AIMD-97-20); Because the success of FAA's 
investment analysis and decision-making process depends in large 
measure on the reliability of ATC project cost information, FAA should 
institutionalize defined processes for estimating ATC projects' costs. 
At a minimum, these processes should include the following six 
institutional process requisites, developed for organizations that are 
building or acquiring software-intensive systems by Carnegie Mellon 
University's Software Engineering Institute (SEI), an institution 
recognized for its expertise in software processes. Each of these 
requisites is described in more detail in this report:

* a corporate memory, or historical database(s), which includes cost 
and schedule estimates, revisions, reasons for revisions, actual cost 
and schedule information, and relevant descriptive information;

* structured approaches for estimating software size and the amount and 
complexity of existing software that can be reused;

* cost models calibrated/ tuned to reflect demonstrated accomplishments 
on past projects;

* audit trails that record and explain all values used as cost model 
inputs;

* processes for dealing with externally imposed cost or schedule 
constraints in order to ensure the integrity of the estimating process;

* data collection and feedback processes that foster capturing and 
correctly interpreting data from work performed; [Empty];

Evidence of policy change: Chapter 19 of FAA's Pricing Handbook 
embodies SEI's philosophy, which maintains that developing credible 
software estimates is a function of how thorough and disciplined an 
organization's estimating processes are. SEI's six institutional 
process requisites are designed to ensure that organizations 
consistently produce reliable cost estimates for software-intensive 
systems. These requisites are as follows:

* a corporate memory, or historical database(s), for cataloging cost 
estimates, revisions, reasons for revisions, actual cost and schedule 
information, and other descriptive information, such as any constraints 
or trends that affect the project;

* structured processes for estimating software size and the amount and 
complexity of existing software that can be reused;

* cost models calibrated/tuned to reflect demonstrated accomplishments 
on similar past projects;

* audit trails that record and explain the values used as cost model 
inputs;

* processes for dealing with externally imposed cost or schedule 
constraints to ensure the integrity of the estimating process;

* data collection and feedback processes that foster capturing and 
correctly interpreting data from work performed;

Rationale for change: We found that FAA's ATC modernization program's 
cost estimating processes do not satisfy recognized estimating 
requisites, and its cost-accounting practices do not provide for proper 
accumulation of actual costs. The result is an absence of reliable 
project cost and financial information that the Congress has 
legislatively specified and that leading public-sector and private-
sector organizations point to as essential to making fully informed 
investment decisions among competing ATC projects. Not having this 
information, increases the likelihood of poor ATC investment decisions, 
not only when a project is initiated but also throughout its life 
cycle. It also means that Congress does not have reliable cost 
information to use in making funding decisions about FAA. Such a 
situation is unacceptable when making small investments, but is 
especially egregious when making multimillion or billion-dollar 
investments in mission-critical ATC systems.

Key recommendation: FAA should immediately begin disclosing the 
inherent uncertainty and range of imprecision in all ATC projects' 
official cost estimates presented to executive oversight agencies or 
Congress;

Evidence of policy change: Chapter 19 of FAA's Pricing Handbook 
incorporates our recommendation and refers explicitly to GAO/AIMD-97-20 
and the work of other experts. The handbook suggests where to 
incorporate audit trails, constraint processes, and the inherent 
uncertainty and range of imprecision in all ATC cost estimates. The 
handbook advocates that staff qualify early project estimates by 
disclosing the level of uncertainty associated with them and refining 
the estimates as the project is completed and the uncertainty 
eliminated.

Key recommendation: FAA should acquire or develop and implement a 
managerial cost-accounting capability that will satisfy the 
requirements of Statement of Federal Financial Accounting Standards no. 
4 (SFFAS 4) Managerial Cost Accounting Concepts and Standards for the 
Federal Government. This system capability should provide the cost-
accounting and financial management information needed by FAA 
management and those who make investment decisions. Such information 
should include full life-cycle costs, which include the costs of 
resources consumed by a project that directly or indirectly contribute 
to the output and the costs of identifiable supporting services 
provided by other organizations within the reporting entity;

Evidence of policy change: The Department of Transportation is in the 
process of meeting key objectives of the Federal Managers' Financial 
Integrity Act (FMFIA) of 1982. A key material weakness was FAA's 
oversight of cost reimbursable contracts. FAA made significant progress 
in the closeout of past cost reimbursable contracts. To resolve this 
material weakness, FAA needs to complete the close out of old contracts 
and increase the use of cost incurred audits. Additionally, FAA needs 
to ensure that appropriate audits are obtained for all active 
contracts. These steps will help reduce the likelihood of cost overruns 
or improper payments for unallowable costs.

Key recommendation: FAA should report its lack of a cost-accounting 
capability for its ATC modernization as a material internal control 
weakness in the Department's fiscal year 1996 Federal Managers' 
Financial Integrity Act (FMFIA) report and in subsequent annual FMFIA 
reports until the problem is corrected.

Key recommendation: FAA should report to the Secretary of 
Transportation and FAA's authorizing and appropriation committees on 
its progress in implementing these recommendations as part of its 
fiscal year 1999 budget submission.

Key recommendation: Air Traffic Control: Complete and Enforced 
Architecture Needed for FAA Systems Modernization; February 3,1997, 
(GAO/AIMD-97-30); FAA should ensure that a complete ATC systems 
architecture is developed and enforced expeditiously before deciding on 
the architectural characteristics of a replacement for the Host 
Computer System. FAA should also take the following steps to establish 
an effective management structure for developing, maintaining, and 
enforcing the complete ATC systems architecture:

* Assign the responsibility and accountability needed to develop, 
maintain, and enforce a complete ATC systems architecture to a single 
FAA organizational entity;

* Provide this single entity with the resources, expertise, and 
budgetary and/or organizational authority needed to fulfill its 
architectural responsibilities;

* Direct this single entity to ensure that every ATC project conforms 
to the architecture unless careful, thorough, and documented analysis 
supports an exception. Given the importance and the magnitude of the IT 
initiative at FAA, a management structure similar to the department-
level chief information officer (CIO) structure prescribed in the 
Clinger-Cohen Act should be established for FAA;

Evidence of policy change: AMS states the National Air Space (NAS) 
Configuration Control Board shall approve changes to NAS technical 
documentation, and shall ensure the traceability of requirements from 
the NAS level to the system and subsystem level. This responsibility 
begins with the approval of the technical architecture by the Joint 
Resources Council at the investment decision and continues throughout 
the life of the program; AMS states that the Joint Resources Council 
approves FAA budget submissions for Research, Engineering and 
Development (RE&D) and Facilities and Equipment (F&E) appropriations, 
participates in the development of FAA budget submissions for the 
operations appropriation, and approves the NAS architecture baseline; 
AMS states that a configuration control board with an approved charter 
and operating procedures shall be the official FAA-wide forum used to 
establish configuration management baselines and to approve or 
disapprove subsequent changes to those baselines;

Rationale for change: FAA lacks a complete system architecture, or 
overall blueprint, to guide and constrain the development and 
maintenance of the many interrelated systems that make up its ATC 
infrastructure. To its credit, FAA is developing one of the two 
principal components of a complete systems architecture, namely, the 
"logical" description of FAA's current and future concept of ATC 
operations as well as descriptions of the ATC business functions to be 
performed, the associated systems to be used, and the information flows 
among systems. However, FAA is not developing, nor does it have plans 
to develop, the second essential component--the ATC-wide "technical" 
descriptions that define all required information technology (IT) and 
telecommunications standards and critical ATC systems' technical 
characteristics; We also found that an architecture is the centerpiece 
of sound systems development and maintenance; FAA is developing a 
logical architecture component for ATC modernization and evolution; FAA 
lacks a technical architectural component to guide and constrain ATC 
modernization and evolution; without a technical ATC architecture, 
costly system incompatibilities have resulted and; will continue; and 
FAA lacks an effective management structure for developing and 
enforcing an ATC systems architecture.

Key recommendation: Air Traffic Control: Immature Software Acquisition 
Processes Increase FAA System Acquisition Risks; March 21, 1997, (GAO/ 
AIMD-97-47); Given the importance and the magnitude of IT at FAA, this 
report reiterates our earlier recommendation calling for the 
establishment at FAA of a CIO management structure similar to the 
department-level CIO structure prescribed in the Clinger-Cohen Act of 
1996. To improve its ability to acquire software for its ATC 
modernization, FAA should;

* assign responsibility for software acquisition process improvement to 
the agency's CIO;

* provide the CIO with the authority needed to implement and enforce 
ATC modernization software acquisition process improvement;

* require the CIO to develop and implement a formal plan for ATC 
modernization software acquisition process improvement that is based on 
the software capability evaluation results contained in this report and 
specifies measurable goals and time frames, prioritizes initiatives, 
estimates resource requirements, and assigns roles and 
responsibilities;

* allocate adequate resources to ensure that planned initiatives are 
implemented and enforced; and;

* require that, before being approved, every; ATC modernization 
acquisition project have software acquisition processes that satisfy at 
least Software Acquisition Capability Maturity Model (SA-CMM) level 2;

Evidence of policy change: FAA states that the CIO:

* serves as the principal adviser to the Administrator, Deputy 
Administrator, and FAA offices on information management and technology 
across the agency. As the agency's senior management official, serves 
as the spokesperson on IT matters before Congress, other agencies, and 
the public;

* leads and directs agencywide strategic planning for IT;

* oversees IT investments to ensure optimization across all agency 
groups and the full range of cost trade-offs;

* creates and maintains an IT strategy to guide research, development, 
maintenance, and sharing of information systems, applications, data, 
and other resources across the lines of business and throughout the 
agency;

* leads the establishment of world-class software and information 
systems engineering methodologies including Capability Maturity 
Models, and applies them to agency systems, operations, and processes 
to provide continuous improvement of IT performance; and;

* leads and directs agencywide efforts on; information systems 
security, ensuring that standards and policies are in place to provide 
security for the critical information architecture of the agency;

Rationale for change: To accommodate forecasted growth in air traffic 
and replace aging equipment, FAA embarked on an ambitious ATC 
modernization program in 1981. FAA estimated that it would spend about 
$20 billion to replace and modernize software-intensive ATC systems 
between 1982 and 2003. Our work over the years has chronicled many FAA 
failures in meeting ATC projects' cost, schedule, and performance 
goals, largely because of software-related problems. As a result of 
these failures as well as the tremendous cost, complexity, and mission 
criticality of FAA's ATC modernization program, we designated the 
program as a high-risk IT initiative in our 1995 and 1997 report series 
on high-risk programs; Software quality is governed largely by the 
quality of the processes involved in developing or acquiring, and 
maintaining it. SEI has developed models and methods that define and 
determine organizations' software process maturity. Together, they 
provide a logical framework for baselining an organization's current 
process capabilities (i.e., strengths and weaknesses) and providing a 
structured plan for incremental rocess improvement. We found that;

* FAA's ATC modernization software acquisitions processes are immature 
and;

* FAA's approach for improving AT; modernization software acquisition 
processes is not effective.

Key recommendation: Air Traffic Control: FAA's Modernization Investment 
Management Approach Could Be Strengthened, April 30, 1999, (GAO/RCED/ 
AIMD-99-88); FAA should implement a comprehensive investment management 
approach through AMS that includes the following actions:

* Establish a complete portfolio of investments--including existing 
systems funded by the operations budget account as well as projects 
funded by the facilities and equipment account--and require the Joint 
Resources Council to periodically review the baseline status and merits 
of each of these investments throughout their entire life cycle. As 
part of this portfolio, cost baselines for operating and maintaining 
all projects should be developed, and this information should be 
included in the agency's financial plan for its investments and in its 
annual budget request to Congress;

Evidence of policy change: FAA's AMS states that five decisions are 
always made at the corporate level by the Joint Resources Council: the 
mission need decision, the investment decision, the decision to approve 
a change to an acquisition program baseline, approval of the RE&D and 
F&E budget submissions, and approval of the NAS Architecture baseline. 
The selection of a solution to satisfy a mission need, the investment 
of resources into a fully funded program, and the possible need to 
cancel other programs to accommodate a new program make the investment 
decision the most important in the life-cycle management process;

Rationale for change: Over the past 17 years, FAA's modernization 
projects have experienced substantial cost overruns, lengthy delays, 
and significant performance shortfalls. Because of FAA's contention 
that some of its modernization problems were caused by federal 
acquisition regulations, the Congress enacted legislation in November 
1995 that exempted the agency from most federal procurement laws and 
regulations and directed FAA to develop a new acquisition management 
system. In response, FAA implemented AMS on April 1, 1996. AMS provides 
high-level acquisition policy and guidance for selecting and 
controlling investments throughout all phases of the acquisition life 
cycle. GAO found that:

* FAA's AMS is designed to provide a discipline, structured process for 
selecting and controlling investments;

* Lack of oversight of the operations portion of projects prevents FAA 
from managing investments as a complete portfolio;

* Weaknesses in selection, control, and evaluation phases limit FAA's 
effectiveness in managing its portfolio.

Key recommendation:

* Improve the selection process by (1) establishing clearly defined 
procedures for validating each project's cost, schedule, benefit, 
performance, and risk information and (2) requiring documentation of 
the results of the validation procedures applied to each project;

Evidence of policy change: FAA's AMS states that the investment 
analysis team develops an initial acquisition program baseline (i.e., 
performance, cost, schedule, benefits, and risk) for each alternative 
solution offering superior value and benefit to FAA and its customers. 
Service organization members of the investment analysis team lead the 
development of cost and schedule baselines using FAA's work breakdown 
structure and other applicable standards.

Key recommendation:

* Strengthen control over investments by (1) revising the acquisition 
program baseline requirements to include project risks and to add 
milestones for project reviews during the operations phase and (2) 
ensuring that project officials fully track and document estimated 
versus actual results for all the elements (i.e., cost, schedule, 
benefit, performance, and risk) contained in the baseline 
documentation;

Evidence of policy change: AMS states that the acquisition program 
baseline should include cost, schedule, performance, benefits, and risk 
information. It also should include all events that are key to 
satisfying mission need, providing intended operational capability, and 
accruing benefits, as well as events crucial to interrelated programs 
or NAS systems. Once an estimate has been completed and a project 
started, FAA establishes reporting and performance measures to compare 
estimated and actual costs, schedules, and performance.

Key recommendation:

* Initiate post implementation evaluations for projects within 3 to 12 
months of deployment or cancellation to compare the completed projects' 
cost, schedule, performance, and mission improvement outcomes with the 
original estimates;

* Incorporate key information from the selection process (e.g., mission 
need statements, cost-benefit analyses, and risk assessments) into 
FAA's management information system for investments;

Evidence of policy change: FAA published a methodology for conducting 
such evaluations entitled An Approach for Developing a Standard Method 
for Conducting Post-Implementation Reviews, Report #2001-13, June 6, 
2001.

Key recommendation: Major Management Challenges and Program Risks, 
Department of Transportation; January 2001, (GAO-01-253); FAA should 
develop a comprehensive plan that would include established checkpoints 
at which the agency would determine, among other things, whether users' 
needs have changed and whether other technologies have matured and 
could better meet users' needs and the agency's requirements for 
satellite navigation. FAA should also have an external organization 
evaluate its progress at established checkpoints and include the 
results of this evaluation in its request for future funding of the 
navigation system;

Evidence of policy change: FAA has appointed an independent board--
consisting of external experts in satellite navigation, safety 
certification, and radio spectrum--that reports directly to the FAA 
Administrator. The board is tasked with reviewing the soundness of the 
panel's recommendations and with revalidating the future path for WAAS. 
However, given the past problems in developing this system and the 
long-term effort that is still required, we believe that continued 
oversight by an independent group of experts is warranted. It is not 
clear whether the current independent board will fulfill this role. We 
will continue to evaluate FAA's progress on this and other system 
acquisition efforts;

Rationale for change: DOT's management of its major acquisitions and 
assets needs improvement in several areas. FAA and the U.S. Coast Guard 
are undertaking costly, long-term programs to modernize and replace 
aging equipment. Over the past 19 years, FAA's multibillion-dollar ATC 
modernization program has experienced cost overruns, delays, and 
performance shortfalls of large proportions. FAA is making progress in 
addressing some of our recommendations, but its reform efforts are not 
complete, and major projects continue to face cost, schedule, and 
performance problems. Because of its size, complexity, cost, and 
problem-plagued past, we designated FAA's IT program as a high-risk IT 
initiative in 1995.

Key recommendation: Status of FAA's Major Acquisitions; DOT/OIG, AV-
2003-045, June 26, 2003; Update the cost, schedule, and performance 
baselines for many of FAA's major acquisition, including STARS, ITWS, 
LAAS, and WAAS at a minimum. Develop--and use--performance goals for 
assessing progress with its major acquisitions. This should involve 
holding staff and contractors accountable for keeping projects within 
cost and schedule, as appropriate;

Evidence of policy change: FAA officials generally agreed with the 
analysis and recommendations in this report. FAA is implementing this 
recommendation. It updated the baseline of STARS in April 2004 and 
updated the baselines of ITWS and WAAS in May 2004. The LAAS program 
was deferred because of budget cuts;

Rationale for change: FAA has made progress with a number of 
acquisitions, including Free Flight Phase 1 and new information 
exchange systems that link FAA and airline operations centers. However, 
other modernization programs have experienced cost, schedule, and 
performance problems. Problems with acquisition efforts have serious 
consequences because they result in costly interim systems, reduce the 
number of units procured, postpone benefits, or "crowd out" other 
modernization projects.

Key recommendation: Status Report on FAA's Operational Evolution Plan; 
DOT/OIG, AV-2003-048, July 23, 2003; Develop realistic cost estimates, 
and link the Operational Evolution Plan (OEP) with FAA's budget in 
order to set priorities for what can be accomplished in the short term; 
Determine--in concert with the aviation community--how to move forward 
(and at what pace) with systems that require airspace users to purchase 
and install new technologies; Determine and maximize the benefits 
associated with airspace design changes, new procedures, and 
capabilities currently onboard aircraft to enhance system capacity;

Evidence of policy change: FAA officials generally agreed with the 
analysis and recommendations in this report. FAA is currently updating 
the OEP, which includes design changes to the National Airspace to, for 
example, enhance capacity;

Rationale for change: The OEP is an important effort because it will 
shape FAA and industry investments over the next decade. However, much 
has changed since the OEP was introduced. The demand for air travel has 
declined, major network carriers are in financial distress, and 
Aviation Trust Fund revenues have declined sharply. The Inspector 
General found that fundamental assumptions about the OEP, such as the 
cost, schedule, and benefits of key efforts as well as the ability of 
airspace users to pay for and equip with new technologies in the near 
term, are no longer valid and need to be revised. 

Source: GAO analysis.

[End of table]

[End of section]

Appendix V: Status of the Seven ATC Modernization Acquisitions That GAO 
Reviewed: 

Standard Terminal Automation Replacement System (STARS):

Purpose and Status:

STARS is a joint FAA and Department of Defense (DOD) program. It will 
replace aging legacy terminal FAA and DOD automation systems with 
terminal ATC systems. Civil and military air traffic controllers across 
the nation are using STARS to direct aircraft near major airports.

In June 2003, FAA commissioned STARS for use at the Philadelphia 
International Airport in Pennsylvania. Currently, STARS is fully 
operational at 24 FAA terminal radar control facilities and 17 DOD 
facilities. Under the ATO's new business model of breaking large and 
complex programs into smaller phases to control cost and schedule, 
STARS is a candidate for further deployment to about 120 FAA and DOD 
operational facilities. In May 2004, FAA changed STARS's cost and 
schedule estimates for the third time and estimates that it will cost 
$1.46 billion to deploy STARS at 50 operational facilities.

Contractor: Raytheon.

[See PDF for image]

Photo caption: STARS Display Monitor. 

Source: FAA 

[End of figure]

Baseline Changes to STARS Scope, Schedule and Cost:

Date: February 1996;
Number of FAA facilities receiving STARS: 172;
Projected date for first deployment: 1998;
Projected date for last deployment: 2005;
Estimated cost (F&E)[A]: $0.94 billion.

Date: October 1999;
Number of FAA facilities receiving STARS: 188;
Projected date for first deployment: 2002;
Projected date for last deployment: 2008;
Estimated cost (F&E)[A]: $1.4 billion.

Date: March 2002;
Number of FAA facilities receiving STARS: 73;
Projected date for first deployment: 2002;
Projected date for last deployment: 2005;
Estimated cost (F&E)[A]: $1.33 billion.

Date: April 2004;
Number of FAA facilities receiving STARS: 50;
Projected date for first deployment: 2003;
Projected date for last deployment: 2008;
Estimated cost (F&E)[A]: $1.46 billion.

Total change;
Number of FAA facilities receiving STARS: -122;
Projected date for first deployment: + 3 years;
Projected date for last deployment: + 3 years;
Estimated cost (F&E)[A]: +$0.52 billion.

Source: GAO's presentation of FAA data;

[A]FAA's Facilities and Equipment (F&E) account funds capital projects.

[End of table]

Risks and Challenges:

Certification issues - FAA also experienced problems in certifying 
STARS, in part because of aggressive scheduling. FAA's approach to 
certifying STARS was oriented to rapid deployment to meet critical 
needs. To meet these needs, FAA compressed its original 32-month 
development and testing schedule into 25 months. This compressed 
schedule left only limited time for human factor evaluations and not 
enough time for computer human; interface issues and involvement of 
controllers and maintenance technicians. 

Airport Surveillance Radar Model-11 (ASR-11):

Purpose and Status:

ASR-11 will provide high-quality digital data to terminal controllers 
in terminal environments. It will also provide a more reliable 
replacement for aging analog radars like ASR-7 and ASR-8; it will also 
provide digitized radar data for the new automation systems such as 
STARS. In addition, ASR-11 will provide six levels of weather 
information, a significant improvement over the current two levels. The 
ASR-11 program is a joint program with DOD-that is, DOD is managing the 
program to joint specifications, and FAA will provide DOD with the 
funds to procure 112 units. ASR-11 is a nondevelopmental item.

The in-service decision was made in 2003, and the radar is being 
deployed to 108 sites. The ASR-11 program is scheduled to be 
rebaselined for cost and schedule in fiscal year 2005.

Contractor: Raytheon.

ASR-11 Scope, Schedule, and Cost:

Date: March 2002;
Number of facilities receiving ASR-11: 112;
Projected date for first deployment: 2000;
Projected date for last deployment: 2005;
Estimated cost (F&E): $743.3 million.

Date: July 2004;
Number of facilities receiving ASR-11: 112;
Projected date for first deployment: 2003;
Projected date for last deployment: 2013;
Estimated cost (F&E): $891.7 million.

Total change;
Number of facilities receiving ASR-11: 0;
Projected date for first deployment: 3 years;
Projected date for last deployment: 8 years;
Estimated cost (F&E): $148.4 million.

[End of table]

Source: GAO's presentation of FAA data.

Risks and Challenges:

The Capital Investment Plan does not support the service as required in 
the current Acquisition Program Baseline, which could put the program 
in jeopardy.

Integrated Terminal Weather System (ITWS):

Purpose and Status:

ITWS provides automated weather information for use by air traffic 
controllers and supervisors in airport terminal airspace (60 miles 
around the airport). It provides products that require no 
meteorological interpretation to air traffic controllers, air traffic 
managers, pilots, and airlines. ITWS provides a comprehensive current 
weather situation and highly accurate forecasts of expected weather 
conditions for the next 30 minutes.

Current FAA plans call for the installation of 34 systems that will 
service various airports. Six systems are operational, and feedback 
from users is satisfactory. In May 2004, the ATO Executive Council 
rebaselined the program to include a weather-forecasting capability in 
the production baseline, a new requirement to provide operational 
support for the New York prototype, and change the operations and 
maintenance cost baseline for the program. However, the council did not 
include additional funding, and therefore, in order to stay within the 
capital improvement program's (CIP) funding levels, the program has 
proposed to defer 12 of the planned 34 systems installations.

Contractor: Raytheon.

Baseline Changes to ITWS Scope, Schedule, and Cost:

Date: June 1997;
Number of facilities receiving ITWS: 34;
Projected date for first deployment: Sep 01-Mar 02;
Projected date for last deployment: Jan 03-Jul 03;
Estimated cost (F&E): $276.1 million.

Date: August 2001;
Number of facilities receiving ITWS: 34;
Projected date for first deployment: December 2002;
Projected date for last deployment: May 2004;
Estimated cost (F&E): $282.2 million.

Date: May 2004;
Number of facilities receiving ITWS: 34;
Projected date for first deployment: December 2002;
Projected date for last deployment: 2009+;
Estimated cost (F&E): $288.3 million.

Total change;
Number of facilities receiving ITWS: 0;
Projected date for first deployment: 1+ years;
Projected date for last deployment: 6+ years;
Estimated cost (F&E): $12.2 million.

Source: GAO's presentation of FAA data:

[End of table]

Risks and Challenges:

Funding issues -The program requested and obtained approval to 
rebaseline. The baseline is being modified to incorporate the Terminal 
Convective Weather Forecasting (TCWF) capability into the production 
baseline. As directed by the ATO Executive Council, responsibility for 
funding operational support for the New York prototype system is also 
being added to the baseline. The ATO Executive Council also directed 
that the cost of the program remain at the current CIP funding levels 
for fiscal years 2005, 2006, and 2007. In order to stay within the CIP 
funding levels, the program proposed to defer 12 of the planned 34 
systems installations.

Schedule issues - Because of constrained funding, 12 airports will not 
receive ITWS capabilities until after 2009.

Local Area Augmentation System (LAAS):

Purpose and Status:

LAAS is a precision approach and landing system that will augment the 
Global Positioning System (GPS) to broadcast highly accurate 
information to aircraft on the final phases of a flight. LAAS consists 
of both ground and avionics components. Ground components include GPS 
reference receivers, which monitor and track GPS signals; very-high-
frequency transmitters for broadcasting the LAAS signal to aircraft; 
and ground station equipment, which generates precision approach data 
and is housed at or near an airport. Aircraft will be equipped with 
avionics to receive LAAS signals.

FAA's fiscal year 2005 budget eliminated funding for LAAS, and 
remaining fiscal year 2004 funds will continue to validate LAAS 
requirements and address radio frequency interference issues. FAA 
officials will reconsider national deployment when more research 
results are completed.

Baseline Changes to LAAS Schedule and Cost:

Date: January 1998;
Projected date for first deployment: 2002;
Projected date for last deployment: TBD;
Estimated cost (F&E): $530.1 million.

Date: September 1999;
Projected date for first deployment: 2003;
Projected date for last deployment: TBD;
Estimated cost (F&E): $696.1 million.

Total change;
Projected date for first deployment: 1 year;
Estimated cost (F&E): +$166 million.

Source: GAO presentation of FAA data.

[End of table]

Risks and Challenges:

Cost issues - LAAS cost estimates are not reliable, reflecting 
inadequate requirements development in the early stages of the program, 
a lack of understanding of a mission degradation issue, incomplete 
software development, and an unrealistic development schedule.

Schedule issues -The LAAS schedule was not realistic. Specifically, FAA 
lacked an understanding of the integrity requirement and software 
development, which were the two biggest technological maturity issues 
facing the LAAS program.

Performance Issues - FAA has not resolved the integrity requirement 
that ensures pilots are alerted in a timely manner when the LAAS signal 
is not reliable. FAA has not been able to prove that the system is safe 
during solar storms. An analysis of the effects of solar storms on the 
LAAS signal's integrity is under way, but an atmospheric monitoring 
device that could address this issue may not be available until fiscal 
year 2009.

Next Generation Air/Ground Communications (NEXCOM):

Purpose and Status:

The Next Generation Air/Ground Communications (NEXCOM) project is to 
replace the existing analog ATC communications system with a new 
digital system that would have greater capabilities. The initial 
development, of a multimodal digital radio (MDR), is to be followed by 
the development of aircraft avionics and ground systems. NEXCOM is 
expected to increase the number of available communications channels, 
provide simultaneous voice and data transmission between controllers 
and pilots, and require a digital form of authentication, designed to 
prevent "phantom controllers" from gaining access to the communications 
system. FAA plans to deploy 6,000 MDR pairs (a radio pair is one 
receiver and one transmitter) during the first phase, which will 
provide voice channels to aircraft in the en route environment.

NEXCOM completed Independent Operational Test and Evaluation assessment 
of the radio component at the Santa Barbara, California, Remote Center 
Air/Ground Communications facility, and radios were approved for in-
service and national deployment in July 2004. The avionics component's 
development is scheduled to be completed by 2006. However, proposed 
funding cuts to FAA's fiscal year 2005 budget required the termination 
of the ground station development, which would enable communications in 
the more efficient digital mode.

Contractor: ITT for MDR.

Baseline Changes to NEXCOM Scope, Schedule, and Cost:

Date: May 2000;
Number of radio pairs deployed: 6,000;
Date first site Initial Operating Capability: July 2002;
Date of In-Service Decision: October 2002;
Estimated cost (F&E): $318.4 million[A].

Date: February 2004;
Number of radio pairs deployed: 6,000;
Date first site Initial Operating Capability: March 2004;
Date of In-Service Decision: July 2004;
Estimated cost (F&E): $318.4 million.

Total change;
Number of radio pairs deployed: 0;
Date first site Initial Operating Capability: 20 months;
Date of In-Service Decision: 21 months;
Estimated cost (F&E): $0.

Source: GAO presentation of FAA data.

[A] Estimated cost is only for the NEXCOM MDR. The NEXCOM ground 
station contract was canceled in March 2004 and is being terminated.

Risks and Challenges:

Schedule issues-FAA planned to base the MDR on a nondevelopmental item 
(NDI), and the initial schedule allowed only limited development. 
However, FAA's requirement that communications channels be free of 
signal interference ("quiet channels") was more demanding than the NDI 
solution was capable of achieving. As a result, further development was 
necessary, delaying the initial operational capability and in-service 
decision by 21 months.

Performance-The NEXCOM radio meets its operational requirement for 
coverage. However, to achieve this requirement FAA determined that the 
NEXCOM radios would have to achieve the same power output level (50 
watts) that the existing radios produced. The contractor is delivering 
radios that put out no more than 34 watts per channel. This posed an 
"unacceptable consequence" and FAA performed additional tests or 
flights checks and determined that the reduced power would not 
adversely affect operations and has approved the use of the lower-
output radios.

Advanced Technologies and Oceanic Procedures (ATOP):

Purpose and Status:

The Advanced Technologies and Oceanic Procedures (ATOP) program 
introduces new controller workstations, data-processing equipment, and 
software designed to enhance the control and flow of oceanic air 
traffic to and from the United States. ATOP processes aircraft position 
updates automatically, whereas currently, oceanic traffic control 
operations are performed manually and updated via paper flight strips.

ATOP is designed to present flight data "electronically" in a format 
similar to these paper strips ATOP completed operational testing at its 
first site, the Oakland Air Route Traffic Control Center (ARTCC) and 
achieved initial operational capability (IOC) on June 30, 2004. 
Currently, ATOP is in limited use for 4 hours a day 5 days a week in 
one of nine sectors under Oakland's control. Plans to fully transition 
ATOP to all nine sectors depend upon feedback from the initial trials 
and sector-by-sector capabilities. Other operational considerations 
still to be resolved are additional staff needs, ATOP's training 
schedule, and coordination with North American Aerospace Defense 
Command on an interface device. FAA is currently in the early phases of 
installing ATOP at the New York ARTCC and is scheduled to achieve IOC 
in March 2005. Additional software that will incorporate radar data 
into ATOP is under development and scheduled to be completed by 
November 2004. This software is expected to be operational at the final 
site, the Anchorage ARTCC, in March 2006.

Contractor: Lockheed Martin Transportation and Security Solutions.

Baseline Changes ATOP Scope, Schedule, and Cost:

Date: May 2001 (baseline);
Number of facilities receiving ATOP: 3;
Projected date for first deployment: June 2004 Oakland ARTCC;
Projected date for last deployment: March 2006;
Estimated cost (F&E): $548.2 million.

Date: July 2004;
Number of facilities receiving ATOP: 3;
Projected date for first deployment: June 2004;
Projected date for last deployment: March 2006;
Estimated cost (F&E): $548.2 million.

Total change;
Number of facilities receiving ATOP: 0;
Projected date for first deployment: None;
Projected date for last deployment: None;
Estimated cost (F&E): $0.

Source: GAO presentation of FAA data:

[End of table]

Risks and Challenges:

Cost issues-Although the contractor's costs to develop ATOP have grown 
by approximately $20 million, FAA is not responsible for these cost 
increases because it has a fixed-price contract arrangement with the 
contractor.

Schedule issues-ATOP achieved its initial operational capability 
milestone of June 2004 but a more aggressive development schedule was 
agreed to with the ATOP contractor to achieve this milestone by April 
2003 or 14 months earlier. An ATOP Assessment Team determined that the 
contractor could not achieve this earlier date due poor requirements 
development, unrealistic schedule estimates, and inadequate evaluation 
by the contractor of the software complexity. The development delay has 
exacerbated the scheduled transition from the current oceanic system to 
the ATOP and would cost an additional $4 million a year to operate and 
maintain the old system until ATOP is fully operational. Program 
officials told us they were not certain when the transition could be 
achieved because several operational issues needed to be resolved 
including ATOP operational trials sector by sector, training schedule, 
and filling new controller positions, and budgetary allocations to 
support these activities.

En Route Automation Modernization (ERAM):

Purpose and Status:

The En Route Automation Modernization (ERAM) program will enable air 
traffic controllers to provide ATC services to users of en route 
airspace (generally, high-altitude airspace at 10,000 feet or above) 
Services provided to users include separation, routing, and advisory 
services needed to meet FAA's mission of providing safe, efficient, and 
reliable air traffic management. Specifically, ERAM is to replace the 
hardware and software in the current en route Host computer system, the 
direct-access radar channel, and associated infrastructure. This 
replacement will result in the installation of new system en route 
automation architecture at each air route traffic control center 
(ARTCC). In concert with other en route programs, ERAM will modernize 
the en route infrastructure to provide a supportable, open-standards-
based system that will be the basis for future capabilities and 
enhancements. ERAM is to be deployed at 20 ARTCCs in the continental 
United States.

FAA awarded a letter contract to Lockheed Martin in December 2002. To 
date, ERAM has not breached any JRC cost or schedule parameters. 
However, the ERAM program is highly software intensive, requiring the 
writing of over 1 million lines of software code. In addition, Lockheed 
Martin is behind schedule because of software design and production 
control issues that Lockheed expects to resolve. Lockheed Martin 
officials stated that it does not expect any downstream impact from the 
current negative schedule variance of about $1 million.

Contractor: Lockheed Martin Transportation and Security Solutions.

Baseline Changes to ERAM Scope, Schedule, and Cost:

Date: June 2003;
Number of facilities receiving ERAM: 20;
Projected date for first deployment: December 2009;
Projected date for last deployment: December 2010;
Estimated cost (F&E and O&M): $3.649 billion.

Total change; None.

Source: GAO's presentation of FAA data:

[Need another set of data to determine any change.]

[End of table]

Risks and Challenges:

Software Issues - Software development is one of ERAM's major risk 
items. The ERAM program is a high-risk effort because of its size and 
the amount of software code - over 1 million lines of software code 
expected. Lockheed Martin is experiencing cost variances because of 
software engineering difficulties. According to its cost performance 
report, software engineering costs are being hampered by lower 
productivity than originally planned and by software code growth across 
the program. However, according to FAA officials, these additional 
software development costs can be easily absorbed within the 
contractor's management reserve that is currently on the contract.

[End of section]

Appendix VI: Key Contributors: 

GAO Contacts: 

JayEtta Z. Hecker, (202) 512-2834; 
Beverly L. Norwood, (202) 512-2834: 

Staff Acknowledgments: 

In addition to the individuals named above, Tamera Dorland, Elizabeth 
Eisenstadt, Brandon Haller, Bert Japikse, Carolyn Kirby, Steve 
Martinez, Richard Scott, Adam Vodraska, and Dale Yuge made key 
contributions to this report.

(540072): 

FOOTNOTES

[1] GAO, Major Management Challenges and Program Risks: Department of 
Transportation, GAO-03-108, (Washington, D.C.: Jan. 2003). 

[2] For example, Best Practices: Capturing Design and Manufacturing 
Knowledge Early Improves Acquisition Outcomes, GAO-02-701, 
(Washington, D.C.: July 15, 2002) and Best Practices: Better Matching 
of Needs and Resources Will Lead to Better Weapon System Outcomes, GAO-
01-288, (Washington, D.C.: Mar. 8, 2001).

[3] In this report, we refer to both commercial product developments 
and federal agency acquisitions as acquisitions. 

[4] The Joint Resources Council is an executive body consisting of 
associate and assistant administrators, acquisition executives, the 
chief financial officer, the chief information officer, and legal 
counsel. The council makes corporate-level decisions, including those 
that determine whether an acquisition meets a mission need and should 
proceed. The council also approves changes to a program's baseline, 
budget submissions, and the National Airspace System's architecture 
baseline. 

[5] The term "federal acquisition system" is used to refer to the 
various statutes and regulations that govern procurement practices by 
federal government agencies--the controlling regulation is the FAR.

[6] Currently, the FAR applies to all federal executive agencies except 
FAA and the Transportation Security Administration.

[7] The six sections provide an overview and address life-cycle 
acquisition management, procurement, configuration management, real 
property, and integrated logistic support. AMS also includes 
implementing guidance, flow charts, handbooks, clauses, forms, and 
other information that expands, illustrates, or supplements policy. 

[8] A management process for establishing and maintaining the 
consistency of a product's performance and physical attributes with its 
requirements, design, and operational information throughout its life.

[9] Integrated logistics support (ILS) is a critical functional 
discipline that establishes and maintains a support system for all FAA 
products and services. Elements of ILS include spare parts, training, 
supply support, manuals and documentation, maintenance, and repair.

[10] 48 C.F.R. pt. 7.

[11] 48 C.F.R. pt. 34.

[12] DOD's 5000 series consists of DOD Directive 5000.1, the Defense 
Acquisition System, and DOD Instruction 5000.2, Operation of the 
Defense Acquisition System.

[13] P. L. 103-62, 107 Stat. 285.

[14] P.L. 89-286, 79 Stat. 1034.

[15] 14 C.F.R. pt. 17.

[16] In two-step sealed bid procurements, the acquisition process is 
divided into two parts. In the first step, proposals are solicited and 
evaluated to determine their acceptability without evaluating price. In 
the second step, offerors who submitted acceptable step-one proposals 
compete for award on the basis of price. Two-phase design-build 
selection procedures are a selection method in which a limited number 
of offerors is selected during the first phase (design) to submit 
detailed proposals for the second phase (construction).

[17] 48 C.F.R. § 2.101 (definition of "full and open competition").

[18] A rational basis for such action may be based on emergencies, 
standardization, or that a source is the only source available to 
satisfy the requirement within the time required, which are necessary 
and important to support FAA's mission. The decision to contract with a 
single source may be made as part of the overall program planning. The 
rational basis must be documented and approved as a part of the 
acquisition strategy paper, a procurement plan, or as a separate 
document. The AMS states that if an acquisition strategy paper is not 
required, and the service organization determines that a procurement 
plan is unnecessary, an independent single-source justification should 
be documented and endorsed by the service organization and approved by 
the contracting officer.

[19] GAO, Rebuilding Iraq: Fiscal Year 2003 Contract Award Procedures 
and Management Challenges, GAO-04-605, (Washington, D.C.: June 1, 
2004).

[20] For example, GAO, Best Practices: Capturing Design and 
Manufacturing Knowledge Early Improves Acquisition Outcomes, GAO-02-
701, (Washington, D.C.: July 15, 2002) and Best Practices: Better 
Matching of Needs and Resources Will Lead to Better Weapon System 
Outcomes, GAO-01-288, (Washington, D.C.: Mar. 8, 2001). 

[21] GAO, Information Technology: FAA Has Many Investment Management 
Capabilities in Place, but More Oversight of Operational Systems Is 
Needed, GAO-04-822, (Washington, D.C.: Aug. 20, 2004). 

[22] Performance deficiencies in relation to the final requirements or 
system specifications are used to assess whether the agency's goals 
have been met. Such deficiencies may not degrade the mission standards 
needed to ensure the safety and efficiency of the National Airspace 
System.

[23] Free Flight Phase 1, completed in 2002, provided new information-
exchange systems and automated controller tools.

[24] We reported in August 2004 that FAA had implemented sound 
requirements development and management practices on four other 
systems, but noted that process improvement initiatives such as these 
were not institutionalized across the agency. See GAO, Air Traffic 
Control: System Management Capabilities Improved, but More Can Be Done 
to Institutionalize Improvements, GAO-04-901, (Washington, D.C.: Aug. 
20, 2004).

[25] Available software refers to commercial-off-the-shelf (COTS) and/
or nondevelopmental items (NDI). AMS defines COTS as a product or 
service that has been developed for sale, lease, or license to the 
general public. The product is currently available at a fair market 
value. AMS defines an NDI as an item that was previously developed for 
use by a government (federal, state, local, or foreign) and requires 
limited further development. For example, the Army's SINCGARS radio is 
the core of FAA's NEXCOM radio, and the software FAA selected for ATOP 
was NDI software from the New Zealand air system.

[26] ATOP is under a fixed-price contract, but the contractor has 
experienced over $20 million in cost overruns during the development 
phase. Also, FAA renegotiated the terms of the contract to ensure that 
the initial software development phase, known as build 1, would meet 
its June 2004 Initial Operating Capability milestone.

[27] AMS does not call for critical design reviews, but they can be 
done at the program's discretion.

[28] GAO, Information Technology: FAA Has Many Investment Management 
Capabilities in Place, but More Oversight of Operational Systems Is 
Needed , GAO-04-822, (Washington, D.C.: Aug. 20, 2004).

[29] FAA is organized into five business units that include: Airports; 
Regulation and Certification; Commercial Space Transportation; the 
Office of Security and Hazardous Materials; and the Air Traffic 
Organization.

[30] Merging the former Air Traffic Services and the Research and 
Acquisitions organizations formed the ATO; individual organizations 
within FAA sponsor specific acquisitions to meet identified needs 
(e.g., controller workstations and radars).

[31] The CIO is not part of the ATO; however, the CIO's efforts to 
improve FAA's acquisition and management of software for major ATC 
systems are directly related to the ATO's efforts to improve the 
agency's acquisition of major ATC systems.

[32] GAO, Air Traffic Control: System Management Capabilities Improved, 
but More Can Be Done to Institutionalize Improvements, GAO-04-901, 
(Washington, D.C.: Aug. 20, 2004).

[33] GAO, Air Traffic Control: FAA's Modernization Investment 
Management Approach Could be Strengthened, GAO/RCED/AIMD-99-88, 
(Washington, D.C.: April 30, 1999).

[34] GAO, Acquisition Management: Agencies Can Improve Training on New 
Initiatives, GAO-03-281, (Washington, D.C.: Jan. 15, 2003).

[35] In October 1997, FAA created the Intellectual Capital Investment 
Plan Council to address the development needs of staff in its research 
and acquisition organization. The council is made up of directors and 
deputy directors from the agency's acquisition and research programs.

[36] Capital Planning and Investment Control is a disciplined process 
that links planning to budgeting to procurement to operations, 
maintenance, and management.

[37] Earned value management compares the actual work performed at 
certain stages of a job to its actual costs--rather than comparing 
budgeted and actual costs, the traditional management approach to 
assessing progress. By measuring the value of the work that has been 
completed at certain stages in a job, earned value management can alert 
program managers, contractors, and administrators to potential cost 
overruns and schedule delays before they occur and to problems that 
need correcting before they worsen.

[38] Through the purchase card program, agency personnel can acquire 
the routine goods and services they need directly from vendors as long 
as the purchase is $2,500 or less.

[39] GAO, Best Practices: Using Spend Analysis to Help Agencies Take a 
More Strategic Approach to Procurement, GAO-04-870, (Washington, D.C.: 
Sept. 16, 2004); Best Practices: Improved Knowledge of DOD Service 
Contracts Could Reveal Significant Savings, GAO-03-661, (Washington, 
D.C.: June 9, 2003); and Best Practices: Taking a Strategic Approach 
Could Improve DOD's Acquisition of Services, GAO-02-230, (Washington, 
D.C.: Jan. 18, 2002). 

[40] GAO, Contract Management: Agencies Can Achieve Significant Savings 
On Purchase Card Buys, GAO-04-430, (Washington, D.C.: Mar. 12, 2004).

[41] GAO, Best Practices: Using Spend Analysis to Help Agencies Take a 
More Strategic Approach to Procurement, GAO-04-870, (Washington, D.C.: 
Sept. 16, 2004). 

[42] P. L. 104-50, § 348, 109 Stat 436 (1995). Included in the United 
States Code as 49 U.S.C. § 40110.

[43] 41 U.S.C. Ch. 4.

[44] 41 U.S.C. Ch. 7.

[45] 15 U.S.C. Ch. 14A.

[46] 14 C.F.R. pt. 17.

[47] AMS views FAA as consisting of numerous service-level 
organizations, which in turn are organizational subunits that deliver 
services within FAA, to industry or to the public, including technical 
as well as nontechnical service providers. 

[48] Investment analysis also includes identifying and analyzing 
alternatives; developing life-cycle cost estimates; assessing net 
present value, return on investment, and benefits; assessing 
affordability; analyzing risk; evaluating the impact of an alternative 
on enterprise architecture; and planning for deployment and 
implementation.

[49] 48 C.F.R. pt. 7.

[50] 48 C.F.R. pt. 34.

[51] DOD's 5000 series consists of DOD Directive 5000.1, The Defense 
Acquisition System and DOD Instruction 5000.2, Operation of the Defense 
Acquisition System.

[52] Section 5 of AMS focuses on the acquisition of real property, a 
subject that is also not covered by FAR.

[53] P. L. 103-62; 107 Stat. 285.

[54] 14 C.F.R. pt. 17. 

[55] And up to $10,000,000 under limited special circumstances.

GAO's Mission: 

The Government Accountability Office, the investigative arm of 
Congress, exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability 
of the federal government for the American people. GAO examines the use 
of public funds; evaluates federal programs and policies; and provides 
analyses, recommendations, and other assistance to help Congress make 
informed oversight, policy, and funding decisions. GAO's commitment to 
good government is reflected in its core values of accountability, 
integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains 
abstracts and full-text files of current reports and testimony and an 
expanding archive of older products. The Web site features a search 
engine to help you locate documents using key words and phrases. You 
can print these documents in their entirety, including charts and other 
graphics.

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as "Today's Reports," on its 
Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order 
GAO Products" heading.

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to: 

U.S. Government Accountability Office

441 G Street NW, Room LM

Washington, D.C. 20548: 

To order by Phone: 

 

Voice: (202) 512-6000: 

TDD: (202) 512-2537: 

Fax: (202) 512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470: 

Public Affairs: 

Jeff Nelligan, managing director,

NelliganJ@gao.gov

(202) 512-4800

U.S. Government Accountability Office,

441 G Street NW, Room 7149

Washington, D.C. 20548: