This is the accessible text file for GAO report number GAO-05-23 
entitled 'Air Traffic Control: FAA's Acquisition Management Has 
Improved, but Policies and Oversight Need Strengthening to Help Ensure 
Results' which was released on December 13, 2004.

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov.

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately.

Report to the Chairman and Ranking Minority Member, Committee on 
Government Reform, House of Representatives: 

November 2004: 

AIR TRAFFIC CONTROL: 

FAA's Acquisition Management Has Improved, but Policies and Oversight 
Need Strengthening to Help Ensure Results: 

GAO-05-23: 

GAO Highlights: 

Highlights of GAO-05-23, a report to the Chairman and Ranking Minority 
Member, Committee on Government Reform, House of Representatives: 

Why GAO Did This Study: 

The Federal Aviation Administration’s (FAA) multibillion-dollar effort 
to modernize the nation’s air traffic control (ATC) system has resulted 
in cost, schedule, and performance shortfalls for over two decades and 
has been on GAO’s list of high-risk federal programs since 1995. 
According to FAA, performance shortfalls were due, in part, to 
restrictions imposed by federal acquisition and personnel regulations. 
In response, Congress granted FAA exemptions in 1995 and directed it to 
develop a new acquisition management system. 

In this report, GAO compared FAA’s AMS with (1) the FAR and (2) 
commercial best practices for major acquisitions, and (3) examined 
FAA’s implementation of AMS and its progress in resolving problems with 
major acquisitions.

What GAO Found: 

FAA’s Acquisition Management System (AMS) is broader and less 
prescriptive than the Federal Acquisition Regulation (FAR), but both 
afford managers flexibility. AMS establishes an acquisition life-cycle 
management system, including both a contracting and program management 
system, whereas the FAR is primarily a contracting system. In addition, 
AMS takes the form of guidance—it is not regulatory, while the FAR is a 
set of published regulations—a legal foundation that has the force and 
effect of law that most federal agencies are required to follow.

[See PDF for image]

[End of figure]

AMS provides some discipline for acquiring major ATC systems; however, 
it does not ensure a knowledge-based approach to acquisition found in 
the best commercial practices for managing commercial and DOD product 
developments that we have identified in numerous past reports. Best 
practices call for (1) use of explicit written criteria to attain 
specific knowledge at key decision points and (2) use of this knowledge 
by executives at the corporate level to determine whether a product is 
ready to move forward. Attainment and use of such knowledge by 
executives helps to avoid cost, schedule, and performance shortfalls 
that can occur if they commit to a system design prematurely. While AMS 
has some good features, including calling for key decision points, it 
falls short of best practices. 

GAO’s review of seven major ATC systems and analysis of FAA’s 
performance in acquiring major systems found that AMS has not resolved 
longstanding problems it experienced prior to its implementation of 
AMS—including developing requirements and managing software—and is just 
beginning to focus on how these acquisitions will improve the 
efficiency of ATC operations. combined found that long-standing 
problems While FAA has made progress by providing guidance for avoiding 
past weaknesses, it has not applied these improvements consistently. 
According to FAA officials, reorganization under and improved oversight 
by FAA’s new performance-based Air Traffic Organization should help 
ensure greater consistency and an increased focus on results. Past GAO 
reports have demonstrated that the success of an acquisition process 
depends on good management, whether it be under AMS or the FAR.

What GAO Recommends: 

GAO recommends that the Secretary of Transportation advise FAA to, 
among other things, (1) improve its development of requirements and 
management of software and (2) more closely align AMS with commercial 
best practices. 

In commenting on a draft of this report, FAA generally agreed with the 
report’s contents and said that our recommendations would be helpful to 
them as they continue to refine AMS. They also provided us with 
technical comments, which we have incorporated as appropriate. 

www.gao.gov/cgi-bin/getrpt?GAO-05-23.

To view the full product, including the scope and methodology, click on 
the link above. For more information, contact JayEtta Z. Hecker 
512-2834 or heckerj@gao.gov.

[End of section]

Contents: 

Letter: 

Results in Brief: 

Background: 

AMS Is Broader and Less Prescriptive Than the FAR: 

AMS Provides Some Discipline but Does Not Ensure a Knowledge-Based 
Approach to Acquisition: 

As Implemented, AMS Has Not Resolved Long-standing Acquisition 
Problems, but FAA Is Beginning to Focus More on Results: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments: 

Appendixes: 

Appendix I: FAA Has Begun Analyzing Spending Trends to Take a More 
Strategic Approach to Procurement: 

Appendix II: Objectives, Scope, and Methodology: 

Appendix III: Comparison of the Scope and Flexibility of FAA's 
Acquisition Management System and the Federal Acquisition Regulation 
Process: 

Background: 

AMS Defines an Investment/Life-Cycle Project Management System: 

Only a Portion of AMS Deals Directly with the Procurement Process: 

AMS States a Nonregulatory FAA Policy: 

AMS Chapter 3 Parallels a Subset of the FAR: 

AMS Includes a Less Rigorous Competition Requirement Than Does the FAR: 

The FAR Gives Procurement Professionals Tighter Control over 
Procurement Decisions: 

Although FAA Project Managers View AMS as More Efficient and Flexible 
Than the FAR, Some Procurement Officials We Interviewed Do Not Agree: 

Appendix IV: How FAA's Acquisition Policy Adapted Key Recommendations 
Made by GAO and DOT (1996-2003): 

FAA Refined AMS in Response to Recommendations: 

Appendix V: Status of the Seven ATC Modernization Acquisitions That GAO 
Reviewed: 

Appendix VI: Key Contributors: 

GAO Contacts: 

Staff Acknowledgments: 

Tables Tables: 

Table 1: Structure of Best Practices Model for Major Product 
Developments: 

Table 2: Structure of AMS: 

Table 3: Knowledge-Based Approach Called for in Our Best Practices 
Model: 

Table 4: AMS's Decision Points, Information Sources, and Oversight 
Reviews: 

Table 5: Description and Status of Seven Selected ATC Acquisitions: 

Table 6: Internal and External Reviews of FAA's Use of AMS for 
Acquiring Major ATC Systems: 

Table 7: Program Costs for the Seven Systems We Reviewed: 

Table 8: Comparison of AMS and the FAR: 

Table 9: Key Recommendations Made to Improve FAA's Acquisition 
Processes: 

Figures: 

Figure 1: Scope of AMS and the FAR: 

Figure 2: Review Process under Our Best Practices Model and under AMS: 

Figure 3: Key Milestones for Selected ATC Acquisitions Initiated before 
and after AMS: 

Figure 4: Our Analysis of FAA's Progress as of 2003 in Implementing Key 
Elements of Training for Its Acquisition Workforce: 

Abbreviations: 

AMS: Acquisition Management System: 

ASR-11: Airport Surveillance Radar, Model 11: 

ATC: air traffic control: 

ATO: Air Traffic Organization: 

ATOP: Advanced Technologies and Oceanic Procedures: 

BOE: Cost Basis of Estimate: 

CFO: Chief Financial Officer: 

CIO: Chief Information Officer: 

COO: Chief Operating Officer: 

CMMI: CMMI Capability Maturity Model Integration: 

COTS: Commercial-off-the-shelf: 

DOD: Department of Defense: 

DOT: Department of Transportation: 

DOTIG: Department of Transportation Office of Inspector General: 

ERAM: En Route Automation Modernization: 

F&E: Facilities and Equipment: 

FAA: Federal Aviation Administration: 

FAR: Federal Acquisition Regulation: 

FMFIA: Federal Managers' Financial Integrity Act of 1982: 

iCMM: integrated Capability Maturity Model: 

ISC: Initial System Configuration: 

IT: Information Technology: 

ITIM: Information Technology Investment Management framework: 

ITWS: Integrated Terminal Weather System: 

JRC: Joint Resources Council: 

LAAS: Local Area Augmentation System: 

NEXCOM: Next Generation Air/Ground Communications System: 

NDI: Non-developmental Item: 

OMB: Office of Management and Budget: 

RE&D: Research, Engineering and Development: 

SEI: Software Engineering Institute: 

SFFAS 4: Statements of Federal Financial Accounting Standards no. 4: 

STARS: Standard Terminal Automation Replacement System: 

WAAS: Wide Area Augmentation System: 

Letter November 12, 2004: 

The Honorable Tom Davis:
Chairman: 
The Honorable Henry A. Waxman: 
Ranking Minority Member: 
Committee on Government Reform: 
House of Representatives: 

In late 1981, the Federal Aviation Administration (FAA) began a 
modernization program to replace and upgrade the National Airspace 
System's (NAS) equipment and facilities to meet the expected increase 
in traffic volume, enhance the margin of safety, and increase the 
efficiency of the air traffic control (ATC) system--the principal 
component of the NAS. Historically, the modernization program has 
experienced cost overruns, schedule delays, and performance shortfalls 
of large proportions and has been on our list of high-risk programs 
since 1995. To date, FAA has spent $41 billion and expects to spend an 
additional $7.6 billion through fiscal year 2007 to, among other 
things, finalize key modernization projects designed to replace radar, 
navigation, communications, and information-processing 
systems.[Footnote 1]

According to FAA, the performance shortfalls in its modernization 
program were due, in part, to restrictions imposed by federal 
acquisition and personnel requirements. In response, Congress passed 
legislation in 1995 that granted FAA unique acquisition and personnel 
exemptions, or flexibilities, and directed FAA to develop a new 
acquisition management policy. FAA issued its new acquisition 
management policy, called the Acquisition Management System (AMS), in 
1996 and began using the new system instead of the Federal Acquisition 
Regulation (FAR). To further address long-standing weaknesses in the 
ATC modernization program, the President and Congress in 2000 directed 
FAA to reorganize and establish a new organization. FAA has just begun 
to do so.

Now that FAA has had several years to implement the earlier procurement 
flexibilities, as well as some time to reorganize, some results of its 
acquisition reform should be discernable. Moreover, FAA's experiences 
in exercising its acquisition flexibilities could provide valuable 
information to Congress in overseeing the use of these flexibilities.

You asked us to review the steps that FAA has taken to reform its 
acquisition of major ATC systems and the impact of the reforms on FAA's 
acquisition outcomes. Specifically, you asked us to (1) compare the 
scope and flexibility of AMS and the FAR, (2) compare AMS with 
commercial best practices for major acquisitions, and (3) examine FAA's 
implementation of AMS and progress in addressing long-standing problems 
with major acquisitions. In addition, you asked us to review FAA's 
general procurement of goods and services; we cover this topic in 
appendix I.

To address the first objective, we compared the topics addressed by, 
and the implementation options afforded to contracting and procurement 
officials under AMS and the FAR. To address the second objective, we 
used a model of best practices that we derived from our body of work on 
how leading private firms manage costly and complex product 
developments and how the Department of Defense (DOD) manages major 
weapon systems acquisitions.[Footnote 2] We used this model to assess 
the extent to which FAA's acquisition management policy mirrors the 
acquisition policies of high-performing organizations in the public and 
private sectors. This model consists of four phases: (1) concept and 
technology development; (2) product development, which includes both 
integration and demonstration activities; (3) production; and (4) 
operations and support. In between these four phases are three key 
knowledge decision points at which commercial firms and the government 
must have sufficient knowledge to make large investment decisions. To 
address the third objective, we selected the seven ATC systems with the 
largest budgets to explore the results of FAA's implementation of its 
acquisition management policy and procedures and to determine how FAA 
has addressed issues found to have contributed to cost, schedule, or 
performance problems. In selecting these seven systems, we ensured that 
some were initiated before and some after April 1996, when FAA 
implemented AMS. While the results of these analyses are not 
generalizable to all of FAA's major ATC acquisitions, they indicate the 
extent to which the agency has made progress in addressing long-
standing problems we have identified. To further assess both the 
implementation and the impact of FAA's acquisition reforms, we reviewed 
our work on FAA's major ATC acquisition efforts since 1996 as well as 
the work of the Department of Transportation's Office of Inspector 
General (DOTIG), FAA, and others. We also reviewed the actions that FAA 
has taken to refine AMS in response to internal and external reviews. 
Finally, to review FAA's procurement of goods and services across the 
agency, we used a commercial best-practices model for taking a more 
strategic approach to procurement, along with interviews with key 
agency officials, to determine whether FAA has begun to analyze 
spending trends to identify opportunities to leverage its buying power. 
We conducted our work from December 2003 through November 2004 in 
accordance with generally accepted government auditing standards. See 
appendix II for additional information on our objectives, scope, and 
methodology.

Results in Brief: 

AMS consists of broad guidance for acquisition life-cycle management--
from defining the requirements for a system through fielding 
(deploying) and decommissioning it (removing it from service). This 
broad guidance contrasts with the rather more detailed and prescriptive 
contract-formation and contract-administration requirements contained 
in the FAR. AMS is broader in scope because it addresses, among other 
areas of life-cycle management, both contract and program management, 
providing both policies and procedures for contracting and a toolset of 
recommended practices for managing individual acquisition projects over 
their life cycles. By contrast, the FAR focuses in far greater detail 
on contracting policies and procedures. FAA managers believe they have 
greater flexibility in interpreting and applying AMS than they would 
have under the FAR, in part because, in areas addressed by both, AMS is 
less directive than the FAR. For example, although AMS states a 
"preference" for competition, FAA personnel may use single-source 
contracting when necessary to fulfill FAA's mission. By contrast, other 
federal agency contracting officials operating under the FAR are 
generally required to seek "full and open competition"--a more rigorous 
standard. These other agency officials can generally use sole-source or 
limited-competition contracting only after higher-level agency 
procurement officials have approved a written justification. In 
addition, FAA contracting personnel operate as part of acquisition 
teams that are responsible to program managers; under the FAR, 
contracting decisions are made by contracting personnel who are 
responsible only to contracting officials. Nonetheless, the FAR also 
affords flexibility because it encourages innovation and addresses a 
wide selection of contracting methods; therefore, procurement officials 
can choose the approach that they consider most appropriate to their 
procurement. According to some current and former FAA procurement 
officials with experience in using both the FAR and AMS, the FAR may 
appear inflexible and cumbersome to inexperienced managers, but those 
who are familiar with it can navigate it effectively.

AMS provides some discipline through its various phases, activities, 
and decision points for acquiring major ATC systems; however, it does 
not ensure the use of a knowledge-based approach found in the best 
practices for managing commercial product developments and DOD 
acquisitions[Footnote 3] that we have identified in numerous past 
reports. Commercial best practices call for specific knowledge to be 
captured and used by corporate-level decision-makers to determine 
whether a product has reached a level of development (product maturity) 
sufficient to demonstrate its readiness to move forward in the 
acquisition process. The capture of such knowledge and its use by 
executives helps to avoid cost overruns, schedule slips, and 
performance shortfalls that can occur if decision-makers commit to a 
system design before acquiring critical technology, design, or 
manufacturing knowledge. AMS has some good features, which indicate a 
process that has some elements of discipline. For example, like the 
best practices model, AMS identifies critical junctures that it terms 
"decision points," the first three of which call for the preparation of 
detailed technical and programmatic information that FAA's corporate 
executive-level body, the Joint Resources Council,[Footnote 4] can use 
to assess whether or not FAA should initiate an acquisition program. 
However, AMS departs from recognized best practices primarily by (1) 
not requiring the attainment of specific knowledge satisfying explicit 
written criteria for decision-makers to use at each key decision point 
and (2) not requiring corporate executive-level oversight at all key 
decisions. For example, AMS allows the Joint Resources Council to 
delegate two key decisions--the decision to begin production and the 
decision to place a system in service. FAA maintains that this approach 
gives program managers flexibility, expedites decision-making, and 
allows those executives with the most knowledge about a major 
acquisition to make key decisions about its continued development. 
FAA's reliance on delegation assumes that managers will inform their 
superiors if they are unable to meet the performance schedules and 
system requirements approved by the Joint Resources Council. However, 
best practices call for more than this, including the use of measurable 
criteria at key points in the acquisition process to ensure that 
specific knowledge has been captured and the independent review of this 
knowledge by corporate executive-level decision-makers before the 
acquisition moves forward in its development. These criteria and 
reviews are particularly important for acquisitions that require a 
large funding commitment, such as those that include the production of 
multiple costly units (e.g., radars and controller workstations). In 
addition, oversight at the corporate-executive or agencywide level is 
needed to ensure consideration of an acquisition's likely impact on 
other agency projects or operations. These departures from best 
practices put FAA's major ATC acquisitions at risk of cost, schedule, 
and performance shortfalls. We are making recommendations to the 
Secretary of Transportation to align AMS more closely with commercial 
best practices.

According to our review of seven major ATC systems and analysis of 
FAA's performance in acquiring major systems, AMS has not resolved 
management problems that FAA experienced before it implemented AMS, but 
the agency is beginning to focus more on the expected results of its 
major acquisitions. (See table 5.) Specifically, our review found that 
AMS did not call for requirements that were specific enough to minimize 
the development of further requirements (requirements growth) or 
unplanned work in five of these systems. This lack of specificity 
resulted in the inadequate development or definition of requirements, 
requirements growth, unplanned work, or a reduction in performance for 
five of these systems. In addition, for three of these systems, FAA 
underestimated the difficulty of modifying available software to 
fulfill its mission needs. Consequently, FAA encountered unexpected 
software development needs, higher costs, and schedule delays. Because 
AMS guidance was not sufficient to account for the risks associated 
with modifying available software, the two systems we reviewed that 
were initiated after AMS's implementation--though currently meeting 
cost and schedule milestones--are nevertheless showing symptoms of 
FAA's past problems with developing requirements and managing software. 
It is too soon to tell if these two systems will remain within their 
cost, schedule, and performance parameters. In addition, our work on 
FAA's major acquisitions, along with that of the DOTIG and others, has 
shown that many of the problems FAA experienced in acquiring major 
systems before 1996 persist under AMS and that effective acquisition 
management, rather than the use of a specific contracting process 
(e.g., the FAR or AMS) is the key to successful acquisitions. To its 
credit, FAA is beginning to focus more on results, largely through its 
new Air Traffic Organization, which has been charged with taking a more 
performance-based approach to managing the agency's major acquisitions. 
This approach includes implementing a training framework for FAA's 
acquisition workforce. While FAA has taken some steps to develop an 
evaluation program with criteria for measuring the extent to which this 
framework is achieving organizational goals by improving the knowledge 
base of FAA's acquisition workforce, at the time of our audit FAA had 
no plans to conduct a comprehensive evaluation. We are making 
recommendations to the Secretary of Transportation to improve FAA's 
development of requirements and management of complex software, and to 
comprehensively evaluate FAA's implementation of the training framework 
to ensure that it is having the intended effect of improving the 
knowledge base of FAA's acquisition workforce. In commenting on a draft 
of this report, FAA said that it generally agreed with the report's 
contents and said that our recommendations would be helpful to them as 
they continue to refine AMS.

Background: 

Maintaining that federal procurement requirements contributed to some 
of its cost, schedule, and performance problems in the 1980s and early 
1990s, FAA sought a statutory exemption from the federal acquisition 
system,[Footnote 5] including the FAR, and those parts of title 5 of 
the United States Code, parts II and III, that govern federal civilian 
personnel management. According to FAA, exemptions from these 
requirements would enable it to streamline its acquisition processes, 
be more responsive to the airline industry's needs, and increase the 
efficiency of ATC operations while maintaining safety. Congress enacted 
legislation in November 1995 that exempted FAA from key federal 
procurement statutes and the FAR, and directed FAA to develop a new 
acquisition management system. In response to these legislative 
initiatives, FAA implemented a new, streamlined acquisition process--
the Acquisition Management System (AMS)--on April 1, 1996.

We developed a knowledge-based model of commercial best practices based 
on our findings about how leading private firms manage costly and 
complex acquisitions effectively--that is, within cost, schedule, and 
performance targets. The use of this knowledge-based model has been 
found to reduce the risks associated with developing products and 
increase the likelihood of successful outcomes. The model divides the 
product development cycle into four phases and related activities. 
Table 1 presents these phases and activities and explains what takes 
place during each.

Table 1: Structure of Best Practices Model for Major Product 
Developments: 

Phase/Activity: 1.Concept and technology development; 
What occurs during this phase or activity: Leading companies work to 
understand their mission needs and confirm that the technologies to be 
used are mature; that is, the technologies needed to meet essential 
product requirements have been demonstrated to work in their intended 
environment.

Phase/Activity: 2. Product development: * Integration; 
What occurs during this phase or activity: Components and subsystems 
are integrated into the product to stabilize the overall system design 
and show that the design can meet the product requirements.

Phase/Activity: 2. Product development: * Demonstration; 
What occurs during this phase or activity: Tests show that the product 
will work as required and can be manufactured within targets.

Phase/Activity: 3. Production; 
What occurs during this phase or activity: Operational test articles 
are built.

Phase/Activity: 4. Operations and support; 
What occurs during this phase or activity: Our best practices model 
does not explicitly cover operations and support activities; however, 
this phase focuses on maintenance of the system through its retirement.

Source: GAO.

[End of table]

AMS provides guidance for selecting and overseeing investments over 
their life cycle. Like our best practices model, it is divided into 
phases and activities, although the divisions sometimes occur at 
different points. Table 2 summarizes AMS's phases and activities.

Table 2: Structure of AMS: 

Phase/Activity: Needs and solution identification: * Mission analysis; 
What occurs during this phase or activity: FAA identifies a capability 
shortfall and determines that it needs an investment to better carry 
out its mission. Recently, FAA began analyzing its mission needs within 
the context of its overall goals for the National Airspace System.

Phase/Activity: Needs and solution identification: * Investment 
analysis; 
What occurs during this phase or activity: FAA, using an investment 
analysis team, evaluates alternatives, selects practical and affordable 
solutions, and develops a baseline of cost, schedule, and performance 
requirements. This document is called the acquisition program baseline.

Phase/Activity: Solution implementation: * System integration; 
What occurs during this phase or activity: Both hardware and software 
components and subsystems are integrated into a product. Also, intra-
and intersystem compatibility are tested and analyzed.

Phase/Activity: Solution implementation: * System demonstration; 
What occurs during this phase or activity: Tests show that the product 
can work as required and be manufactured within targets.

Phase/Activity: Solution implementation: * System production; 
What occurs during this phase or activity: All activities are carried 
out to produce needed quantities. Each end item is tested before it 
leaves the factory to verify that it conforms to specifications and is 
free from manufacturing defects.

Phase/Activity: In-service management; 
What occurs during this phase or activity: All required activities are 
carried out, including directly operating, providing maintenance 
functions (both scheduled and unscheduled), and furnishing technical 
and logistics support for the maintenance of FAA systems, subsystems, 
services, or equipment.

Source: FAA. 

[End of table]

To implement the new, performance-based organization for managing ATC 
modernization and operations, as the President and Congress directed in 
2000, FAA appointed a chief operating officer in August 2003 and 
formally established the Air Traffic Organization (ATO) in February 
2004. ATO, under the direction of a six-member executive council, is 
now responsible for further implementing acquisition reforms for major 
ATC systems.

AMS Is Broader and Less Prescriptive Than the FAR: 

AMS establishes an acquisition life-cycle management system that 
encompasses both contracting and program management, whereas the FAR is 
primarily a contracting system that focuses on contract formation and 
contract administration. As a result, AMS is broader in scope than the 
FAR. See figure 1. In addition, AMS takes the form of guidance. This 
guidance is expressed in documentation of FAA policy, handbooks, 
templates, flowcharts, forms, and standard contract language. It is not 
regulatory. By contrast, the FAR is a set of published regulations--a 
legal foundation that has the force and effect of law for the federal 
agencies that are required to follow it.[Footnote 6] Furthermore, the 
FAR is more detailed and prescriptive in establishing contracting 
requirements and can require more administrative involvement. This 
fundamental difference between AMS and the FAR may suggest to some that 
AMS is more flexible. FAA personnel can choose how to apply AMS's 
provisions to a major acquisition. Nonetheless, procurement officials 
under the FAR also have flexibility because the FAR encourages 
innovation consistent with its direction (and other applicable legal 
requirements), provides a wide selection of contracting solutions, and 
permits contracting officials to choose the methods that they consider 
most suitable for a given situation.

Figure 1: Scope of AMS and the FAR: 

[See PDF for image] 

Note: AMS provides policy for the four phases of life-cycle management, 
as well as 14 functional areas, (e.g., test and evaluation, human 
factors, procurement, real estate, security, and systems engineering).

[A] The NAS in-service decision is a key program milestone that 
authorizes the deployment of a system into the National Airspace System 
after thoroughly testing the system to verify its operational 
readiness.

[End of figure] 

AMS Addresses Both Procurement and Project Management, Whereas the FAR 
Focuses Primarily and in Far Greater Detail on Procurement: 

AMS comprises six policy sections and five appendixes.[Footnote 7] The 
procurement policy section of AMS covers a range of topics, including 
contract funding and administration, contracting with small and 
disadvantaged businesses, and compliance with labor laws. According to 
this section, competition is FAA's preferred method of contracting, but 
single-source contracting is permitted when appropriate to fulfill the 
agency's mission. This policy section also describes the procurement of 
commercially available or nondevelopmental items.

Other sections of AMS cover project management tools that the FAR does 
not address, such as investment analysis, configuration 
management,[Footnote 8] and integrated logistics support.[Footnote 9] 
AMS also addresses areas that fall outside project management and 
procurement, including real property management--an area that becomes 
important when FAA must lease or purchase real property so that it can 
install ATC systems such as radars or antennas on property that it does 
not currently own. FAA's policy directs FAA staff to "conduct this 
business in a fair and equitable manner following best practices." 

Although the FAR includes requirements that address procurement 
planning[Footnote 10] and major systems acquisition,[Footnote 11] it 
does so only in the context of government procurement policy and 
procedure. Agencies subject to the FAR find the broader program 
planning requirements, which appear in AMS but not in the FAR, in 
documents such as the Office of Management and Budget's Circular A-109 
and in their own planning guidance. For example, DOD has issued a 
series of directives and instructions on this subject.[Footnote 12]

The contracting procedures set forth in section 3 of AMS do not 
prescribe detailed contracting procedures for various categories of 
procurements, as do those detailed under the FAR. Instead, AMS provides 
two basic contracting models for obtaining products and services 
through FAA's contracting process. The first model is called "Complex 
and Noncommercial Source Selection" and is used for complex, large-
dollar, developmental, noncommercial items and services. This is the 
model that typically would be used for investments approved by the 
Joint Resources Council. The second model is called the "Commercial and 
Simplified Purchase Method" and is typically used for commercial items 
that are less complex and less costly. Procurements of such products or 
services may be routine in nature and are generally purchased on a 
fixed-price basis. Generally, source selection under AMS follows a 
screening process, with the awardee being selected on a "best value" 
basis from among those who remain in consideration when the selection 
is made.

AMS Provides Broad Guidance While the FAR Establishes Detailed 
Requirements, but Managers Have Flexibility under Both: 

AMS sets out a nonregulatory FAA policy that is binding on FAA 
personnel as FAA employees. AMS also sets out other guidelines that FAA 
states should be followed unless there is a rational basis for doing 
otherwise. AMS is subject to such internal controls and enforcement as 
the Administrator decides and to general overarching legal 
requirements, such as the Government Performance and Results Act of 
1993 (GPRA).[Footnote 13]FAA has also deemed certain acquisition laws 
applicable to its procurements (sometimes with modifications), such as 
the Service Contract Act.[Footnote 14] There is also a legal 
requirement, created by the 1995 legislation exempting FAA from the 
FAR, that small and socially or economically disadvantaged firms be 
given all reasonable opportunities to receive contract awards. FAA has 
adopted a dispute resolution process with some legal 
underpinnings.[Footnote 15] Otherwise, as the preface to AMS states, 
"nothing in this document creates or conveys any substantive [legal] 
rights." In short, although FAA is subject to the general legal 
requirement that government decisions cannot be arbitrary or 
capricious, AMS does not establish regulatory requirements for the 
conduct of procurements and does not create or convey substantive legal 
rights.

In contrast to AMS, the FAR is a set of published regulatory 
requirements. It has the force and effect of law, and agencies that are 
subject to it are bound to follow it. The FAR's requirements provide 
for a range of procurement strategies and approaches. In addition to 
negotiated procurement methods, it allows two-step sealed-bid and two-
phase design-build methods,[Footnote 16] among others. It includes 
streamlined procedures for soliciting and evaluating offers to furnish 
commercial items, as well as permits the use of simplified acquisition 
procedures in a broad range of procurements. Furthermore, the FAR 
supports a diverse selection of available contract types, product-
testing tools, and other tools that an agency's contracting personnel 
may select when conducting an acquisition to meet the agency's needs.

Although contracting personnel in agencies subject to the FAR are 
required to comply with it, they enjoy broad discretion in their 
management of procurements. For example, the FAR allows wide latitude 
in drafting requirements statements, from performance-based statements 
of work to design specifications as necessary. It allows broad 
discretion in framing solicitations and in conducting procurements, 
including scoring proposals, determining how negotiations will be 
conducted, eliminating firms whose proposals are not competitive, and 
selecting the awardees whose proposals afford the government the best 
value when evaluated against the selection criteria established in the 
solicitations.

Because AMS consists of broad guidance while the FAR comprises detailed 
and prescriptive regulatory requirements, FAA managers view AMS as 
giving them more flexibility than they would have under the FAR, 
particularly in two areas--competition and oversight. Whereas the FAR 
generally requires full and open competition, AMS calls for providing 
"reasonable access to" competition to firms interested in obtaining 
contracts--a less rigorous standard than full and open competition. AMS 
further states that the "preferred" method of selecting sources is to 
compete requirements among two or more sources. By contrast, full and 
open competition requires that all responsible sources be permitted to 
compete.[Footnote 17] Under AMS, there is no policy that firms that 
want to participate actually get a chance to do so. Rather, FAA told us 
that its system is beneficial because the agency can use screening 
requests to preselect competing firms, eliminating those firms that FAA 
believes are not likely to receive an award. The following example 
illustrates the differences between AMS and the FAR in their respective 
requirements on exceptions to competition. FAA may contract with a 
single source when this approach is determined to be in the best 
interest of FAA.[Footnote 18] The FAR, however, allows exceptions to 
full and open competition only for certain specified conditions (such 
as unusual and compelling urgency or the availability of only one 
source). The FAR describes in detail the circumstances of these 
conditions and the requirements for using them as justification for not 
providing for full and open competition. The FAR also requires the 
contracting officer to prepare a justification document that must 
generally be approved by higher-level agency procurement officials (up 
to the agency's senior procurement executive) depending on the 
estimated dollar value of the procurement. The content of this 
justification is prescribed by the FAR. When not providing for full and 
open competition, the contracting officer is required under the FAR to 
solicit offers from as many potential sources as is practicable under 
the circumstances. The FAR prohibits contracting if the justification 
for less than full and open competition results from a lack of advanced 
planning. For a more detailed comparison of AMS and the FAR, see 
appendix III.

Although some of the FAA personnel we interviewed see AMS as more 
efficient and flexible than the FAR, other current and former FAA 
procurement officials we interviewed who have experience using both the 
FAR and AMS did not agree that AMS is more flexible than the FAR. 
According to these officials, the FAR may appear inflexible and 
cumbersome to persons who lack experience with it, but those who are 
familiar with it are able to navigate its complexities effectively. The 
FAR requires full and open competition, but as experienced procurement 
personnel know, the system does not break down when emergencies 
necessitate quick and decisive action. For example, we recently 
reported that agencies generally complied with applicable FAR 
requirements in awarding new contracts for work in Iraq using other 
than full and open competition.[Footnote 19]In some circumstances, the 
government's legitimate need for prompt action was sufficient to 
justify selecting a contractor on an expedited basis from among the 
firms that appeared able to meet the government's emergency need. In 
other cases, the agencies reasonably determined that only one source 
could meet their requirements.

AMS Provides Some Discipline but Does Not Ensure a Knowledge-Based 
Approach to Acquisition: 

AMS provides some discipline through its various phases, activities, 
and decision points for acquiring major ATC systems; however, it does 
not ensure the use of a knowledge-based approach found in the best 
practices for managing commercial product developments and DOD 
acquisitions that we have identified in numerous past reports.[Footnote 
20]Commercial best practices call for specific knowledge to be captured 
and used by corporate-level decision-makers to determine whether a 
product has reached a level of development (product maturity) 
sufficient to demonstrate its readiness to move forward in the 
acquisition process. The capture of such knowledge and its use by 
executives helps to avoid cost overruns, schedule slips, and 
performance shortfalls that can occur if decision-makers commit to a 
system design before acquiring critical technology, design, or 
manufacturing knowledge. The absence of these key best practices under 
AMS puts FAA's major ATC acquisitions at risk of cost overruns, 
schedule slips, and performance shortfalls.

Best Practices for Managing Acquisitions Call for a Knowledge-Based 
Approach, Including Criteria for Knowledge Needed and Oversight at the 
Corporate Executive Level: 

Commercial best practices call for managing acquisitions using a 
knowledge-based approach, including (1) using established criteria to 
attain specific knowledge at three critical junctures in the 
acquisition cycle, which we call knowledge points, and (2) requiring 
oversight at the corporate executive level for each of these knowledge 
points. For example, at each knowledge point, successful product 
developers apply specific indicators, or criteria, to determine whether 
they have attained the knowledge they need to move to the next phase or 
activity in the acquisition process. Such developers also conduct 
corporate executive-level reviews to ensure that they obtain the 
insights and perspectives of stakeholders throughout their 
organization. If the knowledge attained does not meet the criteria for 
advancement or if the executive reviewers determine that further 
development is inconsistent with their priorities, the acquisition does 
not move forward. Table 3 summarizes the knowledge points, criteria, 
oversight reviews, and timing of oversight reviews included in our 
model of best practices for major acquisitions.

Table 3: Knowledge-Based Approach Called for in Our Best Practices 
Model: 

Knowledge point: 1. Resources and needs matched; 
Criteria: 
* Match customers' needs with available resources--technology, design, 
time, and funding; 
* Demonstrate that technologies needed to meet essential product 
requirements can work in intended environment; 
* Complete a preliminary product design using systems engineering to 
balance customers' desires and available resources; 
Oversight review: Executive-level review required to initiate the 
program; 
Timing of oversight review: Knowledge point 1 should precede the 
commitment to begin product development.

Knowledge point: 2. Product design stable; 
Criteria: 
* Complete 90 percent of design drawings by critical design review; 
* Obtain stakeholders' concurrence that drawings are complete and 
producible; 
* Review subsystem and system designs; 
* Demonstrate with prototype that design meets users' requirements; 
* Identify critical manufacturing processes; 
Oversight review: Executive-level review required to move to 
demonstration; 
Timing of oversight review: Knowledge point 2 should precede the 
commitment to build prototypes to demonstrate the design.

Knowledge point: 3. Production processes mature; 
Criteria: 
* Demonstrate manufacturing processes; 
* Build and test production prototypes; 
* Test production-representative prototypes to achieve reliability 
goals; 
* Test production-representative prototypes to demonstrate product 
performance in operational environment; 
* Collect statistical process control data; 
Oversight review: Executive-level review required to move to 
production; 
Timing of oversight review: Knowledge point 3 should precede the 
commitment to begin production. 

Source: GAO.

[End of table]

Experience with commercial best practices has shown that to the extent 
that the level of knowledge called for at each knowledge point is not 
attained, organizations take on risks in the form of unknowns that will 
persist into the later stages of development, where they will take more 
time and money to resolve if they become problems. Such problems lead 
to cost increases and schedule delays.

AMS Has Some Good Features but Does Not Ensure That High Levels of 
Knowledge Are Attained Before Major Commitments Are Made: 

AMS has some good features, including phases and key decision points 
indicative of an acquisition process that has some elements of 
discipline; however, AMS does not ensure that high levels of knowledge 
are attained and that corporate executive-level reviews occur before 
major commitments of agency resources are made. For example, like the 
best practices model, AMS identifies critical junctures, which it terms 
"decision points." Three of these decision points occur during the 
initial acquisition phase (mission need, initial investment, and the 
final investment decision). A fourth decision point occurs before 
production, and a fifth decision point occurs before the start of the 
final acquisition phase (in-service management). AMS also calls for 
detailed technical and programmatic information that decision-makers 
can use at the first three decision points to assess whether or not FAA 
should initiate an acquisition program. This information includes a 
final requirements document, a final acquisition program baseline, a 
final investment analysis report, an acquisition strategy paper, and an 
integrated program plan. Finally, AMS, like our best practices model, 
calls for senior executives to review the information and determine 
whether the acquisition is ready to move forward. The FAA executives 
who make the decisions at these points include associate and assistant 
administrators, acquisition executives, the chief financial officer, 
the chief information officer, and legal counsel; they form the Joint 
Resources Council (JRC), FAA's senior decision-making body for major 
ATC acquisitions. Table 4 summarizes this information.

Table 4: AMS's Decision Points, Information Sources, and Oversight 
Reviews: 

Decision point by phase/activity:
Phase: Needs and solution identification; 
* Activity: Mission analysis; 
Decision point: Mission need decision; 
Information sources and oversight reviews: Information sources: Input 
from users in the field and mission need statement; 
Oversight review: JRC review called for to move from mission analysis 
to investment analysis.

Decision point by phase/activity: 
Phase: Needs and solution identification; 
* Activity: Investment analysis; 
Decision Point: Initial investment decision; 
Information sources and oversight reviews: Information sources: Initial 
investment analysis report, initial life-cycle program baseline for the 
most viable alternative, updated initial requirements document and 
action plan for final investment analysis; 
Oversight review: JRC review called for to select a preferred solution.

Decision point: Final investment decision; 
Information sources and oversight reviews: Information sources: Final 
requirements document, final acquisition program baseline, final 
investment analysis report, acquisition strategy paper, integrated 
program plan; 
Oversight review: JRC review called for to move from investment 
analysis to solution implementation.

Decision point by phase/activity: 
Phase: Solution implementation; 
* Activity: System integration; 
* Activity: System demonstration; 
Decision point: Production decision; 
Information sources and oversight reviews: Information sources: 
Determined by JRC; 
Oversight review: JRC may retain or delegate decision making authority.

Decision point by phase/activity: 
* Activity: System production; 
Decision point: In-service decision; 
Information sources and oversight reviews: Information Sources: 
Determined by JRC; 
Oversight Review: JRC review called for to move from solution 
implementation to in-service management; however, the JRC may retain or 
delegate decision making authority.

Decision point by phase/activity: 
Phase: In-service management; 
 
Source: GAO analysis of FAA data.

Note: In this report, we place FAA's "mission analysis" and "investment 
analyses" activities in the "Needs and Solution Identification" phase 
to facilitate comparison with the "concept and technology development" 
phase in our best practices model. Similarly, we place "system 
integration" and "system demonstration" in the solution implementation 
phase for comparative purposes.

[End of table]

AMS departs from the best practices model in two key ways--it does not 
call for high levels of knowledge to be attained at three critical 
junctures (knowledge points), and does not call for corporate 
executive-level oversight at one of five junctures. Specifically, AMS 
does not establish explicit, written criteria for (1) the information 
needed to determine technology maturity at solution implementation, (2) 
releasable drawings at critical design review and production process 
controls at production. Our best practices model calls for attaining 
specific knowledge and setting out criteria for what information should 
be available to help organizations minimize risks in the form of 
unknowns. Risks associated with such unknowns can persist into the 
later stages of development, where they can take more time and money to 
resolve if they become problems, potentially leading to cost increases 
and schedule delays.

In addition, AMS does not provide for corporate executive-level 
oversight reviews at two of the three key junctures where our best 
practices model calls for such reviews. Although AMS calls for three 
Joint Resources Council reviews during the initial acquisition phase--
while our model calls for a single corporate executive-level review--
AMS allows the council to delegate its oversight responsibility later 
in the acquisition process to the program managers within the service 
organization responsible for an acquisition. By contrast, our model 
calls for two corporate executive-level reviews later in the 
acquisition process.

According to FAA, its approach gives program managers flexibility, 
expedites decision-making, and allows the executives with the most 
knowledge about a major acquisition to make key decisions about its 
continued development. FAA's reliance on this approach assumes that the 
program managers will inform higher-level managers if they are unable 
to meet the performance schedules and systems requirements approved by 
the Joint Resources Council. However, although program managers may 
have the most knowledge about their particular acquisition, they may 
not have the agencywide perspective of the Joint Resources Council 
members. Having an agencywide perspective, including a broad 
understanding of an acquisition's potential impact on other agency 
projects and operations, is especially critical when an acquisition 
includes the production of multiple units and requires a substantial 
commitment of agency resources, as do FAA's primarily multimillion-
dollar acquisitions, such as controller workstations and radars.

Because decisions about moving a major acquisition forward require both 
a program manager's specific knowledge of the acquisition itself and a 
senior executive's understanding of the acquisition's potential impact 
on other agency projects and operations, our best practices model calls 
for both measurable criteria at key points in the acquisition process 
to ensure that specific knowledge has been captured and corporate 
executive-level reviews to ensure that senior decision-makers have the 
opportunity to independently consider this knowledge. Without higher-
level reviews such as our best practices model recommends and the Joint 
Resources Council could provide later as well as early in the 
acquisition process, FAA cannot ensure that it has fully considered the 
impact of advancing an acquisition on other agency projects and 
operations. This opportunity for full consideration is a central 
advantage of managing acquisitions as a portfolio, as we concluded in 
our August 2004 report on FAA's information technology investment 
management process.[Footnote 21]

Figure 2 contrasts FAA's process for reviewing an acquisition's 
progress under AMS with the process that we found leads to successful 
commercial acquisitions.

Figure 2: Review Process under Our Best Practices Model and under AMS: 

[See PDF for image] 

[A] To facilitate the comparison of AMS with out best practices model 
in this report, we have done the following: (1) placed FAA's "Mission 
Analysis" and "Investment Analyses" activities in the "Needs and 
Solution Identification" phase to make it comparable with the "concept 
and technology development" phase in our best practices model; (2) 
depicted only the final investment decision point, recognizing that the 
investment analysis phase includes an initial investment decision; and 
(3) placed "system integration" and "system demonstration" in the 
solution implementation phase.

[B] AMS does not explicitly call for a design review decision point, 
which would fall between system integration and system demonstration.

[C] The in-service decision is a key program milestone. It authorizes 
the deployment of a system into the National Airspace System. At times, 
the JRC delegates its decision authority for the production and in-
service decisions to service organizations.

[End of figure] 

To its credit, FAA continues to improve its AMS process. For example, 
the agency is currently modifying its mission needs activity to make 
the selection of major ATC acquisitions more consistent with the 
overall goals of modernizing the National Airspace System. In addition, 
the Air Traffic Organization has established an executive council to 
review major acquisitions before they are sent to the Joint Resources 
Council. This review is designed to screen acquisitions to determine 
which ones are important enough to warrant higher-level review by the 
Council. Finally, FAA is currently revising AMS to bring it in line 
with the Office of Management and Budget's guidance. Specifically, the 
agency is incorporating OMB Exhibit 300, which provides the investment 
justifications and management plans required for major ATC 
acquisitions.

As Implemented, AMS Has Not Resolved Long-standing Acquisition 
Problems, but FAA Is Beginning to Focus More on Results: 

According to our review of seven major ATC systems and analysis of 
FAA's performance in acquiring major systems, AMS has not resolved the 
long-standing problems that FAA experienced before implementing AMS, 
but the agency is beginning to focus more on the expected results of 
its major acquisitions. (See table 5.) Specifically, our review found 
that AMS guidance did not call for requirements that were specific 
enough to minimize requirements growth or unplanned work for five of 
these systems. This lack of specificity resulted in the inadequate 
development or definition of requirements, growth in requirements, 
unplanned work, or a reduction in performance for five of these 
systems. In addition, for three of these systems, FAA underestimated 
the difficulty of modifying available software to fulfill its mission 
needs. Because AMS guidance was not sufficient to account for the risks 
associated with modifying available software, FAA encountered 
unexpected software development needs, higher costs, and schedule 
delays. The two systems we reviewed that were initiated after AMS was 
implemented are currently meeting cost and schedule milestones; 
however, both systems are showing symptoms of FAA's past problems with 
developing requirements and managing software, and it is too soon to 
tell if these programs will remain within their cost, schedule, and 
performance parameters. In addition, our work on FAA's major 
acquisitions, along with that of the DOTIG and others has shown that 
the problems FAA experienced before 1996 in acquiring major systems 
persist under AMS and that effective acquisition management, rather 
than the use of a specific contracting process (e.g., the FAR or AMS) 
is key to successful acquisitions. To its credit, FAA is beginning to 
focus more on results, largely through its new Air Traffic 
Organization, which has been charged with taking a more performance-
based approach to managing the agency's acquisitions.

Table 5: Description and Status of Seven Selected ATC Acquisitions: 

Dollars in millions; 

Project and description: STARS--new controller and maintenance 
workstations to replace the legacy system at terminal air traffic 
control facilities[A]; 
Original cost: $940.0; 
Current cost: $1,460.0; 
Original schedule: 1998; 
Current schedule: 2003; 
Acquisition issues and status: STARS is a joint FAA and DoD program. 
STARS delays and cost increases resulted from poor requirements 
definition and schedule estimates. STARS is fully operational at 25 FAA 
terminal radar facilities and 17 DoD facilities. Only 50 of the planned 
172 systems are being deployed. STARS had difficulties in achieving 
many human factor requirements for improving system efficiency and 
safety.

Project and description: ASR-11--digital radar for terminal 
environments; 
Original cost: $743; 
Current cost: $891.7; 
Original schedule: 1997; 
Current schedule: 2013; 
Acquisition issues and status: ASR-11 was approved for its in-service 
decision in September 2003 and is being deployed at 108 sites. These 
systems are being deployed at a slower pace than originally planned 
because of budget cuts and deferrals.

Project and description: ITWS--computer processors and displays to 
automate weather data near the airport; 
Original cost: $276.1; 
Current cost: $288.3; 
Original schedule: September 2001; 
Current schedule: 2002; 
Acquisition issues and status: Currently, six ITWS systems are 
operational. In May 2004, the ATO Executive Council rebaselined the 
program to include a new weather-forecasting capability into the 
production baseline. FAA proposes to defer 12 of the 34 systems it 
planned to procure.

Project and description: LAAS--a precision approach and landing system 
that augments the Global Positioning System; 
Original cost: $530.1; 
Current cost: $696.1; 
Original schedule: 2002; 
Current schedule: Deferred at least until 2009; 
Acquisition issues and status: LAAS has been adversely affected by poor 
requirements development, a lack of understanding of its technical 
complexity, incomplete software development, and an unrealistic 
development schedule. Unresolved radio interference precludes the safe 
operation of LAAS. As a result, FAA has delayed national deployment to 
continue further research on this issue.

Project and description: NEXCOM--digital radios to improve air traffic 
communications; 
Original cost: $318.4; 
Current cost: $318.4; 
October 2002; 
Current schedule: 2004; 
Acquisition issues and status: NEXCOM program delays were due to 
misunderstanding of a program requirement and testing procedures. 
NEXCOM was recently approved for its in-service decision in July 2004.

Project and description: ATOP--new workstations and processing 
capability to control ocean air traffic; 
Original cost: $548.2; 
Current cost: $548.2; 
Original schedule: June 2004; 
Current schedule: 2004; 
Acquisition issues and status: ATOP achieved its acquisition program 
baseline objectives; however, this baseline does not reflect program 
delays and cost increases resulting from poor requirements development, 
unrealistic schedule estimates, and inadequate evaluation of software 
complexity.

Project and description: ERAM--upgrades the existing en route system 
with improved hardware and software; 
Original cost: $3,649.0; 
Current cost: $3,649.0; 
Original schedule: December 2009; 
Current schedule: December 2009; 
Acquisition issues and status: To date, ERAM has not breached any cost 
and schedule parameters. However, it remains a high-risk program 
because of the large amount of software that must be developed. The 
ERAM contractor is experiencing software engineering difficulties as a 
result of lower-than-expected productivity and software code growth. 

Source: GAO analysis of FAA data.

[A] Terminal air traffic control facilities, known as Terminal Radar 
Approach Control (TRACON) facilities, direct aircraft in the airspace 
that extends from the point where the tower's control ends to about 50 
nautical miles from the airport. A TRACON can be located at or outside 
an airport.

[End of table]

Our Reviews of Seven Major Systems Show That Problems with Requirements 
and Software Management Persist under AMS: 

Our reviews of seven of FAA's costliest ATC system acquisitions found 
that the problems FAA experienced with requirements and software 
management and their related impact on cost, schedule, and performance 
goals persist today under AMS.[Footnote 22]Figure 3 identifies these 
seven acquisitions and their milestones, which are expressed in terms 
of AMS decisions even when the acquisitions were initiated before AMS 
was implemented. (See app. V for a description and the status of each 
of these projects.) Specifically, for 6 of these 7 major ATC 
acquisitions, FAA did not consistently (1) clearly define system 
requirements at the investment decision point or (2) adequately assess 
software complexity. Moreover, as FAA has acknowledged, it has never 
managed its major acquisitions by focusing on how each would improve 
the efficiency of ATC operations while maintaining or improving safety. 
Although FAA has made progress in improving its acquisition of major 
ATC systems--by, for example, improving the maturity of its processes 
for acquiring software, using a "build a little, test a little" 
approach to acquisitions as it did for Free Flight Phase 1,[Footnote 
23] and restructuring its organization to minimize stovepipes--long-
standing problems persist in these areas. In addition, the two systems 
we reviewed that were initiated after AMS's implementation are 
currently operating within cost and schedule goals; however, they are 
showing symptoms of past problems with developing requirements and 
managing software complexity. Moreover, our work for more than two 
decades--before and after AMS's implementation--has cited these types 
of weaknesses as central reasons for the agency's long history of cost, 
schedule, and performance shortfalls. This work has also found that the 
effectiveness of an agency's acquisition management has had a greater 
impact on the success of its major acquisitions than the contracting 
process used (e.g., the FAR or AMS).

Figure 3: Key Milestones for Selected ATC Acquisitions Initiated before 
and after AMS: 

[See PDF for image] 

[End of figure] 

Inadequate Development or Definition of Requirements Led to 
Requirements Growth or Unplanned Work for Five Acquisitions: 

For five of the seven acquisitions we reviewed, AMS guidance did not 
call for requirements that were specific enough to minimize 
requirements growth or unplanned work. For four of these five 
acquisitions--STARS, LAAS, NEXCOM, and ATOP--incomplete and poorly 
defined requirements in the final requirements documents, used at the 
investment decision point to assess an acquisition's readiness to enter 
the development phase, led to requirements growth, unplanned 
development work, or a reduction in system performance.[Footnote 24]For 
the fifth acquisition--ASR-11--FAA misjudged the extent to which the 
high-level requirements that were used to support the commercial-off-
the-shelf/nondevelopmental item (COTS/NDI) procurement by the 
Department of Defense could result in a product capable of meeting 
FAA's mission or user needs. As a result, unplanned software changes 
were required.

* FAA's cost estimate for STARS has grown from its original estimate of 
$0.94 billion in 1996 to $1.46 billion in 2004 and will deploy only 50 
of the 172 STARS initially planned. Much of the cost growth has been 
due to FAA requirements creep. As a result, the STARS program has 
experienced delays of more than five years from its original plan, in 
part due to added requirements to the commercial-off-the-shelf Initial 
System Configuration (ISC). However, the STARS ISC was satisfactory for 
use by the Department of Defense as deployed.

* A final requirements document was approved, and the development of 
LAAS was scheduled to begin in 1999. However, poorly established 
requirements resulted in the addition of 113 new requirements to the 
initial specification, entailing significant software and hardware 
changes. Furthermore, LAAS may not achieve its promised capabilities 
because FAA has been unable to develop technologies necessary to warn 
pilots of a disruption in the LAAS signal. Until this technology is 
developed, LAAS cannot be operated safely. As a result, FAA recently 
cut the fiscal year 2005 funding for LAAS, and the program will revert 
to a research and development effort.

* FAA developed a final requirements document for the NEXCOM system, 
but the requirements lacked the specificity needed to assess the 
development risk. According to a NEXCOM contractor program official, 
this led to miscommunication about the program requirement relating to 
signal interference. This official stated that they misunderstood this 
requirement and had not planned on the additional development work for 
the NDI solution to meet such program objectives and delayed the 
program 21 months. Another program requirement involved the NEXCOM 
radios meeting or exceeding the operational coverage area of the 
existing voice system. The existing radios had power output levels of 
50 watts but the NEXCOM contractor could only achieve 34 watts of power 
to meet the coverage requirement. A program official stated that the 
contractor and FAA had not agreed on the testing procedures to assess 
the power levels. This posed an "unacceptable consequence" and, as a 
result, FAA performed additional testing or flight checks of the 
reduced radio performance (50 watts versus 34 watts) and determined 
that the performance reduction should not affect NEXCOM's mission or 
its coverage requirement.

* FAA did not follow the AMS guidelines that call for completing a 
final requirements document before proceeding to the development phase 
for ATOP. The Joint Resources Council approved a delay in developing 
the final requirements until after contract award. This decision 
resulted in schedule delays and additional unplanned software 
development. The ATOP program office asserted that the requirements 
remained very stable and that the program is within cost and schedule 
objectives established by the Council. However, FAA's internal 
documents revealed that the requirements were not adequately defined. 
For example, the ATOP Investment Analysis Study reported to the Joint 
Resources Council prior to contract award that the lack of more 
detailed ATOP requirements at this stage of acquisition added risk and 
was of concern to the investment analysis team. Under AMS, this team is 
responsible for, among other things, conducting risk analyses for the 
various acquisitions. Furthermore, an ATOP Assessment Team conducted a 
study in March 2003 and determined that at the ATOP contract award, 
"requirements were written at a high level and not mutually understood 
by FAA and the contractor." However, FAA management allowed the ATOP 
program to proceed to solution implementation without the final 
requirements document and, according to the contractor, this resulted 
in schedule delays and growth in the amount of software needing 
development.

* The high-level requirements for ASR-11, jointly generated by FAA and 
the Department of Defense, to support a COTS/NDI acquisition, resulted 
in a product that did not initially meet the FAA mission or user needs. 
The software changes that were required to meet FAA's target detection 
needs, as well as significant hardware design changes, parts 
obsolescence, and production issues, added approximately two years to 
system qualification and acceptance.

FAA Underestimated Software Complexity for Three Systems: 

For three of the seven major ATC acquisitions we reviewed--ITWS, LAAS, 
and ATOP--FAA's AMS guidance was not sufficient to address the risks 
associated with modifying available software[Footnote 25] to fulfill 
FAA's mission needs. In all three cases, FAA officials underestimated 
the difficulty of modifying available software. Our work has shown that 
underestimates are likely to result in unexpected software development, 
higher costs, and schedule delays.

* ITWS experienced delays from the beginning because of the complexity 
of its software development. Although the program appeared to be 
progressing according to its baseline, immediately after the critical 
design review in September 1998, the contractor revealed that it had 
exceeded the target cost by $4 million. In addition, the contractor 
claimed that the program did not recognize that the computer processor 
originally planned for the program was becoming outdated, that the 
manufacturer planned to discontinue its production because the market 
was demanding a processor with greater processing and storage 
capability, and that as a result, the original computer processor would 
not be available to the program. Consequently, ITWS experienced cost 
increases, schedule delays, and performance shortfalls. According to 
the contractor and the original acquisition plan, all systems were 
scheduled for delivery by December 2001, but that date has now 
stretched to after 2009.

* LAAS's technology maturity was not adequately assessed, and further 
development was needed. Specifically, the potential for radio 
interference through the atmosphere was not understood and could limit 
LAAS's operations. FAA has now placed all LAAS activities in research 
and development. FAA did not adequately assess LAAS's software 
development. At the time of the contract award, the contractor and FAA 
estimated that 80 percent of the software that LAAS required had been 
developed. FAA later determined that only 20 percent had been 
developed. FAA and the contractor attribute this discrepancy to a lack 
of communication on the steps necessary to satisfy the program's 
requirements. FAA agrees that it should have conducted a software audit 
and a software capabilities assessment, but pressures to keep LAAS on 
schedule resulted in an inadequate assessment.

* The ATOP contractor underestimated by about half the extent to which 
legacy nondevelopmental item software, which is the core of the ATOP 
system, met the program's 1,036 requirements. As a result, a 
significant amount of unanticipated new software code development and 
other modifications were required.[Footnote 26]

ATC Systems Have Required Multiple Rebaselining Decisions to Address 
Delays and Cost Growth: 

As figure 3 illustrates, FAA initiated at least one rebaselining 
decision for three of the five acquisitions that were begun before AMS 
was implemented and were later transitioned to AMS. These rebaselining 
decisions responded to delays and cost growth--problems that arise when 
requirements are not stable, a program's design is not fixed, or 
software code growth is not controlled. For example, FAA rebaselined 
STARS two times--first in 1999 and again in 2002. Similarly, 2 years 
after the investment decision for ITWS, FAA rebaselined the program 
twice, in 1997 and again in 2001. Given the frequency of these past 
rebaselining decisions for major ATC systems and the number of years 
that elapsed before or between the rebaselining decisions (3 to 4 
years), it is too soon to tell whether the two systems that were 
initiated under AMS--ATOP and ERAM--will require similar rebaselinings 
and ultimately meet their cost, schedule, and performance goals. 
Although both programs are currently operating within their cost and 
schedule goals and have not yet been rebaselined, FAA has had problems 
with managing its major acquisitions in the past and is currently 
having difficulties developing requirements and managing software 
complexity. Furthermore, as we reported in May 2004, FAA's budget 
increased from $9 billion in 1998 to $14 billion in 2004 but will be 
constrained for the foreseeable future. In such a constrained budget 
environment, cost growth and schedule problems can have serious 
negative consequences for ongoing modernization efforts--postponed 
benefits, costly interim systems, delays in funding other systems, or 
reductions in the number of units purchased.

Internal and External Reviews Have Found That FAA Has Made Some 
Progress but Continues to Experience Problems in Acquiring Major ATC 
System under AMS: 

Reviews of FAA's acquisition process, conducted by FAA, GAO, the DOTIG, 
and others have shown that FAA has improved its management of major ATC 
acquisitions in recent years but continues to experience cost overruns, 
schedule slips, and performance shortfalls under AMS. Table 7 
summarizes the results of 22 internal and external reviews of FAA's 
major ATC acquisitions. According to these reviews, issued from 1997 
through 2004, the same problems have persisted over many years, despite 
various initiatives to address them, and FAA needs to strengthen its 
management controls. For example, a key FAA review of eight major ATC 
acquisitions, published in 1999, 3 years after AMS was implemented, 
found that these acquisitions, though on track to meet their 
performance goals, were not meeting their cost and schedule baselines. 
FAA attributed these cost and schedule issues to new or poorly 
understood requirements, underestimates of the acquisitions' technical 
complexity, and funding shortfalls.

In addition, our reviews of major FAA acquisitions--initiated before 
and after AMS was implemented--have found for more than two decades 
that FAA's failure to meet schedule, cost, and performance baselines 
for major ATC acquisitions has been due to shortfalls in planning, weak 
management controls, and a lack of systematic processes for acquiring 
new systems, including inadequate requirements management, cost-
accounting data, and estimates of technical difficulty. As we reported 
in August 2004, judged against the criteria of GAO's framework for 
information technology (IT) investment management, which measures the 
maturity of an organization's investment management processes, FAA has 
established about 80 percent of the basic selection and control 
practices that it needs to manage its mission-critical investments for 
the National Airspace System.[Footnote 27] For example, FAA's business 
units actively monitor projects throughout their life cycles.[Footnote 
28]However, the agency's senior IT investment board does not regularly 
review investments that are in the "in-service management," or 
operational phase, and this creates a weakness in FAA's ability to 
oversee more than $1 billion of its IT investments. In addition, the 
agency has not yet established the practices that would enable it to 
effectively manage its annual IT budget of about $2.5 billion, and 
agency executives lack assurance that they are selecting and managing 
the mix of investments that best meets the agency's needs and 
priorities. DOT has responded to our recommendations to FAA to 
strengthen its IT investment management capability.

Moreover, other reviews, such as those by Booz-Allen & Hamilton and 
MITRE, have identified other shortfalls, which reflect a lack of proper 
management controls and planning. For example, in 1997, Booz-Allen & 
Hamilton found, among other things, that FAA had not clearly defined 
organizational roles and responsibilities within the various phases of 
AMS and that greater guidance and training under AMS were warranted. In 
1999, Booz-Allen & Hamilton reported that FAA had not demonstrated 
improvement in adhering to planned costs and schedules under AMS and 
that the agency needed to better manage its development of requirements 
and address persistent funding shortfalls. Moreover, in 2001, a MITRE 
report on selected major acquisitions found inadequate management 
controls and deficiencies in both contractors' performance and in FAA's 
measurement of acquisition performance. See table 7 for a chronological 
listing of the reviews.

Table 6: Internal and External Reviews of FAA's Use of AMS for 
Acquiring Major ATC Systems: 

Review: GAO, Air Traffic Control: Improved Cost Information Needed to 
Make Billion-Dollar Modernization Investment Decisions, GAO/AIMD-97-
20, (Washington, D.C.: Jan. 22, 1997); 
Selected findings: FAA's cost- estimation practices do not satisfy 
recognized estimating requisites, increasing the likelihood of poor 
acquisition selection decisions; 
Contributing factors: FAA's cost-accounting practices do not provide 
for the proper accumulation of actual project costs.

Review: GAO, Air Traffic Control: Complete and Enforced Architecture 
Needed for FAA Systems Modernization, GAO/AIMD-97-30, (Washington, 
D.C.: Feb. 3, 1997); 
Selected findings: Incompatibilities exist between current and planned 
ATC acquisitions, resulting in high costs and reduced performance; 
Contributing factors: FAA lacks a complete systems architecture or 
overall "blueprint" to guide and constrain the development and 
maintenance of ATC acquisitions.

Review: GAO, Air Traffic Control: Immature Software Acquisition 
Processes Increase FAA System Acquisition Risks, GAO/AIMD-97-47, 
(Washington, D.C.: Mar. 21, 1997); 
Selected findings: Planned acquisitions frequently are not delivered on 
time and within budget; 
Contributing factors: Weaknesses in some key process areas, such as 
planning, requirements development, and management, limit FAA's ability 
to consistently acquire software-intensive ATC systems on time and 
within budget.

Review: FAA, Evaluation of FAA Acquisition Reform--The First Year: 
April 1996 - March 1997, (Washington, D.C.: May 1997); 
Selected findings: AMS addresses 15 of the 17 problems facing 
acquisitions; 
Contributing factors: Inadequate management has not enabled FAA to meet 
its goals of reducing acquisition deployment time by 50 percent and 
cost by 20 percent.

Review: FAA, Evaluation of FAA Acquisition Reform--The First Two Years: 
April 1996 - March 1998, Report #1998-02, (Washington, D.C.: May 29, 
1998); 
Selected findings: Further improvements are necessary if acquisition 
reform is going to allow FAA to meet its cost and schedule goals; 
Contributing factors: Procedural weaknesses limit FAA's ability to 
achieve cost and schedule goals.

Review: GAO, Air Traffic Control: Observations on FAA's Air Traffic 
Control Modernization Program, GAO/T-RCED/AIMD-99-137, (Washington, 
D.C.: Mar. 25, 1999); 
Selected findings: From the inception of its modernization efforts, FAA 
has not consistently followed a disciplined management approach for new 
acquisitions; 
Contributing factors: Weaknesses persist in key areas, such as how FAA 
monitors the status of its acquisitions throughout their life cycles.

Review: GAO, Air Traffic Control: FAA's Modernization Investment 
Management Approach Could Be Strengthened, GAO/RCED/AIMD-99-88, 
(Washington, D.C.: Apr. 30, 1999); 
Selected findings: AMS contained weaknesses in the selection of 
acquisitions and in the review of acquisitions' performance during the 
postimplementation phase; 
Contributing factors: FAA lacked adequate cost data for making 
selection decisions; adequate management controls, and a defined, 
documented process for conducting reviews during the in-service 
management phase.

Review: FAA, Evaluation of FAA Acquisition Reform--The First Three 
Years: April 1996 - March 1999, Report #1999-04,; (Washington, D.C.: 
May 28, 1999); 
Selected findings: FAA's cost and schedule plans were not on track, but 
performance plans were met; 
Contributing factors: Requirements changed or were misunderstood; 
technical difficulties were underestimated; and funding fell short.

Review: Booz-Allen & Hamilton, Independent Assessment of the Federal 
Aviation Administration's Acquisition Management System, (McLean, VA: 
July 6, 1999); 
Selected findings: FAA has yet to implement a seamless life-cycle 
approach to acquisitions management; 
Contributing factors: AMS is not being consistently implemented across 
all life-cycle phases.

Review: GAO, National Airspace System: Persistent Problems in FAA's New 
Navigation System Highlight Need for Periodic Reevaluation, GAO/RCED/
AIMD-00-130, (Washington, D.C.: June 12, 2000); 
Selected findings: FAA experienced delays and cost increases in 
developing its global positioning navigation system; as a result, it is 
unclear whether the benefits of the system will outweigh the cost; 
Contributing factors: FAA lacks a comprehensive plan with checkpoints 
for reviewing the contractor's approach to meeting the system's 
performance requirements.

Review: GAO, National Airspace System: Problems Plaguing the Wide Area 
Augmentation System and FAA's Actions to Address Them, GAO/T-RCED-00-
229, (Washington, D.C.: June 29, 2000); 
Selected findings: FAA experienced cost and schedule problems in 
developing this navigational system because of unplanned software 
development needs and a requirement to warn pilots of any system 
failure that would provide misleading information; 
Contributing factors: FAA underestimated the complexity of developing 
the acquisition.

Review: GAO, National Airspace System: Free Flight Tools Show Promise, 
but Implementation Challenges Remain, GAO-01-932, (Washington, D.C.: 
Aug. 31, 2001); 
Selected findings: Three acquisitions that are components of FAA's 
planned new approach for air traffic management have uncertain 
potential benefits and may not be worth FAA's investment; 
Contributing factors: FAA needs better data collection and analysis 
processes to ensure that benefits are realized.

Review: GAO, National Airspace System: Better Cost Data Could Improve 
FAA's Management of the Standard Terminal Automation Replacement 
System, GAO-03-343, (Washington, D.C.: Jan. 31, 2003); 
Selected findings: The reliability of the life-cycle cost estimate for 
STARS is uncertain because cost data obtained from the contractor do 
not reflect the current status of the contract; 
Contributing factors: The development cost estimate is based on the 
contractor's projections, which FAA has not yet independently analyzed, 
as called for under AMS.

Review: GAO, National Airspace System: Current Efforts and Proposed 
Changes to Improve Performance of FAA's Air Traffic Control System, 
GAO-03-542, (Washington, D.C.: May 30, 2003); 
Selected findings: FAA was unable to hire a chief operating officer to 
head the ATO; 
Contributing factors: Uncertainties about the position's 
responsibilities, reporting relationships, and performance measurement 
criteria hampered the hiring.

Review: DOT/OIG, Status of FAA's Major Acquisitions, AV-2003-045, 
(Washington, D.C.: June 26, 2003); 
Selected findings: Cost growth, schedule delays, and performance 
problems continue with FAA's major acquisitions; 
Contributing factors: Cost and schedule baselines are not reliable, and 
decisions are being made with unclear data.

Review: GAO, Air Traffic Control: FAA's Modernization Efforts--Past, 
Present, and Future, GAO-04-227T, (Washington, D.C.: Oct. 30, 2003); 
Selected findings: Systemic management issues, including inadequate 
management controls and human capital issues, have contributed to major 
ATC acquisitions' persistent cost overruns, schedule delays, and 
performance shortfalls; 
Contributing factors: FAA lacked the information technology and 
financial management systems that would have helped it reliably 
determine the acquisitions' technical requirements and estimate and 
control their costs and schedules; and the agency's organizational 
culture discouraged collaboration among technical experts and users.

Review: GAO, Information Technology: FAA Has Many Investment Management 
Capabilities in Place, but More Oversight of Operational Systems Is 
Needed, GAO-04-822, (Washington, D.C.: Aug. 20, 2004); 
Selected findings: Although weaknesses remain, FAA has established 
about 80 percent of the basic practices needed to manage its mission-
critical acquisitions so that it can be assured that it is selecting 
and managing the mix of investments that best meets its needs and 
priorities; 
Contributing factors: Remaining weaknesses include inadequate 
management controls and the lack of a defined, documented process for 
conducting reviews during the in-service management phase.

Review: GAO, Air Traffic Control: System Management Capabilities 
Improved, but More Can Be Done to Institutionalize Improvements, GAO-
04-901, (Washington, D.C.: Aug. 20, 2004); 
Selected findings: FAA made progress in improving its system management 
capabilities, but can do more to institutionalize process improvement 
initiatives; 
Contributing factors: Process improvement efforts have not been 
institutionalized. 

Source: GAO analysis.

[End of table]

FAA's ATO Is Taking Steps to Improve Major ATC Acquisitions: 

FAA's recent reorganization, which brought ATC acquisitions and 
operations together in the ATO,[Footnote 29] is expected to help the 
agency address many of the concerns we have identified for more than 
two decades, including those identified in this report. For example, 
the ATO is continuing to develop and refine specific guidance for 
critical areas, such as requirements management, software development, 
and cost estimation. In addition, as the overseer of both ATC 
acquisitions and operations, the ATO is in a position to facilitate 
more effective management of major ATC acquisitions than has occurred 
in the past. The ATO is attempting, for example, to link acquisition 
decisions directly with expected improvements in operational efficiency 
without compromising safety. This is important, given that FAA has 
spent about $2.5 billion on ATC modernization per year since 1996 while 
operating costs have continued to rise--from $4.6 billion to $7.5 
billion over the past decade. FAA had not completed its reorganization 
or implemented all of its initiatives at the time of our audit.

Improvements to Requirements Development: 

With the establishment of the ATO, FAA consolidated requirements 
development from two organizations (the organization sponsoring an 
acquisition and the former agencywide acquisition organization) into a 
single new organization--the Air Traffic System Requirements 
Service.[Footnote 30] In addition, the ATO developed guidance to better 
manage requirements during the middle phase of AMS (solution 
implementation). According to FAA officials, some more complex 
development efforts may need to develop systems requirements and a more 
detailed requirements document than AMS currently calls for in the 
final requirements document. More important, in January 2003, FAA 
issued guidance on requirements management, Roles in Requirements 
Management During Solution Implementation Phase, which provides for 
integrated requirements teams that maintain responsibility for 
requirements management throughout an acquisition's life cycle. 
According to this guidance, when the final requirements document is 
accepted by the Joint Resources Council at the investment decision 
point, a requirements baseline is established and any proposed changes 
to the requirements must be assessed for their impact on the program 
and shown to be operationally suitable, affordable, executable, and 
justifiable. An FAA official on an integrated requirements team stated 
that any changes that may affect an acquisition's cost and the schedule 
require approval by the Executive Committee. The FAA official also 
stated that this guidance has already helped to stabilize NEXCOM's 
requirements during the solution implementation phase. Other FAA 
officials representing the Joint Resources Council acknowledged that 
the guidance should ensure greater control over program requirements 
growth, but said that not all program offices have consistently applied 
it.

Improvements to Managing Software and System Acquisition and 
Development: 

To better manage software programs for ATC modernization acquisitions, 
FAA established a centralized process improvement office that reports 
to the Chief Information Officer (CIO).[Footnote 31] This office 
developed an FAA integrated capability maturity model (i-CMM), a 
software development and management model that is similar to a model 
developed by Carnegie Mellon University called the Capability Maturity 
Model Integration (CMMI®), which is used to appraise the maturity of an 
organization's processes for acquiring software. However, FAA's i-CMM 
goes beyond Carnegie Mellon's model to reflect international standards. 
The CMMI® appraisal methodology calls for assessing process areas--such 
as project planning, requirements management, and quality assurance--by 
determining whether key practices are implemented and overarching goals 
are satisfied. Both the i-CMM model and CMMI® appraisal methodologies 
provide a logical framework for measuring and improving key processes 
needed for achieving quality software and systems.

However, as we reported in August 2004,[Footnote 32] FAA projects are 
not required to use the capability maturity model for process 
improvement, and individual projects that use the i-CMM model are 
allowed to choose which process areas they seek to improve and to 
determine when they are ready for an appraisal of their progress. To 
date, fewer than half of FAA's major ATC projects have used this model. 
The recurring weaknesses we identified in our project-specific 
evaluations are due in part to the flexibility these projects were 
given in deciding whether and how to adopt this process improvement 
initiative. Furthermore, after combining its ATC organizations into a 
single performance-based organization (the ATO), FAA is reconsidering 
prior policies, and it is not yet clear whether process improvement 
will remain a priority. Without a strong senior-level commitment to 
process improvement and a consistent, institutionalized approach to 
implementing and evaluating it, FAA cannot ensure that key projects 
will continue to improve systems acquisition and development 
capabilities. As a result, FAA will continue to risk the project 
management problems--including cost overruns, schedule delays, and 
performance shortfalls--that have plagued past acquisitions. To address 
these shortcomings, we recommended that the Secretary of Transportation 
address specific weaknesses and institutionalize FAA's process 
improvement initiatives by establishing a policy and plans for 
implementing and overseeing process improvement initiatives.

Improvements to Estimating Costs: 

FAA has taken steps to improve its cost estimation for major ATC 
projects by issuing guidance on how to develop and use pricing under 
AMS. For example, AMS policy calls for audit trails to record and 
explain the values that are used as inputs to cost models. In addition, 
it calls for agency officials, when reporting to executive oversight 
agencies and Congress, to disclose the level of uncertainty and 
imprecision that are inherent in cost estimates for major ATC systems. 
According to AMS policy, estimators record the procedures, ground rules 
and assumptions, data, environment, and events that underlie their 
development or update of a cost estimate. This information supports the 
credibility of the cost estimate, aids in the analysis of changes in 
program costs, enables reviewers to assess the cost estimate 
effectively, and contributes to the population of FAA databases that 
can be used for estimating the cost of future programs. Finally, 
despite a delay of many years, FAA officials told us that they are in 
the final stages of completing the agency's cost-accounting system and 
plan to have it in place across the agency by the end of this calendar 
year, which will bring FAA into compliance with the Federal Managers' 
Financial Integrity Act of 1982. This measure will help reduce the 
likelihood of cost overruns or improper payments for unallowable costs 
and provide decision-makers with critical information. As we have 
reported in the past,[Footnote 33] a cost-accounting system is critical 
to managing major ATC acquisitions, because without it, FAA lacks the 
information it needs to reliably estimate operating costs over an 
acquisition's life cycle.

Other Improvement Efforts: 

In May 2004, the FAA Administrator testified to Congress that, to date, 
in attempting to improve the efficiency of ATC operations while 
maintaining safety, FAA had not managed its major ATC acquisitions to 
be aware of their cost implications for its operations. The 
Administrator said, however, that the agency was taking its first steps 
to fundamentally change how it makes acquisition decisions by adopting 
a more results-oriented approach. Under this approach, the agency plans 
to link its decisions to fund major acquisitions directly with their 
expected contribution to improving operational efficiency and 
controlling escalating operating costs. Whereas, in the past, FAA 
measured results in terms of its progress in completing and deploying a 
major ATC system, it was now going to focus on how a given system 
improved operational efficiency. Such an approach holds promise for 
helping FAA more effectively manage its portfolio of major ATC 
acquisitions by providing a sound basis for choosing among competing 
priorities. However, because FAA has only recently begun to incorporate 
this type of analysis of acquisitions' costs and operational efficiency 
into its decision-making and management processes, it is still too 
early to assess the results.

In addition, to its credit, FAA has created a training framework for 
its acquisition workforce, which we found mirrors human capital best 
practices that we have identified. In January 2003, we reported on 
FAA's efforts to define and train its workforce to meet the 
requirements of the Clinger-Cohen Act of 1996.[Footnote 34] This act 
required FAA and other civilian agencies to establish education, 
training, and experience requirements for their acquisition workforce. 
Our work on public and private best practices has identified six 
elements of training as critical to acquisition. These elements include 
(1) prioritizing the acquisition initiatives most important to the 
agency, (2) securing top-level commitment and resources, (3) 
identifying those who need training on specific initiatives, (4) 
tailoring training to meet the needs of the workforce, (5) tracking 
training to ensure it reaches the right people, and (6) measuring the 
effectiveness of training. These six elements are crucial for 
successfully implementing acquisition initiatives and reforms. 
Agencies that do not focus their attention on these critical elements 
risk having an acquisition workforce that is ill equipped to implement 
new processes. The probability of success is higher if training is 
well planned rather than left to chance. In 2003, we found that FAA's 
model for training its acquisition workforce largely mirrored public 
and private-sector best practices and that the agency had highly 
developed processes for four of these six elements. See figure 4.

Figure 4: Our Analysis of FAA's Progress as of 2003 in Implementing Key 
Elements of Training for Its Acquisition Workforce: 

[See PDF for image]

[End of figure]

Since 2003, FAA has taken some steps to measure the effectiveness of 
its training. For example, the agency collects and reviews 
participants' assessments of the knowledge they have gained, the extent 
that learning objectives were achieved and the applicability and 
usefulness of the training. In addition, members of FAA's Intellectual 
Capital Investment Plan Council[Footnote 35] have attempted to make 
qualitative judgments about the impact of the training on the 
effectiveness or efficiency of their organizations. However, FAA is 
still developing an evaluation program with metrics to measure the 
extent to which organizational goals are achieved when individual 
training objectives are met. Industry and government experts believe 
training and human capital investments are prerequisites for 
successfully introducing and implementing effective acquisition best 
practices. FAA's acquisition workforce plays a critical role in 
addressing long-standing weaknesses that we and others have identified 
with FAA's acquisition of major ATC systems. Given the importance of 
training for acquisition workforces, it will be important for the ATO 
to put mechanisms in place to comprehensively evaluate the 
effectiveness of the training it provides to improve the knowledge base 
of FAA's acquisition workforce.

To improve its investment management decision-making and oversight of 
major ATC acquisitions, the ATO also initiated the following 
procedures: 

* Integrate AMS and the Office of Management and Budget's Capital 
Planning and Investment Control Process[Footnote 36]to develop a 
process for analyzing, tracking, and evaluating the risks and results 
of all major capital investments made by FAA.

* Conduct Executive Council reviews of project breaches of 5 percent in 
cost, schedule, and performance to better manage cost growth;

* Issue monthly variance reports to upper management to keep them 
apprised of cost and schedule trends.

* Monitor progress in meeting the goals identified in FAA's Flight 
Plan, the agency's blueprint for action through 2008. The Executive 
Council tracks this progress monthly and reports to the Administrator, 
using a color-coded system to keep her apprised of how well FAA is 
meeting its goals. Green denotes that a goal will be met, yellow 
denotes that some of the activities leading to a main goal may be in 
jeopardy but the overall goal can be achieved, and red denotes serious 
concerns about reaching a goal without major intervention. A formal 
progress report is issued quarterly and made publicly available on the 
agency's Web site; and: 

* Increase the use of cost monitoring or earned value management 
systems[Footnote 37] to improve oversight of programs.

Despite FAA's current and planned efforts to improve its acquisition of 
major ATC systems under the ATO, given the newness of these efforts and 
the agency's poor track record in this area for more than two decades, 
it is critical for FAA to (1) modify AMS to more fully reflect the best 
practices followed by high-performing acquisition organizations, (2) 
follow through on planned improvement initiatives, and (3) adopt a 
continuous improvement approach to acquiring new ATC systems.

Conclusions: 

In the early 1990s, FAA contended that it needed relief from the FAR to 
remedy long-standing problems with cost, schedule, and performance 
shortfalls in its major ATC acquisitions; however, our work for more 
than two decades in this area has found that acquiring major ATC 
systems successfully depends more on managing an acquisition process 
well than on using a specific contracting process (e.g., the FAR or 
AMS). While our recent work has shown some improvement in FAA's 
management of major ATC system acquisitions, some key problems that 
existed before 1996 persist under AMS--including difficulty with 
clearly defining system requirements at the investment milestone and 
adequately assessing complex software requirements. These problems 
continue to make these acquisitions vulnerable to cost, schedule, and 
performance shortfalls. Without further measures to improve the 
development and management of requirements and to better estimate the 
complexity of the software development needed for major ATC systems, 
such shortfalls are likely to persist.

Although AMS provides some discipline for acquiring major ATC systems 
through its various phases, activities, and decision points, it does 
not require that (1) specific knowledge be attained using explicit 
written criteria and (2) corporate executive-level oversight be 
provided to determine--independently from the program offices--whether 
a system has reached a level of development (product maturity) 
sufficient to move forward in the acquisition process. Commercial best 
practices call for such knowledge-based decision-making at the 
corporate executive-level to help ensure that acquisitions are not 
moved into the development phase prematurely, to obtain greater 
predictability in ATC system program costs and schedules, to improve 
the quality of the ATC systems that are deployed, and to deliver new 
capability to the National Airspace System faster. A knowledge-based 
approach is also important because it provides assurance that agency 
decision-makers have critical information about an acquisition's 
ability to meet a mission need and FAA's readiness to move forward in 
the acquisition process before making large commitments of agency 
resources. Absent such an approach, FAA lacks assurance that it has 
obtained the critical technological, design, or manufacturing knowledge 
that best practices call for to avoid cost overruns, schedule slips, 
and performance shortfalls. As a result, FAA is not doing all that it 
can to systematically address persistent shortcomings in its management 
of major ATC acquisitions. Moreover, although FAA has established a 
framework for training its acquisition workforce under the ATO, it has 
not yet developed comprehensive performance criteria to evaluate how 
effectively it has implemented this framework. As a result, the agency 
lacks assurance that its use of this framework is having the intended 
effect of improving the knowledge base of this workforce.

Recommendations for Executive Action: 

We are making five recommendations to the Secretary of Transportation. 
To reduce the risk of persistent cost and schedule shortfalls in major 
ATC system acquisition programs, to improve the quality of the ATC 
systems that are deployed, and to deliver new capability to the 
National Airspace System faster, we recommend that the Secretary of 
Transportation advise the FAA Administrator to take the four following 
actions: 

* Modify AMS to specify that requirements be more clearly defined for 
major ATC systems, including providing more detailed guidance on 
setting clear, objective, and measurable requirements that reflect 
customers' needs, before making large investments of agency resources.

* Establish a strategy for identifying and measuring all additional 
development needed for complex software (e.g., commercial-off-the-
shelf or nondevelopmental items) used for major ATC systems.

* Develop explicit written criteria for the key decision points called 
for under best practices, including the capture of specific design and 
manufacturing knowledge.

* Require corporate executive-level decisions at these key decision 
points (before an acquisition moves from integration to demonstration 
and, again, before it moves to production).

In addition, to assure FAA that the training framework it has adopted 
for the ATO's acquisition workforce is improving the knowledge base of 
this workforce as intended, we recommend that the Secretary advise the 
Administrator to develop performance criteria to comprehensively 
evaluate the framework's effectiveness.

Agency Comments: 

We provided copies of a draft of this report to DOT for review and 
comment and met with Department and FAA officials, including the ATO's 
Vice President for Acquisition and Business Services, to obtain their 
comments. FAA officials told us that they have made great strides in 
improving their acquisition of major ATC systems under AMS; however, 
they recognize that there is room for improvement and are firmly 
committed to implementing best practices for acquisitions. These 
officials generally agreed with the report's findings and conclusions 
and said that our recommendations would be useful to them as they 
continue to refine their acquisition management system, including 
training their acquisition workforce. The agency provided us with oral 
comments, primarily technical clarifications, which we have 
incorporated as appropriate.

As agreed with your office, unless you publicly announce the contents 
of this report earlier, we plan no further distribution until 30 days 
from the report date. At that time, we will send copies of this report 
to interested congressional committees, the Secretary of 
Transportation, and the Administrator, FAA. We will also make copies 
available to others upon request. In addition, the report will be 
available at no charge on the GAO Web site at 
[Hyperlink, http://www.gao.gov].

Please call me at (202) 512-2834 if you or your staff have any 
questions concerning this report. Key contributors to this report are 
listed in appendix VI.

Signed by: 

JayEtta Z. Hecker: 
Director, Physical Infrastructure Team: 

[End of section]

Appendixes: 

Appendix I: FAA Has Begun Analyzing Spending Trends to Take a More 
Strategic Approach to Procurement: 

Our review of the Federal Aviation Administration's (FAA) general 
procurement of goods and services focused on the Air Traffic 
Organization (ATO) and its predecessor offices. According to FAA 
officials, the ATO has recently begun to consider ways to better 
leverage its buying power by taking a more strategic approach to 
procurement. While FAA uses the Acquisition Management System (AMS) for 
all FAA acquisitions, including the procurement of such goods and 
services as office supplies, computers, telephone services, and 
engineering and technical support services, these procurement 
activities take place in a decentralized environment of independent, 
transaction-oriented buying processes. Each FAA unit determines its 
need for goods and services and procures them as necessary, leaving 
headquarters with limited oversight of the agency's total procurement 
spending. For example, in 2003, FAA units carried out over 346,000 
procurement actions for goods and services and purchase 
cardholders[Footnote 38] made an additional 335,000 transactions. This 
fragmented environment does not permit the agency to leverage its 
buying power through lower-cost, consolidated contracts, at the local, 
regional, or national level and to rationalize the number of suppliers 
best suited to meet the agency's needs. At the same time, as part of a 
strategic procurement effort, FAA can use spend analysis to monitor 
trends in small and disadvantaged business participation so that it can 
balance the goals of lower-cost contract consolidation and promoting 
small business contracting opportunities.

Spend analysis, a tool used in a strategic approach to procurement, 
provides knowledge about how much is being spent for what goods and 
services, who the buyers are, who the suppliers are, and where the 
opportunities are to leverage buying power. Our past work[Footnote 39] 
shows that private companies are using spend analysis as a foundation 
for employing a strategic approach to procurement. The analysis 
identifies where numerous suppliers are providing similar goods and 
services--often at varying prices--and where purchasing costs can be 
reduced and performance improved by better leveraging buying power and 
reducing the number of suppliers to meet the company's needs. Our 
research on commercial best practices has found that spend analysis is 
an important driver of strategic planning and execution. As part of an 
overall strategic procurement effort, companies use spend analysis to 
(1) define the magnitude and the characteristics of their spending, (2) 
understand their internal clients and supply chain, (3) create lower-
cost consolidated contracts, and (4) monitor spending with small and 
disadvantaged businesses to achieve socioeconomic procurement goals.

We previously reported[Footnote 40]that six agencies, including DOT, 
did not take advantage of opportunities to obtain more favorable prices 
on purchase card buys with frequently used vendors--vendors where an 
agency spends more than $1 million annually. In these six agencies, 
which accounted for over 85 percent of federal government purchase card 
spending, frequently used vendors accounted for purchases totaling 
nearly $3 billion in 2002. We recommended several actions--including 
conducting spend analysis using available data and gathering additional 
information where feasible--that could ultimately help these agencies 
achieve $300 million annually in potential savings.

In fiscal year 2003, FAA procured nearly $4 billion in goods and 
services and spent an additional $132 million using purchase cards. 
According to senior FAA officials, the agency has just begun to 
implement a strategic approach to general procurements. Other federal 
agencies are beginning to use strategic tools such as spend analysis to 
improve their spending for goods and services, and some have 
initiatives under way to obtain more favorable prices on purchase card 
buys. According to a senior FAA acquisition official, FAA has to 
balance the need of its units to independently make purchases that 
pertain solely to unit requirements with the agency's need to aggregate 
purchases of goods and services that are used by more than one unit. 
FAA has hired a consultant to help begin the use of spend analysis. 
This effort could reduce the agencywide costs for mobile wireless 
services by 40 percent--an effort expected to save the agency $8 to $10 
million annually. FAA intends to expand its use of spend analysis to 
target other procurement category savings opportunities, including 
information technologies, training, facilities, and professional 
services, as its accounting systems improve.

FAA has taken some preliminary steps to set up a spend analysis 
program; however, progress has been challenging for FAA because of 
deficiencies in its accounting systems. For example, because the 
agency's accounting system did not identify all of the mobile wireless 
services for which it was being billed, the contractor implementing the 
spend analysis had to obtain this information from the wireless 
providers. FAA will need to expedite its efforts in this area to fully 
realize potential savings. Our prior research has shown that setting up 
a spend analysis program can be challenging. Companies have had 
problems accumulating sufficient data from internal financial systems 
that do not capture information on all of what a company buys or is 
using in different, unconnected parts of the company. Despite these 
challenges, companies that have developed formal, centralized spend 
analysis programs have been able to track their costs and identify 
areas for strategic sourcing and savings opportunities.

In our recent report on spend analysis,[Footnote 41] we found that DOT, 
at the time of our review, had not yet begun to collect the data needed 
for a strategic approach to procurement; however, the department is 
engaged in ongoing efforts to improve procurements, and its top 
leadership is committed to using spend analysis to change the way goods 
and services are purchased. One obstacle to using spend analysis that 
the department cited during our review was a lack of comprehensive and 
reliable spending data. However, since we completed our review, the 
department reports stepping up efforts to use currently available data 
and evaluate business intelligence software to overcome those 
obstacles. In commenting on our report, Transportation's senior 
procurement executive told us that the department is expanding its 
spend analysis efforts. For example, his office recently reviewed 
purchase card spending data to identify volume discount opportunities 
and is now using the results to negotiate new discount agreements with 
several office product vendors. In addition, he told us that to 
facilitate future agencywide purchase card spend analyses, DOT awarded 
a task order in June 2004 to one bank card company that will provide 
purchase-card audit software and enhanced data-mining capabilities. He 
also indicated that the department's leadership supports fiscal year 
2005 funding to enhance spend analysis capabilities and that software 
options for the new agencywide spend analysis system are now being 
evaluated as part of an ongoing financial and procurement review.

[End of section]

Appendix II: Objectives, Scope, and Methodology: 

To compare FAA's Acquisition Management System (AMS) with the Federal 
Acquisition Regulation (FAR), we reviewed AMS and changes in it over 
time. We also compared FAA's acquisition authority under the FAR and 
under AMS. In addition, we identified relevant recommendations from 
reports that we, the Department of Transportation's Inspector General 
(DOTIG), and others have issued to determine which recommendations have 
been implemented, rejected, or left open, and to evaluate how those 
recommendations have modified FAA's acquisition policies and practices. 
We also collected and summarized published reports and analyzed 
available life-cycle management data on the current status of major and 
nonmajor acquisitions being carried out under AMS.

To determine the ways in which FAA's acquisition policies compare with 
our best practices model, we used information from several of our 
products that examine how commercial best practices can improve 
outcomes for acquisition programs. This model consists of four phases: 
(1) concept and technology development; (2) product development, which 
includes both integration and demonstration activities; (3) production; 
and (4) operations and support. In between these four phases are three 
key knowledge points at which commercial firms must have sufficient 
knowledge to make large investment decisions. We also reviewed and 
analyzed AMS, accessible at h [Hyperlink, http://fast.faa.gov] ttp: //
fast.faa.gov. Furthermore, to clarify the content of FAA's acquisition 
process, we met with various FAA vice-presidents and officials from 
FAA's Acquisition Planning and Policy Division. Next, we compared and 
contrasted FAA's acquisition policies with the best practices for 
commercial acquisitions identified in our past reports. Our analysis 
focused on whether FAA's policies contained the measurable criteria and 
management controls necessary to achieve FAA's intent of minimizing 
cost, schedule, and performance risks. We also interviewed current and 
former FAA procurement officials that have experience using both the 
FAR and AMS.

To determine if FAA has effectively implemented its new acquisition 
authority and improved its acquisition outcomes, we reviewed seven of 
FAA's most expensive major ATC acquisitions, including the Airport 
Surveillance Radar 11 (ASR-11), Standard Terminal Automation 
Replacement System (STARS), Integrated Terminal Weather System (ITWS), 
Local Area Augmentation System (LAAS), Next Generation Air/Ground 
Communications System (NEXCOM), Advanced Technologies and Oceanic 
Procedures (ATOP), and En Route Automation Modernization (ERAM). See 
table 7 for specific program costs.

Table 7: Program Costs for the Seven Systems We Reviewed: 

Dollars in millions.

Program: STARS; 
Total program cost as of 9/30/04: $1,460.0.

Program: ASR-11; 
Total program cost as of 9/30/04: $891.7.

Program: ITWS; 
Total program cost as of 9/30/04: $288.3.

Program: LAAS; 
Total program cost as of 9/30/04: $696.1.

Program: NEXCOM; 
Total program cost as of 9/30/04: $318.4.

Program: ATOP; 
Total program cost as of 9/30/04: $548.2.

Program: ERAM; 
Total program cost as of 9/30/04: $2,154.6.

Total; 
Total program cost as of 9/30/04: $6,357.3.

Source: GAO analysis.

Note: These amounts are for facilities and equipment only (not 
operations and maintenance).

[End of table]

We also selected these seven acquisitions because we considered them to 
fall into two basic categories-pre-AMS and post-AMS. Five of the 
acquisitions were initiated before AMS was implemented in April 1996 
and were transitioned into AMS at various times before their 
completion. The two remaining acquisitions--ATOP and ERAM--were 
initiated and have remained completely under AMS. We then reviewed 
program documents and reports and interviewed program and agency 
officials responsible for developing these acquisitions, as well as 
other acquisitions experts in the private sector. For some 
acquisitions, we discussed programmatic issues with representatives of 
the primary contractor for the specific acquisition to obtain 
information on the practices and procedures used for the acquisition. 
In addition, we interviewed some current and former FAA procurement 
officials with experience using both the FAR and AMS to obtain their 
views on the use of each contracting process and how the two compare. 
Furthermore, to see how FAA has progressed in addressing problems with 
its acquisitions, we reviewed our work on acquisitions over the last 20 
years, as well as reports by the DOTIG, FAA, Booz-Allen & Hamilton, and 
MITRE. Because the data in this report on cost, schedule and 
performance are used as background information or to otherwise provide 
a description of acquisitions, we did not assess their reliability.

The effect of the current budget process on FAA's ability to 
successfully modernize the National Airspace System, including 
acquiring major ATC systems is not within the scope of this review.

[End of section]

Appendix III: Comparison of the Scope and Flexibility of FAA's 
Acquisition Management System and the Federal Acquisition Regulation 
Process: 

Background: 

FAA's business processes, including its acquisition of major systems, 
differ significantly from the business processes followed by most other 
federal agencies. FAA relies on its Acquisition Management System 
(AMS), which establishes FAA internal acquisition policy. AMS resulted 
from the adoption of language in the Department of Transportation and 
Related Agencies Appropriations Act,[Footnote 42] which directed the 
FAA Administrator to develop and implement an acquisition management 
system for FAA. The adoption of this language (section 348) followed 
FAA's assertions that the requirement that it conduct procurements in 
accordance with the Federal Acquisition Regulation (FAR) was at least a 
contributing factor in its repeated failure to complete air traffic 
control (ATC) and other modernization programs on schedule. The 
Administrator was directed to put in place a system that would address 
the "unique needs of the agency" that FAA contended prevented its 
acquisitions from being timely and cost-effective.

Section 348 distinguished FAA from other federal agencies by removing 
FAA from the federal acquisition system. Under section 348, FAA was no 
longer subject to title III of the Federal Property and Administrative 
Services Act of 1949,[Footnote 43] which among other things requires 
that the government procure supplies and services competitively. It 
removed FAA as an agency subject to the Office of Federal Procurement 
Policy Act[Footnote 44] and eliminated the requirement that FAA comply 
with the FAR. While mandating that FAA conduct its acquisitions so that 
"all reasonable opportunities to be awarded contracts shall be provided 
to small business concerns and small business concerns owned and 
controlled by socially and economically disadvantaged individuals," 
section 348 eliminated the requirement that FAA comply with the Small 
Business Act.[Footnote 45] Furthermore, it made the procurement protest 
system of the U.S. Government Accountability Office inapplicable to 
FAA, although disappointed offerors can still file protests with FAA's 
Office of Dispute Resolution for Acquisition.[Footnote 46]

AMS Defines an Investment/Life-Cycle Project Management System: 

Much of AMS guidance concerns project, financial, and property life-
cycle management issues. In fact, FAA's policy describes AMS as 
applying to all investment programs regardless of cost or the 
appropriation funding them. It recognizes that a single investment 
program may span multiple procurements and projects. It applies, 
according to its terms, to the activities associated with needs 
analysis, determination of requirements, analysis of investment 
alternatives, establishment of investment programs, allocation and 
expenditure of resources, procurement and deployment of needed products 
and services, in-service management of fielded capability, and eventual 
disposal of obsolete products.

AMS focuses on the following key program milestones: 

* Mission Analysis--encompasses those key corporate and service-level 
processes that define, coordinate, and integrate the work of service 
organizations,[Footnote 47] thereby providing strategic direction to 
keep FAA responsive to the service needs of its customers. Mission 
analysis is used to update a mission need statement, which in turn may 
identify capability shortfalls or technological opportunities, that is, 
unmet needs. Unmet needs are presented to the Joint Resources Council 
(JRC) for a mission need decision. To be approved, the unmet need 
should be supported by the updated mission need statement and the 
initial requirements document, including a concept of use, and the 
initial investment plan.

* Investment Analysis--builds on the results of the mission need 
decision by developing detailed plans and final requirements for each 
proposed investment program and by defining an acquisition program 
baseline that establishes cost, schedule, performance, benefit, and 
risk-management boundaries for the program. AMS calls for planning the 
entire solution--an effort that may use market survey data but is based 
in large measure on FAA's assumptions and data. The service 
organization produces a final implementation and life-cycle support 
strategy. A detailed program plan and an acquisition program baseline 
are also produced. The results are presented to the JRC for a "final 
investment decision."[Footnote 48]

* Solution Implementation--encompasses acquiring, accepting, 
deploying, installing and preparing for the operational use of an 
approved investment. Approval of the investment carries with it 
authorization for the service organization to conduct all acquisitions 
needed to execute the investment decision, subject to any constraints 
established in the final investment decision.

* In-Service Decision--is an FAA system qualification milestone, which 
is achieved when an otherwise operational investment is satisfactorily 
tested to demonstrate its operational effectiveness and suitability 
before it is placed in service in the National Airspace System. The JRC 
designates the decision maker.

* In-Service Management--covers activities throughout a system's life 
cycle, starting at the time that an investment becomes operational. In-
service product improvements may eliminate latent defects, fix systemic 
problems, and enhance the utility of the investment. These changes may 
be made within the approved acquisition program baseline without 
corporate-level approval. In-service management also includes 
planning, programming, and developing supporting budget input; 
monitoring and assessing performance, cost of ownership, and support 
trends; and planning for service-life investment decisions.

* Service Life Extension--seeks a new investment decision by the JRC 
when a current capability is unable to satisfy demand or when another 
solution may be more effective. The JRC can decide to revalidate the 
mission need satisfied by the solution by upgrading or refurbishing 
fielded capability or by replacing that capability with another 
equivalent or new superior solution. The JRC may also decide that the 
capability should be retired.

Only a Portion of AMS Deals Directly with the Procurement Process: 

Although the FAR includes requirements addressing procurement planning 
and major system acquisition, AMS as just outlined differs 
significantly from the FAR in its focus and scope. The FAR addresses 
planning[Footnote 49] and major system acquisition[Footnote 50] in the 
context of government procurement policy and procedure. Agencies other 
than FAA find the broader program planning and management issues 
addressed in AMS outside of the FAR, in documents such as the Office of 
Management and Budget's (OMB) Circular A-109, in their own planning 
guidance, such as the Department of Defense's (DOD) 5000 
series,[Footnote 51]and in established knowledge-based best practices. 
As indicated earlier, much of AMS focuses on just such issues. Only AMS 
section 3 addresses procurement policy and procedure.[Footnote 52]

AMS States a Nonregulatory FAA Policy: 

A further significant foundational difference between AMS and the FAR 
is that AMS sets out a nonregulatory FAA policy, whereas the FAR was 
adopted and is maintained as a set of published governmentwide 
regulatory requirements, which form a legal basis for federal agencies' 
contract decision-making. AMS is binding on FAA personnel as FAA 
employees and establishes other guidelines that FAA states should be 
followed unless there is a rational basis for doing otherwise. AMS is 
subject to such internal controls as the Administrator chooses to 
enforce and general overarching legal requirements, such as the 
Government Performance and Results Act of 1993 (GPRA).[Footnote 53] 
There is a legal requirement, created by section 348, that small and 
socially or economically disadvantaged firms be given all reasonable 
opportunities to receive contract awards. FAA in its Office of Dispute 
Resolution for Acquisition has adopted a dispute resolution process 
with some legal underpinnings.[Footnote 54] Otherwise, as the preface 
to AMS states, "nothing in this document creates or conveys any 
substantive [legal] rights." In short, FAA has assumed no legal 
obligation to follow AMS other than to ensure that its actions are not 
arbitrary and capricious or contrary to law. By contrast, the FAR has 
the force and effect of law, and agencies that are subject to the FAR 
are bound to follow it.

AMS Chapter 3 Parallels a Subset of the FAR: 

When FAA personnel apply the procurement methodology in AMS chapter 3, 
they are applying guidance that closely parallels some of the 
procedures set out in the FAR. The AMS Chapter 3 acquisition process 
parallels a subset of the varied selection of procurement methods 
available under the FAR, requiring that all competitive FAA contracts 
be negotiated with the awardee being selected on a "best value" basis. 
The FAR also provides a much more detailed set of information and 
guidance than does AMS. A comparison of high-level differences and 
similarities between AMS and the FAR is presented in table 8.

Table 8: Comparison of AMS and the FAR: 

Best value source selection; 
AMS: Yes, following screening; 
FAR: Yes, although other methods are also available for use when 
appropriat