This is the accessible text file for GAO report number GAO-04-722 
entitled 'Information Technology: DOD's Acquisition Policies and 
Guidance Need to Incorporate Additional Best Practices and Controls' 
which was released on July 30, 2004.

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov.

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately.

Report to Congressional Requesters: 

July 2004: 

INFORMATION TECHNOLOGY: 

DOD's Acquisition Policies and Guidance Need to Incorporate Additional 
Best Practices and Controls: 

GAO-04-722: 

GAO Highlights: 

Highlights of GAO-04-722, a report to congressional requesters: 

Why GAO Did This Study: 

The way in which the Department of Defense (DOD) has historically 
acquired its business systems has been cited as a root cause for its 
limited success in delivering promised system capabilities and benefits 
on time and within budget. In response, DOD recently revised its 
systems acquisition policies and guidance to incorporate best 
practices, including those pertaining to business systems. 

GAO was asked to determine whether DODís revised systems acquisition 
policies and guidance (1) are consistent with industry best practices, 
including those pertaining to commercial component-based systems, and 
(2) provide the necessary controls to ensure that DOD component 
organizations adhere to the practices.

What GAO Found: 

DODís revised policies and guidance largely incorporate 10 best 
practices for acquiring any type of information technology (IT) 
business system. For example, the revisions include the requirement 
that acquisitions be economically justified on the basis of costs, 
benefits, and risks. However, the revisions generally do not 
incorporate 8 best practices relating to the acquisition of commercial 
component-based systems. For example, they do not address basing any 
decision to modify commercial components on a thorough analysis of the 
impact of doing so or on preparing system users for the business 
process and job roles and responsibilities changes that are embedded in 
the functionality of commercial IT products. In total, GAO found that 
DODís acquisition policies and guidance fully incorporate 8 of the 18 
best practices that they were evaluated against, partially incorporate 
5 practices, and do not incorporate the remaining 5 practices 
(see figure). DOD intends to expand its acquisition guidance to 
incorporate additional best practices by September 30, 2004, but 
department officials cite other priorities as a reason why they have 
not been able to complete this effort and could not provide a plan 
specifying how this will be accomplished. Until DODís revised policies 
and guidance incorporate key systems acquisition best practices, the 
risk that system investments will not consistently deliver promised 
capabilities and benefits on time and within budget is increased. 

DODís revised policies also do not contain sufficient controls to 
ensure that DOD components appropriately follow the best practices that 
are incorporated in its policies and guidance. According to acquisition 
best practices experts, as well as GAOís internal control guidance, 
controls are effective if they are backed by measurements that are 
verified. Although the revised policies and guidance require 
acquisition managers to examine and, as appropriate, adopt best 
practices, they do not cite what that examination entails. DOD believes 
existing controls are sufficient, even though these controls do not 
provide for measuring and validating the practicesí use. Without 
specific requirements to measure and validate the use of best 
practices, the risk that they will not be followed increases, which, in 
turn, increases the risk that system investments will not meet 
expectations. 

DOD Incorporation of Best Practices: 

[See PDF for image]

[End of figure]

What GAO Recommends: 

To improve DODís ability to acquire IT business systems, GAO recommends 
that the Secretary of Defense incorporate additional best practices in 
DODís acquisition policies and guidance, and that the department 
strengthen controls for ensuring that best practices are appropriately 
followed.

In commenting on a draft of this report, DOD agreed or partially agreed 
with our recommendations for incorporating additional best practices, 
but did not agree that it needed (1) a plan to govern its incorporation 
of these practices or (2) stronger controls for ensuring that best 
practices are followed.

www.gao.gov/cgi-bin/getrpt?GAO-04-722.

To view the full product, including the scope and methodology, click on 
the link above. For more information, contact Randolph C. Hite at 
(202) 512-3439 or hiter@gao.gov.

Contents: 

Letter: 

Results in Brief: 

Background: 

DOD's Acquisition Policy and Guidance Are Consistent with Some, but Not 
All, Key Acquisition Best Practices: 

DOD's Acquisition Policies Do Not Contain Sufficient Controls to Ensure 
That the Requirement Is Met for Appropriately Applying Best Practices: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendixes: 

Appendix I: Objectives, Scope, and Methodology: 

Appendix II: Best Practices: 

Best Practices Relevant to Any IT Business Systems Acquisition: 

Complementary Best Practices Relevant to Commercial Component-Based IT 
Business Systems Acquisitions: 

Appendix III: Comments from the Department of Defense: 

GAO Comments: 

Appendix IV: GAO Contact and Staff Acknowledgments: 

GAO contact: 

Staff acknowledgments: 

Tables: 

Table 1: Organizational Responsibilities for the DOD 5000 Series 
Documents: 

Table 2: Summary of Business Systems Acquisition Best Practices: 

Table 3: Activity-by-activity Comparison of the 5000 Series to Best 
Practices Relevant to Any IT Business Systems Acquisition: 

Table 4: Activity-by-Activity Comparison of the 5000 Series to Best 
Practices Relevant to Commercial Component-based Business Systems 
Acquisitions: 

Figures: 

Figure 1: Revisions to the DOD 5000 Series Documents: 

Figure 2: Simplified Diagram of DOD's Acquisition Management Framework	: 

Figure 3: DOD Acquisition Policies and Guidance Incorporation of Best 
Practices: 

Abbreviations: 

AT&L: Acquisition, Technology, and Logistics: 

DOD: Department of Defense: 

IT: information technology: 

NII: Networks and Information Integration: 

OT&E: Operational Test and Evaluation: 

Letter: 

July 30, 2004: 

The Honorable John Ensign: 
Chairman: 
The Honorable Daniel K. Akaka: 
Ranking Minority Member: 
Subcommittee on Readiness and Management Support: 
Committee on Armed Services: 
United States Senate: 

This report responds to your request that we assess the Department of 
Defense's (DOD) recently revised policies and guidance for acquiring 
business systems for incorporation of acquisition best practices. As 
you know, the way in which DOD has historically acquired information 
technology (IT) systems has been cited as a root cause of these systems 
failing to deliver promised capabilities and benefits on time and 
within budget. The use of these best practices--which includes those 
practices pertaining to any business system, whether custom developed 
or based on commercially available product components, as well as those 
unique to commercial component-based systems--is intended to improve on 
this performance.

As agreed with your offices, our objectives were to determine whether 
DOD's recently revised policies and guidance for acquiring business 
systems (1) are consistent with industry best practices, including 
those pertaining to commercial component-based systems, and (2) provide 
the necessary controls to ensure that DOD component organizations 
adhere to best practices. We conducted our work between December 2003 
and May 2004 in accordance with generally accepted government auditing 
standards. Details of our objectives, scope, and methodology are 
included in appendix I.

Results in Brief: 

DOD's revisions to its systems acquisition policies and guidance 
incorporate many best practices for acquiring business systems. For 
example, the revisions recognize such practices as (1) economically 
justifying system investments on the basis of costs, benefits, and 
risks and (2) continually measuring an acquisition's performance, cost, 
and schedule against approved baselines. However, the revised policies 
and guidance do not incorporate a number of other best practices, 
particularly those associated with acquiring commercial component-
based business systems. For example, they do not address basing a 
decision to modify a commercial component on a thorough analysis of the 
impact of doing so; evaluating contractors on their ability to 
implement commercial components and using the results in source 
selection decisions; and preparing system users for the business 
process and job roles and responsibilities changes that are embedded in 
the functionality of commercial software products. According to 
officials responsible for revising DOD acquisition policies and 
guidance, additional best practices and lessons learned will be 
incorporated into the acquisition guidance by September 30, 2004. 
However, documented plans for this task do not exist, and the 
associated resources needed to complete this task have not been 
assigned due to higher priority needs. Until these missing best 
practices are included in DOD's acquisition policies and guidance, the 
risk is increased that systems acquisitions will not deliver planned 
capabilities and benefits on time and within budget.

DOD's revised acquisition policies also do not contain sufficient 
controls to ensure that military services and defense agencies 
appropriately follow the practices. Controls are considered effective 
if they are measured and verified. Although DOD's revised policies 
require both the project manager and the investment decision authority 
to examine the adoption of best practices, neither the policies nor the 
associated guidance provide for measuring and verifying the use of the 
practices. As a result, DOD is increasing the risk that best practice 
adoption and use will not occur, which, in turn, increases the risk 
that systems acquisitions will not deliver what is planned--on time and 
within budget.

This report makes 14 recommendations to the Secretary of Defense that 
are aimed at strengthening DOD's acquisition policy and guidance by 
including additional business systems acquisition best practices and 
controls for ensuring that best practices are followed.

In its written comments (reprinted in app. III) on a draft of this 
report signed by the Principal Director, Deputy Assistant Secretary of 
Defense (Command, Control, Communications, Space and Information 
Technology Programs), DOD agreed with the importance and relevance of 
the best practices that we cite in the report. DOD also agreed or 
partially agreed with most of our recommendations, stating that it 
would either incorporate those practices that we reported as missing 
from the department's acquisition policies and guidance, or consider 
augmenting its coverage of those practices that deserve greater 
emphasis in its policies and guidance. Further, while DOD acknowledged 
that incorporation of additional best practices in its acquisition 
policies and guidance should be undertaken or considered, it did not 
agree that it needed an explicit plan to govern its ongoing and future 
policy and guidance revision activities, stating that our 
recommendation to this effect was inappropriate. We do not agree with 
DOD that a plan governing incorporation of the practices is not needed. 
Given the importance of DOD's acquisition policies and guidance as well 
as best practices, we believe that having an explicit plan that defines 
how and when incorporation of best practices will be added is 
essential. Among other things, a plan would highlight the resource 
constraints that this revision effort has been subject to, would allow 
measurement against defined milestones, and would allow disclosure of 
progress and impediments. In its comments, DOD also did not agree that 
stronger controls are needed for ensuring adherence to the best 
practices contained in its acquisition policies and guidance. We do not 
agree with the department's position on this matter because its 
existing controls do not provide for either the measurement or 
verification of whether the practices are employed--both recognized 
elements of effective process controls.

Background: 

Best practices are tried and proven methods, processes, techniques, and 
activities that organizations define and use to minimize risks and 
maximize chances for success. As we have previously reported, using 
best practices can result in better outcomes--including cost savings; 
improved service and product quality; and, ultimately, a better return 
on investment. For example, two software engineering analyses of nearly 
200 systems acquisitions projects indicated that teams using systems 
acquisition best practices produced cost savings of at least 11 percent 
over similar projects conducted by teams that did not employ the kind 
of rigor and discipline embedded in these practices.[Footnote 1] In 
addition, our research[Footnote 2] shows that best practices are a 
significant factor in successful acquisition outcomes, including 
increasing the likelihood that programs and projects will be executed 
within cost and schedule estimates.

DOD, GAO, and the Congress have all advocated the use of best 
practices. For example, in September 2000, DOD established a steering 
group to promote the use of systems acquisition best practices and 
lessons learned. Further, our 2001 report[Footnote 3] cited the 
benefits of DOD adoption of best practices and provided recommendations 
for establishing a mechanism for sharing best practices throughout DOD. 
In the fiscal year 2003 Defense Authorization Act,[Footnote 4] the 
Congress used our recommendations in directing DOD to expand its use of 
best practices. Specifically, it required the Under Secretary of 
Defense (Acquisition, Technology, and Logistics (AT&L)) and the 
Assistant Secretary of Defense (Command, Control, Communications, and 
Intelligence--now called Networks and Information Integration (NII))--
to identify and serve as a clearinghouse for information regarding 
software acquisition best practices in the public and private sectors. 
In response, DOD assigned AT&L responsibility for serving as that 
clearinghouse. Further, the Defense Information Systems Agency created 
a Web site to provide information about acquisition best practices.

DOD Relies Extensively on IT Systems to Perform a Variety of Business 
Functions: 

DOD is one of the largest and most complex organizations in the world. 
In fiscal year 2003, DOD reported that its operations involved over $1 
trillion in assets, nearly $1.6 trillion in liabilities, disbursements 
of more than $416 billion, and approximately 3.3 million military and 
civilian personnel. Execution of these operations spans a wide range of 
defense organizations, including the military services, defense 
agencies and field activities, and various combatant and joint 
operation commands. To execute these military operations, the 
department performs an assortment of business functions, including 
logistics management, procurement, healthcare management, and 
financial management.

To support its business functions, DOD reports that it currently relies 
on about 2,300 IT systems, including accounting, acquisition, 
logistics, and personnel systems. Moreover, its future investment in 
business systems is expected to be sizable. For fiscal year 2004, DOD 
requested approximately $28 billion in IT funding to support a wide 
range of military and business operations. Approximately $9 billion of 
this amount is to support primarily command and control systems, and 
the remaining $19 billion is to support the operation, maintenance, and 
modernization of business systems.

Overview of DOD's Acquisition Management Framework: 

Since the 1980s, DOD's oversight of its systems acquisitions had been 
defined by a series of three documents--commonly called the 5000 
series--that provided the policies and guidance for departmental 
efforts to acquire service capabilities and systems: 

* DOD Directive 5000.1, The Defense Acquisition System--describes the 
management principles for DOD's acquisition programs.

* DOD Instruction 5000.2, Operation of the Defense Acquisition System-
-outlines the framework for managing acquisition programs.

* DOD 5000.2-R, Mandatory Procedures for Major Defense Acquisition 
Programs (MDAPS) and Major Automated Information System (MAIS) 
Acquisition Programs--provides the mandatory procedures for acquiring 
major defense programs.

These documents have been revised several times. Most recently, in 
October 2002, the Deputy Secretary of Defense determined that the 
existing versions of these three documents required further revisions 
to improve acquisition efficiency, flexibility, creativity, and 
innovation. As a result, the Deputy Secretary canceled the existing 
versions of each document and instructed the Under Secretary for AT&L; 
the Assistant Secretary for NII; and the Director, Operational Test and 
Evaluation (OT&E) to jointly revise the documents. (Table 1 describes 
selected responsibilities of these three entities.) The revised 
directive and instruction were issued in May 2003. Both were shortened 
and modified to focus on required outcomes and legal requirements and 
to eliminate the "how-to" details in the previous versions. In doing 
so, DOD intended the revisions to provide program managers with more 
flexibility in executing their respective programs.

Table 1: Organizational Responsibilities for the DOD 5000 Series 
Documents: 

Organization: Office of the Under Secretary of Defense (AT&L)/Defense 
Acquisition Executive; 
Responsibility: Advises the Secretary of Defense on all matters 
pertaining to DOD's acquisition framework as well as research and 
development; advanced technology; developmental test and evaluation; 
production; logistics; installation management; military construction; 
procurement; environmental security; and nuclear, chemical, and 
biological matters.

Organization: Office of the Assistant Secretary of Defense (NII)/DOD 
Chief Information Officer; 
Responsibility: Advises the Secretary of Defense on achieving and 
maintaining information superiority through the collection, processing, 
and dissemination of an uninterrupted flow of information in support of 
DOD missions while exploiting or denying an adversary's ability to do 
the same; ; Serves as the principal assistant to the secretary for 
electronic business, information management, information operations 
and assurance, and IT.

Organization: Office of the Director (OT&E); 
Responsibility: Advises the Secretary of Defense on OT&E; Issues DOD 
OT&E policy and procedures, and reviews and analyzes OT&E conducted 
for each major acquisition program and provides reports on adequacy 
and results to the Secretary, the Under Secretary for AT&L, and the 
Congress.

Source: GAO, based on DOD documentation.

[End of table]

DOD 5000.2-R was renamed the Interim Defense Acquisition Guidebook and 
was made optional guidance on best practices and lessons learned. (See 
fig. 1.) According to DOD officials, improvements to the guidebook are 
currently under way. Until they are completed, DOD 5000.2-R serves as 
the guidebook.

Figure 1: Revisions to the DOD 5000 Series Documents: 

[See PDF for image] 

[End of figure] 

According to the revised policy, an acquisition is a directed, funded 
effort that provides a new, improved, or continuing materiel, weapon or 
information system, or service capability. The revised 5000 series 
applies to acquisitions conducted by all of the department's 
organizational components.[Footnote 5] These components include the 
military services and the defense agencies, such as the Defense 
Information Systems Agency.

The 5000 series describes a management framework that is intended to 
translate mission needs and requirements into systems acquisition 
programs. To accomplish this, the framework specifies five phases: 

* Concept refinement: Intended to refine the initial system concept and 
produce a strategy for acquiring a system capability. A decision is 
made at the end of this phase (milestone A decision) whether to move to 
technology development.

* Technology development: Intended to determine the appropriate set of 
technologies to be integrated into the system by iteratively assessing 
the viability of various technologies while simultaneously refining 
user requirements. Once the technology has been demonstrated in a 
relevant environment, a decision is made at the end of this phase 
(milestone B decision) whether to move to system development and 
demonstration.

* System development and demonstration: Intended to develop a system or 
a system increment and demonstrate through developer testing that the 
system/system increment can function in its target environment. A 
decision is made at the end of this phase (milestone C decision) 
whether to move to production and deployment.

* Production and deployment: Intended to achieve an operational 
capability that satisfies the mission needs, as verified through 
independent operational test and evaluation, and ensures that the 
system is implemented at all applicable locations.

* Operations and support: Intended to provide a support program to meet 
operational support requirements and sustain the system in the most 
cost-effective manner over its total life cycle.

According to the framework, an acquisition program may begin at 
milestone A, B, or C, and its progress depends on obtaining sufficient 
knowledge to make an informed decision about whether to continue to the 
next acquisition phase. Although the framework permits programs to be 
managed as a single project, DOD Instruction 5000.2 states that the 
department prefers an evolutionary acquisition strategy that delivers a 
mature product in increments. Under such a strategy, the instruction 
states that each increment is to begin with a milestone B decision, and 
the production and deployment phase of each increment is to begin with 
a milestone C decision. Figure 2 provides a simplified diagram of the 
department's acquisition management framework.

Figure 2: Simplified Diagram of DOD's Acquisition Management Framework: 

[See PDF for image] 

[End of figure] 

Typically, the Under Secretary for AT&L or a designee serves as the 
investment decision authority[Footnote 6] for DOD acquisitions, but the 
Assistant Secretary for NII/Chief Information Officer or a designee 
serves as the decision authority for IT systems acquisitions.

Past Evaluations of DOD Business Systems Have Revealed Acquisition 
Management Weaknesses: 

Due in part to its long-standing and pervasive IT acquisition 
management weaknesses, DOD has had limited success in acquiring IT 
resources to replace its outdated business systems. Both inspector 
general and departmental studies have cited these weaknesses on a 
variety of acquisition projects. We have also reported on business 
systems acquisition weaknesses.[Footnote 7] For example, in 2002 we 
reported that the Defense Logistics Agency did not have effective 
corporate processes for consistently acquiring software (the most 
costly and complex component of systems), and that the agency did not 
have a software process improvement program in place to effectively 
strengthen its software acquisition processes. In 2002, we also 
reported acquisition management problems with the Military Health 
System's acquisition of DOD's primary medical information system, 
including weaknesses in incremental economic justification, risk 
management, and contract management. In 2003, we reported that DOD had 
not economically justified four finance and accounting systems that 
have an estimated cost of more than $1 billion. In each of our reports, 
we have made recommendations for strengthening acquisition management 
through the adoption of best practices. DOD has largely agreed with our 
recommendations, but its progress to date in implementing them across 
the department has been uneven.

DOD's Acquisition Policy and Guidance Are Consistent with Some, but Not 
All, Key Acquisition Best Practices: 

We and others, such as Carnegie Mellon University's Software 
Engineering Institute, have identified and promoted the use of a number 
of best practices associated with acquiring IT systems. For the 
purposes of this report, we have identified 18 relevant best practices 
and grouped them into two categories: (1) 10 best practices for 
acquiring any type of business system and (2) 8 complementary best 
practices that relate specifically to acquiring commercial component-
based business systems. Examples of best practices relevant to any 
business systems acquisition include ensuring that (1) reasonable 
planning for all parts of the acquisition occur, (2) a clear 
understanding of system requirements exists, and (3) risks are 
proactively identified and systematically mitigated. Examples of best 
practices relevant to commercial component-based systems acquisitions 
include ensuring that (1) commercial product modification is 
effectively controlled, (2) relationships among commercial products are 
understood before acquisition decisions are made, and (3) the 
organizational impact of using new system functionality is proactively 
managed. Each of these practices is composed of from one to eight 
activities and is described in table 2. DOD officials responsible for 
revising the 5000 series told us that each of these 18 practices are 
relevant to DOD business systems acquisitions. Appendix II provides 
additional details on each of these practices.

Table 2: Summary of Business Systems Acquisition Best Practices: 

Best practices relevant to any business systems acquisition: 

Best practices: Acquisition planning: To ensure that reasonable 
planning for all parts of the acquisition is conducted; 
Activity: 
* Plans are prepared during acquisition planning and maintained 
throughout the acquisition; 
* Planning addresses the entire acquisition process, as well as life 
cycle support of the products being acquired; 
* The acquisition organization has a written policy for planning the 
acquisition; 
* Responsibility for acquisition planning activities is designated.

Best practices: Architectural alignment: To ensure that the acquisition 
is consistent with the organization's enterprise architecture; 
Activity: 
* The system being acquired is assessed for alignment with the 
enterprise architecture at key life cycle decision points, and any 
deviations from the architecture are explicitly understood and 
justified by an explicit waiver to the architecture; 
* Product line requirements---rather than just the requirements for 
the system being acquired---are an explicit consideration in each 
acquisition.

Best practices: Contract tracking and oversight: To ensure that 
contract activities are performed in accordance with contractual 
requirements; 
Activity: 
* The acquiring organization has sufficient insight into the 
contractor's activities to manage and control the contractor and 
ensure that contract requirements are met; 
* The acquiring organization and contractor maintain ongoing 
communication; 
commitments are agreed to and implemented by both parties; 
* All contract changes are managed throughout the life of the 
contract; 
* The acquisition organization has a written policy for contract 
tracking and oversight; 
* Responsibility for contract tracking and oversight activities is 
designated; 
* The acquiring organization involves contracting specialists in the 
execution of the contract; 
* A quantitative set of software and system metrics are used to define 
and measure product quality and contractor performance; 
* In addition to incentives for meeting cost and schedule estimates, 
measurable, metrics-based product quality incentives are explicitly 
cited in the contract.

Best practices: Economic justification: To ensure that system 
investments have an adequate economic justification; 
Activity: 
* System investment decisions are made on the basis of reliable 
analyses of estimated costs, expected benefits, and anticipated risks; 
* Large systems acquisitions are (to the maximum extent practical) 
divided into a series of smaller, incremental acquisition efforts, and 
investment decisions on these smaller efforts are made on the basis of 
reliable analyses of estimated costs, expected benefits, and 
anticipated risks.

Best practices: Evaluation: To ensure that evidence showing that the 
contract products satisfy the defined requirements are provided prior 
to accepting contractor products; 
Activity: 
* Evaluation requirements are developed in conjunction with the 
contractual requirements and are maintained over the life of the 
acquisition; 
* Evaluations are planned and conducted throughout the total 
acquisition period to provide an integrated approach that satisfies 
evaluation requirements and takes advantage of all evaluation results; 
* Evaluations provide an objective basis to support the product 
acceptance decision; 
* The acquiring organization has a written policy for managing the 
evaluation of the acquired products; 
* Responsibility for evaluation activities is designated.

Best practices: Project management: To ensure that the project office 
and its supporting organizations function efficiently and effectively; 
Activity: 
* Project management activities are planned, organized, controlled, 
and communicated; 
* The performance, cost, and schedule of the acquisition are 
continually measured, compared with planned objectives, and 
controlled; 
* Problems discovered during the acquisition are managed and 
controlled; 
* The acquisition organization has a written policy for project 
management; 
* Responsibility for project management is designated.

Best practices: Requirements development and management: To ensure 
that contractual requirements are clearly defined and understood by 
the acquisition stakeholders; 
Activity: 
* Contractual requirements are developed, managed, and maintained; 
* The end user and other affected groups have input to the contractual 
requirements over the life of the acquisition; 
* Contractual requirements are traceable and verifiable; 
* The contractual requirements baseline is established prior to 
release of the solicitation package; 
* The acquisition organization has a written policy for establishing 
and managing the contractual requirements; 
* Responsibility for requirements development and management is 
designated; 
* Requirements that are mandatory versus optional are clearly 
delineated and used in deciding what requirements can be eliminated or 
postponed to meet other project goals, such as cost and schedule 
constraints.

Best practices: Risk management: To ensure that risks are proactively 
identified and systematically mitigated; 
Activity: 
* Projectwide participation in the identification and mitigation of 
risks is encouraged; 
* The defined acquisition process provides for the identification, 
analysis, and mitigation of risks; 
* Milestone reviews include the status of identified risks; 
* The acquisition organization has a written policy for managing 
acquisition risk; 
* Responsibility for acquisition risk management activities is 
designated.

Best practices: Solicitation: To ensure that a quality solicitation is 
produced and a best-qualified contractor is selected; 
Activity: 
* The solicitation package includes the contractual requirements and 
the proposal evaluation criteria; 
* The technical and management elements of proposals are evaluated to 
ensure that the requirements of the contract will be satisfied; 
* The selection official selects a supplier who is qualified to 
satisfy the contract's requirements; 
* The acquiring organization has a written policy for conducting the 
solicitation; 
* Responsibility for the solicitation is designated; 
* A selection official has been designated to be responsible for the 
selection process and decision; 
* The acquiring team includes contracting specialists to support 
contract administration.

Best practices: Transition to support: To ensure proper transfer of 
the system from the acquiring organization to the support organization; 
Activity: 
* The acquiring organization ensures that the support organization has 
the capacity and capability to provide the required support; 
* There is no loss in continuity of support to the products during 
transition from the supplier to the support organization; 
* Configuration management of the products is maintained throughout 
the transition; 
* The acquiring organization has a written policy for transitioning 
the products to the support organization; 
* The acquiring organization ensures that the support organization is 
involved in planning for transition to support; 
* Responsibility for transition to support activities is designated.

Complementary best practices relevant to commercial component-based 
business systems acquisitions: 

Best practices: Component modification: To ensure that commercial 
product modification is effectively controlled; 
Activity: 
* Modification of commercial components is discouraged and allowed 
only if justified by a thorough analysis of life-cycle costs and 
benefits.

Best practices: Configuration management: To ensure the integrity and 
consistency of system commercial components; 
Activity: 
* Project plans explicitly provide for evaluation, acquisition, and 
implementation of new, often frequent, product releases; 
* Modification or upgrades to deployed versions of system components 
are centrally controlled and unilateral user release changes are 
precluded.

Best practices: Dependency analysis: To ensure that relationships 
between commercial products are understood before acquisition 
decisions are made; 
Activity: 
* Decisions about acquisition of commercial components are based on 
deliberate and thorough research, analysis, and evaluation of the 
components' interdependencies.

Best practices: Legacy systems integration planning: To ensure 
reasonable planning for integration of commercial products with 
existing systems; 
Activity: 
* Project plans explicitly provide for the necessary time and 
resources for integrating commercial components with legacy systems.

Best practices: Organization change management: To ensure that the 
organizational impact of using new system functionality is proactively 
managed; 
Activity: 
* Project plans explicitly provide for preparing users on the impact 
that the business processes embedded in the commercial components will 
have on the users' respective roles and responsibilities; 
* The introduction and adoption of changes to how users will be 
expected to execute their jobs are actively managed.

Best practices: Solicitation: To ensure that a quality solicitation is 
produced and a best-qualified contractor is selected; 
Activity: 
* Systems integration contractors are explicitly evaluated on their 
ability to implement commercial components.

Best practices: Tradeoff analysis: To ensure that system requirements 
alone do not drive the system's solution; 
Activity: 
* Investment decisions throughout a system's life cycle are based on 
tradeoffs among the availability of commercial products (current and 
future), the architectural environment in which the system is to 
operate (current and future), defined system requirements, and 
acquisition cost/schedule constraints.

Best practices: Vendor and product research and evaluation: To ensure 
that vendor and product characteristics are understood before 
acquisition decisions are made; 
Activity: 
* Commercial component and vendor options are researched, evaluated/
tested, and understood, both early and continuously; 
* A set of evaluation criteria for selecting among commercial 
component options is established that includes both defined system 
requirements and vendor/commercial product characteristics (e.g., 
customer satisfaction with company and product line). 

Sources: See sources listed in appendix I of this report.

[End of table]

DOD's acquisition policies and guidance largely incorporate the 10 best 
practices that are relevant to any business systems 
acquisition.[Footnote 8] More specifically, they fully incorporate 7 of 
the 10 practices and partially incorporate the other 3 practices. 
However, they generally do not incorporate the 8 best practices that 
relate to acquiring commercial component-based business systems. In 
particular, they fully incorporate 1 best practice, partially 
incorporate 2, and do not incorporate the remaining 5. (See fig. 3 for 
a summary of our analysis.) At our request, DOD officials responsible 
for the 5000 series also assessed it against those 18 practices, and we 
incorporated information from their assessment into ours. These 
officials also told us that the acquisition guidebook is currently 
being expanded to incorporate additional best practices, but they did 
not provide us with a plan for accomplishing this. Until this is 
accomplished, DOD is increasing the risk that important and beneficial 
best practices will not be followed and that DOD business systems 
investments will not deliver promised capabilities and benefits on time 
and within budget.

Figure 3: DOD Acquisition Policies and Guidance Incorporation of Best 
Practices: 

[See PDF for image] 

[End of figure] 

The 5000 Series Largely Incorporates Best Practices Relevant to Any 
Business Systems Acquisition: 

Of the 10 best practices that we categorized as relevant to the 
acquisition of any business system, whether custom-developed or 
developed using commercial packages and products, essentially all have 
been incorporated into DOD's acquisition policies and guidance. (See 
table 3 for our detailed comparative analysis of the 5000 series 
against the 10 best practices.) For example, those practices aimed at 
ensuring that the acquisition is well planned, that the system is 
adequately tested and evaluated against contractual requirements, and 
that the requirements are clearly defined and understood by all 
stakeholders are all addressed in the 5000 series. Similarly, for the 3 
practices that are not fully addressed, this is the case because one 
activity associated with the practice is not addressed. According to 
DOD officials responsible for revising the 5000 series, the policies 
contain those best practices mandated by either law or DOD regulation, 
and other, optional best practices are contained in the interim 
guidance represented by the former DOD 5000.2-R.

Nevertheless, the activities that are missing from the 3 practices are 
important, and their absence increases the risk that the activities, 
and thus the practice, will not be adequately performed. In turn, this 
increases the risk that acquisition projects will fall short of 
expectations. The best practice aimed at ensuring that risks are 
proactively identified and systematically mitigated is a case in point. 
This practice has five activities associated with it, one of which the 
5000 series does not address--project reviews include the status of 
identified risks. As with all the activities under this practice, this 
activity plays an important role in ensuring that the appropriate level 
of attention and visibility is regularly given to risk identification 
and mitigation to ensure that it is effectively executed. Conversely, 
if the activities are not provided for in policy and guidance, it is 
unlikely that they will be performed, and it is likely that acquisition 
risks will become cost, schedule, and performance problems.

Table 3: Activity-by-activity Comparison of the 5000 Series to Best 
Practices Relevant to Any IT Business Systems Acquisition: 

Best practice: Acquisition planning; 
5000 series incorporates this best practice? Fully.

Best practice: Acquisition planning; 
Activity: Plans are prepared during acquisition planning and maintained 
throughout the acquisition; 
5000 series addresses this activity? Yes.

Best practice: Acquisition planning; 
Activity: Planning addresses the entire acquisition process, as well 
as life cycle support of the products being acquired; 
5000 series addresses this activity? Yes.

Best practice: Acquisition planning; 
Activity: The acquisition organization has a written policy for 
planning the acquisition; 
5000 series addresses this activity? Yes.

Best practice: Acquisition planning; 
Activity: Responsibility for acquisition planning activities is 
designated; 
5000 series addresses this activity? Yes.

Best practice: Architectural alignment; 
5000 series incorporates this best practice? Partially.

Best practice: Architectural alignment; 
Activity: The system being acquired is assessed for alignment with the 
enterprise architecture at key life cycle decision points, and any 
deviations from the architecture are understood and justified by an 
explicit waiver to the architecture; 
5000 series addresses this activity? Yes.

Best practice: Architectural alignment; 
Activity: Product line requirements--rather than just the requirements 
for the system being acquired--are an explicit consideration in each 
acquisition; 
5000 series addresses this activity? No.

Best practice: Contract tracking and oversight; 
5000 series incorporates this best practice? Fully.

Best practice: Contract tracking and oversight; 
Activity: The acquiring organization has sufficient insight into the 
contractor's activities to manage and control the contractor and ensure 
that contract requirements are met; 
5000 series addresses this activity? Yes.

Best practice: Contract tracking and oversight; 
Activity: The acquiring organization and contractor maintain ongoing 
communication; 
commitments are agreed to and implemented by both parties; 
5000 series addresses this activity? Yes.

Best practice: Contract tracking and oversight; 
Activity: All contract changes are managed throughout the life of the 
contract; 
5000 series addresses this activity? Yes.

Best practice: Contract tracking and oversight; 
Activity: The acquisition organization has a written policy for 
contract tracking and oversight; 
5000 series addresses this activity? Yes.

Best practice: Contract tracking and oversight; 
Activity: Responsibility for contract tracking and oversight activities 
is designated; 
5000 series addresses this activity? Yes.

Best practice: Contract tracking and oversight; 
Activity: The acquiring organization involves contracting specialists 
in the execution of the contract; 
5000 series addresses this activity? Yes.

Best practice: Contract tracking and oversight; 
Activity: A quantitative set of software and system metrics are used 
to define and measure product quality and contractor performance; 
5000 series addresses this activity? Yes.

Best practice: Contract tracking and oversight; 
Activity: In addition to incentives for meeting cost and schedule 
estimates, measurable, metrics-based product quality incentives are 
explicitly cited in the contract; 
5000 series addresses this activity? Yes.

Best practice: Economic justification; 
5000 series incorporates this best practice? Fully.

Best practice: Economic justification; 
Activity: System investment decisions are made on the basis of 
reliable analyses of estimated costs, expected benefits, and 
anticipated risks; 
5000 series addresses this activity? Yes.

Best practice: Economic justification; 
Activity: Large system projects are (to the maximum extent practical) 
divided into a series of smaller, incremental acquisition efforts, and 
investment decisions on these smaller efforts are made on the basis of 
reliable analyses of estimated costs, expected benefits, and 
anticipated risks; 
5000 series addresses this activity? Yes.

Best practice: Evaluation; 
5000 series incorporates this best practice? Fully.

Best practice: Evaluation; 
Activity: Evaluation requirements are developed in conjunction with the 
contractual requirements and are maintained over the life of the 
acquisition; 
5000 series addresses this activity? Yes.

Best practice: Evaluation; 
Activity: Evaluations are planned and conducted throughout the total 
acquisition period to provide an integrated approach that satisfies 
evaluation requirements and takes advantage of all evaluation results; 
5000 series addresses this activity? Yes.

Best practice: Evaluation; 
Activity: Evaluations provide an objective basis to support the product 
acceptance decision; 
5000 series addresses this activity? Yes.

Best practice: Evaluation; 
Activity: The acquiring organization has a written policy for managing 
the evaluation of the acquired products; 
5000 series addresses this activity? Yes.

Best practice: Evaluation; 
Activity: Responsibility for evaluation activities is designated; 
5000 series addresses this activity? Yes.

Best practice: Project management; 
5000 series incorporates this best practice? Partially.

Best practice: Project management; 
Activity: Project management activities are planned, organized, 
controlled, and communicated; 
5000 series addresses this activity? Partly--communication not cited.

Best practice: Project management; 
Activity: The performance, cost, and schedule of the acquisition are 
continually measured, compared with planned objectives, and controlled; 
5000 series addresses this activity? Yes.

Best practice: Project management; 
Activity: Problems discovered during the acquisition are managed and 
controlled; 
5000 series addresses this activity? Yes.

Best practice: Project management; 
Activity: The acquisition organization has a written policy for 
project management; 
5000 series addresses this activity? Yes.

Best practice: Project management; 
Activity: Responsibility for project management is designated; 
5000 series addresses this activity? Yes.

Best practice: Requirements development and management; 
5000 series incorporates this best practice? Fully.

Best practice: Requirements development and management; 
Activity: Contractual requirements are developed, managed, and 
maintained; 
5000 series addresses this activity? Yes.

Best practice: Requirements development and management; 
Activity: The end user and other affected groups have input into the 
contractual requirements over the life of the acquisition; 
5000 series addresses this activity? Yes.

Best practice: Requirements development and management; 
Activity: Contractual requirements are traceable and verifiable; 
5000 series addresses this activity? Yes.

Best practice: Requirements development and management; 
Activity: The contractual requirements baseline is established prior to 
release of the solicitation package; 
5000 series addresses this activity? Yes.

Best practice: Requirements development and management; 
Activity: The acquisition organization has a written policy for 
establishing and managing the contractual requirements; 
5000 series addresses this activity? Yes.

Best practice: Requirements development and management; 
Activity: Responsibility for requirements development and management is 
designated; 
5000 series addresses this activity? Yes.

Best practice: Requirements development and management; 
Activity: Requirements that are mandatory versus optional are clearly 
delineated and used in deciding what requirements can be eliminated or 
postponed to meet other project goals, such as cost and schedule 
constraints; 
5000 series addresses this activity? Yes.

Best practice: Risk management; 
5000 series incorporates this best practice? Partially.

Best practice: Risk management; 
Activity: Projectwide participation in the identification and 
mitigation of risks is encouraged; 
5000 series addresses this activity? Yes.

Best practice: Risk management; 
Activity: The defined acquisition process provides for the 
identification, analysis, and mitigation of risks; 
5000 series addresses this activity? Yes.

Best practice: Risk management; 
Activity: Milestone reviews include the status of identified risks; 
5000 series addresses this activity? No.

Best practice: Risk management; 
Activity: The acquisition organization has a written policy for 
managing acquisition risk; 
5000 series addresses this activity? Yes.

Best practice: Risk management; 
Activity: Responsibility for acquisition risk management activities is 
designated; 
5000 series addresses this activity? Yes.

Best practice: Solicitation; 
5000 series incorporates this best practice? Fully.

Best practice: Solicitation; 
Activity: The solicitation package includes the contractual 
requirements and the proposal evaluation criteria; 
5000 series addresses this activity? Yes.

Best practice: Solicitation; 
Activity: The technical and management elements of proposals are 
evaluated to ensure that the requirements of the contract will be 
satisfied; 
5000 series addresses this activity? Yes.

Best practice: Solicitation; 
Activity: The selection official selects a supplier who is qualified 
to satisfy the contract's requirements; 
5000 series addresses this activity? Yes.

Best practice: Solicitation; 
Activity: The acquiring organization has a written policy for 
conducting the solicitation; 
5000 series addresses this activity? Yes.

Best practice: Solicitation; 
Activity: Responsibility for the solicitation is designated; 
5000 series addresses this activity? Yes.

Best practice: Solicitation; 
Activity: A selection official has been designated to be responsible 
for the selection process and decision; 
5000 series addresses this activity? Yes.

Best practice: Solicitation; 
Activity: The acquiring team includes contracting specialists to 
support contract administration; 
5000 series addresses this activity? Yes.

Best practice: Transition to support; 
5000 series incorporates this best practice? Fully.

Best practice: Transition to support; 
Activity: The acquiring organization ensures that the support 
organization has the capacity and capability to provide the required 
support; 
5000 series addresses this activity? Yes.

Best practice: Transition to support; 
Activity: There is no loss in continuity of support to the products 
during transition from the supplier to the support organization; 
5000 series addresses this activity? Yes.

Best practice: Transition to support; 
Activity: Configuration management of the products is maintained 
throughout the transition; 
5000 series addresses this activity? Yes.

Best practice: Transition to support; 
Activity: The acquiring organization has a written policy for 
transitioning the products to the support organization; 
5000 series addresses this activity? Yes.

Best practice: Transition to support; 
Activity: The acquiring organization ensures that the support 
organization is involved in planning for transition to support; 
5000 series addresses this activity? Yes.

Best practice: Transition to support; 
Activity: Responsibility for transition to support activities is 
designated; 
5000 series addresses this activity? Yes. 

Source: GAO, based on analysis of DOD data.

[End of table]

The 5000 Series Generally Does Not Incorporate Best Practices Relevant 
to Commercial Component-Based Business Systems Acquisitions: 

Of the 8 best practices relevant to acquiring commercial component-
based business systems, few have been incorporated into DOD systems 
acquisition policies and guidance. (See table 4 for our detailed 
comparative analysis of the 5000 series against the 8 best practices.) 
For example, while the practice aimed at ensuring that adequate 
planning takes place for integrating commercial products with legacy 
systems is incorporated into the 5000 series, practices associated with 
closely controlling any modification to the software of these packages 
and products, thoroughly analyzing and understanding the dependencies 
among commercial products before acquiring them, and proactively 
managing the institutional change that results from implementing the 
functionality in commercial packages and products are not incorporated. 
According to DOD officials responsible for revising the 5000 series, 
these practices were not included in the recently revised version of 
DOD's acquisition policies because they included only those in existing 
law or regulation.

Nevertheless, the absence of these practices from the 5000 series 
increases the risk that the practices will not be performed, which, in 
turn, increases the risk that acquisition projects will fall short of 
expectations. The practice intended to ensure development of a quality 
solicitation and selection of a best-qualified contractor illustrates 
this. Specifically, this practice calls for contract bidders to be 
evaluated on their ability to implement commercial components. This 
evaluation is important because integrating and implementing these 
component products is sufficiently different from developing customized 
system solutions; it requires different core competencies and 
experiences to be successful. By explicitly taking this into 
consideration in evaluating and selecting a contractor, the risk of 
contract award to a less-than-best-qualified contractor is reduced.

Table 4: Activity-by-Activity Comparison of the 5000 Series to Best 
Practices Relevant to Commercial Component-based Business Systems 
Acquisitions: 

Best practice: Component modification; 
Does the 5000 series incorporate this best practice? No.

Best practice: Component modification; 
Activity: Modification of commercial components is discouraged and 
allowed only if justified by a thorough analysis of life-cycle costs 
and benefits; 
Does the 5000 series address this activity? No.

Best practice: Configuration management; 
Does the 5000 series address this activity? Partially.

Best practice: Configuration management; 
Activity: Project plans provide for evaluation, acquisition, and 
implementation of new, often frequent, product releases; 
Does the 5000 series address this activity? Yes.

Best practice: Configuration management; 
Activity: Modification or upgrades to deployed versions of system 
components are centrally controlled and unilateral user release 
changes are precluded; 
Does the 5000 series address this activity? No.

Best practice: Legacy systems integration planning; 
Does the 5000 series address this activity? Fully.

Best practice: Legacy systems integration planning; 
Activity: Project plans explicitly provide for the necessary time and 
resources for integrating commercial components with legacy systems; 
Does the 5000 series address this activity? Yes.

Best practice: Dependency analysis; 
Does the 5000 series address this activity? No.

Best practice: Dependency analysis; 
Activity: Decisions about acquisition of commercial components are 
based on deliberate and thorough research, analysis, and evaluation of 
the components' interdependencies; 
Does the 5000 series address this activity? No.

Best practice: Organization change management; 
Does the 5000 series address this activity? No.

Best practice: Organization change management; 
Activity: Project plans provide for preparing users for the impact 
that the business processes embedded in the commercial components will 
have on the users' respective roles and responsibilities; 
Does the 5000 series address this activity? No.

Best practice: Organization change management; 
Activity: The introduction and adoption of changes to how users will 
be expected to use the system to execute their jobs are actively 
managed; 
Does the 5000 series address this activity? No.

Best practice: Solicitation; 
Does the 5000 series address this activity? No.

Best practice: Solicitation; 
Activity: Systems integration contractors are explicitly evaluated on 
their ability to implement commercial components; 
Does the 5000 series address this activity? No.

Best practice: Tradeoff analysis; 
Does the 5000 series address this activity? No.

Best practice: Tradeoff analysis; 
Activity: Investment decisions throughout a system's life cycle are 
based on tradeoffs among the availability of commercial products 
(current and future), the architectural environment in which the system 
is to operate (current and future), defined system requirements, and 
acquisition cost/schedule constraints; 
Does the 5000 series address this activity? No.

Best practice: Vendor and product research and evaluation; 
Does the 5000 series address this activity? Partially.

Best practice: Vendor and product research and evaluation; 
Activity: Commercial component and vendor options are researched, 
evaluated/tested, and understood, both early and continuously; 
Does the 5000 series address this activity? Yes.

Best practice: Vendor and product research and evaluation; 
Activity: A set of evaluation criteria for selecting among commercial 
component options is established that includes both defined system 
requirements and vendor/commercial product characteristics (e.g., 
customer satisfaction with company and product line); 
Does the 5000 series address this activity? Partly--vendor/commercial 
product characteristics not cited. 

Source: GAO, based on analysis of DOD data.

[End of table]

DOD Officials Report That They Are in the Process of Revising the 
Interim Defense Acquisition Guidebook: 

The DOD officials responsible for revising the 5000 series told us they 
recognize the need for the 5000 series to incorporate additional best 
practices. To this end, they reported that efforts are under way to 
expand the Interim Defense Acquisition Guidebook to include additional 
best practices and lessons learned across the department. However, the 
officials could not provide us with a documented plan and associated 
documentation showing how this task will be accomplished, what 
resources are needed and assigned to accomplish it, when it will be 
accomplished, and where the department stands in accomplishing it. 
Instead, the officials told us that progress on it has been slowed by 
other priorities, such as the need to first revise DOD Directive 5000.1 
and DOD Instruction 5000.2. They also said that there is only a small 
number of staff available to work on what they described as being an 
extensive revision of the guidebook. According to these officials, 80 
to 90 percent of the revision has been completed and reviewed and their 
goal is to publish the initial version of the revised guidebook by 
September 30, 2004. In our view, until the missing best practices that 
we cite in this report are included in DOD's acquisition policies and 
guidance, the chance that business systems acquisitions will follow the 
policies and guidance and consistently produce a successful outcome is 
diminished.

DOD's Acquisition Policies Do Not Contain Sufficient Controls to Ensure 
That the Requirement Is Met for Appropriately Applying Best Practices: 

Federal laws and regulations define the need for effective controls 
over agency programs, and controls are a key factor in achieving 
program results, minimizing operational problems, and managing evolving 
demands and priorities.[Footnote 9] Controls over defined processes, 
procedures, and support activities are considered effective if they 
entail measuring and verifying whether a given practice is followed. 
Without sufficient controls, it is unlikely that practices will be 
consistently employed, which, in turn, increases the probability that 
the positive program and project outcomes these practices are designed 
to produce will not occur.

DOD's acquisition policy requires program managers and investment 
decision authorities to examine and, as appropriate, adopt best 
practices. However, neither the policies nor the accompanying guidance 
explain what "examine" means, including whether practice use is to be 
measured and verified. Instead, the policies state that any issues 
regarding the intent of the 5000 series, which would include whether 
practice adoption is to be measured and validated, shall be resolved by 
the investment decision authority, meaning that it is entirely up to 
this individual what information relative to the use of best practices 
is relevant and necessary to ensure that best practices are 
appropriately followed. According to the Chairman of the Defense 
Acquisition Policy Working Group,[Footnote 10] this control is 
sufficient, and explicit requirements for measuring and validating the 
use of best practices are not necessary.

In our view, not requiring that decision authorities examine the 
measurement and validation of best practices' use increases the chance 
that important best practices will not be appropriately followed, as: 

required by DOD policy. As we have previously reported,[Footnote 11] a 
lack of explicit controls that require review of relevant information 
at key decision points raises the risk of making uninformed project 
decisions, as well as the risk that investments will not meet cost, 
schedule, capability, and benefit commitments.

Conclusions: 

DOD recognizes the importance of business systems acquisition best 
practices by including best practices in their revised policy and 
guidance. However, other practices that, if followed, could increase 
the odds of acquisitions delivering promised system capabilities and 
benefits on time and within budget have yet to be similarly included. 
In particular, those practices associated with the successful 
acquisition of commercial component-based business systems have not 
been sufficiently incorporated into either the policies or the 
guidance. Moreover, effective controls for ensuring that best practices 
are appropriately followed are not adequately provided for in the 
policies. Although DOD officials intend to expand the coverage of best 
practices in future versions of DOD's acquisition guidance, it is 
unclear what the scope, nature, and status of these intentions are 
because explicit plans for revising the guidance and associated 
progress reports were not available. Until DOD incorporates the best 
practices we found missing in the 5000 series, and until it strengthens 
the means by which the appropriate use of these practices will be 
ensured, its business systems acquisitions will be exposed to 
unnecessary risk. Therefore, it is important for DOD to treat further 
revisions of its acquisition policy and guidance relative to business 
systems as a priority and move quickly to incorporate missing best 
practices and associated controls for ensuring that the practices are 
followed.

Recommendations for Executive Action: 

To improve DOD's ability to acquire business systems, we recommend that 
the Secretary of Defense direct the Under Secretary for AT&L, in 
collaboration with the Assistant Secretary for NII and the Director, 
OT&E, to take the following actions: 

1. Develop and implement an explicit plan for incorporating into the 
5000 series the best practices and associated activities currently 
missing from the series. We recommend that the plan specify tasks to be 
performed, resources needed and assigned, and milestones for completing 
tasks.

2. We further recommend that progress against this plan be tracked and 
reported as appropriate, and that the plan, at a minimum, incorporate 
each of the following best practice activities: 

* Product line requirements--rather than just the requirements for the 
system being acquired--are an explicit consideration in each 
acquisition.

* Acquisition project management activities are communicated to all 
stakeholders.

* Acquisition reviews include the status of identified risks.

* Modification of commercial components is discouraged and allowed only 
if justified by a thorough analysis of life-cycle costs and benefits.

* Modification or upgrades to deployed versions of system components 
are centrally controlled, and unilateral user release changes are 
precluded.

* Acquisition decisions about commercial components are based on 
deliberate and thorough research, analysis, and evaluation of the 
components' interdependencies.

* Acquisition plans provide for preparing users for the impact that the 
business processes embedded in the commercial components will have on 
their respective roles and responsibilities.

* Changes affecting how users will be expected to use the system to 
execute their jobs are actively managed.

* Systems integration contractors are explicitly evaluated on their 
ability to implement commercial components.

* Investment decisions throughout a system's life cycle are based on a 
continuous set of tradeoffs among capabilities available in commercial 
components (current and future), the architectural environment in which 
the system is to operate, defined system requirements, and existing 
cost/schedule constraints.

* Evaluation criteria are established for selecting among commercial 
component options that include both defined system requirements and 
vendor/commercial product characteristics.

3. To ensure that the best practices provided for in DOD acquisition 
policies and guidance are appropriately followed, we also recommend 
that the above recommended plan incorporate steps to include in DOD's 
acquisition policies a provision for measurement and verification of 
best practices.

Agency Comments and Our Evaluation: 

In written comments on a draft of this report, signed by the Principal 
Director for Command, Control, Communications, Space, and Information 
Technology Programs in the office of the DOD Assistant Secretary for 
Networks and Information Integration, DOD agreed with the importance 
and relevance of the best practices that we cite in the report. 
Additionally, DOD agreed with 2 of our 13 recommendations for 
incorporating additional best practices, stating that the department 
would incorporate the 2 practices in its policies and guidance.

DOD also partially agreed with 9 of our recommendations for 
incorporating additional practices, stating that it would consider 
augmenting its coverage of 5 of the practices and that it believed that 
4 practices already existed in its policies and guidance. With regard 
to the 5 practices, DOD stated that it needed to review each practice 
further and determine the need for its emphasis or endorsement in the 
5000 series. We understand DOD's desire to carefully consider changes 
to its acquisition policies and guidance, and believe that such careful 
deliberation is consistent with the spirit of our recommendations.

With regard to the remaining 4 practices that DOD partially agreed 
with, we do not agree with the department's comment that these best 
practices adequately exist in the 5000 series. For example, DOD 
commented that because its existing policies and guidance provide for 
the use of integrated product teams, which, according to DOD, are a 
means for promoting collaboration and facilitating communication among 
stakeholders, its policies and guidance therefore already provide for 
communicating information about management of a given acquisition 
project to all relevant project stakeholders. While we do not question 
the use of integrated product teams as a way to communicate 
information, the point of our recommendation is that there needs to be 
an explicit recognition in policy or guidance of the type of 
information to be communicated and with whom it is to be communicated. 
Restated, our recommendation for incorporating the best practice of 
communicating acquisition management activities to all stakeholders is 
intended to permit communication vehicles, such as integrated product 
teams, to be more effective by explicitly providing for this best 
practice in relevant policies and guidance. As another example, we do 
not agree with DOD's comment that 2 of the best practices that we 
recommended for incorporation in its policies and guidance--preparing 
users for the impact that business process changes embedded in 
commercial components will have on their roles and responsibilities, 
and actively managing changes in how users will use new systems--are 
already sufficiently contained in the 5000 series. In particular, while 
we agree that the series references an acquisition management toolkit 
that addresses these 2 best practices, this reference is provided only 
once in the 5000 series, and this reference is only in relation to one 
phase of the acquisition cycle (the technology development phase). 
Given the importance and relevance of these practices to successful 
implementation of commercial component-based systems, our position, and 
thus the basis for our recommendations, is that the practices' 
implementation would be more likely to occur if the practices were 
visible and better recognized in all relevant stages of DOD's 
acquisition cycle.

Also in its comments, DOD did not agree with our recommendations to 
develop and implement an explicit plan to govern its ongoing and future 
policy and guidance revision activities, specifically stating that the 
recommendation was inappropriate and offering updated information on 
the status of and associated milestone for completing its activities. 
While we have updated our report to include the revised status and 
milestone information, we do not agree with DOD that a plan governing 
these efforts is not needed. Given the importance of DOD's acquisition 
policies and guidance, and the need for their continuous review and 
update to reflect new acquisition best practices, we believe that 
having an explicit plan that defines how and when these policies and 
guidance will be incorporated is essential. Among other things, a plan 
would highlight the resource constraints that this revision effort has 
been subject to, would allow measurement against defined milestones, 
and would allow disclosure of progress and impediments.

DOD also did not agree with our recommendation to add stronger controls 
for ensuring adherence to the best practices that are contained in its 
acquisition policies and guidance, stating that its existing oversight 
process includes the necessary compliance activities. We disagree. As 
we state in the report, DOD's existing policy leaves these compliance 
activities to the discretion of the program manager and the investment 
decision authority, and it does not provide for measurement and 
verification of the use of best practices, both of which are recognized 
components of effective control processes.

A copy of DOD's comments is reprinted in appendix III, along with our 
response. 

We are sending copies of this report to the Chairmen and Ranking 
Minority Members of the Senate and House Committees on Armed Services; 
Subcommittees on Defense, Senate and House Committees on 
Appropriations; and the Subcommittee on Military Readiness, House 
Committee on Armed Services. We are also sending copies to the 
Director, Office of Management and Budget; the Secretary of Defense; 
the Under Secretary of Defense (AT&L); the Assistant Secretary of 
Defense (NII)/Chief Information Officer; and the Director, OT&E. We 
will make copies available to others on request. This report will also 
be available at no charge on our Web site at http: //www.gao.gov.

If you or your staff has any questions concerning this report, please 
contact me at (202) 512-3439. I can also be reached by e-mail at 
[Hyperlink, hiter@gao.gov]. Other contacts and key contributors to this 
report are listed in appendix IV.

Signed by: 

Randolph C. Hite: 
Director, Information Technology Architecture and Systems Issues: 

[End of section]

Appendixes: 

Appendix I: Objectives, Scope, and Methodology: 

Our objectives were to determine whether the Department of Defense's 
(DOD) revised systems acquisition policies for acquiring information 
technology (IT) business systems (1) are consistent with industry best 
practices, including those pertaining to commercial component-based 
systems, and (2) provide the necessary controls to ensure that the 
department's component organizations adhere to the practices.

To accomplish the first objective, we identified the DOD policies and 
guidance relevant to business systems. These policies and guidance are 
contained in three documents--DOD Directive 5000.1, DOD Instruction 
5000.2, and the Interim Defense Acquisition Guidebook--and are 
generally referred to as the 5000 series. We then reviewed each of 
these documents and discussed with DOD officials responsible for 
developing and revising the documents what steps were taken to 
incorporate best practices into each document. The DOD officials that 
we interviewed were from the offices of the Under Secretary of Defense 
(Acquisition, Technology, and Logistics (AT&L)); the Assistant 
Secretary of Defense (Networks and Information Integration (NII)); and 
the Director, Operational Test and Evaluation (OT&E). Next, we 
researched prior GAO reports; the work of federally funded research and 
development organizations, such as the Software Engineering Institute 
and The Aerospace Corporation;[Footnote 12] and other authoritative 
sources to identify business systems acquisition best practices. Our 
research produced 18 best practices, including associated activities, 
that we placed into two categories--one category for the practices that 
are relevant to any business systems acquisition and one category for 
the practices that are relevant to commercial component-based business 
systems acquisitions. In particular, we drew extensively from the 
Software Engineering Institute's Software Acquisition Capability 
Maturity Model.[Footnote 13] In doing so, we selected practices from 
the model's repeatable level of process maturity, which is level two on 
the model's five-level scale. We used the repeatable level of process 
maturity because it is intended to provide the necessary process 
discipline to allow an organization to repeat earlier successes on 
similar projects. In addition, we included one Software Acquisition 
Capability Maturity Model level-three process area--risk management--
because many experts consider it to be one of the most important 
process areas. We did not attempt to develop an exhaustive list of best 
practices and, in fact, fully recognize that additional best practices 
exist, such as ensuring that the appropriate level of human capital 
knowledge, skills, and abilities are employed, as well as that 
additional activities for the practices that we have identified exist, 
such as those configuration management activities associated with 
identifying, controlling, reporting on, and auditing configuration 
items and components. For the purposes of this report, we identified 
those practices that are embodied, recognized, and accepted acquisition 
models or frameworks, as well as those practices that are now being 
recognized as being unique to commercial component-based systems and 
for which there appears to be general agreement, including agreement 
with DOD officials responsible for revising the 5000 series, that the 
practices are relevant and important. Last, we analyzed each of the DOD 
5000 series documents to determine whether the documents addressed, 
either directly or indirectly by reference to another authoritative 
document, the 18 best practices that we identified. Based on this 
analysis, we judged whether the 5000 series documents fully, partially, 
or did not incorporate each best practice. In making these judgments, 
we used the following criteria: 

* To fully incorporate the practice, the 5000 series addressed all of 
the practice's activities.

* To partially incorporate the practice, the 5000 series addressed 
some, but not all, of the practice's activities.

* To not incorporate the practice, the 5000 series did not address any 
of the practice's activities.

Additionally, we provided the DOD officials responsible for revising 
the 5000 series with the 18 practices that we identified to obtain 
their views on whether the practices were relevant to DOD business 
systems acquisitions. The officials agreed that they were. We also 
requested that these officials perform their own assessment of the 5000 
series against these practices, and we used these officials' assessment 
in making our judgments as to whether the practices were fully, 
partially, or not incorporated into DOD's acquisition policies and 
guidance. For a number of the activities, DOD identified the Federal 
Acquisition Regulation and the Defense Federal Acquisition Regulation 
Supplement as evidence that the activity was being performed within a 
particular practice. We accepted that as proof that the activity was 
being covered within DOD's business systems acquisition policy.

To address our second objective, we researched federal internal control 
standards and controls inherent in the business systems acquisition 
best practices that we identified. In particular, we reviewed the 
Software Engineering Institute's Software Acquisition Capability 
Maturity Model framework and GAO's internal control standards.[Footnote 
14] We then analyzed DOD's revised acquisition policies and guidance to 
identify whether these controls were cited and to provide assurance 
that relevant best practices were being followed. We also interviewed 
DOD officials responsible for revising the 5000 series to determine 
reasons why controls were addressed or not addressed in the policies 
and guidance.

We conducted our work at DOD offices in Arlington, Virginia, between 
December 2003 and May 2004 in accordance with generally accepted 
government auditing standards: 

[End of section]

Appendix II: Best Practices: 

Additional information on each of the 18 best practices that we 
identified is provided in this appendix.

Best Practices Relevant to Any IT Business Systems Acquisition: 

1. Acquisition Planning: 

Purpose: To ensure that reasonable planning for all parts of the 
acquisition is conducted.

Description: Acquisition planning is the process for conducting and 
documenting acquisition planning activities beginning early and 
covering all parts of the project. It extends to all acquisition areas, 
such as budgeting, scheduling, resource estimating, risk 
identification, and requirements definition, as well as the overall 
acquisition strategy. Acquisition planning begins with the earliest 
identification of a requirement that is to be satisfied through an 
acquisition.

Activities: (1) Plans are prepared during acquisition planning and 
maintained throughout the acquisition. (2) Planning addresses the 
entire acquisition process, including life cycle support of the 
products being acquired. (3) The acquisition organization has a written 
policy for planning the acquisition. (4) Responsibility for acquisition 
planning activities is designated.

2. Architectural Alignment: 

Purpose: To ensure that the acquisition is consistent with the 
organization's enterprise architecture.

Description: Architectural alignment is the process for analyzing and 
verifying that the proposed architecture of the system being acquired 
is consistent with the enterprise architecture for the organization 
acquiring the system. Such alignment is needed to ensure that acquired 
systems can interoperate and are not unnecessarily duplicative of one 
another. Exceptions to this alignment requirement are permitted, but 
only when justified and only when granted an explicit waiver from the 
architecture. A particular architectural consideration is whether 
requirements that extend beyond the specific system being acquired 
should be considered when selecting system components. Such product 
line (i.e., systems that are developed from a common set of assets and 
share a common and managed set of features) considerations can provide 
substantial production economies over acquiring systems from scratch.

Activities: (1) The system being acquired is assessed for alignment 
with the enterprise architecture at key life cycle decision points, and 
any deviations from the architecture are explicitly understood and 
justified by an explicit waiver to the architecture. (2) Product line 
requirements--rather than just the requirements for the system being 
acquired--are an explicit consideration in each acquisition.

3. Contract Tracking and Oversight: 

Purpose: To ensure that contract activities are performed in accordance 
with contractual requirements.

Description: Contract tracking and oversight is the process by which 
contractual agreements are established and contractor efforts to 
satisfy those agreements are supervised. It involves information 
sharing between the acquirer and contractor to ensure that contractual 
requirements are understood, that there are regular measurements to 
disclose overall project status and whether problems exist, and that 
there are appropriate incentives for ensuring that cost and schedule 
commitments are met and that quality products are delivered. Contract 
tracking and oversight begins with the award of the contract and ends 
at the conclusion of the contract's period of performance.

Activities: (1) The acquiring organization has sufficient insight into 
the contractor's activities to manage and control the contractor and 
ensure that contract requirements are met. (2) The acquiring 
organization and contractor maintain ongoing communication; 
commitments are agreed to and implemented by both parties. (3) All 
contract changes are managed throughout the life of the contract. (4) 
The acquiring organization has a written policy for contract tracking 
and oversight. (5) Responsibility for contract tracking and oversight 
activities is designated. (6) The acquiring organization involves 
contracting specialists in the execution of the contract. (7) A 
quantitative set of software and system metrics is used to define and 
measure product quality and contractor performance.[Footnote 15] (8) 
In addition to incentives for meeting cost and schedule estimates, 
measurable, metrics-based product quality incentives are explicitly 
cited in the contract.[Footnote 16]

4. Economic Justification: 

Purpose: To ensure that system investments have an adequate economic 
justification.

Description: Economic justification is the process for ensuring that 
acquisition decisions are based on reliable analyses of the proposed 
investment's likely costs versus benefits over its useful life, as well 
as an analysis of the risks associated with actually realizing the 
acquisition's forecasted benefits for its estimated costs. Moreover, it 
entails minimizing the risk and uncertainty of large acquisitions that 
require spending large sums of money over many years by breaking the 
acquisition into smaller, incremental acquisitions. Economic 
justification is not a one-time event, but rather is performed 
throughout an acquisition's life cycle in order to permit informed 
investment decision making.

Activities: (1) System investment decisions are made on the basis of 
reliable analyses of estimated system life cycle costs, expected 
benefits, and anticipated risks. (2) Large systems acquisitions are (to 
the maximum extent practical) divided into a series of smaller, 
incremental acquisition efforts, and investment decisions on these 
smaller efforts are made on the basis of reliable analyses of estimated 
costs, expected benefits, and anticipated risks.

5. Evaluation: 

Purpose: To ensure that evidence showing that the contract products 
satisfy the defined requirements are provided prior to accepting 
contractor products.

Description: Evaluation is the process by which contract deliverables 
are analyzed to determine whether they meet contract requirements. It 
includes developing criteria such as product acceptance criteria to be 
included into both the solicitation package and the contract. It should 
be conducted continuously throughout the contract period as products 
are delivered. It begins with development of the products' requirements 
and ends when the acquisition is completed.

Activities: (1) Evaluation requirements are developed in conjunction 
with the contractual requirements and are maintained over the life of 
the acquisition. (2) Evaluations are planned and conducted throughout 
the total acquisition period to provide an integrated approach that 
satisfies evaluation requirements and takes advantage of all evaluation 
results. (3) Evaluations provide an objective basis to support the 
product acceptance decision. (4) The acquisition organization has a 
written policy for managing the evaluation of the acquired products. 
(5) Responsibility for evaluation activities is designated.

6. Project Management: 

Purpose: To ensure that the project office and its supporting 
organizations function efficiently and effectively.

Description: Project management is the process for planning, 
organizing, staffing, directing, and managing all project-office-
related activities, such as defining project tasks, estimating and 
securing resources, scheduling activities and tasks, training, and 
accepting products. Project management begins when the project office 
is formed and ends when the acquisition is completed.

Activities: (1) Project management activities are planned, organized, 
controlled, and communicated. (2) The performance, cost, and schedule 
of the acquisition are continually measured, compared with planned 
objectives, and controlled. (3) Problems discovered during the 
acquisition are managed and controlled. (4) The acquisition 
organization has a written policy for project management. (5) 
Responsibility for project management is designated.

7. Requirements Development and Management: 

Purpose: To ensure that contractual requirements are clearly defined 
and understood by the acquisition stakeholders.

Description: Requirements development is the process for developing and 
documenting contractual requirements, including evaluating 
opportunities for reusing existing assets. It involves participation 
from end users to ensure that product requirements are well understood, 
and that optional versus mandatory requirements are clearly delineated. 
Requirements management is the process for establishing and maintaining 
agreement on the contractual requirements among the various 
stakeholders and for ensuring that the requirements are traceable, 
verifiable, and controlled. This involves baselining the requirements 
and controlling subsequent requirements changes. Requirements 
development and management begins when the solicitation's requirements 
are documented and ends when system responsibility is transferred to 
the support organization.

Activities: (1) Contractual requirements are developed, managed, and 
maintained. (2) The end user and other affected groups have input into 
the contractual requirements over the life of the acquisition. (3) 
Contractual requirements are traceable and verifiable. (4) The 
contractual requirements baseline is established prior to release of 
the solicitation package. (5) The acquisition organization has a 
written policy for establishing and managing the contractual 
requirements. (6) Responsibility for requirements development and 
management is designated. (7) Requirements that are mandatory versus 
optional are clearly delineated and used in deciding what requirements 
can be eliminated or postponed to meet other project goals, such as 
cost and schedule constraints.[Footnote 17]

8. Risk Management: 

Purpose: To ensure that risks are identified and systematically 
mitigated.

Description: Risk management is the process for identifying potential 
acquisition problems and taking appropriate steps to avoid their 
becoming actual problems. It includes risk identification and 
categorization based on estimated impact, development of risk 
mitigation strategies, and execution of and reporting on the 
strategies. Risk management occurs early and continuously in the 
acquisition life cycle.

Activities: (1) Projectwide participation in the identification and 
mitigation of risks is encouraged. (2) The defined acquisition process 
provides for the identification, analysis, and mitigation of risks. (3) 
Milestone reviews include the status of identified risks. (4) The 
acquisition organization has a written policy for managing acquisition 
risk. (5) Responsibility for acquisition risk management activities is 
designated.

9. Solicitation: 

Purpose: To ensure that a quality solicitation is produced and a best-
qualified contractor selected.

Description: Solicitation is the process for developing, documenting, 
and issuing the solicitation package; developing and implementing a 
plan to evaluate responses; conducting contract negotiations; and 
awarding the contract. Solicitation ends with contract award.

Activities: (1) The solicitation package includes the contractual 
requirements and the proposal evaluation criteria. (2) The technical 
and management elements of proposals are evaluated to ensure that the 
requirements of the contract will be satisfied. (3) The selection 
official selects a supplier who is qualified to satisfy the contract's 
requirements. (4) The acquiring organization has a written policy for 
conducting the solicitation. (5) Responsibility for the solicitation is 
designated. (6) A selection official has been designated to be 
responsible for the selection process and decision. (7) The acquiring 
team includes contracting specialists to support contract 
administration.

10. Transition to Support: 

Purpose: To ensure proper transfer of the system from the acquisition 
organization to the eventual support organization.

Description: Transition to support is the process for developing and 
implementing the plans for transitioning products to the support 
organization. This includes engaging relevant stakeholders in the 
acquisition and sharing information about the system's supporting 
infrastructure. Transition to support begins with requirements 
development and ends when the responsibility for the products is turned 
over to the support organization.

Activities: (1) The acquiring organization ensures that the support 
organization has the capacity and capability to provide the required 
support. (2) There is no loss in continuity of support to the products 
during transition from the supplier to the support organization. (3) 
Configuration management of the products is maintained throughout the 
transition. (4) The acquiring organization has a written policy for 
transitioning products to the support organization. (5) The acquiring 
organization ensures that the support organization is involved in 
planning for transition to support. (6) Responsibility for transition 
to support activities is designated.

Complementary Best Practices Relevant to Commercial Component-Based IT 
Business Systems Acquisitions: 

1. Component Modification: 

Purpose: To ensure that commercial product modification is effectively 
controlled.

Description: Component modification is the process for limiting the 
chances of a commercial product being modified to the point that it 
becomes a one-of-a-kind solution because doing so can result in 
extensive life cycle costs. Such modifications, if not incorporated 
into the commercially available version of the product by the supplier, 
mean that every product release has to be modified in accordance with 
the custom changes, thus precluding realization of some of the benefit 
of using a commercial product.

Activity: (1) Modification of commercial components is discouraged and 
allowed only if justified by a thorough analysis of life cycle costs 
and benefits.[Footnote 18]

2. Configuration Management: 

Purpose: To ensure the integrity and consistency of system commercial 
components.

Description: Configuration management relative to commercial 
component-based systems is the process for ensuring that changes to the 
commercial components of a system are strictly controlled. It 
recognizes that when using commercial components, it is the vendor, not 
the acquisition or support organization, that controls the release of 
new component versions and that new versions are released frequently. 
Thus, acquisition management needs to provide for both receiving new 
product releases and controlling the implementation of these releases.

Activities: (1) Project plans explicitly provide for evaluation, 
acquisition, and implementation of new, often frequent, product 
releases.[Footnote 19] (2) Modification or upgrades to deployed 
versions of system components are centrally controlled, and unilateral 
user release changes are precluded.

3. Dependency Analysis: 

Purpose: To ensure that relationships between commercial products are 
understood before acquisition decisions are made.

Description: Dependency analysis relative to commercial component-
based systems is the process for determining and understanding the 
characteristics of these products so that inherent dependencies among 
them can be considered before they are acquired. It involves 
recognizing that the logical and physical relationships among products 
impact one another. This is necessary because commercial products are 
built around each vendor's functional and architectural assumptions and 
paradigms, such as approaches to error handling and data access, and 
these assumptions and paradigms are likely to be different among 
products from different sources. Such differences complicate product 
integration. Further, some commercial products have built-in 
dependencies with other products that, if not known, can further 
complicate integration.

Activity: (1) Decisions about the acquisition of commercial components 
are based on deliberate and thorough research, analysis, and evaluation 
of the components' interdependencies.[Footnote 20]

4. Legacy Systems Integration Planning: 

Purpose: To ensure reasonable planning for integration of commercial 
products with existing systems.

Description: Legacy systems integration planning is the process for 
ensuring that the time and resources needed to integrate existing 
systems with the system being acquired are identified and provided for. 
It involves identifying which legacy systems will interact with the 
system being acquired and what kinds and levels of testing are 
required. Integration planning recognizes that, although some 
commercial products may provide mechanisms and information that is 
helpful in integration with legacy systems, the unavailability of the 
source code for commercial products and the different organizations 
that are responsible for the two will likely require additional time 
and effort.

Activity: (1) Project plans explicitly provide for the necessary time 
and resources for integrating commercial components with legacy 
systems.

5. Organization Change Management: 

Purpose: To ensure that the organizational impact of using new system 
functionality is proactively managed.

Description: Organization change management relative to commercial 
component-based systems is the process for preparing system users for 
the business process changes that will accompany implementation of the 
system. It involves engaging users and communicating the nature of 
anticipated changes to system users through training on how jobs will 
change. This is necessary because commercial products are created with 
the developers' expectations of how they will be used, and the 
products' functionality may require the organization implementing the 
system to change existing business processes.

Activities: (1) Project plans explicitly provide for preparing users on 
the impact that the business processes embedded in the commercial 
components will have on the user's respective roles and 
responsibilities. (2) The introduction and adoption of changes to how 
users will be expected to execute their jobs are actively 
managed.[Footnote 21]

6. Solicitation: 

Purpose: To ensure that a quality solicitation is produced and a best-
qualified contractor is selected.

Description: Solicitation relative to commercial component-based 
systems is the process for ensuring that a capable contractor is 
selected. It involves ensuring that the selected contractor has 
experience with integrating commercial component products. This is 
important because expertise in developing custom system solutions is 
different from expertise in implementing commercial components; it 
requires different core competencies and experiences to be successful.

Activity: (1) Systems integration contractors are explicitly evaluated 
on their ability to implement commercial components.[Footnote 22]

7. Tradeoff Analysis: 

Purpose: To ensure that system requirements alone do not drive the 
system solution.

Description: Tradeoff analysis relative to commercial product-based 
systems is the process for analyzing and understanding the tradeoffs 
among competing acquisition variables so as to produce informed 
acquisition decision making. It involves planning and executing 
acquisitions in a manner that recognizes four competing interests: 
defined system requirements, the architectural environment (current and 
future) in which the system needs to operate, acquisition cost and 
schedule constraints, and the availability of products in the 
commercial marketplace (current and future). This analysis should be 
performed early and continuously throughout an acquisition's life 
cycle.

Activity: (1) Investment decisions throughout a system's life cycle are 
based on tradeoffs among the availability of commercial products 
(current and future), the architectural environment in which the system 
is to operate (current and future), defined system requirements, and 
acquisition cost/schedule constraints.[Footnote 23]

8. Vendor and Product Research and Evaluation: 

Purpose: To ensure that vendor and product characteristics are 
understood before acquisition decisions are made.

Description: Vendor and product research and evaluation relative to 
commercial component-based systems is the process for obtaining 
reliable information about both the product being considered and the 
vendor offering the product. It involves taking additional steps beyond 
vendor representations, such as obtaining information about the 
vendor's history, obtaining information on the vendor's business 
strategy relative to evolution and support of the product, and 
evaluating copies of the product in a test environment.

Activities: (1) Commercial component and vendor options are researched, 
evaluated/tested, and understood, both early and continuously. (2) A 
set of evaluation criteria for selecting among commercial component 
options is established that includes both defined system requirements 
and vendor/commercial product characteristics (e.g., customer 
satisfaction with company and product line).

[End of section]

Appendix III: Comments from the Department of Defense: 

OFFICE OF THE ASSISTANT SECRETARY OF DEFENSE:
6000 DEFENSE PENTAGON: 
WASHINGTON, DC 20301-6000:

NETWORKS AND INFORMATION INTEGRATION:

July 1 2004:

Mr. Joel C. Willemssen:
Managing Director: 
Information Technology Issues: 
U.S. General Accounting Office:
441 G Street, N.W. 
Washington, D.C. 20548:

Dear Mr. Willemssen:

This is the Department of Defense (DoD) response to the GAO draft 
report (04-722), "INFORMATION TECHNOLOGY: DOD'S Acquisition Policies 
and Guidance Need To Incorporate Additional Best Practices and 
Controls," dated June 10, 2004 (GAO Code 310274).

We appreciate the opportunity to comment on the draft report and the 
time your staff afforded us during their preparation of the report. Our 
reply to each of the 14 recommendations is attached.

We recognize that the recommended best practices are based on guidance 
published by such organizations as the Software Engineering Institute, 
and we do not disagree that they are best practices. Our partial 
concurrences are based on the fact that in some cases, we disagree with 
the assertion that the proposed best practice does not exist in our 
current directives and guidance system. However, in all cases, we will 
consider including, or further emphasizing, the recommended best 
practice in the Department's directives and guidance system.

Our point of contact is Dave Mullins at 703-602-2585.

Sincerely,

Signed by: 

John R. Landon: 
Principal Director Deputy Assistant Secretary of Defense (C3, Space & 
IT Programs):

DoD Comments to GAO draft report (04-722), "INFORMATION TECHNOLOGY: 
DOD'S Acquisition Policies and Guidance Need To Incorporate Additional 
Best Practices and Controls," dated June 10, 2004 (GAO Code 310274).

RECOMMENDATION 1: The GAO recommended that the Secretary of Defense 
direct the Under Secretary for AT&L, in collaboration with the 
Assistant Secretary for NII and the Director, OT&E, to develop and 
implement an explicit plan for incorporating into the 5000 series the 
best practices and associated activities currently missing from the 
series. (p. 22/GAO Draft Report):

DOD RESPONSE: Nonconcur. The Department believes this recommendation 
and Recommendation 2 below are inappropriate. We agree that the best 
practices with which we concur should be incorporated into the 5000 
series or in related DoD policy or guidance documents, but do not agree 
that a detailed plan with resources and milestones is needed.

Based on discussions with GAO staff, we understand that the intent of 
this recommendation is for the Department to have a written program 
plan for completing the DoD Acquisition Guidebook that is now under 
development. We expect the Guidebook to be completed this summer. About 
80 to 90 percent of it has been through an initial review throughout 
the Department. The remaining sections will undergo such a review 
within the next few weeks. After that, the entire document will be sent 
out for a final review, and the primary authors will devote a few days 
at an off site to make any final changes before final approval and 
publication. The planned date for completion of these activities is not 
later than September 30, 2004.

RECOMMENDATION 2: The GAO recommended that the Secretary of Defense 
direct the Under Secretary for AT&L, in collaboration with the 
Assistant Secretary for NII and the Director, OT&E, to develop and 
implement an explicit plan that specify tasks to be performed, 
resources needed and assigned, and milestones for completing tasks, and 
that progress against the plan be tracked and reported as appropriate. 
(p. 22/GAO Draft Report):

DOD RESPONSE: Nonconcur. See above for explanation.

RECOMMENDATION 3: The GAO recommended that the Secretary of Defense 
direct the Under Secretary for AT&L, in collaboration with the 
Assistant Secretary for NII and the Director, OT&E, to implement a 
specific plan that, at a minimum provides for incorporating product 
line requirements-rather than just the requirements for the system 
being acquired-are an explicit consideration in each acquisition. (p. 
22/GAO Draft Report):

DOD RESPONSE: Partially concur. We find this recommendation somewhat 
confusing, but we believe it relates to considering the reuse of 
products that may already have been developed or acquired within a 
particular functional area or domain. The Enterprise Integration (EI) 
Toolkit, which was developed and is maintained by the Office Deputy 
Under Secretary of Defense for Logistics and Materiel Readiness, 
strongly endorses the reuse of reports, interfaces, conversions, and 
extensions (RICE) that have been built or acquired by other programs. 
In fact, an initial operating capability for a repository of custom 
software components that help adapt commercial components for defense 
use and reuse is available. It can be accessed via the Reports, 
Interfaces, Conversions, Extensions (RICE) Repository in the El Toolkit 
at www.eitoolkit.com. We will research the Software Enterprise 
Institute document that advocates this best practice to decide whether 
it is a practice we want to endorse in either the DoD 5000 series or 
the Guidebook.

RECOMMENDATION 4: The GAO recommended that the Secretary of Defense 
direct the Under Secretary for AT&L, in collaboration with the 
Assistant Secretary for NII and the Director, OT&E, to implement a 
specific plan that, at a minimum communicates acquisition management 
activities to all stakeholders. (p. 22/GAO Draft Report):

DOD RESPONSE: Partially concur. We find this to be another confusing 
recommendation. The primary purposes of the DoD 5000 series and the 
Acquisition Guidebook are to communicate acquisition management 
activities to all stakeholders. The confusion may stem from the fact 
that the report appears to use the terms "acquisition management" and 
"project management" interchangeably. The DoD 5000 series directs 
program managers and acquisition officials to conduct acquisition and 
acquisition oversight through the integrated product team (IPT) 
process. The essence of the IPT process is that acquisition and 
oversight are conducted in a collaborative manner, thus facilitating 
maximum communication among all stakeholders. In light of the above, 
while we agree that communication of acquisition management activities 
to all stakeholders is a best practice, we believe it already exists in 
our directives system and our practices.

RECOMMENDATION 5: The GAO recommended that the Secretary of Defense 
direct the Under Secretary for AT&L, in collaboration with the 
Assistant Secretary for NII and the Director, OT&E to implement a 
specific plan that, at a minimum includes acquisition reviews that 
include the status of identified risks. (p. 22/GAO Draft Report):

DOD RESPONSE: Partially concur. We agree that reviewing the status of 
identified risk is a best practice but disagree that it needs to be 
added to the DoD 5000 series or the Guidebook. DoD Instruction 5000.2 
has many references to various types of risks and the management or 
mitigation of those risks. One of the primary purposes of the 
acquisition phases in the DoD Acquisition Framework is to reduce risk. 
For example, DoDI 5000.2 states that one of the entrance criteria for 
the Systems Development and Demonstration phase is "the management and 
mitigation of technology risk." We will review the 5000 series and the 
Guidebook to detennine if there is a need to further emphasize this 
best practice.

RECOMMENDATION 6: The GAO recommended that the Secretary of Defense 
direct the Under Secretary for AT&L, in collaboration with the 
Assistant Secretary for Nil and the Director, OT&E, to implement a 
specific plan where at a minimum, modification of commercial components 
is discouraged and allowed only if justified by a thorough analysis of 
life-cycle costs and benefits. (p. 22/GAO Draft Report):

DOD RESPONSE: Concur. The current draft of the DoD Acquisition 
Guidebook now under development discourages the modification of 
commercial components. Additional changes to acquisition policy that 
emphasize this policy will be considered.

RECOMMENDATION 7: The GAO recommended that the Secretary of Defense 
direct the Under Secretary for AT&L, in collaboration with the 
Assistant Secretary for NII and the Director, OT&E to implement a 
specific plan where at a minimum, provide modification or upgrades to 
deployed versions of system components are centrally controlled, and 
unilateral user release changes are precluded. (p. 22/GAO Draft 
Report):

DOD RESPONSE: Concur. This best practice will be added to the 
Acquisition Guidebook.

RECOMMENDATION 8: The GAO recommended that the Secretary of Defense 
direct the Under Secretary for AT&L, in collaboration with the 
Assistant Secretary for NII and the Director, OT&E, to implement a 
specific plan where, acquisition decisions about commercial components 
are based on deliberate and thorough research, analysis, and 
evaluation of the components' interdependencies. (p. 22/GAO Draft 
Report):

DOD RESPONSE: Partially concur. The research, analysis and evaluation 
of components' interdependencies is considered in the conduct of the 
Joint Capabilities Integration and Development System (JCIDS) process 
and analyzed in detail in a key acquisition system deliverable; i.e., 
the Information Support Plan. We will emphasize these analyses in the 
Acquisition Guidebook.

RECOMMENDATION 9: The GAO recommended that the Secretary of Defense 
direct the Under Secretary for AT&L, in collaboration with the 
Assistant Secretary for NII and the Director, OT&E to implement a 
specific plan that, at a minimum provide for incorporating: Acquisition 
plans provide for preparing users for the impact that the business 
processes embedded in the commercial components will have on their 
respective roles and responsibilities. (p. 22/GAO Draft Report):

DOD RESPONSE: Partially concur. We agree that change management 
activities such as those described in this recommendation are a best 
practice but disagree that they need to be added to the DoD 5000 
series. The El Toolkit mentioned previously contains a Change 
Management roadmap that addresses organization change, readiness and 
preparing the users. It also includes several samples of communications 
to users, training aides and actual "getting familiar with your new 
job" type of examples from actual programs. The El Toolkit is 
referenced in DoDI 5000.2.

RECOMMENDATION 10: The GAO recommended that the Secretary of Defense 
direct the Under Secretary for AT&L, in collaboration with the 
Assistant Secretary for NII and the Director, OT&E to implement a 
specific plan where, at a minimum changes affecting how users will be 
expected to use the system to execute their jobs are actively managed. 
(p. 22/GAO Draft Report):

DOD RESPONSE: Partially concur. Same explanation as for the previous 
recommendation.

RECOMMENDATION 11: The GAO recommended that the Secretary of Defense 
direct the Under Secretary for AT&L, in collaboration with the 
Assistant Secretary for NII and the Director, OT&E, to implement a 
specific plan where, systems integration contractors are explicitly 
evaluated on their ability to implement commercial components. (p. 22/
GAO Draft Report):

DOD RESPONSE: Partially concur. The FAR and DFARS address the 
evaluation of past performance, and the Department has published a 
guide, titled A Guide to the Collection and Use of Past Performance 
Information. In addition the FAR and DFARS prescribe some evaluation 
factors while giving the contracting officer wide discretion in 
determining other appropriate factors. The Department strives to state 
its requirements in terms of the capabilities required; and commercial 
items are only one means for providing those required capabilities. 
However, acquiring commercial items is not an end in itself. Making the 
ability to implement commercial components a mandatory evaluation 
factor might have the unintended result of selecting a commercial 
component over a better alternative. In light of the above, we believe 
it is best to leave the selection of evaluation factors to the source 
selection authority, who is the person best acquainted with the 
particular circumstances.

Notwithstanding the above view, the Department has taken steps to 
ensure that we have the opportunity to select from a group of qualified 
contractors. The DoD Enterprise Software Initiative recently 
established agreements with five leading systems integration firms to 
provide commercial software integration services on a firm-fixed price 
basis. These Blanket Purchase Agreements, which are available on the 
GSA Schedule, encourage performance-based contracting and identify 
particular performance incentives for each firm. The five integrators 
were chosen based on an exhaustive 18-month market research and 
evaluation period, using industry best practices guides - and 
explicitly proven abilities to implement commercial software 
solutions. Major DoD IT Program Managers are highly encouraged to use 
these agreements.

RECOMMENDATION 12: The GAO recommended that the Secretary of Defense 
direct the Under Secretary for AT&L, in collaboration with the 
Assistant Secretary for NII and the Director, OT&E, to implement a 
specific plan that, at a minimum provides for incorporating: Investment 
decisions throughout a system's life cycle are based on a continuous 
set of tradeoffs among capabilities available in commercial components 
(current and future), the architectural environment in which the system 
is to operate, defined system requirements, and existing cost/schedule 
constraints. (p. 22/GAO Draft Report):

DOD RESPONSE: Partially concur. The JCIDS process described in Chairman 
of the Joint Chiefs of Staff Instruction (CJCSI) 3170.0113 and the DoD 
5000 series already require such analyses. CJCSI 3170.011) requires 
that three analyses be conducted before a needed capability is 
permitted to enter the technology development phase. These are a 
functional area analysis, functional needs analysis and a functional 
solutions analysis (FSA). These analyses and similar analyses required 
by DoDI 5000.2 at subsequent investment decision points, require 
tradeoffs among various available capabilities (including but not 
limited to those available in commercial components) and consideration 
of the architectural environment in which the system is to operate, 
defined system requirements, and existing cost/schedule constraints. 
For example, the FSA required at Milestone A is an analysis of 
available alternatives. The Capabilities Development Document (CDD) 
required at Milestone B addresses the architectural environment in 
which the system is to operate and describes the system requirements. 
The other Milestone B information requirements required by DoDI 5000.2, 
such as the Economic Analysis and the Acquisition Program Baseline 
address cost and schedule constraints. Tradeoff analyses are integral 
to systems engineering, which is required by the DoD 5000 series. At 
the direction of the Defense Acquisition Executive, more detailed 
guidance on systems engineering will be added to the 5000 series and 
the Guidebook. This recommendation will be considered when we develop 
that guidance.

RECOMMENDATION 13: The GAO recommended that the Secretary of Defense 
direct the Under Secretary for AT&L, in collaboration with the 
Assistant Secretary for NII and the Director, OT&E, to implement a 
specific plan where, evaluation criteria are established for selecting 
among commercial component options that include both defined system 
requirements and vendor/commercial products characteristics. (p. 22/
GAO Draft Report):

DOD RESPONSE: Partially concur. As stated in our response to 
Recommendation 11, we are reluctant to limit the contracting officer's 
discretion in choosing evaluation criteria. We are concerned that the 
same unintended consequence cited in the response to Recommendation 11 
could result from requiring the evaluation of commercial component 
options. Moreover, as stated in the reply to Recommendation 13, each 
acquisition is required to develop a CDD that describes system 
requirements. However, we will further review the intent of this 
recommendation, the 5000 series and the Guidebook to determine if there 
is a need to further emphasize this best practice.

RECOMMENDATION 14: The GAO recommended that the Secretary of Defense 
direct the Under Secretary for AT&L, in collaboration with the 
Assistant Secretary for NII and the Director, OT&E, to implement a 
specific plan that, at a minimum includes a provision for measurement 
and verification of DOD's acquisition policies best practices. (p. 22/
GAO Draft Report):

DOD RESPONSE: Nonconcur. Auditing statutory and regulatory compliance 
is the central focus of the oversight process described in DoDI 5000.2. 
Review is accomplished in the context of the Working-level IPT process, 
and non-compliance is corrected at each Milestone decision point. In 
addition, the Department periodically reviews the entire acquisition 
process to ensure that it is achieving desired outcomes and that it 
continues to reflect sound business practice:

The following are GAO's comments on the Department of Defense's letter 
dated July 1, 2004.

GAO Comments: 

1. We disagree. While we have updated our report to reflect the 
additional information provided in DOD's comments on the status of its 
efforts and the associated milestone, the importance of revising and 
maintaining DOD's acquisition policies and guidance, and their 
incorporation of acquisition best practices, makes it essential to have 
an explicit plan. Among other things, a plan would highlight the 
resource constraints that this revision effort has been subject to, 
would allow measurement against defined milestones, and would allow 
disclosure of progress and impediments.

2. See comment 1.

3. We do not question DOD's statement that it has an initial repository 
of custom software components that can help adapt commercial components 
to defense use and reuse. However, this repository does not satisfy the 
product line requirements in this best practice and our recommendation. 
According to the Software Engineering Institute, the product line 
requirements best practice involves more than just reuse. Under the 
approach described in DOD's comments, reuse generally involves items, 
such as software modules or components, that developers are encouraged 
to use. However, the product line requirements best practice is not 
simply encouraging reuse of items in a repository. Rather, it is 
planned, enabled, and enforced reuse of such assets as requirements, 
models, and architectures that have been designed and optimized for use 
in multiple systems. In short, it is proactive rather than reactive 
reuse.

4. We have modified our recommendation to use the terminology 
"acquisition project management activities" instead of "acquisition 
management activities" to eliminate any confusion. Further, we do not 
question DOD's use of integrated product teams as a way to communicate 
information. Further, the point of our recommendation is that there 
needs to be an explicit recognition in policy and guidance of the type 
of information to be communicated and to whom. Incorporating this best 
practice is based on the need to ensure that communication vehicles, 
such as integrated product teams, are effective.

5. We agree that the 5000 series contains information on acquisition 
risk management, as we state in our report, and that one of the 
purposes of DOD's acquisition framework is to reduce risks. However, 
there is no provision in DOD's acquisition policy or guidelines to 
ensure that the status of identified risks are discussed at key 
decision points. For example, DOD policy states that a criterion for 
passing milestone A is "management and mitigation of technology risk." 
However, it does not provide for what is to be done to manage and 
mitigate risks, and it does not provide for reviewing risk status at 
milestones B or C.

6. While we agree that the toolkit provides relevant change management 
information, the toolkit is referenced only once in the 5000 series; 
and the reference is only in relation to one phase of the acquisition 
cycle (the technology development phase). Given the importance and 
relevance of this practice to the successful implementation of 
commercial component-based systems, our position, and thus the basis 
for our recommendation, is that the best practice's implementation 
would more likely occur if it was visible and recognized in all 
relevant stages of DOD's acquisition cycle.

7. See comment 6.

8. While the regulations and guidance that DOD cited and referenced in 
its acquisition policies discuss the use of information on contractors' 
past performance, they do not discuss evaluating systems integration 
contractors on their ability to implement commercial components, which 
is the point of the best practice. Further, DOD's objection to 
incorporating this best practice is not consistent with its own 
comments. Specifically, DOD commented that it has already taken steps, 
through its enterprise systems initiative, to establish blanket 
agreements with five contractors who were evaluated on, among other 
things, "explicitly proven abilities to implement commercial systems 
solutions." Additionally, while we appreciate DOD's concern that 
incorporation of this best practice can have unintended consequences, 
it is important to also recognize that our recommendation is not 
intended to restrict a contracting officer's options. Rather, it is 
intended to be one of the many factors considered in the source 
selection process when it is relevant.

9. We support the Defense Acquisition Executive's decision to add more 
specifics on systems engineering to the 5000 series, including 
provisions that address this best practice and recommendation.

10. See comment 8.

11. We disagree. While we do not question that statutory and regulatory 
compliance are referenced in DOD's integrated process team and 
milestone decision point processes, we do not believe that these 
reviews are adequately defined with respect to implementation of best 
practices because DOD's policy does not require that the practices' use 
be measured and verified. Rather, it leaves these reviews to the 
discretion of the program manager and investment decision authority. As 
we state in our report, not requiring that the use of best practices be 
measured and verified increases the chance that the practices will not 
be followed. Therefore, our position remains that DOD's policies do not 
provide effective controls for ensuring that best practices are 
appropriately followed.

[End of section]

Appendix IV: GAO Contact and Staff Acknowledgments: 

GAO contact: 

Carl L. Higginbotham, (404) 679-1824: 

Staff acknowledgments: 

In addition to the individual named above, key contributorsto this 
report included Nabajyoti Barkakati, Nancy Glover,Madhav Panwar, 
Morgan Walts, and Thomas Wright.

(310274): 

FOOTNOTES

[1] Donald E. Harter, Mayuram S. Krishnan, and Sandra A. Slaughter, 
"Effects of Process Maturity on Quality, Cycle Time, and Effort in 
Software Product Development," Management Science, vol. 46, no. 4, 
2000; and Bradford K. Clark, "Quantifying the Effects of Process 
Improvement on Effort," IEEE Software (November/December 2000).

[2] U.S. General Accounting Office, Defense Acquisitions: Stronger 
Management Practices Are Needed to Improve DOD's Software-Intensive 
Weapon Acquisitions, GAO-04-393 (Washington, D.C.: Mar. 1, 2004).

[3] U.S. General Accounting Office, DOD Information Technology: 
Software and Systems Process Improvement Programs Vary in Use of Best 
Practices, GAO-01-116 (Washington, D.C.: Mar. 30, 2001).

[4] Bob Stump National Defense Authorization Act for 2003 (Pub.L. No. 
107-314).

[5] Collectively, these oversight policy and guidance documents cover 
most--but not all--major acquisitions. The Secretary of Defense has 
delegated authority to the Missile Defense Agency and to the National 
Security Space Team to develop separate guidance for missile defense 
and space systems, respectively.

[6] DOD policy establishes a decision authority, called a milestone 
decision authority, as the designated individual who has overall 
responsibility for an IT investment. This person has the authority to 
approve an investment's move from one phase to the next phase of the 
acquisition process and is responsible for reporting cost, schedule, 
and performance results to higher authorities, including the Congress. 

[7] U.S. General Accounting Office, Information Technology: 
Inconsistent Software Acquisition Processes at the Defense Logistics 
Agency Increase Project Risks, GAO-02-9 (Washington, D.C.: Jan. 10, 
2002); Information Technology: Greater Use of Best Practices Can Reduce 
Risks in Acquiring Defense Health Care System, GAO-02-345 (Washington, 
D.C.: Sept. 26, 2002); and DOD Business Systems Modernization: 
Continued Investment in Key Accounting Systems Needs to be Justified, 
GAO-03-465 (Washington, D.C.: Mar. 28, 2003). 

[8] We defined "fully incorporate" to mean the 5000 series addressed 
all of the practice's activities; "partially incorporate" to mean it 
addressed some, but not all, of the activities; and "not incorporate" 
to mean it did not address any of the activities. 

[9] The Federal Managers' Financial Integrity Act of 1982 (Pub.L. No. 
97-255); Government Performance and Results Act of 1993 (Pub.L. No. 
103-62); Office of Management and Budget Circular A-123, June 21, 1995. 


[10] The Defense Acquisition Policy Working Group is the standing DOD 
acquisition policy working group that revised the 5000 series. 

[11] U.S. General Accounting Office, Defense Acquisitions: DOD's 
Revised Policy Emphasizes Best Practices, but More Controls Are Needed, 
GAO-04-53 (Washington, D.C.: Nov. 10, 2003).

[12] For example, the Software Engineering Institute is a federally 
funded research and development center operated by Carnegie Mellon 
University and sponsored by DOD. The Software Engineering Institute's 
objective is to provide leadership in software engineering and in the 
transition of new software engineering technology into practice. The 
Aerospace Corporation is a private, nonprofit organization that 
operates a federally funded research and development center for DOD 
that focuses on the government's need to develop space-related hardware 
and software. 

[13] Software Engineering Institute, Software Acquisition Capability 
Maturity Modelģ version 1.03, CMU/SEI-2002-TR-010 (Pittsburgh, PA: 
March 2002).

[14] U.S. General Accounting Office, Standards for Internal Control in 
the Federal Government, AIMD-00-21.3.1 (Washington, D.C.: November 
1999).

[15] Richard J. Adams, Suellen Eslinger, Karen L. Owens, and Mary A. 
Rich, Reducing Risk in the Acquisition of Software-Intensive Systems: 
Best Practices from the Space System Domain (Los Angeles, CA: 2003). 

[16] Adams and Eslinger, "COTS-Based Systems: Lessons Learned from 
Experiences with COTS Software Use on Space Systems." (Paper presented 
to the Southern California SPIN: Oct. 6, 2003.)

[17] Software Engineering Institute, Real-Time Systems Engineering: 
Lessons Learned from Independent Technical Assessments, CMU/SEI-2001-
TN-004 (Pittsburgh, PA: June 2001); and Adams, Eslinger, Owens, and 
Rich, Reducing Risk in the Acquisition of Software Intensive-Systems: 
Best Practices from the Space System Domain.

[18] Adams and Eslinger, "COTS-Based Systems: Lessons Learned from 
Experiences with COTS Software Use on Space Systems." (Paper presented 
to the Southern California SPIN: Oct. 6, 2003.)

[19] Donald J. Reifer, Victor R. Basili, Barry W. Boehm, Betsy Clark, 
"COTS-Based Systems--Twelve Lessons Learned about Maintenance." 
(Presentation, 3RD International Conference on COTS-Based Software 
Systems, Redondo Beach, CA, Feb. 4, 2004.)

[20] Tricia Oberndorf, Lisa Brownsword, and Carol A. Sledge, Ph.D., An 
Activity Framework for COTS-Based Systems, Technical Report CMU/SEI-
2000-TR-010 (Pittsburgh, Pa.: Software Engineering Institute, Carnegie 
Mellon University, October 2000).

[21] Suzanne Garcia, John Roberts, and Len Estrin, "Managed Technology 
Adoption Risk." (Presentation, 3RD International Conference on COTS-
Based Software Systems, Redondo Beach, CA, Feb. 2, 2004).

[22] Adams and Eslinger, "COTS-Based Systems: Lessons Learned from 
Experiences with COTS Software Use on Space Systems." (Paper presented 
to the Southern California SPIN: Oct. 6, 2003.)

[23] Software Engineering Institute, Evolutionary Process for 
Integrating COTS-Based Systems (EPIC): An Overview, CMU/SEI-2002-TR-
009 (Pittsburgh, PA: July 2002).

GAO's Mission: 

The Government Accountability Office, the investigative arm of 
Congress, exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability 
of the federal government for the American people. GAO examines the use 
of public funds; evaluates federal programs and policies; and provides 
analyses, recommendations, and other assistance to help Congress make 
informed oversight, policy, and funding decisions. GAO's commitment to 
good government is reflected in its core values of accountability, 
integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains 
abstracts and full-text files of current reports and testimony and an 
expanding archive of older products. The Web site features a search 
engine to help you locate documents using key words and phrases. You 
can print these documents in their entirety, including charts and other 
graphics.

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as "Today's Reports," on its 
Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order 
GAO Products" heading.

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to: 

U.S. Government Accountability Office

441 G Street NW, Room LM

Washington, D.C. 20548: 

To order by Phone: 

	

Voice: (202) 512-6000: 

TDD: (202) 512-2537: 

Fax: (202) 512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470: 

Public Affairs: 

Jeff Nelligan, managing director,

NelliganJ@gao.gov

(202) 512-4800

U.S. Government Accountability Office,

441 G Street NW, Room 7149

Washington, D.C. 20548: