This is the accessible text file for GAO report number GAO-04-386SP 
entitled 'Best Practices: Using A Knowledge-Based Approach To Improve 
Weapon Acquisition' which was released on January 30, 2004.

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 


Best Practices: 

Using A Knowledge-Based Approach To Improve Weapon Acquisition: 

This booklet describes the knowledge-based approach successful 
organizations generally follow to deliver sophisticated products in 
less time and at lower costs. 

The Department of Defense (DOD) is on the threshold of several major 
investments in acquisition programs that are likely to dominate budget 
and doctrinal debates well into the next decade. Over the next 5 years 
alone, DOD’s overall investments are expected to average $150 billion a 
year as DOD works to keep legacy systems as well as transform our 
national defense capabilities for the future. To meet this challenge, 
it is essential that sound foundations for investments in systems be 
laid now so that the resulting programs can be executed within 
estimates of available resources. 

At the request of the Congress, we have been examining ways DOD can
optimize its investment in weapon systems, drawing on lessons learned 
from the best, mostly commercial, product development efforts. Leading
commercial firms we have studied have developed increasingly 
sophisticated products in less time and at lower cost. Key to their 
success is their knowledge-based approach to the acquisition of new 
products. A knowledge-based approach is supported by incentives that 
encourage realism and candor. 

This booklet highlights the results of our work to date. We continue to 
explore additional facets of the acquisition process to identify best 
practices. More details on our work can be found in the reports cited 
in this brochure. 

GAO’s best practices work is done under the direction of Katherine V.
Schinasi. For more on this work, go to [hyperlink,]. If you have questions or would like 
to discuss our reviews, please contact Ms. Schinasi at (202) 512-4841 

Signed by: 

Jack L. Brock Jr.
Managing Director: 
Acquisition and Sourcing Management: 

[End of section] 

A Knowledge-Based Approach Puts Acquisition Programs In A Better 
Position To Succeed: 

Leading commercial firms expect that their managers will deliver high 
quality products on time and within budgets. Doing otherwise could 
result in losing a customer in the short term and losing the company in 
the long term. Thus, these firms have adopted practices that put their 
individual program in a good position to succeed in meeting these 
expectations on individual products. Collectively, these practices 
ensure that a high level of knowledge exists about critical facets of 
the product at key junctures during its development and is used to 
deliver capability as promised. DOD has recognized the need to adopt
ways of doing business that enable it to achieve similar results for 
its weapons acquisitions. 

Our reviews have shown that there are three critical junctures at which 
firms must have knowledge to make large investment decisions. First, 
before a product development is started, a match must be made between 
the customers’ needs and the available resources—technical and 
engineering knowledge, time, and funding. Second, a product’s design 
must demonstrate its ability to meet performance requirements and be 
stable about midway through development. Third, the developer must show 
that the product can be manufactured within cost, schedule, and quality 
targets and is demonstrated to be reliable before production begins. If 
the knowledge attained at each juncture does not confirm the business 
case on which the acquisition was originally justified, the program 
does not go forward. 

In applying the knowledge-based approach, the most-leveraged decision 
point of the three junctures is matching the customer’s needs with the 
developer’s resources. This initial decision sets the stage for the 
eventual outcome—desirable or problematic. The match is ultimately 
achieved in every development program, but in successful development 
programs, it occurs before product development. In successful programs, 
negotiations and tradeoffs occur before product development is started 
to ensure that a match exists between customer expectations and 
developer resources. Leading firms thus make an important distinction 
between technology development and product development. Technologies 
that are not mature continue to be developed in the technology 
base—they are not included in a product development. 

With achievable requirements and commitment of sufficient investment to
complete the development, programs are better able to deliver products 
at cost and on schedule. When knowledge lags, a number of risks are 
introduced into the acquisition process that can result in cost 
overruns, schedule delays, and inconsistent product performance. 

An approach that enables organizations to achieve a match between needs 
and resources is evolutionary product development. Under this approach, 
basic requirements are achieved first, with additional capabilities 
planned for future generations of the product. Because product 
development is incremental, achieving knowledge is more manageable. 
Commercial companies have found that trying to capture the knowledge 
needed to stabilize the design of a product with considerable new 
technical content is an unwieldy task—especially if the goal is to 
reduce development cycle times and get the product to the marketplace 
as quickly as possible. With evolutionary development, a product’s 
design uses only components and subsystems whose reliability has been 
proven through past use or testing. 

Knowledge Points at a Glance: 

The knowledge-based process followed by leading organizations is 
highlighted and further discussed below. 

Figure: Knowledge Points at a Glance: 

[See PDF for image] 

This figure is an illustration of knowledge points, as follows: 

Technology Development: Leads to: 
Product Development: Program launch and Knowledge Point 1 occur at the 
beginning of product development; Knowledge point 2 occurs in the mid-
point of product development, which leads to: 
Production: Knowledge point 3 occurs at the beginning of production. 

* Knowledge point 1: Resources and needs match. Knowledge point 1
occurs when a sound business case is made for the product—that is, a
match is made between the customer’s requirements and the product
developer’s available resources in terms of knowledge, time, and money. 
To determine their available resources, successful companies rely on 
current and valid information from predecessor programs, new 
technologies that have demonstrated a high level of maturity, system 
engineering data, and experienced people. Successful companies also 
communicate extensively with customers to match their wants and needs 
with the firm’s available resources and with its ability to manufacture 
an appropriate product. 

* Knowledge point 2: Product design is stable. Knowledge point 2 occurs
when a company determines that a product’s design is stable—that is, it
will meet customer requirements and cost and schedule targets. A best 
practice is to achieve design stability at the product’s critical 
design review, usually held midway through development. In DOD, the 
critical design review occurs when the first phase of product 
development—product integration—has been completed and the second 
phase—product demonstration—is about to begin. Completion of at least 
90 percent of engineering drawings at the critical design review 
provides tangible evidence that the design is stable. 

* Knowledge point 3: Production processes are mature. This level of
knowledge is achieved when it has been demonstrated that the product can
be manufactured within cost, schedule, and quality targets. A best 
practice is to ensure that all key manufacturing processes are in 
statistical control—that is, they are repeatable, sustainable, and 
capable of consistently producing parts within the product’s quality 
tolerances and standards—at the start of production. It is important 
that the product’s reliability be demonstrated before production 
begins, as investments can increase significantly if defective parts 
need to be repaired or reworked. 

[End of figure] 

Measuring Success at Key Knowledge Points: 
The organizations we have studied apply a variety of tools and measures 
to gauge the level of knowledge that they have attained before making 
major investment decisions. 

* Measuring technology readiness: 
Many programs now use an analytical tool—technology readiness levels
(TRL)—that can assess the maturity level of technology as well as the 
risk that maturity poses if the technology is included in a product 
development. The experiences of the DOD and commercial technology 
development cases we reviewed indicate that demonstrating a high level 
of maturity before allowing new technologies into product development 
programs puts those programs in a better position to succeed. Simply 
put, the more mature technology is at the start of the program, the 
more likely the program will succeed in meeting its objectives. 

There are nine TRLs, each denoting a level of demonstrated performance,
beginning with concept—the lowest level of readiness—to application,
where the technology has been “flight proven” under mission conditions.
The higher the TRL, the smaller the gap between the technology’s 
maturity and the product’s requirements, and the lower the risk of 
including the technology in the product’s development. Our best 
practices work has shown that TRL 7—demonstration of a technology in an 
operational environment—is the level of technology needed to minimize 
risks when launching an acquisition program. 

Using TRLs to assess a technology’s maturity helps decisionmakers make
informed choices about product development. If a technology lacks 
maturity, decisionmakers can choose to delay product development until
the technology has matured sufficiently or lower the product’s 
requirements so that a less advanced but proven technology can be used.
The more a technology has been proven, the more likely the product is to
meet its objectives and the more likely the company can minimize
unexpected problems and avoid potential schedule delays and cost

* Measuring design readiness: 
The organizations we studied also understood the importance of having
disciplined design reviews and getting agreement from the stakeholders
that the product’s design had been demonstrated to meet requirements
before beginning initial manufacturing. Each organization had a design
review process that began at the component level, continued through the
subsystem level, and culminated with a critical design review of the
integrated system to determine if the product was ready to progress to 
the next phase of development. 

In addition to design engineers, a cross-functional team of 
stakeholders in the process included key suppliers, manufacturing 
representatives, and service and maintenance representatives. From past 
experience, leading organizations have discovered that cross-functional 
teams provide a complete perspective of the product. 

At critical design review, the leading organizations we studied 
generally require that at least 90 percent of the engineering drawings 
be completed. They consider engineering drawings to be a good measure 
of the demonstrated stability of a product’s design because the 
drawings represent the language used by engineers to communicate to the 
manufacturers the details of a new product design—what it looks like, 
how its components interface, how it functions, how to build it, and 
what critical materials and processes are required to fabricate and 
test it. 

Other Important Tools and Practices: 

Our reports have identified other tools and practices that enable 
successful implementation of the approach. These include: 

* Systems engineering for identifying gaps between requirements and
resources so that they can be reconciled through effective trade-offs 
before product development. 

* Employment of integrated product teams to bring together in a single
organization the different functions needed to design and manufacture a
product, such as engineering, finance, test and evaluation, and

* Supplier management approaches that optimize relationships with both
contractors and subcontractors. 

* Using earned value management techniques to track a projects progress
and assess its ability to meet cost and scheduling goals. 

* The use of a variety of testing and evaluation tools and techniques to
validate a product’s performance early and throughout development. 

* The use of targeted, hands-on methods to ensure that program offices 
are trained on key practices. 

* Collection of statistical process control data to ensure manufacturing
processes are consistently producing parts within quality standards. 

* Considering reasonable operating and support costs and the readiness 
or availability of equipment as requirements equal in importance to 
other performance characteristics. 

Coupling the knowledge-based approach with sound tools and techniques 
for oversight, strengthening the workforce, and ensuring the right 
knowledge is attained at the right times, increases an organization’s 
potential to meet cost, scheduling, and performance targets. It is 
still essential, however, that the right incentives and resources be in 
place to encourage program managers to employ such measures. 

Following A Knowledge-Based Acquisition Process Is Critical For Weapon 

The majority of weapon system acquisitions we have reviewed over the 
past several decades experienced problems during acquisition that drove 
up costs and schedules and increased technical risks. Many programs 
have been restructured by DOD in the face of delays and cost growth; a 
few have been canceled. We have found that these problems are largely 
rooted in the failure to match customer’s needs with the developer’s 
resources—technical knowledge, timing, and funding--when starting 
product development. In other words, commitments were made to 
delivering capability without knowing whether technologies being 
pursued could really work as intended. Time and costs were consistently 
underestimated. Problems that surfaced early cascaded throughout 
development and magnified the risks facing the program. 

When Knowledge Is Not Attained At Key Junctures: 

* Launching an acquisition program before requirements and available 
resources are matched can result in a product that fails to perform as 
expected, costs more, or takes longer to develop. 

* Failure to ensure design stability about halfway through product 
development can result in design changes that are more costly to 
correct later in development or after the product is fielded. 

* Entering production before manufacturing processes are under control 
can result in product defects that require additional resources to 
rework or scrap—a costly and inefficient practice. 

On a number of programs, DOD has shown that it can manage its weapon
system acquisition process to ensure important knowledge about 
technology and requirements, design, and manufacturing is captured and 
used to make informed and timely decisions before committing to 
substantial development and production investments. Moreover, over the 
past few years, DOD has made constructive changes to its acquisition 
policy to embrace best practices—especially those related to technology 
maturity and separating technology development from product 
development. If faithfully implemented, DOD’s policies of evolutionary 
acquisition and phased requirements will make it easier to match 
resources with needs before starting a new product development. Funding 
pressures and the need to modernize systems across the Department will 
continue to make it paramount for DOD to adopt practices that can save 
money and deliver new capabilities quicker. 

However, a number of incentives—many tied to funding—continue to 
undermine DOD’s ability to achieve a match between needs and resources
at the onset of weapon acquisition programs. Unlike the commercial world
where the focus is on delivering a product to market, DOD’s system 
focuses on competing for resources. In the competition for funding, 
managers are encouraged to launch product developments before 
technologies are mature. Because a proven way to win support for a new 
weapons acquisition program is to promote unprecedented performance 
features and design characteristics, managers have implicit incentives 
to rely on immature technologies. Moreover, because funding is 
competitive and DOD’s forecasts of cost, schedule, and performance are 
largely based on immature technologies and other unknowns, estimates 
tend to be squeezed into insufficient profiles of available funding. 
Other factors, such as short tenures and career pressures, discourage 
program managers from saying no to requirements that are later 
discovered to have been unreasonable. Thus, to meet the investment 
challenge of modernizing its forces, DOD will need not only to 
implement policies that embrace evolutionary, knowledge-based 
acquisitions, it will need to instill incentives that encourage realism
and candor in the acquisition process and sustain its commitment to
improving business practices. 

[End of section] 

Best Practices Reports: 

These reports can be found on the GAO Web site at [hyperlink,]. 

Defense Acquisitions: Assessments of Major Weapon Programs, GAO-03-476,
Washington, D.C.: May 2003. 

Best Practices: Setting Requirements Differently Could Reduce Weapon
Systems’ Total Ownership Costs. GAO-03-57. Washington, D.C.: February 
11, 2003. 

Best Practices: Capturing Design and Manufacturing Knowledge Early
Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: July 15, 

Defense Acquisitions: DOD Faces Challenges in Implementing Best 
Practices. GAO-02-469T. Washington, D.C.: February 27, 2002. 

Best Practices: DOD Teaming Practices Not Achieving Potential Results. 
GAO-01-510, Washington, D.C.: April 2001. 

Best Practices: Better Matching of Needs and Resources Will Lead to 
Better Weapon System Outcomes. GAO-01-288. Washington, D.C.: March 8, 

Best Practices: A More Constructive Test Approach Is Key to Better 
Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.: July 31, 

Defense Acquisitions: Employing Best Practices Can Shape Better Weapon
System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.: April 26, 2000. 

Best Practices: DOD Training Can Do More to Help Weapon System Programs
Implement Best Practices. GAO/NSIAD-99-206. Washington, D.C.: August 16,

Best Practices: Better Management of Technology Development Can Improve
Weapon System Outcomes. GAO/NSIAD-99-162. Washington, D.C.: July 30, 

Best Practices: DOD Can Help Suppliers Contribute More to Weapon System
Programs. GAO/NSIAD-98-87. Washington, D.C.: March 17, 1998. 

Best Practices: Successful Application to Weapon Acquisition Requires
Changes in DOD’s Environment. GAO/NSIAD-98-56. Washington, D.C.:
February 24, 1998. 

Best Practices: Commercial Quality Assurance Practices Offer 
Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.: August 26,