This is the accessible text file for GAO report number GAO-06-883 
entitled 'Best Practices: Stronger Practices Needed to Improve DOD 
Technology Transition Processes' which was released on September 14, 
2006. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to Congressional Committees: 

United States Government Accountability Office: 

GAO: 

September 2006: 

Best Practices: 

Stronger Practices Needed to Improve DOD Technology Transition 
Processes: 

DOD Technology Transition: 

GAO-06-883: 

GAO Highlights: 

Highlights of GAO-06-883, a report to congressional committees 

Why GAO Did This Study: 

The Department of Defense (DOD) relies on its science and technology 
community to develop innovative technologies for weapon systems, 
spending $13 billion on basic, applied, and advanced technology 
research. Several GAO reports have addressed problems in transitioning 
technologies to the acquisition community. This report, which was 
prepared under the Comptroller General’s authority to conduct 
evaluations, compares DOD’s technology transition processes with 
commercial best practices. Specifically, GAO identifies technology 
transition techniques used by leading companies and assesses the extent 
to which DOD uses the techniques. 

What GAO Found: 

Leading commercial companies use three key techniques for successfully 
developing and transitioning technologies, with the basic premise being 
that technologies must be mature before transitioning to the product 
line side. 

* Strategic planning at the corporate level: Strategic planning 
precedes technology development so managers can gauge market needs, 
identify the most desirable technologies, and prioritize resources.
* Gated management reviews: A rigorous process is used to ensure a 
technology’s relevancy and feasibility and enlist product line 
commitment to use the technologies once the labs are finished maturing 
them.
* Corroborating tools: To secure commitment, technology transition 
agreements solidify and document specific cost, schedule, and 
performance metrics labs need to meet for transition to occur. 
Relationship managers address transition issues within the labs and 
product line teams and across both communities. Meaningful metrics 
gauge project progress and process effectiveness. 

Not only does DOD lack the breadth and depth of these techniques, the 
department routinely accepts high levels of technology risk at the 
start of major weapon acquisition programs. The acquisition community 
works with technologies before they are ready to be transitioned and 
takes on responsibility for technology development and product 
development concurrently, as shown in the following figure. A defined 
phase for technology transition is not evident. These shortcomings 
contribute significantly to DOD’s poor cost and schedule outcomes. 

Figure: Path That DOD Follows for Technology Development and Product 
Development: 

[See PDF for Image] 

Source: DOD (data); GAO (analysis and presentation). 

[End of Figure] 

A stark contrast exists between DOD’s and private industry’s 
environments for developing technology. The numerous examples of DOD 
programs that have incurred cost overruns, schedule delays, and reduced 
performance serve as reminders that inserting a few best practices and 
changing the mechanics of technology transition processes without 
changing the environment that determines incentives may not produce 
better outcomes. 

What GAO Recommends: 

GAO recommends that DOD strengthen its technology transition processes 
by developing a gated process with criteria to support funding 
decisions; expanding the use of transition agreements, relationship 
managers, and metrics; and setting aside funding for transition 
activities. DOD generally agreed with GAO’s recommendations with the 
exception of adopting process-oriented metrics and setting aside 
funding for transition. It cited ongoing initiatives it believes 
address several of the recommendations. GAO believes DOD’s actions to 
date are incomplete and all recommendations warrant further attention. 

[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-883]. 

To view the full product, including the scope and methodology, click on 
the link above. For more information, contact Michael J. Sullivan at 
(202) 512-4841 or sullivanm@gao.gov. 

[End of Section] 

Contents: 

Letter: 

Results in Brief: 

Background: 

Leading Companies Rely on Strategic Planning, a Gated Process, and 
Tools to Transition Needed Technologies: 

DOD Lacks Breadth and Depth of Techniques That Leading Companies Use to 
Effectively Transition Technologies: 

Differences in Environment and Incentives Contribute to Different 
Practices and Outcomes: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Objectives, Scope, and Methodology: 

Appendix II: Comments from the Department of Defense: 

Appendix III: DOD Research, Development, Technology, and Engineering 
Budget: 

Appendix IV: DOD Technology Readiness Levels: 

Related GAO Products: 

Tables: 

Table 1: DOD's Two Development Phases with Technology-Related 
Activities: 

Table 2: Metrics Used by Leading Companies to Assess Lab Projects and 
Processes: 

Figures: 

Figure 1: Fiscal Year 2006 Categories for DOD Research, Development, 
Test, and Evaluation Budget (dollars in billions): 

Figure 2: Average Program Research, Development, Test, and Evaluation 
Cost Growth from First Full Estimate (sample of 52 DOD weapon 
programs): 

Figure 3: General Flow of Process Leading Up to Technology Transition: 

Figure 4: Generalized Depiction of Deliverables and Funding under Gated 
Process: 

Figure 5: Linkage between Technology Development and Product 
Development: 

Figure 6: Notional Boeing Technology Maturity Scorecard for a 
Hypothetical Technology: 

Figure 7: Questions That Should Be Answered in Technology Transition 
Agreements: 

Figure 8: Communication Flow for Motorola's Three Levels of 
Relationship Managers: 

Figure 9: Path that DOD Routinely Follows for Technology Development 
and Product Development: 

Figure 10: Criteria for DOD Technology Development Phase: 

Figure 11: Accountability for Management and Funding of Technology: 

Figure 12: Excerpts from a Defense Science Board Task Force Report on 
the Manufacturing Technology Program: 

Figure 13: DARPA, a Success Story: 

Abbreviations: 

DARPA: Defense Advanced Research Projects Agency: 
DDRE: Director, Defense Research and Engineering: 
DOD: Department of Defense: 
RDT&E: research, development, test, and evaluation: 
S&T: science and technology: 
TRL: technology readiness level: 

United States Government Accountability Office: 
Washington, DC 20548: 

September 14, 2006: 

The Honorable John W. Warner: 
Chairman: 
The Honorable Carl Levin: 
Ranking Minority Member: 
Committee on Armed Services: 
United States Senate: 

The Honorable Duncan Hunter: 
Chairman: 
The Honorable Ike Skelton: 
Ranking Minority Member: 
Committee on Armed Services: 
House of Representatives: 

The Department of Defense (DOD) relies on the technological superiority 
of its weapon systems and armed forces to protect U.S. interests at 
home and abroad. The DOD science and technology (S&T) community is 
tasked with ensuring that technologies are mature when DOD's 
acquisition community takes over and integrates the technologies into 
weapon systems. In fiscal year 2006, DOD plans to spend approximately 
$13 billion[Footnote 1] in its science and technology efforts to 
develop technologies that are as innovative as stealth and global 
positioning were when they were first developed. In doing its work, the 
S&T community must strike a balance between meeting the short-term 
needs of today's warfighters and the long-term needs of future years' 
warfighters. Achieving this balance is made more challenging because of 
the need to keep pace with or exceed the pace of innovation and to 
counter technologies developed by potential U.S. adversaries. 

Although the United States has produced the best weapons in the world, 
its acquisition programs often incur cost overruns, schedule delays, 
and performance shortfalls that undermine DOD's buying power. This 
dilemma is due in part to DOD's difficulty transitioning technologies 
from a technology development environment to an acquisition program. 
Because the acquisition community frequently pulls technologies too 
early, it takes on the additional task of maturing the technologies--an 
activity that is the primary responsibility of technology developers-- 
at the start of an acquisition program.[Footnote 2] The start of a 
program ushers in a high-cost, delivery-oriented phase in which the 
acquisition community is supposed to be focused on integrating 
subsystems and working on system development and demonstration. DOD has 
continued to allow the acquisition community to take over this task 
before the S&T community considers the technologies ready for 
transition. Numerous GAO reports have addressed the problems of 
proceeding with immature technologies and have explained what leading 
commercial companies do to ward off such problems.[Footnote 3] 

This report examines DOD's technology transition processes through the 
prism of best practices. Specifically, our objectives are to (1) 
identify techniques that commercial companies use to transition mature 
technologies before the start of product development and (2) assess the 
extent to which DOD is using these techniques. We also describe how the 
environments at private companies and DOD differ with regard to 
technology development. The overall effort to hand off and integrate 
mature technologies is referred to in this report as technology 
transition. Also in this report, the private sector acquisition 
community is generally referred to as a product line or business unit. 
GAO has prepared this report under the Comptroller General's authority 
to conduct evaluations on his own initiative as part of a continued 
effort to assist Congress in overseeing DOD's technology and 
acquisition investments. 

In conducting our work, we interviewed lab and product line managers at 
four leading commercial companies--Boeing, IBM, Motorola, and 3M--to 
identify processes for successfully transitioning mature technologies. 
These companies develop a variety of products such as the 787 airplane, 
network servers, integrated radio communications and information 
solutions, and fuel cell technology for cars, buses, trains, homes, and 
businesses. We also conducted interviews and collected pertinent 
documents from each of DOD's military services and the Defense Advanced 
Research Projects Agency (DARPA), which is the central science and 
technology organization for DOD. We compared commercial and DOD 
practices to identify potential areas for improvement. Appendix I 
includes additional details about how we performed our review, which 
was conducted from June 2005 to July 2006 in accordance with generally 
accepted government auditing standards. 

Results in Brief: 

Successful transition in leading companies starts with strong strategic 
planning followed by a structured technology development process led by 
research labs and supported by tools that pave the way for a smooth 
handoff to the product line. Strategic planning is considered a 
precursor to transition and allows managers to identify market needs so 
the company can quickly adapt its technology portfolios to meet those 
needs. A gated technology development process continually tests for 
relevancy and feasibility of technologies and gauges the commitment of 
product line managers to accept them. Once a technology is ready to 
transition, management and funding responsibilities gradually shift 
from the lab to the product line. By the end of transition, but before 
product development starts, the technology is validated as mature 
enough for use in the intended product. Companies highlighted three 
tools to facilitate transition: technology transition agreements, 
relationship managers, and metrics. The agreements put in writing the 
technology and business-related expectations, such as specific cost, 
schedule, and performance characteristics that labs must demonstrate. 
The agreements also may require documenting manufacturing costs or 
specifying whether certain lab scientists will be loaned to the product 
line to provide continuity in technical knowledge. Relationship 
managers communicate across the labs and product lines to address 
transition issues. Last, metrics gauge the effectiveness of the 
technologies and the process itself. These tools require the active 
involvement of the labs and product lines to ensure successful 
transition of technologies to new products. 

Over the past several years, DOD has taken steps to improve its 
transition processes, but it lacks many of the techniques that are 
hallmarks of leading companies' ability to transition technology 
smoothly onto new products. From a strategic perspective, the 
department lacks strong influence at the corporate level to guide the 
department's technology investments. In addition, DOD does not use a 
gated process with criteria that would allow lab and program managers 
to know when a technology is ready to transition. Consequently, 
technologies are often not ready when needed and acquisition programs 
pull the technologies into their programs too early, leading to 
inefficiency during product development and cost and schedule 
increases. We found that DOD has taken some positive steps to aid 
technology transition. They hold promise, but must be accepted, 
improved, and replicated significantly more than currently to have a 
positive impact. For example, each of the military services has 
established boards to select and oversee some of their technology 
projects and has elevated the importance of meaningful metrics. They 
are also using technology transition agreements. However, use of these 
agreements thus far has been low. With regard to improving 
communication, DARPA is using relationship managers to address 
transition issues. And the Office of the Secretary of Defense has 
initiated a number of new programs, including the Joint Capability 
Technology Demonstration program, which requires the S&T and 
acquisition communities to work more collaboratively earlier in the 
acquisition process. 

The environment and incentives of private world-class companies differ 
dramatically from DOD's. Despite these differences, the practices used 
by commercial companies can help DOD make better progress in 
transitioning technologies to weapon programs. Private companies 
operate in a competitive environment that demands speedy delivery of 
innovative, high-quality products to satisfy market needs or the 
company will go out of business. DOD has a variety of "customers" and 
complex relationships that often hinder the chief technology officer at 
the corporate level from providing the type of strategic leadership 
found at successful companies. DOD puts pressure on itself to develop 
many new technologies. And because competition for funding is fierce, 
the technologies described with many superlatives for speed and 
lethality tend to get more attention than others do. We previously 
reported that to secure funding, DOD program managers frequently make 
overly optimistic promises to the warfighter about technologies' cost, 
feasibility, risk, and delivery schedule.[Footnote 4] The challenge for 
DOD and congressional decision makers lies not only in the "how to" 
aspects of technology transition but also in creating stronger and more 
uniform incentives that encourage the S&T and acquisition communities 
to work together to deliver mature technologies to programs. 

We are making recommendations that DOD strengthen its technology 
development and transition processes by developing a gated process that 
includes a transition phase and identifies criteria that can be used to 
support funding decisions. In addition, we are recommending that DOD 
expand the use of transition agreements, relationship managers, and 
metrics. We also recommend that DOD set aside a portion of research and 
development (advanced component development and prototypes) funds for 
the S&T community to manage the transition of technologies to 
acquisition programs. 

DOD concurred with our recommendations to develop a gated process for 
developing and transitioning technology and to expand the use of 
technology transition agreements, although we do not see its plans as 
fully responsive. DOD partially concurred with the recommendation to 
include additional metrics in technology transition agreements, stating 
that it recently developed metrics to gauge manufacturing readiness and 
that other metrics should be applied on a project-by-project basis. DOD 
also partially concurred with expanding the use of relationship 
managers, stating it already uses written documents to facilitate 
communication. However, DOD agreed that more time should be devoted by 
staff at the execution level on transition activities. We continue to 
believe that additional person-to-person communication needs to take 
place at all levels within the S&T and acquisition communities to 
supplement the written agreements. DOD did not concur with our 
recommendation to adopt more process oriented metrics because it 
believes that other processes it plans to mature over the next 4-5 
years will capture these metrics. We believe the metrics DOD wants to 
use will not allow the department to analyze its investment portfolio 
and make adjustments appropriately. Finally, DOD did not agree with our 
recommendation to set aside a portion of its 6.4 budget activity 
funding, known as advanced component development and prototypes funds, 
for the S&T community to manage for the transition of technology. DOD 
does not believe the S&T community should have additional resources to 
mature technologies to the point of successful transition to programs. 
We found otherwise in the commercial world and believe DOD's approach 
to funding technology development and transition is flawed. 

Background: 

Mature technologies are pivotal in developing new products. Without 
mature technologies at the outset, a product development program will 
almost certainly incur cost and schedule problems. Effectively managing 
technology as a separate process from product development can improve 
the potential for on-time product delivery at reduced cost. Leading 
companies know this and have established disciplined processes to 
prioritize their technology investments based on market needs, 
eliminate technologies that are not relevant or feasible, balance 
technology push and customer pull, and then transition technologies to 
product development efficiently. Overall, effective management of 
technology facilitates the delivery of new, innovative products to the 
user in less time. While DOD is very aware of the need for new advanced 
technology in its weapon systems, it has not always been effective in 
transitioning mature and relevant technologies to product development. 

Development of DOD's new weapon systems depends on two distinct phases: 
technology development and product development. DOD relies on its S&T 
community to identify, pursue, and develop new technologies that 
improve and enhance military operations and ensure technological 
superiority over adversaries. This includes the development of 
technologies for new weapons programs as well as those that will be 
inserted into existing systems. The acquisition community takes these 
new technologies, develops weapon system programs, and delivers the 
products--that is, the weapon systems--to the warfighter. Table 1 shows 
technology-related activities that take place during technology 
development and product development, who performs the work, fiscal year 
2006 funding, and overall goals. 

Table 1: DOD's Two Development Phases with Technology-Related 
Activities: 

Technology development phase: 
Activities: identify, fund, develop, and manage new technology projects 
that warfighter needs; 
Who does the work: government, industry, and academic technologists; 
2006 funding: about $13 billion; 
Goal: demonstrate a technology or subsystem in a relevant environment; 
Product development phase: 
Activities: integrate, demonstrate, support, and upgrade technologies 
on weapon systems; 
Who does the work: prime developer, industry supplier, and acquisition 
community's development lab engineers; 
2006 funding: about $58 billion; 
Goal: develop and deliver weapon systems to the warfighter. 

Source: DOD (data); GAO (presentation and analysis). 

Note: Activities listed under the product development phase focus 
solely on technology maturation and do not include other activities 
that the acquisition community is responsible for, such as overall 
design and manufacturing. 

[End of table] 

DOD uses research, development, test, and evaluation appropriations to 
fund its technology activities. The budget is divided into categories 
that generally follow a sequential path for developing technologies. 
Figure 1 shows the amount of money DOD spends in each category. (App. 
III contains additional details about the budget.) 

Figure 1: Fiscal Year 2006 Categories for DOD Research, Development, 
Test, and Evaluation Budget (dollars in billions): 

[See PDF for image] 

Source: DOD (data); GAO (analysis and presentation). 

[End of figure] 

The S&T community controls the budget for basic research, applied 
research, and advanced technology development. The budget category of 
advanced component development and prototypes involves testing and 
evaluating prototypes of systems or subsystems in a high-fidelity or 
realistic environment before product development starts. DOD officials 
say it is assumed that either the S&T community or the acquisition 
community could carry out this work, but traditionally the acquisition 
side prevails. After this point, additional technology-related 
activities are completed as part of product development under the 
authority of the acquisition community, namely, the program manager for 
a weapon system. 

Prior GAO reports have said that DOD launches new weapon programs with 
immature technology. We found this inability to mature the technology 
before the start of product development to be a major contributor to 
weapon system cost and schedule growth. In our March 2006 review of 52 
major DOD weapon programs, we found that 90 percent of the programs 
started with immature technologies[Footnote 5]. More than half of the 
programs were working with immature technologies at design review, when 
DOD acquisition policy expects the design to be stable. And by the time 
production began, one-third of programs still did not have mature 
technologies. 

Not surprisingly, we found that DOD research, development, test, and 
evaluation cost estimates increased dramatically for programs having 
immature technologies at program start. Figure 2 shows the average cost 
growth of DOD programs we reviewed when technologies were mature and 
immature at program start. 

Figure 2: Average Program Research, Development, Test, and Evaluation 
Cost Growth from First Full Estimate (sample of 52 DOD weapon 
programs): 

[See PDF for image] 

Source: DOD (data as of March 2006); GAO ( analysis and presentation). 

[End of figure] 

Programs that started with mature technologies have averaged a modest 
4.8 percent cost growth above the first full estimate, whereas programs 
that started with immature technologies have averaged about 35 percent 
cost growth. Some programs experienced significantly greater cost 
growth. A consequence of this cost growth is that the services 
typically deliver weapon systems late, have to reduce quantities to 
stay within cost estimates, shift funds away from other projects to 
make up for added costs, or some combination of the three. For example, 
the Air Force has incurred a 189 percent growth in the cost per 
aircraft for its F/A-22 tactical aircraft program, in large part 
because of technology maturation issues. In response, the Air Force has 
reduced the quantity of F/A-22 aircraft it plans to procure by 72 
percent from 648 to 183 to offset escalating costs. In the case of the 
Space-Based Infrared System High satellite, technology and design 
components matured late in development, contributing to research, 
development, test, and evaluation cost growth and four Nunn-McCurdy 
unit cost breaches[Footnote 6]. Instead of purchasing five satellites, 
the Air Force now plans to buy three at a program acquisition unit cost 
of about $3.4 billion, a 315 percent increase. 

Leading Companies Rely on Strategic Planning, a Gated Process, and 
Tools to Transition Needed Technologies: 

To successfully develop and transition technologies from their labs to 
their product lines, leading commercial companies depend on three key 
techniques: 

* strategic planning at the corporate level; 

* a gated management review process that ensures a technology's 
relevancy, feasibility, and transition readiness; and: 

* effective tools to solidify commitment, address transition issues, 
and gauge project progress and process effectiveness. 

Overall, corporations are incentivized to follow these critical 
techniques because the opportunity cost of not meeting customer demand 
is late delivery and lost revenues and market share for the company. 
Through rigorous adherence to these practices, leading companies 
increase chances of eventual success because the strategy for 
developing the technology fits the company's objectives, commitment is 
strong for incorporating the technology, and only after the technology 
is considered mature enough for use in a certain product does product 
development begin. Because the cost of developing new and breakthrough 
technologies can be high, funding typically comes from the corporate 
level rather than from a single product line unit, enabling the 
company's product lines to manage only product development risk, not 
technology risk as well. 

The central focus of this report is on how the lab and the product line 
communities work together to solidify the final steps of technology 
transition. Figure 3 depicts the general flow of technology development 
and where technology transition resides in the process. 

Figure 3: General Flow of Process Leading Up to Technology Transition: 

[See PDF for image] 

Source: GAO analysis and presentation of leading companies' practices. 

[End of figure] 

This report touches briefly on strategic planning, which precedes 
technology development. As the lab community identifies and develops 
high-priority technologies during the technology development phase, 
product line managers develop the business case for a new product by 
identifying the market potential of the new product and using systems 
engineering to set product requirements. This report does not focus on 
these product line activities or on product development.[Footnote 7] 

Strategic Planning Is a Critical Precursor to Successful Transition: 

Strategic planning--the effort to identify desirable technologies and 
prioritize resources--is an important early step in a company's ability 
to eventually deliver the highest-priority technologies to various 
product lines. Leading companies organize their research and 
development activities into technological "thrusts", or competencies 
that represent the core markets of their businesses. 3M and the other 
companies we visited undergo strategic planning at least annually, and 
this process enables corporate management to conduct portfolio 
analysis, identify long-term market needs, and match the projects in 
each thrust area to market needs. Eventually, managers determine which 
projects appear to be relevant and feasible, which ones are applicable 
for which products, whether the right projects are getting the right 
resources, whether the company wants to be first to market, and whether 
the final products should be released to the marketplace as soon as 
possible or several years down the road. Managers may decide to 
establish new thrust areas as new ideas come to light, rely on outside 
suppliers for some technology, or hire new people to fill technology 
voids. Projects that no longer are relevant or feasible are eventually 
terminated. This type of strategic planning is critical to ensuring 
that the right technologies ultimately transition to the right product 
line in an economical and timely way. 

Each company we visited funds technology differently, however, all 
ensure that funding for early technology projects is protected at the 
corporate level and not immediately beholden to product lines. 3M, for 
example, has 17 technological thrust areas, and the projects are 
directly funded with seed money from the corporate level. IBM receives 
the majority of its funding from the corporate level. IBM labs also 
receive and manage funding from product lines for specific projects. 
Both Motorola and Boeing require business units to fund a portion of 
lab costs each year based on their historical usage of lab resources, 
but the labs have freedom to use those funds in line with corporate 
strategy. Decisions are made at least annually on the composition of 
projects in the portfolio so it includes the most relevant and feasible 
projects. 

Leading Companies Use Gated Reviews to Validate Readiness for 
Technology Transition: 

After technology projects are selected for funding, they enter the 
development "pipeline" where a gated process is used to manage and 
oversee technology exploration, development, and transition. At each 
gated review, lab managers assess technology progress and ensure that 
certain criteria are met before technologies can enter the next stage 
of development. The final phase is dedicated to technology transition 
activities. The number and names of the gates vary by company, but the 
type of information collected and knowledge obtained are similar. For 
example, 3M uses a three-gate process, Boeing has a four-gate process, 
and Motorola has a five-gate process. Regardless of where the funds 
came from--a corporate or a product line unit--labs are responsible for 
managing, overseeing, and using the money to fund technology projects 
until the projects have transitioned to a product line. Figure 4 
generally depicts the technology development gates that companies use, 
lists types of deliverables expected during each gate, and shows who 
provides funding. 

Figure 4: Generalized Depiction of Deliverables and Funding under Gated 
Process: 

[See PDF for image] 

Source: GAO analysis based on Boeing Commercial Airplanes, Motorola, 
and 3M process. 

[End of figure] 

Explore gate: Lab technologists turn technology proposals into viable, 
executable projects that ideally will meet future market needs, while 
lab managers determine the amount of resources they need to invest in 
the projects. Funding goes to projects that are deemed relevant. Prior 
to the gated review, technologists deliver to lab managers a 
preliminary assessment of the competitiveness of the technology and a 
road map for completion. Technologists must address key deliverables 
related to such areas as manufacturing, intellectual property, 
marketing, life cycle management, and plans for the next gate. 

Develop gate: Technologists develop prototypes of the project and 
measure performance for relevancy and feasibility on an eventual 
product. Technologists deepen their understanding of the technology, 
refine the technology solution, identify the most attractive market 
segments for introducing the technology, and select a product line that 
will incorporate the technology into a new product. 3M and Motorola 
require product line commitment to transition a technology before the 
gated review draws near. This prevents labs from wasting valuable time 
and resources developing technologies that product lines do not want. 
Projects that do not gain product line commitment will either be 
terminated or go through a different process to demonstrate their 
relevancy and feasibility. Motorola leadership, for example, has a 
special program called an early stage accelerator under which selected 
projects can obtain funds to build additional prototypes to demonstrate 
the value of technology to product line managers or potential 
customers. Senior lab managers carefully monitor these projects through 
quarterly reviews to determine how they are progressing. If the project 
is not making sufficient progress, the project is quickly terminated so 
resources can be spent on other projects. 

Transition gate: Technologists demonstrate a prototype in an 
operational environment. The product line maintains a "customer" role 
until managers are confident that the technology will work in the 
intended product. Then gradually, the responsibility for funding and 
further developing the technology shifts from the lab to the product 
line. This shift in responsibility requires up-front and continuous 
planning by both the lab and product line communities to ensure a 
smooth transition. As shown in figure 5, companies we visited plan for 
technology transition to occur prior to program start. This provides 
product line managers sufficient time to gain additional knowledge 
about a technology's attributes as they are maturing it to its final 
form, fit, and function. Product line managers, who concurrently have 
been working on the business case for the product, are able to validate 
that the technology can be integrated into the preliminary product 
design. In addition, they can develop more reliable product development 
cost and revenue estimates before they are locked into needing this 
technology as a product feature. 

Figure 5: Linkage between Technology Development and Product 
Development: 

[See PDF for image] 

Source: GAO analysis based on leading companies' processes. 

[End of figure] 

During the transition phase, labs and product lines must complete a 
number of activities for transition to go smoothly. For example, labs 
must demonstrate that the technology meets product line cost, schedule, 
and performance requirements. In addition, production costs must be 
identified and acceptable to the product line. According to 3M 
officials, past experience has shown that costly manufacturing is a 
major reason for product line managers deciding not to transition a 
technology. Labs must also address intellectual property concerns, a 
step that is crucial to the company's ability to be a market leader. 
Product lines must address any people or organizational issues, such as 
the transition of jobs and training requirements that may result from 
using the new technology. They must also work out any agreements with 
the labs for on-going support. At the end of the phase, technical 
documentation related to the technology is transitioned from the labs 
to the product line. 

Boeing Commercial Airplanes[Footnote 8] assesses the extent to which 
these activities, as well as others, have been completed to determine 
whether a particular technology is ready to transition. The information 
is then summarized on a scorecard that lab and product line managers 
can use to quickly identify areas that need additional attention. The 
scorecard is updated continually and measures a technology's readiness 
from a business and production standpoint as well as its technological 
readiness. Boeing's labs have three phases for technology development, 
which they call discovery, feasibility, and practicality, plus a 
technology transition phase. Each technology is evaluated in 10 
readiness categories, and bars are plotted horizontally across the 
scorecard to indicate how much objective evidence exists for technology 
readiness. Once a technology has demonstrated full readiness in all 10 
categories, it is ready to be transitioned. Figure 6 shows a notional 
picture of a technology's readiness at one point in time. Because the 
hypothetical technology has reached full readiness in only five 
categories and is halfway through transition in one category, the 
technology is not ready for transition. The lab and product line would 
have to address issues involving readiness categories 3, 4, 6 and 9 
before transition can begin or the technology is in jeopardy of not 
being applied to a product. 

Figure 6: Notional Boeing Technology Maturity Scorecard for a 
Hypothetical Technology: 

[See PDF for image] 

Source: GAO analysis based on The Boeing Company's scorecard. 

[End of figure] 

Technology Transition Is Bolstered by Formal Agreements, Relationship 
Managers, and Metrics: 

Companies we visited use three tools to aid transition, including 
technology transition agreements, relationship managers, and metrics. 
Technology transition agreements are formal documents that detail the 
specific cost, schedule, and performance attributes of the technology 
that labs must demonstrate before transition can occur. Relationship 
managers address the details and issues surrounding transition, and 
metrics allow managers to identify the effectiveness of their 
technology development and transition processes and make adjustments 
when necessary. These tools require the active involvement of the labs 
and product lines to ensure successful transition of technologies to 
new products. 

Technology Transition Agreements: 

Labs and product lines use technology transition agreements to 
facilitate each technology's transition to product development. What 
the agreements specify varies by company but typically consists of 
specific technology and business readiness metrics, such as cost, 
schedule, and performance characteristics that labs must demonstrate 
for product line managers to agree to accept the technology. As figure 
7 shows, the agreements should answer some basic questions. 

Figure 7: Questions That Should Be Answered in Technology Transition 
Agreements: 

Is it real?; 

Is it relevant?; 

Is it marketable?; 

Where will it transition?; 

Do we have product line support?. 

Source: GAO. 

[End of Figure/Text Box]

These formal agreements may also include identification of product line 
resources needed to transition the technology, such as nonrecurring 
costs to further mature the technology, as well as consideration of 
recurring manufacturing costs associated with integrating the new 
technology on the product. Metrics collected in the agreements are used 
by product line managers to manage technology risk. For example, if the 
labs cannot develop needed technology within cost and schedule 
parameters or with specified performance characteristics, the product 
lines can terminate the agreement and go with an alternative 
technology. 

At Motorola, agreements enable technology project leaders to customize 
their work for a particular product line's needs. Product lines may 
request that the labs identify alternative technologies for a 
particular product, produce a detailed report about a specific 
technology, or conduct various tests to demonstrate the relevancy or 
feasibility of a technology. Each of these items becomes a 
"deliverable" in the agreement. 3M uses transition agreements only for 
technologies that are expected to go into a product. 3M's agreements 
incorporate many of the same metrics used by Motorola, including 
feasibility, relevancy, and application of the technology. In addition, 
agreements include an assessment of the product line's ability and 
commitment to transition technology. For example, they evaluate a 
product line's resource constraints that could inhibit a technology 
from transitioning, such as asking if the technology is too expensive 
or if manufacturing it would be too costly. 3M officials told us that 
some agreements include loaning key lab technologists to the product 
line for a period of time so product line teams can maintain momentum 
after the lab has signed off and moved on to developing technology for 
other parts of the company. 

Relationship Managers: 

3M, IBM, and Motorola use lab and product line relationship managers to 
smooth transition. Relationship managers foster effective transition 
practices by preventing the labs from pushing technologies that product 
line managers do not want and by preventing product line managers from 
pulling immature technologies from the labs. A Motorola official 
referred to relationship managers as the most important communication 
tool because communication occurs more frequently, thus allowing 
problems to be identified and addressed more quickly. 

Motorola designates key people at three different levels in both the 
labs and the product line to serve as relationship managers. As a 
technology is matured and demonstrated, communication intensifies and 
additional levels of relationship managers become involved. Figure 8 
depicts how Motorola's relationship managers communicate with each 
other within the lab or product line and with their designated 
counterparts in the other community. 

Figure 8: Communication Flow for Motorola's Three Levels of 
Relationship Managers: 

[See PDF for image] 

Source: Motorola; GAO (analysis and presentation). 

[End of figure] 

Executive managers in Motorola's labs and product lines are responsible 
for interfacing with each other on a periodic basis. Altogether, 
Motorola has eight executive managers--four in the labs and four on the 
product line side. Motorola officials told us that this number provides 
a one-to-one match between the labs and the product lines. Lab 
executive managers are responsible for ensuring product line needs are 
identified and new technology projects are started or existing projects 
are reprioritized to meet those needs. Their counterparts have the 
final word on what priority each technology project has with respect to 
the needs of the product line. Lab and product line executive managers 
are required to sign off on key technology projects and milestones for 
the current year. In addition, they try to remove any roadblocks within 
their own units and work with their counterparts to address any 
roadblocks between the two communities. 

Motorola considers liaisons the most critical in the process. Again, 
there are four liaisons in the labs and four on the product line side. 
They are the primary interface for coordination, collaboration, and 
communication during technology development and transition. Liaisons in 
the lab have broad oversight of technologies being developed and share 
information about technology breakthroughs with their counterparts on 
the product line side, and remove roadblocks for lab technologists. 
These lab liaisons work intimately with their product line counterparts 
to approve technology transition agreements and assess technology 
readiness. The lab liaisons oversee 40 to 60 technology transition 
agreements at any one time. Liaisons on the product line side are 
responsible for providing information about the changing needs of the 
product lines on a more real-time basis. They determine product line 
needs and priorities during the annual planning process and remove 
roadblocks for technologists on the product line side. Lab and product 
line liaisons are incentivized through their annual performance 
assessments and pay increases to work together to ensure the successful 
transition of technologies. 

The most direct and constant communication is between the lab's 
technologist, who developed the technology, and the product line's 
technologist, who is responsible for maturing the technology for 
inclusion into the product. These two technologists stay in continuous 
contact because they are the ones who have the most working knowledge 
about the technology. The lab technologist is expected to spend as much 
time as needed to make sure that transition happens as smoothly as 
possible. While the lab technologist does not become an official asset 
of a product line, he or she might have to spend a period of time 
working in the product line's development lab. 

Metrics: 

Leading companies use product metrics--such as weight, power, and 
reliability measurements--to assess the readiness of transitioning 
technologies and process metrics--such as profit growth and cycle time-
-to gauge the impact of their technology development processes and 
identify areas for improvement. The companies analyze data gained from 
the use of these metrics to evaluate how well they deliver on what is 
promised, better understand the value of their respective science and 
technology activities, and identify cases of inefficient investment in 
technologies or underutilization of lab technologies. Table 2 provides 
a composite example of metrics used at the companies we visited and the 
value of the metrics to management. 

Table 2: Metrics Used by Leading Companies to Assess Lab Projects and 
Processes: 

Categories of metrics: Project: Technology-specific; 
Examples of metrics: Nonrecurring development costs Scheduled delivery 
Recurring manufacturing costs Performance characteristics; 
* Size; 
* Weight; 
* Power; 
* Reliability; 
Head count; 
Use of the metrics: Allows lab and product line managers to assess the 
status of technology development and whether the technology meets the 
needs of the product. If technologies do not meet agreed-upon goals, 
the technology project may be terminated or the product line manager 
may decide to include it on a future product. 

Categories of metrics: Process: Status; 
Examples of metrics: Number of ongoing projects; 
Number of projects with a technology transition agreement; 
Number of projects completed by labs; 
Number of technologies transitioned; 
Number of projects terminated; 
Use of the metrics: Provides lab managers information on how many 
technology projects transitioned, were terminated, and are still 
ongoing. Companies expect that almost all technologies that make it to 
the final stages of technology development will transition. If they 
experience a lower transition rate than expected, officials will 
examine their processes to determine what changes are necessary to 
improve transition or determine why the project was not terminated 
earlier. 

Categories of metrics: Process: Timeliness; 
Examples of metrics: Development cycle time; 
Percentage of tasks on time; 
Task slippage; 
Time to market; 
Use of the metrics: Measures the amount of time it takes labs to 
develop technologies. The metrics allow lab managers to identify and 
focus on projects that are moving slower than expected. A lab project 
may be terminated or additional resources may be allocated to speed up 
development. Use of the metrics allows product line managers to decide 
whether a technology will be ready in time to include on a given 
product. 

Categories of metrics: Process: Impact; 
Examples of metrics: Number of technologies commercialized; 
Number of multiple transitions; 
Return on investment; 
Profit growth; 
Market share growth; 
Orders captured; 
Cost reduction; 
Number of patents/influential papers; 
Customer satisfaction; 
People rotation; 
Use of the metrics: Provides lab and corporate managers feedback on the 
market impact of their technology investments in terms of revenue and 
market share. The metrics also provide information on product line 
satisfaction of lab performance. Satisfaction survey results are a 
useful tool for identifying development and transition problem areas 
that need management attention. 

Source: Interviews with leading companies (data); GAO (presentation and 
analysis). 

[End of table] 

Motorola's labs use a suite of metrics, many of which are shown above, 
to monitor technology projects and lab performance. For example, the 
company tracks the estimated and actual development time and costs for 
each project to determine estimation accuracy. In addition, it 
identifies and tracks the number of lab and product line people 
(headcount) needed for development and transition. These metrics are 
not treated with the same rigor as metrics that are used after product 
development begins. Corporate managers understand that technology 
development must be managed with more latitude to accommodate 
calculated risk taking. In addition, Motorola asks each product line to 
fill out satisfaction surveys twice a year to assess lab performance. 
Because relationships are at the heart of the process, the product 
line's perceptions are of paramount importance, making internal 
customer satisfaction a key metric. A Motorola official indicated that 
satisfaction surveys are a ubiquitous part of the Motorola culture. Not 
only do they help the labs improve, but they help the company improve 
overall, the official said. The surveys also provide compelling 
anecdotal information about performance problems that need to be 
addressed by certain labs. 

In addition to technology-specific metrics, Boeing tracks the number of 
technologies that were actually integrated into products. While the 
intent is that all completed technologies will make it into a product, 
this does not always happen because of changes in market requirements, 
timing, or funding constraints. Boeing also has metrics that assess the 
technology's impact on orders for new airplanes and on its ability to 
reduce manufacturing costs. The labs, for example, have a 
responsibility for improving the way Boeing designs and builds its 
products. As such, they have goals for reducing recurring and 
nonrecurring costs and cycle time. Finally, Boeing uses a metric that 
encourages the transfer of people, along with the technologies, to the 
product line side. The company tracks the rotation of people from its 
labs to its product lines specifically to improve its transition 
processes and to refresh staff with "program people." As a result, 
company officials believe they can better align their labs to their 
product lines strategies. 

Company officials also told us that, similar to DOD, they have 
struggled to assess return on investment for technologies they develop. 
This is because technology can undergo a metamorphosis after it leaves 
the labs and can be applied to multiple products or altered before 
going into the intended product. Company officials said they are 
actively trying to track technology utilization and the impact their 
technology investments have had on company revenues. 3M has a 
technology database it uses to track each project it funds. The 
database contains cost, schedule, and performance data and summarizes 
other information, such as how long a project took to complete and lab 
and product line funding used to mature the technology. 3M then 
compares the cost information with worldwide sales, tracked through bar 
codes, to determine its return on investment for a particular 
technology. Company officials indicated that there are still problems 
with this methodology for estimating return on investment and that they 
are continuing to refine their approach. 

DOD Lacks Breadth and Depth of Techniques That Leading Companies Use to 
Effectively Transition Technologies: 

DOD has taken some steps over the past few years to improve its 
technology transition processes, but the practice of accepting high 
levels of technology risk at the start of major weapon system 
acquisition programs continues to be the norm. This shortcoming is a 
major contributor to DOD's poor cost and schedule outcomes. Many of 
DOD's problems can be attributed to deficiencies in strategic planning 
for critical technologies, processes for technology development and 
transition, and tools that support transition. A recent Defense Science 
Board report states that DOD is not as well positioned as it should be 
from a strategic standpoint to meet the challenges and exploit the 
opportunities offered by technology. The report identified several 
opportunities for improvement at the Director, Defense Research and 
Engineering level (DDRE)[Footnote 9] to help the department jointly 
identify, prioritize, develop, and deliver the technologies most 
relevant and critical to meeting weapon system requirements in a timely 
manner. DOD's technology development process is undisciplined and lacks 
criteria for maintaining a close connection between fledgling 
technologies and the products that will need them to meet the 
warfighter's future needs. As a result, technologies are often not 
ready when they are needed, and acquisition programs pull the 
technologies into their programs too early, leading to inefficiency 
during product development and cost and schedule increases. DOD's 
process for transitioning technologies to product development is funded 
and managed by its acquisition community, the opposite approach to that 
taken by the companies we visited. 

We identified some initiatives within DOD's S&T community that emulate 
some of the tools we found in the commercial world. They hold promise, 
but must be accepted, improved, and replicated significantly more than 
currently to have a positive impact. For example, the military services 
are using technology transition agreements and have established boards 
to select and oversee some of its technology portfolio. Also, DARPA 
uses relationship managers to expedite the efficient transition of 
technologies to products and DOD has established other programs that 
institutionalize some best practices. The collection and use of 
meaningful metrics, however, remain a problem. 

On the basis of our previous review of 52 major weapons programs, we 
found that DOD's typical path involves starting programs with immature 
technologies and concurrently working on technology development and 
product development. Figure 9 is a depiction of this path. 

Figure 9: Path that DOD Routinely Follows for Technology Development 
and Product Development: 

[See PDF for image] 

Source: DOD (data); GAO (analysis and presentation). 

[End of figure] 

Strategic Planning Affects DOD's Ability to Meet Warfighters' Needs: 

DOD has an annual strategic planning process that involves the 
development of several plans at the corporate level and within the 
services and defense agencies for determining how to invest its S&T 
funding, which amounted to about $13 billion in fiscal year 2006. 
Despite these efforts, a recent Defense Science Board report[Footnote 
10] stated that DOD is not as well positioned as it should be at the 
DDRE level (the corporate level) to meet the challenges and exploit the 
opportunities offered by technology. The report identified several 
opportunities for improvement. Until improvements are made, it is 
likely that the there will be strategic gaps and overlaps in technology 
coverage as the services and agencies develop their own approaches to 
meet critical warfighter needs. 

Similar to private industry, the department goes through an annual 
strategic planning process led by its corporate level. As part of its 
responsibilities, DDRE develops the Defense Science and Technology 
Strategy, which serves as the foundation for DOD's science and 
technology strategic planning process. The strategy identifies five 
generic technology thrust areas--information assurance, battlespace 
awareness, force protection, reduced cost of ownership, and maintaining 
basic research--that have high priority in the department, and is 
supported by four other documents, including the: 

* Basic Research Plan, which presents the DOD objectives and investment 
strategy for DOD-sponsored basic research performed by universities, 
industry, and service laboratories; 

* Defense Technology Area Plan, which presents the DOD objectives and 
the strategies for applied research and advanced technology development 
investments; 

* Joint Warfighting Science and Technology Plan, which takes a joint 
perspective across the applied research and advanced technology 
development plans to ensure that the science and technology program 
supports priority future joint warfighting capabilities; and: 

* Defense Technology Objectives, which identify specific technology 
advancements that will be developed or demonstrated, the anticipated 
date of availability, and specific expected benefits. 

Together, these documents, as well as the supporting S&T plans of the 
military services and defense agencies, are supposed to provide the 
framework for decisionmaking throughout the science and technology 
community. 

However, a recent Defense Science Board report states over time there 
has been a relative decline in the influence of DDRE at the corporate 
level on strategic matters. In the 1960s and 1970s, the corporate level 
was more proactive and provided high-level direction that drove several 
decisive technological developments, including stealth; standoff 
precision strike; and tactical intelligence, surveillance, and 
reconnaissance systems that have transformed U.S. military 
capabilities. Since the 1980's DDRE has used a decentralized management 
approach, relying on the services and agencies to determine and 
prioritize the most needed technology projects. The report states that 
as a result, the department is not wellpositioned to implement new 
operational capabilities needed for future warfighters, including: 

* identifying and tracking terrorists, 

* addressing commercialization and globalization issues, 

* rapidly evolving technologies such as bio-and nanotechnologies, and: 

* identifying the proper mix of short-and long-term projects to work 
on. 

One of the recommendations included in the report is for the 
Undersecretary of Defense (Acquisition, Technology, and Logistics) to 
develop a strategic technology plan and that DDRE be tasked with 
assuring that all research and development organizations are 
implementing DDRE's strategic technology guidance. The board also 
believes DDRE needs to be more involved in strategic challenges related 
to (1) gathering and nurturing technology from a variety of sources, 
(2) developing and exploiting technology to enable new disruptive 
capabilities; (3) identifying and countering disruptive capabilities 
developed by adversaries using readily available or advanced 
technology, and (4) ensuring an adequate level of long-term research 
for DOD needs. Without effective corporate level leadership, warfighter 
needs are addressed in a decentralized, uncoordinated manner, thereby 
increasing the risk that the department will not fulfill these needs. 

DOD Does Not Use a Disciplined, Gated Process to Test for Relevancy, 
Feasibility, and Transition Readiness: 

DOD's 5000 series acquisition policy specifies that technology 
development should be separated from product development and that a 
project shall not exit the technology development phase until the 
technology has been demonstrated in a relevant environment. This is in 
line with the practices we found at leading companies, with the 
exception that the companies require technologies to be demonstrated in 
an operational environment before program start, which is a higher 
degree of readiness. However, the department lacks a structured, gated 
process for managing technology development and transition, as well as 
criteria that would allow decision makers to know when technology is 
ready to progress from the technology development environment into an 
acquisition program to begin product development. As a result, the 
services continue to launch new programs with immature technologies, 
and acquisition programs take on technology development 
responsibilities, an activity that is considered too risky for 
commercial companies we visited. The following figure shows where the 
technology development phase occurs in DOD's acquisition process and 
the entrance and exit criteria for that phase. 

Figure 10: Criteria for DOD Technology Development Phase: 

[See PDF for image] 

Source: GAO analysis and presentation of DOD processes and practices. 

[End of figure] 

As shown in the figure above, DOD has two phases prior to program start 
(also referred to in DOD nomenclature as milestone B). The first is the 
concept refinement phase, when acquisition programs refine the initial 
concept and develop a technology development strategy that is supposed 
to guide activities during the technology development phase. According 
to the acquisition policy, the strategy should include: 

* a discussion of the planned acquisition approach, including a summary 
of the considerations and rationale concerning the approach; 

* a discussion of the planned strategy to manage research and 
development; 

* a complete description of the first technology demonstration; and: 

* a test plan. 

The policy, however, does not address the role of the S&T community in 
designing or supporting a technology development strategy. And 
according to S&T officials, their role in this phase is minimal. We 
recently reported that DOD skipped a formal milestone meeting that 
should take place at the end of the concept refinement phase, referred 
to as milestone A, for 80 percent of the programs we reviewed.[Footnote 
11] This inhibits the S&T and acquisition communities' ability to 
create a realistic plan for developing and maturing needed technologies 
prior to program start and executing a plan once the program begins. 

During the technology development phase, the acquisition policy calls 
for DOD to focus on the development, maturation, and demonstration of 
the technologies needed for the capability under consideration. To exit 
this phase, technologies should be demonstrated to be sufficiently 
mature. DOD defines this as demonstrating technologies in a relevant 
environment. (See app. IV for definitions of technology readiness.) 

When comparing the DOD process with that used by commercial companies 
we visited, we identified two major differences. First, although DOD 
policy states that the S&T community "shall enable rapid, successful 
transition … to useful military products," there is no defined 
transition phase with criteria to facilitate assessment of a 
technology's readiness to transition. Instead, the services have senior-
level boards that review projects on an annual basis to determine if 
they are on track relative to cost, schedule, and performance goals set 
out for the program and rely on a technology readiness tool to gauge 
development progress. The reviews do not include a formal assessment of 
many of the technical and business criteria contained in the Boeing 
scorecard, such as determining if the costs, benefits, and risk are 
well understood and technology is affordable. As a result, program 
managers often pull a technology from technology developers before the 
program manager has the opportunity to validate that the technology can 
be integrated into the preliminary product design and to develop good 
product development estimates prior to the start of system development. 
As stated earlier, commercial companies we visited had a dedicated 
phase at the end of the technology development process for shifting 
technology responsibilities to their product lines. We recently 
reported that almost three-fourths of the programs started since the 
acquisition policy was revised in 2000 began with critical technologies 
that were not ready for product development.[Footnote 12] Seven of the 
nine programs we reviewed more indepth, about 80 percent, were approved 
to begin development even though program officials reported that 
technologies were below readiness levels required for entering product 
development. This included programs like the Future Combat System and 
the Joint Strike Fighter. 

We found DOD generally views immature critical technologies at the 
beginning of development as an acceptable risk as long as there is a 
plan to mature the technologies by the time the program reaches its 
design readiness review. In effect, the department views risk 
management plans as an acceptable substitute for demonstrated 
knowledge. For example, the Navy's Multi-Mission Maritime Aircraft 
program had none of its critical technologies mature at program 
initiation. Instead of holding the program to the acquisition policy 
criteria for entering development, the decision maker simply directed 
the Navy to work with the Office of the Secretary of Defense to 
implement risk mitigation and technology maturation plans during the 
integration phase of system development. 

Second, under DOD's current funding structure, the transition of 
technology--which should occur prior to the beginning of product 
development and milestone B--is funded and managed by acquisition 
programs. This contrasts with the approach used by commercial 
companies, where the lab manages and funds these activities. Commercial 
firms find that holding their labs accountable for the management and 
funding of technology as it transitions to a new product forces them to 
deliver more quickly and efficiently and allows the product lines to 
focus on product development and the risks associated with it. Figure 
11 illustrates the difference in responsibility. 

Figure 11: Accountability for Management and Funding of Technology: 

[See PDF for image] 

Source: GAO analysis and presentation of landing companies' and DOD 
practices.

[End of figure] 

As shown above, in the commercial companies we visited, product 
developers are allowed to act like customers for emerging technologies; 
they are not required to accept, manage, and fund technology risk. This 
significantly improves the chances of their products reaching the 
market quickly, at predictable cost, and with high quality. In DOD, 
major weapon system acquisition programs pull technologies that are not 
yet ready for a product in order to meet that product's requirements. 
The S&T community, although still called upon to review technology 
readiness, has no funds at stake and is no longer responsible for the 
risk from technology at that point. This is a major contributor to the 
significant cost overruns and late deliveries of major weapon systems 
to the warfighter in recent years. 

DOD Is Adopting Transition Tools, but Use Is Not Widespread: 

DOD is using several tools we found were also being used by commercial 
companies to facilitate the transition of technology to weapons 
programs. For example, the services are using technology transition 
agreements for some of their technology projects, DARPA is using 
relationship managers to address transition issues, the services have 
established boards to oversee and manage a portion of their technology 
portfolios, and DOD has several new programs, including the Joint 
Capability Technology Demonstration program to speed a technology's 
transition to a weapon system. However, because of the newness or small 
amount of funding associated with these tools, widespread use has yet 
to occur. We also found that DOD is using technology-specific and 
status metrics, but does not have sufficient measures in place to 
assess the impact of their technology investments or technology 
transition processes. 

Technology Transition Agreements: 

The services have recently begun using technology transition agreements 
to formalize technical expectations labs must demonstrate in order for 
acquisition programs to transition technologies into their programs. 
DOD science and technology officials believe the agreements are a 
useful way to hold the technology developers accountable and gauge lab 
progress toward meeting technology specifications. We are encouraged by 
their use of agreements, particularly in the Navy. Agreements we 
reviewed, however, did not contain some of the information or metrics 
commercial companies believe is valuable to track, and use thus far has 
been limited. 

The agreements we reviewed contained some of the same elements included 
in agreements used by leading companies we visited, such as a 
description of the technology project, key technology developer and 
weapon system personnel associated with the project, and specific 
performance characteristics that the lab must meet for transition to 
occur. In addition, they identify the amount of funding acquisition 
programs will have for transition and when that funding will be 
available. Unlike the leading companies we visited, however, these 
agreements did not typically require the technology developer to 
demonstrate cost metrics for the technology to be included in a weapons 
program. Commercial companies, for example, include an assessment of 
manufacturing costs that could be expected if a technology were to be 
included on a new product and they usually demand prototypes of the 
technology as demonstration of readiness to transition. A recent 
Defense Science Board Report highlighted the importance of addressing 
manufacturing concerns during technology development. Excerpts are 
shown in figure 12. 

Figure 12: Excerpts from a Defense Science Board Task Force Report on 
the Manufacturing Technology Program: 

"Immature technologies and manufacturing challenges have a significant 
impact on DOD's ability to rapidly and affordably transition technology 
to the war fighter."; "S&T program managers often believe that 
affordability and manufacturing issues are not relevant concerns in 6.3 
programs, focusing instead on fabrication of test and evaluation and 
prototype articles. But this line of thinking leads to higher costs 
later in a program, when manufacturing concerns are addressed after 
technical designs are considered 'ready.' "; "In order to achieve the 
objective of lower cost equipment, manufacturing concerns must be 
addressed earlier in the program life cycle. Production and support 
costs need to become a component of key technical design requirements, 
before the final stages of development when technologies are released 
for prototyping.". 

Source: Defense Science Board study (data); GAO (presentation and 
analysis). 

[End of Figure/text Box]

We also found that the services' use of technology transition 
agreements has been very limited. For example, we found that the 
services have a combined total of 224 applied and advanced technology 
projects that they have identified as candidates for transition, of 
which 146--or about 65 percent--have technology transition agreements. 
The Navy accounted for the greatest percentage, with 90 of its 115 
projects having an agreement, almost 80 percent. However, there are 
hundreds of other military services applied and advanced technology 
research projects that were not selected as candidates where agreements 
may also be useful. As stated earlier, commercial companies we visited 
had agreements for nearly all technology projects. The agreements were 
updated at least annually or sooner if particular tasks were completed. 
Lab officials use metrics included in the agreements, as well as other 
criteria, to determine if the project should be continued. 

It should also be pointed out that while technology transition 
agreements are useful tools for firming weapons program commitment to 
transition, they do not guarantee transition success. For example, a 
Navy official told us that about one-third of the projects that had 
been expected to transition in fiscal year 2005 did not transition for 
the following reasons: 

* The technology did not meet the cost parameters required for 
transition to occur. 

* Weapon system requirements changed, forcing changes to the original 
agreement. 

* Program offices did not identify dollars in its budget to transition 
the technology. 

* A program office decided the technology was too risky compared with 
current hardware in production. 

Air Force and Army officials told us they do not track information on 
why their technologies do not transition. However, they have 
experienced similar problems as the Navy. They also stated that some 
lab technologies do not transition because prime contractors decide to 
use technologies developed in-house, even though DOD technology 
developers met the metrics included in the transition agreements. 
Because agreements are usually established between the technology 
developers and program offices prior to the selection of prime 
contractors, there is no assurance that the technologies will 
transition. Even in cases where a prime contractor is known, such as 
for ongoing development programs, the prime contractor is not a party 
in the agreement. 

Relationship Managers: 

With the exception of DARPA, DOD does not use relationship managers in 
the same manner as leading private companies. According to an Air Force 
lab official, relationship managers market technologies being developed 
by the labs or gather data about ongoing projects for senior lab 
management. Most communication about technology transition in DOD takes 
place through integrated product teams or during annual reviews of 
technology projects by the senior-level oversight boards for each of 
the services. Use of relationship managers for these purposes are 
helpful, but the managers do not necessarily serve as points of contact 
within the labs and acquisition communities, do not devote time toward 
efficiently transitioning technologies to multiple weapon system 
programs, and do not help identify and address systemic transition 
problems. 

Within DARPA, senior officers, called operational liaisons, focus on 
marketing and transitioning DARPA-sponsored technologies. According to 
the DARPA director, the liaisons have been very helpful with 
transitioning technologies because they are well practiced at using the 
command chain of their respective services and finding the right 
service contact at the right time. The liaisons: 

* provide operational advice for planning and strategy development; 

* provide an understanding of service perspectives, issues and needs so 
that potential customers can be identified and effective agreements can 
be written; 

* draft and coordinate agreements between DARPA and the services; and: 

* direct technology insertion in the services. 

Figure 13 describes the impact that operational liaisons have had on 
transitioning a DARPA-sponsored program called Boomerang. We believe 
that DARPA's approach could serve as a model for how the military 
services might establish more formal roles for communication between 
the S&T and acquisition communities. 

Figure 13: DARPA, a Success Story: 

The DARPA director credits operational liaisons for the quick 
transition of the Boomerang, an acoustic shot-detection system, from 
the lab to troops in Iraq. DARPA developed the system in response to 
feedback from Iraq that convoys were being engaged by snipers yet 
remained unaware of sniper attacks until a windshield was broken, a 
soldier was hit, or a vehicle was visibly damaged upon inspection at 
the end of the convoy mission; Within 60 days of an urgent Army 
request, DARPA fielded the first Boomerang system. But DARPA's director 
said the system did not hold up well in the extreme weather conditions 
and under wartime conditions. As Boomerang II was being prepared for 
fielding, the director said the operational liaisons helped craft a 
more realistic concept of operations, training package, and logistical 
support package to ensure that Boomerang II not only was 
technologically ready for combat but was properly supported with spare 
parts, maintenance facilities, maintenance personnel, training, and 
lessons-learned feedback to DARPA and the Army. The liaisons also 
ensured that the Army acquisition community was alerted to Boomerang 
II's deployment so product developers would be ready to evaluate the 
final product for movement into the more traditional acquisition 
process. 

Source: DARPA (data); GAO (presentation and analysis). 

[End of Figure/Text Box]

DOD Programs to Aid Transition: 

The military services and the Office of the Secretary of Defense have a 
variety of programs to help transition technologies to weapons systems 
or directly to the warfighter. These programs, some of which are 
relatively new and others that have been around for several years, have 
met with some success. However, these programs represent a small 
portion of $13 billion DOD spends on science and technology 
development. As such, they cannot single-handedly overcome transition 
problems, but rather demonstrate various ways to ease transition. 

In recent years, each of the military services has established senior- 
level boards to oversee technology programs that include advanced and 
in some cases applied research projects. This includes the Air Force's 
Advanced Technology Demonstration program, the Army Technology 
Objectives program, and the Navy's Future Naval Capabilities program. 
The boards, which are comprised of representatives from the lab, 
acquisition, and warfighter communities, are responsible for selecting 
projects to be developed, allocating funding to those projects, and 
reviewing the projects' progress on an annual basis. The boards 
emphasize transition planning and can address obstacles to successful 
transition. According to officials, these programs represent about $1.3 
billion of DOD's S&T budget. 

The Advanced Concept Technology Demonstration program and the newly 
established Joint Capability Technology Demonstration program were 
initiated by DOD as a way to get technologies that meet critical 
military needs into the hands of users faster and at less cost than by 
the traditional acquisition process. Under these programs, military 
operators test prototypes that have already been developed and matured 
in realistic settings. If they find the items to have military utility, 
DOD may choose to buy additional quantities or just use the items 
remaining after the demonstration. Fiscal year 2006 was the first year 
of a 3-to 5-year period where the current Advanced Concept Technology 
Demonstration program will be phased out in favor of the new Joint 
Capability Technology Demonstration program. Department officials 
believe the new joint program offers improvements over the earlier 
program in that capabilities will be demonstrated 1-2 years earlier, 
there is a greater focus on combatant command needs during the 
selection process, and the Office of the Secretary of Defense provides 
significantly more funding during the first 2 years of the 
demonstration project. 

A big difference between the two programs is that the new joint program 
is expected to provide a better path for the transition of technologies 
because it includes funding for both advanced technology development 
and advanced component development and prototypes. The Advanced Concept 
Technology Demonstration program only includes funding for advanced 
technology development. It must rely on the acquisition community to 
identify advanced component development and prototype funding for 
transitioning the technologies. An additional benefit is that the 
Office of the Secretary of Defense is partnering with the Naval 
Postgraduate School to conduct business case analysis for completed 
demonstration projects. This type of analysis is expected to aid 
decision makers in evaluating alternative approaches to the allocation 
of scarce resources competing for transition funds. 

The Manufacturing Technology Program is aimed at quickly identifying 
and solving production problems associated with technology transition. 
It focuses on the needs of weapon system programs for affordable, low- 
risk development and production. For example, one project, which was 
given $730,000, was geared toward improving the reliability and 
strength of large-diameter fasteners used to attach various components 
on the Seawolf and Virginia Class submarines. Corrosion concerns 
required that the existing fasteners be replaced at periodic intervals 
to preclude catastrophic failure. Navy and commercial companies 
evaluated a wide variety of materials for strength and corrosion 
resistance to determine the best candidate material and then adjusted 
the material manufacturing process to obtain the required strength and 
toughness levels to ensure the fasteners could be produced without 
extraordinary measures or a deterioration of the material properties. 
Among other things, program officials expect the project to result in 
cost avoidance of $1.1 million per fielded submarine per year and 
improved reliability. 

There are other, smaller programs that also focus on transition. Two of 
these are the Foreign Comparative Testing Program, which focuses on 
identifying, evaluating, and procuring technologies that have already 
been developed and tested in other countries, and the Technology 
Transition Initiative, which focuses on speeding the transition of 
technologies developed by DOD's S&T programs into acquisition programs. 
These programs received about $37 million and $29 million in fiscal 
2006, respectively. The Foreign Comparative Testing Program, for 
example, successfully evaluated a South African mine-protected 
clearance vehicle that will protect soldiers from the effects of 
landmine explosions during route clearance operations. As of June 2006, 
61 of these vehicles have been delivered to the Army, Navy, and Marine 
Corps. 

Metrics: 

For the most part, the military services track and use only a few of 
the metrics we found at commercial companies. As stated earlier, they 
have some technology-specific metrics in their technology transition 
agreements. In addition, they have status metrics to track some of the 
on-going, completed, and transitioned projects that are funded by the 
Air Force, Army, and Navy S&T communities as well as the number of 
projects with technology transition agreements. However, the services 
have few metrics that would enable them to gauge the impact of their 
investments and the effectiveness of their processes for developing and 
transitioning technology. 

DOD officials told us that establishing good, quantifiable metrics to 
measure transition success is difficult to do. Some of the challenges 
they face include determining: 

* how long to track a technology after it has transitioned, 

* how to track and distinguish transition success when the same 
technology is used on multiple weapons programs, and: 

* how much credit to take when only portions of a technology transition 
to a weapon program. 

Further, DOD S&T officials do not believe that they have enough 
resources to track technology once technology developers finish working 
on a project. They say that it would be very labor-intensive to track 
long-term measures, such as the impact of transitioned technology in 
terms of cost savings and improved performance. Instead, the services 
rely heavily on projected cost, schedule, and performance improvements 
that are required by S&T program management as part of the project 
selection process. Although progress in meeting technology expectations 
is monitored throughout technology development, once technology 
developers are finished working on a technology, little is done to 
determine if it actually went onto a weapon system and if it is being 
used in the field. 

Last year, we reported that the Office of the Secretary of Defense had 
difficulty tracking the impact of three small technology transition 
programs it oversees--the Technology Transition Initiative, the Defense 
Acquisition Challenge Program, and the Quick Reaction Fund.[Footnote 
13] Nevertheless, we pointed out that there may already be readily 
available starting points within the department to capture more 
information regarding return on investments. For example, we pointed 
out that the DOD Foreign Comparative Testing Program has established 
metrics to measure the health, success, and cost-effectiveness of the 
program and has a database to facilitate return on investment analyses. 
Further, a study by the Naval Postgraduate School to identify metrics 
for the Advanced Concept Technology Demonstration program may also be 
useful to the military services for assessing transition success and 
impact. 

Differences in Environment and Incentives Contribute to Different 
Practices and Outcomes: 

There are critical differences between the environments and cultures of 
private world-class companies and DOD that must be recognized before 
tangible progress can be made in establishing more efficient practices 
for transitioning technologies to major weapon system acquisition 
programs. Examples from past initiatives serve as reminders that just 
changing the mechanics of technology transition processes, without 
changing the environment that determines incentives, may not produce 
better outcomes. Private companies operate in a competitive environment 
that demands speedy delivery of innovative, high-quality technologies 
and products to satisfy market needs. If the company cannot meet those 
criteria, it will cease to exist in that environment. On the other 
hand, DOD operates in an environment with a complex variety of 
"customers." These complex relationships often hinder the ability of 
DDRE, its "corporate" component, to provide strategic leadership 
similar to that of the private company. Success in this environment is 
often based on a single service's ability to launch a program to 
address critical needs and to secure annual funding for the program. 
The challenge for DOD and congressional decision makers may not lie so 
much in the "how to" aspects of technology transition as in creating 
stronger and more uniform incentives that encourage the S&T and 
acquisition communities to work together to deliver mature technologies 
to programs. 

Private firms must continually deliver innovative, reliable products to 
market very quickly to prosper. The delivery-oriented environment that 
private firms live in creates a need for strategic simplicity and 
directness from leadership and forces product lines and labs to measure 
success in terms of lower costs and increased revenue and market share. 
The ability to deliver new, innovative, reliable products of high 
quality to market as soon as possible drives revenue and market share. 
This environment creates incentives to maintain an efficient technology 
base that is focused on market needs and can efficiently transition 
feasible technologies into new products, keeping costs down while 
achieving greater revenues. Because commercial firms understand that 
they do not succeed until a new, innovative product is delivered to the 
customer, they view successful transition of technologies to products 
as a critical part of their value chain. The corporation, therefore, 
insists on accountability from and supports its technology developers 
with a strong strategic process for ensuring that its product portfolio 
is balanced. It does not want too many programs vying for scarce 
investment dollars. 

For example, IBM officials stated that poor strategic planning was one 
of the problems the company had to address after it incurred an $8 
billion loss in 1992-1993. According to company officials, technologies 
no longer could be developed for the sake of development; they had to 
be aligned with a product line and have market potential to be funded. 
IBM labs now conduct a situation analysis that takes an internal and 
external look at technologies being developed and a gap analysis to 
determine where IBM might be behind in technology development. 

At 3M, meeting market imperatives is critical and senior leadership has 
set a goal to increase sales 5 to 8 percent annually through new 
products. In this environment, the company has established a 
companywide initiative referred to as 2X/3X. Generally, the goal is to 
double the number of new ideas going into the product development 
pipeline and to triple the market success of the products coming out of 
the pipeline. Lab and product line managers are evaluated and held 
accountable for the success of the initiative based upon their ability 
to meet sales goals and properly resource their projects. According to 
3M's 2004 annual report, the initiative is producing strong results. 3M 
officials believe their culture is well aligned with this competitive 
environment. For example, technologists participate in companywide 
technology forums to discuss problems and share information. A 
technology council, comprised of about 80 lab and technical directors, 
meets monthly to share ideas, experiences, and best practices among the 
labs. 

At Motorola, the corporate vision is known as "seamless mobility." 
Senior leaders have aligned the company's global resources--including 
its $3.1 billion annual research and development budget and a worldwide 
research network--to help achieve this goal. According to Motorola 
officials, the seamless mobility vision has been instrumental in 
uniting Motorola labs and product lines over the past 3 years toward a 
common goal. 

On the other hand, DOD operates in an environment that often hinders 
the ability of its corporate component--DDRE--to provide strategic 
leadership similar to the private company. Not only must the department 
worry about national security imperatives, it must also answer to 
Congress and its oversight agencies; manage a stove-piped, parochial 
culture; ensure the public's trust that tax dollars are wisely and 
fairly spent; and incentivize an industrial base to deliver cutting- 
edge, often very risky technologies and products. This is a complex set 
of deliverables by any measure. It creates an environment that makes it 
difficult for DDRE to lead and has contributed to an undisciplined 
strategic planning process. Success in this environment is often based 
on a service's ability to launch a program to address critical needs 
and to secure annual funding for the program. It is characterized by 
fierce competition among the services to win scarce funding to begin 
major weapon system acquisition programs, regardless of the readiness 
of technologies to meet far-reaching requirements. As a result, 
programs are pressured to distinguish themselves from other programs 
previously developed by promising new, enhanced features such as being 
faster and/or more lethal than anything else. It is very common for 
these new programs to include requirements for new technologies with 
overly optimistic assessments of technological feasibility, risk, and 
delivery schedules. This environment and these incentives breed 
uncertainty and risk and, as a result, cost overruns and delivery 
delays. 

Conclusions: 

In DOD, delivering mature technologies to weapon system acquisition 
programs at the right time continues to be a challenge. Rather than 
addressing technology issues in a science and technology environment 
before product development starts, acquisition programs carry 
significant technology risk into product development. This brings with 
it a high risk that costs will rise and deliveries to the warfighter 
will be delayed. In fact, there is strong evidence that acquisition 
programs that start with immature technologies encounter significantly 
poorer acquisition outcomes than others. This approach is in sharp 
contrast to the approach taken in the commercial world. High-performing 
companies solve technology challenges in the S&T environment. To do 
this, they have put in place processes and adopted techniques that are 
pertinent to DOD. Strong strategic planning defines critical investment 
priorities, and a structured process defines the path towards a 
technology's transition to product development. This transition is 
supported by technology transition agreements that hold the research 
labs accountable for what they must deliver. They also include 
clarification of the responsibilities the product developer has in 
accepting new technologies. Metrics are used to force demonstration of 
relevancy and feasibility at key points in the process and gauge the 
success of individual projects and the process itself. Funding for 
technology development largely comes from the corporate level, with the 
research labs having responsibility for technology development until 
technology is matured and transitioned to the product line. This is 
critical to success, because it allows product developers to play a 
customer role. They are allowed to say no to technologies that are not 
ready for their programs and can focus on their job at hand--product 
integration, supplier management, and quality. 

DOD has adopted some of these practices. The department has begun using 
technology transition agreements and relationship managers and has 
initiated programs that place greater emphasis on technology transition 
planning. However, the reach of these initiatives is limited, and there 
is no unified, corporate approach to using them at this point. We 
recognize that the environment and incentives for DOD are very 
different than those of commercial firms we visited. For commercial 
practices to work on a broad scale, the DOD environment must be 
conducive for applying such practices. DOD senior leadership is a 
critical factor in providing this direction and vision as well as in 
maintaining the culture of the organization. The department must devote 
greater attention to strategic planning and technology development 
processes so that resources are spent on technologies that can and will 
be transformed into capabilities for the warfighter. In addition, the 
S&T and acquisition communities must work together to expand the use of 
technology transition agreements and relationship managers. Finally, 
the department should examine the way it currently funds technology 
transition. It may benefit from an examination of commercial companies' 
methods in this regard. They hold their technology developers 
accountable for delivering relevant technologies that are ready to be 
integrated into new products, but they empower them to succeed by 
providing adequate funding to get the job done. 

Recommendations for Executive Action: 

DOD should take steps to improve its transition of technologies to more 
efficiently deliver capabilities to its warfighters. The Defense 
Science Board has recommended strengthening of DOD's strategic planning 
function, and we believe this would move the department in the right 
direction. We believe a disciplined, gated approach for technology 
development, supported by technical and business criteria, would 
provide adequate knowledge to acquisition program managers about the 
risk of including particular technologies on specific weapons programs. 
It would also provide a more systematic way for the S&T community to 
continually assess the relevancy, feasibility, and potential transition 
commitment of its technologies and make decisions about future 
investments. DOD's current process lacks specific decision points with 
"go/no go" decisions. As such, we recommend that the Secretary of 
Defense take the following actions: 

* develop a gated process for developing and transitioning technologies 
that establishes a transition phase and defines activities that should 
occur during this phase, and: 

* include specific criteria to support continued funding of specific 
projects in that process. 

We also believe greater use of tools, such as technology transition 
agreements, relationship managers, and metrics, could help the 
department improve its ability to deliver mature technologies when 
needed, address transition issues more quickly, and gauge the impact of 
their science and technology investments and lab processes. Therefore, 
we recommend that the Secretary of Defense: 

* expand the use of technology transition agreements to applied and 
advanced development projects; 

* include additional metrics in technology transition agreements to 
provide S&T and acquisition program managers demonstrated knowledge 
about the manufacturing readiness, producibility, other benefits, and 
risks of including the technology on a weapons program; 

* expand the use of relationship managers by designating people at 
various levels in both the S&T and acquisition communities to address 
systemic transition issues and those related to specific weapon system 
programs. Also, define responsibilities for each level of relationship 
manager; 

* adopt additional process-oriented metrics, such as the percentage of 
advanced technologies that--once past milestone A of the acquisition 
process--transitioned into a weapons program or were fielded and the 
cycle time from milestone A to milestone B as a way to measure the 
effectiveness of S&T processes and the impact of science and technology 
investments; 

Commercial companies fund technology development and transition 
activities in their labs and hold the labs accountable for delivering 
mature technology to their product lines. As such, we are recommending 
that the Secretary of Defense: 

* Set aside a portion of advanced component development and prototype 
funds for the S&T community to manage the transition of technologies to 
acquisition programs. For this funding to be used effectively, it will 
require the discipline provided by corporate leadership in defining 
priorities, processes, and metrics. 

Agency Comments and Our Evaluation: 

DOD provided us with written comments on a draft of this report. DOD 
generally concurred with the recommendations in our report to improve 
its ability to transition technology to our warfighters. In doing so, 
it emphasized that that commercial industry program managers operate in 
an environment driven by profit and market opportunities, while DOD is 
organized to support our warfighters. We understand the basic 
differences in the environments, but believe there are many lessons 
that can be gleaned from our commercial visits. While it is true that 
private companies are motivated by profit, they can not achieve a 
profit without being focused on the timely delivery of products to the 
market. We note in this and other best practices reports that 
commercial firms have significant pressures on them to deliver cutting 
edge technologies quickly. They often bet their very existence on 
delivering new products and technologies before anyone else does so. In 
some cases, such as medical systems, their customers have urgent, life- 
or-death needs, similar to those of the warfighters, that demand very 
complex technological solutions. The real difference between these 
companies and the department is that they understand the futility of 
promising more than can be delivered to the market at any given point. 
They understand that, in the final analysis, they only succeed--or 
survive--only by delivering needed capability. On the other hand, the 
department's acquisition programs consistently accept immature 
technologies into product development that become a major cause of cost 
increases, schedule delays, and ultimately very late delivery to the 
warfighter who needs the equipment. The list of programs that have 
delivered late is extensive. 

Rather than focus on the different environment in the commercial world, 
DOD should focus on the practices used in the commercial world that 
allow it to deliver mature technology to product development (then, to 
the user) quickly and efficiently, an outcome woefully lacking in the 
department. 

Specially, of the six recommendations we made, DOD concurred with two 
recommendations, partially concurred with two others, and did not 
concur with the final two. DOD's comments appear in appendix II. 

DOD concurred with the recommendation that called for establishing a 
gated process for developing and transitioning technologies. In its 
response, the department stated that it has a gated process for 
programs that have a formal milestone A decision point and that 
technology maturity and technology readiness assessments are used in 
this process. DOD explained that the technology maturity assessments, 
in particular, often lead to risk mitigation plans, which include 
explicit gates that immature technologies must pass through to become 
qualified for adoption. 

We are not confident that DOD's implementation plans are fully 
responsive to this recommendation. First, the department does not have 
a process similar to those that we found at commercial companies to 
actively manage and make investment decisions across all technologies 
and determine if an individual technology is ready to transition. In 
its comments, DOD refers to a process for programs beginning at 
milestone A. However, we have reported that 80 percent of the programs 
we reviewed that passed milestone B since 2000 did not have a milestone 
A. 

Second, we found that even when technology readiness assessments 
indicated that technologies were not ready to be included in an 
acquisition program, DOD often decided to use them anyway. Third, risk 
mitigation plans used by DOD are subjective engineering judgments and 
are of limited value when evaluating the transition readiness of 
technologies. 

The department also concurred with our recommendation that it expand 
the use of technology transition agreements to applied and advanced 
development projects. However, in its response, DOD did not identify 
specific actions it plans to take to implement this recommendation. 

DOD partially concurred with the recommendation that it include 
additional metrics in technology transition agreements. In its 
response, the department indicated that it has taken steps to 
incorporate manufacturing readiness assessments as a means to improve 
technology readiness and that any additional metrics should be project- 
specific and included as exit criteria as part of the technology 
transition agreement. We are encouraged by DOD's recent development of 
manufacturing readiness levels and believe they should be a mandatory 
tool for the S&T and acquisition program managers to use to assess the 
manufacturability of a new technology. While we agree that the 
department should develop additional metrics on a project-by-project 
basis, we also believe that a more holistic approach, as depicted 
earlier in the report, is needed to determine the transition readiness 
of a particular technology. Transition readiness is not solely 
dependent upon the technical maturity achieved by S&T, but is also 
dependent upon the readiness of an acquisition program to accept the 
new technology and therefore should include an assessment of other 
factors such as cost, benefit, risk, scalability, and acquisition 
program endorsement. 

DOD also partially concurred with the recommendation to expand the use 
of relationship managers to address systemic transition issues and 
those related to specific weapon system programs. In its response, DOD 
indicated that it relies on written documents--the technology maturity 
assessments and technology transition agreements--to facilitate 
communication between the S&T and acquisition communities, particularly 
at the executive level. As stated earlier, these documents do not 
address many of the factors that could hinder transition. As DOD states 
in its comments, technology transition is referred to as a "body- 
contact sport." We agree, and this is why our recommendation is aimed 
at more direct person-to-person communication. Leading companies 
believe relationship managers play a key role in their ability to 
successfully transition technology. While DOD did acknowledge that its 
S&T investment would benefit from expanded emphasis by staff at the 
execution levels on technology transition issues, it did not directly 
address the need for midlevel relationship managers. We continue to 
believe DOD would benefit from the use of midlevel relationship 
managers. These managers provide the back and forth communications 
between the S&T and acquisition communities that help ensure that the 
right technology is being developed in the time frame needed. These 
managers are also uniquely positioned to ensure that technology is 
applied to multiple platforms when applicable and can identify systemic 
transition problems that should be addressed. 

DOD did not concur with the recommendation that it should develop 
additional process-oriented metrics, even though it stated that it is 
installing a process to do this through the use of technology maturity 
and technology readiness assessments. DOD stated that it does not want 
to commit to overly burdensome metrics that may be more oriented to 
measuring the process for the sake of measurement. Like DOD, we do not 
believe the department should develop metrics just for the sake of 
measurement. Rather, we believe DOD should develop and use metrics that 
allow DOD, Congress, and taxpayers to gauge the effectiveness of DOD 
S&T investments, which are expected to reach about $13 billion this 
year. The metrics that DOD references--those derived from technology 
maturity and technology readiness assessments--are project-level 
metrics rather than portfolio and process metrics that would allow the 
department to analyze its investment and make adjustments 
appropriately. 

DOD also did not concur with the recommendation to set aside a portion 
of advanced component development and prototype funds for the S&T 
community to manage the transition of technologies to acquisition 
programs. In its response, DOD stated that acquisition programs are 
best suited for transitioning technology because they have the training 
and discipline to field systems and have the responsibility to ensure a 
stable design, identify a responsive and responsible contractor, manage 
execution, and plan for life cycle support of the system. While DOD 
discusses transition to the warfighter, our recommendation is aimed at 
transitioning technology from S&T to acquisition programs. We found 
that in the commercial world, the training and discipline for 
transitioning technologies to product development is managed by the 
technology development community and is adequately funded. DOD believes 
its current approach of setting aside a small portion of S&T funds for 
transition through programs such as the Advanced Concept Technology 
Demonstration program, Joint Capability Technology Demonstration 
program, the Technology Transition Initiative, Manufacturing Technology 
program, and the Foreign Comparative Testing program is appropriate. We 
continue to believe DOD's approach to funding transition is flawed and 
that small pots of money for specific transition activities offer a 
piecemeal solution to a more systemic problem. DOD currently uses some 
of its advanced component development and prototype funds for S&T 
activities, including transition. However, acquisition programs 
typically carry out and fund these activities. We believe the S&T 
community should be responsible for these activities and as such, 
should be given the appropriate level of funding to carry them out. We 
believe DOD would be better positioned to develop and deliver weapon 
systems more quickly to its warfighters if the S&T community was 
responsible for developing, maturing, and transitioning mature 
technologies to the acquisition community and if the acquisition 
community focused solely on product development activities and 
delivering weapon systems to the warfighter That being said, we 
recognize that the acquisition and warfighting communities play 
critical roles in this process and therefore must continue to work 
toward setting realistic program requirements and establishing an 
evolutionary approach for developing new weapon programs. 

We are sending copies of this report to the Secretary of Defense, the 
Director of the Office of Management and Budget, and interested 
congressional committees. We will also make copies available at no 
charge on the GAO Web site at http://www.gao.gov. 

If you have any questions about this report or need additional 
information, please contact me at (202) 512-4841 or sullivanm@gao.gov. 
Contact points for our Offices of Congressional Relations and Public 
Affairs may be found on the last page of this report. Key contributors 
to this report were Karen Zuckerstein, Assistant Director; Cheryl 
Andrew; Lily Chin; Sameena Ismailjee; and Sean Merrill. 

Signed by: 

Michael Sullivan: 
Director: 
Acquisition and Sourcing Management: 

[End of section] 

Appendix I: Objectives, Scope, and Methodology: 

This report examines the Department of Defense's (DOD) efforts to 
improve its technology transition processes, with a focus on 
identifying specific management, funding, and organizational practices 
that could improve technology transition and weapon system outcomes. 
Specifically, our objectives were to (1) identify techniques that 
commercial companies use to transition mature technologies before the 
start of product development and (2) assess the extent to which DOD is 
using these techniques. 

We used a case study approach to compare and contrast DOD and leading 
commercial companies' practices. Companies were selected on the basis 
of such factors as the amount of money spent on research and 
development activities over the past several years and the percentage 
of change in research and development spending. For the most part, we 
selected Fortune 500 companies that were in the top 100 for research 
and development spending and had not experienced major cutbacks in this 
funding. We also took into account the type of products each company 
develops and selected companies representing several different business 
sectors. Below are descriptions of the four companies featured in this 
report. 

Boeing: 

Boeing is a leading aerospace company and the largest manufacturer of 
commercial jetliners and military aircraft combined, with capabilities 
in rotorcraft, electronic and defense systems, missiles, satellites, 
launch vehicles, and advanced information and communication systems. 
Boeing Phantom Works develops advanced systems solutions, such as 
advanced homeland security and air traffic management, as well as 
breakthrough technologies, such as advanced avionics and composite 
materials that are intended to significantly improve the performance, 
quality, and affordability of aerospace products and services. We met 
with research officials, as well as product line officials representing 
both its commercial and defense sectors in Seattle, Washington. 

3M: 

3M is a diversified technology company with a worldwide presence in 
various markets, including consumer and office; display and graphics; 
electronics and communications; health care; industrial; safety, 
security, and protection services; and transportation. With more than 
55,000 products, 3M invests more than $1 billion annually in research 
and development and related activities associated with 30-plus core 
technologies. The company was awarded nearly 500 U.S. patents in 2005. 
We met with research, business unit, and government programs officials 
at 3M headquarters in St. Paul, Minnesota. 

IBM: 

IBM is one of the world's largest technological companies, spending 
about $3 billion annually on research and development activities. It is 
the largest supplier of hardware, software, and information technology 
services and pioneered the development and implementation of on-demand 
business. With 3,248 U.S. patents, IBM earned more patents than any 
other company for the 12th consecutive year in 2004. In the past 4 
years, IBM inventors received more than 13,000 patents--approximately 
5,400 more than any other patent recipient. We met with research and 
product development officials at the Watson Research Center in 
Hawthorne, New York. 

Motorola: 

Motorola is a Fortune 500 global communications leader that provides 
seamless mobility products and solutions across broadband, embedded 
systems and wireless networks for products in homes, automobiles, and 
workplaces. Motorola spent about $3.5 billion in 2005 on research and 
development activities. Approximately 22,000 professional employees 
were engaged in research activities during 2005. We met with research 
officials and product development officials, as well as partnership 
development managers at its offices in Schaumburg, Illinois. 

For each of the companies, we interviewed senior management officials 
knowledgeable about research and development activities to gather 
consistent information about processes, practices, and metrics the 
companies use to transition technology smoothly. In particular, we 
discussed their (1) strategic planning process for identifying and 
prioritizing customer needs, (2) technology development process used to 
fund and mature technologies required to meet customer needs, (3) tools 
used to facilitate communication between labs and product lines to 
transition technology, and (4) technology transition process, including 
when transition occurs, the organizations involved, and how technology 
is funded throughout the transition phase. We synthesized information 
from GAO's past best practices work about technology and product 
development. 

To determine DOD's practices for transitioning technology, we met with 
science and technology (S&T) and acquisition officials to discuss the 
same categories of questions listed above that we asked leading 
commercial companies. The following is a list of the organizations we 
met with: 

* Director, Defense Research and Engineering, Alexandria, Virginia: 

- Director, Plans and Programs: 

- Deputy Under Secretary of Defense, Advanced Systems and Concepts: 

- Director, Defense Advanced Research Projects Agency: 

* Air Force: 

- Office of the Assistant Secretary of the Air Force for Acquisition 
(Science, Technology, and Engineering), Alexandria, Virginia: 

- Air Force Material Command, Wright Patterson Air Force Base, Ohio: 

- Air Force Research Labs, Wright Patterson Air Force Base, Ohio: 

- Joint Unmanned Combat Air Systems Program Office, Wright Patterson 
Air Force Base, Ohio: 

* Army: 

- Assistant Secretary of the Army Acquisition, Logistics and 
Technology, Alexandria, Virginia: 

- Deputy Assistant Secretary for Research and Technology Chief 
Scientist, Alexandria, Virginia: 

- Joint Tactical Radio System Cluster 5 Program Office, Alexandria, 
Virginia: 

* Navy: 

- Deputy Assistant Secretary of Navy for Research, Development, Test 
and Evaluation, Alexandria, Virginia: 

- Office of Naval Research, Alexandria, Virginia: 

- Naval Air Systems Command, Patuxent River Naval Air Station, 
Maryland: 

- Naval Sea Systems Command, Washington Navy Yard, D.C. 

- CVN-21 Program Office, Washington Navy Yard, D.C. 

- Multi-mission Maritime Aircraft Program Office, Patuxent River Naval 
Air Station, Maryland: 

- Aircraft Launch and Recovery Equipment Program Office, Patuxent River 
Naval Air Station, Maryland: 

At each of these locations, we collected appropriate documents that 
describe the various programs, organizations, responsibilities, and 
funding. We obtained examples of key documents, such as technology 
transition agreements or memorandums of agreement that are used to 
solidify agreements made between labs and weapon programs. We reviewed 
DOD and military service strategic plans and research, development, 
test, and evaluation funding documents. In addition, we reviewed 
documents required as part of the selection and oversight process for 
the Air Force Advanced Technology Development program, the Army 
Technology Objectives program, and the Future Naval Capabilities 
program. We relied on previous GAO best practices and weapon system 
reports that highlight cost and schedule impacts of launching new 
weapon programs with immature technology. A list of these reports can 
be found at the end of this report. 

[End of section] 

Appendix II: Comments from the Department of Defense: 

Office Of The Director Of Defense Research And Engineering: 
3040 Defense Pentagon Washington, D.C. 20301-3040: 

SEP 5 2006: 

Mr. Michael J. Sullivan: 
Director, Acquisition and Sourcing Management: 
U.S. Government Accountability Office: 
441 G Street, N. W. 
Washington, DC 20548: 

Dear Mr. Sullivan: 

This is the Department of Defense (DoD) response to the General 
Accountability Office (Gao) Draft Report, "Best Practices: Stronger 
Practices Needed to Improve DOD Technology Transition Processes," dated 
August 2006 (GAO Code 120454/ GAO-06-883). 

While we generally concur with recommendations in the report that would 
encourage DoD to improve its ability to transition technology to our 
warfighters, we believe it is important to recognize there are 
differences between the Government and industry. Industry program 
managers operate in an environment driven by profit and market 
opportunities while DoD is organized to support our warfighters. Some 
of the ways DoD provides warfighter capabilities is through technology 
upgrades, leap-aheads, and transformation driven by the near and mid- 
term operational priorities. The essential governance process of annual 
appropriations and authorizations is also a factor that private sector 
technology investment does not experience. We adopt industry best 
practices where practical and in a manner consistent with both our 
legislative mandates and the DoD mission. Comments are provided at the 
enclosure. 

The staff has worked closely with the GAO team and appreciates their 
inputs for transitioning technology to the warfighter. Thank you for 
the opportunity to comment on the draft report. 

Sincerely, 

Signed by: 

Alan R. Shaffer: 
Director, Plans and Program: 

GAO Draft Report Mated August 2006 GAO Code 120454/gao-06-883: 

"Best Practices: Stronger Practices Needed to Improve DOD Technology 
Transition Processes" 

Department Of Defense Comments To The Recommendations: 

The Department has reviewed the GAO's report concerning implementation 
of industry best practices for technology transition and partially 
agrees with the findings. However, as discussed during the exit 
briefing, private sector industry program managers operate in a 
significantly different environment than DoD managers, with profit 
objectives driving commercial investments, vice our DoD-mission focus 
on meeting warfighter objectives for technology upgrades, leap-aheads, 
and transformation. These objectives drive many of the inherent 
cultural changes which we are continuing to address with periodic 
process changes across multiple programs and organizations in the 
Department. 

Recommendation 1: The GAO recommended that the Secretary of Defense 
develop a gated process for developing and transitioning technologies 
that establishes a transition phase and defines activities that should 
occur during this phase, and include specific criteria to support 
continued funding of specific projects in that process. (p. 32/GAO 
Draft Report): 

DOD Response: Concur. 

The Department has, and recognizes the value of, a gated process for 
programs that have a formal Milestone A. That process involves the use 
of Technology Maturity Assessments (TMA's) and Technology Readiness 
Assessments (TRA's). This formal process is under the responsibility of 
the Science and Technology community and helps ensure that critical 
technologies are proven prior to delivery to the Acquisition community. 
The use of TMAs and TRAs has been in use for about two years, and we 
would expect that another 4-5 years of assessments will be needed in 
order to realize the desired results. TMA's often lead to technology 
risk mitigation plans, which include explicit "gates" that immature 
technologies must pass through to become qualified for adoption. The 
Department also supports technology transition agreements, signed by 
the developer and acquisition managers, with specific, mutually agreed 
upon exit criteria, as a viable alternative for those programs that 
have transition as a specific objective. 

Recommendation 2: The GAO recommended that the Secretary of Defense 
expand the use of technology transition agreements to applied and 
advanced development projects. 
(p. 32/GAO Draft Report): 

DOD Response: Concur. 

Technology transition agreements have been used successfully by the 
Services and OSD to document and facilitate technology transition 
between the Science & Technology (S&T) and acquisition communities. 

Recommendation 3: The GAO recommended that the Secretary of Defense 
include additional metrics in technology transition agreements to 
provide S&T and acquisition program managers demonstrated knowledge of 
manufacturing readiness, producibility, other benefits, and risks of 
including the technology on a weapons program. (p. 32/GAO Draft 
Report): 

DOD Response: Partially Concur. 

The Department has taken steps to incorporate manufacturing readiness 
assessments as a means to improve technology transition. Recently the 
Joint Defense Manufacturing Technology Panel, with the support of 
industry and the defense acquisition community, developed a set of 
Manufacturing Readiness Level (MRL) definitions and supporting tools. 
MRLs are designed to serve the same purpose for manufacturing risk/ 
maturity as the Technology Readiness Levels (TRLs) serve for technology 
readiness, and incorporate a similar numbering system. They are 
described in the DoD Technology Readiness Assessment Deskbook, at 
http://www.dod.mil/ddre/doc.tra_deskbook.pdf Any additional metrics 
should be project specific and determined by the S&T and acquisition 
managers, and be documented as mutually agreed upon exit criteria as 
part of the technology transition agreement. 

Recommendation 4: The GAO recommended that the Secretary of Defense 
expand the use of relationship managers by designating people at 
various levels in both the S&T and acquisition communities to address 
systemic transition issues and those related to specific weapon system 
programs. Also, define responsibilities for each level of relationship 
manager. (p. 32/GAO Draft Report): 

DOD Response: Partially Concur. 

The DoD already has a process in place with defined responsibilities 
for both the S&T and acquisition communities. TMAs are provided to the 
Program Executive Officers, the Defense Acquisition Board (DAB), and 
the Overarching Integrated Process Teams for Acquisition Category 1 
(ACAT 1) programs. The TMAs are part of the DAB reviews to ensure a 
transition path is established. For technology not part of an ACAT 1 
program, the Department uses Technology Transition Agreements to 
accomplish this purpose thereby causing the S&T and Acquisition 
managers to negotiate exit criteria that must be met before the 
technology transitions. Technology transition is frequently referred to 
as a "body-contact sport," in recognition of the important role that 
personal communication and relationships usually play in successful 
efforts. Throughout DoD'S S&T enterprise, there are many staff who 
focus on fostering transitions by establishing relationships between 
technology developers and potential technology users, and by breaking 
down barriers to technology transition. Many of these efforts are 
successful and move certain technologies forward from research to 
adoption reasonably quickly. However, it's probably a fair observation 
that a substantial fraction of DOD's S&T investment would benefit from 
expanded emphasis by staff at the execution level in the labs, 
Agencies, and Program Managers, etc. in terms of technology transition 
versus technology development. 

Recommendation 5: The GAO recommended that the Secretary of Defense 
adopt additional process-oriented metrics, such as the percentage of 
advanced technologies that--once past Milestone A of the acquisition 
process-transitioned onto a weapons program or were fielded and the 
cycle time from Milestone A to Milestone B as a way to measure the 
effectiveness of S&T processes and the impact of science and technology 
investments. 
(p. 32/GAO Draft Report): 

DOD Response: Nonconcur. 

We are installing a process to do this through the use of TMAs and 
TRAs. As we mature this process over the next 4-5 years, we would not 
want to commit to overly burdensome metrics that may be more oriented 
to measuring the process for the sake of measurement rather than using 
this process to improve transition. Technology has transitioned when: 
it has moved into an acquisition program of record; it can be acquired/ 
procured through normal DoD procurement programs; it has addressed or 
met the requirements of DoD doctrine, operations, training, 
maintenance, procurement, logistics, and facilities and; it provides 
sustainable capability. With this transition we also transition from 
S&T funding to Acquisition funding to reflect the proper alignment of 
programs and funding. Technology transition is the result of a 
collaboration between the requirements, technology, and acquisition 
communities. Each of the Military Services have a process in place to 
bring these collaborative elements together. The technology is already 
certified at Acquisition Milestone A with a technology pathway for 
achieving Technology Readiness Level (TRL) 6 at Acquisition Milestone 
B. 

Recommendation 6: The GAO recommended that the Secretary of Defense set 
aside a portion of advanced component development and prototype funds 
for the S&T community to manage the transition of technologies to 
acquisition programs. (p. 32/GAO Draft Report): 

DOD Response: Nonconcur. 

GAO recognizes the transition programs established by the Department to 
make technologies ready for transition, including the Advanced Concept/ 
Joint Concept Technology Demonstrations (AC/JCTDs), Technology 
Transition Initiative (TTI), Manufacturing Technology (ManTech) and 
Foreign Comparative Testing (FCT). But these, in practice, reflect a 
collaborative contract between the S&T community, the acquisition 
community procurement program of record, and the warfighter to deploy 
the technology when it is ready to be fielded. In addition, the 
training and discipline to field systems resides in the acquisition 
community, as they have the responsibility to ensure a stable design, 
identify a responsive and responsible contractor, manage execution, and 
plan for life cycle support of the system. While there may be examples 
of stand-alone systems where the S&T community could manage the 
transition to a fielded system, this is not a desirable or feasible 
approach for larger system-of-system development programs with broad 
involvement from many stakeholders and longer system life cycles. 
However, to assist the transition of those technologies not 
specifically tied to a program of record, the DDR&E invests 
approximately $500 million annually in programs like but not limited to 
ACTDs/JCTDs, TTI, ManTech, and FCT as mentioned above. Accordingly, DoD 
is setting aside a portion of S&T funds for transition. We believe the 
current balance is correct, but we will continue to assess to assure 
ready technologies are offered to our warfighters rapidly, efficiently, 
and affordably. 

[End of section] 

Appendix III: DOD Research, Development, Technology, and Engineering 
Budget: 

Dollars in billions. 

Name: Basic research; 
Budget activity: 1; 
Description: Basic research is systematic study directed toward greater 
knowledge or understanding of the fundamental aspects of phenomena and 
of observable facts without specific applications towards processes or 
products in mind. It includes all scientific study and experimentation 
directed towards increasing fundamental knowledge and understanding in 
those fields of the physical, engineering, environmental, and life 
sciences related to long-term national security needs. It is farsighted 
high-payoff research that provides the basis for technological 
progress; 
Which DOD community controls spending: S&T; 
Budget (fiscal year 2006): $1.5. 

Name: Applied research; 
Budget activity: 2; 
Description: Applied research is systemic study to understand the means 
to meet a recognized and specific need. It is a systematic expression 
and application of knowledge to develop useful materials, devices, and 
systems or methods. Applied research may translate promising basic 
research into solutions for broadly defined military needs, short of 
system development. Applied research precedes system- specific 
technology investigations or development; 
Which DOD community controls spending: S&T; 
Budget (fiscal year 2006): $5.2. 

Name: Advanced technology development; 
Budget activity: 3; 
Description: Advanced technology development includes development of 
subsystems and components and efforts to integrate them into system 
prototypes for field experiments and/or tests in a simulated 
environment. The results of this type of effort are proof of 
technological feasibility and assessment of subsystem and component 
operability and producibility rather than the development of hardware 
for service use. Projects in this category have a direct relevance to 
identified military needs. Program elements in this category involve 
pre-acquisition efforts, such as system concept demonstration, joint 
and service-specific experiments, or technology demonstrations, and 
generally have technology readiness levels (TRLs) of 4, 5, or 6. 
Projects in this category do not necessarily lead to subsequent 
development or procurement phases, but should have the goal of moving 
out of science and technology and into the acquisition process within 
years defense program. Upon successful completion of projects that have 
military utility, the technology should be available for transition; 
Which DOD community controls spending: S&T; 
Budget (fiscal year 2006): $6.6. 

Name: Advanced component development and prototypes; 
Budget activity: 4; 
Description: Advanced component development and prototypes consists of 
efforts necessary to evaluate integrated technologies or prototype 
systems in a high fidelity and realistic operating environment. These 
activities include system-specific efforts that help expedite 
technology transition from the laboratory to operational use. Emphasis 
is on proving component and subsystem maturity prior to integration in 
major and complex system sand may involve risk reduction initiatives. 
Advanced component development and prototypes efforts are to occur 
before an acquisition program starts product; 
Which DOD community controls spending: Acquisition; 
Budget (fiscal year 2006): $13.9. 

Name: System development and demonstration; 
Budget activity: 5; 
Description: System development and demonstration consists of newly 
initiated acquisition programs and includes engineering and 
manufacturing development tasks aimed at meeting validated requirements 
prior to full-rate production. Characteristics of this activity involve 
mature system development, integration, and demonstration to support a 
production decision; 
Which DOD community controls spending: Acquisition; 
Budget (fiscal year 2006): $19.3. 

Name: Research, development, test and evaluation management support; 
Budget activity: 6; 
Description: RDT&E management support includes efforts to sustain 
and/or modernize the installations or operations required for general 
RDT&E. Such efforts may related to test ranges, military construction, 
maintenance support of laboratories, operation and maintenance of test 
aircraft and ships, and studies and analyses I support of the RDT&E 
program; 
Which DOD community controls spending: Acquisition; 
Budget (fiscal year 2006): $4.0. 

Name: Operational system development; 
Budget activity: 7; 
Description: Operational system development includes development 
efforts to upgrade systems that have been fielded or have received 
approval for full-rate production and anticipate production funding in 
the current or subsequent fiscal year; 
Which DOD community controls spending: Acquisition; 
Budget (fiscal year 2006): $20.6. 

Source: DOD (data); GAO (presentation and analysis). 

[End of table] 

[End of section] 

Appendix IV: DOD Technology Readiness Levels: 

Technology readiness level: 1. Basic principles observed and reported; 
Description: Lowest level of technology readiness. Scientific research 
begins to be translated into applied research and development. Examples 
might include paper studies of a technology's basic properties. 

Technology readiness level: 2. Technology concept and/or application 
formulated; 
Description: Invention begins. Once basic principles are observed, 
practical applications can be invented. Applications are speculative 
and there may be no proof or detailed analysis to support the 
assumption. Examples are still limited to analytic studies. 

Technology readiness level: 3. Analytical and experimental critical 
function and/or characteristic proof of concept; 
Description: Active research and development is initiated. This 
includes analytical studies and laboratory studies to physically 
validate analytical predictions of separate elements of the technology. 
Examples include components that are not yet integrated or 
representative. 

Technology readiness level: 4. Component and/or breadboard. Validation 
in laboratory environment; 
Description: Basic technological components are integrated to establish 
that they will work together. This is relatively "low fidelity" 
compared to the eventual system. Examples include integration of "ad 
hoc" hardware in a laboratory. 

Technology readiness level: 5. Component and/or breadboard validation 
in a relevant environment; 
Description: Fidelity of breadboard technology increases significantly. 
The basic technological components are integrated with reasonably 
realistic supporting elements so it can be tested in a simulated 
environment. Examples include "high fidelity" laboratory integration of 
components. 

Technology readiness level: 6. System/subsystem model or prototype 
demonstration in a relevant environment; 
Description: Representative model or prototype system, which is well 
beyond that of TRL 5, is tested in a relevant environment. Represents a 
major step up in a technology's demonstrated readiness. Examples 
include testing a prototype in a high-fidelity laboratory environment 
or in simulated operational environment. 

Technology readiness level: 7. System prototype demonstration in an 
operational environment; 
Description: Prototype near, or at, planned operational system. 
Represents a major step up from TRL 6, requiring demonstration of an 
actual system prototype in an operational environment such as in an 
aircraft, vehicle, or space. Examples include testing the prototype in 
a test bed aircraft. 

Technology readiness level: 8. Actual system completed and qualified 
through test and demonstration; 
Description: Technology has been proven to work in its final form and 
under expected conditions. In almost all cases, this TRL represents the 
end of true system development. Examples include developmental test and 
evaluation of the system in its intended weapon system to determine if 
it meets design specifications. 

Technology readiness level: 9. Actual system proven through successful 
mission operations; 
Description: Actual application of the technology in its final form and 
under mission conditions, such as those encountered in operational test 
and evaluation. Examples include using the system under operational 
mission conditions. 

Source: DOD (data); GAO (presentation and analysis). 

[End of table] 

[End of section] 

Related GAO Products: 

Defense Acquisitions: Assessments of Selected Major Weapon Programs. 
GAO-06-391. Washington, D.C.: March 31, 2006. 

Best Practices: Better Support of Weapon System Program Managers Needed 
to Improve Outcomes. GAO-06-110. Washington, D.C.: November 1, 2005. 

DOD Acquisition Outcomes: A Case for Change. GAO-06-257T. Washington, 
D.C.: November 15, 2005. 

Defense Acquisitions: Stronger Management Practices Are Needed to 
Improve DOD's Software-Intensive Weapon Acquisitions. GAO-04-393. 
Washington, D.C.: March 1, 2004. 

Best Practices: Setting Requirements Differently Could Reduce Weapon 
Systems' Total Ownership Costs. GAO-03-57. Washington, D.C.: February 
11, 2003. 

Defense Acquisitions: Factors Affecting Outcomes of Advanced Concept 
Technology Demonstration. GAO-03-52. Washington, D.C.: December 2, 
2002. 

Best Practices: Capturing Design and Manufacturing Knowledge Early 
Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: July 15, 
2002. 

Defense Acquisitions: DOD Faces Challenges in Implementing Best 
Practices. GAO-02-469T. Washington, D.C.: February 27, 2002. 

Best Practices: Better Matching of Needs and Resources Will Lead to 
Better Weapon System Outcomes. GAO-01-288. Washington, D.C.: March 8, 
2001. 

Best Practices: A More Constructive Test Approach Is Key to Better 
Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.: July 31, 
2000. 

Defense Acquisition: Employing Best Practices Can Shape Better Weapon 
System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.: April 26, 2000. 

Best Practices: DOD Training Can Do More to Help Weapon System Programs 
Implement Best Practices. GAO/NSIAD-99-206. Washington, D.C.: August16, 
1999. 

Best Practices: Better Management of Technology Development Can Improve 
Weapon System Outcomes. GAO/NSIAD-99-162. Washington, D.C.: July 30, 
1999. 

Defense Acquisitions: Best Commercial Practices Can Improve Program 
Outcomes. GAO/T-NSIAD-99-116. Washington, D.C.: March 17, 1999. 

Defense Acquisition: Improved Program Outcomes Are Possible. GAO/T- 
NSIAD-98-123. Washington, D.C.: March 17, 1998. 

Best Practices: DOD Can Help Suppliers Contribute More to Weapon System 
Programs. GAO/NSIAD-98-87. Washington, D.C.: March 17, 1998. 

Best Practices: Successful Application to Weapon Acquisition Requires 
Changes in DOD's Environment. GAO/NSIAD-98-56. Washington, D.C.: 
February 24, 1998. 

Best Practices: Commercial Quality Assurance Practices Offer 
Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.: August 26, 
1996. 

FOOTNOTES 

[1] This represents the amount of money Congress authorized DOD for 
basic research, applied research, and advanced technology development. 

[2] GAO, Defense Acquisitions: Assessments of Selected Major Weapon 
Programs. GAO-06-391 (Washington, D.C.: March 31, 2006). 

[3] GAO, Best Practices: Better Management of Technology Development 
Can Improve Weapon System Outcomes. GAO/NSIAD-99-162 (Washington, D.C.: 
July 30, 1999) and Best Practices: Better Matching of Needs and 
Resources Will Lead to Better Weapon System Outcomes GAO-01-288 
(Washington, D.C.: March 8, 2001). Other best practices reports are 
listed in the Related GAO Products section at the back of this report. 

[4] GAO, Best Practices: Better Support of Weapon System Program 
Managers Needed to Improve Outcomes. GAO-06-110 (Washington, D.C.: Nov. 
1, 2005). 

[5] GAO-06-391 

[6] 10 U.S.C § 2433 establishes the requirement for unit cost reports 
if certain thresholds for program costs are exceeded (known as unit 
cost or Nunn-McCurdy breaches). DOD is required to report to Congress 
and, if applicable, certify the program to Congress. 

[7] Numerous GAO reports address issues surrounding product 
development. See Related GAO Products at the end of this report. 

[8] Boeing has two primary businesses: Boeing Commercial Airplanes and 
Boeing Integrated Defense systems. Both businesses are supported by the 
same science and technology lab. However, the two businesses have 
different tools for transitioning technology. 

[9] DDRE is the principal staff adviser to the Under Secretary of 
Defense for Acquisition, Technology, and Logistics and the Secretary 
and Deputy Secretary of Defense for research and engineering matters. 
DDRE serves as the chief technology officer for the department. 

[10] DOD. Defense Science Board Task Force on The Roles and Authorities 
of the Director of Defense Research and Engineering, (Washington, D.C.: 
Oct. 28, 2005). 

[11] GAO, Defense Acquisitions: Major Weapon Systems Continue to 
Experience Cost and Schedule Problems under DOD's Revised Policy. GAO-
06-368 (Washington, D.C.: April 13, 2006). 

[12] GAO-06-368. 

[13] GAO, Defense Technology Development: Management Process Can Be 
Strengthened for New Technology Transition Programs. GAO-05-480 
(Washington, D.C.: June 17, 2005). 

GAO's Mission: 

The Government Accountability Office, the investigative arm of 
Congress, exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability 
of the federal government for the American people. GAO examines the use 
of public funds; evaluates federal programs and policies; and provides 
analyses, recommendations, and other assistance to help Congress make 
informed oversight, policy, and funding decisions. GAO's commitment to 
good government is reflected in its core values of accountability, 
integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains 
abstracts and full-text files of current reports and testimony and an 
expanding archive of older products. The Web site features a search 
engine to help you locate documents using key words and phrases. You 
can print these documents in their entirety, including charts and other 
graphics. 

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as "Today's Reports," on its 
Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order 
GAO Products" heading. 

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to: 

U.S. Government Accountability Office 

441 G Street NW, Room LM 

Washington, D.C. 20548: 

To order by Phone: 

Voice: (202) 512-6000: 

TDD: (202) 512-2537: 

Fax: (202) 512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm 

E-mail: fraudnet@gao.gov 

Automated answering system: (800) 424-5454 or (202) 512-7470: 

Public Affairs: 

Jeff Nelligan, managing director, 

NelliganJ@gao.gov 

(202) 512-4800 

U.S. Government Accountability Office, 

441 G Street NW, Room 7149 

Washington, D.C. 20548: