This is the accessible text file for GAO report number GAO-06-634 
entitled 'NASA's James Webb Space Telescope: Knowledge-Based 
Acquisition Approach Key to Addressing Program Challenges' which was 
released on July 13, 2006. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to Congressional Committees: 

United States Government Accountability Office: 

GAO: 

July 2006: 

NASA'S James Webb Space Telescope: 

Knowledge-Based Acquisition Approach Key to Addressing Program 
Challenges: 

James Webb Space Telescope: 

GAO-06-634: 

GAO Highlights: 

Highlights of GAO-06-634, a report to congressional committees 

Why GAO Did This Study: 

The National Aeronautics and Space Administrationís (NASA) James Webb 
Space Telescope (JWST) is being designed to explore the origins and 
nature of the universe. It should allow scientists to look deeper into 
spaceóand thus farther back in timeóthan ever before. The program, 
however, has experienced cost growth of more than $1 billion and its 
schedule has slipped nearly 2 years. NASA recently restructured the 
program and now anticipates a launch no sooner than June 2013. Because 
of the cost and schedule problems, under the Comptroller Generalís 
authority, we reviewed the JWST program to determine the extent to 
which this procurement follows NASA acquisition policy and GAO best 
practices for ensuring that adequate product knowledge is used to make 
informed investment decisions 

What GAO Found: 

Although the JWST program recently revised its acquisition strategy to 
conform to NASAís acquisition policies, the program still faces 
considerable challenges because it has not fully implemented a 
knowledge-based approach, which our past work has shown is often a key 
factor in program success. In a recent report, we made recommendations 
that NASA take steps to ensure that projects follow a knowledge-based 
approach for product development. NASA concurred and revised its 
acquisition policy. When we initiated our work and before the JWST 
programís recently revised acquisition strategy, program officials 
intended to have NASA commit to program start, which is the end of the 
formulation phase and the beginning of the implementation phase, with 
immature technologies, according to best practices, and without a 
preliminary design. During our review, we discussed these shortfalls 
with NASA officials, and they revised their acquisition strategy to 
conform to NASA policy. However, the current strategy still does not 
fully incorporate a knowledge-based approach which ensures that 
resources match requirements in terms of knowledge, time, and money 
before program start. If program officials follow the current plan, the 
maturity of key technologies may not be adequately tested prior to 
program start. In addition, it appears the program will not have 
sufficient funding resources to ensure the programís success. In light 
of the fiscally constrained environment the federal government and NASA 
will face in the years ahead, adopting a knowledge-based approach will 
not only increase the JWST programís chances for success but also lay 
the foundation for comparison between competing programs. 

Figure: Conceptual Drawing of NASA's JWST. 

[See PDF for Image] 

[End of Figure] 

What GAO Recommends: 

GAO recommends that the NASA administrator: (1) direct the JWST program 
to fully apply a knowledge-based acquisition approach to ensure that 
adequate knowledge is attained at key decision points and also to hold 
the program accountable and 
(2) instruct the JWST program to continue to adhere to NASA acquisition 
policy and go forward only after demonstrating that it is meeting 
incremental knowledge markers and has sufficient funds to execute the 
program. NASA concurred with GAOís recommendations. 

[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-634]. 

To view the full product, including the scope and methodology, click on 
the link above. For more information, contact Allen Li at (202) 512-
4841 or lia@gao.gov. 

[End of Section] 

Contents: 

Letter: 

Results in Brief: 

Background: 

JWST's Revised Strategy Does Not Fully Incorporate a Knowledge-Based 
Approach That Could Reduce Risks and Better Inform Decision Making: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Scope and Methodology: 

Appendix II: Technology Readiness Levels: 

Appendix III: Comments from the National Aeronautics and Space 
Administration: 

Appendix IV: GAO Contact and Staff Acknowledgments: 

Related GAO Products: 

Figures: 

Figure 1: Conceptual Drawing of NASA's JWST: 

Figure 2: Comparison of NASA's Life Cycle with a Knowledge-Based 
Acquisition Life Cycle: 

Figure 3: Technology Maturity Levels for Product Development: 

Abbreviations: 

CSA: Canadian Space Agency: 
ESA: European Space Agency: 
JWST: James Webb Space Telescope: 
KP1: Knowledge point 1: 
NARN: on-advocate Review: 
NASA: National Aeronautics and Space Administration: 
PDR: Preliminary Design Review: 
TRL: Technology Readiness Levels: 

United States Government Accountability Office: 
Washington, DC 20548: 

July 14, 2006: 

Congressional Committees: 

As the expected follow-on to the tremendously successful Hubble Space 
Telescope, the National Aeronautics and Space Administration's (NASA) 
James Webb Space Telescope (JWST) is being designed to explore the 
early universe and allow scientists to shed light on the origins and 
nature of the universe by allowing them to look deeper into space--and 
thus farther back in time--than ever before. Recently, however, NASA 
acknowledged that the program[Footnote 1] has experienced cost growth 
exceeding $1 billion--which increased its life-cycle cost estimate from 
$3.5 billion to $4.5 billion--and its schedule has slipped nearly 2 
years. The agency restructured the program and is now anticipating a 
launch no sooner than June 2013. 

Because of the restructuring and past cost and schedule problems, we 
reviewed the program to determine the extent to which the JWST 
program's acquisition strategy follows NASA acquisition policy and 
Government Accountability Office (GAO) best practices for ensuring that 
adequate product knowledge is used to make informed investments. We 
conducted our work under the Comptroller General's authority and are 
addressing this report to you because of your committee's or 
subcommittee's interest in NASA activities. 

To assess the extent to which the JWST acquisition strategy follows 
NASA policy and GAO best practices for ensuring readiness to proceed 
into implementation, we reviewed NASA policy guidance and compared it 
with the JWST program's acquisition strategy. We also benchmarked the 
JWST acquisition strategy to best practices. We interviewed NASA and 
contractor officials to clarify our understanding of the program's 
management approach and technology development plan. We analyzed cost 
and schedule information and discussed the impact of the investment in 
the JWST on other NASA programs with NASA officials. We attended two 
design reviews, including one at the prime contractor's facility. We 
performed our review from August 2005 through May 2006 in accordance 
with generally accepted government auditing standards. 

Results in Brief: 

Although the JWST program recently revised its acquisition strategy to 
conform to NASA's acquisition policies, the program still faces 
considerable challenges because it has not fully implemented a 
knowledge-based approach. Our past work on the best practices of 
product developers in government and industry has found that using a 
knowledge-based approach is often a key factor in program success. We 
recently made recommendations that NASA take steps to ensure that 
projects follow a knowledge-based approach for product 
development.[Footnote 2] NASA concurred and revised its acquisition 
policy. When we initiated our work and before the JWST program's 
recently revised acquisition strategy, program officials intended to 
have NASA commit to the program and start implementation with immature 
technologies, according to best practices, and without a preliminary 
design. During our review, we discussed these shortfalls with NASA 
officials, and they revised their acquisition strategy to conform to 
NASA policy. However, the current strategy still does not fully 
incorporate a knowledge-based approach that ensures that resources 
match requirements in terms of knowledge, time, and money before 
program start, which is the end of the formulation phase and the 
beginning of the implementation phase. If program officials follow the 
current plan, the maturity of key technologies may not be adequately 
tested prior to program start. For example, a test to demonstrate 
critical performance parameters is scheduled to occur after the program 
start decision and some planned test items may not provide the validity 
needed to adequately verify technology maturity. In addition, it 
appears the program will not have sufficient funding resources to 
ensure the program's success. According to a review conducted by NASA's 
Independent Program Assessment Office, the program's contingency 
funding is too low and phased in too late in the program to support the 
planned launch date and provide the necessary resources to address as 
yet unforeseen problems. In light of the fiscally constrained 
environment the federal government and NASA will face in the years 
ahead, adopting a knowledge-based approach will not only increase the 
JWST program's chances for success but also lay the foundation for 
comparison between competing programs. As more programs, such as the 
JWST, move into implementation, using a knowledge-based approach will 
allow NASA to assess these development efforts in a consistent format 
to confirm the continued viability of the investment. 

To increase the JWST program's chances of successful product 
development and to better inform NASA's decision-making process, we are 
recommending that the NASA Administrator (1) direct the JWST program to 
apply a knowledge-based acquisition approach, including incremental 
markers, to ensure that adequate knowledge is attained at key decision 
points and to hold the program accountable and (2) instruct the JWST 
program to continue to adhere to NASA acquisition policy and base the 
program's go/no-go decision not only on adherence to that policy, but 
also on demonstrating that it is meeting incremental knowledge markers 
and that adequate funds are available to execute the program. 

In written comments on a draft of this report, NASA concurred with our 
recommendations. NASA's comments are included in their entirety in 
appendix III. 

Background: 

The JWST--identified by the National Research Council as the top 
priority new initiative for astronomy and physics for the current 
decade--is a large deployable space-based observatory being developed 
to study and answer fundamental questions ranging from the formation 
and structure of the universe to the origin of planetary systems and 
the origins of life. Often referred to as the replacement to Hubble, 
the JWST is more of a next generation telescope--one that scientists 
believe will be capable of seeing back to the origins of the universe 
(Big Bang). The JWST will have a large, segmented primary mirror--6.5 
meters (about 21 feet) in diameter--which is a leap ahead in technology 
over the last generation of mirrors. The observatory requires a 
sunshield approximately the size of a tennis court to allow it to cool 
to the extremely cold temperature (around 40 degrees Kelvin, or minus 
388 degrees Fahrenheit) necessary for the telescope and science 
instruments to work. The mirror and the sunshield--both critical 
components--must fold up to fit inside the launch vehicle and open to 
their operational configuration once the JWST is in orbit. In addition, 
the observatory will house science instruments--such as a near- 
infrared[Footnote 3] camera, a near-infrared spectrograph,[Footnote 4] 
a mid-infrared instrument, and a fine guidance sensor--to enable 
scientists to conduct various research activities. 

Figure 1: Conceptual Drawing of NASA's JWST: 

[See PDF for image] 

Source: Northrup Grumman Corporation. 

[End of figure] 

The JWST is an international collaboration among the United States, the 
European Space Agency (ESA), and the Canadian Space Agency (CSA). ESA 
will provide the near-infrared spectrograph science instrument, the 
optical bench assembly of the mid-infrared instrument, and the launch 
of the JWST by means of an Ariane 5 expendable launch vehicle. CSA's 
contribution will be the fine guidance sensor to enable stable 
pointing. 

Recently, the JWST program recognized significant cost growth and 
schedule slippage. In March 2005, NASA identified about $1 billion cost 
growth, which increased the JWST's life-cycle cost estimate from $3.5 
billion to $4.5 billion. In addition, the program's schedule slipped 
nearly 2 years. As a result, the program began a series of re- 
baselining efforts to revise its acquisition strategy. In summer 2005, 
NASA Headquarters chartered two independent review teams--an 
Independent Review Team from NASA's Independent Program Assessment 
Office and a Science Assessment Team--to evaluate the program. The 
Independent Review Team was charged with examining the program's new 
cost/schedule/ technical baseline and reported in mid-April 2006 that 
(1) the JWST's scientific performance met the expectations of the 
science community, (2) the technical content was complete and sound, 
and (3) the Goddard Space Flight Center and contractor teams were 
effective. However, the team was concerned about the program's early 
year funding constraints. 

The Science Assessment Team, an international team of outside experts, 
was established to evaluate scientific capabilities of the JWST in the 
2015 time frame in light of other astronomical facilities that would be 
available. The team concluded that the financial savings gained from 
the reduction in the size of the primary mirror area would not be worth 
the resultant loss of scientific capabilities. The team recommended 
relaxing some science requirements and simplifying other aspects of the 
mission, such as integration and testing, to reduce the program's cost 
risk. For example, the team recommended relaxing the contamination 
requirements, allowing the project to test the mirrors using an 
innovative approach that will reduce costs. The team also recommended 
that the JWST de-emphasize the shorter wavelengths, since other 
astronomical facilities would be available to cover that range. 

JWST's Revised Strategy Does Not Fully Incorporate a Knowledge-Based 
Approach That Could Reduce Risks and Better Inform Decision Making: 

The JWST program recently revised its acquisition strategy to conform 
to NASA's acquisition policies; however, the program still faces 
considerable challenges. GAO best practices work has found that using a 
knowledge-based approach is a key factor in program success. When we 
initiated our work and before the program's recently revised 
acquisition strategy, program officials intended to have NASA commit to 
the program and start implementation with immature technologies, 
according to best practices, and without a preliminary design. During 
our review, we discussed these shortfalls with NASA officials, and they 
revised their acquisition strategy to align their decision milestones 
in accordance with NASA acquisition policy. While this is a good step, 
the current strategy does not fully incorporate a knowledge-based 
approach that could reduce the program's risks by ensuring that 
resources match requirements at program start. By closely following a 
knowledge-based approach, the JWST program will increase its chances 
for success and better inform NASA's decision making. 

Immature Technologies, Design Challenges, and Testing Restrictions 
Still Pose Risks: 

The JWST contains several innovations, including lightweight optics, a 
deployable sunshield, and a folding segmented mirror. Although the 
program began risk reduction activities early to develop and mature 
some technologies, such as the lightweight segmented folding mirror, 
the program is challenged with maturing some of its other critical 
technologies. For example, the sunshield, which consists of five layers 
of membranes, must be folded for launch but then unfurled to its 
operational configuration--with enough tension to prevent wrinkle 
patterns that could interfere with the telescope's mirrors, but not so 
much tension to cause tears in the fabric. The sunshield must also be 
aligned with the rest of the observatory so that only the top layer of 
the sunshield is visible to the primary mirror and a correct angle 
between the observatory and the sun and other heat-radiating bodies is 
maintained to enable the telescope and science instruments to preserve 
the very cold temperature--about 40 degrees Kelvin--critical for 
achieving the JWST's mission. In addition, using passive cooling 
devices, such as heat switches, to allow specific areas of the 
telescope to cool down, represent additional challenges since these 
items will be used in new configurations. NASA also recently 
substituted the cryo-cooler used for the mid-infrared instrument for a 
lower technology component to save mass. According to JWST officials, 
the program recently awarded the development contract for the cryo- 
cooler. In addition, the micro shutter array, which will allow the JWST 
to program specific patterns of the electromagnetic spectrum for 
viewing, is a new technology being developed by the Goddard Space 
Flight Center and is still at a relatively low level of maturity. JWST 
officials acknowledge that they are concerned about maturing the cryo- 
cooler and the micro shutter array. 

In addition, the program also faces design challenges related to the 
launch vehicle and the observatory's stability. For example, program 
officials told us that they may need to request a waiver because the 
telescope will not fit within the criteria limits of the launch 
vehicle's envelop without making design modifications. Furthermore, due 
to the late selection of the launch vehicle, the project office and 
prime contractor are just beginning to discuss interfaces, 
transportation at the launch site, and the additional space issue with 
Ariane 5 officials. Also, the project faces the unresolved problem of 
finding the best way to keep the observatory stable. The large 
sunshield, observatory attitude changes, and other effects conspire to 
produce unbalanced torques, which can make the observatory unstable. 
The project continues to look at ways to resolve this problem, 
including thrusters to rebalance the observatory, but project officials 
say this will continue to be a challenge. 

Another overriding concern is NASA's inability to test the entire 
observatory in its operational environment, since there is no test 
facility in the United States large enough to perform this test. The 
plan is to incrementally test components and subsystems on the ground 
in laboratories simulating the observatory's operational environment 
and to make extensive use of modeling and simulation. According to the 
memorandum summarizing the January 2006 System Definition Review, a key 
concern is that the JWST is pushing the limits of ground test 
facilities and cannot be tested at the observatory level; therefore, 
requiring complicated integration and testing with a series of 
subsystem tests and analyses. In its April 2006 assessment of the JWST 
program, the Independent Review Team reported that there are several 
exceptions to the "test as you fly"[Footnote 5] guideline and that 
mitigation strategies need to be developed before the end of the 
preliminary design phase. 

Containing Further Cost Growth and Schedule Slippage: 

In March 2005, the JWST program recognized that its cost had grown by 
about $1 billion, increasing the JWST's life-cycle cost estimate from 
$3.5 billion to $4.5 billion. About half of the cost growth was due to 
schedule slippage--a 1-year schedule slip because of a delay in the 
decision to use an ESA-supplied Ariane 5 launch vehicle and an 
additional 10-month slip caused by budget profile limitations in fiscal 
years 2006 and 2007. More than a third of the cost increase was caused 
by requirements and other changes. An increase in the program's 
contingency funding accounted for the remainder--about 12 percent--of 
the growth. 

Despite an increase in the program's contingency funding, the 
Independent Review Team found that the contingency funding is still 
inadequate. In its April 2006 assessment of the JWST program's re- 
baselining, the Independent Review Team expressed concern over the 
program's contingency funding, stating that it is too low and phased in 
too late. According to the team, the program's contingency from 2006 
through 2010 of only $29 million, or about 1.5 percent,[Footnote 6] 
after "liens" and "threats"[Footnote 7] is inadequate.[Footnote 8] The 
team also stated that a 25 percent to 30 percent total contingency is 
appropriate for a program of this complexity. The program's total 
contingency is only about 19 percent. The team warned that because of 
the inadequate contingency, the program's ability to resolve issues, 
address program risk areas, and accommodate unknown problems is very 
limited. Therefore, the team concluded that from a budget perspective, 
the re-baselined program is not viable for a 2013 launch. The team 
recommended that before the Non-Advocate Review (NAR)[Footnote 9] 
leading to program start, steps should be taken by the Science Mission 
Directorate to assure that the JWST program contains an adequate time- 
phased funding contingency to secure a stable launch date. 

The JWST program remains at risk of incurring additional cost growth 
and schedule slippage because of the technical challenges that must be 
resolved--immature technologies, design challenges, and testing 
restrictions. Our best practices work indicates that immature 
technology increases the risk of cost increases and schedule slips. 
Unresolved technology challenges can cascade through a product 
development cycle often resulting in an unstable design that will 
require more testing and thus more time and money to fix the problems. 
Subsequently, it will be difficult to prepare a reliable cost estimate 
until these challenges are resolved. 

Knowledge-Based Approach Key to Overcoming Challenges: 

Our past work on the best practices of product developers in government 
and industry has found that the use of a knowledge-based approach is a 
key factor in successfully addressing challenges such as those faced by 
the JWST program. Over the last several years, we have undertaken a 
body of work on how leading developers in industry and government use a 
knowledge-based approach to deliver high quality products on time and 
within budget.[Footnote 10] A knowledge-based approach to product 
development efforts enables developers to be reasonably certain that, 
at critical junctures or "knowledge points" in the acquisition life 
cycle, their products are more likely to meet established cost, 
schedule, and performance baselines and therefore provides them with 
information needed to make sound investment decisions. The marker for 
the first juncture--knowledge point 1 (KP1)--occurs just prior to 
program start. At KP1, the customer's requirements match the product 
developer's resources in terms of knowledge, time, and money. At KP 2, 
the product design is stable, and production processes are mature at KP 
3. Product development efforts that have not followed a knowledge-based 
approach can frequently be characterized by poor cost, schedule, and 
performance outcomes. 

We recently reported that NASA's revised acquisition policy for 
developing flight systems and ground support projects incorporates some 
aspects of the best practices used by successful developers.[Footnote 
11] For example, NASA policy requires projects to conduct a major 
decision review--NAR--before moving from formulation to implementation. 
Further, before moving from formulation to implementation, projects 
must validate requirements and develop realistic cost and schedule 
estimates, human capital plans, a preliminary design, and a technology 
plan--all key elements for matching needs to resources before 
commitment to a major investment is made at project start. Figure 2 
compares NASA's life cycle with a knowledge-based acquisition life 
cycle. 

Figure 2: Comparison of NASA's Life Cycle with a Knowledge-Based 
Acquisition Life Cycle: 

[See PDF for image] 

Sources: NASA and GAO analysis. 

[End of figure] 

While the policy incorporates elements of a knowledge-based approach, 
we also reported that NASA's acquisition policies lack the necessary 
requirements to ensure that programs proceed and are funded only after 
an adequate level of knowledge at key junctures. For example, NASA 
policy does not require that programs demonstrate technologies at high 
levels of maturity at program start. Further, although NASA policy does 
require project managers to establish a continuum of technical and 
management reviews, the policy does not specify what these reviews 
should be nor does it require major decision reviews at other key 
points in a product's development. These best practices could be used 
to further reduce program risks. 

In order to close the gaps between NASA's current acquisition 
environment and best practices on knowledge-based acquisition, we 
recommended that NASA take steps to ensure that NASA projects follow a 
knowledge-based approach for product development. Specifically, we 
recommended that NASA (1) in drafting its systems engineering policy, 
incorporate requirements for flight systems and ground support projects 
to capture specific product knowledge by key junctures in project 
development and use demonstration of this knowledge as exit criteria 
for decision making at key milestones and (2) revise NASA Procedural 
Requirements 7120.5C to institute additional major decision reviews 
following the NAR for flight systems and ground support projects, which 
result in recommendations to the appropriate decision authority at key 
milestones. NASA concurred with our recommendations and agreed to 
revise its policies. 

One of the resources needed at program start is mature technology. Our 
best practices work has shown that technology readiness levels 
(TRL)[Footnote 12]--a concept developed by NASA--can be used to gauge 
the maturity of individual technologies. Specifically, TRL 6-- 
demonstrating a technology as a fully integrated prototype in a 
realistic environment--is the level of maturity needed to minimize 
risks for space systems entering product development. To achieve TRL 6, 
technology maturity must be demonstrated in a relevant environment 
using a prototype or model. (See app. II for a detailed description and 
definition of TRLs and test environments.) 

Figure 3: Technology Maturity Levels for Product Development: 

[See PDF for image] 

Source: HAO> 

[End of figure] 

A knowledge-based approach also involves the use of incremental markers 
to ensure that the required knowledge has been attained at each 
critical juncture. For example, exit criteria at KP1 should include 
demonstrated maturity of critical technologies, completed trade-offs 
and finalized requirements, and initial cost and schedule estimates 
using results from the preliminary design review. The approach ensures 
that managers will (1) conduct activities to capture relevant product 
development knowledge, (2) provide evidence that knowledge was 
captured, and (3) hold decision reviews to determine that appropriate 
knowledge was captured to allow a move to the next phase. If the 
knowledge attained at each juncture does not justify the initial 
investment, the project should not go forward and additional resources 
should not be committed. 

Risks Not Fully Addressed by Recently Revised Acquisition Strategy: 

Prior to the program's recent acquisition strategy revision, program 
officials were not following NASA acquisition policy[Footnote 13] and 
were set to commit to the program and start implementation with 
immature technologies, according to best practices, and without a 
preliminary design. For instance, the schedule called for convening the 
NAR before the end of preliminary design. NASA policy indicates that 
the NAR and Preliminary Design Review (PDR) should be aligned. Even at 
the pre-NAR[Footnote 14] in July 2003, the plan had been to have the 
NAR before the PDR,[Footnote 15] although the two reviews were closer 
together than the more recent plan. 

During our review, we discussed these shortfalls with NASA officials. 
To their credit, they revised their acquisition strategy to conform to 
NASA policy. Currently, the mission NAR--upon which the program start 
decision will be based--will be aligned with the mission PDR (scheduled 
for March 2008). We believe this is a positive step, since it will 
ensure that a preliminary design--a key element for matching needs to 
resources--is established before program start. The revised strategy 
also splits the NAR into two parts--a technical NAR and a mission NAR. 
The purpose of the technical NAR (scheduled for January 2007) will be 
to determine whether the project has successfully retired its invention 
risk, i.e., critical technologies have achieved TRL 6, according to a 
NASA official. Technology issues will not be revisited after the 
technical NAR unless problems arise. However, it is unclear if the 
critical technologies will be demonstrated to a level of fidelity 
required by best practices at the technical NAR. Furthermore, the 
strategy does not fully incorporate a knowledge-based approach that 
could address the program's risks by ensuring--through the use of exit 
criteria--that resources match requirements in terms of knowledge, 
time, and money before program start. For example: 

* Under a knowledge-based approach, adequate testing is required to 
demonstrate that key technologies are mature--at TRL 6--prior to 
program start. This is particularly important for the JWST, given the 
program's challenges with testing restrictions and the fact that the 
observatory cannot be serviced in space. In some cases, such as the 
sunshield, backup technologies do not exist, thus increasing the 
importance of adequately maturing and testing critical technologies. If 
key components--like the sunshield--fail, then the entire observatory 
will be lost. This requires greater fidelity in the testing, even as 
early as demonstrating the maturity of key technologies prior to 
program start. 

To achieve TRL 6 (the maturity level required by best practices for 
program start), technology maturity must be demonstrated as a 
representative model or prototype--which is very close to the actual 
system in form, fit, and function--in a relevant environment. However, 
there is risk that the current JWST technology development plan will 
not result in the appropriate demonstration of technology maturity. For 
example, the half-scale thermal vacuum test of the entire 
observatory[Footnote 16] at Johnson Space Center is currently planned 
for September 2008, and so the knowledge gained regarding the maturity 
of the sunshield's thermal and dynamic performance[Footnote 17] is 
pushed out 6 months beyond the PDR/NAR/program start date of March 
2008. When JWST program officials briefed us in August 2005, the TRL 
levels for thermal and dynamic performance of the sunshield were both 
assessed to be at TRL 4, and the plan to get to TRL 6 was to test these 
subsystems during this half-scale thermal vacuum test. However, in fall 
2005 program officials reviewed the technology development plan and 
concluded that only the materials for the sunshield's membrane are 
technology development items, while other items affecting the 
configuration and deployment of the sunshield--such as thermal and 
dynamic performance--are considered engineering challenges. JWST 
officials stated that earlier testing of sample materials demonstrated 
the sunshield's thermal performance and a demonstration using a 1/10th 
scale model demonstrated dynamic performance[Footnote 18] and satisfied 
TRL 6 requirements. However, we have found in our best practices work 
that demonstrating a technology to a TRL 6 typically involves 
demonstrating that a prototype--close to the form, fit, and 
functionality intended for the product--has been demonstrated in an 
environment that closely represents the anticipated operational 
environment. In our past review of development programs, we have found 
that if this level of maturity is not demonstrated before a product 
development effort is launched, a program increases the likelihood of 
cost growth and schedule delays as it tries to close the knowledge gap 
between the technologies' maturity level and the product's design 
requirements. 

* The JWST program's inadequate contingency runs contrary to another 
premise of a knowledge-based approach--having sufficient resources in 
terms of funding available to ensure a program's success. As discussed 
in an earlier section, the Independent Review Team stated that the 
program's contingency from 2006 through 2010 of only about 1.5 percent 
after "liens" and "threats" is inadequate. The team warned that, 
because of the inadequate contingency, the program's ability to resolve 
issues, address program risk areas, and accommodate unknown problems is 
very limited. The team concluded that, from a budget perspective, the 
re-baselined program is not viable for a 2013 launch. 

Knowledge-Based Approach Would Allow the JWST Program to Better Inform 
NASA's Decision-Making Process: 

A good basis for making informed investment decisions is essential in 
the fiscally constrained environment that now exists across the federal 
government. Our nation faces large, growing, and structural long-term 
fiscal imbalances. Given the severity of those fiscal challenges and 
the wide range of federal programs, hard choices need to be considered 
across the government, and NASA is no exception. NASA must compete with 
other departments and agencies for part of a constricted discretionary 
spending budget. 

In the near future, NASA will need to determine the resources necessary 
to develop the systems and supporting technologies to achieve the 
President's Vision for Space Exploration--while simultaneously 
financing its other priority programs--and structure its investment 
strategy accordingly. Initial implementation of the Vision as explained 
in NASA's Exploration Systems Architecture Study calls for completing 
the International Space Station, developing a new crew exploration 
vehicle, and returning to the moon no later than 2020. NASA estimates 
that it will cost approximately $104 billion over the next 13 years to 
accomplish these initial goals. These priorities, along with NASA's 
other missions, will be competing within NASA for funding. It will 
likely be difficult for decision makers to agree on which projects to 
invest in and which projects, if any, to terminate. The NASA 
Administrator has acknowledged that NASA faces difficult choices about 
its missions in the future--for example, between human space flight, 
science, and aeronautics missions. 

In the President's fiscal year 2007 budget request for NASA, the JWST 
has the largest budget allocation of all programs in the Science 
Mission Directorate's Astrophysics Division for the 5-year budget 
horizon from fiscal year 2007 through fiscal year 2011--nearly $2 
billion of the division's $6.9 billion total budget, or about 29 
percent. An inadequately informed decision to commit to the estimated 
$4.5 billion total funding for the JWST would significantly impact 
NASA's science portfolio, since funding given to the JWST will not 
available for other programs. Early in the planning for how to handle 
the JWST program's cost growth, NASA officials recognized the impact 
that the JWST's cost growth could have on other programs. In a July 
2005 briefing to the Agency Program Management Council[Footnote 19] 
soon after the cost growth was identified, NASA officials stated that 
"something must give if JWST stays in the portfolio." The choices 
discussed were (1) relaxing requirements or (2) adding budget and 
schedule, which would mean that other missions would be deferred or 
deleted from the portfolio. 

In addition, committing to the JWST program obligates the government 
contractually, since it allows the prime contractor to begin 
implementation tasks on the very long prime contract extending from 
October 2002 through launch--currently planned for June 2013--plus one 
year. The contract states that until the project achieves the 
implementation milestone, contract spending is limited to formulation 
activities, except for long-lead items and other activities approved in 
writing. After the implementation milestone is achieved at program 
start, the contracting officer will notify the contractor by letter to 
proceed to implementation. According to the contracting officer, the 
assumption is that this is the go-ahead for the whole program. 

To make well-informed decisions, NASA needs the knowledge to assess the 
value of its programs--like the JWST program--in relationship to each 
other. In May 2004, we reported that, of 27 NASA programs we examined, 
17 had cost increases averaging about 31percent.[Footnote 20] One of 
the programs in our sample was another infrared telescope program--the 
Spitzer Space Telescope--and it was plagued by schedule slippages 
caused by delays in the delivery of components, flight software, the 
mission operation system, and launch delays, all contributing to a 29.3 
percent increase in program costs. In general, we found the programs in 
the sample lacked sufficient knowledge needed to make informed 
acquisition decisions. Insufficient knowledge to make informed 
investment decisions can further complicate the already-difficult 
choices that NASA faces. Conversely, sufficient knowledge at key 
junctures can facilitate well-informed investment decisions and protect 
the government from incurring contractual liabilities before it is 
appropriate. A knowledge-based approach ensures that comprehensive and 
comparable programmatic data are obtained. 

Conclusions: 

Within the JWST program, NASA officials have accomplished a great deal, 
such as the development of the large, segmented mirror that is a leap 
ahead in technology. Moreover, the program has support from the larger 
scientific community. To enhance the program's chances for success, 
program officials have chosen a path forward which follows NASA's 
policies for ensuring readiness to proceed into implementation/product 
development. However, the JWST program's revised strategy does not 
fully address the risks associated with the many challenges that the 
program still faces--including maturing technology, mitigating testing 
restrictions, and ensuring that adequate funding is available for 
contingencies. This puts the program at risk of further cost growth and 
schedule slippage. The program needs to have sufficient knowledge at 
key junctures to successfully address its challenges and use 
incremental markers to make certain that resources in terms of 
knowledge, time, workforce, and money match the requirements. Given the 
severity of the fiscal challenges our nation faces and the wide range 
of competing federal programs, hard choices need to be considered 
across the government, and NASA is no exception. Using a knowledge- 
based approach for NASA's new development programs such as the JWST 
could help the agency make the difficult choices about how to allocate 
its limited budget resources among competing priorities by utilizing 
common and consistent criteria in program evaluations. 

Recommendations for Executive Action: 

To increase the JWST program's chances of successful product 
development, we recommend that the NASA Administrator take the 
following actions: 

≤ Direct the JWST program to fully apply a knowledge-based acquisition 
approach--to include incremental markers--that will not only ensure 
that adequate knowledge is attained at key decision points, but also 
hold the program accountable. These markers should include, but not be 
limited to: 

* schedules that demonstrate the maturity of all critical technologies 
prior to program start; 

* criteria to ensure the validity of test articles; 

* criteria to demonstrate that mature component designs being used in 
new configurations meet form, fit, and function standards; and: 

* criteria to ensure that sufficient contingency funding can be 
provided and phased appropriately. 

≤ Instruct the JWST program to continue to adhere to NASA acquisition 
policy and base the program's go/no-go review (NAR) decision not only 
on adherence to that policy, but also on (1) the program's ability to 
demonstrate whether it is meeting the knowledge markers outlined 
earlier and (2) whether adequate funds are available to execute the 
program. 

Agency Comments and Our Evaluation: 

In written comments on a draft of this report, NASA concurred with our 
two recommendations and outlined actions that the agency plans to take 
to implement such recommendations. NASA said that it endorses the 
knowledge-based approach recommended and that it believes the current 
JWST program plan is consistent with that approach. NASA's recognition 
of the value of obtaining knowledge prior to moving to subsequent 
acquisition phases and acknowledgment that it plans to use exit 
criteria as knowledge markers for other JWST mission-level reviews are 
welcome steps toward establishing an agency-wide risk reduction 
culture. Now, it will be critical for NASA decision makers to enforce 
adherence to the discipline of the knowledge-based approach and ensure 
that critical product knowledge is indeed demonstrated before allowing 
the JWST program to proceed. In the years ahead, NASA decision makers 
will likely face pressures to grant waivers for going forward with 
immature technologies, allow programs to be restructured, and thus 
marginalize accountability. For a program such as the JWST, whose 
investment is already substantial and successful outcome eagerly 
anticipated by the science community, adherence to such knowledge-based 
principles will need to be strictly enforced. As identified in this 
report, NASA would be well served by applying its own technology 
readiness standards (reprinted in appendix II) as part of its exit 
criteria, and demonstrating that critical technologies are at the TRL 6 
level prior to program start using a representative model or prototype-
-which is very close to the actual system in form, fit, and function-- 
in a relevant environment. Emphasis by decision makers on the 
application of "form, fit, and function standards" and "validity of 
test articles" as exit criteria for the JWST program start and entry 
into Phase C will help address our concern that the current JWST 
technology development plan may not result in the appropriate 
demonstration of technology maturity prior to program start. NASA's 
comments are reprinted in appendix III. 

We are sending copies of this report to interested congressional 
committees and to the NASA Administrator. We will make copies available 
to others upon request. In addition, the report will be available at no 
charge on the GAO Web site at [Hyperlink, http://www.gao.gov]. 

If you or your staff have any questions concerning this report, please 
contact me at (202) 512-4841 or lia@gao.gov. Contact points for our 
Offices of Congressional Relations and Public Affairs may be found on 
the last page of this report. Key contributors to this report are 
acknowledged in appendix IV. 

Signed by: 

Allen Li: 
Director Acquisition and Sourcing Management: 

List of Congressional Committees: 

The Honorable Kay Bailey Hutchison: 
Chairman: 
The Honorable Bill Nelson: 
Ranking Minority Member: 
Subcommittee on Science and Space: 
Committee on Commerce, Science, and Transportation: 
United State Senate: 

The Honorable Richard C. Shelby: 
Chairman: 
The Honorable Barbara A. Mikulski: 
Ranking Minority Member: 
Subcommittee on Commerce, Justice, Science, and Related Agencies: 
Committee on Appropriations: 
United States Senate: 

The Honorable Sherwood L. Boehlert: 
Chairman: 
The Honorable Bart Gordon: 
Ranking Minority Member: 
Committee on Science: 
House of Representatives: 

The Honorable Ken Calvert: 
Chairman: 
The Honorable Mark Udall: 
Ranking Minority Member: 
Subcommittee on Space and Aeronautics: 
Committee on Science: 
House of Representatives: 

The Honorable Frank R. Wolf: 
Chairman: 
The Honorable Alan B. Mollohan: 
Ranking Minority Member: 
Subcommittee on Science, the Departments of State, Justice, and 
Commerce, and, Related Agencies: 
Committee on Appropriations: 
House of Representatives: 

[End of section] 

Appendix I: Scope and Methodology: 

To assess the extent to which the JWST acquisition strategy follows 
NASA policy and GAO best practices for ensuring readiness to proceed 
into implementation, we reviewed NASA policy on program management and 
compared the JWST project office's management approach to NASA policy. 
Additionally, we analyzed the JWST acquisition strategy and benchmarked 
it to best practices. We interviewed NASA and contractor officials to 
clarify our understanding of the JWST management approach and 
technology development plan in relation to NASA policy and guidelines 
and best practices. To deepen our understanding of JWST technical 
issues, we attended the 3-day Sunshield Subsystem Concept Design Review 
as well as the 4-day JWST System Definition Review. 

To evaluate the impact of the JWST acquisition strategy on NASA's 
ability to assess the program and make informed investment decisions in 
the context of its other priorities, we analyzed available JWST cost 
and schedule data and conducted interviews with program officials to 
clarify our understanding of the information. Furthermore, we requested 
and reviewed documentary support breaking out the components of the 
cost increases and schedule slippage. We also interviewed program 
officials to clarify our understanding of the potential impact that 
investment in the JWST will have on other NASA programs. In addition, 
we reviewed statements of the NASA Administrator, budget documents, 
GAO's High-Risk Series, and GAO's 21st Century Challenges to better 
evaluate the JWST's significance in the larger NASA and federal 
government context. 

To accomplish our work, we visited NASA Headquarters, Washington, D.C; 
Goddard Space Flight Center, Greenbelt, Maryland; Marshall Space Flight 
Center, Huntsville, Alabama; Northrop Grumman Space Technology, Redondo 
Beach, California; and Ball Aerospace and Technologies Corporation, 
Boulder, Colorado. 

We performed our review from August 2005 through May 2006 in accordance 
with generally accepted government auditing standards. 

[End of section] 

Appendix II: Technology Readiness Levels: 

Technology Readiness Level: TRL 1: Basic principles observed and 
reported; 
Description: Lowest level of technology readiness. Scientific research 
begins to be translated into applied research and development. Examples 
might include paper studies of a technology's basic properties; 
Hardware Software: None. (Paper studies and analysis.); 
Demonstration Environment: None. 

Technology Readiness Level: TRL 2: Technology concept and/or 
application formulated; 
Description: Invention begins. Once basic principles are observed, 
practical applications can be invented. The application is speculative 
and there is no proof or detailed analysis to support the assumption. 
Examples are still limited to paper studies; 
Hardware Software: None. (Paper studies and analysis.); 
Demonstration Environment: None. 

Technology Readiness Level: TRL 3: Analytical and experimental critical 
function and/or characteristic proof of concept; 
Description: Active research and development is initiated. This 
includes analytical studies and laboratory studies to physically 
validate analytical predictions of separate elements of the technology. 
Examples include components that are not yet integrated or 
representative; 
Hardware Software: Analytical studies and demonstration of nonscale 
individual components (pieces of subsystem); 
Demonstration Environment: Lab. 

Technology Readiness Level: TRL 4: Component and/or breadboard. 
Validation in laboratory environment; 
Description: Basic technological components are integrated to establish 
that the pieces will work together. This is relatively "low fidelity" 
compared to the eventual system. Examples include integration of "ad 
hoc" hardware in a laboratory; 
Hardware Software: Low fidelity breadboard. Integration of nonscale 
components to show pieces will work together. Not fully functional or 
form or fit but representative of technically feasible approach 
suitable for flight articles; 
Demonstration Environment: Lab. 

Technology Readiness Level: TRL 5: Component and/or breadboard 
validation in relevant environment; 
Description: Fidelity of breadboard technology increases significantly. 
The basic technological components are integrated with reasonably 
realistic supporting elements so that the technology can be tested in a 
simulated environment. Examples include "high fidelity" laboratory 
Integration of components; 
Hardware Software: High fidelity breadboard. Functionally equivalent 
but not necessarily form and/or fit (size weight, materials, etc.) 
Should be approaching appropriate scale. May include integration of 
several components with reasonably realistic support elements/ 
subsystems to demonstrate functionality; 
Demonstration Environment: Lab demonstrating functionality but not form 
and fit. May include flight demonstrating breadboard in surrogate 
aircraft. Technology ready for detailed design studies. 

Technology Readiness Level: TRL 6: System/subsystem model or prototype 
demonstration in a relevant environment; 
Description: Representative model or prototype system, which is well 
beyond the breadboard tested for TRL 5, is tested in a relevant 
environment. Represents a major step up in a technology's demonstrated 
readiness. Examples include testing a prototype in a high fidelity 
laboratory environment or in simulated operational environment; 
Hardware Software: Prototype--Should be very close to form, fit and 
function. Probably includes the integration of many new components and 
realistic supporting elements/subsystems if needed to demonstrate full 
functionality of the subsystem; 
Demonstration Environment: High-fidelity lab demonstration or limited/ 
restricted flight demonstration for a relevant environment. Integration 
of technology is well defined. 

Technology Readiness Level: TRL 7: System prototype demonstration in an 
operational environment; 
Description: Prototype near or at planned operational system. 
Represents a major step up from TRL 6, requiring the demonstration of 
an actual system prototype in an operational environment, such as in an 
aircraft, vehicle or space. Examples include testing the prototype in a 
test bed aircraft; 
Hardware Software: Prototype. Should be form, fit and function 
integrated with other key supporting elements/subsystems to demonstrate 
full functionality of subsystem; 
Demonstration Environment: Flight demonstration in representative 
operational environment such as flying test bed or demonstrator 
aircraft. Technology is well substantiated with test data. 

Technology Readiness Level: TRL 8: Actual system completed and "flight 
qualified" through test and demonstration; 
Description: Technology has been proven to work in its final form and 
under expected conditions. In almost all cases, this TRL represents the 
end of true system development. Examples include developmental test and 
evaluation of the system in its intended weapon system to determine if 
it meets design specifications; 
Hardware Software: Flight qualified hardware; 
Demonstration Environment: Developmental test and evaluation in the 
actual system application. 

Technology Readiness Level: TRL 9: Actual system "flight proven" 
through successful mission operations; 
Description: Actual application of the technology in its final form and 
under mission conditions, such as those encountered in operational test 
and evaluation. In almost all cases, this is the end of the last "bug 
fixing" aspects of true system development. Examples include using the 
system under operational mission conditions; 
Hardware Software: Actual system in final form; 
Demonstration Environment: Operational test and evaluation in 
operational mission conditions. 

Source: GAO and its analysis of NASA data. 

[End of table] 

[End of section] 

Appendix III: Comments from the National Aeronautics and Space 
Administration: 

National Aeronautics and Space Administration:
Office of the Administrator: 
Washington, DC 20546-0001:

June 26, 2006:

Mr. Allen Li:

Director, Acquisition and Sourcing Management:
United States Government Accountability Office: 
Washington, DC 20548:

Dear Mr. Li:

NASA appreciates the opportunity to comment on your draft report 
General Accountability Office (GAO) GAO-06-634 entitled "Knowledge- 
Based Acquisition Approach Key to Addressing Program Challenges," which 
pertains to the James Webb Space Telescope (JWST) program. NASA 
endorses the knowledge-based acquisition approach recommended by the 
GAO. In part due to earlier GAO recommendations and NASA management 
changes, NASA believes the current JWST program plan is consistent with 
a knowledge-based approach and that the appropriate maturity of JWST 
technologies will be demonstrated well in advance of an Agency decision 
to proceed into Phase C.

The draft report references an earlier GAO report (GAO-06-218), 
entitled "Implementing a Knowledge-Based Acquisition Framework Could 
Lead to Better Investment Decisions and Project Outcomes," in which the 
GAO recommended that NASA take steps to ensure that NASA projects 
follow a knowledge-based approach for product development. In a letter 
to GAO dated December 15, 2005, NASA agreed with the GAO 
recommendations, noting that while NASA was already employing many of 
the recommended practices, some of those practices were not apparent in 
existing NASA acquisition policy documents. NASA is currently in the 
process of revising relevant NASA acquisition policy documents in 
accordance with the commitments made in the letter to the GAO.

In the current draft report, GAO recommends that the NASA Administrator 
take the following actions:

Recommendation 1* Direct the JWST program to apply a knowledge-based 
acquisition approach to include incremental markers-that will not only 
ensure that adequate knowledge is attained at key decision points, but 
also hold the program accountable. These markers should include, but 
not be limited to:

* schedules that demonstrate the maturity of all critical technologies 
prior to program start;

* criteria to ensure the validity of test articles;

* criteria to demonstrate that mature component designs being used in 
new configurations meet form, fit, and function standards; and:

* criteria to ensure that sufficient contingency funding can be 
provided and phased appropriately.

Concur - NASA concurs with this recommendation. The recently replanned 
JWST program includes a set of mission-level reviews that exceed the 
minimum set of reviews required by NASA Procedural Requirements 7123. 
Explicit exit criteria (including the criteria listed in this GAO 
report recommendation), are developed for each mission-level review to 
serve as incremental knowledge markers to ensure that adequate 
knowledge has been attained before proceeding to the next mission 
phase. Major JWST mission-level reviews include:

* Technology Non-Advocate Review (T-NAR) planned for January 2007:

* Preliminary Design Review (PDR) planned for March 2008 

* Non-Advocate Review (NAR) planned for March 2008 Critical Design 
Review (CDR) planned for July 2009 
* Test Readiness Review:

A successful PDR/NAR will be required for Agency approval to proceed 
into Phase C, and a successful CDR will be required for Agency approval 
to proceed to Phase D.

Recommendation 2: Instruct the JWST program to continue to adhere to 
NASA acquisition policy and base the program's go/no-go review (NAR) 
decision not only on adherence to that policy, but also on (1) the 
program's ability to demonstrate whether it is meeting the knowledge 
markers outlined earlier and (2) whether adequate funds are available 
to execute the program.

Concur - NASA concurs with this recommendation. NASA will employ the 
monthly JWST program status reporting processes, the annual budget 
planning processes, and the mission-level reviews (listed above) to 
confirm that the JWST program continues to adhere to NASA acquisition 
policy. NASA will also ensure that Agency approval to proceed to Phase 
C will be based on the program's ability to demonstrate that it is 
meeting the appropriate knowledge markers, as well as on whether 
adequate funds are available to execute the program.

Thank you for the opportunity to respond to this draft report.

Signed by: 

Shana Dale:
Deputy Administrator: 

[End of section] 

Appendix IV: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Allen Li (202) 512-4841: 

Staff Acknowledgments: 

In addition to the individual named above, Jim Morrison, Assistant 
Director; Greg Campbell; Keith Rhodes; Sylvia Schatz; Erin Schoening; 
Hai Tran; and Ruthie Williamson made key contributions to this report. 

[End of section] 

Related GAO Products: 

NASA Reports: 

NASA: Implementing a Knowledge-Based Acquisition Framework Could Lead 
to Better Investment Decisions and Project Outcomes. GAO-06-218. 
Washington, D.C.: December 21, 2005. 

NASA's Space Vision: Business Case for Prometheus 1 Needed to Ensure 
Requirements Match Available Resources. GAO-05-242. Washington, D.C.: 
February 28, 2005. 

Space Reports: 

Space Acquisitions: Stronger Development Practices and Investment 
Planning Need to Address Continuing Problems. GAO-05-891T. Washington, 
D.C.: July 12, 2005. 

Defense Acquisitions: Incentives and Pressures That Drive Problems 
Affecting Satellite and Related Acquisitions. GAO-05-570R. Washington, 
D.C.: June 23, 2005. 

Defense Acquisitions: Space-Based Radar Effort Needs Additional 
Knowledge before Starting Development. GAO-04-759. Washington, D.C.: 
July 23, 2004. 

Defense Acquisitions: Risks Posed by DOD's New Space Systems 
Acquisition Policy. GAO-04-379R. Washington, D.C.: January 29, 2004. 

Space Acquisitions: Committing Prematurely to the Transformational 
Satellite Program Elevates Risks for Poor Cost, Schedule, and 
Performance Outcomes. GAO-04-71R. Washington, D.C.: December 4, 2003. 

Defense Acquisitions: Improvements Needed in Space Systems Acquisition 
Policy to Optimize Growing Investment in Space. GAO-04-253T. 
Washington, D.C.: November 18, 2003. 

Defense Acquisitions: Despite Restructuring, SBIRS High Program Remains 
at Risk of Cost and Schedule Overruns. GAO-04-48. Washington, D.C.: 
October 31, 2003. 

Defense Acquisitions: Improvements Needed in Space Systems Acquisition 
Management Policy. GAO-03-1073. Washington, D.C.: September 15, 2003. 

Military Space Operations: Common Problems and Their Effects on 
Satellite and Related Acquisitions. GAO-03-825R. Washington, D.C.: June 
2, 2003. 

Military Space Operations: Planning, Funding, and Acquisition 
Challenges Facing Efforts to Strengthen Space Control. GAO-02-738. 
Washington, D.C.: September 23, 2002. 

Polar-Orbiting Environmental Satellites: Status, Plans, and Future Data 
Management Challenges. GAO-02-684T. Washington, D.C.: July 24, 2002. 

Defense Acquisitions: Space-Based Infrared System-Low at Risk of 
Missing Initial Deployment Date. GAO-01-6. Washington, D.C.: February 
28, 2001. 

Best Practices Reports: 

Defense Acquisitions: Assessments of Selected Major Weapon Programs. 
GAO-05-301. Washington, D.C.: March 31, 2005. 

Defense Acquisitions: Stronger Management Practices Are Needed to 
Improve DOD's Software-Intensive Weapon Acquisitions. GAO-04-393. 
Washington, D.C.: March 1, 2004. 

Defense Acquisitions: Assessments of Selected Major Weapon Programs. 
GAO-04-248. Washington, D.C.: March 31, 2004. 

Defense Acquisitions: DOD's Revised Policy Emphasizes Best Practices, 
but More Controls Are Needed. GAO-04-53. Washington, D.C.: November 10, 
2003. 

Defense Acquisitions: Assessments of Selected Major Weapon Programs. 
GAO-03-476. Washington, D.C.: May 15, 2003. 

Best Practices: Setting Requirements Differently Could Reduce Weapon 
Systems' Total Ownership Costs. GAO-03-57. Washington, D.C.: February 
11, 2003. 

Best Practices: Capturing Design and Manufacturing Knowledge Early 
Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: July 15, 
2002. 

Defense Acquisitions: DOD Faces Challenges in Implementing Best 
Practices. GAO-02-469T. Washington, D.C.: February 27, 2002. 

Best Practices: Better Matching of Needs and Resources Will Lead to 
Better Weapon System Outcomes. GAO-01-288. Washington, D.C.: March 8, 
2001. 

Best Practices: A More Constructive Test Approach Is Key to Better 
Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.: July 31, 
2000. 

Defense Acquisition: Employing Best Practices Can Shape Better Weapon 
System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.: April 26, 2000. 

Best Practices: DOD Training Can Do More to Help Weapon System Program 
Implement Best Practices. GAO/NSIAD-99-206. Washington, D.C.: August 
16, 1999. 

Best Practices: Better Management of Technology Development Can Improve 
Weapon System Outcomes. GAO/NSIAD-99-162. Washington, D.C.: July 30, 
1999. 

Defense Acquisitions: Best Commercial Practices Can Improve Program 
Outcomes. GAO/T-NSIAD-99-116. Washington, D.C.: March 17, 1999. 

Defense Acquisition: Improved Program Outcomes Are Possible. GAO/T- 
NSIAD-98-123. Washington, D.C.: March 18, 1998. 

Best Practices: Successful Application to Weapon Acquisition Requires 
Changes in DOD's Environment. GAO/NSIAD-98-56. Washington, D.C.: 
February 24, 1998. 

Major Acquisitions: Significant Changes Underway in DOD's Earned Value 
Management Process. GAO/NSIAD-97-108. Washington, D.C.: May 5, 1997. 

Best Practices: Commercial Quality Assurance Practices Offer 
Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.: August 26, 
1996. 

FOOTNOTES 

[1] The JWST is a one-project program, according to a NASA official. 
The terms "program" and "project" are used interchangeably throughout 
this report. 

[2] GAO, NASA: Implementing a Knowledge-Based Acquisition Framework 
Could Lead to Better Investment Decisions and Project Outcomes, GAO-06-
218 (Washington, D.C.: Dec. 21, 2005). 

[3] Infrared radiation is one of the many types of "light" that 
comprise the electromagnetic spectrum. Infrared light is situated 
outside of the visible spectrum and has wavelengths longer than visible 
light. Astronomers generally divide the infrared portion of the 
electromagnetic spectrum into three regions: near infrared, mid 
infrared, and far infrared. The JWST will be sensitive to near-infrared 
and mid-infrared radiation. 

[4] A spectrograph is an instrument for dispersing radiation (as 
electromagnetic radiation or sound waves) into a spectrum and 
photographing or mapping the spectrum. 

[5] "Test as you fly" means performing the final performance and 
environmental test with the spacecraft fully integrated in the same 
configuration that it will be in when it launches, according an agency 
official. 

[6] Some budget cuts were restored after the Independent Review Team's 
assessment, increasing this amount to about 3 percent. 

[7] A "lien" is a potential cost to a project, direct or indirect, 
which may or may not come to fruition, for which a portion of funding 
reserves is set aside. According to a JWST project official, "threats" 
are things that concern a project or engineer, which may or may not 
come true, but which bear watching to see if they have validity; 
however, they do not require the same rigor as "liens." 

[8] According to a member of the Independent Review Team, "threats" 
were included in the analysis because after examining the project 
office's "threat" list, the team concluded that the "threats" had a 
high probability of occurring and were therefore more like "liens." 

[9] The NAR--a program/project milestone review prescribed by NASA 
Procedural Requirements 7120.5C--is intended to provide NASA management 
with an independent assessment of a program's readiness to move into 
implementation and the final design phase. 

[10] Our best practice reviews are identified in the "Related GAO 
Products" section at the end of this report. 

[11] GAO-06-218. 

[12] TRLs characterize the readiness of technologies for hand-off to 
project implementers. Nine levels are defined representing concepts 
from fundamental research level through technologies fully qualified 
and demonstrated in flight. 

[13] NASA Procedural Requirements 7120.5C, which states that its 
requirements are applicable to all programs and projects currently in 
formulation as of the effective date of March 22, 2005. 

[14] The pre-NAR is an independent review of programs/projects 
conducted at the end of the concept development phase to assess 
readiness to proceed into the preliminary design phase. 

[15] The PDR is the project milestone review which establishes the 
basis for proceeding with a detailed design. The purpose of the PDR is 
to demonstrate that the preliminary design meets all system 
requirements with an acceptable level of risk within the planned cost 
and schedule. 

[16] According to the mission systems engineer, the half-scale thermal 
vacuum test will be done using a half-scale model of the entire 
observatory. Deployments, including the sunshield, will be tested, and 
the sunshield membrane will be vibrated during the test. 

[17] The purpose of dynamic testing is to determine how the sunshield 
behaves structurally when shaken at different frequencies in order to 
predict the influence of disturbances on the pointing control of the 
JWST's optics. 

[18] The main components of the 1/10th scale model test article were a 
central mounting block, four support tubes, and four Kapton film 
layers. Therefore, the 1/10th scale model was not a scale version of 
the current JWST sunshield, which consists of five layers of Kapton 
membranes with special coatings, booms, hinges, deployment motors, edge 
cables, stowed boom restraints, stowed membrane containment structure, 
and other mechanisms. 

[19] The Agency Program Management Council is one of a system of 
Governing Program Management Councils responsible for assessing program 
and project formulation and implementation as well as providing 
oversight and direction. 

[20] GAO, NASA: Lack of Disciplined Cost-Estimating Processes Hinders 
Effective Program Management, GAO-04-642 (Washington, D.C.: May 28, 
2004). 

GAO's Mission: 

The Government Accountability Office, the investigative arm of 
Congress, exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability 
of the federal government for the American people. GAO examines the use 
of public funds; evaluates federal programs and policies; and provides 
analyses, recommendations, and other assistance to help Congress make 
informed oversight, policy, and funding decisions. GAO's commitment to 
good government is reflected in its core values of accountability, 
integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains 
abstracts and full-text files of current reports and testimony and an 
expanding archive of older products. The Web site features a search 
engine to help you locate documents using key words and phrases. You 
can print these documents in their entirety, including charts and other 
graphics. 

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as "Today's Reports," on its 
Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order 
GAO Products" heading. 

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to: 

U.S. Government Accountability Office 

441 G Street NW, Room LM 

Washington, D.C. 20548: 

To order by Phone: 

Voice: (202) 512-6000: 

TDD: (202) 512-2537: 

Fax: (202) 512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm 

E-mail: fraudnet@gao.gov 

Automated answering system: (800) 424-5454 or (202) 512-7470: 

Public Affairs: 

Jeff Nelligan, managing director, 

NelliganJ@gao.gov 

(202) 512-4800 

U.S. Government Accountability Office, 

441 G Street NW, Room 7149 

Washington, D.C. 20548: