Orion Multi-Purpose Crew Vehicle:
Action Needed to Improve Visibility into Cost, Schedule, and Capacity to Resolve Technical Challenges
GAO-16-620: Published: Jul 27, 2016. Publicly Released: Jul 27, 2016.
What GAO Found
The National Aeronautics and Space Administration's (NASA) Orion Multi- Purpose Crew Vehicle (Orion) program has overcome several technical challenges and made design changes to the crew capsule to reduce risk. Known challenges, however, remain—such as development of the service module and the crew capsule heatshield, among others—that could cause cost increases and schedule delays as the program undergoes integration and test. Technical challenges are inherent in complex programs such as Orion, but if not carefully managed, they could result in cost overruns and schedule delays. For example, the program has identified software development as an area of substantial risk with a potential cost impact of more than $90 million and which may result in schedule delays.
GAO found that the Orion program's cost and schedule estimates are not reliable based on best practices for producing high-quality estimates. Cost and schedule estimates play an important role in addressing technical risks. In September 2015, NASA established a commitment baseline of $11.3 billion and an April 2023 launch readiness date for the program's second exploration mission. NASA used a joint cost and schedule confidence level (JCL) analysis—a point-in-time estimate that, among other things, includes all cost and schedule elements and incorporates and quantifies known risks—to establish the commitment baselines at a 70 percent confidence level, as required by NASA policy. However, NASA's JCL analysis was informed by its unreliable cost and schedule estimates. GAO found that the Orion cost estimate met or substantially met 7 of 20 best practices and its schedule estimate met or substantially met 1 of 8 best practices. For example, the cost estimate lacked necessary support and the schedule estimate did not include the level of detail required for high-quality estimates. Without sound cost and schedule estimates, decision makers do not have a clear understanding of the cost and schedule risk inherent in the program or important information needed to make programmatic decisions.
NASA and the Orion program have made some programmatic decisions that could further exacerbate cost and schedule risks. The Orion program is executing to an internal schedule with a launch readiness date of August 2021, which has a lower confidence level than its commitment baseline. This means that NASA is accepting higher cost and schedule risk associated with executing this schedule. Working toward a more aggressive goal is not a bad practice; however, increasing cost and schedule risk to the program in order to pursue such a goal may not be a beneficial strategy to the program in the long term. According to program officials, the program employs most of its available budget to fund current work and holds most of its cost reserves at the end of the internal schedule. The lack of cost reserves has caused the program to defer work to address technical issues and stay within budget. As a result, the Orion program's reserves in future years could be overwhelmed by work being deferred. Program officials told GAO that they have not performed a formal analysis to understand the impact that delaying work might have on the available reserves since the program was confirmed. Without this type of analysis, program management may not have a complete understanding of how decisions made now will affect the longer-term execution of the program.
Why GAO Did This Study
Orion is NASA's first crew capsule that could transport humans beyond the moon. Recent programs, such as Constellation, were canceled in the face of acquisition problems and funding-related issues. The $11 billion that NASA estimates it will need to develop Orion through 2023, along with the funding necessary for other human spaceflight programs, represents a significant portion of NASA's anticipated budget during that period.
The House Committee on Appropriations included a provision in its report for GAO to review the acquisition progress of Orion, among other human spaceflight programs. This report assesses (1) technical challenges facing the Orion program that may affect cost and schedule, (2) the reliability of Orion's cost and schedule estimates, and (3) agency and program programmatic decisions that may affect cost and schedule risks. To do this work, GAO examined documents supporting the cost and schedule estimates, contractor performance data, and other relevant program documentation, and interviewed relevant officials.
What GAO Recommends
To provide the Congress and NASA a reliable estimate of program cost and schedule, the program should perform an updated JCL analysis with cost and schedule estimates in line with best practices. The program should also perform an analysis to understand the impact of deferred work on program reserves. NASA partially concurred with the first recommendation and concurred with the second.
For more information, contact Cristina Chaplain at (202) 512-4841 or email@example.com.
Recommendations for Executive Action
Comments: NASA partially agreed with this recommendation, stating that the agency reviewed, in detail, the Orion integrated cost/schedule and risk analysis methodology and determined the rigor to be a sufficient basis for the agency commitments. In January 2018, NASA officials stated that they have no plans to update the joint confidence level analysis for the Orion program. We maintain that NASA should update its analysis that informed its baseline because we found that the cost and schedule estimates underlying those baselines are not reliable, as they did not conform to best practices. Further, an updated analysis would be beneficial given numerous conditions and risks have changed since the analysis was completed, including unexpected delays to the European Service Module and launch date for the first exploration mission.
Recommendation: To provide the Congress and NASA a reliable estimate of program cost and schedule that are useful to support management and stakeholder decisions, the NASA Administrator should direct the Orion program to perform an updated JCL analysis including updating cost and schedule estimates in adherence with cost and schedule estimating best practices.
Agency Affected: National Aeronautics and Space Administration
Comments: NASA concurred with this recommendation, but characterized its deferral of work to date as task-level deferrals, lasting only several months and not affecting major program milestone or the critical path. NASA did agree to include an analysis of how these deferrals affect budget reserves and program performance in future routine management reporting. NASA officials told us that they are currently evaluating work flow for the first and second mission as the agency revisits the launch date for the first mission. Officials were not able to provide any analysis at this time about the potential cost impact of changes in scheduled work given that this is currently under review.
Recommendation: To have a full understanding of the cost, schedule, and safety impact of deferring work, the NASA Administrator should direct the Orion program to perform an analysis on the cost of deferred work in relation to levels of management reserves and unallocated future expenses and actual contractor performance, and report the results of that analysis to NASA management.
Agency Affected: National Aeronautics and Space Administration