NASA:

Earned Value Management Implementation across Major Spaceflight Projects Is Uneven

GAO-13-22: Published: Nov 19, 2012. Publicly Released: Nov 19, 2012.

Additional Materials:

Contact:

Cristina T. Chaplain
(202) 512-4859
chaplainc@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

What GAO Found

The National Aeronautics and Space Administration's (NASA) 10 major spaceflight projects discussed in this report have not yet fully implemented earned value management (EVM). As a result, NASA is not taking full advantage of opportunities to use an important tool that could help reduce acquisition risk. GAO assessed the 10 projects against three fundamental EVM practices that, according to GAO's best practices cost guide, are necessary for maintaining a reliable EVM system. GAO found shortfalls in two of three fundamental practices. Specifically, we found that

  • More than half of the projects did not use an EVM system that was fully certified as compliant with the industry EVM standard.
  • Only 4 of the 10 projects established formal surveillance reviews, which ensure that key data produced by the system was reliable. The remaining 6 projects provided evidence of monthly EVM data reviews; however, the rigor of both the formal and informal surveillance reviews is questionable given the numerous data anomalies GAO found.

GAO also found that 3 projects had reliable EVM data while 7 had only partially reliable data. For the EVM data to be considered reliable per best practices it must be complete and accurate with all data anomalies explained.

NASA EVM focal points, headquarters officials, project representatives, and program executives cited cultural and other challenges as impediments to the effective use of EVM at the agency. Traditionally, NASA's culture has focused on managing science and engineering challenges and not on monitoring cost and schedule data, like an effective EVM system produces. As a result, several representatives said this information traditionally has not been valued across the agency. This sentiment was also echoed in a NASA study of EVM implementation. Also cited as a challenge to the effective use of EVM was NASA's insufficient number of staff with the skills to analyze EVM data. Without a sufficient number of staff with such skills, NASA's ability to conduct a sound analysis of the EVM data is limited. However, NASA has not conducted an EVM skills gap analysis to determine the extent of its workforce needs.

NASA has undertaken several initiatives aimed at improving the agency's use of EVM. For example, NASA strengthened its spaceflight management policy to reflect the industry EVM standard and has developed the processes and tools for projects to meet these standards through its new EVM system. While these are positive steps, the revised policy contains only the minimum requirements for earned value management. For example, it lacks a requirement for rigorous surveillance of how projects are implementing EVM and also does not require use of the agency's newly developed EVM system to help meet the new requirements. NASA has attempted to address EVM shortcomings through policy changes over the years, but these efforts have failed to adequately address the cultural resistance to implementing EVM.

Why GAO Did This Study

NASA historically has experienced cost growth and schedule slippage in its portfolio of major projects and has taken actions to improve in this area, including adopting the use of EVM. EVM is a tool developed to help project managers monitor risks. GAO was asked to examine (1) the extent to which NASA is using EVM to manage its major space flight acquisitions, (2) the challenges that NASA has faced in implementing an effective EVM system, and (3) NASA's efforts to improve its use of EVM. To address these questions, GAO obtained contractor and project EVM data and used established formulas and tools to analyze the data and assess NASA's implementation of EVM on 10 major spaceflight projects; interviewed relevant NASA headquarters, center and mission directorate officials on their views on EVM; and reviewed prior reports on EVM and organizational transformations. GAO compared NASA policies and guidance on EVM to best practices contained in GAO's cost estimating best practices guide.

What GAO Recommends

GAO recommends that NASA establish a time frame for requiring new spaceflight projects to implement its new EVM system; conduct an EVM skills gap assessment; develop a change management plan for EVM; and strengthen its EVM requirements by requiring projects to implement formal EVM surveillance. NASA concurred with two recommendations and partially concurred with two others citing resource constraints. Despite NASA's plans to address some issues GAO identified, not addressing all key issues lessens the usefulness of EVM at NASA.

For more information, contact Cristina Chaplain at (202) 512-4841 or chaplainc@gao.gov.

Recommendations for Executive Action

  1. Status: Open

    Comments: In fiscal year 2013, NASA started its phased rollout by implementing the EVM Capability process on the Space Launch System project at the Marshall Space Flight Center and the Ice, Clouds and Land Elevation Satellite project at Goddard Space Flight Center. The projects receive implementation support from the agency as part of the rollout. During implementation, the respective Center is to develop an institutional capability to support future projects. In fiscal year 2014, the EVM Capability process was implemented by the Ground Systems Development Office (GSDO) at the Kennedy Space Center (KSC). This included acquiring EVM support, in addition, to providing EVM training to project personnel. KSC has also setup their program baselines in the EVM tool. They are using Cobra and the Empower (replaces wInsight) EVM tool for analysis and reporting. As part of this process, the GSDO was required to support an independent review of its Project Planning and Control capabilities, which included EVM. The EVM Steering Committee has identified the Multi-purpose Crew Vehicle (Orion) project for rollout of the EVM Capability process in fiscal year 2015. EVM tool setup, training and the EVM Assessment review are ongoing. The plan is to support implementation of the common agency EVM process at each NASA center. However, NASA has not established a time frame by which all new spaceflight projects will be required to implement NASA's newly developed EVM process.

    Recommendation: To improve NASA management and oversight of its spaceflight projects, the NASA Administrator should direct the appropriate offices to establish a time frame by which all new spaceflight projects will be required to implement NASA's newly developed EVM system, unless the project is proposing to use a certified system, to ensure that in-house efforts are compliant with ANSI/EIA-748. The time frame selected should take in to account the need to increase NASA's institutional capability for conducting EVM and analyzing and reporting the data.

    Agency Affected: National Aeronautics and Space Administration

  2. Status: Closed - Implemented

    Comments: NASA completed the EVM skill gap survey. In addition, NASA developed an EVM training plan and briefed to the EVM Working Group which is comprised of organizational EVM Focal Points. They have responsibility for supporting the projects within their organizations with EVM support and training. Some of the new courses, i.e., EVM tools, within the training plan have already been taught at the Marshall Space Flight Center, Kennedy Space Center, Johnson Space Center and Goddard Space Flight Center. One of the outcomes of the skill gap survey was that it identified that 98 percent of NASA's EVM requirements are at these four Centers. Therefore, the focus in the short-term will be to provide EVM training to those four Centers. Other training courses will be taught at the other Centers over the next few months/years. In addition, some of the existing courses were updated to ensure that the agency's training packages are current. Also, the agency also is working closely with the Project Planning and Control working group to include the EVM training as part of the agency's overall Project Planning and Control training curriculum.

    Recommendation: To improve NASA management and oversight of its spaceflight projects, the NASA Administrator should direct the appropriate offices to conduct an EVM skills gap analysis to identify areas requiring augmented capability across the agency. Based on the results of the assessment, develop a workforce training plan to address any deficiencies.

    Agency Affected: National Aeronautics and Space Administration

  3. Status: Closed - Implemented

    Comments: NASA developed a change management plan and it was approved by the EVM Steering Committee. The EVM Steering Committee is comprised of senior level NASA officials and provides guidance to the agency on the implementation of EVM. The plan has been briefed to the EVM Focal Points and is expected to be implemented by the EVM Focal Points within their perspective organizations. The plan focuses on the rollout of the EVM Capability, EVM surveillance, and training.

    Recommendation: To improve NASA management and oversight of its spaceflight projects, the NASA Administrator should direct the appropriate offices to develop an EVM change management plan to assist managers and employees throughout the agency with accepting and embracing earned value techniques while reducing the operational impact on the agency. The plan should include a strategy for having the agency's senior leadership communicate their commitment to implementation of EVM.

    Agency Affected: National Aeronautics and Space Administration

  4. Status: Open

    Comments: NASA has not modified its NASA Procedural Requirements (NPR) 7120.5 to require projects to implement a formal surveillance program. NASA has, however, taken steps to provide information on implementing EVM surveillance and assessment. For example, NASA finalized its EVM Handbook which contains an acceptance and surveillance process. The EVM handbook is now available on the NASA EVM website for all NASA projects, though use of the handbook is not required. In addition, NASA has conducted EVM Assessments to test the maturity of its projects. For example, the agency has conducted two EVM Assessments or Project Planning and Control Assessments in fiscal year 2014 and is in the process of conducting a third review. The completed reviews were for the Ground Systems Development and Operations and OSIRIS-Rex projects. According to NASA, both reviews were well received and provided feedback to the projects to help strengthen their EVM implementations. The Multi-purpose Crew Vehicle (Orion) EVM Assessment is ongoing and will be completed in early fiscal year 2015. In addition, the agency conducted an initial EVM Validation Review at the Applied Physics Laboratory facility. The final EVM Validation review is scheduled for early April 2015. This will increase the likelihood that the data being supplied by Applied Physics Laboratory to NASA projects is reliable. Automated anomalies reports have been briefed to the EVM Focal Points and are available to the projects across the agency. Part of the change management plan is for the organizational EVM Focal Points to educate their various projects on the tools, how to access and use them to strengthen surveillance across the agency.

    Recommendation: To improve NASA management and oversight of its spaceflight projects, and to improve the reliability of project EVM data, the NASA Administrator should direct the appropriate offices to modify the NASA Procedural Requirements (NPR) 7120.5 to require projects to implement a formal surveillance program that: (1) Ensures anomalies in contractor-delivered and in-house monthly earned value management reports are identified and explained, and report periodically to the center and mission directorate's leadership on relevant trends in the number of unexplained anomalies. (2) Ensures consistent use of work breakdown structures (WBS) for both the EVM report and the schedule. (3) Ensures that lower level EVM data reconcile to project level EVM data using the same WBS structure. (4) Improves underlying schedules so that they are properly sequenced using predecessor and successor dependencies and are free of constraints to the extent practicable so that the EVM baseline is reliable.

    Agency Affected: National Aeronautics and Space Administration

 

Explore the full database of GAO's Open Recommendations »

Jul 23, 2014

Jul 16, 2014

Jun 20, 2014

May 15, 2014

May 8, 2014

Apr 15, 2014

Mar 12, 2014

Feb 4, 2014

Jan 13, 2014

Jan 8, 2014

Looking for more? Browse all our products here