Information Technology

Agencies Need to Improve the Implementation and Use of Earned Value Techniques to Help Manage Major System Acquisitions

GAO-10-2, Nov 6, 2009

Additional Materials:

Contact:

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

In fiscal year 2009, the federal government planned to spend about $71 billion on information technology (IT) investments. To more effectively manage such investments, in 2005 the Office of Management and Budget (OMB) directed agencies to implement earned value management (EVM). EVM is a project management approach that, if implemented appropriately, provides objective reports of project status, produces early warning signs of impending schedule delays and cost overruns, and provides unbiased estimates of anticipated costs at completion. GAO was asked to assess selected agencies' EVM policies, determine whether they are adequately using earned value techniques to manage key system acquisitions, and eval- uate selected investments' earned value data to determine their cost and schedule performances. To do so, GAO compared agency policies with best practices, performed case studies, and reviewed documenta- tion from eight agencies and 16 major investments with the highest levels of IT development-related spending in fiscal year 2009.

While all eight agencies have established policies requiring the use of EVM on major IT investments, these policies are not fully consistent with best practices. In particular, most lack training requirements for all relevant personnel responsible for investment oversight. Most policies also do not have adequately defined criteria for revising program cost and schedule baselines. Until agencies expand and enforce their EVM policies, it will be difficult for them to gain the full benefits of EVM. GAO's analysis of 16 investments shows that agencies are using EVM to manage their system acquisitions; however, the extent of implementation varies. Specifically, for 13 of the 16 investments, key practices necessary for sound EVM execution had not been implemented. For example, the project schedules for these investments contained issues--such as the improper sequencing of key activities--that undermine the quality of their performance baselines. This inconsistent application of EVM exists in part because of the weaknesses contained in agencies' policies, combined with a lack of enforcement of policies already in place. Until key EVM practices are fully implemented, these investments face an increased risk that managers cannot effectively optimize EVM as a management tool. Furthermore, earned value data trends of these investments indicate that most are currently experiencing shortfalls against cost and schedule targets. The total life-cycle costs of these programs have increased by about $2 billion. Based on GAO's analysis of current performance trends, 11 programs will likely incur cost overruns that will total about $1 billion at contract completion--in particular, 2 of these programs account for about 80 percent of this projection. As such, GAO estimates the total cost overrun to be about $3 billion at program completion (see figure). However, with timely and effective management action, it is possible to reverse negative trends so that the projected cost overruns may be reduced.

Status Legend:

More Info

Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.

  • In Process
  • Open
  • Closed - implemented
  • Closed - not implemented

Recommendations for Executive Action

Recommendation: In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should modify policies governing EVM to ensure that they address the weaknesses that we identified, taking into consideration the criteria used in this report.

Agency Affected: Department of Homeland Security

Status: Open

Comments: As of May 2011, the Department of Homeland Security stated that an earned value management (EVM) Working Group was formed on January 25, 2011 to update the existing EVM Guidance, V 1.2, dated 2007. The final draft is scheduled for review on July 18, 2011. We will continue to monitor the department's progress in implementing this recommendation.

Recommendation: In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should modify policies governing EVM to ensure that they address the weaknesses that we identified, taking into consideration the criteria used in this report.

Agency Affected: Department of Agriculture

Status: Open

Comments: The Department of Agriculture updated its policy governing earned value management (EVM), including establishing criteria for program rebaselining; however, the updated policy does not include a standard structure for defining work products. In addition, the department developed an EVM training for executives; however, this training is not required in policy, nor has it been offered. We will continue to monitor the department's progress in implementing this recommendation.

Recommendation: In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should modify policies governing EVM to ensure that they address the weaknesses that we identified, taking into consideration the criteria used in this report.

Agency Affected: Department of Commerce

Status: Open

Comments: As of August 2010, the Department of Commence had not updated its policies governing EVM to include a standard structure for defining work products. We will continue to monitor the department's progress in implementing this recommendation.

Recommendation: In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should modify policies governing EVM to ensure that they address the weaknesses that we identified, taking into consideration the criteria used in this report.

Agency Affected: Department of Defense

Status: Open

Comments: As of February 2011, the Department of Defense (DOD) stated that an Earned Value Management (EVM) integrated program team is currently undertaking a review of the Defense Acquisition University EVM curricula. Further, DOD stated that, upon completion of this review, that the integrated program team will provide recommendations on ways for the university's EVM curricula to better meet the EVM needs of the DOD services and agencies. We will continue to monitor the department's progress in implementing this recommendation.

Recommendation: In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should modify policies governing EVM to ensure that they address the weaknesses that we identified, taking into consideration the criteria used in this report.

Agency Affected: Department of Justice

Status: Closed - Implemented

Comments: In November 2009, the Department of Justice updated its earned value management (EVM) policy to require training for all personnel with investment oversight and program management responsibilities. In addition, the updated policy requires the use of a standard structure for defining work products.

Recommendation: In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should modify policies governing EVM to ensure that they address the weaknesses that we identified, taking into consideration the criteria used in this report.

Agency Affected: Department of Transportation

Status: Closed - Implemented

Comments: In July 2010, the Department of Transportation updated its earned value management (EVM) policy to include consistent requirements for complying with industry EVM standards. In addition, the updated policy requires EVM training for executive-level staff and the use of a standard structure for defining work products.

Recommendation: In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should modify policies governing EVM to ensure that they address the weaknesses that we identified, taking into consideration the criteria used in this report.

Agency Affected: Department of Veterans Affairs

Status: Open

Comments: Veterans Affairs' officials stated that the agency is incorporating the critical earned value management (EVM) practices of into its new Program Management Accountability System (PMAS). When we confirm how PMAS addresses the weaknesses we identified in the agency's EVM policy, we will provide updated information.

Recommendation: In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should modify policies governing EVM to ensure that they address the weaknesses that we identified, taking into consideration the criteria used in this report.

Agency Affected: National Aeronautics and Space Administration

Status: Open

Comments: NASA officials stated that the agency is in the process of developing an interim directive for its IT project management policy that establishes a standard work breakdown structure and defines rebaselining criteria for IT projects. When we confirm these and other actions the agency has taken in response to this recommendation, we will provide updated information.

Recommendation: In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should direct key system acquisition programs to implement the EVM practices that address the detailed weaknesses that we identified in appendix II, taking into consideration the criteria used in this report.

Agency Affected: Department of Homeland Security

Status: Open

Comments: As of May 2011, the Department of Homeland Security stated that the agency was preparing a policy that would direct key system acquisition programs to implement earned value management (EVM) and will include requirements for contracts over $20 million in value regardless of contract type. The final draft of the policy is expected to be completed by fourth quarter in fiscal year 2011. We will continue to monitor the department's progress in implementing this recommendation.

Recommendation: In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should direct key system acquisition programs to implement the EVM practices that address the detailed weaknesses that we identified in appendix II, taking into consideration the criteria used in this report.

Agency Affected: Department of Agriculture

Status: Open

Comments: The Modernize and Innovate the Delivery of Agriculture Services (MIDAS) program conducted an integrated baseline review in October 2010, but is currently in the process of revising its integrated master schedule. When we confirm what further actions the program has taken, we will update the recommendation status with new information.

Recommendation: In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should direct key system acquisition programs to implement the EVM practices that address the detailed weaknesses that we identified in appendix II, taking into consideration the criteria used in this report.

Agency Affected: Department of Commerce

Status: Closed - Implemented

Comments: The Field Data Collection Automation (FDCA) program improved its ability to establish a comprehensive EVM system by implementing metrics to analyze the program schedule and determine if actions are needed to address scheduling issues, such as the improper sequencing of tasks. Further, the program improved its ability to ensure reliable data by integrating EVM data analysis into tools used for monitoring contractor performance. Lastly, the FDCA program has taken steps to improve its use of EV data to manage the acquisition. For example, the program took action to mitigate risks by integrating risk review and analysis into the Decennial Census performance dashboard.

Recommendation: In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should direct key system acquisition programs to implement the EVM practices that address the detailed weaknesses that we identified in appendix II, taking into consideration the criteria used in this report.

Agency Affected: Department of Defense

Status: Closed - Implemented

Comments: The Warfighter Information Network-Tactical (WIN-T) and the Joint Tactical Radio System-Handheld, Manpack, Small Form Fit (JTRS-HMS) programs took actions to implement key earned value management (EVM) practices. For example, in May 2010, the WIN-T program conducted an integrated baseline review and identified a list of action items intended to improve, among other things, the measurement of earned value and the quality of the program's cost and schedule baselines. The program also improved its ability to use EVM data for decision-making purposes by updating the performance baseline as program changes occurred. Specifically, the program management team documented baseline changes requests to identify work activities that had increased or decreased in value and the rationale for the adjustments, which were subsequently approved by the WIN-T Program Manager. In addition, the JTRS-HMS program sought assistance from the Defense Contract Management Agency to evaluate the program's master schedule on a monthly-basis to identify weaknesses for corrective action.

Recommendation: In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should direct key system acquisition programs to implement the EVM practices that address the detailed weaknesses that we identified in appendix II, taking into consideration the criteria used in this report.

Agency Affected: Department of Transportation

Status: Open

Comments: As of June 2011, the En Route Automation Modernization (ERAM) program was in the process of rebaselining the cost and schedule of the program. ERAM officials stated that, upon completion of this effort, the program plans to revise the program schedule and establish a program-wide performance measurement baseline. We will continue to monitor the department's progress in implementing this recommendation.

Recommendation: In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should direct key system acquisition programs to implement the EVM practices that address the detailed weaknesses that we identified in appendix II, taking into consideration the criteria used in this report.

Agency Affected: Department of Veterans Affairs

Status: Closed - Implemented

Comments: The Department of Veterans Affairs formulated policies as part of its Program Management and Accountability System that incorporate critical practices of EVM. Based on these new policies, the department found the VistA-FM program to be poorly planned and managed, areas of which we also reported as having weaknesses. As a result of this evaluation and our recommendation, the department eliminated the program.

Recommendation: In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should direct key system acquisition programs to implement the EVM practices that address the detailed weaknesses that we identified in appendix II, taking into consideration the criteria used in this report.

Agency Affected: National Aeronautics and Space Administration

Status: Open

Comments: NASA officials described several improvements in its programs' EVM practices. For example, officials stated that the James Webb Space Telescope program has modified its contract to require monthly variance analysis. For the Juno program, officials stated that an integrated baseline review had been held for the Flight System. For the Mars Science Laboratory, officials stated that an estimate-to-complete analysis had been conducted. When we confirm these and other actions taken by the agency, we will provide updated information.

Recommendation: In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should direct key system acquisition programs to take action to reverse current negative performance trends, as shown in the earned value data, to mitigate the potential cost and schedule overruns.

Agency Affected: Department of Homeland Security

Status: Open

Comments: As of May 2011, the Department of Homeland Security stated that the agency is in the process of developing new earned value management (EVM) training requirements that extend beyond a practitioner level course, to include senior leaders and program managers. Officials stated that the extended courses are intended demonstrate why EVM is critical to effective program management and how it offers program managers the ability to forecast negative trends and events. In addition, officials stated that a Cost and Schedule Working Group is considering using "tiger teams" to work with troubled programs. We will continue to monitor the department's progress in implementing this recommendation.

Recommendation: In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should direct key system acquisition programs to take action to reverse current negative performance trends, as shown in the earned value data, to mitigate the potential cost and schedule overruns.

Agency Affected: Department of Agriculture

Status: Open

Comments: As of July 2010, the Modernize and Innovate the Delivery of Agriculture Services (MIDAS) program had not yet taken action to reverse negative performance trends to mitigate potential cost and schedule overruns. When we confirm actions taken to mitigate and reverse these trends, we will provide updated information.

Recommendation: In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should direct key system acquisition programs to take action to reverse current negative performance trends, as shown in the earned value data, to mitigate the potential cost and schedule overruns.

Agency Affected: Department of Commerce

Status: Closed - Implemented

Comments: The Field Data Collection Automation (FDCA) program improved its ability to assess and to make timely decisions about contractor performance by integrating earned value data analysis into key program management tools. As a result, by July 2010, the FDCA program had reversed negative performance trends and had outperformed it planned cost target by about $5 million.

Recommendation: In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should direct key system acquisition programs to take action to reverse current negative performance trends, as shown in the earned value data, to mitigate the potential cost and schedule overruns.

Agency Affected: Department of Defense

Status: Closed - Implemented

Comments: The Warfighter Information Network-Tactical (WIN-T) program and the Joint Tactical Radio System-Handheld, Manpack, Small Form Fit (JTRS-HMS) programs took actions to improve their negative performance trends. For example, in May 2010, the WIN-T program conducted a review of the performance baseline and identified a list of action items intended to improve, among other things, the measurement of earned value and the quality of the program's schedule baseline. The WIN-T Program Office and prime contractor also conducted comprehensive "deep dive" reviews at a major subcontractor where schedule performance data for planned work were discussed. By January 2011, the WIN-T program had reduced its schedule variance by approximately $4.5 million. In addition, the JTRS-HMS program restructured the contractor management team overseeing system development efforts by replacing key personnel, such as the JTRS Program Director, and reorganizing staff to better align with specific work products. Further, the program identified significant schedule drivers and took action to address them by, for example, revising the program schedule to focus on the delivery of smaller incremental capabilities. By April 2011, the JTRS-HMS program had reduced its schedule variance by approximately $6.3 million.

Recommendation: In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should direct key system acquisition programs to take action to reverse current negative performance trends, as shown in the earned value data, to mitigate the potential cost and schedule overruns.

Agency Affected: Department of Justice

Status: Closed - Implemented

Comments: In August 2010, the Next Generation Identification (NGI) program established an Executive Change Control Board to better prevent contract scope increases. This board is responsible for, among other things, reviewing and approving any change requests that affect the scope, milestones, and/or cost of the NGI development contract. In addition, the department established an independent verification and validation contractor with responsibility for independently reporting on a monthly basis the status of the NGI program, including earned value data. As a result, by February 2011, the NGI had reduced the program's cost variance to approximately 1.6 percent of the cumulative cost of work performed, thus improving cost performance and reducing the potential cost overruns expected on this key system acquisition program.

Recommendation: In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should direct key system acquisition programs to take action to reverse current negative performance trends, as shown in the earned value data, to mitigate the potential cost and schedule overruns.

Agency Affected: Department of Transportation

Status: Closed - Implemented

Comments: The Surveillance and Broadcast System (SBS) program reported on earned value data trends to an executive-level council on a monthly basis and, by July 2010, had reduced its negative schedule variance.

Recommendation: In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should direct key system acquisition programs to take action to reverse current negative performance trends, as shown in the earned value data, to mitigate the potential cost and schedule overruns.

Agency Affected: Department of Veterans Affairs

Status: Closed - Implemented

Comments: The Department of Veterans Affairs formulated policies as part of its Program Management and Accountability System that incorporate critical practices of EVM. Based on these new policies, the department found the VistA-FM program to be poorly planned and managed, areas of which we also reported as having weaknesses. As a result of this evaluation and our recommendation, the department eliminated the program.

Recommendation: In order to address the weaknesses identified in agencies' policies and practices in using EVM, the Secretaries of the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Transportation, and Veterans Affairs and the Administrator of the National Aeronautics and Space Administration should direct key system acquisition programs to take action to reverse current negative performance trends, as shown in the earned value data, to mitigate the potential cost and schedule overruns.

Agency Affected: National Aeronautics and Space Administration

Status: Open

Comments: When we confirm what actions the agency has taken in response to this recommendation, we will provide updated information.