Information Technology Cost Estimation:

Agencies Need to Address Significant Weaknesses in Policies and Practices

GAO-12-629: Published: Jul 11, 2012. Publicly Released: Jul 11, 2012.

Additional Materials:

Contact:

Valerie C. Melvin
(202) 512-3000
melvinv@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

What GAO Found

While the eight agencies GAO reviewed—the Departments of Agriculture, Commerce, Defense, Homeland Security, Justice, Labor, and Veterans Affairs, and the Environmental Protection Agency—varied in the extent to which their cost-estimating policies and procedures addressed best practices, most had significant weaknesses. For example, six of the eight agencies had established a clear requirement for programs to develop life-cycle cost estimates. However, most of the eight agencies’ policies lacked requirements for cost-estimating training, a standard structure for defining work products, and a central, independent cost-estimating team, among other things. The weaknesses in agencies’ policies were due, in part, to the lack of a priority for establishing or enhancing department or agency-level cost-estimating functions. Until agencies address weaknesses in their policies, it will be difficult for them to make effective use of program cost estimates for informed decision making, realistic budget formation, and meaningful progress measurement.

The 16 major acquisition programs had developed cost estimates and were using them, in part, to support program and budget decisions. However, all but 1 of the estimates were not fully reliable—meaning that they did not fully reflect all four characteristics of a reliable cost estimate identified in the GAO cost-estimating guide: comprehensive, well-documented, accurate, and credible. For example, the estimates for many of these investments did not include all life-cycle costs, such as costs for operating and maintaining the system; did not adequately document the source data and methodologies used to develop the estimate; were not regularly updated so that they accurately reflected current status; and lacked credibility because they were not properly adjusted to account for risks and uncertainty. The inadequate implementation of cost-estimating best practices was largely due to weaknesses in agencies’ policies. Until cost-estimating best practices are fully implemented, these programs face an increased risk that managers will not be able to effectively use their cost estimates as a sound basis for informed program and budget decision making.

Why GAO Did This Study

The federal government plans to spend at least $75 billion on information technology (IT) investments in fiscal year 2012. The size of this investment highlights the importance of reliably estimating the costs of IT acquisitions. A reliable cost estimate is critical to the success of any IT program, providing the basis for informed decision making and realistic budget formation. Without the ability to generate such estimates, programs risk missing their cost, schedule, and performance targets.

GAO was asked to (1) assess selected federal agencies’ implementation of cost-estimating policies and procedures, and (2) evaluate whether selected IT investments at these agencies have reliable cost estimates to support budget and program decisions. To do so, GAO compared policies and procedures to best practices at eight agencies. GAO also reviewed documentation supporting cost estimates for 16 major investments at these eight agencies—representing about $51.5 billion of the planned IT spending for fiscal year 2012.

What GAO Recommends

GAO is recommending that the selected agencies modify cost-estimating policies to be consistent with best practices and update future cost estimates of the selected acquisition programs to address identified weaknesses. The seven agencies that commented on a draft of this report generally agreed with GAO’s results and recommendations, although the Environmental Protection Agency disagreed with the assessment of one of its investments. However, GAO stands by its assessment.

For more information, contact Valerie C. Melvin at (202) 512-6304 or melvinv@gao.gov.

Recommendations for Executive Action

  1. Status: Closed - Implemented

    Comments: The Department of Agriculture (USDA) has implemented this recommendation. USDA modified its cost estimating policy by creating a departmental regulation (Departmental Regulation (DR) 3130-012, dated March 4, 2016) as policy for governing information technology (IT) cost estimating procedures. The regulation requires the use of the GAO Cost Estimating and Assessment Guide and its 12-step process for all IT investments. The GAO guide, which represents a comprehensive presentation of GAO identified best practices that are essential for effective cost estimating guidance, is now a principal USDA IT cost estimating policy element. As a result, USDA has addressed the cost estimating policy weaknesses that we identified in our report. For example, USDA policy now includes ensuring that estimates identify and require compliance with cost-estimating best practices, and validating their use; and requiring that estimates be reviewed and approved by management, among other practices. The use of these practices will lead to cost estimates that are more reliable with more comprehensive, well-documented, accurate, and credible attributes.

    Recommendation: To address weaknesses identified in the Department of Agriculture's policies for cost estimating, we recommend that the Secretary of Agriculture direct responsible officials to modify policies governing cost estimating to ensure that they address the weaknesses that we identified.

    Agency Affected: Department of Agriculture

  2. Status: Closed - Implemented

    Comments: The Department of Commerce has implemented this recommendation by modifying its policy governing cost estimates. In June 2014, the department issued a revised policy on cost estimation and independent cost estimates. This policy presented a clear requirement for cost estimating; stated that cost estimates use GAO best practices; and specified that the GAO Cost Estimating and Assessment Guide is a key resources for use in cost estimating activities. The policy required that cost estimates contain attributes that allow them to be reliable, comprehensive, well-documented, accurate and credible as proscribed in our report. Additionally, in May 2015, Commerce issued a Departmental Administrative Order entitled Acquisition Project Management (DAO 208-16) that further improved its cost estimating policy. This guidance specified requirements that addressed weaknesses that we identified in our report including management review and training requirements. The use of these practices will lead to cost estimates that are more reliable with more comprehensive, well-documented, accurate, and credible attributes.

    Recommendation: To address weaknesses identified in the Department of Commerce's policies for cost estimating, we recommend that the Secretary of Commerce direct responsible officials to modify policies governing cost estimating to ensure that they address the weaknesses that we identified.

    Agency Affected: Department of Commerce

  3. Status: Closed - Implemented

    Comments: The Department of Homeland Security has addressed our recommendation. As of October 2014, the department issued an updated Cost Estimating Handbook. The Handbook includes policies addressing the weaknesses we identified in our report. For example, it established requirements for cost estimating training, a standard structure for defining work products, and a process to collect and store cost-related data. The updated policies established in the handbook provide department officials with sound program cost estimates that may be used for informed decision making on how to most effectively manage their portfolios of projects.

    Recommendation: To address weaknesses identified in the Department of Homeland Security's policies for cost estimating, we recommend that the Secretary of Homeland Security direct responsible officials to modify policies governing cost estimating to ensure that they address the weaknesses that we identified.

    Agency Affected: Department of Homeland Security

  4. Status: Closed - Implemented

    Comments: The Department of Labor has generally implemented this recommendation by updating its policies governing cost estimates and establishing tools for defining work products and collecting cost related data. For example, Labor's updated System Development Life-cycle guide requires the use of standard work beak-down structures (WBS) at the beginning of the Conceptual Planning Phase and updated as necessary for each of the remaining life cycle phases. The guidance also requires the project managers to create a WBS dictionary that defines the work activities, deliverables and estimated resources needed to complete the work activities which help the department track cost and schedule by defined deliverables, and results in more consistent cost estimates. In addition, project managers are required to conduct a gate review at the end of each phase to determine whether all required WBS tasks have been completed successfully. Labor also established a cost benefit analysis tool which provides the department a standardized approach for performing quantitative analysis of the life cycle costs and benefits associated with each alternative to its IT investments. Additionally, Labor's IT Cost Estimate guide which is based on the GAO Cost Estimating and Assessment Guide, establishes a process for collecting and storing cost-related data which allows the cost data to be available to support future estimates. Labor, however, only partially addressed the policy on providing cost-estimating training. Labor's policy provides cost-estimating training and a mechanism to track participation, but the department did not provide evidence that its policy addressed providing training to personnel with investment oversight responsibility. In addition, while the guidance calls for the project manager (PM) to have the estimates independently reviewed by another PM, the guidance does not require an independent cost estimate conducted by a group outside the acquiring organization. Nevertheless, fully implementing its cost estimating guidance should provide Labor a basis for effective program cost estimating and result in more reliable cost estimates that management can use for making informed decisions.

    Recommendation: To address weaknesses identified in the Department of Labor's policies for cost estimating, we recommend that the Secretary of Labor direct responsible officials to modify policies governing cost estimating to ensure that they address the weaknesses that we identified.

    Agency Affected: Department of Labor

  5. Status: Closed - Implemented

    Comments: The Department of Veterans Affairs has addressed our recommendation. Specifically, in response to our recommendation, the department informed GAO updated its Cost Estimating Guide. The cost estimating policies and procedures in the updated guide addressed the weaknesses we found with its cost estimating policy. For example, the guide established a process to collect and store cost related data. Specifically, all program cost estimates must be uploaded to a registry that is used to establish a central repository and knowledge base to inform future cost analysis. Establishing an agency-wide process to collect and store cost related data will improve the data available to all programs and increase the efficiency of developing cost estimates. As a result, program officials across the department will have reliable cost estimates to make informed program management decisions reducing the likelihood of cost overruns, missed deadlines, and performance shortfalls.

    Recommendation: To address weaknesses identified in the Department of Veterans Affairs' policies for cost estimating, we recommend that the Secretary of Veterans Affairs direct responsible officials to modify policies governing cost estimating to ensure that they address the weaknesses that we identified.

    Agency Affected: Department of Veterans Affairs

  6. Status: Open

    Comments: We have not yet validated agency action on this recommendation; we are awaiting agency information.

    Recommendation: To address weaknesses identified in the Department of Justice's policies for cost estimating, we recommend that the Attorney General direct responsible officials to modify policies governing cost estimating to ensure that they address the weaknesses that we identified.

    Agency Affected: Department of Justice

  7. Status: Closed - Implemented

    Comments: The Environmental Protection Agency (EPA) has implemented this recommendation by modifying its cost estimating policy. Specifically, EPA updated its System Life Cycle Management Procedure document that defines the requirements and processes for implementation of the Agency system life cycle management (SLCM) policy. It directs EPA Project Managers to use the GAO Cost Estimating and Assessment guidance as reference for cost estimating activities; and specifies that the GAO Guide is part of EPA related policy, standards and guidance. The GAO guide specifically addresses the EPA cost estimating weaknesses described in our report, including establishment of an agency-wide process to collect and store cost-related data; a system to enforce training requirements for personnel with program management and investment oversight responsibilities; and a standard structure for defining work products. Consequently, usage of the best practice GAO guide will reduce the likelihood that the EPA cost estimates will only partially reflect key activities for developing a comprehensive, well-documented, and credible estimate. As a result, the risks of cost overruns, missed deadlines, and performance shortfalls will be significantly reduced.

    Recommendation: To address weaknesses identified in the Environmental Protection Agency's policies for cost estimating, we recommend that the Administrator of the Environmental Protection Agency direct responsible officials to modify policies governing cost estimating to ensure that they address the weaknesses that we identified.

    Agency Affected: Environmental Protection Agency

  8. Status: Closed - Implemented

    Comments: The Department of Agriculture has implemented this recommendation by directing that a departmental regulation requiring use of the GAO cost estimating guide and its 12-step process be utilized for all IT cost estimating activities. The GAO guide, which represents a comprehensive presentation of GAO identified best practices that are essential for effective cost estimating guidance and served as criteria for our assessment of program cost estimates, is now a principal feature of USDA IT cost estimating activities. By specifying that the GAO Guide be used, the USDA regulation presents the best practices governing characteristics of a reliable cost estimate. For example, among other practices, the GAO Guide requires that program cost estimates be updated to reflect actual costs and cost changes. As a result of this action, responsible USDA officials are positioned to update future life-cycle cost estimates using cost-estimating practices that will lead to cost estimates that have more comprehensive, well-documented, accurate, and credible attributes and program officials will have a more reliable basis for increasing the likelihood that program cost estimates account for all possible costs.

    Recommendation: To address weaknesses identified in the Department of Agriculture's practices for cost estimating, we recommend that the Secretary of Agriculture direct responsible officials to update future life-cycle cost estimates of the system acquisition programs discussed in this report using cost-estimating practices that address the detailed weaknesses that we identified.

    Agency Affected: Department of Agriculture

  9. Status: Closed - Implemented

    Comments: The Department of Commerce has implemented this recommendation by directing program managers to follow GAO cost estimating principles and best practices and by specifying that the GAO Cost Estimating Guide be used as a key resource governing cost estimating activities. This action applies to all acquisition programs, projects and cost estimating activities throughout the Department; addresses the need to regularly update estimates; and requires that cost estimates contain attributes that allow them to be reliable, comprehensive, well-documented, accurate and credible as described in our report. By incorporating all practices advocated by GAO, including usage in developing future cost estimations, future life-cycle cost estimates of the system acquisition programs discussed in our report will be directed to use cost-estimating practices that will improve the basis for budgetary decisions.

    Recommendation: To address weaknesses identified in the Department of Commerce's practices for cost estimating, we recommend that the Secretary of Commerce direct responsible officials to update future life-cycle cost estimates of the system acquisition programs discussed in this report using cost-estimating practices that address the detailed weaknesses that we identified.

    Agency Affected: Department of Commerce

  10. Status: Closed - Implemented

    Comments: The Department of Homeland Security has addressed our recommendation. Specifically, in response to our recommendation, in December 2014, we confirmed that DHS officials directed IPAWS and the Rescue 21 management to use the revised DHS Cost Estimating Handbook for future cost estimations. This updated handbook established practices governing the characteristics of a reliable cost estimate and addressed the weaknesses of the IPAWS and Rescue 21 program cost estimates we identified in our report. For example, the handbook establishes the required practices of identifying data and data sources, and normalizing the data to improve its consistency and comparability, among others. In addition, the IPAWS program office updated its cost estimate in December 2015 and included improvements such as having 10 years of operations and maintenance costs beyond the end of development ensuring a more comprehensive cost estimate. As a result, program officials have a more robust basis for informed budget making decisions reducing the likelihood that IPAWS and Rescue 21 will have cost overruns.

    Recommendation: To address weaknesses identified in the Department of Homeland Security's practices for cost estimating, we recommend that the Secretary of Homeland Security direct responsible officials to update future life-cycle cost estimates of the system acquisition programs discussed in this report using cost-estimating practices that address the detailed weaknesses that we identified.

    Agency Affected: Department of Homeland Security

  11. Status: Closed - Implemented

    Comments: The Department of Labor has generally addressed this recommendation. As of May 2016, the Department revised its cost estimating guidance and practices used when updating the life-cycle estimate to address weaknesses we identified. For example, the guidance requires the cost estimate to include both government and contractor costs of the program over its full life cycle. The cost estimate should also be based on a work breakdown structure that allows a program to track cost and schedule by defined deliverables which helps ensure a comprehensive and well documented cost estimate. In addition, Labor's revised cost benefit analysis tool guidance requires that cost estimates be updated regularly so that they are accurate and reflect current status. Further, Labor's revised guidance requires risk analysis and sensitivity analysis to determine the level of risk associated with the cost estimate and requires risk adjusted figures to be included in the estimate which helps ensure that the estimate is credible. However, while the guidance calls for the project manager (PM) to have the estimates independently reviewed by another PM who is experienced with similar projects, the guidance does not require an independent cost estimate conducted by a group outside the acquiring organization. Nevertheless, fully implementing the guidance should help Labor address weaknesses we identified and guide consistent and reliable cost estimates that management can use as a basis for making informed decisions.

    Recommendation: To address weaknesses identified in the Department of Labor's practices for cost estimating, we recommend that the Secretary of Labor direct responsible officials to update future life-cycle cost estimates of the system acquisition programs discussed in this report using cost-estimating practices that address the detailed weaknesses that we identified.

    Agency Affected: Department of Labor

  12. Status: Closed - Implemented

    Comments: The Department of Veterans Affairs(VA)has implemented this recommendation by creating a VA Cost Estimating Guide in response to our report. The VA Guide directs project managers to use the GAO Cost Estimating and Assessment guidance as reference for cost estimating activities; and specifies that the GAO Guide is part of VA intentions to establish requirements and standards for its cost estimating activities. The VA guidance applies to all programs and sub-programs subject to cost estimates; requires usage of the GAO Cost Estimating Guide and its 12-step cost estimating process and incorporates all practices advocated by GAO; and is intended for use throughout the department; including usage in developing future cost estimations. As a result of this action, responsible VA officials are positioned to update future life-cycle cost estimates of the system acquisition programs discussed in our report using cost-estimating practices that will allow estimates to be more comprehensive, well-documented, accurate and credible; and reduce the likelihood of inaccurate or unreliable cost estimates.

    Recommendation: To address weaknesses identified in the Department of Veterans Affairs' practices for cost estimating, we recommend that the Secretary of Veterans Affairs direct responsible officials to update future life-cycle cost estimates of the system acquisition programs discussed in this report using cost-estimating practices that address the detailed weaknesses that we identified.

    Agency Affected: Department of Veterans Affairs

  13. Status: Open

    Comments: We have not yet validated agency action on this recommendation; we are awaiting agency information.

    Recommendation: To address weaknesses identified in the Department of Justice's practices for cost estimating, we recommend that the Attorney General direct responsible officials to update future life-cycle cost estimates of the system acquisition programs discussed in this report using cost-estimating practices that address the detailed weaknesses that we identified.

    Agency Affected: Department of Justice

  14. Status: Closed - Implemented

    Comments: The Environmental Protection Agency (EPA) has implemented this recommendation by directing project managers to use the GAO Cost Estimating and Assessment guidance as reference for cost estimating activities; and specifying that the GAO Guide is part of EPA related policy, standards and guidance. The EPA guidance, which incorporates all practices advocated by GAO, is intended for use throughout the department including usage in developing future cost estimations. As a result of this action, responsible EPA officials are positioned to update future life-cycle cost estimates of the system acquisition programs discussed in our report using cost-estimating practices that will allow estimates to be more comprehensive, well-documented, accurate and credible.

    Recommendation: To address weaknesses identified in the Environmental Protection Agency's practices for cost estimating, we recommend that Administrator of the Environmental Protection Agency direct responsible officials to update future life-cycle cost estimates of the system acquisition programs discussed in this report using cost-estimating practices that address the detailed weaknesses that we identified.

    Agency Affected: Environmental Protection Agency

  15. Status: Closed - Implemented

    Comments: The Pension Benefit Guaranty Corporation has addressed our recommendation. As of October 2014, the agency revised its cost estimating practices used when updating the life-cycle cost estimate to address the weaknesses we identified. For example, the program office now tracks IT program data centrally ensuring the formation of a well documented, accurate, and credible cost estimate. As a result, program officials have a sound basis for informed budget making decisions and reducing the likelihood of underfunding and cost overruns.

    Recommendation: To address weaknesses identified in the Pension Benefit Guaranty Corporation's practices for cost estimating, we recommend that the Director of the Pension Benefit Guaranty Corporation direct responsible officials to update future life-cycle cost estimates of the system acquisition program discussed in this report using cost-estimating practices that address the detailed weaknesses that we identified.

    Agency Affected: Pension Benefit Guaranty Corporation

  16. Status: Closed - Not Implemented

    Comments: The Department does not plan to implement this recommendation. The Department agreed with the provisions in the GAO Cost Estimating and Assessment Guide used as criteria and the methodological basis for assessing cost estimating practices of its Tactical Mission Command program. However, our recommendation to update future life-cycle cost estimates is specific to the Tactical Mission Command program and only its future life-cycle cost estimates. This program is in the system deployment phase of its acquisition lifecycle, with segments nearing lifecycle end, being migrated out of the Army Architecture; and the program has no significant life-cycle cost changes anticipated. Consequently, there is no plan to update the Tactical Mission Command life-cycle cost estimate.

    Recommendation: To address weaknesses identified in the Department of Defense's practices for cost estimating, we recommend that the Secretary of Defense should direct responsible officials to update future life-cycle cost estimates of the Tactical Mission Command program using cost-estimating practices that address the detailed weaknesses that we identified.

    Agency Affected: Department of Defense

 

Explore the full database of GAO's Open Recommendations »

Jun 29, 2016

Jun 22, 2016

Jun 10, 2016

Jun 9, 2016

Jun 2, 2016

May 25, 2016

May 18, 2016

May 17, 2016

May 5, 2016

Looking for more? Browse all our products here