This is the accessible text file for GAO report number GAO-15-37 entitled 'DOE and NNSA Project Management: Analysis of Alternatives Could Be Improved by Incorporating Best Practices' which was released on December 11, 2014. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. United States Government Accountability Office: GAO: Report to the Committee on Armed Services, U.S. Senate: December 2014: DOE and NNSA Project Management: Analysis of Alternatives Could Be Improved by Incorporating Best Practices: GAO-15-37: GAO Highlights: Highlights of GAO-15-37, a report to the Committee on Armed Services, U.S. Senate. Why GAO Did This Study: NNSA manages numerous programs that require the design and construction of one-of-a-kind facilities. NNSA's selection of preferred alternatives for these projects is governed by DOE requirements and guidance related to conducting an AOA. In recent years, NNSA has incurred substantial cost increases and schedule delays for such projects. GAO was mandated to review the AOA process applied by NNSA. This report (1) identifies and describes AOA best practices, (2) determines the extent to which DOE requirements and guidance for conducting an AOA conform to AOA best practices, and (3) determines the extent to which NNSA conformed to best practices and followed certain DOE requirements and guidance in conducting the AOAs for recent NNSA projects. To do this work, GAO examined relevant AOA guidance from the public and private sectors and DOE's AOA requirements and guidance, sought input from AOA experts, and interviewed agency officials. GAO reviewed three AOAs that NNSA completed or was scheduled to complete between November 2010 and September 2014. What GAO Found: GAO identified 24 best practices for analysis of alternatives (AOA)-—a process that is a key first step in capital asset acquisition. The process entails identifying, analyzing, and selecting a preferred alternative to best meet the mission need by comparing the operational effectiveness, costs, and risks of potential alternatives. Because no single set of best practices for AOAs was broadly recognized by government and private-sector entities, GAO developed a set of practices by reviewing AOA policies and guidance used by seven public and private-sector entities with experience in the AOA process, and verified these practices with subject matter experts. These best practices include, among other things, defining functional requirements based on mission need, conducting the AOA without a predetermined solution, including the status-quo alternative, and conducting an independent review of the entire AOA process. These practices can be applied to a wide range of activities, projects, and programs. The Department of Energy's (DOE) requirements and guidance-—found in the agency's orders and associated guides—-for conducting an AOA do not conform to the 24 best practices GAO identified. Therefore, DOE does not have assurance that applying these requirements and guidance may lead to reliable AOAs. GAO's review of DOE's requirements for AOAs found that they conform to only 1 of the 24 best practices: the practice of defining functional requirements based on mission need. GAO' s review of DOE's requirements combined with associated guidance-— which includes nonmandatory approaches for meeting requirements-—found that they conform to 9 of the 24 best practices. For example, DOE's guidance suggests identifying and considering at least three viable alternatives, including the status quo—-a best practice that is not included in the requirements. Federal standards for internal control related to risk assessment call for agency management to decide on actions to mitigate identified risks. Without developing a reliable AOA process, DOE may not be successful in mitigating the risk it has identified related to this process. DOE's requirements and guidance for AOAs also apply to the National Nuclear Security Administration (NNSA), a separately organized agency in DOE. For three recent NNSA projects that GAO reviewed, NNSA did not conform to most of the best practices GAO identified for conducting AOAs, therefore, raising concerns about the reliability of these AOAs. Specifically, for the three projects' AOAs, NNSA conformed to 6, 8, and 11 of the 24 identified best practices, respectively. For all three projects, NNSA consistently followed the one DOE requirement that conformed to a best practice—-to define functional requirements based on mission need. DOE and NNSA officials acknowledge that unreliable AOAs are a risk factor for major cost increases and schedule delays for NNSA projects. As GAO has previously reported, NNSA has spent billions of dollars designing and partially constructing projects with an estimated cost of $750 million or more, only to later reassess alternatives. Without a process to develop reliable AOAs, NNSA may continue on this path and continue to have limited assurance that it is selecting alternatives that best meet its mission needs and will not result in major cost increases and schedule delays in the future. What GAO Recommends: GAO recommends that DOE incorporate best practices into its AOA requirements. DOE agreed with this recommendation. View [hyperlink, http://www.gao.gov/products/GAO-15-37]. For more information, contact David C. Trimble at (202) 512-3841 or trimbled@gao.gov. [End of section] Contents: Letter: Background: GAO Identified 24 Best Practices for the Analysis of Alternatives Process: DOE's AOA Requirements and Guidance Do Not Conform to Best Practices: NNSA's AOAs for Three Recent Projects Did Not Conform to Best Practices and Did Not Consistently Follow Certain DOE Guidance: Conclusions: Recommendation for Executive Action: Agency Comments and Our Evaluation: Appendix I: Scope and Methodology: Appendix II: The Department of Energy's Analysis of Alternatives Process: Appendix III: Best Practices for the Analysis of Alternatives Process: Appendix IV: DOE's AOA Requirements and Guidance Compared with Best Practices: Appendix V: NNSA's Analysis of Alternatives Conducted at the High Explosive, Science, Technology and Engineering Project Compared with Best Practices: Appendix VI: NNSA's 2013 Analysis of Alternatives Conducted at the Radioactive Liquid Waste Treatment Facility Compared with Best Practices: Appendix VII: NNSA's 2012 Analysis of Alternatives Conducted at the Uranium Processing Facility Compared with Best Practices: Appendix VIII: Comments from the Department of Energy: Appendix IX: GAO Contact and Staff Acknowledgments: Tables: Table 1: Best Practices for the Analysis of Alternatives (AOA) Process Included in the General Principles Category: Table 2: Best Practices for the Analysis of Alternatives (AOA) Process Included in the Identifying Alternatives Category: Table 3: Best Practices for the Analysis of Alternatives (AOA) Process Included in the Analyzing Alternatives Category: Table 4: Best Practices for the Analysis of Alternatives (AOA) Process Included in the Selecting a Preferred Alternative Category: Table 5: Department of Energy (DOE) Reviews Required in Order 413.3B, Which Cover Mission Need and Cost Estimating Aspects of the Analysis of Alternatives (AOA): Table 6: Assessment of Department of Energy (DOE) Requirements Compared with Best Practices by Category of Best Practices and Overall: Table 7: Assessment of Department of Energy (DOE) Requirements Combined with Guidance Compared with Best Practices by Category of Best Practices and Overall: Table 8: Assessment of the Analysis of Alternatives for the High Explosive Science, Technology and Engineering Project, by Category of Best Practices and Overall: Table 9: Assessment of the Analysis of Alternatives for the Radioactive Liquid Waste Treatment Facility, by Category of Best Practices and Overall: Table 10: Assessment of the Analysis of Alternatives for the Uranium Processing Facility, by Category of Best Practices and Overall: Table 11: Extent to Which the National Nuclear Security Administration Followed Optional Department of Energy (DOE) Guidance That Conforms to Best Practices for Analysis of Alternatives: Table 12: Best Practices for the Analysis of Alternatives (AOA) Process: Table 13: The Department of Energy's (DOE) Requirements and Guidance Compared with Analysis of Alternatives (AOA) Best Practices: Table 14: Comparison of the AOA Conducted at the High Explosive, Science, Technology and Engineering Project with AOA Best Practices: Table 15: Comparison of the AOA Conducted at the Radioactive Liquid Waste Treatment Facility with AOA Best Practices: Table 16: Comparison of the AOA Conducted at the Uranium Processing Facility with AOA Best Practices: Figure: Figure 1: Conducting the Analysis of Alternatives as Part of Department of Energy's Project Management Process for Capital Asset Projects: Abbreviations: APM: Office of Acquisition and Project Management: AOA: analysis of alternatives: CD: critical decision: DOE: Department of Energy: FPD: Federal Project Director: IPT: Integrated Project Team: NEPA: National Environmental Policy Act: NNSA: National Nuclear Security Administration: UPF: Uranium Processing Facility: [End of section] United States Government Accountability Office: GAO: 441 G St. N.W. Washington, DC 20548: December 11, 2014: The Honorable Carl Levin: Chairman: The Honorable James M. Inhofe: Ranking Member: Committee on Armed Services: United States Senate: To accomplish its nuclear security and nonproliferation mission, the National Nuclear Security Administration (NNSA)--a separately organized agency within the Department of Energy (DOE)--manages numerous programs that require NNSA to design and construct one-of-a- kind facilities through a process known as capital asset acquisition.[Footnote 1] When NNSA is acquiring such assets, the use of the analysis of alternatives (AOA) process included in DOE's requirements and guidance is a key first step to help ensure that the selected alternative best meets the agency's mission need. In 2009, we defined the AOA process as an analytical study that is intended to compare the operational effectiveness, cost, and risks of a number of potential alternatives to address valid needs and shortfalls in operational capability.[Footnote 2] This process helps ensure that the best alternative that satisfies the mission need is chosen on the basis of the selection criteria, such as safety, cost, or schedule. In recent years, as we previously reported, NNSA has used its requirements and guidance to select preferred alternatives for many projects, and then spent billions of dollars designing and partially constructing several major capital asset projects (facilities with an estimated cost of $750 million or more), only to later reassess alternatives for each project. These major projects are the Chemistry and Metallurgy Research Replacement Nuclear Facility[Footnote 3] at NNSA's Los Alamos National Laboratory in New Mexico, the Mixed Oxide Fuel Fabrication Facility[Footnote 4] and the Pit Disassembly and Conversion Facility[Footnote 5] at NNSA's Savannah River Site in South Carolina, and the Uranium Processing Facility[Footnote 6] at NNSA's Y- 12 site in Tennessee, all projects for which NNSA has recently reassessed or is now in the process of reassessing alternatives. Cost increases and schedule delays for these projects led to the reassessment of alternatives and increased scrutiny of NNSA's project management process, including requirements and guidance for the AOA. Senate Report 113-44 accompanying the National Defense Authorization Act for Fiscal Year 2014 mandated that GAO review the AOA process applied by NNSA for capital asset acquisition.[Footnote 7] In response to this mandate, this report (1) identifies and describes AOA best practices, (2) determines the extent to which DOE's requirements and guidance for conducting an AOA conform to AOA best practices, and (3) determines the extent to which NNSA conformed to best practices and followed certain DOE requirements and guidance in conducting the AOAs for recent NNSA projects. To identify and describe AOA best practices, we first identified AOA handbooks, guidebooks, requirements, and other AOA-related information from the federal government and private-sector entities and also sent a request to subject-matter experts including experts from DOE, NNSA and various other government or private-sector entities to help us identify further relevant information.[Footnote 8] We then reviewed information related to the AOA process included in our Cost Estimating Guide[Footnote 9] and from seven additional entities[Footnote 10] and compiled a draft set of practices commonly mentioned across these different entities' AOA policies and guidance. We sent this draft set of AOA best practices to the experts for review and received comments by e-mail. We also received comments on this set of best practices during a GAO semiannual meeting, in which some of these experts participated, including officials from DOE and NNSA. We finalized the set of 24 best practices for the AOA process by incorporating the experts' comments. We developed this set of best practices because no one single set of practices existed that was broadly recognized by government and private-sector entities. We grouped these 24 best practices into four categories: (1) general principles, (2) identifying alternatives, (3) analyzing alternatives, and (4) selecting a preferred alternative. To determine the extent to which DOE's requirements and guidance for conducting an AOA conform to AOA best practices, we examined DOE's project management order, its associated guides and other documentation, and interviewed DOE and NNSA officials to identify the process NNSA is required to follow when conducting an AOA. We then confirmed these requirements with DOE and NNSA officials. We performed two sets of analysis: we compared the best practices to (1) DOE's requirements and (2) DOE's requirements combined with the guidance. We used a five-point scoring system to determine the extent to which DOE's AOA process conforms to best practices.[Footnote 11] We first used this scoring system to determine how well DOE's requirements and guidance conform to each best practice. We then used the average of the scores for the best practices in each of the four categories to determine an overall score for each category, and we then used the average of the scores for the four categories as the final score for the overall DOE AOA process. If the score for each best practice, the average score for each category, or the final score for the AOA process was "fully met" or "substantially met," we concluded that the AOA process conformed to best practices and therefore could be considered reliable.[Footnote 12] In contrast, if the score was "partially met," "minimally met," or "not met," we concluded that the AOA process did not conform to best practices and therefore could not be considered reliable. To determine the extent to which NNSA conformed to best practices and followed certain DOE requirements and guidance in conducting the AOAs for recent NNSA projects, we chose projects for which NNSA completed the AOA process since November 2010, when DOE's most recent version of the project management order came into effect, or for which NNSA was scheduled to complete the AOA process by the end of fiscal year 2014. We identified these projects by examining DOE's data from its February 2014 Monthly Project Portfolio Status Report. The three projects that met our criteria were (1) the High Explosive Science, Technology and Engineering Project at NNSA's Pantex site in Texas, (2) the Radioactive Liquid Waste Treatment Facility at NNSA's Los Alamos National Laboratory in New Mexico, and (3) the Uranium Processing Facility at NNSA's Y-12 site in Tennessee.[Footnote 13] We checked with knowledgeable DOE and NNSA officials, reviewed recent GAO reports, and compared the data in the project status report with information received from NNSA on the projects to assess the reliability of the data in this status report, and we determined that the data were sufficiently reliable to ensure these projects met our criteria. For each of these projects, we reviewed project documentation, and we interviewed NNSA and contractor officials in charge of the AOA process to compare the process followed by NNSA with best practices. In April 2014, as we were conducting our engagement, NNSA released a peer review[Footnote 14] report that recommended a new alternative for the Uranium Processing Facility,[Footnote 15] and we examined whether this peer review reflected any characteristics of an AOA. In addition, we reviewed project documentation and interviewed NNSA and contractor officials in charge of the AOA process to determine the extent to which NNSA conformed to certain DOE requirements and guidance--those DOE requirements and guidance that conformed to best practices--in conducting the AOAs for these projects. Appendix I presents a more detailed description of our scope and methodology. We conducted this performance audit from February 2014 to December 2014 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Background: This section describes DOE's requirements and guidance for AOA; DOE and NNSA offices responsible for conducting AOAs for capital asset projects; and NNSA's recent projects that have completed, or are nearing completion of, an AOA. DOE's Requirements and Guidance for Analysis of Alternatives: DOE's mandatory requirements and optional guidance for identifying, analyzing, and selecting alternatives when conducting AOAs also apply to NNSA. DOE's requirements for the AOA process are outlined in Order 413.3B governing the Program and Project Management for the Acquisition of Capital Assets (hereafter referred to as Order 413.3B).[Footnote 16] This order includes requirements for the acquisition of capital assets, with the stated goal of delivering fully capable projects within the planned cost, schedule, and performance baseline. In addition to the order's requirements, DOE has guidance for identifying, analyzing, and selecting alternatives (hereafter referred to as "guidance") that is found throughout seven guides associated with the order.[Footnote 17] DOE states at the beginning of each of these guides that they include nonmandatory approaches for meeting requirements, that guides are not requirements documents, and that they are not to be construed as requirements in any audit for appraisal of compliance with the parent policy or order. DOE's Order 413.3B establishes five critical decision processes of project development that each end with a major approval milestone--or "critical decision point"; these decision processes cover the life of a project. Under Order 413.3B, an AOA occurs during the span of the first two critical decision processes--the preconceptual design process (CD-0) and the conceptual design process (CD-1)--with the majority of the AOA being conducted during the conceptual design process and ending with CD-1 approval. Figure 1 illustrates when DOE conducts the analysis of alternatives as part of its project management process. Figure 1: Conducting the Analysis of Alternatives as Part of Department of Energy's Project Management Process for Capital Asset Projects: [Refer to PDF for image: illustration] Department of Energy's (DOE) critical decision (CD) process: Conducting the analysis of alternatives (AOA): CD-0: Identify need: Preconceptual design process. CD-1: Select alternative: Conceptual design process. CD-2: Establish the baseline: Preliminary design process. Reassessment of alternatives[A]. CD-3: Begin work: Final design process. CD-4: Complete the project: Construction. [A] If the top end of the approved CD-1 cost range grows by more than 50 percent, the program office must conduct another AOA and obtain a new CD-1 approval. Source: GAO analysis of DOE's Order 413.3B. GAO-15-37. [End of figure] DOE's Order 413.3B and its associated guides include the AOA-related requirements and guidance listed below, among others. Appendix II includes a full description of DOE's requirements and guidance for conducting an AOA. * DOE requirements: - As part of the preconceptual design (CD-0) approval process, the mission need--which DOE defines in Order 413.3B as a credible gap between current capabilities and those required to meet the goals articulated in the strategic plan--and functional requirements--the general parameters that the selected alternative must have to address the mission need--must be identified. The mission need must be independent of a particular alternative, and the program office responsible for the capital asset project must explore a variety of alternatives. - As part of the conceptual design (CD-1) approval process, reliable cost and schedule range estimates for the alternatives considered must be developed, and whatever figure or range is provided at the CD-0 and CD-1 stages must explicitly note relevant caveats concerning risks and uncertainties inherent in early estimates. In addition, a conceptual design report must be developed that includes, among other things, a clear and concise description of the alternatives analyzed, the basis for the selected alternative, how the selected alternative meets the mission need, the functional requirements that define the alternative and demonstrate that the alternative can be successful, and life-cycle cost assumptions.[Footnote 18] - During the CD-0 and CD-1 approval process, several independent reviews must be conducted, depending on the estimated cost of the project, related to two aspects of the AOA process: validation of (1) the mission need statement and (2) the cost estimates. - As a project moves toward approval of CD-2, if the top end of the approved CD-1 cost range for the selected alternative grows by more than 50 percent, the program office must reassess alternatives by conducting another AOA and obtaining another CD-1 approval. * DOE guidance: - DOE's guides suggest summarizing a planned approach to conduct an analysis of alternatives; considering at least three viable alternatives for analysis, including one that represents the status quo; developing cost estimates that are explicit ranges instead of point estimates; including life-cycle cost estimates of the alternatives being considered; quantifying the benefits of alternatives over their life cycle; adjusting life-cycle cost and benefit estimates for risk; considering various selection criteria for the alternatives; weighting the selection criteria for relative importance; and comparing alternatives using net present value.[Footnote 19] The guidance suggests that, at the end of the analysis, the AOA team present the recommended alternative based on the preceding analysis in an integrated form, summarizing why an alternative is preferred and supporting the recommendation of the preferred alternative with facts from the analysis. * In addition, DOE's guidance suggests that the program office review the alternative selection and cost range to assess whether the AOA process (1) evaluates a range of appropriate attributes for each alternative, including cost, risks, safety, technology and regulatory requirements, and (2) is reasonable and provides best value to the government. Order 413.3B also requires prior to CD-1 approval that DOE conduct a separate analysis of alternatives under the National Environmental Policy Act of 1969 (NEPA).[Footnote 20] Under NEPA, federal agencies must identify and assess the likely environmental effects of proposed projects, and reasonable alternatives, using an environmental assessment or, if the projects likely would significantly affect the quality of the human environment, a more detailed environmental impact statement. Recognizing the need for better integration between the AOA conducted under Order 413.3B and the AOA conducted under NEPA, in early 2012, DOE established a multidisciplinary team of NEPA- compliance and program-and project-management specialists, and in June 2012, the Secretary of Energy issued a memo highlighting the importance of integrating these two AOAs. DOE officials who were part of this team stated that DOE has not yet decided how to connect the NEPA process with the AOA process under Order 413.3B. DOE and NNSA Offices Responsible for Conducting AOAs for Capital Asset Projects: Within DOE and NNSA, several groups are responsible for various aspects of conducting an AOA: * DOE Office of Acquisition and Project Management (DOE APM). This office is responsible for, according to DOE APM officials, writing DOE's project management Order 413.3B and its associated guides, including requirements and guidance for conducting an AOA, and for reviewing some DOE projects. * NNSA Office of Acquisition and Project Management (NNSA APM). This office is responsible for managing construction of capital asset projects within approved cost and schedule estimates, and for conducting at least yearly reviews of the construction projects to evaluate technical, cost, scope, and other aspects of the projects. According to NNSA APM's Concept of Operations, this office leads the development of the analysis of alternatives for NNSA projects. * NNSA program offices. Program offices are responsible for the projects that fall within their portfolios. For projects, according to NNSA APM's Concept of Operations, the program offices lead the development of the mission needs and functional requirements, support the analysis of alternatives, and lead the selection of the preferred alternative. * NNSA Office of Cost Estimating and Program Evaluation. The National Defense Authorization Act for Fiscal Year 2014 mandated the permanent establishment of this office to provide an independent review of cost estimates and to advise the NNSA Administrator on AOA policies and procedures, among other responsibilities.[Footnote 21] NNSA officials stated that they expected this office to work with other NNSA offices to define its roles and responsibilities as they relate to conducting AOAs during fiscal year 2015. NNSA's Recent Projects That Have Completed, or Are Nearing Completion of, an AOA: The following describes the three recent NNSA projects that we reviewed that had completed, or are nearing completion of an AOA. High Explosive Science, Technology and Engineering Project: This project is intended to replace and upgrade existing facilities used for NNSA's high explosive manufacturing support, surveillance, testing, and technology development program. NNSA began conducting the AOA in November 2011 and expects to complete the AOA process by obtaining CD-1 approval in December 2014.[Footnote 22] As of September 2014, NNSA estimated that the total project cost[Footnote 23] would range from $100 million to $155 million at an 85 percent confidence level. Radioactive Liquid Waste Treatment Facility: This project is intended to replace an existing facility at the Los Alamos National Laboratory to treat and dispose of two types of radioactive waste. According to this project's mission need statement document, the existing facility was nearing its designed life span and had suffered numerous failures. NNSA completed an initial AOA in 2006, selecting the alternative to place the treatment capabilities for both types of waste in the same structure. NNSA estimated in 2006 that the total project cost would likely range between $82 million and $104 million. After significant cost increases were incurred during the preliminary design process of the project, NNSA completed a reassessment of alternatives in September 2013, which we examine in this report. NNSA indicated in this reassessment that the existing facility was beyond its designed life span. The selected alternative for this reassessment separates treatment capabilities into two separate structures, and NNSA estimated that the revised total project cost would likely range between $168 million and $220 million. Uranium Processing Facility: This project is intended to replace existing facilities for enriched uranium capabilities that, according to this project's mission need statement document, require intensive maintenance and are experiencing escalating operating costs. NNSA completed an initial AOA in 2007, selecting the alternative to construct a new building with full capability to process enriched uranium.[Footnote 24] NNSA estimated in 2007 that the total project cost would likely range between $1.4 billion and $3.5 billion. During the preliminary design phase, the estimated cost of the project increased significantly, and NNSA completed a reassessment of alternatives in June 2012. This second AOA, which we examine in this report, selected a new alternative to construct the same building, but included fewer capabilities, while deferring significant portions of the project's original scope to be added in the facility at a later time. The approved estimated cost increased to a range of $4.2 billion to $6.5 billion. In December 2013, an independent review by the Department of Defense's Office of Cost Assessment and Program Evaluation estimated that this selected alternative would cost $9.8 billion, which was over 50 percent more than the top end of the approved cost range.[Footnote 25] According to NNSA's Associate Administrator for Acquisition and Project Management, in December 2013, NNSA project management officials proposed to NNSA and DOE senior management a new alternative of building two new, smaller facilities to include all capabilities approved in the 2012 AOA, and senior management approved this new alternative. NNSA began to consider alternatives other than building a single facility because, according to the UPF Federal Project Director, the project was facing budget constraints, rising costs, and competition from other high-priority projects within NNSA.[Footnote 26] In January 2014, the NNSA Administrator asked for a peer review to develop and recommend an alternative approach to UPF. In April 2014, the peer reviewers recommended an alternative similar to the one approved by DOE and NNSA senior management in December 2013. This alternative would require NNSA to construct two new, smaller facilities and upgrade existing facilities to cover the same capabilities approved in the 2012 AOA, all within the $6.5 billion approved top end of the cost range.[Footnote 27] According to NNSA officials, this peer review was not an AOA.[Footnote 28] According to the NNSA Deputy Federal Project Director for this project, the Federal Project Director provided direction in April 2014 to proceed with the alternative proposed by the peer review. According to NNSA officials in October 2014, the alternative was further modified. At this time, NNSA's plan was to build three new, smaller facilities and upgrade existing facilities to provide the same capabilities approved in 2012 and still remain within the approved cost range of $4.2 billion to $6.5 billion. GAO Identified 24 Best Practices for the Analysis of Alternatives Process: Because no one single set of practices existed that was broadly recognized by government and private-sector entities for the AOA process, we identified 24 best practices for identifying, analyzing, and selecting alternatives. We identified these best practices by (1) compiling and reviewing commonly mentioned AOA policies and guidance used by different government and private-sector entities and (2) incorporating experts' comments on our draft set of practices to develop a final set of practices. These practices can be applied to a wide range of activities in which an alternative must be selected from a set of possible options, as well as to a broad range of capability areas, projects, and programs. These practices can provide a framework to help ensure that entities consistently and reliably select the project alternatives that best meet mission needs. We grouped these 24 best practices into four categories: (1) general principles, (2) identifying alternatives, (3) analyzing alternatives, and (4) selecting a preferred alternative. The four categories of best practices address the AOA process from defining the mission need and functional requirements to independently reviewing its results. On the basis of our reviews and experts' comments, we believe that these best practices can be generally applied from the beginning of the AOA process with practices from the general principles category, through practices in the identifying and analyzing alternatives categories, and ending with practices in the selecting a preferred alternative category. We also believe that these best practices do not necessarily have to be followed in order and that some of them can be applied concurrently with other best practices. For example, the best practice of defining the selection criteria based on the mission need in the selecting a preferred alternative category could be addressed at the same time as the best practice of creating a study plan in the general principles category. The following sections describe the 24 best practices by category. General Principles Category: The general principles category contains best practices that would need to be applied before starting the process of identifying, analyzing, and selecting alternatives, such as determining the mission need and functional requirements, developing the study time frame and creating a study plan, and determining who conducts the analysis. It also includes best practices that would need to be applied throughout the AOA process, such as documenting all steps taken to identify, analyze, and select alternatives in a single document. Table 1 lists the best practices in the general principles category. Table 1: Best Practices for the Analysis of Alternatives (AOA) Process Included in the General Principles Category: 1. The customer defines the mission need and functional requirements without a predetermined solution. 2. The customer defines functional requirements based on the mission need. 3. The customer provides the team conducting the analysis of alternatives (AOA) with enough time to complete the AOA process to ensure a robust and complete analysis. 4. The team includes members with diverse areas of expertise including, at a minimum, subject matter expertise, project management, cost estimating, and risk management. 5. The team creates a plan, including proposed methodologies, for identifying, analyzing, and selecting alternatives, before beginning the AOA process. 6. The team documents all steps taken to identify, analyze, and select alternatives in a single document. 7. The team documents and justifies all assumptions and constraints used in the analysis. 8. The team conducts the analysis without a predetermined solution. Source: GAO. GAO-15-37. [End of table] Identifying Alternatives Category: The identifying alternatives category contains best practices that help ensure the alternatives to be analyzed are sufficient, diverse, and viable. Table 2 lists the best practices in the identifying alternatives category. Table 2: Best Practices for the Analysis of Alternatives (AOA) Process Included in the Identifying Alternatives Category: 9. The team identifies and considers a diverse range of alternatives to meet the mission need. 10. The team describes alternatives in sufficient detail to allow for robust analysis. 11. The team includes one alternative representing the status quo to provide a basis of comparison among alternatives. 12. The team screens the list of alternatives before proceeding, eliminates those that are not viable, and documents the reasons for eliminating any alternatives. Source: GAO. GAO-15-37. [End of table] Analyzing Alternatives Category: The analyzing alternatives category contains best practices that are necessary to compare the alternatives selected for analysis. The best practices in this category help ensure that the team conducting the analysis uses a standard, quantitative process to assess the alternatives. This category includes best practices related to estimating the costs and benefits of each alternative over its life cycle and understanding the impacts of risks and key assumptions on these estimates. Table 3 lists the best practices in the analyzing alternatives category. Table 3: Best Practices for the Analysis of Alternatives (AOA) Process Included in the Analyzing Alternatives Category: 13. The team develops a life-cycle cost estimate for each alternative, including all costs from inception of the project through design, development, deployment, operation, maintenance and retirement[A]. 14. The team presents the life-cycle cost estimate for each alternative as a range or with a confidence interval, and not solely as a point estimate. 15. The team expresses the life-cycle cost estimate in present value[B] terms and explains why it chose the specific discount rate used[C]. 16. The team uses a standard process to quantify the benefits/ effectiveness of each alternative and documents this process. 17. The team quantifies the benefits/effectiveness resulting from each alternative over that alternative's full life cycle, if possible. 18. The team explains how each measure of benefit/effectiveness supports the mission need. 19. The team identifies and documents the significant risks and mitigation strategies for each alternative. 20. The team tests and documents the sensitivity of both the cost and benefit/effectiveness estimates for each alternative to risks and changes in key assumptions. Source: GAO. GAO-15-37. [A] GAO-09-3SP contains a separate set of best practices to assess the reliability of life-cycle cost estimates included in the AOA. [B] The present value of an estimate reflects the time value of money, the concept that a dollar in the future is worth less than a dollar today because the dollar today can be invested and earn interest. [C] The discount rate is the interest rate used to calculate the present value of an estimate. [End of table] Selecting a Preferred Alternative Category: The category for selecting a preferred alternative contains best practices that help ensure the team selects a preferred alternative that best meets the mission needs. This category includes best practices related to defining selection criteria, differentiating among the selection criteria, and independently reviewing the AOA process. Table 4 lists the best practices in this category. Table 4: Best Practices for the Analysis of Alternatives (AOA) Process Included in the Selecting a Preferred Alternative Category: 21. The team or the decision maker defines selection criteria based on the mission need. 22. The team or the decision maker weights the selection criteria to reflect the relative importance of each criterion. 23. The team or the decision maker compares alternatives using net present value,[A] if possible. 24. An entity independent of the AOA process reviews the extent to which all best practices have been followed (for certain projects, additional independent reviews may be necessary at earlier stages of the process such as for reviewing the study plan or for reviewing the identification of viable alternatives). [End of table] [A] The net present value is the difference between the discounted present value of benefits and the discounted present value of costs. Source: GAO. GAO-15-37. On the basis of our reviews and experts' comments, we believe that conforming to these best practices helps ensure that the preferred alternative that is selected is the one that best meets the agency's mission needs. Not conforming to best practices may lead to an unreliable AOA, and the agency will not have assurance that the preferred alternative best meets its mission needs. Appendix III lists the best practices from all four categories. DOE's AOA Requirements and Guidance Do Not Conform to Best Practices: Neither DOE's AOA requirements nor its guidance conform to best practices and, therefore, DOE does not have assurance that applying these requirements and guidance may lead to reliable AOAs.[Footnote 29] Our review of DOE's requirements contained within Order 413.3B found that they minimally meet best practices overall, and that they do not fully or substantially meet best practices in any of the four categories. Our analysis also found that, even when DOE's requirements are combined with guidance contained in the guides associated with Order 413.3B, they only partially meet best practices overall, and they only fully or substantially meet best practices in the identifying alternatives and selecting a preferred alternative categories but not in the general principles or analyzing alternatives categories. DOE's Order 413.3B Requirements Do Not Conform to Best Practices: DOE's Order 413.3B requirements for the AOA process minimally meet best practices. Therefore, DOE does not have assurance that applying these requirements may lead to reliable AOAs. DOE's requirements do not fully or substantially meet best practices in any of the four AOA categories, and they conform to only 1 of the 24 best practices we identified--the practice of having the customer define functional requirements based on the mission need. Our assessment of DOE's Order 413.3B requirements found the following: * General principles. DOE's Order 413.3B requirements minimally meet best practices for the general principles category by substantially meeting 1 of the 8 best practices in this category, partially or minimally meeting 4, and not meeting the other 3. DOE requires that the program office define a project's mission need without a predetermined solution and that the Integrated Project Team must represent a diverse range of disciplines, develop functional requirements to satisfy the mission need, and document some of the steps and assumptions required as part of conducting the AOA. DOE's Order 413.3B does not make any reference to appropriate AOA time frames, an AOA study plan, or conducting the analysis without a predetermined solution. * Identifying alternatives. DOE's Order 413.3B requirements minimally meet best practices for identifying alternatives by partially or minimally meeting 2 of the 4 best practices in this category and not meeting the other 2. DOE requires that the project's customer, not the AOA team, explore a variety of alternatives and that the alternatives be defined clearly and concisely. DOE's Order 413.3B does not make any reference to including a status quo alternative in the AOA or pre- screening the list of identified alternatives for viability. * Analyzing alternatives. DOE's Order 413.3B requirements minimally meet best practices for analyzing alternatives by partially or minimally meeting 3 of the 8 best practices in this category and not meeting the other 5. DOE requires that life-cycle cost assumptions be included in the conceptual design report, which leaves room for interpretation because it could be interpreted as requiring the inclusion of life-cycle cost estimates for all alternatives or, as DOE and NNSA officials said that they interpret this statement, as requiring a life-cycle cost estimate for only the selected alternative after completion of the AOA process. DOE also requires that project risks and mitigation strategies are assessed. DOE's Order 413.3B does not make any reference to presenting life-cycle cost estimates in present value terms, quantifying benefits, explaining how measures of benefit support the mission need, or testing the sensitivity of cost and benefit estimates to risks and changes in key assumptions. * Selecting a preferred alternative. DOE's Order 413.3B requirements do not meet best practices for selecting a preferred alternative by minimally meeting 1 of the 4 best practices in this category and not meeting the other 3. The best practice that DOE minimally meets is having an entity independent of the AOA process reviewing the extent to which all best practices have been followed. Order 413.3B contains four separate reviews that might be required during CD-0 and CD-1 processes depending on the amount of the project's cost estimate, but these reviews are limited to two aspects of the AOA process: the validation of (1) the mission need statement and (2) the cost estimates. In addition, one of these reviews is not an independent review because it is conducted by the program office, which is on the project's chain of command. Table 5 lists these required reviews. DOE's requirements do not meet the other best practices in this category because DOE's Order 413.3B does not make any reference to defining selection criteria based on the mission need; weighting selection criteria to reflect the relative importance of each; or comparing the alternatives considered, using net present value. Table 5: Department of Energy (DOE) Reviews Required in Order 413.3B, Which Cover Mission Need and Cost Estimating Aspects of the Analysis of Alternatives (AOA): Name of review[A]: Mission need statement document review; Review required for projects with: Total project cost greater than or equal to $100 million; Office conducting the review: DOE's Office of Acquisition and Project Management; Timing of review: Prior to CD-0; Aspects of the AOA covered by the review: Reviews the mission need statement document and provides a recommendation to the decision maker. Name of review[A]: Mission validation independent review; Review required for projects with: Total project cost greater than or equal to $750 million; Office conducting the review: Program office; Timing of review: Prior to CD-0; Aspects of the AOA covered by the review: Validates the mission need and the rough order of magnitude cost range. Name of review[A]: Independent cost review; Review required for projects with: Total project cost greater than or equal to $750 million; Office conducting the review: DOE's Office of Acquisition and Project Management; Timing of review: Prior to CD-0; Aspects of the AOA covered by the review: Validates the basis of the rough order of magnitude cost range and assesses whether the range reasonably bounds the alternatives to be analyzed prior to CD-1. Name of review[A]: Independent cost estimate/Independent cost review; Review required for projects with: Total project cost greater than or equal to $100 million; Office conducting the review: DOE's Office of Acquisition and Project Management; Timing of review: Prior to CD-1; Aspects of the AOA covered by the review: Validates the basis of the preliminary cost range of the selected alternative for reasonableness and executability. Also includes a full accounting of life-cycle costs to support the alternative selection process. Source: GAO analysis of DOE's Order 413.3B. GAO-15-37. Note: The selecting a preferred alternative category contains a best practice that suggests having an independent entity reviewing the extent to which all best practices have been followed. This table shows that DOE's requirements minimally meet this best practice because these reviews are limited to two aspects of the AOA process: the validation of (1) the mission need statement and (2) the cost estimates. [A] In addition to the reviews listed in this table, DOE Order 413.3B also requires an Acquisition Strategy Review to be conducted by DOE's Office of Acquisition and Project Management for projects with a total project cost equal or greater than $750 million and a Conceptual Design Review to be conducted by a party independent of the project, but the order does not define what these reviews entail and, according to DOE officials, these reviews are not related to the AOA process. [End of table] Table 6 and appendix IV summarize our assessment of DOE's requirements. Table 6: Assessment of Department of Energy (DOE) Requirements Compared with Best Practices by Category of Best Practices and Overall: Best practice categories: General principles; Category score: Minimally meet; Number of practices: 8; Fully or substantially meet: 1; Partially or minimally meet: 4; Does not meet: 3. Best practice categories: Identifying alternatives; Category score: Minimally meet; Number of practices: 4; Fully or substantially meet: 0; Partially or minimally meet: 2; Does not meet: 2. Best practice categories: Analyzing alternatives; Category score: Minimally meet; Number of practices: 8; Fully or substantially meet: 0; Partially or minimally meet: 3; Does not meet: 5. Best practice categories: Selecting a preferred alternative; Category score: Does not meet; Number of practices: 4; Fully or substantially meet: 0; Partially or minimally meet: 1; Does not meet: 3. Best practice categories: Overall best practices; Category score: Minimally meet; Number of practices: 24; Fully or substantially meet: 1; Partially or minimally meet: 10; Does not meet: 13. Source: GAO analysis of DOE information. GAO-15-37. Note: The five-point scoring system that we used was as follows: Fully meet means that DOE's documentation demonstrated that DOE completely meets the best practice. Substantially meet means that the DOE's documentation demonstrated that DOE meets a large portion of the best practice. Partially meet means that DOE's documentation demonstrated that DOE meets about half of the best practice. Minimally meet means that the DOE's documentation demonstrated that DOE meets a small portion of the best practice. Does not meet means that DOE's documentation did not demonstrate that DOE meets the best practice. [End of table] DOE's Requirements Combined with Guidance Do Not Conform to Best Practices: DOE's Order 413.3B AOA requirements combined with guidance partially meet best practices. Therefore, DOE does not have assurance that applying its requirements combined with its guidance may lead to reliable AOAs. DOE's requirements combined with guidance fully or substantially met best practices in the identifying alternatives and selecting a preferred alternative categories, but not in the general principles or analyzing alternatives categories. DOE's requirements combined with guidance conform to 9 of 24 best practices. Our assessment of DOE's requirements combined with guidance found the following: * General principles. DOE's requirements combined with guidance partially meet best practices for the general principles category by fully or substantially meeting 3 of the 8 best practices in this category, partially or minimally meeting 3, and not meeting the other 2. DOE's guidance supplements the requirements discussed in the previous section by suggesting that the Federal Project Director and the Integrated Project Team conduct different parts of the AOA, but the guidance does not suggest specific areas of expertise to be represented on the team. The guidance also suggests that a study plan be developed but does not suggest what to include in that plan. Neither DOE's requirements nor guidance refer to appropriate AOA time frames or conducting the analysis without a predetermined solution. * Identifying alternatives. DOE's requirements combined with guidance substantially meet best practices for the identifying alternatives category by fully meeting 1 of the 4 best practices in this category and partially or minimally meeting the other 3. DOE's guidance supplements the requirements by suggesting that at least three alternatives be considered, including one representing the status quo, and by listing the advantages and disadvantages of each alternative. Additionally, DOE's guidance suggests pre-screening the alternatives for viability, but the guidance does not suggest documenting the reasons for eliminating any alternatives. * Analyzing alternatives. DOE's requirements combined with guidance partially meet best practices for the analyzing alternatives category by fully or substantially meeting 3 of the 8 best practices in this category, partially or minimally meeting 3, and not meeting the other 2. DOE's guidance supplements the requirements by suggesting that life- cycle cost estimates be developed for each of the alternatives, the life-cycle cost estimates be presented as ranges instead of point estimates, life-cycle benefit estimate be developed for each alternative, risk be taken into consideration when evaluating alternatives, and life-cycle cost estimates be adjusted for risk. Neither DOE's requirements nor guidance refer to presenting life-cycle cost estimates in present value terms, explaining how each measure of benefit supports the mission need, or testing the sensitivity of the cost and benefit estimates to changes in key assumptions. * Selecting a preferred alternative. DOE's requirements combined with guidance substantially meet best practices for the selecting a preferred alternative category by fully or substantially meeting 2 of the 4 best practices in this category and partially or minimally meeting the other 2. DOE's guidance supplements the requirements by suggesting that a variety of discriminators are considered when selecting a preferred alternative, the selection criteria are weighted to reflect the relative importance of each, and the alternatives are compared using net present value. The guidance also suggests another review of more aspects of the AOA process, such as whether the alternative selection process evaluates a range of appropriate attributes for each alternative including cost, maintainability, safety, technology requirements, risks, and regulatory requirements, but the guidance suggests that this review be performed by the program office and not by a party independent of the project. Table 7 below and appendix IV summarize our assessment of DOE's requirements combined with guidance. Table 7: Assessment of Department of Energy (DOE) Requirements Combined with Guidance Compared with Best Practices by Category of Best Practices and Overall: Best practice categories: General principles; Category score: Partially meet; Number of practices: 8; Fully or substantially meet: 3; Partially or minimally meet: 3; Does not meet: 2. Best practice categories: Identifying alternatives; Category score: Substantially meet; Number of practices: 4; Fully or substantially meet: 1; Partially or minimally meet: 3; Does not meet: 0. Best practice categories: Analyzing alternatives; Category score: Partially meet; Number of practices: 8; Fully or substantially meet: 3; Partially or minimally meet: 3; Does not meet: 2. Best practice categories: Selecting a preferred alternative; Category score: Substantially meet; Number of practices: 4; Fully or substantially meet: 2; Partially or minimally meet: 2; Does not meet: 0. Best practice categories: Overall best practices; Category score: Partially meet; Number of practices: 24; Fully or substantially meet: 9; Partially or minimally meet: 11; Does not meet: 4. Source: GAO analysis of DOE information. GAO-15-37. Note: The five-point scoring system that we used was as follows: Fully meet means that DOE's documentation demonstrated that DOE completely meets the best practice. Substantially meet means that the DOE's documentation demonstrated that DOE meets a large portion of the best practice. Partially meet means that DOE's documentation demonstrated that DOE meets about half of the best practice. Minimally meet means that the DOE's documentation demonstrated that DOE meets a small portion of the best practice. Does not meet means that DOE's documentation did not demonstrate that DOE meets the best practice. [End of table] DOE and NNSA officials told us that unreliable AOAs are a risk factor for major cost increases and schedule delays for NNSA projects. DOE APM and NNSA APM officials acknowledged that DOE does not have definitive guidance on conducting AOAs. According to these officials, the absence of adequate guidance has led to some projects pursuing alternatives that subsequently proved to be unaffordable. For example, each of NNSA's ongoing major projects (e.g., the Chemistry and Metallurgy Research Replacement Nuclear Facility, the Mixed Oxide Fuel Fabrication Facility, or the Uranium Processing Facility) has undergone reassessments of alternatives due to cost increases and schedule delays. As we previously reported, federal standards for internal control related to risk assessment call for agency management to assess the risks faced entity-wide and at the activity level, and that once risks have been identified, management should decide what actions should be taken to mitigate them.[Footnote 30] Without developing a reliable AOA process, DOE may not be successful in mitigating the risk it has identified related to this process. NNSA's AOAs for Three Recent Projects Did Not Conform to Best Practices and Did Not Consistently Follow Certain DOE Guidance: For three recent projects, our analysis found that NNSA did not conform to most of the best practices in conducting the AOAs,[Footnote 31] therefore, raising concerns about the reliability of these AOAs. NNSA also did not consistently follow certain DOE optional guidance for these AOAs. NNSA's AOAs for the Three Recent Projects Did Not Conform to Best Practices: NNSA did not conform to best practices overall in conducting the AOAs for the three recent projects we examined--the High Explosive Science, Technology and Engineering Project, the Radioactive Liquid Waste Treatment Facility, and the Uranium Processing Facility, therefore, raising concerns about the reliability of these AOAs. NNSA also conducted each AOA differently. (See apps. V through VII for more detailed summaries of our assessment of the AOAs for each project.) High Explosive Science, Technology and Engineering Project AOA: For the High Explosive, Science, Technology and Engineering Project, NNSA partially met best practices overall in conducting the AOA. Therefore, this AOA may not be reliable. NNSA fully or substantially met best practices in the general principles and identifying alternatives categories but not in the analyzing alternatives or selecting a preferred alternative categories. NNSA conformed to 11 of 24 best practices when conducting this AOA. Our assessment of NNSA's AOA for this project identified the following: * General principles. NNSA substantially met best practices for the general principles category by fully or substantially meeting 4 of the 8 best practices in this category and by partially or minimally meeting the other 4. NNSA convened a team, which included federal and contractor project officials to conduct the analysis. Before conducting the analysis, NNSA developed a 17-step plan for conducting the AOA. When conducting the analysis, the team described in project documentation many, but not all, of the steps it took, and listed most of the assumptions made in the analysis but did not justify them. NNSA also undertook the analysis having an "originally envisioned" alternative. NNSA ultimately did not select this alternative as the preferred alternative, but NNSA did not explain how the analysis was conducted without having a predetermined solution. * Identifying alternatives. NNSA fully met best practices for the identifying alternatives category by fully or substantially meeting all 4 best practices in the category. NNSA identified nine alternatives that represented a diverse range of potential solutions to the mission need, including one that represented the status quo. NNSA defined these alternatives in detail, including providing descriptions of the specific characteristics used to create cost estimates. Before proceeding to the analysis, NNSA eliminated four alternatives it did not consider viable and described the general reasons for eliminating them but did not provide specific reasons for the scores it gave each alternative as part of the screening process. * Analyzing alternatives. NNSA minimally met best practices for the analyzing alternatives category by fully meeting 1 of the 8 best practices in this category, partially or minimally meeting 4, and not meeting the remaining 3. To assess the alternatives, NNSA developed cost estimates for each alternative, and presented these cost estimates with a stated accuracy range of 15 percent below to 50 percent above, but they were not life-cycle cost estimates because NNSA did not include such costs as decontamination and decommissioning in the estimates because it assumed those costs would be identical among the alternatives. NNSA did not develop quantitative estimates of the benefits that would result from each alternative, and mentioned the benefits of the alternatives only qualitatively through a discussion of some of the pros and cons of each alternative. To assess the risks associated with the alternatives, NNSA developed a list of risks for each alternative, but it did not develop potential mitigation strategies for all of the risks and did not test the sensitivity of the cost estimates to these risks or to changes in key assumptions. * Selecting a preferred alternative. NNSA partially met best practices for the selecting a preferred alternative category by fully or substantially meeting 2 of the 4 best practices in this category, partially meeting 1 and not meeting 1. To select a preferred alternative, NNSA developed 21 selection criteria such as life-cycle costs, schedules, risks, flexibility, and ability to meet the mission need. NNSA then weighted these criteria on a five-point scale to reflect their relative importance, and selected the alternative that received the highest overall score. As of June 2014, NNSA planned to conduct an independent project review to assess, among other things, whether the selected preferred alternative remained the most viable and feasible alternative for meeting the approved mission need, but it did not plan on assessing other parts of the AOA process, including the identification of alternatives, the criteria used for selecting the alternative, or the benefit estimates for each alternative. Table 8 below and appendix V summarize our assessment of the AOA conducted for this project. Table 8: Assessment of the Analysis of Alternatives for the High Explosive Science, Technology and Engineering Project, by Category of Best Practices and Overall: Best practice categories: General principles; Score: Substantially met; Number of practices: 8; Fully or substantially met: 4; Partially or minimally met: 4; Did not meet: 0. Best practice categories: Identifying alternatives; Score: Fully met; Number of practices: 4; Fully or substantially met: 4; Partially or minimally met: 0; Did not meet: 0. Best practice categories: Analyzing alternatives; Score: Minimally met; Number of practices: 8; Fully or substantially met: 1; Partially or minimally met: 4; Did not meet: 3. Best practice categories: Selecting a preferred alternative; Score: Partially met; Number of practices: 4; Fully or substantially met: 2; Partially or minimally met: 1; Did not meet: 1. Best practice categories: Overall best practices; Score: Partially met; Number of practices: 24; Fully or substantially met: 11; Partially or minimally met: 9; Did not meet: 4. Source: GAO analysis of National Nuclear Security Administration information. GAO-15-37. Note: The five-point scoring system that we used was as follows: Fully met means that NNSA's documentation demonstrated that NNSA completely met the best practice. Substantially met means that the NNSA's documentation demonstrated that NNSA met a large portion of the best practice. Partially met means that NNSA's documentation demonstrated that NNSA met about half of the best practice. Minimally met means that the NNSA's documentation demonstrated that NNSA met a small portion of the best practice. Did not meet means that NNSA's documentation did not demonstrate that NNSA met the best practice. [End of table] Radioactive Liquid Waste Treatment Facility AOA: For the Radioactive Liquid Waste Treatment Facility project, NNSA partially met best practices overall in conducting the most recent AOA in 2013. Therefore, this AOA also may not be reliable. NNSA fully or substantially met best practices in the general principles and identifying alternatives categories but not in the analyzing alternatives or selecting a preferred alternative categories. NNSA conformed to 8 of 24 best practices when conducting this AOA. Our assessment of the AOA conducted for this project identified the following: * General principles. NNSA substantially met best practices for the general principles category by fully or substantially meeting 3 of the 8 best practices in this category and by partially or minimally meeting the other 5. NNSA convened a team consisting of project management and technical experts. NNSA developed detailed study documentation for some areas of the AOA process, such as for risk and cost analysis, but not for others, such as for describing how NNSA screened alternatives for viability or how NNSA developed the selection criteria. NNSA also listed but did not justify the assumptions used in the AOA report, and did not discuss any potential constraints associated with the analysis. This analysis also may have been conducted with having a predetermined solution because one of the two alternatives considered was developed from a previous design that had already incurred significant cost increases and so was unlikely to be selected. * Identifying alternatives. NNSA substantially met best practices for the identifying alternatives category by fully or substantially meeting 2 of the 4 best practices in the category, and by partially or minimally meeting the other 2. NNSA identified thirty-seven options for various pieces of this project, and then combined those options into two complete alternatives to be analyzed: one that was based on the design selected in the 2006 AOA and another with a design similar to a previously-considered alternative. This approach compared an alternative developed from a design that had already incurred cost increases significant enough to warrant a reassessment of alternatives to only one other alternative. NNSA did, however, define the two alternatives in detail, assess the viability of the status quo and screen the list of thirty-seven options for viability before combining them into the two final alternatives. * Analyzing alternatives. NNSA partially met best practices for the analyzing alternatives category by fully or substantially meeting 3 of the 8 best practices in this category, partially or minimally meeting 2, and not meeting 3. To assess the alternatives, NNSA developed full life-cycle cost estimates for both of the alternatives, which it presented in present-value terms, but which it did not present with a confidence interval or range. NNSA did not develop quantitative estimates of the benefits that would result from each alternative, and did not explain how any measure of benefit used would support the mission need. To assess the risks associated with the alternatives, NNSA developed a list of risks and mitigation strategies for each alternative, and NNSA adjusted the total project cost estimates for each alternative, but not the life-cycle cost estimates, for these risks. NNSA did not, however, test the sensitivity of the full life- cycle cost estimates to risks or changes in key assumptions. * Selecting a preferred alternative. NNSA minimally met best practices for the selecting a preferred alternative category by partially or minimally meeting 2 of the 4 best practices in this category and by not meeting the other 2. To select a preferred alternative, NNSA compared the life-cycle cost estimates, the total project cost estimates, estimated schedules, and the scope of the alternatives. NNSA did not weight these selection criteria and did not compare the alternatives using net present value. NNSA conducted an independent project review of only the selected alternative, not of each alternative considered. NNSA also conducted a cost estimate review of total project cost estimates, not the life-cycle cost estimates, of each alternative. Table 9 below and appendix VI summarize our assessment of the AOA conducted for this project. Table 9: Assessment of the Analysis of Alternatives for the Radioactive Liquid Waste Treatment Facility, by Category of Best Practices and Overall: Best practice categories: General principles; Score: Substantially met; Number of practices: 8; Fully or substantially Met: 3; Partially or minimally met: 5; Did not meet: 0. Best practice categories: Identifying alternatives; Score: Substantially met; Number of practices: 4; Fully or substantially Met: 2; Partially or minimally met: 2; Did not meet: 0. Best practice categories: Analyzing alternatives; Score: Partially met; Number of practices: 8; Fully or substantially Met: 3; Partially or minimally met: 2; Did not meet: 3. Best practice categories: Selecting a preferred alternative; Score: Minimally met; Number of practices: 4; Fully or substantially Met: 0; Partially or minimally met: 2; Did not meet: 2. Best practice categories: Overall best practices; Score: Partially met; Number of practices: 24; Fully or substantially Met: 8; Partially or minimally met: 11; Did not meet: 5. Source: GAO analysis of National Nuclear Security Administration information. GAO-15-37. Note: The five-point scoring system that we used was as follows: Fully met means that NNSA's documentation demonstrated that NNSA completely met the best practice. Substantially met means that the NNSA's documentation demonstrated that NNSA met a large portion of the best practice. Partially met means that NNSA's documentation demonstrated that NNSA met about half of the best practice. Minimally met means that the NNSA's documentation demonstrated that NNSA met a small portion of the best practice. Did not meet means that NNSA's documentation did not demonstrate that NNSA met the best practice. [End of table] Uranium Processing Facility AOA: For the Uranium Processing Facility project, NNSA partially met best practices overall in conducting the most recent AOA in 2012. Therefore, this AOA also may not be reliable. NNSA fully or substantially met best practices in the identifying alternatives category but not in the general principles, analyzing alternatives, or selecting a preferred alternative categories. NNSA conformed to 6 of 24 best practices when conducting this AOA. Our assessment of NNSA's 2012 AOA for this project identified the following observations: * General principles. NNSA partially met best practices for the general principles category by fully or substantially meeting 2 of the 8 best practices in this category and by partially or minimally meeting the other 6. NNSA convened a team composed of federal project management staff and contractors to conduct the AOA, and they conducted the bulk of the analysis over 3 days in January 2012. NNSA did not create a detailed plan for conducting the AOA. Additionally, NNSA conducted the analysis for this project with, according to NNSA officials, a fundamental assumption that the selected alternative would be a single-facility solution such as the alternative selected in the project's previous AOA conducted in 2007. * Identifying alternatives. NNSA substantially met best practices for the identifying alternatives category by fully meeting 1 of the 4 best practices in the category, and by partially or minimally meeting the other 3. NNSA identified six alternatives, all of which were variations of the single-facility solution because, according to project officials, they were "wedded" to that idea. NNSA also did not consider the status quo as an alternative. NNSA did determine that one alternative was not viable before proceeding with the analysis but did not document the screening of all the alternatives against viability criteria. NNSA described the alternatives in sufficient detail. * Analyzing alternatives. NNSA minimally met best practices for the analyzing alternatives category by fully meeting 1 of the 8 best practices in the category, partially or minimally meeting 5, and by not meeting the other 2. To assess the alternatives, according to UPF project officials, NNSA developed relative cost estimates for each alternative that were updates of a 2010 total project cost estimate and that were not life-cycle cost estimates because they did not include costs that were the same across all alternatives. NNSA also did not develop quantitative estimates of the benefits that would result from each alternative. To assess the risks associated with the alternatives, NNSA listed categories of risk, such as cost, schedule, and technical risks in the description of each alternative but did not discuss specific risks or mitigation strategies. NNSA adjusted the design and construction elements of the cost estimates based on the risks identified in developing the 2010 total project cost estimate, but NNSA did not document these risks in project documentation and did not test the sensitivity of the estimates to changes in key assumptions. * Selecting a preferred alternative. NNSA partially met best practices for the selecting a preferred alternative category by fully or substantially meeting 2 of the 4 best practices in this category and by partially or minimally meeting the other 2. To select a preferred alternative, according to UPF project officials, NNSA used commercially available software to compare each alternative to every other alternative. NNSA used six selection criteria, including life- cycle cost, execution flexibility, and ability to achieve benefits from modernizing and consolidating enriched uranium processing, but NNSA did not compare the alternatives using net present value. NNSA weighted these criteria on a percentage scale and selected the alternative that scored the highest. NNSA also conducted an independent review after conducting the AOA to determine if the project was technically ready, but this was not a review of the AOA process. Table 10 below and appendix VII summarize our assessment of the AOA conducted for this project. Table 10: Assessment of the Analysis of Alternatives for the Uranium Processing Facility, by Category of Best Practices and Overall: Best practice categories: General principles; Score: Partially met; Number of practices: 8; Fully or substantially met: 2; Partially or minimally meet: 6; Did not meet: 0. Best practice categories: Identifying alternatives; Score: Substantially met; Number of practices: 4; Fully or substantially met: 1; Partially or minimally meet: 3; Did not meet: 0. Best practice categories: Analyzing alternatives; Score: Minimally met; Number of practices: 8; Fully or substantially met: 1; Partially or minimally meet: 5; Did not meet: 2. Best practice categories: Selecting a preferred alternative; Score: Partially met; Number of practices: 4; Fully or substantially met: 2; Partially or minimally meet: 2; Did not meet: 0. Best practice categories: Overall best practices; Score: Partially met; Number of practices: 24; Fully or substantially met: 6; Partially or minimally meet: 16; Did not meet: 2. Source: GAO analysis of National Nuclear Security Administration information. GAO-15-37. Note: The five-point scoring system was as follows: Fully met means that NNSA's documentation demonstrated that NNSA completely met the best practice. Substantially met means that the NNSA's documentation demonstrated that NNSA met a large portion of the best practice. Partially met means that NNSA's documentation demonstrated that NNSA met about half of the best practice. Minimally met means that the NNSA's documentation demonstrated that NNSA met a small portion of the best practice. Did not meet means that NNSA's documentation did not demonstrate that NNSA met the best practice. [End of table] NNSA Conducted Each AOA Differently: NNSA conducted each AOA differently and applied best practices differently among the projects. For example, as mentioned above in the description of the AOAs conducted at these projects, NNSA conducted the risk analysis differently for each project. The different ways that NNSA conducted the AOAs for the projects we examined may be due to the nonprescriptive nature of the Order 413.3B requirements. NNSA project officials stated that the AOA requirements in Order 413.3B are very high level and do not prescribe specific methods to follow. According to these officials, Order 413.3B provides a very general set of requirements for what the AOA process should accomplish. For example, DOE's Order 413.3B states that life-cycle cost assumptions should be included in the conceptual design report. The word "assumptions" leaves room for interpretation because it could be interpreted as requiring the inclusion of life-cycle cost estimates for all alternatives or as requiring a life-cycle cost estimate for only the selected alternative after completion of the AOA process. In the absence of specific requirements, each AOA team developed its own approach to conducting an AOA. For example, NNSA officials who worked on the AOA for the High Explosive, Science, Technology and Engineering Project stated that they had to develop much of the AOA process themselves and that they based their process on historical AOAs conducted at the Pantex site in Texas.[Footnote 32] Officials working on the AOA for the Radioactive Liquid Waste Treatment Facility project at Los Alamos Nuclear Laboratory told us that they also used site- specific guidance including the laboratory's engineering standards manual as well as their own professional experience. In contrast, officials working on the Uranium Processing Facility project stated that they conducted the AOA based on guidance contained within a commercially available, off-the-shelf software suite. In addition, there may be some confusion about roles and responsibilities for leading the AOA process at NNSA. NNSA officials in the Office of Acquisition and Project Management, who are responsible for managing the construction process, stated that their office does not have lead responsibility for the AOA process. However, the NNSA Concept of Operations for the Office of Acquisition and Project Management states that, while the program offices lead the development of the mission need and functional requirements and the selection of the preferred alternatives, the NNSA Office of Acquisition and Project Management has the lead for the development of the analyses of alternatives. NNSA officials stated that they will reexamine the Concept of Operations to clarify who has responsibility for the AOA process. The officials explained that NNSA APM has begun looking at, among other things, DOE's current directives and guidance and NNSA's policies and procedures relating to AOA, the Office of Management and Budget's guidance relating to AOAs, and the legislation Congress has enacted in response to other federal agencies' AOA challenges. NNSA Did Not Consistently Follow Certain DOE AOA Guidance for Its Projects: For the three AOAs we examined, NNSA consistently followed the DOE AOA requirement that conformed to best practices but did not consistently follow the elements of optional guidance that conform to best practices. As noted above, DOE has one requirement that conforms to best practices, the best practice of defining functional requirements based on the mission need. NNSA followed this requirement for all three AOAs we reviewed. Similarly, DOE has eight other elements of guidance that conform to best practices (see table 11). For the three projects we reviewed, NNSA followed from one to three of these eight elements of guidance. The specific elements of guidance NNSA followed for the AOAs differed substantially by project. Specifically, NNSA followed three elements of DOE guidance that conform to best practices for the High Explosive, Science, Technology and Engineering Facility Project AOA and followed two such elements of guidance for the Radioactive Liquid Waste Treatment Facility project and only one such element of guidance for the Uranium Processing Facility project AOAs. For example, the AOA for NNSA's High Explosive, Science, Technology and Engineering Project followed the DOE guidance that suggests identifying and considering at least three viable alternatives, including the status quo; the AOA identified nine alternatives, including one that represented the status quo. In contrast, the 2012 AOA for the Uranium Processing Facility identified six alternatives, but none represented the status quo, and the Radioactive Liquid Waste Treatment Facility project AOA identified two alternatives. Table 11 below shows the extent to which NNSA's AOAs for the projects we reviewed followed DOE's guidance that conform to best practices. Table 11: Extent to Which the National Nuclear Security Administration Followed Optional Department of Energy (DOE) Guidance That Conforms to Best Practices for Analysis of Alternatives: DOE elements of guidance that conform to best practices: The Federal Project Director and the Integrated Project Team are responsible for conducting different parts of the AOA process[A]; Analysis of Alternatives (AOA) reviewed: AOA for the High Explosives Science, Technology and Engineering Project: [Empty]; AOA for the Radioactive Liquid Waste Treatment Facility: [Empty]; AOA for the Uranium Processing Facility: [Empty]. DOE elements of guidance that conform to best practices: The Mission Need Statement should summarize the planned approach to conducting this analysis; Analysis of Alternatives (AOA) reviewed: AOA for the High Explosives Science, Technology and Engineering Project: [Empty]; AOA for the Radioactive Liquid Waste Treatment Facility: [Check]; AOA for the Uranium Processing Facility: [Empty]. DOE elements of guidance that conform to best practices: Identify and consider at least three viable alternatives, including the status quo; Analysis of Alternatives (AOA) reviewed: AOA for the High Explosives Science, Technology and Engineering Project: [Check]; AOA for the Radioactive Liquid Waste Treatment Facility: [Empty]; AOA for the Uranium Processing Facility: [Empty]. DOE elements of guidance that conform to best practices: Discuss and summarize the life-cycle cost of the alternatives; Analysis of Alternatives (AOA) reviewed: AOA for the High Explosives Science, Technology and Engineering Project: [Empty]; AOA for the Radioactive Liquid Waste Treatment Facility: [Check]; AOA for the Uranium Processing Facility: [Empty]. DOE elements of guidance that conform to best practices: Prior to CD-2 approval, DOE Order 413.3B requires the use of ranges to express project cost estimates; Ranges may be determined or based upon various project alternatives, project identified risks, and confidence levels; It is suggested that life-cycle costs of likely alternatives being considered have expected accuracy ranges from a low of -20% to -50% to a high of +30% to +100%; Analysis of Alternatives (AOA) reviewed: AOA for the High Explosives Science, Technology and Engineering Project: [Check]; AOA for the Radioactive Liquid Waste Treatment Facility: [Empty]; AOA for the Uranium Processing Facility: [Empty]. DOE elements of guidance that conform to best practices: The benefits of each alternative should be quantified over that alternative's full life-cycle[B]; Analysis of Alternatives (AOA) reviewed: AOA for the High Explosives Science, Technology and Engineering Project: [Empty]; AOA for the Radioactive Liquid Waste Treatment Facility: [Empty]; AOA for the Uranium Processing Facility: [Empty]. DOE elements of guidance that conform to best practices: The selection criteria should be weighted; Analysis of Alternatives (AOA) reviewed: AOA for the High Explosives Science, Technology and Engineering Project: [Check]; AOA for the Radioactive Liquid Waste Treatment Facility: [Empty]; AOA for the Uranium Processing Facility: [Check]. DOE elements of guidance that conform to best practices: Normally, analyses require comparing alternatives using net present value or annuities; Analysis of Alternatives (AOA) reviewed: AOA for the High Explosives Science, Technology and Engineering Project: [Empty]; AOA for the Radioactive Liquid Waste Treatment Facility: [Empty]; AOA for the Uranium Processing Facility: [Empty]. Sources: GAO analysis of DOE and NNSA information. GAO-15-37. Note: A check mark indicates that NNSA received a score of "fully met" or "substantially met," because NNSA followed the element of DOE guidance that conformed to the best practice. No check mark indicates that NNSA received a score of "partially met," "minimally met," or "not met," because NNSA did not follow the element of the DOE guidance that conformed to a best practice. [A] This statement is our summary of language included in multiple guides suggesting different tasks to be performed by the Federal Project Director and the Integrated Project Team. [B] This language reflects our understanding of this guidance based on an example included in one of DOE's guides. [End of table] Conclusions: The AOA process is a key first step in the project management process at which DOE and NNSA have the opportunity to put projects on the right path forward. DOE has some requirements and guidance for this process in its project management Order 413.3B and associated guides. However, DOE's AOA requirements in Order 413.3B conform to only 1 of the 24 best practices we identified from different government and private-sector entities, and the additional suggestions provided in DOE's guides are still not sufficient to create a reliable AOA process. The AOAs for the three recent NNSA projects we reviewed--the High Explosive Science, Technology and Engineering Project, the Radioactive Liquid Waste Treatment Facility, and the Uranium Processing Facility--did not conform to best practices overall and may therefore not be reliable. DOE and NNSA officials acknowledge that unreliable AOAs are a risk factor for major cost increases and schedule delays for NNSA projects. Federal standards for internal control related to risk assessment call for management to decide on actions to mitigate identified risks. Developing requirements that incorporate best practices would help DOE mitigate the risk for major cost increases and schedule delays by creating a framework to ensure a reliable AOA process. Developing reliable AOAs is particularly important at this time because NNSA has recently reassessed or is in the process of reassessing alternatives for all its major construction projects, after spending billions of dollars on designing and partially constructing these projects. Without a process to develop reliable AOAs, NNSA may continue on this path and continue to have limited assurance that it is selecting alternatives that best meet its mission needs and will not result in major cost increases and schedule delays in the future. Recommendation for Executive Action: To minimize the risk of developing unreliable AOAs and incurring major cost increases and schedule delays on projects, we recommend that the Secretary of Energy direct DOE's Office of Acquisition and Project Management to update its project management order requirements to incorporate best practices for conducting an AOA. Agency Comments and Our Evaluation: We provided DOE with a draft of this report for its review and comment. In its written comments, reproduced in appendix VIII, DOE agreed with the report's recommendation. In its written comments, DOE stated that DOE's order for project management (DOE O 413.3B) will be assessed for revision following the issuance of the revision to DOE- STD1189, Integration of Safety into the Design Process, currently scheduled for November 2016. DOE stated that it will consider the AOA best practices when the policy is updated. In the interim, DOE stated that it has strengthened guidance in the area of AOA, including issuing a Lifecycle Cost Handbook, and that in fiscal year 2015 it will develop and issue an AOA Handbook that will reflect the best practices for AOAs. DOE also provided technical comments that were incorporated, as appropriate. We are pleased that DOE agreed with our recommendation and that it will take action to issue an AOA Handbook. However, while an AOA Handbook may be a useful interim measure, the unspecified, open-ended date for updating the project management order that contains requirements (i.e., sometime after November 2016) and the statement that the AOA best practices will be considered, not incorporated, may indicate DOE's lack of urgency in implementing this recommendation. We are sending copies of this report to the appropriate congressional committees, the Secretary of Energy, and other interested parties. In addition, the report is available at no charge on the GAO website at [hyperlink, http://www.gao.gov]. If you or your staff have any questions about this report, please contact me at (202) 512-3841 or trimbled@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IX. David C. Trimble: Director, Natural Resources and Environment: [End of section] Appendix I: Scope and Methodology: To identify and describe best practices for the analysis of alternatives (AOA) process, we first searched for a source of generally accepted best practices that we could use as criteria for comparison with the Department of Energy's (DOE) AOA process, but we could not identify a single source that was broadly recognized by government and private-sector entities and that could serve as the definitive source of best practices for the AOA process. In the absence of such a definitive source, we identified AOA handbooks, guidebooks, requirements, and other AOA-related information from federal and private-sector entities. We also sent a request to subject- matter experts, including experts from DOE, the National Nuclear Security Administration (NNSA), and various other government or private-sector entities, to help us identify further relevant information.[Footnote 33] These subject-matter experts are part of GAO's Projects Controls Expert Working Group, which contains members with expertise in program and project management, capital acquisition, cost estimation, risk and sensitivity analysis, earned value management and scheduling, and represent a diverse range of government and private-sector entities. We reviewed the information related to the AOA process, including information from our Cost Estimating Guide,[Footnote 34] the Association for the Advancement of Cost Engineering International, Department of Defense, Department of Homeland Security, National Aeronautics and Space Administration, Office of Management and Budget, United States Air Force, and a private-sector entity that specializes in acquisition management. We compiled a draft set of best practices commonly mentioned across these different entities' AOA policies and guidance, and we sent this draft set of AOA best practices to the experts for review in advance of our semiannual meeting of the Projects Controls Expert Working Group that took place in March 2014. More than 90 experts participated, including officials from DOE and NNSA. We received comments from some of these experts both during this meeting and by e-mail after the meeting. We developed a final set of 24 best practices for the AOA process based on the comments received. We grouped these best practices into four categories: (1) general principles, (2) identifying alternatives, (3) analyzing alternatives, and (4) selecting a preferred alternative. To determine the extent to which DOE's requirements and guidance for conducting an AOA conform to AOA best practices, we first identified the process NNSA is required to follow when conducting an AOA. To do so, we reviewed DOE's project management order and its associated guides, as well as other documentation received from NNSA's Office of Acquisition and Project Management, including a compilation of DOE's requirements and guidance related to conducting an AOA. We also interviewed officials from DOE's Office of Acquisition and Project Management, NNSA's Office of Acquisition and Project Management, NNSA's Office of Cost Estimating and Program Evaluation, DOE's Office of Science, and a former DOE official familiar with DOE's and NNSA's project management process, as well as NNSA and contractor officials from the programs and projects we selected for review. We then confirmed DOE's requirements and guidance with DOE and NNSA officials, from whom we received technical comments. We performed two sets of analyses: we compared the best practices to (1) DOE's requirements and (2) DOE's requirements combined with the guidance. We used a five-point scoring system to determine the extent to which DOE's AOA process conforms to best practices.[Footnote 35] We first used this scoring system to determine how well DOE's requirements and guidance conform to each best practice. We then used the average of the scores for the best practices in each of the four categories to determine an overall score for each category, and we then used the average of the scores for the four categories as the final score for the overall DOE AOA process. If the score for each best practice, the average score for each category, or the final score for the AOA process was "fully met" or "substantially met," we concluded that the AOA process conformed to best practices and therefore could be considered reliable. In contrast, if the score was "partially met," "minimally met," or "not met," we concluded that the AOA process did not conform to best practices and therefore could not be considered reliable. For us to consider the AOA process reliable, the entire AOA process had to receive an average score of "fully met" or "substantially met," and each individual category--(1) general principles, (2) identifying alternatives, (3) analyzing alternatives, and (4) selecting a preferred alternative--had to receive an average score of "fully met" or "substantially met" to ensure that the agency adequately and consistently performed all parts of the AOA process. GAO does not weight its cost estimating, scheduling, earned value management and AOA best practices because it is not possible to quantitatively determine the relative weights of each of the criteria. Weighting has the potential to vary across programs, as well as due to where a program may be in its different stages of its life cycle. Therefore the fairest standardized methodology to evaluate programs against is to have all criteria weighted equally. To determine the extent to which NNSA conformed to best practices in conducting the AOAs for recent NNSA projects, we used as criteria the best practices identified as part of this engagement. We chose projects for which NNSA had completed the AOA process since November 2010, when DOE's most recent version of the project management order came into effect, or for which NNSA was scheduled to complete the AOA process by the end of fiscal year 2014. We chose projects for which NNSA had completed, or was nearing completion of, an AOA process under this version of the order because this version included significant revisions, including more specific and detailed requirements for conducting an AOA. We also selected NNSA projects that were scheduled to complete an AOA by the end of fiscal year 2014 because most of the AOA documentation would be available for these projects. We identified these projects by examining DOE's data from its February 2014 Monthly Project Portfolio Status Report. The three projects listed in this report that met our criteria were: (1) the High Explosive Science, Technology and Engineering Project at NNSA's Pantex site in Texas, for which NNSA was scheduled to finish the AOA process by the end of September 2014 at the time of our project selection; (2) the Radioactive Liquid Waste Treatment Facility at NNSA's Los Alamos National Laboratory in New Mexico, for which NNSA completed its most recent reassessment of alternatives in September 2013; and (3) the Uranium Processing Facility at NNSA's Y-12 site in Tennessee, for which NNSA completed its most recent reassessment of alternatives in June 2012.[Footnote 36] We checked with knowledgeable DOE and NNSA officials, reviewed recent GAO reports, and compared the data in the project status report with information received from NNSA on the projects to assess the reliability of the data in this status report, and we determined that the data were sufficiently reliable to ensure these projects met our criteria. For each of these projects, we reviewed project documentation, and we interviewed NNSA and contractor officials in charge of the AOA to compare the process followed by NNSA with best practices. To score each AOA process, (1) two GAO analysts separately examined the AOA documentation received from the agency and then agreed on a score for each of the 24 best practices, then, (2) an AOA specialist independent of the engagement team reviewed the AOA documentation and the scores assigned by the analysts for accuracy and cross-checked the scores in all the analyses for consistency. After completing this process for each individual best practice in all the analyses, we calculated the scores for each category--(1) general principles, (2) identifying alternatives, (3) analyzing alternatives, and (4) selecting a preferred alternative--and the overall score for each analysis. We sent our analyses to DOE and NNSA for review twice, and we used the same scoring process when we revised the scores based on their technical comments and any additional evidence received. In April 2014, as we were conducting our engagement, NNSA released a peer review report that recommended a new alternative for the Uranium Processing Facility. We examined whether this peer review reflected any characteristics of an AOA by interviewing NNSA officials and comparing the peer review with the AOA best practices. In addition, as part of this objective, we determined the extent to which NNSA conformed to certain DOE requirements and guidance--those DOE requirements and guidance that conformed to best practices--in conducting the AOAs for these projects. We used the results of the earlier analysis comparing DOE's requirements and guidance to best practices to determine which DOE requirements and guidance conformed to best practices. We then reviewed project documentation and interviewed NNSA and contractor officials in charge of the AOA process to assess the extent to which NNSA projects followed DOE's requirements and guidance that conformed to best practices. We conducted this performance audit from February 2014 to December 2014 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. [End of section] Appendix II: The Department of Energy's Analysis of Alternatives Process: The Department of Energy's (DOE) analysis of alternatives (AOA) process includes the following requirements from DOE Order 413.3B and suggested guidance from seven guides associated with this order. The source of the requirements or guidance is listed in parentheses at the end of the item. DOE requirements: * The Integrated Project Team--the team supporting the Federal Project Director to deliver a capital asset project and conduct the AOA-- consists of professionals representing a diverse range of disciplines and specific knowledge to successfully execute the project.[Footnote 37] (O 413.3B p. C-11): - As part of the preconceptual design (CD-0) approval process: - The conceptual design must develop the scope required to satisfy the program's mission requirements and identify requirements and features. A Mission Need Statement document is developed to identify the mission need--the gap that needs to be addressed--and functional requirements-- the general parameters that the selected alternative must have to address the mission need. (O 413.3B pp. A-4, C-5): - The mission need must be independent of a particular solution. (O 413.3B p. A-4): - The program office responsible for the capital asset project must identify a credible performance gap between its current capabilities and capacities and those required to achieve the goals articulated in its strategic plan. (O 413.3B p. A-4): - The program office is afforded the flexibility to explore a variety of solutions and not limit potential solutions. (O 413.3B p. A-4): * As part of the conceptual design (CD-1) approval process: - The conceptual design must develop reliable cost and schedule range estimates for the alternatives considered. The order also requires that whatever figure or range that is provided at the CD-0 and CD-1 stages must explicitly note relevant caveats concerning risks and uncertainties inherent in early estimated given the immature requirements definition at these stages. (O 413.3B pp. C-5, C-16): - The conceptual design also must develop an assessment of project risks and identification of appropriate risk handling strategies. (O 413.3B p. C-4): - Develop a conceptual design report that includes, among other things, a clear and concise description of the alternatives analyzed, the basis for the selected alternative, how the selected alternative meets the mission need, the functions and requirements that define the alternative and demonstrate the capability that the alternative can be successful, and life-cycle cost assumptions.[Footnote 38] (O 413.3B pp. A-7, C-5): * The order also requires that: - The project's sponsor must never be the sole cost estimator at any stage, and that the second cost estimator must be from outside the chain of command, to avoid conflict of interest. (O 413.3B pp. A-1, C- 16): - Conduct several independent reviews during CD-0 and CD-1 approval processes, depending on the estimated cost of the project, related to two aspects of the AOA process: the validation of the mission need statement and of the cost estimates. (O 413.3B pp. A-7, C-7, C-17, C- 18): - The selected alternative must be the optimum solution that must provide the essential functions and capabilities at an optimum life- cycle cost, consistent with required cost, scope, schedule, performance, and risk considerations. (O 413.3B pp. A-2, A-5): Additionally, DOE's order requires that as a project moves toward approval of CD-2, if the top end of the approved CD-1 cost range for the selected alternative grows by more than 50 percent, the program must reassess alternatives. (O 413.3B p. A-6): The order also requires as part of CD-1 approval that an analysis of alternatives must be conducted under the National Environmental Policy Act of 1969 (NEPA).[Footnote 39] (O 413.3B p. A-7): DOE guidance: * Summarize a planned approach to conduct an analysis of alternatives.[Footnote 40] (G 413.3-17 p. 4): - The guides provide numerous examples of optional tasks for the Integrated Project Team and the Federal Project Director to perform as part of the AOA process, such as identifying functional requirements, evaluating alternatives for satisfying the requirements, conducting the appropriate analyses, and recommending a preferred alternative. (G 413.3-1 pp. 5, 7, 14, 15)[Footnote 41] (G 413.3-5A p. 8)[Footnote 42] (G 413.3-13 pp. 2, 3, 17)[Footnote 43] (G 413.3-18A p. 8)[Footnote 44] (G-413.3-21 p. 9)[Footnote 45] - Explore concepts and consider alternatives for meeting the mission need until a set of viable, affordable, and sustainable alternatives is reached; consider at least three viable alternatives for analysis, including one that represents the status quo; and develop and list the primary advantages and disadvantages for each alternative. (G-413.3-13 p. 9, G 413.3-21 p. B-2): - Develop cost estimates prior to the CD-2 approval milestone that are explicit ranges instead of point estimates, including life-cycle cost estimates of likely alternatives being considered that have a broad accuracy range from a low of -20 to -50 percent to a high of +30 to +100 percent. (G 413.3-21 p. 15): - Quantify the benefits of alternatives over their life cycle. (G 413.3-13 p. 10): - Adjust life-cycle cost and benefit estimates for risk to ensure consideration of the alternative with the best cost-benefit ratio and generally the lowest life-cycle cost to the federal government. (G 413.3-21 p. 9): - Consider various selection criteria for the alternatives, including cost and schedule, funding and budget, and technology and engineering. (G 413.3-13 p. 9): - Weight the selection criteria used to select a preferred alternative for relative importance and compare alternatives using net present value or annuities.[Footnote 46] (G 413.3-13 p. 10; G 413.3-21 p. 69): - At the end of the analysis, the AOA team presents the recommended alternative based on the preceding analysis in an integrated form, summarizing why an alternative is preferred, and supporting the recommendation of the preferred alternative with facts from the analysis. (G 413.3-13 p. 13): - The program office performs an alternative selection and cost-range review to assess whether the AOA process (1) evaluates a range of appropriate attributes for each alternative, including cost, risks, safety, technology, and regulatory requirements and (2) is reasonable and provides best value to the federal government.[Footnote 47] (G 413.3-9 pp. 16-19): [End of section] Appendix III: Best Practices for the Analysis of Alternatives Process: We identified 24 best practices for identifying, analyzing, and selecting alternatives. These practices can be applied to a wide range of activities in which an alternative must be selected from a set of possible options, and to a broad range of capability areas, projects, and programs. These practices provide a framework to ensure that entities consistently and reliably select the alternatives that best meet mission needs. We grouped these 24 best practices into four categories: (1) general principles, (2) identifying alternatives, (3) analyzing alternatives, and (4) selecting a preferred alternative. The four categories of best practices address the entire analysis of alternatives (AOA) process from defining the mission need and functional requirements to independently reviewing its results. On the basis of our reviews and experts' comments, we believe that these best practices can be generally applied from the beginning of the AOA process with practices from the general principles category, through practices in the identifying and analyzing alternatives categories, and ending with practices in the selecting a preferred alternative category. We also believe that these best practices do not necessarily have to be followed in order and that some of them can be applied concurrently with other best practices. For example, the best practice of defining the selection criteria based on the mission need in the selecting a preferred alternative category could be addressed at the same time as the best practice of creating a study plan in the general principles category. Table 12 below provides a complete list of best practices. Table 12: Best Practices for the Analysis of Alternatives (AOA) Process: I. General principles: 1. The customer defines the mission need and functional requirements without a predetermined solution. 2. The customer defines functional requirements based on the mission need. 3. The customer provides the team conducting the analysis of alternatives (AOA) with enough time to complete the AOA process to ensure a robust and complete analysis. 4. The team includes members with diverse areas of expertise including, at a minimum, subject matter expertise, project management, cost estimating, and risk management. 5. The team creates a plan, including proposed methodologies, for identifying, analyzing, and selecting alternatives, before beginning the AOA process. 6. The team documents all steps taken to identify, analyze and select alternatives in a single document. 7. The team documents and justifies all assumptions and constraints used in the analysis. 8. The team conducts the analysis without a predetermined solution. II. Identifying alternatives: 9. The team identifies and considers a diverse range of alternatives to meet the mission need. 10. The team describes alternatives in sufficient detail to allow for robust analysis. 11. The team includes one alternative representing the status quo to provide a basis of comparison among alternatives. 12. The team screens the list of alternatives before proceeding, eliminates those that are not viable, and documents the reasons for eliminating any alternatives. III. Analyzing alternatives: 13. The team develops a life-cycle cost estimate for each alternative, including all costs from inception of the project through design, development, deployment, operation, maintenance, and retirement[A]. 14. The team presents the life-cycle cost estimate for each alternative as a range or with a confidence interval, and not solely as a point estimate. 15. The team expresses the life-cycle cost estimate in present value terms and explains why it chose the specific discount rate used.[B]. 16. The team uses a standard process to quantify the benefits/effectiveness of each alternative and documents this process. 17. The team quantifies the benefits/effectiveness resulting from each alternative over that alternative's full life cycle, if possible. 18. The team explains how each measure of benefit/effectiveness supports the mission need. 19. The team identifies and documents the significant risks and mitigation strategies for each alternative. 20. The team tests and documents the sensitivity of both the cost and benefit/effectiveness estimates for each alternative to risks and changes in key assumptions. IV. Selecting a preferred alternative: 21. The team or the decision maker defines selection criteria based on the mission need. 22. The team or the decision maker weights the selection criteria to reflect the relative importance of each criterion. 23. The team or the decision maker compares alternatives using net present value,[C] if possible. 24. An entity independent of the AOA process reviews the extent to which all best practices have been followed (for certain projects, additional independent reviews may be necessary at earlier stages of the process such as for reviewing the study plan or for reviewing the identification of viable alternatives). Source: GAO. GAO-15-37. [A] GAO-09-3SP contains a separate set of best practices to assess the reliability of life-cycle cost estimates included in the AOA. [B] The present value of an estimate reflects the time value of money, the concept that a dollar in the future is worth less than a dollar today because the dollar today can be invested and earn interest. The discount rate is the interest rate used to calculate the present value of an estimate. [C] The net present value is the difference between the discounted present value of benefits and the discounted present value of costs. [End of table] [End of section] Appendix IV: DOE's AOA Requirements and Guidance Compared with Best Practices: Overall, the Department of Energy's (DOE) requirements for analysis of alternatives (AOA) in Order 413.3B minimally meet the best practices we identified. DOE's requirements combined with its guidance in the guides associated with the order partially meet the best practices. Table 13 below describes our analysis of DOE's AOA requirements and guidance compared with best practices. Table 13: The Department of Energy's (DOE) Requirements and Guidance Compared with Analysis of Alternatives (AOA) Best Practices: Best Practices in the General Principles Category[A]: 1. The customer defines the mission need and functional requirements without a predetermined solution. DOE requirements: The mission need is independent of a particular solution [part of CD-0 approval]. (O 413.3B p. A-4); The program office will identify a credible performance gap between its current capabilities and capacities and those required to achieve the goals articulated in its strategic plan. (O413.3B p. A-4); DOE guidance: No information found. GAO's assessment of DOE requirements alone: Partially meet. DOE's requirements indicate that the program office will define the mission need without a predetermined solution, but DOE does not have a similar requirement for the functional requirements; GAO's assessment of DOE requirements combined with guidance: Partially meet. The score does not change because DOE's guidance does not include any additional information. 2. The customer defines functional requirements based on the mission need. Best Practices in the General Principles Category[A]: DOE requirements: The conceptual design shall develop the scope required to satisfy the program's mission requirements and identify requirements and features. The Mission Need Statement is the translation of this gap into functional requirements that cannot be met through other than material means [part of CD-0 approval] (O 413.3B pp. A-4, C-5); DOE guidance: No information found. GAO's assessment of DOE requirements alone: Substantially meet. DOE requirements indicate that functional requirements should be developed to satisfy the mission need, and define key elements of those requirements in the Mission Need Statement. However, DOE's requirements are not clear that the customer will perform this task; GAO's assessment of DOE requirements combined with guidance: Substantially meet. The score does not change because DOE's guidance does not include any additional information. 3. The customer provides the team conducting the analysis of alternatives (AOA) with enough time to complete the AOA process to ensure a robust and complete analysis. Best Practices in the General Principles Category[A]: DOE requirements: No information found; DOE guidance: No information found. GAO's assessment of DOE requirements alone: Do not meet. DOE's requirements do not include any language related to a time frame for conducting an AOA; GAO's assessment of DOE requirements combined with guidance: Do not meet. The score does not change because DOE's guidance does not include any additional information. 4. The team includes members with diverse areas of expertise including, at a minimum, subject matter expertise, project management, cost estimating, and risk management. DOE requirements: The Integrated Project Team (IPT) consists of professionals representing diverse disciplines with the specific knowledge, skills and abilities to support the Federal Project Director (FPD) in successfully executing a project. (O 413.3B p. C-11); DOE guidance: NNSA's Office of Acquisition and Project Management (APM) has the lead for the development of the alternatives analysis. (NNSA APM Conc. Ops. p. 6)[B]; [The FPD and IPT are responsible for conducting different parts of the AOA] (G 413.3-5A p. 8; DOE G 413.3- 18A, p. 8; DOE G 413.3-21, p. 9; DOE G 413.3-1, pp. 5, 7, 14, 15; DOE G 413.3-13, pp. 2, 3, 17). GAO's assessment of DOE requirements alone: Partially meet. DOE's requirements do not specify who must conduct the AOA. The requirements specify that the IPT must represent a diverse range of disciplines but do not specify what those should be; GAO's assessment of DOE requirements combined with guidance: Substantially meet. DOE's guidance adds a suggestion for the FPD and IPT to conduct the AOA but does not specify which areas of expertise should be represented on those teams. 5. The team creates a plan, including proposed methodologies, for identifying, analyzing, and selecting alternatives, before beginning the AOA process. DOE requirements: No information found; DOE guidance: The Mission Need Statement should summarize the planned approach to conducting this analysis. (DOE G 413.3-17 p. 4). GAO's assessment of DOE requirements alone: Do not meet. DOE's requirements do not include any language related to an AOA study plan; GAO's assessment of DOE requirements combined with guidance: Substantially meet. DOE's guidance adds a suggestion for a study plan but does not specify what should be included in that plan. 6. The team documents all steps taken to identify, analyze and select alternatives in a single document. DOE requirements: A Conceptual Design Report shall be developed that includes a clear and concise description of the alternatives analyzed, the basis for the alternative selected, how the alternative meets the approved mission need, the functions and requirements that define the alternative and demonstrate the capability for success, and the facility performance requirements, planning standards and life-cycle cost assumptions. (O 413.3B pp. A-7, C-5); DOE guidance: No information found. GAO's assessment of DOE requirements alone: Partially meet. DOE requirements indicate that the conceptual design report must include details of the steps taken in several aspects of conducting the AOA but do not require documenting all the steps required as part of conducting the AOA; GAO's assessment of DOE requirements combined with guidance: Partially meet; The score does not change because DOE's guidance does not include any additional information. 7. The team documents and justifies all assumptions and constraints used in the analysis. DOE requirements: The Conceptual Design Report shall include life-cycle cost assumptions. (O 413.3B pp. A-7, C-5); DOE guidance: No information found. GAO's assessment of DOE requirements alone: Minimally meet. DOE's requirements state that the team must document the assumptions related to the life-cycle cost estimates but do not mention including any other assumptions or constraints; GAO's assessment of DOE requirements combined with guidance: Minimally meet. The score does not change because DOE's guidance does not include any additional information. 8. The team conducts the analysis without a predetermined solution. DOE requirements: No information found; DOE guidance: No information found. GAO's assessment of DOE requirements alone: Do not meet. DOE's requirements do not include any language related to conducting the analysis without a predetermined solution; GAO's assessment of DOE requirements combined with guidance: Do not meet. The score does not change because DOE's guidance does not include any additional information. Best Practices in the Identifying Alternatives Category: 9. The team identifies and considers a diverse range of alternatives to meet the mission need. DOE requirements: The program office [the customer] must explore a variety of solutions and not limit potential solutions [part of CD-0 approval]. (O 413.3B p. A-4); DOE guidance: No information found. GAO's assessment of DOE requirements alone: Partially meet. DOE's requirements indicate that the project's customer must explore a variety of alternatives, but the best practice suggests that the AOA team identify these alternatives; GAO's assessment of DOE requirements combined with guidance: Partially meet. The score does not change because DOE's guidance does not include any additional information. 10. The team describes alternatives in sufficient detail to allow for robust analysis. DOE requirements: The Conceptual Design Report shall include a clear and concise description of the alternatives analyzed, including the functions and requirements that define the alternatives. (O 413.3B p. C-5); DOE guidance: List and describe the alternatives reasonably used to meet the required capability and the primary advantages and disadvantages for each. (G 413.3-13 p. 9). GAO's assessment of DOE requirements alone: Partially meet. DOE's requirements indicate that the alternatives must be clearly and concisely defined but do not include a requirement for the definitions to be detailed enough to allow for a robust analysis; GAO's assessment of DOE requirements combined with guidance: Partially meet. DOE's guidance adds a suggestion to list the primary advantages and disadvantages of each alternative but does not mention anything about providing detailed enough descriptions to allow for a robust analysis of the alternatives. 11. The team includes one alternative representing the status quo to provide a basis of comparison among alternatives. DOE requirements: No information found; DOE guidance: Identify and consider at least three viable alternatives, including the status quo. (G 413.3-13 p. 9). GAO's assessment of DOE requirements alone: Do not meet. DOE's requirements do not include any language on including a status quo alternative; GAO's assessment of DOE requirements combined with guidance: Fully meet. DOE's guidance adds a suggestion to consider at least three viable alternatives, including one representing the status quo. 12. The team screens the list of alternatives before proceeding, eliminates those that are not viable, and documents the reasons for eliminating any alternatives. DOE requirements: No information found; DOE guidance: During conceptual design, concepts for meeting a mission need are explored and alternatives considered before arriving at the set of alternatives that are technically viable, affordable, sustainable. (G 413.3-21 p. B-2). GAO's assessment of DOE requirements alone: Do not meet. DOE's requirements do not include any language on pre- screening the list of alternatives for viability; GAO's assessment of DOE requirements combined with guidance: Partially meet. DOE's guidance suggests pre-screening the list of alternatives but does not suggest documenting the reasons for eliminating any alternatives. Best Practices in the Analyzing Alternatives Category: 13. The team develops a life-cycle cost estimate for each alternative, including all costs from inception of the project through design, development, deployment, operation, maintenance, and retirement. DOE requirements: The Conceptual Design Report shall include life- cycle cost assumptions. (O 413.3B p. C-5); Life-cycle costs: The sum total of all direct, indirect, recurring, nonrecurring, and other related costs incurred or estimated to be incurred in the planning, design, development, procurement, production, operations and maintenance, support, recapitalization, and final disposition of real property over its anticipated life span for every aspect of the program, regardless of funding source. (O 413.3B Attachment 2 p. 8); DOE guidance: Discuss and summarize the life-cycle cost of the alternatives. (G 413.3-13 p. 12). GAO's assessment of DOE requirements alone: Partially meet. DOE's requirements state that the conceptual design report must include the assumptions related to the life-cycle cost. This requirement could be interpreted to mean that life-cycle cost estimates must be included in this report for all alternatives, but this is not explicitly stated. DOE and NNSA officials interpret this statement to require a life-cycle cost estimate only for the selected alternative after the AOA process is finished; GAO's assessment of DOE requirements combined with guidance: Fully meet. DOE's guidance adds a suggestion that life-cycle cost estimates be developed for each of the alternatives. 14. The team presents the life-cycle cost estimate for each alternative as a range or with a confidence interval, and not solely as a point estimate. DOE requirements: The conceptual design should develop reliable cost and schedule range estimates for the alternatives considered. Whatever figure or range is provided should explicitly note relevant caveats concerning risks and uncertainties inherent in early estimates at CD-0 and CD-1 stages given the immature requirements definition at this juncture. (O 413.3B p. C-5); DOE guidance: Prior to CD-2 approval, DOE Order 413.3B requires the use of ranges to express project cost estimates. Ranges may be determined or based upon various project alternatives, project identified risks, and confidence levels. (G 413.3-21 p. 17); It is suggested that life-cycle costs of likely alternatives being considered be Class 5 estimates (which have expected accuracy ranges from a low of -20% to -50% to a high of +30% to +100%). (G 413.3-21 p. 15). GAO's assessment of DOE requirements alone: Partially meet. DOE's requirements are somewhat contradictory with one sentence requiring "cost range estimates" and another saying "whatever figure or range is provided." The requirements do state that the team must note uncertainties and risks involved with these estimates; GAO's assessment of DOE requirements combined with guidance: Fully meet. DOE's guidance adds an explicit suggestion to use cost ranges to express the project cost estimates, and recommends a specific class of estimate to use. 15. The team expresses the life-cycle cost estimate in present value terms and explains why it chose the specific discount rate used[C]. DOE requirements: No information found; DOE guidance: No information found. GAO's assessment of DOE requirements alone: Do not meet. DOE's requirements do not contain any language related to expressing life-cycle cost estimates in present value terms or related to discount rates; GAO's assessment of DOE requirements combined with guidance: Do not meet. The score does not change because DOE's guidance does not include any additional information. 16. The team uses a standard process to quantify the benefits/effectiveness of each alternative and documents this process. DOE requirements: No information found; DOE guidance: [An example of an analysis of alternatives shows that benefits should be quantified for each alternative over its life- cycle.] (G413.3-13 p. 10). GAO's assessment of DOE requirements alone: Do not meet. DOE's requirements do not contain any language related to quantifying benefits; GAO's assessment of DOE requirements combined with guidance: Minimally meet. DOE's guidance implies the development of a life-cycle benefit estimate, but the guidance does not mention using or documenting this process. 17. The team quantifies the benefits/effectiveness resulting from each alternative over that alternative's full life cycle, if possible. DOE requirements: No information found; DOE guidance: [An example of an analysis of alternatives shows that benefits should be quantified for each alternative over its life cycle.] (G 413.3-13 p. 10). GAO's assessment of DOE requirements alone: Do not meet. DOE's requirements do not contain any language related to quantifying benefits over the alternative's life cycle; GAO's assessment of DOE requirements combined with guidance: Substantially meet. DOE's guidance implies including a life-cycle benefit estimate for each alternative. 18. The team explains how each measure of benefit/effectiveness supports the mission need. DOE requirements: No information found; DOE guidance: No information found. GAO's assessment of DOE requirements alone: Do not meet. DOE's requirements do not contain any language related to measures of benefit/effectiveness and mission need; GAO's assessment of DOE requirements combined with guidance: Do not meet. The score does not change because DOE's guidance does not include any additional information. 19. The team identifies and documents the significant risks and mitigation strategies for each alternative. DOE requirements: The conceptual design shall develop an assessment of project risks and identification of appropriate risk handling strategies. (O 413.3B p. C-4); DOE guidance: Risk should be taken into consideration when evaluating the alternatives. (G 413.3-13 p. 13). GAO's assessment of DOE requirements alone: Partially meet. DOE's requirements indicate development of an assessment of project risks and mitigation strategies but do not clarify if this must be done for each alternative or only for the selected alternative; GAO's assessment of DOE requirements combined with guidance: Partially meet. DOE's guidance adds a suggestion that risk be taken into account when evaluating the alternatives but does not mention identifying mitigation strategies. 20. The team tests and documents the sensitivity of both the cost and benefit/effectiveness estimates for each alternative to risks and changes in key assumptions. DOE requirements: No information found; DOE guidance: A risk-adjusted life-cycle cost estimate should be prepared for each alternative under consideration to ensure the alternative with the best cost/benefit ratio (and generally the lowest life-cycle cost) to the government is considered. (G 413.3-21 p. 9); [An example of an analysis of alternatives shows that the life-cycle cost and benefit estimates should be risk adjusted.] (G 413.3-13 p. 10). GAO's assessment of DOE requirements alone: Do not meet. DOE's requirements do not contain any language related to testing the sensitivity of estimates to risks and changes in key assumptions; GAO's assessment of DOE requirements combined with guidance: Partially meet. DOE's guidance adds suggestions to adjust the life- cycle cost estimates for risk, and an example shows adjusting the benefit estimates for risk, but the guidance does not mention testing the estimates for sensitivity to changes in key assumptions. Best Practices in the Selecting a Preferred Alternative Category: 21. The team or the decision maker defines selection criteria based on the mission need. DOE requirements: No information found; DOE guidance: Consider various discriminators among alternatives, including: cost and schedule, funding and budget, technology and engineering, etc.(G 413.3-13 p. 9). GAO's assessment of DOE requirements alone: Do not meet. DOE's requirements do not contain any information about defining the selection criteria based on the mission need; GAO's assessment of DOE requirements combined with guidance: Partially meet. DOE's guidance adds a suggestion to consider a variety of discriminators when selecting a preferred alternative but does not mention that these should be based on the mission need. 22. The team or the decision maker weights the selection criteria to reflect the relative importance of each criterion. DOE requirements: No information found; DOE guidance: The selection criteria should be weighted. (G 413.3-13 p. 10). GAO's assessment of DOE requirements alone: Do not meet. DOE's requirements do not contain any information about weighting selection criteria to reflect the relative importance of each; GAO's assessment of DOE requirements combined with guidance: Fully meet. DOE's guidance adds an explicit suggestion to weight selection criteria. 23. The team or the decision maker compares alternatives using net present value,[D] if possible. DOE requirements: No information found; DOE guidance: Normally, analyses require comparing alternatives using net present value or annuities. (G 413.3-21 p. 69). GAO's assessment of DOE requirements alone: Do not meet. DOE's requirements do not contain any information about comparing alternatives using net present value; GAO's assessment of DOE requirements combined with guidance: Fully meet. DOE's guidance adds a suggestion to compare the alternatives using net present value or annuities. 24. An entity independent of the AOA process reviews the extent to which all best practices have been followed (for certain projects, additional independent reviews may be necessary at earlier stages of the process such as for reviewing the study plan or for reviewing the identification of viable alternatives). DOE requirements: Mission Need Statement Document Review (prior to CD- 0): DOE APM will review the Mission Need Statement Document and provide a recommendation to the decision maker for a project with a total project cost greater or equal to $100 million. (O 413.3B p. C- 17); Independent Cost Review (prior to CD-0) for major systems projects [with a total project cost greater than or equal to $750 million]: DOE APM will conduct this review to validate the basis of the rough-order-of-magnitude cost range and provide an assessment of whether the range reasonably bounds the alternatives to be analyzed in the next project phase. (O 413.3B p. C-17); Independent Cost Estimate/Independent Cost Review (prior to CD-1) for projects with a total project cost greater or equal to $100 million: This review validates the basis of the preliminary cost range for reasonableness and executability. It also includes a full accounting of life-cycle costs to support the alternative selection process and budgetary decision. (O413.3B pp. A-6, C-18); DOE guidance: Alternative Selection and Cost Range Review: This is an optional Independent Project Review conducted by the program office and not by a party independent of the project to assess whether the alternative selection process evaluates a range of appropriate attributes for each alternative including cost, maintainability, safety, technology requirements, risks, and regulatory requirements. (G 413.3-9 pp. 16-19). GAO's assessment of DOE requirements alone: Minimally meet. DOE's requirements specify reviews of the mission need and cost estimates but do not require a review of the entire AOA including identifying alternatives, estimates of benefit/effectiveness, risk and sensitivity analysis, and alternative selection; GAO's assessment of DOE requirements combined with guidance: Partially meet. DOE's guidance adds a suggestion for a review of additional aspects of the AOA but does not recommend a review of certain aspects of the AOA including identifying alternatives, estimates of benefit/effectiveness, and sensitivity analysis. Additionally, the guidance suggests the customer--not a party independent of the project--conduct the review. Sources: GAO analysis of DOE's Order 413.3B and its associated guides. GAO-15-37. [A] The five-point scoring system was as follows: "fully met" means that DOE's documentation demonstrated that DOE completely met the best practice; "substantially met" means that the DOE's documentation demonstrated that DOE met a large portion of the best practice; "partially met" means that DOE's documentation demonstrated that DOE met about half of the best practice; "minimally met" means that the DOE's documentation demonstrated that DOE met a small portion of the of the best practice; and "did not meet" means that DOE's documentation did not demonstrate that DOE met the best practice. [B] In this instance, the guidance is included in NNSA's Office of Acquisition and Project Management's Concept of Operations. This guidance applies only to NNSA and not the entire DOE. [C] The present value of an estimate reflects the time value of money, the concept that a dollar in the future is worth less than a dollar today because the dollar today can be invested and earn interest. The discount rate is the interest rate used to calculate the present value of an estimate. [D] The net present value is the difference between the discounted present value of benefits and the discounted present value of costs. [End of table] [End of section] Appendix V: NNSA's Analysis of Alternatives Conducted at the High Explosive, Science, Technology and Engineering Project Compared with Best Practices: Overall, the National Nuclear Security Administration's (NNSA) analysis of alternatives (AOA) conducted for the High Explosive Science, Technology and Engineering Project partially met best practices. The mission need for this project--to replace aging high explosive facilities at NNSA's Pantex site in Texas--was approved in November 2011. NNSA expects to complete the AOA process by obtaining CD-1 in December 2014. As of September 2014, NNSA estimated that the total project cost would range from $100 to $155 million at an 85 percent confidence level. NNSA expects to finalize the AOA process by the end of 2014. Table 14 below compares the AOA conducted at the High Explosive, Science, Technology and Engineering Project with AOA best practices. Table 14: Comparison of the AOA Conducted at the High Explosive, Science, Technology and Engineering Project with AOA Best Practices: Best practice category and score[A]: General principles; Substantially met; Best practice: 1. The customer defines the mission need and functional requirements without a predetermined solution; Detailed assessment[A]: Fully met. NNSA's documents describing the mission need and functional requirements did not mention any particular alternative and stated that they apply to any alternative. Best practice: 2. The customer defines functional requirements based on the mission need; Detailed assessment[A]: Substantially met. The functional requirements followed from the mission need, but this is not directly stated in project documentation. Best practice: 3. The customer provides the team conducting the analysis of alternatives (AOA) with enough time to complete the AOA process to ensure a robust and complete analysis; Detailed assessment[A]: Fully met. The AOA team completed the AOA over a time span of more than one year. Best practice: 4. The team includes members with diverse areas of expertise including, at a minimum, subject matter expertise, project management, cost estimating, and risk management; Detailed assessment[A]: Minimally met. Project documentation listed the organizations represented on the AOA team but did not indicate the specific individuals and their areas of expertise. Best practice: 5. The team creates a plan, including proposed methodologies, for identifying, analyzing, and selecting alternatives, before beginning the AOA process; Detailed assessment[A]: Substantially met. NNSA created a 17-step analysis plan that covered most of the areas of the AOA. Best practice: 6. The team documents all steps taken to identify, analyze, and select alternatives in a single document; Detailed assessment[A]: Partially met. NNSA listed in the AOA report many, but not all, of the steps it took as part of conducting the AOA, but NNSA did not describe many of them in detail. Best practice: 7. The team documents and justifies all assumptions and constraints used in the analysis; Detailed assessment[A]: Partially met. NNSA listed in project documentation the assumptions it made as part of conducting the AOA but did not provide justifications. Best practice: 8. The team conducts the analysis without a predetermined solution; Detailed assessment[A]: Partially met. NNSA listed one of the alternatives as an originally envisioned alternative to satisfy the mission need. NNSA did not end up selecting this alternative but also did not explain how the analysis was conducted without a predetermined solution. Best practice category and score[A]: Identifying alternatives: Fully met; Best practice: 9. The team identifies and considers a diverse range of alternatives to meet the mission need; Detailed assessment[A]: Substantially met. NNSA identified and considered nine different alternatives for meeting the mission need. Best practice: 10. The team describes alternatives in sufficient detail to allow for robust analysis; Detailed assessment[A]: Fully met. NNSA described the alternatives considered in detail, including providing descriptions of the specific characteristics of each alternative used to create cost estimates. Best practice: 11. The team includes one alternative representing the status quo to provide a basis of comparison among alternatives; Detailed assessment[A]: Fully met. NNSA considered a no-action alternative that would have maintained the status quo. Best practice: 12. The team screens the list of alternatives before proceeding, eliminates those that are not viable, and documents the reasons for eliminating any alternatives; Detailed assessment[A]: Substantially met. NNSA followed a screening process to eliminate four of the nine alternatives initially identified, and described the general reasons for eliminating them but did not provide specific reasons for the scores it gave each alternative as part of the screening process. Best practice category and score[A]: Analyzing Alternatives; Minimally met; Best practice: 13. The team develops a life-cycle cost estimate for each alternative, including all costs from inception of the project through design, development, deployment, operation, maintenance, and retirement; Detailed assessment[A]: Partially met. NNSA developed cost estimates for each alternative that were used for comparison purposes among alternatives, but not life-cycle cost estimates, because NNSA did not include costs related to the eventual retirement of the facility, which NNSA estimated as being the same across the alternatives. Best practice: 14. The team presents the life-cycle cost estimate for each alternative as a range or with a confidence interval, and not solely as a point estimate; Detailed assessment[A]: Fully met. NNSA included cost estimates for each alternative that were listed with an accuracy range of -15% to +50%. Best practice: 15. The team expresses the life-cycle cost estimate in present value terms and explains why it chose the specific discount rate used; Detailed assessment[A]: Minimally met. NNSA did not present the life- cycle cost estimates for each alternative in present value terms but did present a post-AOA life-cycle cost estimate for only the selected alternative in present values terms. Best practice: 16. The team uses a standard process to quantify the benefits/effectiveness of each alternative and documents this process; Detailed assessment[A]: Not met. NNSA did not mention quantifying benefits in any of the project documentation provided. Best practice: 17. The team quantifies the benefits/effectiveness resulting from each alternative over that alternative's full life cycle, if possible; Detailed assessment[A]: Minimally met. NNSA did not quantify benefits when analyzing alternatives. It included a qualitative discussion of some of the pros and cons of each alternative. Best practice: 18. The team explains how each measure of benefit/effectiveness supports the mission need; Detailed assessment[A]: Not met. NNSA did not state how or if it used any measures of benefit/effectiveness or how they would have supported the mission need. Best practice: 19. The team identifies and documents the significant risks and mitigation strategies for each alternative; Detailed assessment[A]: Partially met. NNSA developed a list assessing a number of risks for each alternative, and it briefly described mitigation strategies for some of the risks in some of the alternatives but did not include mitigation strategies for all risks. After conducting the AOA, NNSA developed a register of 76 risks and mitigation strategies for only the selected alternative. Best practice: 20. The team tests and documents the sensitivity of both the cost and benefit/effectiveness estimates for each alternative to risks and changes in key assumptions; Detailed assessment[A]: Not met. NNSA project documentation did not include any information about sensitivity analysis. Best practice category and score[A]: Selecting alternatives; Partially met; Best practice: 21. The team or the decision maker defines selection criteria based on the mission need; Detailed assessment[A]: Substantially met. NNSA developed 21 selection criteria that were based on the stated mission need but the linkages between the criteria and mission need were not clear in the documentation. Best practice: 22. The team or the decision maker weights the selection criteria to reflect the relative importance of each criterion; Detailed assessment[A]: Fully met. NNSA weighted the selection criteria using a five-point scale, with a 5 indicating most important and a 1 indicating least important. Best practice: 23. The team or the decision maker compares alternatives using net present value, if possible; Detailed assessment[A]: Not met. NNSA project documentation did not include any information about the use of net present value for comparison. Best practice: 24. An entity independent of the AOA process reviews the extent to which all best practices have been followed (for certain projects, additional independent reviews may be necessary at earlier stages of the process such as for reviewing the study plan or for reviewing the identification of viable alternatives); Detailed assessment[A]: Partially met. As of June 2014, NNSA was planning a review of select parts of the AOA, such as the viability and feasibility of the selected alternative and the cost and schedule estimates of the alternatives considered but not of other parts, such as the identification of alternatives, the selection criteria, or the benefit estimates for each alternative. Source: GAO analysis of NNSA information. GAO-15-37. [A] The score for each category represents the average scores of the best practices included in that category. The overall score for the AOA represents the average score of the four categories. An overall score or "fully met" or "substantially met" means that the AOA was reliable. The five-point scoring system that we used was as follows: "fully met" means that NNSA's documentation demonstrated that it completely met the best practice; "substantially met" means that NNSA's documentation demonstrated that it met a large portion of the best practice; "partially met" means that NNSA's documentation demonstrated that it met about half of the best practice; "minimally met" means that the NNSA's documentation demonstrated that it met a small portion of the best practice; and "did not meet" means that NNSA's documentation did not demonstrate that it met the best practice. [End of table] [End of section] Appendix VI: NNSA's 2013 Analysis of Alternatives Conducted at the Radioactive Liquid Waste Treatment Facility Compared with Best Practices: Overall, the National Nuclear Security Administration's (NNSA) analysis of alternatives (AOA) conducted for the Radioactive Liquid Waste Treatment Facility project partially met best practices. The mission need for this project--to replace the current, aging facility-- was approved in October 2004. NNSA approved an initial AOA for this project in 2006, and after substantial cost increases, conducted a second AOA (analyzed here) in 2013. NNSA currently estimates the project will cost between $168 million and $220 million. Table 15 below compares the AOA conducted at the Radioactive Liquid Waste Treatment Facility with AOA best practices. Table 15: Comparison of the AOA Conducted at the Radioactive Liquid Waste Treatment Facility with AOA Best Practices: Best practice category and score[A]: General principles; Substantially met; Best practice: 1. The customer defines the mission need and functional requirements without a predetermined solution; Detailed assessment[A]: Fully met. NNSA did not discuss alternatives while developing the documentation for the development of functional requirements and the mission need. Best practice: 2. The customer defines functional requirements based on the mission need; Detailed assessment[A]: Substantially met. NNSA's description of the functional requirements follows from the mission need, but this linkage was not clearly shown in project documentation. Best practice: 3. The customer provides the team conducting the analysis of alternatives (AOA) with enough time to complete the AOA process to ensure a robust and complete analysis; Detailed assessment[A]: Fully met. The AOA team completed the AOA study over a time span of more than one year. Best practice: 4. The team includes members with diverse areas of expertise including, at a minimum, subject matter expertise, project management, cost estimating, and risk management; Detailed assessment[A]: Partially met. NNSA's AOA report listed the individuals who participated in the AOA study, but it did not list their areas of expertise. Best practice: 5. The team creates a plan, including proposed methodologies, for identifying, analyzing, and selecting alternatives, before beginning the AOA process; Detailed assessment[A]: Partially met. NNSA's project documentation described some elements that had to be conducted during the AOA such as developing a life-cycle cost estimate. Best practice: 6. The team documents all steps taken to identify, analyze, and select alternatives in a single document; Detailed assessment[A]: Partially met. NNSA described the identification of alternatives, as well as risk and cost analysis in project documentation, but did not include a detailed narrative of how the analysis was conducted or described how it prescreened alternatives for viability, why other alternatives were not included in the analysis, or the development of selection criteria. Best practice: 7. The team documents and justifies all assumptions and constraints used in the analysis; Detailed assessment[A]: Partially met. NNSA listed the assumptions used in the analysis but did not provide justifications for these assumptions or discussed the constraints associated with most sections of the analysis. Best practice: 8. The team conducts the analysis without a predetermined solution; Detailed assessment[A]: Minimally met. NNSA considered only two alternatives in the analysis, and one of them was a previously selected alternative that had experienced significant cost increases, and so was unlikely to be selected. Best practice category and score[A]: Identifying alternatives; Substantially met; Best practice: 9. The team identifies and considers a diverse range of alternatives to meet the mission need; Detailed assessment[A]: Partially met. NNSA considered a wide range of partial capability options but combined them into only two final alternatives with similar designs. Best practice: 10. The team describes alternatives in sufficient detail to allow for robust analysis; Detailed assessment[A]: Substantially met. NNSA conducted detailed engineering studies for the two alternatives analyzed but defined the partial capability options initially considered in significantly less detail. Best practice: 11. The team includes one alternative representing the status quo to provide a basis of comparison among alternatives; Detailed assessment[A]: Fully met. NNSA included two partial capability options that represented the status quo in the list of partial capability options that it combined to arrive at the two final alternatives analyzed. Best practice: 12. The team screens the list of alternatives before proceeding, eliminates those that are not viable, and documents the reasons for eliminating any alternatives; Detailed assessment[A]: Partially met. NNSA conducted several screening steps on the list of partial capability options initially identified, and used these steps to eliminate non-viable partial capability options and combine the partial capability options into full alternatives. NNSA did not fully document some of these steps, such as how and why it consolidated some of these options into two alternatives. Best practice category and score[A]: Analyzing alternatives; Partially met; Best practice: 13. The team develops a life-cycle cost estimate for each alternative, including all costs from inception of the project through design, development, deployment, operation, maintenance, and retirement; Detailed assessment[A]: Fully met. NNSA developed full life-cycle cost estimates for each of the two alternatives analyzed. Best practice: 14. The team presents the life-cycle cost estimate for each alternative as a range or with a confidence interval, and not solely as a point estimate; Detailed assessment[A]: Minimally met. The life-cycle cost estimates that NNSA developed included design and construction costs as range estimates, but the remainder of the life-cycle cost elements, including the final estimates, were listed as point estimates. Best practice: 15. The team expresses the life-cycle cost estimate in present value terms and explains why it chose the specific discount rate used; Detailed assessment[A]: Fully met. NNSA used the discount rate specified by the Office of Management and Budget to create present values of the life-cycle cost estimates for each of the two alternatives. Best practice: 16. The team uses a standard process to quantify the benefits/effectiveness of each alternative and documents this process; Detailed assessment[A]: Not met. NNSA project documentation did not include any information about quantifying the benefits resulting from each alternative. Best practice: 17. The team quantifies the benefits/effectiveness resulting from each alternative over that alternative's full life cycle, if possible; Detailed assessment[A]: Not met. NNSA project documentation did not include any information about quantifying the benefits resulting from each alternative over the alternative's life cycle. Best practice: 18. The team explains how each measure of benefit/effectiveness supports the mission need; Detailed assessment[A]: Not met. NNSA project documentation did not include any information about measures of benefit. Best practice: 19. The team identifies and documents the significant risks and mitigation strategies for each alternative; Detailed assessment[A]: Fully met. NNSA identified and documented potential risks for each alternative, and assessed likelihood of occurrence, impact, and mitigation strategies for each risk. Best practice: 20. The team tests and documents the sensitivity of both the cost and benefit/effectiveness estimates for each alternative to risks and changes in key assumptions; Detailed assessment[A]: Partially met. NNSA conducted tests on elements of the life-cycle cost estimates to adjust them for risk but did not adjust the entire estimates, and did not include any information related to testing sensitivity of projected costs or benefits to changes in key assumptions. Best practice category and score[A]: Selecting a preferred alternative; Minimally met; Best practice: 21. The team or the decision maker defines selection criteria based on the mission need; Detailed assessment[A]: Partially met. NNSA included in the project documentation brief summaries of the selection criteria used but did not describe how these were based on the mission need. NNSA included only one of these selection criteria--the scope--in the mission need statement. Best practice: 22. The team or the decision maker weights the selection criteria to reflect the relative importance of each criterion; Detailed assessment[A]: Not met. NNSA did not include weighting selection criteria in project documentation. Best practice: 23. The team or the decision maker compares alternatives using net present value, if possible; Detailed assessment[A]: Not met. NNSA did not mention the use of net present value in project documentation. Best practice:24. An entity independent of the AOA process reviews the extent to which all best practices have been followed (for certain projects, additional independent reviews may be necessary at earlier stages of the process such as for reviewing the study plan or for reviewing the identification of viable alternatives); Detailed assessment[A]: Minimally met. NNSA conducted an independent review of the total project cost for the two alternatives analyzed, but NNSA did not conduct an independent analysis of the life-cycle cost estimate because the independent reviewers stated it was not possible for them to determine the reasonableness of the life-cycle cost estimates. NNSA also conducted another independent review of the selected alternatives. These two were not reviews of the entire AOA. Source: GAO analysis of NNSA information. GAO-15-37. [A] The score for each category represents the average scores of the best practices included in that category. The overall score for the AOA represents the average score of the four categories. An overall score or "fully met" or "substantially met" means that the AOA was reliable. The five-point scoring system that we used was as follows: "fully met" means that NNSA's documentation demonstrated that it completely met the best practice; "substantially met" means that NNSA's documentation demonstrated that it met a large portion of the best practice; "partially met" means that NNSA's documentation demonstrated that it met about half of the best practice; "minimally met" means that the NNSA's documentation demonstrated that it met a small portion of the best practice; and "did not meet" means that NNSA's documentation did not demonstrate that it met the best practice. [End of table] [End of section] Appendix VII: NNSA's 2012 Analysis of Alternatives Conducted at the Uranium Processing Facility Compared with Best Practices: Overall, the National Nuclear Security Administration's (NNSA) analysis of alternatives (AOA) conducted for the Uranium Processing Facility (UPF) project partially met best practices. The mission need for this project--to consolidate and replace existing enriched uranium processing capabilities at NNSA's Y-12 site--was approved in 2004. NNSA conducted a first AOA for this project in 2007, and after substantial cost increases, conducted a second AOA (analyzed here) in 2012. NNSA estimated the project will cost between $4.2 billion and $6.5 billion. Table 16 below compares the AOA conducted at the Uranium Processing Facility with AOA best practices. Table 16: Comparison of the AOA Conducted at the Uranium Processing Facility with AOA Best Practices: Best practice category and score[A]: General principles; Partially met; Best practice: 1. The customer defines the mission need and functional requirements without a predetermined solution; Detailed assessment[A]: Partially met. NNSA approved the mission need in 2004 and based the AOA analysis on that mission need, but the AOA report did not mention functional requirements. Best practice: 2. The customer defines functional requirements based on the mission need; Detailed assessment[A]: Substantially met. NNSA officials stated that NNSA has a functional requirements document; however, we were not able to review them. Best practice: 3. The customer provides the team conducting the analysis of alternatives (AOA) with enough time to complete the AOA process to ensure a robust and complete analysis; Detailed assessment[A]: Minimally met. NNSA asked for an AOA report at the end of November 2011 and needed the analysis to be conducted as quickly as possible to support the CD-1 reaffirmation decision. The AOA team conducted planning for the AOA in December 2011, and the bulk of the analysis was conducted over a three-day time frame in mid- January 2012. Part-time work continued until the final addendum was issued in May 2012. Best practice: 4. The team includes members with diverse areas of expertise including, at a minimum, subject matter expertise, project management, cost estimating, and risk management; Detailed assessment[A]: Partially met. NNSA's documentation did not provide enough detail to identify the AOA team members' areas of expertise. Best practice: 5. The team creates a plan, including proposed methodologies, for identifying, analyzing, and selecting alternatives, before beginning the AOA process; Detailed assessment[A]: Minimally met. NNSA did not create a formal study plan, and recorded only some planning activities. Best practice: 6. The team documents all steps taken to identify, analyze, and select alternatives in a single document; Detailed assessment[A]: Substantially met. NNSA documented in the AOA report most aspects of the analysis, including identification of alternatives, selection criteria and the basis of the cost estimates. Best practice: 7. The team documents and justifies all assumptions and constraints used in the analysis; Detailed assessment[A]: Partially met. NNSA listed some of the assumptions used in its analysis but did not justify them or identify the constraints associated with the analysis. Best practice: 8. The team conducts the analysis without a predetermined solution; Detailed assessment[A]:Minimally met. NNSA conducted this analysis with a fundamental assumption that a single facility would be selected and, therefore, it considered only alternatives that were related to the single-facility design selected by NNSA in the previous 2007 AOA. Best practice category and score[A]: Identifying alternatives; Substantially met; Best practice: 9. The team identifies and considers a diverse range of alternatives to meet the mission need; Detailed assessment[A]: Partially met. NNSA identified and considered six alternatives that were variations of the single-facility design, and did not consider any alternatives that were not variations on this idea. Best practice: 10. The team describes alternatives in sufficient detail to allow for robust analysis; Detailed assessment[A]: Fully met. NNSA described the alternatives in sufficient detail. NNSA described the scope, design, and schedule of each alternative considered. Best practice: 11. The team includes one alternative representing the status quo to provide a basis of comparison among alternatives; Detailed assessment[A]: Partially met. NNSA did not include a status quo alternative in the analysis, and NNSA stated in project documentation that the status quo was no longer considered viable, but NNSA did not demonstrate the lack of viability by explicitly comparing the status quo with the selection criteria. Best practice: 12. The team screens the list of alternatives before proceeding, eliminates those that are not viable, and documents the reasons for eliminating any alternatives; Detailed assessment[A]: Partially met. NNSA determined that one alternative was not viable before proceeding with the analysis but did not document the screening of all the alternatives against viability criteria. Best practice category and score[A]: Analyzing alternatives; Minimally met; Best practice: 13. The team develops a life-cycle cost estimate for each alternative, including all costs from inception of the project through design, development, deployment, operation, maintenance, and retirement; Detailed assessment[A]: Partially met. NNSA developed cost estimates for each alternative that were updates of a July 2010 total project cost estimate for a similar design, not an actual accounting of the anticipated life-cycle costs of each alternative. Best practice: 14. The team presents the life-cycle cost estimate for each alternative as a range or with a confidence interval, and not solely as a point estimate; Detailed assessment[A]: Not met. NNSA project documentation did not present the life-cycle cost estimates as ranges or confidence intervals. Best practice: 15. The team expresses the life-cycle cost estimate in present value terms and explains why it chose the specific discount rate used; Detailed assessment[A]: Fully met. NNSA expressed its life-cycle cost estimates in both constant 2011 dollar and present value terms. Best practice: 16. The team uses a standard process to quantify the benefits/effectiveness of each alternative and documents this process; Detailed assessment[A]: Minimally met. NNSA assessed the alternatives by comparing two alternatives at a time against each other for each selection criteria. This method did not attempt to assign a numerical value to the actual benefits that would result from each alternative. Best practice: 17. The team quantifies the benefits/effectiveness resulting from each alternative over that alternative's full life cycle, if possible; Detailed assessment[A]: Not met. NNSA project documentation did not present benefit estimates over the alternative's life cycle. Best practice: 18. The team explains how each measure of benefit/effectiveness supports the mission need; Detailed assessment[A]: Partially met. NNSA considers its selection criteria to be measures of benefit for each alternative, and these criteria follow from their approved mission need, but this linkage was not explicitly stated in the AOA report. Best practice: 19. The team identifies and documents the significant risks and mitigation strategies for each alternative; Detailed assessment[A]: Minimally met. NNSA listed categories of risk, such as cost, schedule, and technical risks in the description of each alternative but did not discuss specific risks or mitigation strategies. Best practice: 20. The team tests and documents the sensitivity of both the cost and benefit/effectiveness estimates for each alternative to risks and changes in key assumptions; Detailed assessment[A]: Minimally met. NNSA did not test the sensitivity of the cost estimates to changes in key assumptions. NNSA adjusted the design and construction elements of the cost estimates based on the risks included in the development of the 2010 cost estimate, but these risks were not documented in the AOA report. Best practice category and score[A]: Selecting a preferred alternative; Partially met; Best practice: 21. The team or the decision maker defines selection criteria based on the mission need; Detailed assessment[A]: Substantially met. NNSA developed six selection criteria that were derived from the approved mission need. However, NNSA updated the AOA to include an additional selection criterion after conducting the AOA, but NNSA provided no explanation whether this additional criterion was based on the mission need, or whether its introduction could have changed the result of the analysis. Best practice: 22. The team or the decision maker weights the selection criteria to reflect the relative importance of each criterion; Detailed assessment[A]: Fully met. NNSA weighted the six selection criteria to reflect their relative importance, with the highest weights given to the selection criteria of risk minimization and an alternative's ability to modernize and consolidate enriched uranium processing. Best practice: 23. The team or the decision maker compares alternatives using net present value, if possible; Detailed assessment[A]: Minimally met. NNSA compared alternatives using the discounted present value of their life-cycle costs but did not include the discounted present value of expected benefits. Best practice: 24. An entity independent of the AOA process reviews the extent to which all best practices have been followed (for certain projects, additional independent reviews may be necessary at earlier stages of the process such as for reviewing the study plan or for reviewing the identification of viable alternatives); Detailed assessment[A]: Minimally met. NNSA conducted an independent review after conducting the AOA of the technical readiness, but it was not a review of the AOA. Source: GAO analysis of NNSA information. GAO-15-37. [A] The score for each category represents the average scores of the best practices included in that category. The overall score for the AOA represents the average score of the four categories. An overall score or "fully met" or "substantially met" means that the AOA was reliable. The five-point scoring system that we used was as follows: "fully met" means that NNSA's documentation demonstrated that it completely met the best practice; "substantially met" means that NNSA's documentation demonstrated that it met a large portion of the best practice; "partially met" means that NNSA's documentation demonstrated that it met about half of the best practice; "minimally met" means that the NNSA's documentation demonstrated that it met a small portion of the of the best practice; and "did not meet" means that NNSA's documentation did not demonstrate that it met the best practice. [End of table] [End of section] Appendix VIII: Comments from the Department of Energy: Department of Energy: Washington, DC 20585: November 17, 2014: Mr. David Trimble: Director, Natural Resources and Environment: U.S. Government Accountability Office: Washington, D.C. 20548: Dear Mr. Trimble: This letter provides the U.S. Department of Energy's (DOE's) response to the recommendations contained in the U.S. Government Accountability Office (GAO) report entitled, DOE Project Management - Analysis of Alternatives Could be Improved by Incorporate Best Practices (GAO-15- 37, December 2014). We appreciate GAO's review of Departmental guidance and its identification of best practices in the area of Analysis of Alternatives. The recommendation is consistent with our commitment to continuous improvement in project management and the process of analyzing the appropriate alternative for capital asset acquisition. The best practices identified by GAO in the report will inform future Departmental policy updates. Enclosed is our response to the recommendation and technical comments on the report. Sincerely, Signed by: Ingrid Kolb: Director: Office of Management: Enclosure: Attachment 1: DOE Responses to Recommendations: The Department has the following responses regarding the draft report's recommendations. Recommendation: "To minimize the risk of developing unreliable AOAs and incurring major cost increases and schedule delays on projects, we recommend that the Secretary of Energy direct DOE's Office of Acquisition and Project Management to update its project management order requirements to incorporate all best practices for conducting an AOA." Page 33 of the GAO Report. Response: The Departmental order for project management (DOE 0 413.3B) will be assessed for revision following the issuance of the revision to DOE-STD-1189, Integration of safety into the Design Process, currently scheduled for November 2016. As the Department prepares for the Order's revision, a revision log has been established. GAO's recommendation has been logged for action, and the best practices for conducting an AOA, as identified by the GAO, will be considered when the policy updated. In the interim, the Department has strengthened guidance in the area of AOA including issuing a Life Cycle Cost Handbook that provides guidance for Life Cycle Cost Estimation and Life Cycle Cost Analysis, key inputs into the AOA process. Additionally, in FY 2015, DOE's Office of Acquisition and Project Management will develop and issue an AOA Handbook that will reflect the best practices for AOAs. [End of section] Appendix IX: GAO Contact and Staff Acknowledgments: GAO Contact: David C. Trimble, (202) 512-3841 or trimbled@gao.gov: Staff Acknowledgments: In addition to the individual named above, Daniel Feehan, Assistant Director; Bethany Benitez; Antoinette Capaccio; Jennifer Echard; Alex Fedell; Cristian Ion; Kirsten Lauber; Jennifer Leotta; Cynthia Norris; Karen Richey; Dan C. Royer; and Kiki Theodoropoulos made key contributions to this report. [End of section] Footnotes: [1] DOE defines a program as an organized set of activities directed toward a common purpose or goal in support of an assigned mission area. DOE defines a capital asset as land, structures, equipment, and intellectual property, which are used by the federal government and have an estimated useful life of 2 years or more. [2] GAO, Defense Acquisitions: Many Analyses of Alternatives Have Not Provided a Robust Assessment of Weapon System Options, [hyperlink, http://www.gao.gov/products/GAO-09-665] (Washington, D.C.: Sept. 24, 2009). [3] GAO, Modernizing the Nuclear Security Enterprise: Observations on NNSA's Options for Meeting Its Plutonium Research Needs, [hyperlink, http://www.gao.gov/products/GAO-13-533] (Washington, D.C.: Sept. 11, 2013). [4] GAO, Plutonium Disposition Program: DOE Needs to Analyze the Root Causes of Cost Increases and Develop Better Cost Estimates, [hyperlink, http://www.gao.gov/products/GAO-14-231] (Washington, D.C.: Feb. 13, 2014). [5] [hyperlink, http://www.gao.gov/products/GAO-14-231]. [6] GAO, Nuclear Weapons: Some Actions Have Been Taken to Address Challenges with the Uranium Processing Facility Design, [hyperlink, http://www.gao.gov/products/GAO-15-126] (Washington, D.C.: Oct. 10, 2014). [7] S. Rep. No. 113-44, at 258-259 (2013); Pub. L. No. 113-66, 127 Stat. 672 (codified in scattered sections of 10, 50, and 37 U.S.C.). [8] We developed this list of experts through past work. [9] GAO, GAO Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Capital Program Costs, [hyperlink, http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: March 2009). [10] These entities are the Association for the Advancement of Cost Engineering International, Department of Defense, Department of Homeland Security, National Aeronautics and Space Administration, Office of Management and Budget, United States Air Force, and a private entity that specializes in acquisition management. [11] The five-point scoring system was as follows: "fully met" means that DOE's documentation demonstrated that DOE completely met the best practice; "substantially met" means that the DOE's documentation demonstrated that DOE met a large portion of the best practice; "partially met" means that DOE's documentation demonstrated that DOE met about half of the best practice; "minimally met" means that the DOE's documentation demonstrated that DOE met a small portion of the best practice; and "did not meet" means that DOE's documentation did not demonstrate that DOE met the best practice. [12] For us to consider the AOA process reliable, the entire AOA process had to receive an average score of "fully met" or "substantially met," and each individual category--(1) general principles, (2) identifying alternatives, (3) analyzing alternatives, and (4) selecting a preferred alternative--had to receive an average score of "fully met" or "substantially met" to ensure that the agency adequately and consistently performed all parts of the AOA process. [13] We chose the High Explosive Science, Technology and Engineering Project because NNSA was scheduled to complete the AOA process for this project by the end of September 2014 at the time of our project selection. We chose the Radioactive Liquid Waste Treatment Facility and Uranium Processing Facility because NNSA completed its most recent reassessment of alternatives for these projects in September 2013 and June 2012, respectively. [14] At DOE and NNSA, peer reviews are performed by peers (with relevant experience and expertise) independent of the project, to evaluate technical, managerial, cost, scope, and other aspects of the project, as appropriate. [15] Oak Ridge National Laboratory, Final Report of the Committee to Recommend Alternatives to the Uranium Processing Facility Plan in Meeting the Nation's Enriched Uranium Strategy (Oak Ridge, TN: Apr. 15, 2014). [16] DOE, Program and Project Management for the Acquisition of Capital Assets, DOE Order 413.3B (Washington, D.C.: Nov. 29, 2010). This order superseded DOE Order 413.3A, Program and Project Management for the Acquisition of Capital Assets (Washington, D.C.: July 28, 2006). [17] Guidance for conducting an AOA is included in DOE G 413.3-17: Mission Need Statement Guide (Washington, D.C.: June 20, 2008); DOE G 413.3-13: U.S. Department of Energy Acquisition Strategy Guide for Capital Asset Projects (Washington, D.C.: July 22, 2008); DOE G 413.3- 9: U.S. Department of Energy Project Review Guide for Capital Asset Projects (Washington, D.C.: Sept. 23, 2008); DOE G 413.3-1: Managing Design and Construction Using Systems Engineering for Use with DOE Order 413.3A (Washington, D.C.: Sept. 23, 2008); DOE G 413.3-21: Cost Estimating Guide (Washington, D.C.: May 9, 2011); DOE G 413.3-5A: Performance Baseline Guide (Washington, D.C.: Sept. 23, 2011); and DOE G 413.3-18A: Integrated Project Team: Guide for Formation and Implementation (Washington, D.C.: Feb. 3, 2012). [18] Order 413.3B defines life-cycle cost as the sum total of all direct, indirect, recurring, nonrecurring, and other related costs incurred or estimated to be incurred in the planning, design, development, procurement, production, operations and maintenance, support, recapitalization, and final disposition of real property over its anticipated life span for every aspect of the program, regardless of funding source. [19] DOE defines net present value as the difference between the discounted present values of benefits and costs. DOE, Cost Estimating Guide, DOE G 413.3-21 (May 9, 2011), at Appendix B. [20] Pub. L. No. 91-190, § 102, 83 Stat. 852, 853 (codified as amended at 42 U.S.C. § 4332). [21] Pub. L. No. 113-66, § 3112, 127 Stat. 672, 1050 (codified at 50 U.S.C. § 2411). [22] At the time of our project selection, NNSA was scheduled to complete the AOA process for this project by the end of September 2014. According to NNSA officials, this date was subsequently extended to December 2014 because the project is a new start and cannot be initiated until passage of the fiscal year 2015 budget. [23] DOE's Order 413.3B defines the total project cost as all costs between CD-0 and CD-4, including, among other things, costs needed to start operations. The total project cost is not the same as a life- cycle cost, because it does not include operational costs after the facility is constructed and other future costs, such as for decontamination and decommissioning. [24] Enriched uranium is uranium that has been processed to increase the concentration of uranium fissile isotopes for use in nuclear applications. [25] This independent cost estimate was requested under the National Defense Authorization Act of 2013. The $9.8 billion is a point estimate and not the full cost range. The top end of the Department of Defense's Office of Cost Assessment and Program Evaluation cost range estimate was at least $10.5 billion with the potential of higher cost depending on constrained or unconstrained funding profile. [26] [hyperlink, http://www.gao.gov/products/GAO-15-126]. [27] Oak Ridge National Laboratory, Final Report of the Committee to Recommend Alternatives to the Uranium Processing Facility Plan in Meeting the Nation's Enriched Uranium Strategy (Oak Ridge, TN: Apr. 15, 2014). [28] Because the NNSA Administrator asked for an evaluation of alternatives, we compared this peer review with AOA best practices. Based on our analysis, we concurred with NNSA officials that this peer review was not an AOA because it reflected only one characteristic of an AOA--being conducted by a team with diverse expertise. [29] As noted earlier and discussed in appendix I, if the overall score was "partially meet" or lower, or if any category received a score of "partially meet" or lower, we concluded that DOE's AOA process did not conform to best practices and therefore could not be considered reliable. [30] GAO, Managing Critical Isotopes: Weaknesses in DOE's Management of Helium-3 Delayed the Federal Response to a Critical Supply Shortage, [hyperlink, http://www.gao.gov/products/GAO-11-472] (Washington, D.C.: May 12, 2011). [31] All three AOAs received an overall score of "partially met." As noted earlier, and discussed in appendix I, if the final score was "partially met" or lower, or if any category score was "partially met" or lower, we concluded that the AOA process did not conform to best practices, and therefore could not be considered reliable. [32] The Pantex site, along with the Kansas City Plant in Missouri and the Y-12 National Security Complex in Tennessee, is one of NNSA's three nuclear weapons production sites. [33] We developed this list of experts through past work. [34] [hyperlink, http://www.gao.gov/products/GAO-09-3SP]. [35] The five-point scoring system was as follows: "fully met" means that DOE's documentation demonstrated that DOE completely met the best practice; "substantially met" means that the DOE's documentation demonstrated that DOE met a large portion of the best practice; "partially met" means that DOE's documentation demonstrated that DOE met about half of the best practice; "minimally met" means that the DOE's documentation demonstrated that DOE met a small portion of the of the best practice; and "did not meet" means that DOE's documentation did not demonstrate that DOE met the best practice. [36] We initially identified five NNSA projects that were listed in the project status report that met the selection criteria, but after further review, we determined that one project's cost estimate was under the $50 million threshold for which Order 413.3B applies, and another project never received CD-1 approval, and NNSA was in the process of canceling it. [37] DOE, Program and Project Management for the Acquisition of Capital Assets, DOE Order 413.3B (Washington, D.C.: Nov. 29, 2010). [38] Order 413.3B defines life-cycle cost as the sum total of all direct, indirect, recurring, nonrecurring, and other related costs incurred or estimated to be incurred in the planning, design, development, procurement, production, operations and maintenance, support, recapitalization, and final disposition of real property over its anticipated life span for every aspect of the program, regardless of funding source. [39] Pub. L. No. 91-190, § 102(2)(C), 83 Stat. 852, 853 (codified at 42 U.S.C. § 4332(2)(C)). Under NEPA, agencies evaluate the likely environmental effects of projects they are proposing, and reasonable alternatives, using an environmental assessment or, if the projects likely would significantly affect the environment, a more detailed environmental impact statement. [40] DOE G 413.3-17: Mission Need Statement Guide (Washington, D.C.: June 20, 2008). [41] DOE G 413.3-1: Managing Design and Construction Using Systems Engineering for Use with DOE Order 413.3A (Washington, D.C.: Sept. 23, 2008). [42] DOE G 413.3-5A: U.S. Department of Energy Performance Baseline Guide (Washington, D.C.: Sept. 23, 2011). [43] DOE G 413.3-13: U.S. Department of Energy Acquisition Strategy Guide for Capital Asset Projects (Washington, D.C.: July 22, 2008). [44] DOE G 413.3-18A: Integrated Project Team: Guide for Formation and Implementation (Washington, D.C.: Feb. 3, 2012). [45] DOE G 413.3-21: Cost Estimating Guide (Washington, D.C.: May 9, 2011). [46] DOE defines net present value as the difference between the discounted present values of benefits and costs. DOE, Cost Estimating Guide, DOE G 413.3-21 (May 9, 2011), at Appendix B. [47] DOE G 413.3-9: U.S. Department of Energy Project Review Guide for Capital Asset Projects (Washington, D.C.: Sept. 23, 2008). [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's website [hyperlink, http://www.gao.gov]. Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Phone: The price of each GAO publication reflects GAO's actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO's website, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO: Connect with GAO on facebook, flickr, twitter, and YouTube. Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts. Visit GAO on the web at [hyperlink, http://www.gao.gov]. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; E-mail: fraudnet@gao.gov; Automated answering system: (800) 424-5454 or (202) 512-7470. Congressional Relations: Katherine Siggerud, Managing Director, siggerudk@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, DC 20548. Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, DC 20548. [End of document]