This is the accessible text file for GAO report number GAO-12-347 entitled 'Nuclear Safety: DOE Needs to Determine the Costs and Benefits of Its Safety Reform Effort' which was released on May 22, 2012. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. United States Government Accountability Office: GAO: Report to Congressional Requesters: April 2012: Nuclear Safety: DOE Needs to Determine the Costs and Benefits of Its Safety Reform Effort: GAO-12-347: GAO Highlights: Highlights of GAO-12-347, a report to congressional requesters. Why GAO Did This Study: DOE carries out many of the nation’s most critical missions, including stewardship of the nation’s nuclear weapons stockpile and the environmental remediation of radioactive and hazardous legacy waste left over from the Cold War. DOE uses a system of regulations and internal directives that lay out requirements and guidance for ensuring the safety of staff and contractors, the public, and the environment. Over the past 10 years, GAO and others have repeatedly made recommendations for DOE to improve safety performance. In March 2010, DOE announced a reform effort to revise safety-related directives to increase productivity and reduce costs while maintaining safety. This report examines (1) how DOE revised safety directives under its reform effort, (2) the costs of the reform effort and the benefits DOE hoped to achieve, and (3) the extent to which its reform effort addresses safety concerns GAO and others have identified. GAO reviewed relevant DOE reform effort documents, visited selected DOE sites to interview site office and contractor officials, and analyzed past GAO and other reports on DOE’s safety problems. What GAO Found: Under its safety reform effort, the Department of Energy (DOE) reduced the number of safety directives by eliminating or combining requirements it determined were unclear, duplicative, or too prescriptive and by encouraging the use of industry standards. DOE reduced the number of its safety directives from 80 to 42, and for some of the directives DOE retained, it made extensive revisions. For example, DOE deleted requirements from its quality assurance directive addressing a corrective action program because another safety directive adequately covered these requirements. DOE obtained comments on its proposed revisions from DOE and contractor staff and from the Defense Nuclear Facilities Safety Board (Safety Board). The benefits of DOE’s reform effort are not clear. DOE intended to enhance productivity and reduce costs while maintaining safety, but DOE did not determine how the original requirements contained in safety directives impaired productivity or added costs before undertaking the reform effort. Moreover, DOE did not assess whether the cost to implement the revised directives would exceed the benefits, but officials said they had launched an initial study to determine, among other things, the costs associated with implementing selected safety requirements. DOE also did not develop performance measures in order to assess how the reform effort will lead to improved productivity or lower costs while maintaining safety. Instead, DOE is measuring success by using output-oriented measures, such as the number of directives eliminated, and not outcome measures, such as specific productivity improvements or cost savings. In the absence of clear measures linking the reform effort to productivity and safety improvements, DOE is not well positioned to know that its reform effort will achieve the intended benefits. DOE’s reform effort did not fully address safety concerns GAO and others have identified in three key areas: (1) quality assurance, (2) safety culture, and (3) federal oversight. Regarding quality assurance, DOE strengthened its quality assurance directive by clarifying that contractors must follow specific industry quality assurance standards, but quality assurance problems persist. For example, DOE proposed a nearly $250,000 fine against a contractor in July 2011 after identifying quality assurance problems in an incident where a worker punctured his hand with a sharp object contaminated with plutonium. With regard to safety culture, DOE revised its Integrated Safety Management directives to attempt to strengthen the safety culture at its sites, but DOE removed requirements for contractors to follow the directives because contractors already had to comply with safety management requirements in federal regulation. Safety Board officials raised concerns that the requirements in federal regulation are less detailed and, as a result, contractors may not implement safety practices as rigorously as if they were subject to the more specific requirements in DOE’s directives. Finally, regarding federal oversight, DOE revised its approach to place greater emphasis on having its independent oversight staff review safety design documents before facilities are constructed, rather than after they are built. Other changes, however, such as requiring oversight staff to coordinate their assessment activities with DOE site office and contractor staff, raise concerns about the oversight staff’s ability to provide a critical review of safety at DOE’s sites that is independent from DOE site office and contractor staff. What GAO Recommends: GAO recommends that DOE analyze the costs and benefits of its safety reform effort and identify how the effort will help address safety concerns. DOE agreed with the recommendations but commented that it had significant concerns about the accuracy of the report’s findings and conclusions. GAO stands by its findings and conclusions for the reasons discussed in the report. To view the full product, click on [hyperlink, http://www.gao.gov/products/GAO-12-347]. For more information, contact Gene Aloise at (202) 512-3841 or aloisee@gao.gov. [End of section] Contents: Letter: Background: DOE's Reform Effort Streamlined Directives, but Views on the Revisions Were Not Sought from the Public or Federal Agencies Other than the Safety Board: Benefits of DOE's Reform Effort Are Unclear: DOE's Reform Effort Did Not Fully Address Safety Concerns We and Others Have Identified: Conclusions: Recommendations for Executive Action: Agency Comments and Our Evaluation: Appendix I: Scope And Methodology: Appendix II: Comments From The Department Of Energy: Appendix III: GAO Contact And Staff Acknowledgments: Related GAO Products: Table: Table 1: Four Key Practices of Management Improvement Initiatives: Abbreviations: DOE: Department of Energy: NNSA: National Nuclear Security Administration: NRC: Nuclear Regulatory Commission: OSHA: Occupational Health and Safety Administration: [End of section] United States Government Accountability Office: Washington, DC 20548: April 20, 2012: Congressional Requesters: The Department of Energy (DOE) carries out many of the nation's most critical missions, including stewardship of the nation's nuclear weapons stockpile and the environmental remediation of radioactive and hazardous legacy waste left over from the Cold War era. Work carried out at DOE's sites across the country involves, among other things, (1) research on dangerous nuclear materials, such as plutonium, and (2) the handling and disposal of radioactive and hazardous waste that, if not handled safely, could cause nuclear accidents or expose the public and the environment to heavy doses of radiation. DOE's sites also conduct a wide range of other activities, including construction and routine maintenance and operation of equipment and facilities that also run the risk of accidents, such as those involving heavy machinery or electrical mishaps. The consequences of such accidents could be less severe than those involving nuclear materials, but they could also lead to long-term illnesses, injuries, or even deaths among workers. DOE relies on contractors and subcontractors to perform day-to-day operations at most of its nuclear and nonnuclear facilities located at approximately 40 sites across the country.[Footnote 1] To help ensure safety at these sites, the department requires its contractors to adhere to federal safety laws; departmental regulations;[Footnote 2] and DOE requirements that are provided in the department's system of directives--including DOE's policies, orders, guides, and manuals. Among other things, DOE directives address the safe operation of nuclear facilities as well as worker safety. The department incorporates directives into contracts and holds contractors accountable for meeting them. DOE's program offices--both at headquarters and its sites--are responsible for directly overseeing the contractors to ensure safety requirements are being met. The department's Office of Health, Safety, and Security independently oversees and enforces, among other things, nuclear and worker safety at DOE's sites. Over the years, federal officials, Congress, and others have expressed concerns about safety problems at DOE's sites. For example, the Defense Nuclear Facilities Safety Board (Safety Board)--an independent executive branch agency created by Congress to independently assess safety conditions and operations at defense nuclear facilities at DOE's sites--has held 11 public hearings over the past 10 years to address concerns about DOE's safety practices. Additionally, in January 2005, the director of DOE's Lawrence Livermore National Laboratory suspended all programmatic work at the laboratory's plutonium-handling facility, largely because of numerous unresolved safety issues and a failure to address these issues adequately. Moreover, in October 2007, we reported that nearly 60 serious accidents or near misses had occurred at DOE's nuclear weapons laboratories since 2000.[Footnote 3] For instance, at DOE's Los Alamos National Laboratory in July 2004, an undergraduate student who was not wearing required eye protection was partially blinded in a laser accident. Accidents such as this one and nuclear safety violations focused attention on needed improvements in safety performance across DOE's sites. In March 2010, the Deputy Secretary of Energy announced a new effort-- the 2010 Safety and Security Reform Plan (reform effort)--to revise DOE's safety and security directives and reform its oversight approach to "provide contractors with the flexibility to tailor and implement safety and security programs without excessive federal oversight or overly prescriptive departmental requirements" while "maintaining the highest standards of safe and secure operations" at DOE's sites. In the memorandum announcing this effort, the Deputy Secretary noted that burdensome safety requirements were affecting the productivity of work at DOE's sites and that reducing this burden on contractors would lead to measurable productivity improvements. In this context, you asked us to evaluate the department's current reform effort. Although DOE's 2010 Safety and Security Reform effort addresses both safety and security directives, this report focuses on revisions to DOE's safety-related directives, which constitute 80 of the 107 directives included in the reform effort. Our objectives were to examine (1) how DOE revised safety directives under its reform effort, (2) the costs of the reform effort and the benefits DOE hoped to achieve, and (3) the extent to which DOE's reform effort addresses safety concerns we and others have identified. To conduct this work, we reviewed documents on DOE's reform effort, including the overall strategy and implementation plan for the reform effort. DOE's reform effort addresses directives both in the safety and security areas. This report focuses on safety, including nuclear safety, worker safety, quality assurance, oversight, and operating experience.[Footnote 4] We interviewed DOE officials at DOE headquarters in the Office of Health, Safety, and Security and in the National Nuclear Security Administration (NNSA), a semiautonomous agency within the department that is responsible for, among other things, the management and security of the nation's nuclear weapons programs. To gain an understanding of the potential impact of the reform effort on DOE's sites, we visited sites that were in various stages of implementing revised safety requirements. These sites were the Kansas City Plant in Missouri, the Nevada National Security Site in Nevada, the Lawrence Livermore National Laboratory in California, and the Hanford Site in Washington state.[Footnote 5] We interviewed both DOE and contractor officials at these sites. We also reviewed documents relating to safety performance measures,[Footnote 6] efforts to address safety problems, and oversight. We interviewed officials at other governmental agencies, including the Nuclear Regulatory Commission (NRC), which licenses and regulates commercial nuclear power plants, and the Department of Labor's Office of Occupational Health and Safety (OSHA), which generally regulates worker safety and commercial industrial facilities. We also interviewed officials with the Safety Board to gain an understanding of the process of changing regulatory safety requirements. In addition, we interviewed officials from the American Society of Industrial Standards, an industry standards-making organization, to determine how industry standards are developed. To gain an understanding of DOE's long-standing safety problems, we reviewed 51 reports from the past 10 years by GAO, DOE's Office of Inspector General, and the Safety Board. Appendix I presents a detailed description of our scope and methodology. We conducted this performance audit from February 2011 through April 2012 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Background: DOE generally regulates the safety of its own nuclear facilities and operations at its sites. In contrast, NRC generally regulates commercial nuclear facilities, and OSHA generally regulates worker safety at commercial industrial facilities.[Footnote 7] DOE manages the safety of its facilities and operations at its sites through federal safety laws; departmental regulations; and a system of internal directives--that is, DOE policies, orders, guides, and manuals. DOE policies communicate top-level expectations for safety issues, such as conducting safe operations at high-hazard nuclear facilities. DOE orders are used as the primary directives for communicating requirements. DOE guides detail acceptable, but not mandatory, methods for implementing requirements contained in DOE orders, while manuals may contain both requirements and guidance. [Footnote 8] In total, DOE's directives lay out requirements to be followed and guidance for implementing requirements to help ensure the safety of workers, the public, and the environment at DOE's sites. To ensure adherence to the requirements in DOE's directives, the department oversees its own operations. This oversight has three main components. First, DOE's program offices--such as DOE's Office of Environmental Management[Footnote 9] and NNSA--have primary responsibility to safely carry out their program missions. Second, the site offices oversee the contractors that manage and operate DOE's sites. DOE's contractors are responsible for safely designing, constructing, and operating the facilities at these sites that they use to carry out their work. Third, DOE's Office of Health, Safety, and Security is responsible for, among other things, the development of safety policy and conducting independent oversight of compliance with DOE's safety regulations and directives, and enforcement activities. Within DOE's Office of Health, Safety, and Security, the Office of Independent Oversight conducts periodic appraisals of the environment, safety, and health programs at DOE's sites to determine if DOE officials and contractors are complying with DOE's safety regulations and directives.[Footnote 10] DOE also receives safety assessments and recommendations from other organizations, most prominently the Safety Board. The Safety Board provides independent safety oversight of DOE's defense nuclear facilities, which are located at 13 DOE sites across the United States. The Safety Board has broad oversight responsibilities at defense nuclear facilities located at DOE's sites and seeks to use informal interactions with DOE to resolve safety issues that are of concern to the Safety Board. However, the Safety Board also uses formal communications, such as recommendations, to address more substantial safety issues at a facility or site as well as broader safety issues across DOE's sites. Other organizations that provide assessments to DOE on the management of its sites include DOE's Office of Inspector General, the National Academy of Sciences, and GAO. In March 2010, the Deputy Secretary of Energy issued a memorandum that announced a reform effort aimed at revising DOE's 80 safety-related directives and the department's approach to regulating and overseeing safety at its sites.[Footnote 11] In announcing the reform effort, the Deputy Secretary noted that it was aimed at (1) providing increased recognition of program management's responsibility for safety, (2) reviewing opportunities to streamline safety requirements, and (3) eliminating safety directives that do not add value. The reform effort was intended to provide contractors with the flexibility to tailor and implement safety programs without excessive federal oversight or overly prescriptive departmental requirements, while also maintaining safety at DOE's sites. Key elements of the reform effort included reviewing the department's safety directives to (1) clarify and streamline wording in them; (2) increase the use of external (non-DOE) standards, such as industry standards and those from other regulatory agencies;[Footnote 12] (3) focus oversight activities on high-risk activities and contractors with poor safety records; and (4) provide more reliance on contractor systems that reliably measure the effectiveness of their safety management systems and ability to implement effective corrective actions. DOE's reform effort mirrors another reform effort underway at NNSA. Known as "governance transformation," NNSA's effort involves revising the agency's business model to, among other things: * more clearly establish roles of federal oversight staff and contractors; * place more reliance on contractor's self-oversight through its contractor assurance systems to ensure such things as safety performance; * allow contractors to tailor safety and other requirements to their work without overly prescriptive DOE requirements; and: * promote the use of national and international standards as appropriate, rather than departmentally unique requirements. * NNSA's Kansas City Plant has completed implementation of this new model, and other NNSA sites--such as the Nevada National Security Site in Nevada and Y-12 National Security Complex at Oak Ridge, Tennessee-- are currently implementing the changes. DOE's Reform Effort Streamlined Directives, But Views On The Revisions Were Not Sought From The Public Or Federal Agencies Other Than The Safety Board: Under its reform effort, DOE reduced the number of safety directives by eliminating or combining, among other things, requirements it determined were unclear, duplicative, or too prescriptive; allowing contractors additional flexibility; and encouraging the use of industry standards. In revising its directives, DOE followed a systematic process that included obtaining the views of staff from within DOE and its contractor community and from the Safety Board, but it did not seek input from the public or other federal agencies. DOE Revised Its Safety Directives To Reduce Duplication, Allow Contractors Additional Flexibility,And Encourage The Use Of Industry Standards: In total, under its reform effort, DOE reduced the number of its safety directives from 80 to 42. When deciding whether to keep, revise, or eliminate a safety directive, DOE identified a justification for the directive--that is, why the directive was first established and what it should accomplish. For example, DOE's safety directive requiring the development of a departmentwide system to collect, communicate, and evaluate lessons learned about operating nuclear and other facilities was established in 2006 in response to, among other things, the 2003 explosion of the Space Shuttle Columbia where inadequate collecting and sharing of operating lessons learned was found to have contributed to the accident.[Footnote 13] According to DOE documents, the department determined that it would not eliminate this directive because it helped establish good work practices and enabled the sharing of information across DOE's sites. In contrast, DOE eliminated its policy directive on recruiting and training federal employees that ensured the employees had sufficient technical knowledge of safety issues. This directive was established in 1998 to provide departmental expectations for maintaining sufficient federal technical safety capability. In undertaking its reform effort, DOE concluded this policy had served its purpose and was no longer needed as a stand-alone directive because a companion directive implementing the requirements for recruiting and training staff to safely carry out work was sufficient.[Footnote 14] If DOE decided to retain rather than eliminate an entire safety directive, DOE officials undertook a line-by-line examination of the specific requirements contained within the directive. This examination was intended to determine whether revisions to the requirements were necessary to make them more easily understandable and to eliminate confusion arising from having similar requirements in multiple directives. As mentioned, the examination included determining why each requirement was initially established. For example, some specific requirements in a DOE directive may have been the result of recommendations from the Safety Board or the result of a long-standing DOE or industry practice. Establishing the reason why a requirement was initially created assisted the DOE officials who were undertaking the line-by-line examination to determine whether revisions were necessary. Within the safety directives that DOE retained, DOE eliminated some requirements that department officials determined were unclear, duplicative, or too prescriptive. In particular, DOE removed requirements that it determined already existed in other safety directives or regulations. In its quality assurance directive, for example, DOE deleted 61 of 245 requirements for a corrective action program because it determined that the requirements were adequately addressed in revisions to other directives on oversight practices.[Footnote 15] DOE also eliminated requirements that it identified as being too prescriptive. For example, in revising its oversight directives, DOE modified requirements that defined specific components that a contractor's oversight system must have. DOE revised the requirements to allow contractors the flexibility to develop their own oversight systems, as long as those oversight systems could generally provide assurance that work was being performed safely. [Footnote 16] DOE also revised directives to promote greater use of industry standards. For example, DOE modified its quality assurance directive to require its contractors to follow industry standards--or its equivalent--for nuclear facility quality assurance that were developed by the American Society of Mechanical Engineers. In addition, DOE revised directives to refocus federal oversight on the performance of high-risk activities and placed greater responsibility on contractors to oversee lower-risk activities. For example, DOE revised a directive detailing how accident investigations should be carried out by eliminating one of two categories under which a federally led accident investigation would take place. Under the revision, a federal investigation would be initiated using some, but not all, of the criteria previously established under the two categories. Accidents not meeting these criteria, including some that would have initiated federal accident investigations, are the responsibility of the contractors themselves to investigate.[Footnote 17] DOE Obtained Views On Proposed Changes From DOE And Contractor Staff And From The Safety Board, But Did Not Seek The Views Of The Public Or Other Federal Agencies: As part of its process to revise its directives, DOE obtained comments from federal and contractor officials on proposed revisions. For example, DOE used "red teams"--groups of management-level staff who reviewed each of the proposed revisions before a revised directive received departmentwide review. DOE's red teams reviewed proposed revisions to directives to determine (1) whether the original requirements provided adequate safety protection, (2) the impacts of the proposed revisions and whether unintended consequences were analyzed, and (3) whether the original requirements provided sufficient flexibility in implementation without increasing safety risks or whether further revisions were needed. The red teams were intended to provide an independent review of proposed revisions to ensure they met the objectives of the reform effort. DOE generally obtained comments on proposed revisions to its directives from affected DOE federal and contractor staff by using its departmentwide review system. This review system--called RevCom--is an online system that disseminates proposed revisions to DOE federal and contractor staff for their review and comment on the revisions' content, relevance, applicability, accuracy, impact, and implementation cost. DOE also obtained comments from the Safety Board, which is responsible for overseeing nuclear safety at DOE and for independently reviewing proposed revisions to relevant safety directives. DOE officials told us they generally discussed the reform process with trade unions whose members are employed at DOE's sites. However, DOE did not provide other stakeholders or the public with an opportunity to comment on proposed revisions to the directives, according to DOE officials we spoke with. Other external stakeholders that could have provided input included agencies that have similar safety expertise, such as NRC or OSHA. NRC and OSHA officials told us that they are required by law to obtain the public's views on any changes in their safety regulations, which generally contain requirements for private employers. Although DOE is required to obtain comments on changes to its safety regulations, DOE is not required to obtain the public's views on its internal directives, many of which contain requirements for DOE's contractors. NRC and OSHA officials told us that obtaining a wide range of comments on proposed revisions can be very helpful in developing quality regulations that balance cost with the expected benefits of the regulations. They said that obtaining a wide range of comments also assists them to build support for the changes. Benefits Of DOE's Reform Effort Are Unclear: Under its reform effort, DOE intended to enhance productivity and reduce costs while maintaining safety, but it is unclear whether its effort will achieve these benefits, or whether the benefits will outweigh the costs to implement the reform. For example, DOE did not determine how the original requirements contained in safety directives impaired productivity or added costs before undertaking the reform effort. Moreover, DOE did not establish clear goals and performance measures to determine the potential benefits of its reform effort. As a result, DOE is not well positioned to know that its reform effort will achieve its intended benefits. DOE Sought To Enhance Productivity And Reduce Costs But Did Not First Analyze Burden Or Costs Of The Original Requirements: According to the Deputy Secretary's March 2010 memorandum announcing the reform effort and discussions with DOE officials, DOE undertook the reform effort to realize productivity and cost benefits while maintaining safety. According to the memorandum, achieving these benefits would be accomplished, in part, by revising requirements considered to be burdensome and overly prescriptive.[Footnote 18] To identify safety requirements that were burdensome and overly prescriptive, DOE's Associate Deputy Secretary told us that DOE and contractor officials used their experience and judgment. It is unclear, however, whether the safety requirements these officials identified were indeed burdensome because DOE and contractor officials we spoke with could not provide clear examples of how these requirements negatively affected productivity or costs or criteria that they used for making a determination that they were burdensome. Specifically: * At the request of the Secretary of Energy, senior managers at DOE's national laboratories participated in an informal May 2011 study of the DOE policies they considered to be the most burdensome. Four of the 18 most burdensome policies the laboratory managers identified in this study addressed safety-related issues--excessive oversight; duplication between directives and industry standards; directives with requirements that provide no value, including DOE's quality assurance directive; and excessive safety reporting. In the study, however, the laboratory managers did not provide specific examples of burdensome safety requirements or criteria that could be used for making a determination of what is burdensome.[Footnote 19] For example, the senior laboratory official who led the study could not provide us examples of specific safety requirements that were burdensome from the survey data collected from the laboratories' managers. The official also could not provide an explanation of how burdensome requirements affected productivity or costs or how changes to them would be an improvement. * In response to the concerns of several senior contractor and federal staff in NNSA that DOE's nuclear safety requirements may be burdensome or overly prescriptive, NNSA launched a study to determine the strength of these concerns. In April 2011, as a result of the study, NNSA reported that the original requirements did not place an undue burden on the contractors.[Footnote 20] The study found that senior contractor managers considered that DOE nuclear safety requirements provided a sound framework to safely operate nuclear facilities. These managers noted, however, that DOE had not clearly communicated its safety requirements, which resulted in uneven implementation of these requirements across NNSA's sites. In terms of productivity, DOE officials indicated that productivity improvements from reducing burdensome requirements will be achieved. For example, several senior DOE officials told us that increases in productivity would result from raising the threshold at which a nuclear facility, such as a facility treating radioactive waste, would be required to undergo a rigorous review process before starting up to ensure it will operate safely. This review process involves federal staff review of a contractor's detailed set of facility startup procedures, contractor preparation of a detailed plan to correct any deficiencies found during that review, and preparation of a detailed report indicating how any deficiencies were resolved. This rigorous review was previously required whenever any facility's operations were restarted, even after routine maintenance. Under the reform effort, this level of review is now only required for facilities and activities posing a higher risk, such as starting up a newly constructed nuclear facility, and not required for facilities and activities posing lesser risk, such as restarting facilities after routine maintenance. According to two officials from DOE's Office of Health, Safety, and Security, the revised requirement will help free up federal resources to focus on higher-risk activities, such as the startup of a newly constructed nuclear waste processing facility, instead of lower-risk activities, which are the responsibility of the contractor to manage safely. DOE did not analyze the costs of the original requirements or attempt to quantify how revising or eliminating requirements would reduce costs. DOE officials told us that cost savings are expected as a result of having a more streamlined, clear, and nonduplicative set of requirements. However, according to DOE officials, the department had not developed estimates of the cost savings it expected. According to DOE's Associate Deputy Secretary, the department has launched an initial study of selected directives to determine how changes to these directives have impacted mission work, schedule, and costs, among other things. Moreover, it is not clear whether the reform's benefits will exceed the costs of implementing the revised directives, as DOE's reform effort is likely to add costs in the short term. Specifically, several officials at DOE's Hanford Site told us that contractors will have to perform cost-impact analyses prior to implementing the revised directives in their contracts. These officials estimated that it can cost $20,000 to $50,000 to conduct this type of analysis for each revised directive. DOE Did Not Determine The Potential Benefits Of Its Reform Effort By Establishing Clear Goals And Performance Measures: As we have previously reported, high-performing organizations typically follow four key practices to ensure management improvement initiatives are successfully implemented and achieve their intended benefits.[Footnote 21] These practices include: (1) establishing clearly defined goals, (2) developing an implementation strategy that sets milestones and establishes responsibility, (3) establishing results-oriented outcome measures early in the process, and (4) systematically using results-oriented data to evaluate the effectiveness of the initiative and make additional changes where warranted. Table 1 describes characteristics of these four key practices, as identified by us and others, including DOE. Table 1: Four Key Practices of Management Improvement Initiatives: Key practices: Clearly defined goals; Characteristics of key practices: * Goals are targeted levels of performance, expressed as tangible, measurable objectives against which achievement can be compared; * Good goals should, among other things, be understandable and expressed in simple, clear terms so as to avoid misinterpretations or vagueness. Key practices: Implementation strategy; Characteristics of key practices: * Strategy sets milestones, assigns roles and responsibilities, describes key tasks, and establishes timelines for the implementation of the reform; * Strategy provides a guideline for near-term, as well as longer-term, implementation of the initiative to ensure that the long-term goals of the initiative are achieved. Key practices: Outcome measures; Characteristics of key practices: * Measures assess the results of a program activity or effort compared to its intended purpose; * Good performance measures, among other things, reflect results, not the activities used to produce results; relate directly to a performance goal; are practical and easily understood by all; and provide a benefit that exceeds the cost. Key practices: Effectiveness evaluation; Characteristics of key practices: * The evaluation uses results-oriented measures and supporting performance data to assess the effectiveness of an initiative and to identify any necessary changes, corrective actions, or follow-up actions to better achieve the underlying goals or objectives of the initiative. Source: GAO analysis of prior GAO reports, DOE documents, and Office of Management and Budget guidance. [End of table] DOE did not completely follow these key practices to assess benefits when revising or planning for the implementation of the department's safety directives. Specifically: * Clearly defined goals. DOE's stated goals--enhancing productivity and reducing costs while maintaining safe operations--are defined so generally that they do not lead to the development of tangible, measurable objectives against which achievement can be compared. For example, the goal of reducing costs does not specify those safety areas where DOE believes savings can be achieved. Moreover, because the goals are so general, DOE site office and contractor officials expressed concern about the lack of effective communication about what the reform effort is trying to achieve. For example, some NNSA officials at the Nevada National Security Site told us that the goals of the reform effort are unclear. Without clear goals that are objective, measurable, and effectively communicated to its sites, DOE will find it difficult to fully assess whether the reform effort is making progress toward achieving its intended results. * Implementation strategy. DOE developed a near-term implementation strategy for its reform effort--which included how safety directives would be revised--but did not develop a long-term strategy on how those revised directives would be implemented. For near-term implementation, DOE developed a project management plan, which focuses on the process to revise the directives but not on how to implement most of them. In the plan, DOE identifies priority actions and milestones to be achieved; establishes a process for the directives reform; and assigns responsibility for leading and managing the directives reform to DOE's Office of Health, Safety, and Security. DOE's implementation strategy, however, did not provide further information on how these changes would be implemented over the long term at DOE's sites. According to DOE officials, it is up to each program office within DOE to determine how best to implement the revised directives.[Footnote 22] DOE's Office of Health, Safety, and Security, however, has developed an implementation strategy for changing its approach to independent oversight that details how these changes will be implemented at DOE's sites. * Outcome measures. DOE has not developed results-oriented outcome measures for productivity, cost, or safety under its reform effort. Instead, DOE's reform effort has focused on output-oriented measures, such as the number of directives canceled or revised, and not on outcome measures, such as specific improvements in productivity or cost savings. DOE indicated that the reform is expected to result in more than a 50 percent reduction in the number of directives but has not explained how this reduction will improve productivity, reduce costs, or maintain or improve safety. DOE's focus on output-oriented measures such as the number of directives reduced does not align with its own guidance on performance measures. DOE's guidance states that there is a danger when performance measures become numerical quotas because the setting of numerical goals and quotas does nothing to accomplish process improvements.[Footnote 23] For safety performance, DOE has measures that it uses to monitor safety problems at its sites. However, according to DOE officials, the department has not developed a plan showing how it could use these measures, if at all, to evaluate the benefits of its reform effort. * Effectiveness evaluation. DOE did not develop a strategy for evaluating the effectiveness of its reform effort and therefore is not collecting the results-oriented performance data necessary to identify benefits achieved or areas where further changes are needed. To date, DOE officials have only provided anecdotes on the benefits the reform is expected to achieve. For example, in response to written questions we provided to the department, DOE's Office of Health, Safety, and Security stated that it expects measurable productivity improvements because it significantly streamlined the requirements for contractor oversight systems, which provide information on safety performance and other management issues. Some officials at DOE sites we visited, however, stated that DOE's reform effort may not produce any productivity or safety improvements and may have little impact on the way they conduct their work. For instance, DOE and contractor officials at DOE's Hanford Site told us that, while they are still working on implementing the revised directives at their site, they have neither seen any changes in project safety performance, nor do they expect to see any changes in the future as a result of the reform effort. Additionally, as discussed previously, DOE has launched an initial study of selected directives to determine how changes to these directives have impacted mission work, schedule, and costs, among other things. However, without results-oriented performance data, it will be difficult for DOE to identify benefits achieved by the reform effort. DOE's Reform Effort Did Not Fully Address Safety Concerns We And Others Have Identified: DOE's reform effort did not fully address safety concerns that we, DOE's Office of Inspector General, and the Safety Board have repeatedly identified in three key areas--(1) quality assurance, (2) safety culture, and (3) federal oversight. In each of these three areas, some changes that DOE has made to its safety directives may begin to address safety concerns, but other changes it made are potentially problematic. DOE Strengthened Its Quality Assurance Directive By Requiring Contractors To Follow Established Industry Standards, But Some Problems Persist: Quality assurance entails ensuring that proper work processes and procedures are in place to ensure all work meets certain standards of quality. Such processes include correcting identified problems and following up to ensure that these actions are effective. As we, DOE's Office of Inspector General, and the Safety Board have reported in 20 reports since 2001, DOE has experienced numerous quality assurance problems, ranging from improper documentation detailing quality assurance procedures to insufficient monitoring of actions to correct identified problems. Under its reform effort, DOE revised a key quality assurance directive in April 2011 to update and clarify how the directive applies to facilities with and without nuclear materials as well as to encourage the use of industry quality assurance standards. Specifically: * In its revised directive, DOE clarified responsibilities to help ensure that contractors at facilities without nuclear materials at DOE's sites are better aware that they can tailor implementation of their quality assurance programs to their nonnuclear needs. According to DOE officials, because problems at nonnuclear facilities may pose different levels of risk than those at nuclear facilities, quality assurance programs can be more flexible at DOE's nonnuclear facilities and sites. * To support the reform effort's goal of greater use of industry safety quality assurance standards, the revised directive includes a new requirement for contractors overseeing high-hazard nuclear facilities to use a uniform version of a quality assurance standard developed by the American Society of Mechanical Engineers. Previously, some sites were using different versions of the standard. As a result, according to DOE documents we reviewed, assessments of the quality assurance practices at DOE's sites sometimes resulted in citations that did not concern safety issues but rather arose from differences in the way that the different versions of the quality assurance standard were written. By eliminating the use of multiple versions of the quality assurance standard, DOE believes it can save time and costs when doing assessments. * The revised quality assurance directive now helps to clarify quality assurance requirements for DOE's safety software, which is consistent with commitments DOE made in response to the Safety Board's September 2002 recommendation to improve the quality of DOE's safety software [Footnote 24] and our February 2011 recommendation to clarify specific quality assurance requirements for computer models used in environmental cleanup decisions.[Footnote 25] DOE expects this clarification to improve the performance of software used on safety- related systems, according to DOE documentation we reviewed. These revisions appear to have strengthened the directive, but it is not yet clear if they will fully address quality assurance problems, some of which continue. For example, DOE's Office of Enforcement identified quality assurance problems following a June 2010 incident during which a worker was exposed to radiation after puncturing his hand with a sharp object contaminated with plutonium at DOE's Savannah River Site. As a result, in July 2011, this office proposed a fine of $243,750 against the contractor. Furthermore, in June 2011, DOE's Office of Inspector General reported that both the Livermore Site Office and the laboratory's contractor lacked adequate quality assurance processes to ensure that corrective actions were effective. [Footnote 26] In the report, DOE's Office of Inspector General recommended that the site office perform more timely inspections to verify that corrective actions had been instituted prior to closing the finding that led to the corrective actions. In addition, at a nuclear facility at NNSA's Nevada National Security Site, NNSA found in August 2011 that the contractor had not effectively identified, controlled, or prevented the recurrence of quality assurance problems related to the installation and inspection of fire safety and other components in nuclear testing facilities, as required in nuclear safety regulations, and proposed a nearly $180,000 fine against the contractor. DOE's Revisions To Its Directives Were Intended To Strengthen Its Safety Culture, But The Safety Board Raised Concerns That Less Vigorous Safety Practices Could Result: A strong safety culture includes the quality that health and safety are integrated into all work decisions so that workers and managers are better aware of and better implement safety practices, which helps to minimize accidents. DOE defines safety culture as an organization's values and behaviors modeled by leadership and practiced by employees to strive to make the safe performance of work a priority. To promote a strong safety culture at its sites, DOE has federal regulations [Footnote 27] and a set of directives--collectively known as Integrated Safety Management--for implementing the department's key safety management practices. These directives lay out high-level expectations and implementing methods for safety management and include several guides and manuals with suggested methods for implementing safety management practices. Many of these directives resulted from DOE's response to October 1995 and September 1998 recommendations by the Safety Board to establish clear and systematic processes for addressing and resolving safety issues and to establish clear lines of responsibility for safety.[Footnote 28] Nevertheless, DOE has experienced problems maintaining an effective safety culture at some of its sites, as we, DOE's Office of Inspector General, and the Safety Board have repeatedly reported. Problems have included not following required safety practices, inadequate testing of key nuclear waste processing systems to ensure they operate safely, and not developing safety strategies for extremely hazardous events. For example, in October 2007, we identified weaknesses in the safety culture at DOE's three nuclear research laboratories, and recommended that DOE strengthen safety management at its weapons laboratories and ensure that safety improvements are carried out in a systematic manner.[Footnote 29] DOE generally agreed with our recommendations, but it noted that, in the department's view, the safety management systems DOE had in place adequately addressed safety culture issues and, as a result, the department took no further action to address those recommendations. In addition, in September 2009, DOE's Office of Inspector General reported that fire fighters who respond to incidents at Los Alamos were not provided with enough opportunity to tour and become familiar with the site, which would impede response time in an emergency.[Footnote 30] Furthermore, in October 2009, the Safety Board reported that Los Alamos lacked a safety strategy to minimize the potential consequences of earthquakes to stored nuclear material and recommended actions to mitigate these potential consequences.[Footnote 31] Under its reform effort, DOE revised its Integrated Safety Management directives to attempt to strengthen the safety culture at its sites. For example, DOE's revisions more clearly emphasized DOE and contractor senior management roles and responsibilities for building a strong safety culture at DOE's sites. Specifically, the new directives lay out tasks that each level of management should complete, such as developing and maintaining safety documents, approving site office safety initiatives, and establishing safety expectations across DOE's sites. These revisions address past issues, such as lack of clarity in roles and responsibilities for ensuring safety practices at DOE's sites. In addition, DOE has undertaken several efforts aimed at improving safety practices across its sites. For example, the department has worked closely with a contractor group--Energy Facilities Contractors Group--to issue guidance on assessing and strengthening safety culture. In addition, DOE issued a new Integrated Safety Management guide in September 2011 to provide additional and extensive information to assist the programs in implementing integrated safety management requirements. Finally, in 2011, DOE established its Nuclear Safety and Security Council, which advises the department's senior management on nuclear safety issues with the aim to improve safety performance. However, some revisions DOE made to its Integrated Safety Management directives raised concerns with Safety Board officials that safety culture may be undermined. For example: * When it revised its Integrated Safety Management directives, DOE removed language that required contractors to follow the directives and instead revised the directives to apply only to federal employees. This is because contractors already had to comply with the department's acquisition regulations, which the Safety Board noted contain similar but less detailed requirements. According to Safety Board officials, because the acquisition regulation does not contain the specificity of the Integrated Safety Management directives, contractors may not implement the requirements as vigorously as they would if they had to follow the more specific practices in the original directives. * Because DOE removed language for contractors to follow the department's Integrated Safety Management directives, contractors will no longer need to consider best practices for implementing a safety culture based on lessons learned over the years that are integrated into the directives. Specifically, Safety Board officials stated that they are concerned that contractors will therefore overlook these lessons learned since the contractors are no longer required to follow the Integrated Safety Management directives. It is also not clear how revisions to DOE's Integrated Safety Management directives will affect safety culture problems that persist at DOE's sites. For example, in June 2011, the Safety Board reported that workers at DOE's Waste Treatment and Immobilization Plant at its Hanford Site hesitated to raise safety concerns for fear of management reprisal and recommended that the Secretary of Energy act to ensure a strong safety culture at the site.[Footnote 32] DOE's Office of Health, Safety, and Security has also recently reported on continuing safety culture problems at the Waste Treatment and Immobilization Plant.[Footnote 33] Furthermore, DOE's Office of Enforcement issued a notice of violation in September 2011 to the contractor at DOE's Separations Process Research Unit site in Niskayuna, New York, after a building demolition accident potentially exposed at least 100 employees to low-level radioactive contamination in September 2010. DOE's Office of Enforcement found that the accident was tied to the contractor's failure to consistently use safe work practices at the site and management's failure to assess work processes to ensure a safe working environment. DOE Clarified Roles And Responsibilities For Its Oversight Of Contractor Activities, But Its Reform May Result In Weakened Federal Oversight: We and others have reported on DOE's problems overseeing contractors' performance. These problems have ranged from the department not ensuring that it had sufficient resources to perform oversight at its sites to not consistently applying such oversight. For example, in October 2008, we reported that DOE's authority to impose civil penalties on contractors for violating nuclear safety requirements had not reduced some recurring safety problems, and we recommended that DOE increase its independent oversight presence at DOE's sites to provide more frequent observations to facilitate enforcement actions. [Footnote 34] In response to our recommendations, and under its reform effort, DOE has changed its approach to conducting independent oversight that is reflected in its revised oversight directives. For example: * DOE has placed greater emphasis on having its Office of Independent Oversight staff review safety design documents, which lay out the safety systems for facilities at DOE's sites, before their construction. Our October 2008 report noted that lack of oversight meant that these documents did not receive an independent review and therefore the safe operation of newly constructed facilities at DOE's sites could not be reasonably assured. In response to our recommendations and under its reform effort, DOE's Office of Independent Oversight staff now review the safety design documents before the facility is constructed, rather than after it is built, and when the facility undergoes major modification. * DOE implemented a new approach to oversight that designates headquarters staff responsibility for oversight at key sites, such as DOE's Hanford and Savannah River sites and its Idaho National Laboratory. These "site leads" monitor and assess all aspects of a site's operations in order to assist the Office of Independent Oversight in planning assessment activities. According to Office of Independent Oversight officials, under the site lead approach, oversight staff now visit a site several times per year rather than approximately once every 3 years under the previous approach. These officials said that they believe this change will result in more frequent reviews at these key sites, enabling them to catch potential safety problems earlier. However, some of the revisions DOE made to its oversight directives under its reform effort raise concerns about the department's ability to effectively execute independent oversight. According to DOE's independent oversight directive, independent oversight is an integral part of DOE's ability to help ensure the effectiveness of DOE and contractor safety performance. However, some of DOE's revisions place accountability for key oversight functions with the site offices and contractors, which may reduce confidence in an independent oversight process. For example: * DOE revised its oversight approach to give site office, rather than Office of Independent Oversight staff, increased responsibility for managing actions to correct problems raised in independent assessments. Previously, issues identified during Independent Oversight assessments had to be addressed with a formal corrective action plan submitted to and tracked by the Office of Independent Oversight. Under the reform, site offices have responsibility for determining which issues will be formally addressed and monitored, and which problems are dealt with without the need for a formal plan. According to some Office of Independent Oversight officials, under this revised approach, independent oversight staff try to limit the concerns raised during assessments to higher priority issues in order to better ensure these receive site management attention. Site office determinations of whether issues required more formal contractor responses may be influenced by the fact that the site offices also have responsibility for keeping costs under control and work on schedule. * In giving greater responsibility to the site offices, DOE has allowed them to use staff from the Office of Independent Oversight to help assess contractor performance. In addition, Office of Independent Oversight staff must now coordinate their assessment activities with site office management in order to maximize the use of resources. This potentially raises concerns about whether Office of Independent Oversight staff will be sufficiently independent from site office management. Furthermore, Safety Board officials told us that the coordination requirement could reduce the effectiveness of Office of Independent Oversight assessments because surprise "drop-in" assessments would be lost, allowing site office and contractor staff to prepare ahead of the assessment. We have raised similar concerns in the past regarding the independence of DOE's oversight, including concerns that the department's focus on providing program offices a greater role in overseeing contractor oversight resulted in weakening independent review.[Footnote 35] Even after DOE's oversight directives were revised, effective oversight of contractor activities continues to be a problem for DOE. For example, in June 2011, DOE's Office of Inspector General reported that the Livermore site office was not sufficiently overseeing the contractor to ensure that corrective actions were fully and effectively implemented.[Footnote 36] In September 2011, DOE's Office of Enforcement proposed a fine of $412,500 against the Separations Process Research Unit contractor in Niskayuna, New York, due, in part, to oversight inadequacies during the September 2010 accident discussed earlier in this report. Moreover, the Office of Enforcement proposed a $150,000 fine in early October 2011 against the Idaho National Laboratory contractor, in part, because of inadequate management oversight that resulted in the tipping over of a hoisting rig loaded with 7,800 pounds. Conclusions: DOE's reform effort was driven by a belief that some requirements in its safety directives have placed excessive burdens on its contractors and that revisions would give contractors more flexibility in operating safely. The elimination of potentially burdensome, duplicative, and costly requirements that offer little or no contribution to improved safety is a worthy goal. If high levels of safety could be assured at DOE's sites while unnecessary requirements were eliminated, then DOE's reform effort would be considered a success. However, DOE did not first determine whether the directives it planned to revise were indeed burdensome or costly. DOE also does not have results-oriented outcome measures to help the department gauge the effectiveness of its effort. Simply counting the number of directives revised or eliminated does not indicate the benefit of the reform on productivity and safety performance at DOE's sites. Safety should not be measured by the amount of paper that is saved but by actual improvements in safety performance across the department. Before embarking on management reforms, high-performing organizations (1) set clear goals, (2) develop an implementation strategy that sets milestones and establishes responsibility, (3) establish results- oriented outcome measures, and (4) use results-oriented data to evaluate the reform's effectiveness and make additional changes where needed. However, DOE did not fully follow these practices to assess the costs and benefits of revising its safety directives. Furthermore, our discussions with DOE and contractor officials at DOE's sites indicate that there is some question about the benefits the reform effort can actually achieve. It is also unclear how revising safety directives will address past and recurring safety concerns with quality assurance, safety culture, and federal oversight of contractor activities that we and others have identified over the years. Many of the directives DOE revised were originally developed to correct problems in these areas. In revising these directives, DOE could be undermining hard-won progress over the years in safety performance at its sites. DOE has not effectively communicated with its site officials or its contractors about the overall goals of the reform effort, how progress will be measured, and whether benefits of the reform will exceed the costs involved with revising directives. In addition, changes to DOE's oversight directives raise concerns about the ability of DOE's Office of Independent Oversight staff to provide a critical review of safety at DOE's sites that is independent from DOE site office and contractor staff, a concern we have raised in the past. As a result, it is unclear whether DOE's reform effort will result in actual productivity or safety improvements at its sites. Recommendations For Executive Action: To help ensure that DOE's reform of its safety directives results in improved productivity and safety at its sites, we recommend that the Secretary of Energy take the following four actions prior to fully implementing revisions to its directives across all of the department's sites: * systematically analyze the costs and benefits associated with implementing the revised safety directives to ensure that the costs do not exceed the benefits that the department expects to achieve; * provide DOE's sites and contractors with a plan that details (1) the reform effort's goals, (2) the effort's long-term implementation strategy, (3) results-oriented outcome measures, and (4) how DOE will use results-oriented data to evaluate the reform's effectiveness and to determine whether additional changes are needed; * ensure that the plan developed for DOE's sites and contractors identifies how the reform effort will help address past and recurring safety concerns with quality assurance, safety culture, and federal oversight of contractor activities; and: * clearly define the oversight roles and responsibilities of DOE's Office of Independent Oversight staff to ensure that their work is sufficiently independent from the activities of DOE site office and contractor staff. Agency Comments And Our Evaluation: We provided DOE with a draft of this report for its review and comment. In its written comments, DOE generally agreed with our recommendations and noted that it will use the report to build on its commitment to continually improve its approach to safety management. However, DOE commented that it had significant concerns with the accuracy of the report's findings and the validity of its conclusions. Specifically, DOE commented that it never assumes its existing directives are perfect and cannot be improved. Instead, DOE stated that its management principles require it to constantly evolve, engaging with its employees, contractors, and stakeholders to recognize impediments and work to mitigate them. We agree with DOE and, as our draft report noted, the elimination of potentially burdensome, duplicative, and costly requirements that offer little or no contribution to improved safety is a worthy goal. Our draft report also clearly described the process DOE followed and the involvement of its employees; contractors; and stakeholders, such as the Safety Board. However, in our view, it is not possible to identify impediments and work to mitigate them without first determining whether and how existing requirements are burdensome. As our draft report stated, an April 2011 NNSA study found that DOE's nuclear safety requirements did not place an undue burden on the contractors. Instead, NNSA reported that these requirements provided a sound framework to safely operate nuclear facilities. Officials we spoke with at some of DOE's sites also told us that they lacked sufficient data to determine whether the safety requirements the department was revising were burdensome. We agree that continuous improvement to remove impediments is beneficial, but DOE did not determine whether its safety directives were, in fact, impediments prior to revising them. In addition, DOE's letter provided information on the roles of its Office of Health, Safety, and Security and noted that its oversight and regulatory enforcement is independent from line management, guaranteeing the office's autonomy without potential conflicts of interest. Our draft report clearly described the roles and responsibilities of DOE's Office of Health, Safety, and Security and its Office of Independent Oversight. However, some of DOE's revisions to its directives place accountability for key oversight functions with DOE site offices and contractors, which may reduce confidence in an independent oversight process. For example, as we noted in our draft report, Office of Independent Oversight staff must now coordinate assessment activities with site office management in order to maximize the use of resources. Safety Board officials told us that the coordination requirement could reduce the effectiveness of assessments because surprise "drop-in" assessments would be lost, allowing site offices and contractor staff to prepare ahead of the assessment. This problem echoes similar concerns we have raised in the past regarding the independence of DOE's oversight, including that the department's focus on providing program offices a greater role in overseeing contractor oversight resulted in weakening independent review. DOE also enclosed a longer summary of its concerns with the draft report's findings and conclusions. DOE's letter, summary, and our detailed responses are reproduced in appendix II. DOE also provided technical comments, which we incorporated as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of to the appropriate congressional committees; the Secretary of Energy; the Director, Office of Management and Budget; and other interested parties. In addition, the report will be available at no charge on the GAO website at [hyperlink, http://www.gao.gov]. If you or your staff have any questions regarding this report, please contact me at (202) 512-3841 or aloisee@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix III. Signed by: Gene Aloise: Director: Natural Resources and Environment: List of Requesters: The Honorable Fred Upton: Chairman: The Honorable Henry A. Waxman: Ranking Member: Committee on Energy and Commerce: House of Representatives: The Honorable Cliff Stearns: Chairman: Subcommittee on Oversight and Investigations: Committee on Energy and Commerce: House of Representatives: The Honorable Joe Barton: House of Representatives: The Honorable Michael C. Burgess: House of Representatives: [End of section] Appendix I: Scope and Methodology: To examine how the Department of Energy (DOE) revised safety directives under its reform effort, we reviewed DOE documents describing the directives review process, including the March 2010 Safety and Security Reform Plan, the June 2010 Project Management Plan, and the September 2010 HSS Requirements Framework and Systems Approach: Integrating Requirements Framework and Overview. We also reviewed documents used to revise safety directives, including justification memorandums and crosswalks between the original directives and their revisions. We also reviewed comments to directive revisions that were submitted to the department's directives review system. This review system--called RevCom--is an online system that disseminates proposed revisions to DOE federal and contractor staff for their review and comment on the revisions' content, relevance, applicability, accuracy, impact, and implementation cost. We interviewed officials at DOE headquarters, including officials in DOE's Office of Health, Safety, and Security and the National Nuclear Security Administration (NNSA), about the directives review process. In addition, we visited the Kansas City Plant in Kansas City, Missouri; the Nevada National Security Site in Las Vegas, Nevada; the Lawrence Livermore National Laboratory in Livermore, California; and the Hanford Site near Richland, Washington state; and interviewed DOE and contractor officials at those sites to determine how the safety directives and oversight reform affected the sites and contractors, if at all. Each of these sites is in different stages of implementing revised directives, from full implementation of revised directives at the Kansas City Plant to revised directives not yet implemented at the Lawrence Livermore National Laboratory. We also interviewed officials at other federal regulatory agencies, including the Nuclear Regulatory Commission and the Department of Labor's Office of Occupational Health and Safety, to gain an understanding of the process of changing safety requirements and federal regulations. We reviewed comments on proposed revisions to DOE's safety directives made by the Defense Nuclear Facilities Safety Board (Safety Board)--which was created by Congress to independently assess safety conditions and operations at defense nuclear facilities at DOE's sites and interviewed Safety Board officials about those changes. We also interviewed officials from the American Society of Industrial Standards to determine how industry standards are developed. To examine the benefits DOE hoped to achieve through its reform effort, we reviewed the reform documents mentioned previously, among others. We interviewed officials at DOE headquarters, including the Associate Deputy Secretary of Energy and staff from the Office of Health, Safety, and Security, to discuss how DOE intended to gauge the success of its reform effort. We also interviewed officials with NNSA to discuss the DOE reform effort as it affected NNSA's simultaneous governance and oversight reform effort. NNSA's effort involves revising the agency's business model to, among other things, place more reliance on contractor's self-oversight through its contractor assurance systems. Furthermore, we visited the Kansas City Plant, the Nevada National Security Site, Lawrence Livermore National Laboratory, and the Hanford Site and interviewed DOE site office and contractor officials about the 2010 reform, site performance measures, expected productivity improvements, and whether cost savings were anticipated. To identify and develop best practices criteria for managing improvement initiatives and performance measures, we reviewed our prior work, DOE guidance on managing improvement initiatives and performance measures, and documentation on performance measures and performance evaluation by industry groups such as the Energy Facility Contractors Group. We then compared these best practices criteria to DOE's documents describing its reform effort, including the March 2010 Safety and Security Reform Plan, the June 2010 Project Management Plan, and the August 2010 Program Plan for Enhancing Independent Oversight of Nuclear Safety, in order to determine how these best practices criteria were applied to these documents. We also verified this information by speaking with DOE officials about these best practices criteria to obtain their perspectives on how these criteria were applied to the reform effort. To identify other safety performance measures DOE uses to monitor safety performance at DOE's sites, we reviewed DOE's Office of Health, Safety, and Security worker safety performance reports and data, as well as performance measures, performance data, and performance evaluation plans provided to us by site office and contractor officials during our site visits. We did not validate these data, but we did examine the reasonableness of these data based on information in our prior work and DOE reports and concluded that these were sufficient for purposes of our report. We also interviewed officials from the Safety Board on issues pertaining to performance measures and performance evaluation. To examine the extent to which DOE's reform effort will address concerns we and others identified, we reviewed reports over the past 10 years by GAO, DOE's Office of Inspector General, and the Safety Board that contain recommendations related to safety and oversight issues at the department. To identify the safety-related concerns of these report recommendations, we conducted a content analysis of the recommendations. Specifically, we developed categories for identifying safety-related concerns--including quality assurance, nuclear safety, worker safety, and oversight--using DOE's HSS Requirements Framework and Systems Approach: Integrating Requirements Framework and Overview document. After discussions with our methodologist and a review of DOE safety directives, we combined the nuclear safety and worker safety categories under the category of safety culture. Two analysts independently reviewed the report recommendations and coded them into the three categories. In cases where differences between the two reviewers regarding the coding of the recommendations into these safety-related categories were found, all differences were resolved through reviewer discussion. To determine how DOE's reform effort addressed past safety concerns, we reviewed key safety directives, crosswalks between the previous version of the directive and the current version, comments provided on those changes from DOE staff and contractors, and comments provided by the Safety Board. We also visited and interviewed staff at the Kansas City Plant, the Nevada National Security Site, Lawrence Livermore National Laboratory, and the Hanford Site to determine how the reform effort was affecting their day-to-day operations and strategies to address past concerns on- site. We also interviewed DOE officials, including at the Office of Health, Safety, and Security, as well as Safety Board officials. We conducted this performance audit from February 2011 through April 2012 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. [End of section] Appendix II: Comments from the Department of Energy: Note: GAO comments supplementing those in the report text appear at the end of this appendix: Department of Energy: Washington, DC 20585: April 9, 2012: Mr. Gene Aloise, Director: Natural Resources and Environment: U.S. Government Accountability Office: 441 G Street, NW: Room 2T23A: Washington, DC 20548: Dear Mr. Aloise: This letter provides the U.S. Department of Energy's (DOE or Department) response to the March 2012, Government Accountability Office (GAO) report, GAO-12-347, Nuclear Safety: DOE Needs to Determine The Costs and Benefits of Its Safety Reform Effort. The Department recognizes and appreciates the important role the GAO plays in providing oversight of federal agencies and programs. While the Department will use this GAO report to build on our commitment to continually improve our approach to safety management, we have significant concerns and take issue with the accuracy of this particular report's findings, the validity of its conclusions, and some of its stated recommendations. As the Secretary and Deputy Secretary have repeatedly stated, nuclear safety is a core value of the DOE. We believe that there is never a need to choose between mission and safety; safety is, in fact, an integral part of our mission. As the Department seeks to improve the way we do business across the complex, we are committed to doing so in a way that is completely and entirely consistent with our management principles, including our commitment to "pursue our mission in a manner that is safe, secure, legally and ethically sound, and fiscally responsible.” Our management principles further state that "we will only succeed through teamwork and continuous improvement." We therefore never assume that our existing directives arc perfect, or cannot he improved. Rather, our management principles require us to constantly evolve, engaging with our employees, contractors, and stakeholders to recognize impediments and work to mitigate them. That commitment to continuous improvement is what led us to a sustained died to improve and streamline our internal directives. As the Chief Health, Safety and Security Officer, my office plays two essential roles in implementing the Department's safety reform. First, the Office of Health, Safety and Security (HSS), in close collaboration with DOE Central Technical Authorities and line management, is responsible for developing DOE nuclear safety policy, Federal Rules, Orders, and the associated standards and guidance; for reviewing safety issues complex-wide; and for assuring that all program managers receive the benefit of HSS expertise in safety and related-best practices. The second HSS function is to conduct oversight and regulatory enforcement that is independent from line management. The independence of HSS, which reports not through the Under Secretaries but directly to the Secretary and Deputy Secretary, guarantees HSS's autonomy to exercise its oversight and regulatory role without potential conflicts of interest. We believe this GAO report does not accurately reflect the intent, the structure or the impact of the Department's safety reform efforts. Nor does this report reflect a full understanding of the disciplined and systematic approach the Department took to revising and streamlining its internal management directives process in order to improve safety across the complex. As requested by the Secretary and Deputy Secretary, this reform effort was undertaken with great care to ensure that the Department's safety program will continue to support effective safety performance and improve safety at our sites. I am concerned that, if not revised to correct significant mischaracterizations, the GAO report could mislead the Department's stakeholders and the public. The Department would like to call to GAO's attention the concerns and perspectives outlined in the enclosed summary. More detailed comments have also been provided separately. If you have any questions about our planned actions and improvement initiatives, please contact me at (202) 586-0271. Sincerely, Signed by: Glenn S. Podonsky: Chief Health, Safety and Security Officer: Office of Health, Safety and Security: Enclosure: cc: Daniel B. Poneman, Deputy Secretary of Energy: Melvin G, Williams Jr., Associate Deputy Secretary of Energy: Arun Majumdar, Under Secretary of Energy: Thomas P. D'Agostino, Administrator, National Nuclear Security Administration: William F. Brinkman, Director, Office of Science: [End of letter] Enclosure: Summary of the Department of Energy's Concerns and Perspectives on GAO Report, GAO-12-347, Nuclear Safety: DOE Needs to Determine the Costs and Benefits of Its Safety Reform Effort: The Department of Energy (DOE) provides the following comments with respect to the Findings, Conclusions, and Recommendations contained in the Government Accountability Office (GAO) Report. GAO Findings: The report does not identify any significant flaws in the revised directives that resulted from the Department's safety reform initiative, or any way in which the revised directives are inconsistent with improved safety. If effectively implemented, the Department is confident that these safety directives, combined with our long-standing safety regulations, provide adequate protection for the public and our workers and will effectively support mission requirements in a safe and efficient manner. [See comment 1] The Defense Nuclear Facilities Safety Board (Defense Board), a federal agency that was statutorily chartered, in part, to evaluate the adequacy of DOE directives, was systematically involved in the review of revised DOE directives at multiple stages in the process. The Department approved its revised directives only after a full consideration of all comments received from the Defense Board. There is only one current DOE safety management directive, DOE Order 450.2, Integrated Safety Management, where the Defense Board provided substantive comments that were not fully resolved. Some of these comments were subsequently addressed with the development and issuance of the related Guide, which the Defense Board reviewed prior to issuance and offered no concerns. [See comment 2] As a safety regulator, DOE solicits public input on its safety regulations pursuant to the Administrative Procedures Act, consistent with the practices of the Occupational Safety and Health Administration and the Nuclear Regulatory Commission. As the owner of the facilities undertaking our mission work, DOE establishes additional expectations in internal safety management directives. DOE's methodical approach to revising internal safety directives requires the involvement of a broad array of our highly experienced safety professionals and the concurrence of all Departmental programs and functional offices. Changes are made very deliberately, and only after thorough review. The GAO report suggests applying the requirements for public comment in the Administrative Procedures Act to the revision of DOE's internal management directives. Nothing in that Act, or its legislative history, suggests that applying its provisions beyond the scope of its statutory reach is either appropriate or warranted. [See comment 3] As part of the reform process, highly experienced subject matter experts performed a detailed qualitative analysis of DOE's prior safety directives to clarify expectations, eliminate duplication, and abolish specific low-value requirements that burdened operations without enhancing safety. In some areas, additional requirements were needed to address previously existing gaps. The GAO report does not recognize the merits of DOE's detailed analysis of requirements for each directive that served as the basis of the Department's reform decisions. The report states that "DOE did not first determine whether the directives it planned to revise were indeed burdensome or costly" and implies that the Department should have first conducted a quantitative analysis of the costs and benefits of the requirements in all of DOE's previous 107 directives, as implemented at dozens of large, complex industrial facilities with highly divergent safety hazards. This approach would be philosophically inconsistent with the Department's approach to safety and to our management principles, which call for a culture of continuous improvement. We never assume that our existing directives are perfect, or cannot be improved. We reject complacency, or insisting on creating a high bar which must be hurdled to justify a comprehensive effort to improve our directives. We also reject the idea that we must engage in a costly and time- consuming study before seeking to improve our safety, diverting time, attention, and resources away from the energetic pursuit of safety excellence in favor of a static cost-benefit analysis. [See comment 4] In short, our commitment to continuous improvement requires that we take a dynamic rather than static approach to evaluating and improving directives. The Department chose a more efficient and a more effective approach that has led to appropriate and timely changes to its safety directives. [See comment 4] The Department's safety reform resulted in more concise safety directives, with requirements that are clearer and add value in protecting our workers and members of the public, and are consistent with the goals established by the Deputy Secretary for the safety directives reform initiative. While some short term costs were associated with analysis and implementation of changes, the process of analysis is itself part of our safety culture of continuous improvement, and the Department's reformed safety directives will contribute to a more efficient operating environment at DOE sites over the long term, as reduced redundancy will mean fewer, but clearer, requirements to implement. [See comment 5] The GAO report criticizes the Department for measuring success by using output-oriented measures, such as the number of directives eliminated, and not outcome measures, but the Department does not, in fact, measure success by the number of directives eliminated. Rather, the Department evaluates the effectiveness of revised directives in supporting safe performance of work that we are confident will also reduce burdens and costs. [See comment 6] The GAO report asserts, without analysis or technical basis, that inadequate directive requirements caused a variety of performance weaknesses at Departmental operating sites over the last several years in the areas of quality assurance, safety culture, and federal oversight. Safety performance problems are often not related to deficiencies in the requirements, but rather in their implementation. The Department strives to continuously improve its performance and has strong programs to investigate operational events, identify performance weaknesses, evaluate root causes, determine the extent of condition of problems, and apply lessons learned across the Department to prevent the recurrence of such events. [See comment 7] The GAO report's significant criticism of the Department under the topic of "Safety Culture" is misguided. The report uses the terms safety management and safety culture as if they were synonymous. They are in fact different, but related, concepts. Most of the issues cited in the report are safety management issues, and instances of failure to flawlessly execute a Safety Management System, do not, in themselves, imply a weak or immature safety culture. Nevertheless, as part of this reform effort, the Department has substantially strengthened its policy expectations and guidance in the area of safety culture. The Integrated Safety Management (ISM) Policy and Order were revised to further clarify and strengthen expectations and requirements for ISM and safety culture. DOE issued the ISM Guide in September 2011 to provide additional and extensive information to assist programs in implementing these ISM requirements. Additionally, the Secretary and Deputy Secretary distributed a memorandum on safety culture to all Departmental Elements in December 2011 and have both been strongly engaged on improving our safety culture around the DOE complex. [See comment 8] The Department maintains a strong Independent Oversight program within its Office of Health, Safety and Security to objectively evaluate, on behalf of the Secretary of Energy, the performance of DOE line management organizations in the areas of safety and security. The Secretary and Deputy Secretary of Energy recently reinforced the importance and independence of the independent oversight function in the aforementioned December 5, 2011, memorandum to all Departmental Elements. The reform effort sharpened the Independent Oversight program's focus on the Department's high-hazard operations and high value national security assets through a revision of the Department's Independent Oversight directive (DOE Order 227.1, Independent Oversight Program). The GAO report incorrectly asserts that the safety reform effort weakened the Independent Oversight program, in part because the revised order requires oversight staff to coordinate their assessments with site offices and contractors. However, it has been DOE's long-standing practice to coordinate oversight activities with line organizations to ensure adequate planning and support for assessments and to promote efficiency in oversight activities. This coordination is not a change from previous order expectations. [See comment 19] The GAO report should be updated to recognize the ongoing improvement initiative at DOE, led by the Associate Deputy Secretary, to support its strategic goal of management and operational excellence. A key action completed in 2011 was establishment of the DOE Nuclear Safety and Security Council, which advises the Department's senior leadership on cross-cutting technical and programmatic nuclear safety issues. The Department continues to pursue a broad agenda to achieve operational and technical excellence and implement a performance-based culture. [See comment 10] GAO Conclusions: The conclusions arrived at in the GAO report should be revised based on the concerns and perspectives provided above in order to provide an accurate perspective on the principles underpinning our safety reform initiative as well as its effectiveness. We believe the GAO report's conclusions, as currently worded, are inaccurate and could mislead stakeholders. [See comments 11-13] Indeed, we strongly believe that the very title of the GAO report is, in fact, a conclusion, suggests bias in the report even before the subject matter is presented for the reader's consideration. [See comment 11] GAO's conclusions do not acknowledge the value and thoroughness of DOE's extensive line-by-line review of each safety directive by highly experienced federal and contractor safety professionals, in consultation with the Defense Board staff, in order to sustain an adequate and effective set of requirements. [See comment 12] GAO inappropriately compares the project management measures used to track progress in the reform effort with the measures uses to monitor safety performance. The Department tracked its progress by a series of metrics as it progressed through the revision of its safety directives. The Department monitors its safety performance metrics closely and will continue to do so to identify and correct any potential unintended outcomes. The Department's safety metrics continue to demonstrate that the Department has excellent safety performance in comparison to industries with similar hazards. [See comment 13] The GAO report did not identify any significant flaws in the revised directives resulting from the Department's safety reform initiative. The report offers no evidence to substantiate its assertion that DOE "could be undermining hard-won progress over the years in safety performance at its sites." [See comment 1] GAO Recommendations: While we do not agree with aspects of the underlying findings and conclusions, we will use GAO recommendations as part of our commitment to continuous improvement of the Department's approach to safety. The Department's responses to the specific GAO recommendations (as currently worded) are provided below: * Systematically analyze the costs and benefits associated with implementing the revised safety directives to ensure that the costs do not exceed the benefits that the Department expects to achieve. The Department will pursue this recommendation consistently with the goals in EO 13563, Improving Regulation and Regulatory Review, which directs each agency "to use the best available techniques to quantify anticipated present and future benefits and costs as accurately as possible. Where appropriate and permitted by law, each agency may consider (arid discuss qualitatively) values that are difficult or impossible to quantify, including equity, human dignity, fairness, and distributive impacts." While not applicable to the Department's internal management directives, that is guidance we applied in this case. In any event, the directives reform effort is nearly complete; only one order revision remains, and most revisions have already been implemented in DOE federal and contractor organizations. As indicated in the GAO report, qualitative costs and benefits were considered, as part of the reform process, by expert DOE and contractor personnel with many years of applicable technical knowledge and management experience. Additionally, as stated above, we take a dynamic rather than static approach to evaluating and improving directives. [See comment 14] DOE is performing a sampling assessment of the results to provide insights that may be used to evaluate how fully the reform objectives were met and to inform future directive revisions. The Department is currently pursuing an assessment on five safety and security directives and will share the outcomes with the GAO. To the extent possible, such outcomes will be quantitative, but they will also have qualitative attributes. The Department accepts this recommendation with a focus on a sampling review of the benefits achieved. * Provide DOE's sites and contractors with a plan that details (I) the reform effort's goals; (2) the effort's long-term implementation strategy; (3) results-oriented outcome measures; and (4) how DOE will use results-oriented data to evaluate the reform's effectiveness and to determine whether additional changes are needed. The Department already had well established, effective processes for implementing directive changes. The goals of this particular reform effort were clearly established and communicated by the Deputy Secretary and well known throughout the DOE organization, and the desired end state of safety directives reform was clearly stated in terms of a set of end-state outcomes. [See comment 14] The Department will monitor the status of ongoing implementation and request any specific feedback for consideration during the one-year directive review called for by DOE Order 251.1C, Departmental Directives Program. The Department accepts this recommendation, with a focus on driving the implementation of directive reforms to completion, identifying any training that needs to be provided, and soliciting feedback regarding further changes needed to ensure that the directives are accurate and effective. * Ensure that the plan developed for DOE's sites and contractors identifies how the reform effort will help address past and recurring safety concerns with quality assurance, safety culture, and federal oversight of contractor activities. The Department is fully committed to vital safety management elements, including quality assurance, safety culture, and federal oversight. The Department's internal directives on these topics, as well as numerous ongoing activities and communications, reflect this commitment. The Department believes that its directives provide appropriate expectations for quality assurance, safety management (including safety culture), and federal oversight. The Department fully agrees that effective implementation of these vital aspects of safety management need continuing emphasis. [See comment 14] The Department accepts this recommendation, in the context of its pursuit of continuous improvement in its safety management program through future planed updates of its directives and implementation of its operations experience program. * Clearly define the oversight roles and responsibilities of DOE's Office of Independent Oversight staff to ensure that their work is sufficiently independent from the activities of DOE site office and contractor staff. HSS has a strong independent oversight role that has served the Department and the taxpayers well for years. The Department remains firmly committed to an independent oversight function that is sufficiently independent from the DOE program and site offices it reviews. The Department believes this objective is being actively pursued and is adequately reflected in the approved Department directive, recent Secretarial communications, and actual practice. Nevertheless, the Department accepts this recommendation and will periodically reinforce these top-level management expectations for the roles and responsibilities of the DOE's Independent Oversight office, including the expectation for independence. [See comment 14] The following are GAO's comments on the Department of Energy's (DOE) letter dated April 9, 2012. GAO Comments: 1. DOE is incorrect that the draft report did not identify any significant flaws in the revised directives that resulted from the department's safety reform effort. On the contrary, the draft report discussed concerns that the Defense Nuclear Facilities Safety Board (Safety Board) and we had with revisions to DOE's Integrated Safety Management and independent oversight directives. In addition, DOE is confident that the directives will, among other things, effectively support mission requirements in a safe and efficient manner. In our view, however, DOE cannot demonstrate that the reform effort will result in benefits because it failed to develop outcome-based performance metrics. 2. Our draft report clearly described the role of the Safety Board in DOE's safety reform effort, including the Safety Board's concerns with DOE's revisions to its Integrated Safety Management directive. However, contrary to DOE's comment, Safety Board officials with whom we spoke stated that, during the reform process, they had expressed concerns about some of the changes DOE made to its safety directives, including its Integrated Safety Management and independent oversight directives. The Safety Board staff indicated that they did not believe those original concerns had been fully addressed and that they continue to have concerns about some of the changes made. Our draft report discussed these concerns, particularly as they relate to independent oversight. 3. Our draft report did not suggest requiring the department to seek public comment when revising its internal directives. Instead, our draft report clearly stated that DOE is not required to obtain the public's views on its internal directives. However, we did note that officials at NRC and OSHA--federal agencies with significant experience in nuclear and worker safety, respectively--told us that obtaining a wide range of comments on proposed revisions can be very helpful in developing quality requirements that balance cost with benefits and assist them to build support for the changes. We revised our report to further emphasize that DOE is not required to obtain public comments when making changes to its internal directives. However, we maintain that obtaining a wide range of comments from a variety of stakeholders is beneficial. 4. DOE agrees that it did not conduct an analysis of the costs and benefits of its requirements prior to revising them, stating that such an approach would be inconsistent with the department's approach to safety and to its management principles. In our view, DOE's comments indicate that the department's priority when revising its directives was to act quickly rather than to use the key practices of high- performing organizations when undertaking management improvement initiatives. As our draft report noted, high-performing organizations set clearly defined goals that are targeted levels of performance, expressed as tangible, measurable objectives against which achievement can be compared. DOE did not do this--as DOE itself notes in its comments--opting for speed instead of effectiveness. We maintain, as our draft report noted, that simply counting the number of directives revised or eliminated does not indicate the benefit of the reform on productivity and safety performance at DOE's sites and that safety should not be measured by the amount of paper that is saved but by actual improvements in safety performance. In our view, when it comes to rigorous safety requirements, the speed of a reform effort is far less important than the effectiveness of the reform. 5. DOE states that its safety reform resulted in more concise safety directives with requirements that are clearer and add value in protecting its workers and the public, and that the results are consistent with the goals established by the Deputy Secretary for the safety reform effort. As our draft report noted, however, one of the goals expressed by the Deputy Secretary in his March 2010 memorandum announcing the reform effort was that "measurable productivity improvements" would result from reducing burdensome safety requirements. Because DOE made no attempt to determine whether these requirements were burdensome--which DOE itself acknowledges in its comments (see comment 4)--and has no outcome-oriented metrics to assess the performance of the safety reform effort, there is no way for the department to demonstrate that measurable productivity improvements will result from its revisions to its safety directives. 6. DOE's comment that it does not measure success by the number of directives eliminated is not supported by the evidence. For example, the Deputy Secretary's March 2010 memorandum announcing the reform effort stated that the reform was expected to result in more than a 50 percent reduction in the number of directives. Furthermore, in numerous meetings, DOE officials cited examples of the accomplishments of the safety reform effort. In particular, during a November 2011 meeting, the Associate Deputy Secretary of Energy provided us with a copy of DOE's fiscal year 2011 "Management and Operational Excellence Performance Results" that cited the elimination of more than 1,650 pages from DOE directives, the cancellation of 22 directives, and the revision of 26 other directives as an accomplishment of the reform effort. 7. Our draft report did not assert that inadequate directive requirements caused a variety of performance weakness at DOE sites over the last several years. Our point was not that inadequate safety directives caused these weaknesses, but that DOE's revisions to these directives did not correct these weaknesses. We agree with DOE that safety performance problems are often not related to deficiencies in requirements, but rather in the implementation of the requirements. In our view, this increases the importance of ensuring directives are adequately implemented rather than spending inordinate time and staff energy revising those directives to eliminate "burdens" that have not been clearly defined. 8. DOE is correct that safety management and safety culture are different, but related, concepts. Nevertheless, as DOE itself acknowledges in its comments, the department has used the revisions to its Integrated Safety Management directives as one means to further clarify and strengthen expectations and requirements for safety culture. For example, our draft report stated DOE revised its Integrated Safety Management directives to, among other things, attempt to strengthen safety culture at its sites by more clearly emphasizing DOE and contractor senior management roles and responsibilities for building a strong safety culture at DOE's sites. Nevertheless, as our draft report noted, Safety Board officials raised concerns that some of these changes--in particular the removal of language that required contractors to follow the directives and allowing them to follow similar but less detailed requirements in DOE's acquisition regulations--could weaken safety culture at DOE's sites. However, to address DOE's concerns that the report does not adequately acknowledge the department's efforts to improve safety culture, we revised the report to add additional examples of recent DOE safety culture improvement initiatives. 9. DOE stated that it has been the department's long-standing practice to coordinate oversight activities with line organizations to ensure adequate planning and that this coordination is not a change from previous order expectations. However, the revised order states that "Independent Oversight appraisal activities must be coordinated with affected DOE line management and staff offices to promote efficient and effective use of resources."[Footnote 37] The previous order that the revised order replaced did not contain similar language. The previous order only required coordination between its oversight activities and DOE's sites on the development of corrective actions to any deficiencies found.[Footnote 38] Although both orders note that independent oversight activities must be independent of all DOE elements that have line management responsibilities for safety and security programs, Safety Board officials with whom we spoke stated that they were concerned that this revision would result in surprise "drop-in" assessments being lost, allowing site office and contractor staff to prepare ahead of the assessment. 10. We revised the report to discuss the establishment of the Nuclear Safety and Security Council. 11. As DOE itself acknowledges in its comments (see comment 4), the department did not analyze the costs and benefits of its requirements prior to revising them. Therefore, we maintain that the draft report's title is accurate. 12. Our draft report discussed in detail the process DOE followed in revising its directives, including the line-by-line review of each safety directive and its consultations with Safety Board staff, and we therefore made no changes to the report in response to this comment. 13. Our draft report noted that DOE has measures that it uses to monitor safety problems at its sites. However, the department has not been able to demonstrate how it could use these measures, if at all, to evaluate the benefits of its reform effort, nor has it developed other outcome-based metrics that could also be used. In our view, this is not consistent with one of the Deputy Secretary's goals for the reform effort--namely, that measurable improvements in productivity would result from the elimination of burdensome requirements while maintaining safety. Because DOE never defined "burdensome" through an analysis of costs and benefits and never developed outcome-based measures of the performance of the safety reform effort, we are left with merely DOE's assurances that improved productivity and safety performance will result rather than objective and verifiable data that could adequately demonstrate improvement. 14. We acknowledge DOE's acceptance of our recommendations and will monitor the department's implementation of them. [End of section] Appendix III: GAO Contact and Staff Acknowledgments: GAO Contact: Gene Aloise, (202) 512-3841 or aloisee@gao.gov: STAFF Acknowledgments: In addition to the individual named above, Ryan T. Coles, Assistant Director; Pamela Davidson; Carlos Gomez; Nancy Kintner-Meyer; Cheryl Peterson; Kevin Remondini; and Carol Herrnstadt Shulman made key contributions to this report. [End of section] Related GAO Products: 2012 Annual Report: Opportunities to Reduce Duplication, Overlap and Fragmentation, Achieve Savings, and Enhance Revenue. [hyperlink, http://www.gao.gov/products/GAO-12-342SP]. Washington, D.C.: February 28, 2012. National Nuclear Security Administration: Observations on NNSA's Management and Oversight of the Nuclear Security Enterprise. [hyperlink, http://www.gao.gov/products/GAO-12-473T]. Washington, D.C.: February 16, 2012. Excess Uranium Inventories: Clarifying DOE's Disposition Options Could Help Avoid Further Legal Violations. [hyperlink, http://www.gao.gov/products/GAO-11-846]. Washington, D.C.: September 26, 2011. Nuclear Material: DOE's Depleted Uranium Tails Could Be a Source of Revenue for the Government. [hyperlink, http://www.gao.gov/products/GAO-11-752T]. Washington, D.C.: June 13, 2011. Nuclear Waste: Disposal Challenges and Lessons Learned from Yucca Mountain. [hyperlink, http://www.gao.gov/products/GAO-11-731T]. Washington, D.C.: June 1, 2011. DOE Nuclear Waste: Better Information Needed on Waste Storage at DOE Sites as a Result of Yucca Mountain Shutdown. [hyperlink, http://www.gao.gov/products/GAO-11-230]. Washington, D.C.: March 23, 2011. Nuclear Waste: DOE Needs a Comprehensive Strategy and Guidance on Computer Models that Support Environmental Cleanup Decisions. [hyperlink, http://www.gao.gov/products/GAO-11-143]. Washington, D.C.: February 10, 2011. Nuclear Waste: Actions Needed to Address Persistent Concerns with Efforts to Close Underground Radioactive Waste Tanks at DOE's Savannah River Site. [hyperlink, http://www.gao.gov/products/GAO-10-816]. Washington, D.C.: September 14, 2010. Recovery Act: Most DOE Cleanup Projects Appear to Be Meeting Cost and Schedule Targets, but Assessing Impact of Spending Remains a Challenge. [hyperlink, http://www.gao.gov/products/GAO-10-784]. Washington, D.C.: July 29, 2010. Department of Energy: Actions Needed to Develop High-Quality Cost Estimates for Construction and Environmental Cleanup Projects. [hyperlink, http://www.gao.gov/products/GAO-10-199]. Washington, D.C.: January 14, 2010. Nuclear Waste: Uncertainties and Questions about Costs and Risks Persist with DOE's Tank Waste Cleanup Strategy at Hanford. [hyperlink, http://www.gao.gov/products/GAO-09-913]. Washington, D.C.: September 30, 2009. Department of Energy: Contract and Project Management Concerns at the National Nuclear Security Administration and Office of Environmental Management. [hyperlink, http://www.gao.gov/products/GAO-09-406T]. Washington, D.C.: March 4, 2009. Nuclear Waste: DOE Lacks Critical Information Needed to Assess Its Tank Management Strategy at Hanford. [hyperlink, http://www.gao.gov/products/GAO-08-793]. Washington, D.C.: June 30, 2008. Hanford Waste Treatment Plant: Department of Energy Needs to Strengthen Controls over Contractor Payments and Project Assets. [hyperlink, http://www.gao.gov/products/GAO-07-888]. Washington, D.C.: July 20, 2007. Nuclear Waste: DOE Should Reassess Whether the Bulk Vitrification Demonstration Project at Its Hanford Site Is Still Needed to Treat Radioactive Waste. [hyperlink, http://www.gao.gov/products/GAO-07-762]. Washington, D.C.: June 12, 2007. Hanford Waste Treatment Plant: Contractor and DOE Management Problems Have Led to Higher Costs, Construction Delays, and Safety Concerns. [hyperlink, http://www.gao.gov/products/GAO-06-602T]. Washington, D.C.: April 6, 2006. Nuclear Waste: Absence of Key Management Reforms on Hanford's Cleanup Project Adds to Challenges of Achieving Cost and Schedule Goals. [hyperlink, http://www.gao.gov/products/GAO-04-611]. Washington, D.C.: June 9, 2004. Nuclear Waste: Challenges to Achieving Potential Savings in DOE's High- Level Waste Cleanup Program. [hyperlink, http://www.gao.gov/products/GAO-03-593]. Washington, D.C.: June 17, 2003. Nuclear Waste: Department of Energy's Hanford Tank Waste Project-- Schedule, Cost, and Management Issues. [hyperlink, http://www.gao.gov/products/GAO-RCED-99-13]. Washington, D.C.: October 8, 1998. [End of section] Footnotes: [1] Nuclear facilities include research and test reactors; facilities storing special nuclear materials, including plutonium; facilities that perform laboratory experiments on nuclear materials; and nuclear waste processing facilities. Nonnuclear facilities include office and administration buildings, emergency management facilities, and facilities performing experiments on chemical and other hazardous materials. [2] Federal safety laws and regulations include 10 C.F.R. § 830, which addresses nuclear safety; 10 C.F.R. § 835, which addresses radiological safety; 10 C.F.R. § 851, which addresses worker safety; and 48 C.F.R. § 970.5223-1, which addresses integrated safety management. Contractor compliance with federal laws and regulations is mandatory and enforceable under law. For directives, DOE incorporates these into contracts. [3] GAO, Nuclear and Worker Safety: Actions Needed to Determine the Effectiveness of Safety Improvement Efforts at NNSA's Weapons Laboratories, [hyperlink, http://www.gao.gov/products/GAO-08-73] (Washington, D.C.: Oct. 31, 2007). [4] Revisions to DOE's security directives under its reform effort will be the subject of a follow-on review. [5] Because this was a nonprobability sample of DOE's sites, the information we collected is not generalizeable to all of DOE's sites; however, this information is illustrative of various stages of the implementation of the revised safety requirements. [6] Performance measures allow an agency to track the progress it is making toward its mission and goals, provide managers information on which to base their organizational and management decisions, and create powerful incentives to influence organizational and individual behavior. [7] DOE regulates the safety of most of its own sites with nuclear operations, NRC regulates several DOE nuclear facilities, and OSHA regulates occupational safety at DOE sites that have no nuclear functions. [8] For purposes of this report, we refer to all these components as directives. [9] DOE's Office of Environmental Management is responsible for the treatment and disposal of radioactive and hazardous waste created as a by-product of nuclear weapons production and energy research. [10] During our review, DOE reorganized offices within the Office of Health, Safety, and Security. The Office of Independent Oversight merged with the Office of Enforcement and was renamed the Office of Enforcement and Oversight. For the purposes of this report, we refer to it as the Office of Independent Oversight. [11] As discussed previously, DOE's reform effort addressed directives in both the safety and security areas. This report focuses on the effort's reform of DOE's safety directives. [12] The use of standards and requirements developed by private standards-making organizations is required for federal agencies. The Office of Management and Budget's Circular A-119 directs government agencies to use voluntary consensus standards in lieu of government- unique standards except where they are inconsistent with law or are otherwise impractical. Voluntary consensus standards are developed by domestic or international organizations that use consensus to adopt agreed-upon procedures. Other standards, such as industry standards, are developed by private sector organizations, but not using a consensus process. An example of a standards-making organization is the International Standards Organization. [13] DOE, DOE Corporate Operating Experience Program, DOE Order 210.2 (Washington, D.C.: June 12, 2006). [14] DOE, Federal Technical Capability Policy for Defense Nuclear Facilities, DOE Policy 426.1 (Washington, D.C.: Oct. 10, 1998). [15] DOE, Quality Assurance, DOE Order 414.1C (Washington, D.C.: June 17, 2005). [16] DOE, Implementation of Department of Energy Oversight Policy, DOE Order 226.1A (Washington, D.C.: July 31, 2007). [17] DOE, Accident Investigations, DOE Order 225.1A (Washington, D.C.: Nov. 26, 1997). [18] On January 18, 2011, the President issued Executive Order 13563, Improving Regulation and Regulatory Review, to ensure that federal regulations seek more affordable, less intrusive means to achieve policy goals and that agencies give careful consideration to the benefits and costs of those regulations. Exec. Order No. 13,563, 76 Fed. Reg. 3821 (Jan. 21, 2011). [19] National Laboratory Directors Council, NLDC Prioritization of Burdensome Policies and Practices, (May 31, 2011). The study discussed a list of 18 policies and practices identified as the most burdensome by senior managers at each of DOE's 18 national laboratories. The four safety-related burdensome policies included: (1) excessive oversight through a "piling on" of audits and assessments without clear risk prioritization, coordination, or value; (2) unnecessary requirements in DOE's Worker Safety and Health Program, which establishes the requirements for a worker safety and health program that reduces or prevents occupational injuries, illnesses, and accidental losses, among other things, under 10 C.F.R. § 851; (3) maintaining directives that add no value, such as the quality assurance order, and become audit-bait for overseers; and (4) requiring unnecessary reporting on safety and other issues. [20] NNSA, Improving Nuclear Safety Requirements--Discussions with NNSA and Contractor and Site-Office Management (Washington, D.C.: Apr. 14, 2011). [21] GAO, Contract Reform: DOE Has Made Progress, but Actions Needed to Ensure Initiatives Have Improved Results, [hyperlink, http://www.gao.gov/products/GAO-02-798] (Washington, D.C.: Sept. 13, 2002). [22] DOE has formal processes for incorporating directives revisions into existing contracts in its DOE Order 251.1C. This order, however, does not include milestones, specific tasks, or roles and responsibilities for implementing the revised safety directives. [23] DOE, Guidelines for Performance Measurement, DOE G 120.1-5 (Washington, D.C.: June 30, 1996). [24] Defense Nuclear Facilities Safety Board, Recommendation 2002-1 to the Secretary of Energy: Quality Assurance for Safety-Related Software (Washington, D.C.: Sept. 23, 2002). [25] GAO, Nuclear Waste: DOE Needs a Comprehensive Strategy and Guidance on Computer Models that Support Environmental Cleanup Decisions, [hyperlink, http://www.gao.gov/products/GAO-11-143] (Washington, D.C.: Feb. 10, 2011). [26] DOE, Office of Inspector General, Implementation of Beryllium Controls at Lawrence Livermore National Laboratory, DOE/IG-0851 (Washington, D.C.: June 17, 2011). [27] DOE's regulatory requirements for promoting safety culture for contractors are contained in DOE's acquisition regulation 48 C.F.R. § 970.5223-1, Integration of Environment, Safety and Health Into Work Planning, and Execution. The regulation requires that contracts provide for contractors to comply with all applicable laws and regulations. The original set of DOE's Integrated Safety Management directives provided supplemental requirements and best practices for contractors to follow in order to implement a strong safety culture. [28] Defense Nuclear Facilities Safety Board, Recommendation 1995-2 to the Secretary of Energy: Safety Management (Washington, D.C.: Sept. 28, 1998), and Recommendation 1998-1 to the Secretary of Energy: Integrated Safety Management at the Department of Energy (DOE) Facilities (Washington, D.C.: Sept. 28, 1998). [29] [hyperlink, http://www.gao.gov/products/GAO-08-73]. [30] DOE, Office of Inspector General, Fire Suppression and Related Services at Los Alamos National Laboratory, DOE/IG-0821 (Washington, D.C.: Sept. 11, 2009). [31] Defense Nuclear Facilities Safety Board, Recommendation 2009-2 to the Secretary of Energy: Los Alamos National Laboratory Plutonium Facility Seismic Safety (Washington, D.C.: Oct. 26, 2009). [32] Defense Nuclear Facilities Safety Board, Recommendation 2011-1 to the Secretary of Energy: Safety Culture at the Waste Treatment and Immobilization Plant (Washington, D.C.: June 9, 2011). [33] DOE, Office of Health, Safety, and Security, Independent Review of Nuclear Safety Culture at the Hanford Site Waste Treatment and Immobilization Plant Project (Washington, D.C.: October 2010). [34] GAO, Nuclear Safety: Department of Energy Needs to Strengthen Its Independent Oversight of Nuclear Facilities and Operations, [hyperlink, http://www.gao.gov/products/GAO-09-61] (Washington, D.C.: Oct. 23, 2008). [35] [hyperlink, http://www.gao.gov/products/GAO-09-61]. [36] DOE, Office of Inspector General, Implementation of Beryllium Controls at Lawrence Livermore National Laboratory, DOE/IG-0851 (Washington, D.C.: June 17, 2011). [37] DOE, Independent Oversight Program, DOE Order 227.1 (Washington, D.C.: Aug. 30, 2011). [38] DOE, Independent Oversight and Performance Assurance Program, DOE Order 470.2B (Washington, D.C.: Oct. 31, 2002). [End of section] GAO’s Mission: The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO’s website [hyperlink, http://www.gao.gov]. Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, go to [hyperlink, http://www.gao.gov] and select “E- mail Updates.” Order by Phone: The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s website, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO: Connect with GAO on facebook, flickr, twitter, and YouTube. Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts. Visit GAO on the web at [hyperlink, http://www.gao.gov]. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; E-mail: fraudnet@gao.gov; Automated answering system: (800) 424-5454 or (202) 512-7470. Congressional Relations: Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-4400 U.S. Government Accountability Office, 441 G Street NW, Room 7125 Washington, DC 20548. Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800 U.S. Government Accountability Office, 441 G Street NW, Room 7149 Washington, DC 20548.