Findings From Prior GAO Work

GAO has issued several reports over the last decade that shed light on the various ways agencies may conduct retrospective reviews, the outcomes of such reviews, and factors that may impede or facilitate these reviews (see the Related GAO Products tab for more information). For example, GAO's 2007 report on the retrospective review activities of nine federal agencies from 2001 to 2006 revealed that agencies were conducting more retrospective reviews of regulations, and a greater variety of reviews, than was commonly understood.[1] Although agencies conduct retrospective reviews in response to various mandatory requirements, most agencies conduct the majority of these reviews at their own initiative, often to determine how effectively their regulations have been implemented.

How have retrospective reviews been conducted and how do evaluation practices vary among agencies?

Retrospective reviews of regulations, which can have a range of purposes, are conducted in various ways. In our 2007 report on opportunities to improve retrospective reviews, we reported that the use of systematic evaluation practices and the development of formal retrospective regulatory review processes varied among and even within the agencies. Additionally, there was variation among and within agencies as to whether regulations were reviewed by the program offices responsible for the regulation or a centralized office. GAO reviewed how selected agencies chose the regulations to be reviewed, conducted the reviews, and reported the results. GAO identified three practices that were important to ensuring the credibility of these reviews: (1) using standards to guide review processes, (2) involving the public, and (3) documenting the process and results of each phase of the review. However, the use of these evaluation practices in agencies' review processes, and the development of formal policies and procedures to guide their review processes, varied among the agencies. Furthermore, agencies' use of these practices often varied according to whether they conducted discretionary or mandatory reviews.

What have been the outcomes of retrospective reviews? Back to top

Agency officials reported that reviews of existing regulations have resulted in various outcomes including

  • changes to regulations;
  • changes or additions to guidance and other related documents;
  • decisions to conduct additional studies;
  • validation that existing regulations were working as planned;
  • determinations that statutory changes were needed; and
  • determinations of cost savings to the agency and the regulated community.

For example, in 1999, GAO reported that Environmental Protection Agency (EPA) officials and other experts stated that retrospective reviews provided insights into a new, market-based regulatory approach to reduce emissions that cause acid rain.[2] These reviews found that the actual costs of reducing emissions were lower than initially estimated. Based on EPA's findings, a legislative proposal to further limit emissions was introduced. In addition, GAO reported in 2005 on a workshop on economic performance measures that GAO held in 2004.[3] At that workshop, GAO presented, for discussion purposes, an example of a retrospective review of the Occupational Safety and Health Administration's scaffold standards. That discussion indicated that the program's actual net benefits, while still positive, were significantly less than expected when the standards were proposed. Retrospective reviews, therefore, can be useful to decision makers by providing information on whether a program, as implemented, has produced the predicted benefits and costs and whether the benefits have justified the costs. In our 2007 report, we found that some agency officials believed that retrospective reviews had resulted in cost savings to their agencies and to regulated parties. Agencies and nonfederal parties (such as academics and business leaders) generally found all of the various reviews useful.

What factors may impede the conduct and usefulness of retrospective reviews?Back to top

In 2007, GAO reported that agencies and nonfederal parties (academia, business, public advocacy, and state government) identified multiple factors that may impede the conduct and usefulness of retrospective reviews.

Competing Priorities and Overlapping Reviews

In our 2007 report, we evaluated the activities of nine agencies covering health, safety, environmental, financial, and economic regulations. The agencies reported that, because of competing mission-critical demands, they were required to respond to other legislative priorities that limited their retrospective review activities. In addition, overlapping or duplicative reviews may have further strained their resources. For example, although some agencies had already reviewed and modified existing regulations, they had to expend additional time and resources to comply with predetermined or generic schedules for retrospective reviews and review factors, which led to reviews that were duplicative and less useful. Of note, agencies indicated that their discretionary reviews generated additional action more often than mandatory reviews. Given the trade-offs agencies face in allocating resources, we concluded that it makes sense to consider the appropriate mix of mandatory and discretionary reviews, and other mission-critical activities. Moreover, participants at a GAO workshop on economic performance analyses indicated that barriers to economic performance analyses may include agencies' lack of resources and limited demand from decision makers for benefit-cost analyses. Finally, in 1999, EPA reported focusing its limited resources on prospectively estimating the future costs and benefits of feasible alternatives rather than conducting retrospective reviews.

Data Limitations

GAO has also reported that the availability of data can hinder or facilitate retrospective regulatory reviews. In 2007, GAO reported that, according to agency officials, a major barrier to conducting effective reviews is a lack of baseline data. In some cases, this lack of data may be caused by a lack of consideration, during the promulgation of certain new regulations, of whether and how the agency would measure the performance of the regulation and how it would collect the data needed to do so. In addition, agencies reported difficulties in obtaining data on their regulations because of the length of time it takes to see the impact of some regulations and the scarcity of data related to the areas that they regulate. Agencies and nonfederal parties also considered requirements of the Paperwork Reduction Act (PRA) to be a potential limiting factor in agencies' ability to collect sufficient data to assess their regulations. The PRA established standards and an OMB approval process to, among other purposes, ensure that agencies minimize the paperwork burden on the public when collecting data. Agencies and nonfederal parties generally agreed that the act was an important consideration in agency data collection.

Further, in 2007, GAO convened a Comptroller General's Forum on financial regulation and reported on the difficulties in the financial services industry of measuring the costs and benefits of regulations, and the potential burden on U.S. financial services firms.[4] The forum also highlighted the difficulty of separating the costs of complying with regulations from other costs, such as costs related to risk management or recordkeeping. For example, GAO reported that banks often could not separate out costs related directly to the implementation of capital requirements and risk management techniques, because the systems for collecting this information often served multiple purposes, including reporting for many kinds of regulations and for internal, risk management purposes. GAO's work in assessing EPA's regulations has also pointed out the difficulties in obtaining valid cost data and quantifying benefits.

Overly Broad Review Scope

Various agencies included in GAO's 2007 report identified the broad scope of many retrospective reviews as a barrier to the usefulness of these reviews. Officials reported significant delays in completing reviews and making timely modifications, as well as in obtaining meaningful input when reviews involved multiple regulations. Similarly, agency officials reported that, when reviews were scoped too broadly, it also affected their ability to conduct the reviews expediently. Agencies' officials suggested that having a narrow and focused unit of analysis, such as a specific standard or regulation, is a more effective approach to conducting retrospective reviews quickly.

Lack of Transparency and Limited Public Participation

In our 2007 report, we stated that federal agencies cited limited public participation as a barrier to the usefulness of retrospective reviews of regulations. Nonfederal parties that GAO consulted during this work were mostly unaware of agency retrospective reviews and cited a lack of transparency in agencies' review processes, results, and related follow-up activities as a barrier to the usefulness of such reviews to the public. Some mandatory reviews do not require public reporting, and many agencies did not consistently report the results of their discretionary reviews, especially if the reviews resulted in no changes to regulations. In addition, agencies stated that, despite extensive outreach efforts, they received very little participation from the public in the review process. Some nonfederal parties told GAO that a lack of transparency was the primary reason for the lack of public participation in agencies' retrospective review processes.

Statutory Constraints

In our 2007 report, agency officials indicated that statutory requirements were a major barrier to modifying or eliminating regulations in response to retrospective regulatory reviews. This is because some agency-specific statutes direct agencies to regulate in a certain manner. In such instances, agencies may have no authority to take a different approach. Similarly, in our 1999 report on regulatory burden, we found that agencies often had no discretion, because of statutory provisions, when they imposed requirements that businesses reported as burdensome.

Gaps in the Use and Quality of Benefit-Cost Analyses

Participants in GAO's 2004 workshop on the use of benefit-cost or cost-effectiveness analyses in federal programs identified gaps in the use of such analyses, including the lack of mechanisms for retrospective reviews of regulations. Workshop participants indicated that evaluating existing programs, including regulatory programs, retrospectively provides an opportunity to expand the use of benefit-cost or cost-effectiveness analyses. Participants nonetheless cited some challenges with the quality and use of these analyses and said that, while agencies have improved such analyses, the quality of analysis is highly variable. In addition, workshop participants reported that there is not agreement on how to quantify some benefits, such as the reduced risk of mortality.

What factors facilitate the conduct and usefulness of retrospective reviews?Back to top

In GAO's 2007 report, agencies and nonfederal parties identified lessons learned and methods of improving retrospective reviews.

Pre-planning

Agencies and nonfederal parties identified pre-planning of retrospective reviews as a practice that aids agencies in identifying the data and analyses needed to conduct effective outcome-based performance reviews. Some agencies included in GAO's 2007 report planned how they would collect performance data on their regulations before or during the promulgation of the relevant regulations or prior to the reviews. They cited this technique as a method for reducing data collection barriers. Furthermore, we observed that pre-planning for data collection could address some challenges agencies reported facing, such as the length of time required to obtain OMB approval under the PRA.

Prioritizing

Agencies reported that prioritizing which regulations to review facilitated the conduct of and improved the usefulness of their reviews. Agencies that developed programs with detailed processes for prioritizing which regulations to review reported that prioritization facilitated their ability to address time and resource barriers and allowed them to target their efforts to more useful reviews of regulations that were likely to need modification. Officials from two agencies reported that their prioritization processes allowed them to focus on the most useful retrospective review activities. Nonfederal parties also asserted that it is not necessary or even desirable for agencies to spend time and resources reviewing all regulations. Instead, they reported that it would be more efficient and valuable for agencies to conduct substantive reviews of a small number of regulations that agencies and the public identify as needing attention. Nonfederal parties and agency officials suggested that factors for agencies to consider when prioritizing their review activities could include economic impact, risk, public feedback, and length of time since the last review of the regulation, among other things.

Assuring Independent Review

Nonfederal parties believed that reviews would be more credible and effective if the parties that conduct them were independent. However, they recognized that it is important to include input from those involved in day-to-day implementation of the regulations and those responsible for producing initial benefit-cost estimates for the regulations. Almost all of the nonfederal parties GAO interviewed expressed concern that agency officials who promulgated and implemented regulations were the same officials responsible for evaluating the performance of these regulations. Although the nonfederal parties acknowledged that it is important for officials with critical knowledge about the program to be involved in retrospective reviews, they were concerned that these officials may not be objective. Nonfederal parties also expressed concerns about agencies' capacity to conduct certain types of analyses for their reviews, such as benefit-cost analyses. The nonfederal parties suggested that agencies consider having an independent body, such as another agency, Inspector General, or a centralized office within the agency, conduct the reviews. Agency officials reported that they sometimes contract out these reviews if they do not have the expertise needed to conduct the analyses. However, agency officials told us it was difficult to find a knowledgeable, independent body to conduct retrospective reviews, and they noted that even contracted reviewers may not be considered fully independent because they are paid by the agencies to conduct the studies.

Obtaining High-Level Management Support

Agencies and nonfederal parties agreed that high-level management support of the review process is important not only to the successful implementation of individual reviews, but also to sustaining the agency's commitment to a review program and following up on the results of the retrospective reviews. Almost all of the selected agencies reported receiving some high-level management attention of their reviews, but where and how they used this involvement varied. For example, almost all of the selected agencies reported receiving some high-level management attention to the decision-making processes that resulted from their retrospective reviews. However, only two of the nine agencies involved high-level managers early in their decision-making processes to determine which regulations to review. Overall, agencies and nonfederal parties indicated that having high-level management attention is important in obtaining and sustaining the resources needed to conduct these reviews and ensuring the credibility of the reviews.

Grouping Related Regulations

While an overly broad scope may cause a delay in conducting a retrospective review, some agency officials stated that grouping related regulations together was a technique that more often generated meaningful comments and suggestions from the public. For example, officials from one agency stated that categorizing regulations for review proved to be a more effective way of receiving public input than requesting that each regulation be reviewed individually. Officials from another agency reported that reviewing related regulations together can be an effective technique because a change to one regulation can have an impact on related regulations. However, one agency official cautioned that, while grouping regulations is an effective technique for obtaining useful public input, defining the categories too broadly can lead to an effort that is too intensive and may be less convenient and practical for agencies that write very specific standards. For those agencies, it may be more effective to pick related characteristics of the regulations in order to group them for review. Similarly, a 2010 GAO report on financial regulation discussed efforts to harmonize regulatory approaches to address concerns about overlap, gaps, and inconsistencies in oversight of certain financial products.[5]

Targeting Products to Meet Needs of Various Audiences

Nonfederal parties suggested that agencies need to be more aware of the different audiences that might be interested in their retrospective reviews, and target the level of detail and type of product to be used in reporting the results to meet the needs of these audiences. Further, both agency officials and nonfederal parties identified methods for improving communication, including better use of information technology tools, such as agency Web sites, electronic docket systems, e-mail listserves, Web-based forums, or other Web-based technologies.

GAO's 2007 report made a number of recommendations for improving the transparency and effectiveness of agencies' retrospective reviews. See the Related GAO Recommendations tab for more information on these recommendations.

Back to top

[1]GAO-07-791.

[2]GAO, Environmental Protection: Assessing the Impacts of EPA's Regulations Through Retrospective Studies, GAO/RCED-99-250 (Washington, D.C.: Sept. 14, 1999).

[3]GAO, Economic Performance: Highlights of a Workshop on Economic Performance Measures, GAO-05-796SP (Washington, D.C.: July 2005).

[4]GAO, Financial Regulation: Industry Trends Continue to Challenge the Federal Regulatory Structure, GAO-08-32 (Washington, D.C.: Oct. 12, 2007).

[5]GAO, Financial Regulation: Clearer Goals and Reporting Requirements Could Enhance Efforts by CFTC and SEC to Harmonize Their Regulatory Approaches, GAO-10-410 (Washington, D.C.: Apr. 22, 2010).