|Page last updated April 9, 2001|
We emphasize the importance of reliable and valid information in our work through (1) standards, policies, and procedures; (2) management's use of performance information; and (3) independent reviews of our work.
First, our standards, policies, and procedures provide guidance on assessing the reliability and validity of performance information. Specifically, our Government Auditing Standards (often referred to as generally accepted government auditing standards) contain the core standards governing our work. Our Policy and Procedures Manual provides additional guidance, including procedures on verifying and validating the information used in specific performance measures. We reinforce the content and application of these standards, policies, and procedures by training all staff conducting GAO work.
Second, management's use of our performance information on a routine basis further helps to ensure its reliability and validity. Data are provided to managers for decision-making, and their feedback on these data helps to ensure that the data are properly recorded.
Third, additional reviews--conducted by both internal and external groups--help ensure that all of our work is consistent with generally accepted government auditing standards and with our policies and procedures. We are also in the process of identifying possible external entities to conduct a peer review of our performance auditing. These reviews include the following:
The sections below cover
Data Limitations and Responses
Generally, our measures are better suited to examining trends in performance over a number of years than to making conclusions about our overall level of performance in any given year. We rely on trends for several reasons. The benefits our work produces may not be realized for a number of years because of the complexity of issues we address and the schedules during which the Congress and the executive branch may act on our recommendations. Also, opportunities to produce benefits vary and can influence the volume of accomplishments recorded in any given year.
To provide a clear indication of trends, we report results that are averaged over a 4-year period. However, we also report yearly totals to allow comparisons between any 2 specific years and to more readily identify underlying factors affecting trends. We also ensure that our criteria are consistently applied each year to allow these comparisons. In addition, because a simple enumeration of our performance does not adequately capture the breadth and depth of our work, we provide each year a qualitative assessment of the extent to which we have successfully met our multiyear performance goals. This assessment considers how our work contributes to the potential outcomes identified in our strategic objective plans.
Current Performance Measures
Background and Context: Our findings and recommendations directly or indirectly contribute to congressional decision-making and executive branch actions that result in significant financial benefits to taxpayers. These benefits include budget reductions, costs avoided, resources reallocated, and revenue enhancements that are documented as either directly attributable to, or significantly influenced by, our work. The funds made available in response to our findings and recommendations may be used to reduce government expenditures or may be reinvested in other areas.
Data Limitations: Not every financial benefit from our work can be readily estimated or directly attributed to GAO. Estimates are based on both objective and subjective data, and as a result, judgment is required. We use data provided by an agency or an independent third party--such as the Congressional Budget Office or a congressional committee--to make our estimates. Moreover, GAO's policy requires conservative estimation of financial benefits. Therefore, we believe that the total of the estimated benefits from our findings and recommendations understates our overall contribution to congressional decision-making and executive branch actions.
Verification/Validation: Policies and procedures guide the estimation of financial benefits and their attribution to GAO. We require estimates to be based on independent sources, reduced by any identifiable offsetting costs, and limited to the first 2 years of implementation. Benefits are estimated in internal written reports that are formally reviewed to ensure they meet the same documentation and quality standards as any external GAO product. In addition, our Quality and Risk Management office reviews benefit claims in excess of $100 million, and our Office of the Inspector General reviews claims in excess of $1 billion. Benefits are revised if new information significantly affects the estimated values.
Data Sources: Internal listing of accomplishment reports.
Background and Context: Our findings and recommendations also contribute to congressional decision-making and executive branch actions that result in significant improvements to agency management or performance--for example, by strengthening internal control processes--but do not have directly measurable financial benefits. This measure is the number of actions that the Congress or agencies have taken in response to our findings and recommendations.
Data Limitations: Other benefits vary in significance. Also, because not all benefits can be directly attributed to our findings and recommendations or documented, this measure understates our overall contribution toward improving government.
Verification/Validation: Policies and procedures require internally written reports to record the other benefits of our findings and recommendations. These reports are formally reviewed within GAO to ensure the appropriateness of the claimed accomplishment, including attribution to GAO's work. These reports must meet the same documentation and quality standards as any GAO product.
Data Sources: Internal listing of accomplishment reports.
Background and Context: As part of our audit responsibilities under generally accepted government auditing standards, we follow up and report yearly to the Congress on the status of actions taken by the Congress and agencies in response to our recommendations. This measure is the percentage rate of implementation of recommendations made 4 years prior to a given fiscal year. For example, the fiscal year 2002 implementation rate is the percentage of recommendations made in fiscal year 1998 that were implemented by fiscal year 2002. Prior experience has shown that if a recommendation has not been implemented within 4 years, it is not likely to be implemented.
Data Limitations: Because the measure is based on the implementation of recommendations made 4 years prior to any given fiscal year, the measured value for any given year will not reflect the results of GAO's activities undertaken within that year. In addition, this measure may not include all actions proposed or initiated by agencies. Specifically, agencies may report actions in response to our recommendations, but we may determine that these actions are insufficient or do not adequately implement our recommendations. In these cases, recommendations will be recorded as not implemented even though the agency has proposed or taken some actions.
Verification/Validation: GAO policies and procedures specify that staff must verify with sufficient supporting documentation that an agency's reported actions are adequately being implemented. Our staff may interview agency officials, obtain agency documents, access agency databases, or obtain information from the agency's Office of the Inspector General. Internal review procedures are intended to ensure that claims regarding the implementation of our recommendations are consistent and meet our quality requirements. Information on recommendations implemented is maintained in a database managed by an external contractor that routinely conducts software-based checks of data consistency and completeness and annually performs more exhaustive checks for data integrity.
Data Sources: The percentage of recommendations implemented is derived from GAO's document database. Information entered into the database is collected through our recommendation follow-up system.
Background and Context: The Congress may request that GAO testify at hearings on various issues. Testimony is one of our most important forms of communication with the Congress, and the total number of testimonies reflects the importance and value of our institutional knowledge in assisting congressional decision-making.
Data Limitations: The number of testimonies in any given year may reflect congressional interest in work completed that year, in the previous year, and work in progress. Additionally, the number each year depends on the Congress's agenda. Therefore, year-to-year variations in the total number of testimonies may be influenced by factors other than the quality of our performance in any specific year.
Verification/Validation: Teams are responsible for notifying GAO's Congressional Relations office of upcoming hearings. Notices of these hearings are entered into a tracking system. Staff are assigned responsibility for monitoring the progress and status of planned hearings within their teams.
Data Sources: Internal listing of hearings planned and held.
Background and Context: Recommendations in our products help to ensure that benefits will result from our work. These recommendations reflect specific actions that can be taken to improve federal programs. Where appropriate, we strive for recommendations that are directed at resolving the cause of identified problems; that are addressed to parties who have the authority to act; and that are specific, feasible, and cost-effective to the extent practical.
Data Limitations: We provide a variety of products and services that meet the needs of our congressional clients but may not lead to recommendations. For example, the Congress may require descriptive information on federal programs or analyses of the potential consequences of alternative program design options. This information is intended to assist the Congress in its oversight of federal agencies or in its formulation of policy and legislation but does not lend itself to recommendations. Consequently, this measure underestimates the extent to which GAO assists the Congress and federal agencies.
Verification/Validation: An external contractor reviews all GAO products distributed through a formal process, prepares summaries that identify products containing recommendations, and verifies this information through our recommendation follow-up system. Also, GAO managers are provided with reports on the recommendations being tracked to help ensure that the contractor has correctly identified the recommendations contained in reports.
Data Sources: GAO's document database.
Background and Context: The likelihood that GAO's products will be used is enhanced if they are produced when needed to support congressional and agency decision-making about government programs. We monitor the extent to which our products are completed by dates agreed to with our clients. This measure is the proportion of GAO's products that are issued by the date to which we have formally committed. In our revised final fiscal year 2001 performance plan, we made the timeliness measure an overall GAO measure. This will provide greater focus and emphasis on meeting client needs.
Data Limitations: We measure the timeliness of key external products but exclude internal products.
Verification/Validation: Aggregate and job-specific timeliness data are given to managers monthly, who advise of any anomalies. The software used to prepare the monthly reports is verified by comparing job-specific detail from the reports with the same detail on original data files maintained by an external contractor. At job completion, data on job target and completion dates are reported to the manager, who reviews and signs the report to confirm its accuracy.
Data Sources: Our automated Mission and Assignment Tracking System, which is used to monitor job progress on an ongoing basis.
Qualitative Performance Measures
Background and Context: Our work is of value to different audiences and is used in a variety of ways that may not be reflected in our quantitative performance measures. For this reason, we complement our quantitative goals with qualitative goals to be achieved over a 3-year period, currently from fiscal year 2000 through fiscal year 2002. These goals are listed in appendixes I through IV, by strategic goal and objective. The qualitative measures are whether we meet, exceed, or fail to meet the performance goals for which results are assessed every 3 years. Our performance meets expectations when we provide information or make recommendations on the key efforts when viewed collectively. Our performance exceeds expectations when we provide information or make recommendations that congressional decisionmakers and others use toward achieving the potential outcomes described in the relevant strategic objective plan. The key efforts and potential outcomes are listed in the relevant strategic objective plans covering fiscal years 2000 through 2002. Gauges of "use" include, among other things, congressional decisionmakers' requests for other support, such as assisting in the development of oversight agendas, commenting on bills, helping to craft hearings, or providing questions for deliberations; citations in congressional documents, such as bills, laws, committee reports, or the Congressional Record; and information showing how agencies use our products. We also plan to develop a congressional feedback system and to track references to our work by the media, universities, and other organizations. We have provided an assessment of progress toward these goals in this report and will do so again next year. Our performance report for fiscal year 2002 will provide a final assessment of the extent to which performance has met, exceeded, or failed to meet expectations for each of these goals over the 3-year period.
Data Limitations: Because our use of qualitative goals is new, we do not yet have sufficient experience to determine their limitations. Success will depend upon the continued refinement of the goals, definitions of key terms, and standards for making assessments.
Verification/Validation: The assessments of progress against each 3-year goal will be supported by specific examples in internal written reports, receive formal internal review, and meet the same documentation and quality standards as any external GAO product. In addition, GAO's Quality and Risk Management office will review the reports for consistency and ensure that requirements are met.
Revising GAO's Performance Measures
In fiscal year 2001, as laid out in our strategic plan, we began assessing our performance measures and, as appropriate, began developing new measures to track progress toward our strategic goals and annual targets. We believe this assessment is appropriate, given that we have a new strategic plan, a newly realigned organization that is consistent with our strategic goals and objectives, and two new management strategies to improve how we accept and manage engagements. The assessment will focus on how we can establish a "balanced scorecard" of performance measures to evaluate our performance in three key areas:
To move to a balanced scorecard, GAO will be developing new measures based on client feedback and our performance in key human capital activities. For the "results" category, we will refine a number of our current measures, for example, financial benefits, other benefits, and timeliness. Most of the new measures and the refinements to existing measures are scheduled to be implemented in fiscal year 2002, which means we will report results in our fiscal year 2002 performance report.
Quality Review Scores
Our final performance plan for fiscal year 2001 included a performance measure of how well GAO's products adhered to applicable auditing policies and procedures. At the end of the second quarter of fiscal year 2000, however--after the plan was published--we discontinued this measure. As a post-issuance measure, it did not reflect the evolving environment at GAO, where the increased emphasis on risk and matrix management requires building quality into our work and products prior to issuance. These management strategies emphasize a more proactive and systematic method to ensure that our products comply with GAO's core values, applicable professional standards, and reporting requirements. In addition, while this measure attempted to put a quantitative value on quality, the results were very similar for all of our operating units, meaning that it was not very useful in our efforts to improve our products and increase our ability to serve our clients.
A measure of product quality should look at our products through the eyes of their users. Thus, we are developing a mechanism to obtain client feedback on our products as part of our balanced scorecard strategy. We will, however, continue our quality control practice of having senior managers review a statistically valid sample of products to make sure they comply with our core values and professional standards. The results will be one of the inputs used to continuously build quality into our products.
final performance plan for fiscal year 2001 included a performance measure
for multiunit products. This was our first attempt at measuring matrix
management by focusing on published products issued jointly by more
than one of our teams and offices. We decided not to use this performance
measure in fiscal year 2001 for three reasons. First, we were concerned
that it did not adequately capture the extensive collaboration needed
within and across units, which goes beyond that required to simply produce
a published report. Second, instead of just measuring the extent to
which matrix management is occurring, we need to assess how well it
is working. Finally, and most importantly, the new risk and matrix management
strategies we implemented last year provide more proactive senior management
involvement in our work processes. For instance, the new engagement
acceptance meetings and engagement review meetings we conduct involve
senior managers early in key decisions, such as whether to accept an
engagement and what resources to allocate to it and determining the
level of senior management involvement. As a result, we believe we have
a more proactive approach to ensuring that the appropriate resources
are devoted to each engagement regardless of where in our agency those
resources are housed. Moreover, because this measure had relatively
little meaning outside of GAO, we are seeking a better way to measure
matrix management efforts.