This is the accessible text file for GAO report number GAO-06-28 
entitled 'Performance Budgeting: PART Focuses Attention on Program 
Performance, but More Can Be Done to Engage Congress' which was 
released on October 31, 2005. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to the Chairman, Subcommittee on Government Management, Finance, 
and Accountability, Committee on Government Reform, House of 
Representatives: 

October 2005: 

Performance Budgeting: 

PART Focuses Attention on Program Performance, but More Can Be Done to 
Engage Congress: 

GAO-06-28: 

GAO Highlights: 

Highlights of GAO-06-28, a report to the Chairman, Subcommittee on 
Government Management, Finance, and Accountability, Committee on 
Government Reform, House of Representatives: 

Why GAO Did This Study: 

GAO was asked to examine (1) the Office of Management and Budget’s 
(OMB) and agency perspectives on the effects that the Program 
Assessment Rating Tool (PART) recommendations are having on agency 
operations and program results; (2) OMB’s leadership in ensuring a 
complementary relationship between the PART and the Government 
Performance and Results Act of 1993 (GPRA); and (3) steps OMB has taken 
to involve Congress in the PART process. To do this, we also followed 
up on issues raised in our January 2004 report on the PART. 

What GAO Found: 

The PART process has aided OMB’s oversight of agencies, focused 
agencies’ efforts to improve program management, and created or 
enhanced an evaluation culture within agencies. Although the PART has 
enhanced the focus on performance, the PART remains a labor-intensive 
process at OMB and agencies. 

Most PART recommendations are focused on improving outcome measures and 
data collection, and are not designed to result in observable short-
term performance improvements. Since these necessary first steps on the 
path to long-term program improvement do not usually lead to improved 
short-term results, there is limited evidence to date of the PART’s 
influence on outcome-based program results. Moreover, as of February 
2005—the date of the most recent available OMB data—the majority of 
follow-on actions have not yet been fully implemented. By design OMB 
has not prioritized them within or among agencies. Because OMB has 
chosen to assess nearly all federal programs, OMB and agency resources 
are diffused across multiple areas instead of concentrated on those 
areas of highest priority both within agencies and across the federal 
government. This strategy is likely to lengthen the time it will take 
to observe measurable change compared with a more strategic approach. 
OMB has used the PART as a framework for several crosscutting reviews, 
but these have not always included all relevant tools, such as tax 
expenditures, that contribute to related goals. Greater focus on 
selecting related programs and activities for concurrent review would 
improve their usefulness. 

OMB has taken some steps to clarify the PART-GPRA relationship but many 
agencies still struggle to balance the differing needs of the budget 
and planning processes and their various stakeholders. Unresolved 
tensions between GPRA and the PART can result in conflicting ideas 
about what to measure and how to measure it. Finally, we remain 
concerned that the focus of agencies’ strategic planning continues to 
shift from long-term goal setting to short-term executive budget and 
planning needs. 

OMB uses a variety of methods to communicate PART results, but 
congressional committee staff we spoke with had concerns about the tool 
itself, how programs were defined, and the usefulness of goals and 
measures. Most said that the PART would more likely inform their 
deliberations if OMB consulted them early on regarding the selection 
and timing of programs; the methodology and evidence to be used; and 
how PART information can be communicated and presented to best meet 
their needs. It is also important that Congress take full advantage of 
the benefits arising from the executive reform agenda. While Congress 
has a number of opportunities to provide its perspective on specific 
performance issues and performance goals, opportunities also exist for 
Congress to enhance its institutional focus to enable a more systematic 
assessment of key programs and performance goals. 

What GAO Recommends: 

GAO recommends that OMB solicit congressional views on the performance 
issues and program areas most in need of review; the most useful 
performance data and the presentation of those data; and select PART 
reassessments and crosscutting reviews based on factors including the 
relative priorities, costs, and risks associated with clusters of 
related programs, and reflective of congressional input. GAO also 
suggests that Congress consider a structured approach to articulating 
its perspective and oversight agenda on performance goals and 
priorities for key programs. 

OMB generally agreed with our findings, conclusions, and 
recommendations, and outlined several actions it is taking to address 
some of the issues raised in the report. 

www.gao.gov/cgi-bin/getrpt?GAO-06-28. 

To view the full product, including the scope and methodology, click on 
the link above. For more information, contact Susan J. Irving at (202) 
512-9142 or irvings@gao.gov. 

[End of section] 

Contents: 

Letter: 

Results in Brief: 

Background: 

The PART Encourages a Focus on Performance Measurement and Program 
Review: 

To Date, the PART Appears to Have Had Limited Impact on Outcome-Based 
Program Results: 

The PART-GPRA Relationship Does Not Adequately Consider the Different 
Needs of the Budget and Planning Processes and Their Various 
Stakeholders: 

Tailoring Outreach to Meet Congressional Needs Is Key to Increasing the 
Likelihood of Congress's Considering the PART in Its Deliberations: 

Conclusions and General Observations: 

Matter for Congressional Consideration: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendixes: 

Appendix I: Scope and Methodology: 

Appendix II: The 2004 PART Questionnaire: 

Appendix III: Sample PART Summary Worksheets: 

Appendix IV: Comments from the Executive Office of the President: 

Appendix V: GAO Contact and Staff Acknowledgments: 

Related GAO Products: 

Tables: 

Table 1: Overview of Sections of the PART Questions: 

Table 2: Overview of the PART Program Types, Cumulative Data 2002-2004: 

Table 3: The Cumulative PART Program Results by Rating Category, 2002- 
2004: 

Figures: 

Figure 1: Selected PART Follow-on Actions: 

Figure 2: PART Recommendations by Type, 2002-2004: 

Figure 3: Federal Evaluators' Views on Tailoring Designs for Program 
Effectiveness Evaluations: 

Letter October 28, 2005: 

The Honorable Todd R. Platts: 
Chairman: 
Subcommittee on Government Management, Finance, and Accountability: 
Committee on Government Reform: 
House of Representatives: 

Dear Mr. Chairman: 

Given current trends and challenges facing the nation--including the 
federal government's long-term fiscal imbalance--we must take advantage 
of opportunities to enhance performance, ensure accountability, and 
position the nation for the future. To this end, it is critical to 
reexamine the relevance of federal programs and their fit with national 
priorities, while maximizing program performance within current and 
expected resource levels.[Footnote 1] 

In the 1990s, Congress and the executive branch laid out a statutory 
and management framework that provides the foundation for strengthening 
government performance and accountability, with the Government 
Performance and Results Act of 1993[Footnote 2] (GPRA) as its 
centerpiece. GPRA is designed to inform congressional and executive 
decision making by providing objective information on the relative 
effectiveness and efficiency of federal programs and spending. A key 
purpose of the act is to create closer and clearer links between the 
process of allocating scarce resources and the expected results to be 
achieved with those resources--also known as performance budgeting. 
Performance budgeting holds promise for reexamining existing federal 
government programs and aligning them with current needs. It can 
increase the government's capacity to assess competing claims for 
federal dollars and has the potential to enrich the budget debate by 
arming decision makers with credible information on both individual 
programs and the relationship between policies, programs, and other 
tools designed to address related goals. 

This administration has made budget and performance integration one of 
five governmentwide management priorities under its President's 
Management Agenda (PMA).[Footnote 3] The Executive Branch Management 
Scorecard, which employs a red-yellow-green stoplight grading system, 
tracks how well departments and major agencies are executing the five 
governmentwide management initiatives. A central element in this 
initiative is the Office of Management and Budget's (OMB) Program 
Assessment Rating Tool (PART), which OMB describes as a diagnostic tool 
meant to provide a consistent approach to evaluating federal programs 
as part of the executive budget formulation process. 

At your request, we reported in January 2004 on OMB and agency 
experiences during the first year of PART implementation.[Footnote 4] 
Specifically, we examined how the PART changed OMB's decision-making 
process in developing the President's fiscal year 2004 budget request; 
the PART's relationship to GPRA; and the PART's strengths and 
weaknesses as an evaluation tool. Our findings and recommendations from 
the January 2004 report are detailed later in this report. In light of 
the challenges we identified, you asked us to examine (1) OMB's and 
agencies' perspectives on the effects the PART recommendations are 
having on agency operations and program results, including issues 
encountered in responding to the PART recommendations; (2) OMB's 
leadership in ensuring an integrated, complementary relationship 
between the PART and GPRA, including instances in which multiple 
programs or agencies are involved in meeting goals and objectives; and 
(3) steps OMB has taken to involve Congress in the PART process. To 
help us fulfill these objectives, we also followed up on the issues and 
challenges we identified during our January 2004 review. Lastly, we 
agreed to issue a separate report that explores (1) progress agencies 
have made in responding to OMB's PART recommendations that they obtain 
program evaluations, (2) what factors have facilitated or impeded 
agency progress toward obtaining the program evaluations that OMB 
recommended, and (3) whether the evaluations appear designed to yield 
the desired information on program results.[Footnote 5] 

To address the first two objectives, we reviewed OMB materials on the 
implementation, application, and revision of the PART for calendar 
years 2002 through 2004.[Footnote 6] We also interviewed OMB branch 
chiefs and OMB staff on the Performance Evaluation Team whose role is 
to provide guidance to budget examiners and help ensure consistent 
application of the PART across OMB offices. To better understand OMB's 
experience with crosscutting reviews, we interviewed OMB staff 
responsible for coordinating the Community and Economic Development and 
Rural Water crosscutting reviews conducted for the fiscal year 2006 
President's budget request. To obtain agency perspectives on the 
relationship between the PART and GPRA and their interactions with OMB 
concerning that relationship, we interviewed department and agency 
officials, including senior managers, and program, planning, and budget 
staffs at (1) the Department of Health and Human Services (HHS), (2) 
the Department of Energy (DOE), (3) the Department of Labor (DOL), and 
(4) the Small Business Administration (SBA). We also interviewed 
officials from these departments and agencies concerning their 
perspectives and activities in response to the PART recommendations and 
the effects of implementing those recommendations on operations and 
results. 

We selected these three departments and one independent agency for a 
number of reasons. Collectively, they offered examples of all seven of 
the PART program types (e.g., block/formula grants, competitive grants, 
direct federal, and research and development) for review. These 
examples covered about a fifth of all the programs subject to the PART 
as of 2004 and thus could provide a broad-based perspective on how the 
PART was applied. Our selection of these four agencies was also 
influenced by our intent to integrate this work with our related work 
examining progress in addressing the PART program evaluation 
recommendations. Approximately half of the evaluation recommendations 
in the 2002 PART were encompassed in our four case selections. 

As part of our work on the second objective, we also performed various 
analyses of the PART recommendations made in all 3 years to discern 
possible changes or trends in recommendations over time and 
relationships between the type of recommendations made, type of 
program, overall rating, the total PART score, and answers to selected 
PART questions. In addition, we also examined relevant OMB and agency 
documents to help determine how recommendations are tracked and their 
impact evaluated by OMB and the selected agencies. 

To address our third objective of examining the steps OMB has taken to 
involve Congress in the PART process, we interviewed OMB and agency 
officials and asked questions about the steps OMB and agencies have 
taken to involve Congress in the PART process or in using the results 
of the PART. To obtain documented instances of Congress's uses and 
views of the PART, we interviewed House and Senate committee staff 
(minority and majority) for the authorization and appropriations 
subcommittees with jurisdiction over our selected agencies as well as 
OMB and officials from the four selected agencies. Finally, we reviewed 
fiscal years 2005 and 2006 House and Senate congressional hearing 
records and reports as well as conference reports for mentions of the 
PART. In addition, where possible, we corroborated testimonial evidence 
with documentary evidence of OMB and agencies' strategies for involving 
the Congress as well as evidence of collaboration and coordination, 
such as planning documents, briefing materials, or other evidence of 
contact with Congress. 

While our summary analyses include all or almost all programs subject 
to the PART for the years 2002 to 2004 or all or almost all programs 
within a specified subset of programs (e.g., program type, specific 
year, specific rating, etc.), the information obtained from OMB, 
congressional, and agency officials as well as documentary materials 
from the selected agencies is not generalizable to the PART process for 
all years or all programs. The information obtained from various 
officials reflects their views and experiences with the PART. However, 
the consistency and frequency with which officials identified certain 
issues and concerns in our prior review and in this one suggest that 
they are both significant and continuing aspects of the PART's use as a 
budget and evaluation tool. 

We conducted our audit work from January 2005 through September 2005 in 
accordance with generally accepted government auditing standards. 
Detailed information on our scope and methodology appears in appendix 
I. 

Results in Brief: 

The PART process continues to aid OMB's oversight of agencies and 
encourage improvements in executive budget formulation and agency 
program management. The PART has helped to structure and discipline 
OMB's use of performance information for internal program analysis and 
budget review, and made its use of this information more transparent. 
Many agency officials told us that the PART helped to create or 
strengthen an evaluation culture within agencies by providing external 
motivation for program review and focused attention on performance 
measurement and its importance in daily program management. Some 
officials said that the PART and the PMA helped them move away from 
"analysis by anecdote" and refocused their attention on the impact 
their programs have, instead of largely on output measures. Others 
echoed a similar sentiment--one indicated that the PART scores helped 
to create "a new sense of urgency" about performance measures and 
completing the changes to performance systems that were already 
underway. Although the PART has enhanced the focus on performance, the 
PART remains a labor-intensive process at OMB and agencies. 

As we reported in our January 2004 report, senior OMB managers 
recognized the increased workload the PART initially placed on 
examiners; however, they expected the workload to decline as both OMB 
and agency staff became more familiar with the PART tool and process, 
and as issues with the timing of the PART reviews were resolved. During 
this review, we found that while the learning curve appears to have 
flattened, it did not compensate for the time needed to assess and 
reassess programs. Although a senior OMB official acknowledged 
continuing capacity issues regarding the PART, he said that the PART is 
still a better way for examiners to accomplish their traditional 
program assessment responsibilities because it is more objective and 
transparent. 

OMB has said that a primary purpose of the PART is program improvement. 
Our analysis supports OMB's statements that most of the PART 
recommendations to date were aimed at improving outcome and efficiency 
measures, and in collecting performance data. Implementing these 
recommendations can lay the groundwork for long-term program 
improvement by improving managers' ability to assess program outcomes, 
identify information gaps, and assess next steps--these are necessary 
first steps on the path to long-term program improvement, but are not 
expected to result in observable program improvement in the short term. 
According to OMB's most recent available data (February 2005), the 
majority of the PART recommendations have not yet been fully 
implemented; nearly half of the fully implemented recommendations are 
of the type described above. For these reasons, it is too soon to tell 
whether the PART has fully produced the intended results. 

Although agency officials appreciated the flexibility OMB provided by 
not making prescriptive recommendations, some were so general that it 
was difficult for them to understand what change was required or how 
progress could be measured. OMB has made a conscious decision not to 
prioritize the recommendations within each agency or across the more 
than 1,700 recommendations governmentwide. Because OMB has chosen to 
assess nearly all federal programs, OMB and agency resources are 
diffused across multiple areas instead of concentrated on those areas 
of highest priority both within agencies and across the federal 
government. This strategy is likely to lengthen the time it will take 
to observe measurable change compared with a more strategic approach. 

Federal agencies are increasingly expected to demonstrate effectiveness 
in achieving agency or governmentwide goals. Our previous work has 
shown that the accuracy and quality of evaluation information necessary 
to make the judgments called for when rating programs is highly uneven 
across the federal government. To help explain linkages between program 
activities, outputs, and outcomes, program evaluation designs are 
tailored to address various types of programs and questions. For 
example, a process evaluation reviews various aspects of program 
operations to assess the quality or efficiency of services. 
Alternatively, an impact evaluation depends on scientific research 
methods to establish a causal connection between program activities and 
outcomes, and, in light of important influences in the program 
environment, to isolate the program's contributions to the observed 
outcomes. 

OMB and agencies' expectations sometimes differed on the scope and 
purpose of evaluations conducted in response to the PART 
recommendations. Some of the difficulties seemed to derive from OMB 
expecting to find, in the agencies' external evaluation studies, the 
types of comprehensive judgments about program design, management, and 
effectiveness found in the PART assessments. Because evaluations 
designed for internal and external audiences often have a different 
focus, differences of opinion on the usefulness of evaluations is 
perhaps not surprising. This raises the strong possibility that-- 
without consultation--the evaluations that agencies conduct may not 
provide OMB with the information it wants. 

The PART's focus on individual programs can yield useful information. 
At the same time, it is also often critical to understand how 
individual programs fit within a broader portfolio of tools and 
strategies--such as regulations, credit programs, and tax expenditures-
-to accomplish federal goals.[Footnote 7] OMB reported on the two 
crosscutting PART assessments--Rural Water programs and Community and 
Economic Development (CED) programs--for the fiscal year 2006 budget 
and it plans to conduct three additional crosscutting reviews on block 
grants, credit programs, and small business innovation research for the 
fiscal year 2007 budget. Cognizant agency officials involved with the 
CED were pleased with the collaborative interagency process. They found 
value in leveraging existing efforts within agencies and benefited from 
OMB staff consultation. To date, however, crosscutting assessments have 
not always addressed all programs identified by OMB as relevant to a 
common goal. Also, little information on the CED crosscutting 
assessment was initially available beyond the brief description 
contained in the Analytical Perspectives volume of the Fiscal Year 2006 
President's budget request. This caused some confusion among budget 
stakeholders, particularly Congress. 

Both the PART and GPRA aim to improve the focus on program results. 
However, the different purposes and time frames they serve continue to 
create tensions. Some agencies have made progress over the past several 
years in reconciling the two processes, but OMB and agencies often have 
conflicting ideas about what to measure, how to measure it, and how to 
report program results. Although less of a problem than it was during 
our January 2004 review of the PART process, difficulties in defining a 
unit of analysis useful for both OMB's budget analysis and program 
management purposes remain. For example, agency officials said that the 
PART sometimes defined a "program" in overly narrow terms which did not 
provide an understanding of how the entire program works. Agency 
officials also noted there can be difficulties if a PART "program" is 
defined so broadly that unrelated or loosely related programs and/or 
programs with uneven success levels are combined. 

Measuring outcomes has long been challenging, especially for certain 
types of programs, and can complicate efforts to arrive at goals useful 
to multiple parties for multiple purposes. During the PART process, 
this was particularly true for programs whose outcomes take many years 
to observe, areas in which the federal government is one of many 
actors, and basic research programs. Differences of opinion about 
efficiency measures highlighted the challenges that can result from the 
difficult but potentially useful process of comparing the costs of 
programs related to similar goals. Although these issues are clearly 
not new or unique to the PART, they are aggravated by the difficulties 
of developing measures useful for multiple purposes and audiences and 
often remain a point of friction in agencies and sometimes within OMB. 

The PART was designed for and is used in the executive branch budget 
preparation and review process; as such, the goals and measures used in 
the PART must meet OMB's needs. GPRA is a broader process involving the 
development of strategic and performance goals and objectives to be 
reported in strategic and annual plans. Compared to the relatively open-
ended GPRA planning process any executive budget formulation process is 
likely to seem closed. Without truly integrating agency budget 
formulation and strategic planning in a way that considers the 
differing needs of these processes and their various stakeholders, 
OMB's ability to strengthen and further the performance-resource 
linkages for which GPRA laid the groundwork will continue to be 
hampered. Although OMB has made efforts to ensure that the PART and 
GPRA are complementary processes, we continue to find evidence that the 
closed nature of the executive budget formulation process may not allow 
for the type of stakeholder involvement in strategic and annual 
planning envisioned by GPRA. Because the executive budget formulation 
process--to which the PART belongs--is "predecisional," information 
required in agencies' annual GPRA plans is embargoed within the 
executive branch until the President's budget request is transmitted to 
Congress. Several of our case study agencies described instances in 
developing annual and long-term goals where interaction and/or 
consultation with key stakeholders was limited or absent. They said 
because of timing issues or the explicit instructions of OMB, they had 
to present new or revised program goals and measures to their 
stakeholders after the fact, and in some cases stakeholders disagreed 
with the goals but had no choice but to accept them. We remain 
concerned that the focus of agency strategic planning is shifting from 
long-term goal setting to short-term budget and planning needs. 

OMB uses a variety of tools to communicate the PART assessment results 
to Congress, including the President's budget request documents, OMB's 
Web site, and meetings with some congressional staff. OMB said that 
they also sent packages including the PART summary sheets for programs 
that fell within each committee's jurisdiction and a list of the 
programs OMB planned to review for the fiscal year 2006 budget request 
to all relevant House and Senate committees, but received little 
response. Committee staff told us that OMB's efforts have not been 
sufficient to convince Congress of the value of the PART. Most 
congressional committee staff we spoke with did not find either the 
PART information or the way it was communicated suited to their needs. 
Many had concerns about the usefulness of the goals and measures OMB 
used to assess program performance and some questioned the "units of 
analysis" used for the PART as well as the design of the tool itself. 
Most reported that they would be more likely to use the PART results to 
inform their deliberations if OMB consulted them early in the PART 
process regarding the selection and timing of programs to assess, the 
methodology and evidence used or to be used, and how the PART 
information can best be communicated and presented to meet their needs. 

That said, OMB has sometimes engaged Congress when it has communicated 
the selected PART results through traditional means of signaling 
executive branch priorities, such as legislative proposals. Congress 
has also held hearings in response to the PART, and has mentioned the 
PART in committee reports. Clearly, certain PART reviews have captured 
congressional attention and contributed to the policy debate. While 
Congress has a number of opportunities to provide its perspective on 
specific performance issues and performance goals, opportunities also 
exist for Congress to enhance its institutional focus to enable a more 
systematic assessment of key programs and performance goals. 

We make recommendations and suggestions in this report to OMB and 
Congress, respectively, to promote greater discussion with 
congressional stakeholders and emphasize the PART's role in assessing 
programs related to common outcomes. We believe that implementing these 
recommendations will help ensure an integrated, complementary approach 
between GPRA and the PART. 

In commenting on a draft of this report, OMB generally agreed with our 
findings, conclusions, and recommendations. OMB outlined several 
actions it is taking to address some of the issues raised in the 
report, including implementing information technology solutions to make 
application of the PART less burdensome and more collaborative. OMB 
also suggested some technical changes throughout the report that we 
have incorporated as appropriate. OMB's comments appear in appendix IV. 
We also received technical comments on excerpts of the draft from the 
Departments of Labor and Health and Human Services, which are 
incorporated as appropriate. 

Background: 

Federal interest in performance information and its potential 
relationship to budgeting practices has existed to varying degrees for 
over 50 years.[Footnote 8] More recently, this interest culminated in 
the passage of GPRA and related management reforms of the 1990s. GPRA 
mandates that federal agencies develop performance information 
describing the relative effectiveness and efficiency of federal 
programs as a means of improving the congressional and executive 
decision-making processes. Among other statutory obligations, GPRA 
requires federal agencies to publish strategic and annual plans 
describing specific program activities with the intention of 
establishing a more tangible link between performance information for 
these programs and agency budget requests.[Footnote 9] 

The current administration has taken several steps to strengthen 
performance-resource linkages for which GPRA laid the groundwork. The 
budget and performance initiative of the PMA contains the criteria 
agencies must meet in order to achieve "green" status on the 
initiative. The criteria include elements relating to budgeting and 
strategic planning and also ties those elements to individual 
performance. As we have previously reported, creating a "line of sight" 
between individual performance and organizational success is a leading 
practice used in public sector organizations to become more results- 
oriented.[Footnote 10] 

Central to the budget and performance integration initiative of the 
PMA, the PART is a means to strengthen the process for assessing the 
effectiveness of programs by making that process more robust, 
transparent, and systematic. The PART is a series of diagnostic 
questions designed to provide a consistent approach to rating federal 
programs. (See app. II for the PART questionnaire.) Drawing on 
available performance and evaluation information, OMB staff use the 
questionnaire to rate the strengths and weaknesses of federal programs 
with a particular focus on individual program results. The PART asks, 
for example, whether a program's long-term goals are specific, 
ambitious, and focused on outcomes, and whether annual goals 
demonstrate progress toward achieving long-term goals. It is designed 
to be evidence-based, drawing on a wide array of information, including 
authorizing legislation; GPRA strategic plans, annual performance 
plans, and reports; financial statements; inspectors general and GAO 
reports; and independent program evaluations. The PART questions are 
divided into four sections; each section is given a specific weight in 
determining the final numerical rating for a program. Table 1 shows an 
overview of the four PART sections and the weights OMB assigned. 

In addition, each PART program is assessed according to one of seven 
major approaches to delivering services. Table 2 provides an overview 
of these program types and the number and percentage of programs 
covered by each type in the 2002 through 2004 performance assessments. 

Table 1: Overview of Sections of the PART Questions: 

Section: I. Program Purpose and Design; 
Description: To assess whether; 
* the purpose is clear, and; 
* the program design makes sense; Weight (percentage): 20. 

Section: II. Strategic Planning; 
Description: To assess whether the agency sets valid programmatic; 
* annual goals, and; 
* long-term goals; Weight (percentage): 10. 

Section: III. Program Management; 
Description: To rate agency management of the program, including; 
* financial oversight, and; 
* program improvement efforts; Weight (percentage): 20. 

Section: IV. Program Results/Accountability; 
Description: To rate program performance on goals reviewed in; 
* the strategic planning section, and; 
* through other evaluations; Weight (percentage): 50. 

Source: GAO. 

[End of table] 

Table 2: Overview of the PART Program Types, Cumulative Data 2002-2004: 

Program type: 1. Direct federal; 
Description: Programs in which support and services are provided 
primarily by federal employees; 
Number/percentage of programs[A]: 2002: 67; 29%; 
Number/percentage of programs[A]: 2003: 117; 29%; 
Number/percentage of programs[A]: 2004: 179; 29%. 

Program type: 2. Block/formula grant; 
Description: Programs that distribute funds to state, local, and tribal 
governments and other entities by formula or block grant; 
Number/percentage of programs[A]: 2002: 41; 18%; 
Number/percentage of programs[A]: 2003: 71; 18%; 
Number/percentage of programs[A]: 2004: 98; 16%. 

Program type: 3. Competitive grant; 
Description: Programs that distribute funds to state, local, and tribal 
governments, organizations, individuals, and other entities through a 
competitive process; 
Number/percentage of programs[A]: 2002: 37; 16%; 
Number/percentage of programs[A]: 2003: 62; 16%; 
Number/percentage of programs[A]: 2004: 105; 17%. 

Program type: 4. Capital assets and service acquisition; 
Description: Programs in which the primary means to achieve goals is 
the development and acquisition of capital assets (such as land, 
structures, equipment, and intellectual property) or the purchase of 
services (such as maintenance and information technology) from a 
commercial source; 
Number/percentage of programs[A]: 2002: 34; 15%; 
Number/percentage of programs[A]: 2003: 43; 11%; 
Number/percentage of programs[A]: 2004: 58; 10%. 

Program type: 5. Research and development; 
Description: Programs that focus on creating knowledge or applying it 
toward the creation of systems, devices, methods, materials, or 
technologies; 
Number/percentage of programs[A]: 2002: 32; 14%; 
Number/percentage of programs[A]: 2003: 42; 11%; 
Number/percentage of programs[A]: 2004: 54; 9%. 

Program type: 6. Regulatory based; 
Description: Programs that employ regulatory action to achieve program 
and agency goals through rule making that implements, interprets, or 
prescribes law or policy, or describes procedure or practice 
requirements. These programs issue significant regulations, which are 
subject to OMB review; 
Number/percentage of programs[A]: 2002: 15; 6%; 
Number/percentage of programs[A]: 2003: 23; 6%; 
Number/percentage of programs[A]: 2004: 32; 5%. 

Program type: 7. Credit; 
Description: Programs that provide support through loans, loan 
guarantees, and direct credit; 
Number/percentage of programs[A]: 2002: 4; 2%; 
Number/percentage of programs[A]: 2003: 14; 4%; 
Number/percentage of programs[A]: 2004: 23; 4%. 

Program type: 8. Mixed[B]; 
Description: Programs that contain elements of different program types; 
Number/percentage of programs[A]: 2002: 4; 2%; 
Number/percentage of programs[A]: 2003: 27; 7%; 
Number/percentage of programs[A]: 2004: 58; 10%. 

Source: GAO analysis of the PART data. 

[A] Percentages do not add to 100 percent because of rounding. 

[B] OMB noted that in certain cases, drawing questions from two or more 
of the seven main PART program types--that is, creation of a "mixed" 
program type--yields a more informative assessment. 

[End of table] 

As of February 2005, the PART ratings have been published for 607 
programs (according to OMB, this represents about 60 percent of the 
federal budget). Each program received one of four overall ratings: (1) 
"effective," (2) "moderately effective," (3) "adequate," or (4) 
"ineffective" based on program design, strategic planning, management, 
and results. A fifth rating, "results not demonstrated," was given-- 
independent of a program's numerical score--if OMB decided that a 
program's performance information, performance measures, or both were 
insufficient or inadequate. Table 3 shows the distribution of ratings 
for 2002-2004. During the next 2 years, the administration plans to 
assess all remaining executive branch programs with limited 
exceptions.[Footnote 11] 

Table 3: The Cumulative PART Program Results by Rating Category, 2002- 
2004: 

Ratings: Effective; 
2002: 6%; 
2003: 11%; 
2004: 15%. 

Ratings: Moderately effective; 
2002: 24%; 
2003: 26%; 
2004: 26%. 

Ratings: Adequate; 
2002: 15%; 
2003: 20%; 
2004: 26%. 

Ratings: Ineffective; 
2002: 5%; 
2003: 5%; 
2004: 4%. 

Ratings: Results not demonstrated; 
2002: 50%; 
2003: 38%; 
2004: 29%. 

Ratings: Total programs; 
2002: 234; 
2003: 399; 
2004: 607. 

Source: OMB. 

[End of table] 

In our January 2004 report on the PART, you asked us to examine (1) how 
the PART changed OMB's decision-making process in developing the 
President's fiscal year 2004 budget request; (2) the PART's 
relationship to the GPRA planning process and reporting requirements; 
and (3) the PART's strengths and weaknesses as an evaluation tool, 
including how OMB ensured that the PART was applied consistently. We 
found that the PART helped structure OMB's use of performance 
information for its internal program and budget analysis, made the use 
of this information more transparent, and stimulated agency interest in 
budget and performance integration. Our analysis confirmed that one of 
the PART's major impacts was its ability to highlight OMB's recommended 
changes in program management and design. We noted that while much of 
the PART's potential value lies in the related program recommendations, 
realizing these benefits would require sustained attention to 
implementation and oversight to determine if desired results are 
achieved, and that OMB needs to remain cognizant of this as it 
considers capacity and workload issues in the PART. 

We also recognized that while there are inherent challenges in 
assigning a single rating to programs having multiple purposes and 
goals, OMB devoted considerable effort to promoting consistent ratings 
but challenges remain in addressing inconsistencies among OMB staff, 
such as interpreting the PART guidance and defining acceptable 
measures. OMB senior officials recently told us that inconsistencies in 
the PART process could also be attributed to agency staff, given the 
shared agency-OMB responsibilities in the PART process. Limited 
credible evidence on results also constrained OMB's ability to rate 
program effectiveness, as evidenced by the almost 50 percent of 
programs rated "results not demonstrated." 

We also found that the PART is not well integrated with GPRA--the 
current statutory framework for strategic planning and reporting. We 
said that by using the PART process to review and sometimes replace 
GPRA goals and measures, OMB substituted its judgment for a wide range 
of stakeholder interests. The PART/GPRA tension was further highlighted 
by challenges in defining a unit of analysis useful for both program- 
level budget analysis and agency planning purposes. Although the PART 
can stimulate discussion on program-specific measurement issues, it 
cannot substitute for GPRA's focus on thematic goals and department-and 
governmentwide crosscutting comparisons, and was not used to evaluate 
similar programs together to facilitate trade-offs or make relative 
comparisons. 

Lastly, we said that while the PART clearly must serve the President's 
interests, the many actors whose input is critical to decisions will 
not likely use performance information unless they feel it is credible 
and reflects a consensus on goals. Our work showed that it if OMB 
wanted to expand the understanding and use of the PART beyond the 
executive branch, it would be important for OMB to discuss in a timely 
fashion with Congress the focus of the PART assessments and clarify the 
results and limitations of the PART and the underlying performance 
information. On the other hand, we noted that a more systematic 
congressional approach to providing its perspective on performance 
issues and goals could facilitate OMB's understanding of congressional 
priorities and thus increase the PART's usefulness in budget 
deliberations. 

In light of these issues, we recommended that OMB address the capacity 
demands of the PART, strengthen the PART guidance, address evaluation 
information availability and scope issues, focus program selection on 
crosscutting comparisons and critical operations, broaden the dialogue 
with congressional stakeholders, and articulate and implement a 
complementary relationship between the PART and GPRA. We also suggested 
that Congress consider the need for a structured approach to 
articulating its perspective and oversight agenda on performance goals 
and priorities for key programs. 

OMB took several steps to implement many of our recommendations. For 
example, OMB clarified its PART guidance on defining the PART programs, 
using outcome and output measures, and expanded the discussion of 
evaluation quality; began to use the PART as a framework for 
crosscutting assessments; and expanded its discussion about the 
relationship between the PART and GPRA. The guidance notes that the 
PART strengthens and reinforces performance measurement under GPRA by 
encouraging the careful development of performance measures according 
to GPRA's outcome-oriented standards. It also requires that PART goals 
be "appropriately ambitious" and that GPRA and the PART performance 
measures be consistent. They have also begun reporting on the status of 
each program's recommendations and implemented PARTWeb, a Web-based 
data collection tool to, among other things, improve collaboration 
between OMB and agencies and centrally track the implementation and 
status of the PART recommendations. 

The PART Encourages a Focus on Performance Measurement and Program 
Review: 

The PART process has aided OMB's oversight of agencies, and has focused 
agencies' efforts to improve performance measurement. According to OMB, 
the PART is a framework for program assessment and informs its budget 
decisions. Many agency officials told us that the PART helped either 
create or strengthen a culture of evaluation within the agencies by 
providing external motivation for program review. Not surprisingly, 
agency officials used the PART results to make a case for increased 
resources in general and for program evaluation specifically. This 
increased focus on performance is often reflected in improved ratings 
when "results not demonstrated" programs get reassessed by the PART--86 
percent of programs previously rated "results not demonstrated" were 
subsequently rated adequate, moderately effective, or effective when 
reassessed. This focus is not without cost, however; the PART remains a 
labor-intensive process for both OMB and agencies. 

OMB senior officials describe the PART as providing a consistent 
framework for assessing federal programs, and as a means to inform its 
budget decisions. As a major component of the PMA, OMB clearly relies 
on the PART--a significant component of the PMA--as a major oversight 
tool and finds information from the PART reviews useful. As we 
previously reported, the PART has helped to structure and discipline 
OMB's use of performance information for internal program analysis and 
budget review, and made their use of this information more transparent. 
Given the PART's use in the budget process, the high profile of the PMA 
scorecard, and the strong connection between the PART and successful 
performance on the PMA's budget and performance integration initiative, 
agencies have clear incentives to take the PART seriously. 

Many agency officials told us that the PART helped either create or 
strengthen a culture of evaluation within the agencies by providing 
external motivation for program review. The PART question that asks 
whether a program has undergone regular, independent evaluations sends 
the message that program assessment and evaluation is an important 
management tool. For example, according to one agency official at the 
Health Resources Services Administration in HHS, this requirement 
encouraged staff to think more broadly about using different types of 
program evaluations and how they could get the most out of their 
evaluation dollars. Another HHS official reported that the PART 
provided an impetus for finishing strategic and evaluation plans for 
his program, which in turn helped inform the division's planning 
process. Our companion report on the PART evaluation recommendations 
reports similar findings, noting that most program officials 
interviewed for that report said that the PART recommendations directed 
senior management's attention to the need for evaluation.[Footnote 12] 

Not surprisingly, agency officials used the PART results in some cases 
successfully--to argue for increased resources in general as well as 
specifically for program evaluation. For example, officials in one 
agency said that a good rating on the PART is a powerful aid in gaining 
bipartisan support for budget increases. DOL agency officials told us 
that absent the PART, they might not have received funding to evaluate 
the Youth Activities program--a program they felt had been in need of 
an evaluation for a long time.[Footnote 13] 

Agency officials we spoke with credited the PART with increasing 
attention to the use of performance measurement in day-to-day program 
management, which they considered to be of greater consequence than the 
PART's bottom-line ratings and recommendations. For example, agency 
officials at DOL credited the first year's PART assessments with 
encouraging managers to take steps prior to assessments to identify and 
address program weaknesses, develop and improve performance measures, 
and train staff on the PART. Officials from DOL's Trade Assistance 
program said that the PART forced them to look at the program in a new 
light, and be objective about what they are doing and how they are 
doing it. SBA officials said that the PART and the PMA helped them move 
away from "analysis by anecdote" and refocused their attention on the 
impact their programs have on small businesses, instead of largely on 
output measures such as the number of loans they have made. One 
official at HHS said that the PART allowed him to "evangelize" on the 
importance of good performance data and the perils of bad data. Other 
officials echoed a similar sentiment, one of them indicating that the 
PART scores helped to create "a new sense of urgency" about performance 
measures and completing the changes to performance systems that were 
already underway. The link between program ratings and the PMA 
scorecard also provided an incentive for change. 

"Results not demonstrated" ratings have implications beyond the PART. 
For an agency to achieve "green" on the Performance and Budget 
Integration initiative of the PMA scorecard, less than 10 percent of 
its programs could have received a results not demonstrated PART rating 
for more than 2 years in a row. According to OMB's PMA scorecard update 
as of June 30, 2005, only nine agencies have met this particular 
criterion.[Footnote 14] 

This increased focus on performance is often reflected in improved 
ratings when programs originally rated "results not demonstrated" are 
reassessed. When reassessed, 86 percent of programs previously rated 
"results not demonstrated" were rated adequate, moderately effective, 
or effective. Because programs were only reassessed when OMB determined 
that significant changes had been made to address deficiencies, this 
result is not surprising. However, it was, on average, the "results not 
demonstrated" programs with initially higher section IV scores (section 
IV measures program results) that, when reassessed, showed the greatest 
improvement in rating. While there were programs with low section IV 
scores that received an "adequate" rating when reassessed, lower 
scoring programs generally remained in the "results not demonstrated" 
category or received an "ineffective" rating when reassessed. 

PART Remains a Labor-Intensive Process at OMB and Agencies: 

Although the PART has enhanced the focus on performance, this has not 
come without a cost. As we reported in our January 2004 report, senior 
OMB managers recognized the increased workload the PART initially 
placed on examiners; however, they expected the workload to decline as 
both OMB and agency staff became more familiar with the PART tool and 
process, and as issues with the timing of the PART reviews were 
resolved. During this review we found that while the learning curve did 
appear to flatten, it did not seem to compensate for either the 
increased workload due to the sheer number of programs being assessed 
or reassessed each year or the amount of time an individual assessment 
takes. This finding is consistent with views expressed by OMB staff 
during our 2004 review. They told us that they were surprised that 
reassessments took almost as long as assessing programs for the first 
time. OMB limited the scope of reassessments to include only those 
programs where there is significant evidence of change. Programs that 
received a "results not demonstrated" rating received priority for 
reassessment. According to OMB officials, this change was made partly 
due to resource constraints. 

Officials in some of our case study agencies expressed concern that 
OMB's growing workload affects how the PART programs are defined. They 
said that as more programs are assessed OMB has less time to focus on 
the PART units that "make sense" and instead is creating larger PART 
units to help control the number of the PART assessments that need to 
be completed. One official recognized that getting into too much detail 
can be time consuming, but nonetheless noted that reviewing a larger 
"program" can lead to missing some important details; another said it 
can lead to recommendations that are not specific enough to be useful 
to a program. 

One agency official said that the PART assessments can be thoughtful 
when OMB is knowledgeable about a program and has enough time to 
complete the reviews, but the assessments are less useful when OMB 
staff are unfamiliar with programs or have too many of the PART 
assessments to complete. Officials across all of our case study 
agencies reported these types of issues. For example, one official said 
that although the PART reviews were to be completed by the cognizant 
OMB examiner for the program, this was not always the case. He said 
that due to turnover at OMB, programs in his department were assessed 
by several people even within a single PART cycle, resulting in a lack 
of continuity. In several cases, agencies reported that OMB was not 
able to reassess programs because of resource constraints. Some 
officials told us that the heavy workload meant that the PARTs were not 
completed in a timely enough fashion to allow agencies to appeal 
ratings or present new performance measures, sometimes resulting in 
lower PART scores. OMB officials noted that OMB policy permits agencies 
to appeal answers to individual questions, not entire ratings, and that 
in practice, ratings may be appealed at any time during the PART 
process whether the ratings are in draft form or completed. 

Although a senior OMB official acknowledged continuing capacity issues 
regarding the PART, he said that the PART is still a better way for 
examiners to accomplish their traditional program assessment 
responsibilities because it is more objective and transparent. He noted 
that OMB is devoting more people to help administer the PART tool and 
that PARTWeb, OMB's new on-line Web-based data collection system for 
PART, is also designed to ease the management of the process. For 
example, the official said that PARTWeb will automate the production of 
PART summary sheets. 

The PART is a resource-intensive process for agencies as well. Some 
agency officials at DOL noted that the PART process is "one size fits 
all" in that a small program at DOL is supposed to have the same 
resources to devote to helping the budget examiner through the process 
and have the same analytic and evaluation resources as a large 
organization like the Social Security Administration. They said that 
agency staff is diverted from mission work to the PART work and in some 
cases is spending significant time on helping OMB staff understand the 
history and context of the programs. 

To Date, the PART Appears to Have Had Limited Impact on Outcome-Based 
Program Results: 

OMB has said that a major purpose of the PART is program improvement. 
Our analysis supports OMB's statements that most of the PART 
recommendations to date were aimed at improving performance assessment, 
such as developing outcome and efficiency measures, and in collecting 
performance data. Improving managers' ability to assess program 
outcomes, identify information gaps, and assess next steps are 
necessary first steps on the path to long-term program improvement, but 
are not expected to result in observable program improvement in the 
short term. Moreover, as of February 2005--the date of the most recent 
available OMB data--the majority of the PART recommendations have not 
yet been fully implemented. Consequently, there is limited evidence to 
date of outcome-based program results. Implementing the PART 
recommendations has proven challenging. Although some agency officials 
appreciated the flexibility OMB provided by not making prescriptive 
recommendations, some follow-on actions were so general that it was 
difficult to understand what change was required or how progress could 
be measured. Some agencies did not discuss with OMB the evaluation 
plans created in response to the PART recommendations; combined with 
different expectations on the scope and purpose of the evaluations and 
the quality of evaluation designs, it is not certain whether these 
evaluations will meet OMB's needs. Lastly, OMB has used the PART as a 
framework for several crosscutting reviews, but these generally do not 
include all tools, such as tax expenditures, that contribute to related 
goals. Greater focus on selecting related programs and activities for 
concurrent review would improve their usefulness. 

Most PART Recommendations Will Not Result in Observable Short-term 
Performance Improvements: 

For each program assessment, the PART summary worksheets were published 
in a separate volume with the President's fiscal year 2004 budget 
request. For the fiscal year 2005 and 2006 budgets, similar information 
was provided in an accompanying CD-ROM. The detailed, supporting 
worksheets for each program were posted on OMB's Web site.[Footnote 15] 
The PART summary sheets display the program's key performance measures, 
budget information, significant findings, and follow-up actions (also 
known as recommendations. See fig. 1 for examples of follow-on 
actions). In the fiscal year 2006 budget, summary sheets for programs 
that have been previously assessed also include information on when the 
program was last assessed and the status of the follow-up actions. 
Status markers include "no action taken," "action taken but not 
completed," and "completed." (See appendix III for examples of summary 
worksheets for programs assessed for the first time and for programs 
that were reassessed.) 

Figure 1: Selected PART Follow-on Actions: 

The PART recommendations are generally aimed at improving program 
design, assessment, management, and/or funding. They can be general or 
very specific. Examples of recommendations for each of the four 
categories include: 

Program Design: 

* "Reduce unnecessary subsidies to lenders and other…program 
participants." 
* "The 2006 Budget proposes to restructure the grant allocation 
process, providing the Secretary with greater discretion to award funds 
based on risks, threats, and vulnerabilities." 

The PART recommendations are generally aimed at improving program 
design, assessment, management, and/or funding. They can be general or 
very specific. Examples of recommendations for each of the four 
categories include: 

Program Management: 

The PART recommendations are generally aimed at improving program 
design, assessment, management, and/or funding. They can be general or 
very specific. Examples of recommendations for each of the four 
categories include: 
* "Work with the agency to coordinate efforts with other federal 
agencies to improve violent-crime reducing services." 
* "Increase the number of accounts supporting this program to quicken 
the transfer of funds with contractors and increase management 
flexibility to address changing security conditions and mission 
priorities. This will significantly improve the obligation and costing 
process of funds." 

The PART recommendations are generally aimed at improving program 
design, assessment, management, and/or funding. They can be general or 
very specific. Examples of recommendations for each of the four 
categories include: 

Funding: 

The PART recommendations are generally aimed at improving program 
design, assessment, management, and/or funding. They can be general or 
very specific. Examples of recommendations for each of the four 
categories include: 
* "Maintaining funding at the 2005 enacted level until the agency can 
show how it will use additional funds to improve performance." 
* "Directly related to the PART findings, the Budget includes $37 
million, a $3 million decrease." 

The PART recommendations are generally aimed at improving program 
design, assessment, management, and/or funding. They can be general or 
very specific. Examples of recommendations for each of the four 
categories include: 

Program Assessment: 

* "Develop a means of regularly performing independent evaluations to 
examine program effectiveness." 
* "Develop baselines for its proposed long-term measures. Without 
baselines for the measures, it was impossible to verify the performance 
of the program." 

Source: OMB. 

[End of figure] 

As figure 2 shows, the distribution of recommendations between program 
management, assessment, and design is fairly consistent over the 3 
years; however, the percentage of recommendations with explicit funding 
references in a given year have steadily declined since the PART's 
inception from 20 percent in 2002 to 12 percent in 2004. 

Figure 2: PART Recommendations by Type, 2002-2004: 

[See PDF for image] 

[End of figure] 

A major goal of the PART is to identify program strengths and 
weaknesses and make recommendations to improve program results. 
However, we found that the link between problems identified by the PART 
assessments and the recommendations intended to address them is 
sometimes unclear. Regardless of what types of deficiencies were 
identified by the PART, the most frequent recommendations in each of 
the three years were related to performance assessments, such as 
developing outcome measures and/or goals, and improving data 
collection. While especially true for "results not demonstrated" 
programs, it also held true for programs rated "effective" and 
"moderately effective." Moreover, programs assessed for the first time 
in 2004--the most recent year for which data is available--received 
recommendations to improve performance assessments, such as outcome 
measures, as frequently as programs assessed during the first PART 
cycle. 

More than half of all the PART recommendations made since the PART's 
inception are aimed at improving the "process" of program assessment. 
This includes developing meaningful and robust performance goals and 
collecting quality data to measure progress against those goals. Of the 
797 follow-on recommendations made in the first 2 years[Footnote 16] 
for which OMB provided status information, 30 percent were considered 
fully implemented.[Footnote 17] Of these, 47 percent are geared toward 
performance assessment. For example, the Animal and Plant Health 
Inspection Service Plant and Animal Health Monitoring Programs within 
the Department of Agriculture received three recommendations, one of 
which would create efficiency measures and the other which would update 
the program's measures and accomplishments. Such measures improve 
managers' ability to assess program outcomes, identify information 
gaps, and assess next steps, but are not expected to result in 
observable program improvement in the short term. OMB claims that many 
programs are getting better every year--which it defines as improving 
program outcomes, taking steps to address the PART findings, improving 
program management, and becoming more efficient--but, as noted above, 
these claims have not yet been fully born out. 

Some recommendations are aimed at changing a program's purpose or 
design and/or implicitly or explicitly require action by Congress. For 
example, the Department of Agriculture's Commodity Credit Corporation's 
Marketing Loan Payments program received a recommendation to have the 
"House and Senate Agricultural committees examine the issue of payment 
limits for marketing loan and LDP gains and how they could be 
tightened." A Department of Education special education program was 
told to "work with Congress on the IDEA[Footnote 18] reauthorization to 
increase the act's focus on accountability and results, and reduce 
unnecessary regulatory and administrative burdens." Even in cases where 
there is general agreement that legislative action or statutory changes 
are needed, making them takes time. 

OMB has said that if statutory provisions impede effectiveness, one 
result of a PART review could be recommendations for legislative 
changes. The responsibility to implement the PART recommendations lies 
with agency and program managers. Successfully implementing 
recommendations that require legislative action or statutory changes 
requires the additional step of positively engaging Congress. A 
perceived disconnect between what one is held accountable for and what 
one has the authority to accomplish is not unusual. Our 2003 survey of 
governmentwide federal managers supports this view.[Footnote 19] We 
found that while 57 percent of non-Senior Executive Service (SES) 
managers and 61 percent of SES mangers believed they were held 
accountable for results to a great or very great extent, only 38 
percent and 40 percent, respectively, believed that managers at their 
level had the decision-making authority they needed to achieve agency 
goals. 

The Lack of Prioritization of PART Recommendations Has Implications for 
Resource Allocation: 

Although OMB has given agencies discretion to define strategies to 
implement recommendations, OMB officials told us that, as a matter of 
policy, they have generally not prioritized the recommendations within 
each agency or across the more than 1,700 recommendations 
governmentwide because they do not want to dilute attention paid to any 
of the recommendations by deeming them a lower priority. Realistically, 
though, agencies cannot act on all of them concurrently. Because OMB 
has chosen to assess nearly all federal programs, resources are 
diffused across multiple areas instead of concentrated on those areas 
of highest priority both within agencies and across the federal 
government. This strategy is likely to lengthen the time it will take 
to observe measurable change. 

Moreover, as we report in our companion report on the PART evaluation 
recommendations, agency officials questioned the PART's assumption that 
all programs should have evaluations. Agency officials in one of our 
case study agencies said that they were able to fund some evaluations 
for small programs without cutting into program budgets, but other 
agency officials pointed out that spending several hundred thousand 
dollars for an evaluation study was a reasonable investment for large 
programs; they questioned whether all programs, regardless of size or 
importance, need to be evaluated, especially in times of tight 
resources and suggested instead a risk-based approach to prioritizing 
programs to be evaluated.[Footnote 20] We also noted that the lack of 
prioritization meant that agencies were free to choose which programs 
to evaluate first, and were likely to be influenced by such factors as 
the potential effect of the PART reassessments on their PMA scores. 

PART Recommendations Provide Wide Latitude for Agency Actions: 

OMB gives agencies wide latitude to implement the PART recommendations, 
which had both positive and negative effects on agency actions. Some 
officials appreciated the flexibility that OMB provided by not making 
prescriptive recommendations. They said that they were generally able 
to devise implementation strategies that suit programmatic needs and in 
most cases were able to devise implementation strategies that fit 
within existing agency plans and procedures. While they discuss their 
strategies with OMB, it is generally up to agency staff to determine 
the best course of action to implement the recommendations. In other 
cases, though, agency officials said that the recommendations were so 
broad as to be vague. This sometimes hampered implementation. 

For example, a DOE program received a recommendation to "improve 
performance reporting by grantees and contractors by September, 2004." 
DOE officials told us that in this case, the desired result is unclear. 
The PART requires that they report grantee performance both aggregated 
on a programwide level and disaggregated at the grantee level. DOE 
officials said that because they already report grantee information in 
each of these ways for both the PART and their Performance and 
Accountability Report (PAR), and because the recommendation does not 
describe the deficiencies in the reporting, they are unsure how to 
change their reporting practices to meet OMB's needs. Our review of 
this program's PART worksheet supports this view. Although we found one 
reference to "inadequate communication in the PAR of program-level 
aggregate data on the impact of the grants program" in the detailed 
supporting worksheet for this program, the reason for the inadequacy is 
unclear. In cases such as these, it is difficult to see how OMB and 
agencies can monitor progress. Given the importance OMB places on 
implementing the PART recommendations, it is important that 
recommendations clearly identify deficiencies and provide a basis for 
determining whether they are complete. 

OMB's and Agencies' Expectations Differed on Evaluations Conducted in 
Response to PART Recommendations: 

Federal agencies are increasingly expected to demonstrate effectiveness 
in achieving agency or governmentwide goals. Our previous work has 
shown that the accuracy and quality of evaluation information necessary 
to make the judgments called for when rating programs is highly uneven 
across the federal government. To help explain linkages between program 
activities, outputs, and outcomes, program evaluation designs are 
tailored to address various types of programs and questions. For 
example, a process evaluation reviews various aspects of program 
operations to assess the extent to which a program is operating as 
intended. Alternatively, an impact evaluation depends on scientific 
research methods to assess the net effect of a program by comparing 
program outcomes with an estimate of what would have happened in the 
absence of the program, in order to isolate the program's contributions 
to the observed outcomes. In other words, evaluations are useful to 
specific decisionmakers to the degree that the evaluations are credible 
and address their information needs. 

Our companion report notes that although the evaluation recommendations 
provided agencies with flexibility to interpret what evaluation 
information OMB expected and which evaluations to fund, a few programs 
did not discuss their evaluation plans with OMB; combined with 
different expectations on defining the scope and purpose of evaluations 
and disagreements about the quality of evaluation designs, it is not 
certain whether these evaluations will meet OMB's needs.[Footnote 21] 

Disagreements about the Scope and Purpose of Evaluations: 

OMB and our case study agencies significantly differed in defining 
evaluation scope and purpose. Some of the difficulties seemed to derive 
from the OMB examiners' expecting to find, in the agencies' external 
evaluation studies, comprehensive judgments about program design, 
management, and effectiveness, similar to the judgments made in the 
PART examinations. 

Because evaluations designed for internal and external audiences often 
have a different focus, differences of opinion on the usefulness of 
evaluations are perhaps not surprising. Evaluations that agencies 
initiate typically aim to identify how to improve the allocation of 
program resources or the effectiveness of program activities. Studies 
requested by program authorizing or oversight bodies, such as OMB, are 
more likely to address external accountability--to judge whether the 
program is properly designed or is solving an important problem. 

HHS departmental officials reported having numerous differences with 
OMB examiners over the acceptability of their evaluations. HHS 
officials were particularly concerned that OMB sometimes disregarded 
their studies and focused exclusively on OMB's own assessments. One 
program official complained that OMB did not adequately explain why the 
program's survey of refugees' economic adjustment did not qualify as an 
"independent, quality evaluation," although an experienced, independent 
contractor conducted the interviews and analysis. In the published PART 
review, OMB acknowledged that the program surveyed refugees to measure 
outcomes and monitored grantees on site to identify strategies for 
improving performance. In our subsequent interview, OMB staff explained 
that the outcome data did not show the mechanisms by which the program 
achieved its outcomes and grantee monitoring did not substitute for 
obtaining an external evaluation, or judgment, of the program's 
effectiveness. Other HHS officials said that OMB had been consistent in 
applying the standards for independent evaluation, but these standards 
were set extremely high. 

In reviewing a vaccination program, OMB did not accept the several 
research and evaluation studies offered, since they did not meet all 
key dimensions of "scope." OMB acknowledged that the program had 
conducted several management evaluations of the program to see whether 
the program could be improved but found their coverage narrow and 
concluded "there have previously been no comprehensive evaluations 
looking at how well the program is structured/managed to achieve its 
overall goals." The examiner also did not accept an external Institute 
of Medicine evaluation of how the government could improve its ability 
to increase immunization rates because the evaluation report had not 
looked at the effectiveness of the individual federal vaccine programs 
or how the program complemented the other related programs. However, in 
reviewing recommendation status, OMB credited the program with having 
contracted for a comprehensive evaluation that was focused on the 
operations, management, and structure of this specific vaccine program. 

Disagreements about the Quality of Evaluation Designs: 

OMB and agencies differed in identifying which evaluation methods were 
sufficiently rigorous to provide high-quality information on program 
effectiveness. OMB guidance encouraged the use of randomized controlled 
trials, or experiments, to obtain the most rigorous evidence of program 
impact but also acknowledged that these studies are not suitable or 
feasible for every program. However, without guidance on which--and 
when--alternative methods were appropriate, examiners and agency staff 
disagreed on whether specific evaluations were of acceptable quality. 
To help develop shared understandings and expectations, federal 
evaluation officials and OMB examiners held several discussions on how 
to assess evaluation quality according to the type of program being 
evaluated. 

When external factors such as economic or environmental conditions are 
known to influence a program's outcomes, an impact evaluation attempts 
to measure the program's net effect by comparing outcomes with an 
estimate of what would have occurred in the absence of the program 
intervention. A number of methodologies are available to estimate 
program impact, including experimental and quasi-experimental designs. 
Experimental designs compare the outcomes for groups that were randomly 
assigned to either the program or to a nonparticipating control group 
prior to the intervention. The difference in these groups' outcomes is 
believed to represent the program's impact, assuming that random 
assignment has controlled for any other systematic difference between 
the groups that could account for any observed difference in outcomes. 
Quasi-experimental designs compare outcomes for program participants 
with those of a comparison group not formed through random assignment, 
or with participants' experience prior to the program. Systematic 
selection of matching cases or statistical analysis is used to 
eliminate any key differences in characteristics or experiences between 
the groups that might plausibly account for a difference in outcomes. 

Randomized experiments are best suited to studying programs that are 
clearly defined interventions that can be standardized and controlled, 
and limited in availability, and where random assignment of 
participants and nonparticipants is deemed feasible and ethical. Quasi- 
experimental designs are also best suited to clearly defined, 
standardized interventions with limited availability, and where one can 
measure, and thus control for, key plausible alternative explanations 
for observed outcomes. In mature, full-coverage programs where 
comparison groups cannot be obtained, program effects may be estimated 
through systematic observation of targeted measures under specially 
selected conditions designed to eliminate plausible alternative 
explanations for observed outcomes.[Footnote 22] 

Following our January 2004 report recommendation that OMB better define 
an "independent, quality evaluation," OMB revised and expanded its 
guidance on evaluation quality. The guidance encouraged the use of 
randomized controlled trials as particularly well suited to measuring 
program impacts but acknowledged that such studies are not suitable or 
feasible for every program, so it recommended that a variety of methods 
be considered. OMB also formed an Interagency Program Evaluation 
Working Group in the summer of 2004 which discussed this guidance 
extensively to provide assistance on evaluation methods and resources 
to agencies undergoing a PART review. Evaluation officials from several 
federal agencies expressed concern that the OMB guidance materials 
defined the range of rigorous evaluation designs too narrowly. In the 
spring of 2005, representatives from several federal agencies 
participated in presentations about program evaluation purposes and 
methods with OMB examiners. They outlined the types of evaluation 
approaches they considered best suited for various program types and 
questions. (See fig. 3.)[Footnote 23] However, OMB did not 
substantively revise its guidance on evaluation quality for the fiscal 
year 2007 reviews beyond recommending that "agencies and OMB should 
consult evaluation experts, in-house and/or external, as appropriate, 
when choosing or vetting rigorous evaluations."[Footnote 24] 

Figure 3: Federal Evaluators' Views on Tailoring Designs for Program 
Effectiveness Evaluations: 

[See PDF for image] 

Source: Eric Bernholz and others, Evaluation Dialogue between OMB Staff 
and Federal Evaluation Leaders: Digging a Bit Deeper into Evaluation 
Science, (Washington, D.C.: Apr. 2005). 

[End of figure] 

The PART Framework Has Been Applied to Crosscutting Programs, but More 
Needs to Be Done: 

Although assessing programs in isolation can yield useful information, 
it is often critical to understand how individual programs fit within a 
broader portfolio of tools and strategies--such as regulations, credit 
programs, and tax expenditures--to accomplish federal goals. Such an 
analysis is necessary to capture whether a program complements and 
supports other related programs, whether it is duplicative and 
redundant, or whether it actually works at cross-purposes to other 
initiatives. 

Although variations of the PART tool are meant to capture the different 
approaches to service delivery, such as grants versus direct federal 
activities, not all approaches--such as tax expenditures--are 
systematically assessed by the PART. Tax expenditures may be aimed at 
the same goals as spending programs but little is known about their 
effectiveness or their relative efficacy when compared to related 
spending programs in achieving the objectives intended by Congress. 
Since in some years, tax expenditures are about the same order of 
magnitude as discretionary programs and in some program areas tax 
expenditures may be the main tool used to deliver services, this is a 
significant gap. We recently recommended that OMB require that tax 
expenditures be included in the PART and any future such budget and 
performance review processes so that tax expenditures are considered 
along with related outlay programs in determining the adequacy of 
federal efforts to achieve national objectives.[Footnote 25] 

Consistent with recommendations in our January 2004 report, OMB has 
begun to use the PART framework to conduct crosscutting assessments. 
OMB reported on two crosscutting PART assessments--Rural Water programs 
and Community and Economic Development (CED) programs--for the fiscal 
year 2006 budget and it plans to conduct three additional crosscutting 
reviews on block grants, credit programs, and small business innovation 
research for the fiscal year 2007 budget. 

According to both OMB and agency participants in the cooperative CED 
assessment, several aspects worked well. For example, the CED effort 
leveraged federal governmentwide community and development expertise 
housed in the OMB Interagency Collaborative on Community and Economic 
Development (ICCED).[Footnote 26] The group focused on four elements: 
(1) determining the programs to be included in such a comparison; (2) 
reaching agreement on the goals and objectives of similar programs; (3) 
identifying opportunities to better coordinate, target, leverage, and 
increase efficiency and effectiveness of similar programs; and (4) 
establishing a common framework of performance measures and 
accountability.[Footnote 27] Cognizant agency officials were pleased 
with this collaborative interagency process. They found value in 
leveraging existing efforts within agencies and benefited from OMB 
staff consultation. 

The CED crosscutting assessment examined the performance of 18 of the 
35 federal community and economic development programs identified by 
OMB and account for the majority of the $16.2 billion OMB estimates is 
spent annually in this area.[Footnote 28] Although OMB identified three 
tax expenditures in the CED portfolio, it did not assess all of them 
with the PART instrument even though the Department of the Treasury's 
(Treasury) estimate of their combined "cost" is nearly $1.4 billion, or 
about 57 percent of Treasury's revenue loss estimates for community 
development.[Footnote 29] 

Little information on the CED crosscutting assessment was initially 
available beyond the brief description contained in the Analytical 
Perspectives volume of the Fiscal Year 2006 President's budget request. 
Some OMB documents and administration officials stated that all 18 
programs had been assessed by the PART. However, 8 of the 18 programs 
proposed for consolidation were actually assessed by the PART. Because 
PART programs do not always clearly align with the individual programs 
proposed for consolidation, it can be difficult to determine which 
programs were assessed with the PART and which were not. As the CED 
team itself recognized, the results of a crosscutting assessment need 
to be communicated to stakeholders and the public. Unless the scope, 
purpose, and results are clear to stakeholders, the fruits of 
crosscutting assessments will likely not be realized. 

In choosing programs to assess and reassess with the PART, OMB 
considers a variety of factors, including continuing presidential 
initiatives, whether a program is up for reauthorization, and whether a 
program received a rating of "results not demonstrated" in a previous 
PART review. Although these are reasonable criteria, a greater emphasis 
on selecting programs related to common or similar outcomes for review 
in a given year would enable decision makers to better analyze the 
efficacy of programs related to such outcomes, and improve the 
usefulness of crosscutting reviews conducted under the PART framework. 
Moreover, using PART assessments to review the relative contributions 
of similar programs to common or crosscutting goals and outcomes 
established through the GPRA process could improve the connection 
between the PART and GPRA. Developing a more strategic approach to 
selecting and prioritizing programs to be reassessed under the PART can 
also help ensure that OMB and agencies are using limited staff 
resources to the best advantage. 

The PART-GPRA Relationship Does Not Adequately Consider the Different 
Needs of the Budget and Planning Processes and Their Various 
Stakeholders: 

Although both the PART and GPRA aim to improve the focus on program 
results, the different purposes and time frames they serve continue to 
create tensions. Some agencies have made progress over the past several 
years in reconciling the PART and GPRA, however, unresolved tensions 
can result in conflicting ideas about what to measure, how to measure 
it, and how to report program results. We continue to find evidence 
that the closed nature of the executive budget formulation process may 
not allow for the type of consultative stakeholder involvement in 
strategic and annual planning envisioned by GPRA. We remain concerned 
that the focus of agency strategic planning is shifting from long-term 
goal setting to short-term budget and planning needs. 

Agencies Continue to Struggle with Integrating the PART and GPRA: 

OMB attempted to clarify the relationship between the PART and GPRA in 
its PART guidance in 2004. The guidance notes that the PART strengthens 
and reinforces performance measurement under GPRA by encouraging the 
careful development of performance measures according to GPRA's outcome-
oriented standards. It also requires that the PART goals be 
"appropriately ambitious" and that GPRA and PART performance measures 
be consistent. 

Some agencies have made progress over the past several years in 
reconciling the PART and GPRA. For example, DOE and SBA officials told 
us that their existing GPRA measures now relate to or are generally 
accepted for PART purposes. Officials from DOE's Office of Science and 
Labor's Employment Standards Administration told us that OMB actively 
encouraged them to use their GPRA measures in the PART. HHS's Breast 
and Cervical Cancer, Diabetes, and Foster Care programs as well as the 
Administration on Developmental Disabilities were able to use some 
existing GPRA measures as annual PART measures. These experiences are 
consistent with OMB's view that although the PART and GPRA are often 
focused on different sets of measures, the characteristics of both sets 
should be the same (e.g., outcome-oriented, ambitious) and support 
OMB's belief that they have adequately clarified the relationship 
between the PART and GPRA. 

However, some agency officials we spoke with described persistent 
difficulties in integrating the two processes. Some described the PART 
and GPRA as duplicative processes that strain agency resources; others 
said they conflicted. As described below, defining a "unit of analysis" 
and performance measures useful for both budget and performance 
purposes remains a challenge. One official noted, "there is almost an 
unavoidable conflict between data that is useful from a governmentwide 
perspective and data that is useful to program managers." Although the 
Breast and Cervical Cancer and Diabetes programs had some success 
marrying their annual GPRA measures with short-term PART measures, they 
found that OMB did not consider their long-term GPRA goals to be 
aggressive enough; the measures were revised to meet OMB's needs. 

Defining a "Unit of Analysis" Useful for Both Budget and Planning 
Purposes Remains a Challenge: 

OMB acknowledges that to improve performance and management decisions, 
OMB and agencies should determine an appropriate "unit of analysis" for 
a PART assessment. The PART guidance notes that although the budget 
structure is not perfect for program review in every instance, the 
budget structure is the most readily available and comprehensive system 
for conveying PART results. In response to our January 2004 report, OMB 
expanded its guidance on how the unit of analysis is to be determined. 
The guidance notes that interdependent programs or program activities 
can be combined for purposes of the PART as long as the aggregated unit 
of analysis for the PART is able to illuminate meaningful management 
distinctions among programs that share common goals but are managed 
differently. Moreover, it notes that several factors should be 
considered when deciding whether to combine programs, such as a 
program's purpose, design, and administration; budgeting; and whether 
the programs support similar outcome goals. 

Although less of a problem than it was during our January 2004 review 
of the PART process, difficulties in defining a unit of analysis useful 
for both OMB's budget analysis and program management purposes remain, 
and continue to highlight the tension between the PART and GPRA. Some 
agency officials acknowledged that overly disaggregating programs for 
the PART sometimes does not provide an understanding of how the entire 
program or service delivery system works before attempting to assess 
the performance of component pieces. One official described it as 
"putting the cart before the horse." Some agency officials noted that 
difficulties can also arise when unrelated programs and programs with 
uneven success levels are combined for the PART. For example, OMB 
combined programs authorized under titles VII and VIII of the Public 
Health Services Act to create the Health Professions PART program. As 
required by the PART guidance, the entire PART program received a "no" 
for each question where any of the PART program components did not meet 
the requirements for a "yes" answer. As agency officials recognized, 
assessing the programs separately would have made each program's 
successes and weaknesses more transparent. They felt this was 
important, as the individual programs have different underlying program 
authorities, goals, and attempt to address the maldistribution of 
health professionals in a variety of ways. In other words, although 
they complement each other, they serve different needs. OMB senior 
officials acknowledged that combining programs could theoretically make 
each component's successes and challenges less apparent, but that in 
this case it is hard to argue that programs authorized by different 
titles of the Public Health Services Act are unrelated to each other. 

The goals and recommendations developed in a PART review, and hence the 
overall quality of the review, may suffer when the unit of analysis is 
not properly targeted. For example, the Centers for Disease Control and 
Prevention's (CDC) National Immunization Program (NIP) includes both 
the 317 program--which provides funding to support 64 state, local, and 
territorial public health immunization programs for program operations 
and vaccine purchase--and the Vaccines for Children (VFC) program-- 
which provides publicly purchased vaccines to participating providers 
which are then given to eligible children without cost to the provider 
or the parent. Only the 317 program has been assessed by the PART to 
date. In its PART assessment of the 317 Program, OMB noted that the 
administration was including a legislative proposal in the fiscal years 
2004 and 2005 budget requests to "make it easier for uninsured children 
who are eligible for the CDC Vaccines for Children (VFC) program to 
receive immunizations in public health clinics, to improve program 
efficiency in the overall childhood immunization program. This proposal 
will expand the VFC program and result in $110 million in savings to 
the 317 discretionary childhood immunization program." According to HHS 
officials, these proposals are outside the scope of the 317 program. 
They said that the 317 program's stakeholders believe that OMB 
penalized the 317 program by recommending a change in that program that 
only the VFC program could accomplish. Program officials were unable to 
convince OMB to remove the VFC legislative proposal from the 317 
program PART assessment summary sheet. Similarly, when OMB proposed a 
goal related to the global eradication of polio, HHS officials were 
unable to convince OMB that while global eradication of polio is a goal 
of the NIP overall, it is not within the scope of the individual 317 
program, which is solely a domestic program. Although one of the 
program's annual measures is still the "number of polio cases 
worldwide," OMB responded to the agency's concern in the most recent 
PART summary sheet for the 317 program, noting that "the global polio 
measure will be tracked by the global immunization program, which will 
be assessed separately in the future, and not by the 317 program." 

Concerns about PART Measures Reflect Long-standing Tensions over 
Performance Measurement: 

We have long recognized the difficulties of developing useful results- 
oriented performance measures for programs such as those geared toward 
long-term health outcomes and research and development (R&D) programs. 
However, in a June 1997 report discussing the challenges of GPRA 
implementation, we also said that although such performance measurement 
efforts were difficult, they have the potential to provide important 
information to decision makers. We noted that agencies were exploring a 
number of strategies to address these issues, such as using program 
evaluations to isolate program impact, developing intermediate 
measures, employing a range of measures to assess progress, and working 
with stakeholders to seek agreement on appropriate measures.[Footnote 
30] OMB recognizes several of these approaches as appropriate 
alternatives to outcome measures for PART purposes but as described 
below, agencies have had mixed success reaching agreement with OMB in 
these areas. Although these types of measurement challenges are clearly 
not new or unique to the PART, they are aggravated by the difficulties 
of developing measures useful for multiple purposes and audiences and 
often remain a point of friction in agencies and sometimes within OMB. 

For programs that can take years to observe program results, it can be 
difficult to identify performance measures that will provide 
information on the annual progress they are making toward achieving 
program results. This can complicate efforts to arrive at goals useful 
to multiple parties for multiple purposes. For example, CDC officials 
told us that long-term health outcome measures favored by the PART are 
often not as useful to them as data on preventative measures, which 
tell managers where more efforts are needed and allows them to respond 
more quickly. 

Programs where the federal government is one among many actors present 
similar challenges--when an outcome is beyond the scope of any one 
program, any changes made to a single federal program will not 
necessarily have an immediate effect. For example, for the Diabetes 
program OMB expressed interest in a long-term health outcome measure 
that tracks changes in national blindness and amputation rates. Program 
officials said that these types of changes generally cannot be 
attributed solely to the Diabetes program because it serves a 
relatively small portion of the population and works with many 
partners. The Breast and Cervical Cancer program--which screens low- 
income women and provides public education, quality assurance, 
surveillance, partnerships, and evaluation regarding breast cancer 
screening among low-income women--has similar concerns about OMB's 
interest in linking the program to changes in the overall mortality 
rates of cancer patients. 

Agency experiences with the PART's emphasis on efficiency measures 
presented a more varied picture. Some programs had success by defining 
efficiency in terms of program administration rather than program 
operations. For example, HHS's foster care officials said that because 
children's safety could have been compromised by moving children too 
quickly out of foster care, OMB agreed that an administrative 
efficiency measure would be appropriate instead of the type of outcome- 
oriented efficiency measure cited above. DOE officials told us that the 
Strategic Petroleum Reserves program is well suited to the PART's view 
of "efficiency" because it can show (1) how every dollar from its 
budget is spent, (2) that it is spent efficiently, and (3) that the 
program results related to spending those dollars. 

In other cases, differences of opinion about efficiency measures 
highlighted the challenges that can result from the difficult but 
potentially useful process of comparing the costs of programs related 
to similar goals. For example, DOL agency officials told us that since 
Job Corps is a self-paced program, participants are permitted to remain 
in the program for up to 2 years (or up to 3 years with special 
approval). They consider this to be adequate time for students to 
complete their education and/or vocational training, which, as studies 
indicate, generally results in higher wages. DOL agency officials noted 
that since costs per participant increase the longer a student remains 
in the program, Job Corps appears less "efficient" compared with other 
job training programs, which reflects poorly in Job Corps' PART 
assessment. They suggested cost per student day as a cost measure with 
less inherent perverse incentive, but OMB did not accept the 
suggestion. 

Similarly, DOL agency officials explained that whereas Job Corps' 
current GPRA measures track the percentage of job/education placements 
for program exiters who graduate, the common measures--which OMB uses 
to gauge performance across all job-training programs--track entered 
employment/education for all program exiters, regardless of their 
graduate status at exit.[Footnote 31] Although there are significant 
differences in the time frames, the placement criteria, and the pool of 
participants for these measures, these officials told us that the 
measures are treated as interchangeable in the PART review. In other 
words, the same benchmark set for the "graduate placement" GPRA 
indicator is also used for the "placement of all participants" common 
measure indicator. Consequently, agency officials said, Job Corps is in 
the position of either (1) failing to meet the common measure goal or 
(2) being labeled un-ambitious by OMB if the goal is changed to be 
attainable yet--in DOL's view--still aggressive. Either way, the agency 
officials said that their PART assessment is negatively affected. 

Several R&D officials noted that prior to the PART, there had been a 
collaborative effort to develop OMB's R&D investment criteria to better 
assess the value of R&D programs. However, these managers believed that 
the investment criteria--which R&D program managers find useful to 
manage their programs--have been overshadowed by the PART--which OMB 
finds useful in its budget development process. Part of the trouble 
seems to be that the PART explicitly requires all programs to have or 
be developing an efficiency measure, while the investment criteria do 
not. The investment criteria focus on improving the management of basic 
research programs. One agency official noted that this is a management 
efficiency question, not a cost question; therefore it should be 
captured in the PART's management section instead of the results 
section. Such a change could affect a program's PART score because the 
management section represents 20 percent of the total weighted score 
whereas the results section represents 50 percent of the total weighted 
score. 

In the investment criteria published with the 2004 PART guidance, OMB 
noted that it had worked to clarify the implementation of the 
investment criteria, stating that the investment criteria are broad 
criteria for all R&D programs while the PART is used to determine 
compliance with the investment criteria at the program level. OMB also 
recognized that while programs must demonstrate an ability to manage in 
a manner that produces identifiable results, taking risks and working 
toward difficult-to-attain goals are important aspects of good research 
management, especially for basic research. They further note that the 
intent of the investment criteria is not to drive basic research 
programs to pursue less risky research that has a greater chance of 
success, and that the administration will focus on improving the 
management of basic research programs. 

Disagreements over when and how to revise and communicate information 
about federal programs further highlight tensions between OMB and 
agencies. OMB Circular A-11 states that the performance targets 
included in the PARTs and congressional justifications need to be 
updated to reflect the budgetary resources and associated performance 
targets decided for the President's budget, and that budget and 
performance reports should identify changes to performance goals that 
primarily stemmed from assessing actual performance. However, several 
agency officials reported problems with adjusting or retiring goals. 
For example, agency officials told us that sometimes goals need to be 
retired or consolidated, and cited instances in which they were not 
permitted to do so even after intense negotiation with OMB. They said 
that changing goals disrupts the ability to observe historical trends, 
making it hard for OMB to measure against a baseline. We recognize the 
value of baseline information and that changing goals and measures can 
limit the ability to observe trends over time. However, this is not 
always possible. Revised performance information can also further 
enhance performance assessments. 

Nature of the Budget Formulation Process Limits Stakeholder 
Involvement: 

As we have previously reported, successful organizations base their 
strategic planning to a large extent on the interests and expectations 
of their stakeholders, since they recognize that stakeholders will have 
a lot to say in determining whether their programs succeed or fail. 
Congress, the executive branch, and other stakeholders may all strongly 
disagree about a given agency's missions and goals--in fact, full 
agreement among stakeholders on all aspects of an agency's efforts is 
relatively uncommon because stakeholders' interests can differ 
significantly. Still, stakeholder involvement is important to help 
agencies ensure that their efforts and resources are targeted at the 
highest priorities. Just as important, involving stakeholders-- 
especially Congress--in strategic planning efforts can help create a 
basic understanding among stakeholders of the competing demands that 
confront most agencies. Because of Congress's constitutional power to 
create and fund programs, congressional involvement is indispensable to 
defining each agency's mission and establishing its goals. 

Some tension between the level of stakeholder involvement in the 
development of performance measures in the GPRA strategic planning 
process and the process of developing performance measures for the PART 
executive is inevitable. Compared to the relatively open-ended GPRA 
process, any executive budget formulation process is likely to seem 
closed. An agency's communication with stakeholders, including 
Congress, about goals and measures created or modified during the 
formulation of the President's budget, is likely to be less than during 
the development of the agency's own strategic or performance plan. 

Although OMB's PART guidance discusses the need to integrate the PART 
and GPRA, we continue to find evidence that the closed nature of the 
executive budget formulation process may not allow for the type of 
stakeholder involvement in strategic and annual planning envisioned by 
GPRA. Beginning with the fiscal year 2005 budget submission, OMB 
required agencies to submit a performance budget, which is expected to 
satisfy all statutory requirements of the GPRA annual performance plan. 
It is generally expected to include the PART performance goals 
(including annual and long-term performance measures with targets and 
time frames) for programs that have been assessed by the PART.[Footnote 
32] The PART guidance recognizes stakeholder involvement in strategic 
planning as required by GPRA by saying agencies must consult with 
Congress and solicit and consider the views of interested and 
potentially affected parties. 

At the same time, the executive budget formulation process--to which 
the PART belongs--is "predecisional." This means that information from 
the annual budget process, including information required in agencies' 
annual GPRA plans, is embargoed within the executive branch until the 
President's budget request is transmitted to Congress. Agencies may 
therefore be prevented from consulting with their stakeholders when 
developing annual and long-term goals and measures. Some of our case 
study agencies described similar experiences. Their interaction with 
key stakeholders was limited. Sometimes they had to present new or 
revised program goals and measures to their stakeholders after the 
fact, and in some cases stakeholders disagreed with the goals, or had 
no choice but to accept them after the fact. 

Discussions of how performance information is being used are important 
because GPRA performance reports are intended to be one of Congress's 
major accountability documents. As such, these reports are to help 
Congress assess agencies' progress in meeting goals and determine 
whether planned actions will be sufficient to achieve unmet goals, or, 
alternatively, whether the goals should be modified. Because 
predecisional performance information must be excluded from the 
reports, their potential as a source of information to Congress is 
limited. However, this embargo conflicts with OMB's own reporting 
requirements regarding the issuance of agency Performance and 
Accountability Reports (PAR). OMB's Circular A-11 guidance notes that 
the transmittal date for an agency's PAR is November 15, and that 
because this precedes the transmittal of the President's budget, an 
agency may need to omit certain "privileged" information from its 
PAR.[Footnote 33] As described in Circular A-11, this privileged 
information includes specifically required elements of agency PARs, 
including an evaluation of performance for the current fiscal year; 
schedules for achieving established performance goals; and, if a 
performance goal is impractical or infeasible, an explanation of why 
that is the case and what action is recommended. However, OMB senior 
officials told us that the only information that cannot be included in 
a PAR is that related to target levels of funding and/or policy 
changes. 

Tailoring Outreach to Meet Congressional Needs Is Key to Increasing the 
Likelihood of Congress's Considering the PART in Its Deliberations: 

While the PART has been useful to OMB to achieve its own budget 
formulation goals, OMB acknowledges the need to work to gain 
congressional acceptance of the tool and its results. In response to 
our January 2004 report on the first year of implementing the PART, OMB 
said that it was working to "generate, early in the PART process, an 
ongoing, meaningful dialogue with congressional appropriations, 
authorization, and oversight committees about what they consider to be 
the most important performance issues and program areas warranting 
review." Although OMB uses a variety of methods to communicate the PART 
assessment results, congressional committee staff said these methods 
have not facilitated this early consultation on the PART. An absence of 
early consultation has contributed to several areas of disagreement 
between OMB and Congress about this executive branch tool, resulting in 
most congressional staff we spoke with not using the PART information. 
Most congressional staff reported that they would more likely use the 
PART results to inform their deliberations if OMB (1) consulted them 
early in the PART process regarding the selection and timing of 
programs to assess, (2) explained the methodology and evidence used or 
to be used to assess programs, and (3) discussed how the PART 
information can best be communicated and leveraged to meet their needs. 
Although OMB will be less likely to demonstrate the value of the PART 
beyond executive branch decision making without early consultation, OMB 
has had some success in engaging Congress when it has communicated 
selected PART results through legislative proposals and other 
traditional methods that clearly signal an executive branch priority. 
Although Congress currently has a number of opportunities to provide 
its perspective on specific performance issues and performance goals, 
opportunities also exist for Congress to enhance its institutional 
focus to enable a more systematic assessment of key programs and 
performance goals. 

OMB Used a Variety of Methods to Communicate PART Scores, but 
Congressional Staff Raised Concerns about These Methods: 

OMB uses a variety of methods to communicate PART results to both the 
public and to Congress, primarily through the President's budget 
request documents, OMB's Web site, and meetings with some congressional 
staff. For example, OMB provides the single, bottom-line PART ratings 
in the Analytical Perspectives volume of the President's budget 
request, while the one-page PART summary sheets are available on a CD- 
ROM accompanying the President's budget request or on OMB's Web site. 
The Web site also contains the detailed supporting worksheets as well 
as other information about the tool itself. Certainly, OMB has provided 
more extensive information on program performance than in the past. 

OMB's PART guidance also directed agencies to address the PART 
findings--from both current and prior year's PARTs--in their fiscal 
year 2006 budget submissions to OMB and budget justifications to 
Congress, as well as in testimony to Congress, in particular when a key 
budget or policy recommendation was influenced by a PART analysis. 
Agency witnesses testifying before the appropriations subcommittees did 
in fact include the results of the PART assessments in their written 
statements, and in some instances the PART was discussed during the 
"Q&A" portions of these hearings. 

In addition to requiring agencies to inform Congress about the PART, 
OMB offered to brief congressional committees about the PART in 2004. 
According to OMB, packages including the PART summary sheets for 
programs that fell within each committee's jurisdiction and a list of 
the programs OMB planned to review for the fiscal year 2006 budget 
request were sent to all relevant House and Senate committees. An OMB 
senior official also said he followed up on these packages with phone 
calls, but received very little response. His records show that between 
February 2005 and June 2005 there were about 21 congressional meetings 
(bicameral and bipartisan) about the PART. In February 2005, upon the 
release of the Major Savings and Reforms in the President's 2006 Budget 
document, OMB held what it termed a briefing on the PART, inviting all 
appropriations staff. 

OMB has set an ambitious benchmark for involving Congress in the PART 
process. In recent testimonies,[Footnote 34] OMB's Deputy Director for 
Management stated that OMB's responsibility is to convince Congress 
that the PART assessments have correctly identified whether a program 
is working and, if not, what to do about it. In the past 3 years OMB 
states it has conducted 607 the PART assessments (about 60 percent of 
federal programs) that have generated nearly 1,800 recommendations. 
However, it is not clear that the PART has had any significant impact 
on congressional authorization, appropriations, and oversight 
activities to date. Moreover, it is unlikely that performance 
information will be used unless it is believed to be credible and 
reliable and reflects a consensus about performance goals among a 
community of interested parties. The PART likely has required a 
significant additional commitment of OMB's and agencies' resources, but 
demonstrating the value of the assessments beyond the executive branch 
will require further efforts. 

Despite OMB's Efforts, Congressional Staff Said There Is Little Early 
Consultation on the PART: 

According to OMB senior officials, OMB's efforts generally focused on 
providing an overview of the PART process or communicating program 
assessment results to Congress rather than seeking early consultation 
about how the tool can best serve congressional needs. For example, 
upon the release of the Major Savings and Reforms in the President's 
2006 Budget document, OMB said they invited leadership, appropriations, 
and budget committee staff to a presentation about it. However, some 
subcommittee staff said that the presentation was primarily intended to 
provide the Major Savings document that proposed program funding 
reductions and terminations, some of which were based on the PART 
assessments. Although some subcommittee staff said that they met with 
OMB and that OMB officials asked for their input about the PART, they 
did not see subsequent evidence that their views had been considered. 
Overall, most committee staff said that OMB generally did not involve 
them in the PART process. 

The need for early consultation is clearly demonstrated by the strong 
interest House appropriators expressed in being consulted early in the 
PART process about the programs, activities, or projects that OMB 
intends to assess for the fiscal years 2007 and 2008 budget requests, 
including approval of the methodology to be used to conduct each 
assessment. Congress went so far as to express these concerns in 
committee report language related to OMB's fiscal year 2006 
appropriations.[Footnote 35] Similar views were also echoed by many 
appropriations and authorizing committee staff we spoke with. As we 
have noted, some tension about the amount of stakeholder involvement in 
the internal deliberations surrounding the development of the PART 
measures and the broader consultations more common to the GPRA 
strategic planning process is inevitable.[Footnote 36] Compared to the 
relatively open-ended GPRA process, any executive budget formulation 
process is likely to seem closed. However, if the PART is to be 
accepted as something more than an executive branch budget formulation 
tool, congressional understanding and acceptance of the tool and its 
analysis will be critical. 

Because of Limited Agreement between OMB and Congress about the PART, 
Most Congressional Staff We Spoke with Do Not Use PART Information: 

A lack of early consultation has contributed to both congressional 
skepticism about the PART and to several areas of disagreement between 
OMB and Congress. As a result, most congressional staff we spoke with 
do not use PART information. Many committee staff we spoke with 
expressed frustration with the lack of available detail on how OMB 
arrived at their ratings of a program's performance. Many had concerns 
about the goals and measures used to assess program performance. Some 
subcommittee staff questioned the "unit of analysis" for the purposes 
of the PART as well as the design of the tool itself. The PART is OMB's 
tool of choice for assessing program performance and as such serves the 
administration's needs. However, it is only one source of information 
available to congressional committees. 

Concerns Raised about Lack of Detail in PART Summary Worksheets and 
Unconvincing PART Assessments: 

Several committee staff were frustrated with the lack of detail 
provided on the PART summary sheets as to why a program was rated a 
certain way. They were unlikely to accept conclusions about a program's 
performance without seeing the evidence used to support them, 
particularly when the rating was contrary to what they believed to be 
true about a program. For example, some appropriations subcommittee 
staff expressed concerns about the "ineffective" PART rating given to 
the Health Professions program, which assists in paying for health 
professionals' education in exchange for their working in underserved 
areas. They said OMB could have made a stronger case for this rating if 
it had provided information showing that the program is unsuccessful in 
placing participating health professionals in underserved areas. In 
general, many committee staff we spoke with said they do not use the 
Web site containing the detailed supporting worksheets, primarily 
because finding this information on the Web site is too time-consuming, 
or the Web site is difficult to use.[Footnote 37] 

Although the detailed supporting worksheet for the Health Professions 
program notes that the agency has not conducted evaluations necessary 
to measure the program's performance--thus a factor for the 
"ineffective" rating--OMB's explanation of this rating is not clearly 
stated on the one-page summary sheet. Several committee staff said they 
wanted detailed information or criteria used to evaluate the program so 
that they could reach their own conclusions about program 
effectiveness. Some subcommittee staff felt that if OMB intends to 
request funding reductions or program eliminations based on PART 
assessments, a special burden exists to prove that these programs are 
ineffective. 

In other cases, committee staff remained unconvinced about the PART 
ratings and the evidence used to support them. House appropriations 
subcommittee staff said that the Agricultural Credit Insurance Fund-- 
Direct Loans, which they had held hearings on, was rated "moderately 
effective;" however, the subcommittee staff questioned the basis on 
which this program was given this rating since the agency has written 
off many of its loans. Committee staff also cited a PART assessment 
that stated that SBA's 7(a) loan program and its 504 program overlap 
because both provide long-term financing for similar borrowers. The 
committee staff disagreed with this assessment.[Footnote 38] 

Lack of Agreement about PART Measures and Defining an Appropriate "Unit 
of Analysis" 

A lack of consultation early in the PART process has contributed to 
congressional committee staff not agreeing with or not finding useful 
OMB's choice or use of certain measures to determine the effectiveness 
of certain programs. Some committee staff reported that not all 
programs are well suited to being assessed by a tool like the PART. For 
example, a House subcommittee held a hearing in March 2004 that 
addressed concerns about defining acceptable PART measures for 
environmental research programs.[Footnote 39] Hearing witnesses noted 
that OMB permitted some research programs to use output or process 
measures while it held similar programs to stricter standards, 
requiring them to use outcome measures. During a recent House Budget 
Committee hearing on performance budgeting, an OMB senior official 
agreed with committee members that the PART needs a set of goals and 
measures useful to OMB and Congress.[Footnote 40] He added that 
consulting Congress early in the PART process, including discussions 
about how to make the PART useful for Congress, can better take place 
now that the PART has generated a critical mass of performance 
information. 

Some congressional staff were troubled by OMB's definition of certain 
programs--the "unit of analysis"--used for the PART assessments. They 
noted that what was useful for congressional budget deliberations 
sometimes differed from the unit of analysis OMB used to assess program 
performance in the PART. For example, appropriations subcommittee staff 
said that they often look at the performance of a particular project in 
determining how much funding to provide it. When OMB combines projects 
that are only loosely related by their authorizing statutes and rates 
them all as "ineffective" or "effective," this arrangement does not 
help Congress make trade-offs among those projects. 

Committees Use a Variety of Performance Information: 

A few committee staff we talked with said that they use the PART 
information as one of many sources of information about program 
performance, including inspectors general reports, agency-commissioned 
evaluations, National Academy of Sciences reports, GAO reports, and 
National Academy of Public Administration reports. Several indications 
of congressional attention to the PART results were reflected in recent 
appropriations committee reports. For example, a House Appropriations 
Committee report on fiscal year 2006 appropriations cites a PART 
assessment stating among other things that performance measures have 
still not been developed and that effects on Pacific salmon stocks are 
still unknown.[Footnote 41] The same committee applauds the Department 
of State's educational and cultural exchange programs (ECA), noting 
that "ECA received from the Office of Management and Budget Program 
Assessment Rating Tool [PART] ratings of 98 percent and 97 percent, the 
highest in the State Department and in the top one-percent in the 
Executive Branch."[Footnote 42] Another House Appropriations Committee 
report for fiscal year 2006 noted that DOE's natural gas and 
petroleum/oil research and development programs received a poor PART 
score. In response, the committee encouraged the department to develop 
a strategic planning process that "demonstrates a clear path of 
investment that will yield demonstrable results, and better reflect the 
successes of these programs."[Footnote 43] 

The PART's focus on outcome measures may not fully appreciate 
congressional needs for other types of measures, such as output and 
workload information. Committee staff said they consider a variety of 
performance information such as outcome, output, and input measures to 
help gauge program performance. We have previously reported that 
congressional staff are interested in using a diverse array of 
information to address key questions on program performance, such as 
recurring information on spending priorities within programs; the 
quality, quantity, and efficiency of program operations; as well as the 
populations served or regulated.[Footnote 44] Our recent work examining 
performance budgeting efforts at both the state and federal levels also 
bears this out. We found that appropriations committees consider 
workload and output measures important for making resource allocation 
decisions.[Footnote 45] Workload and output measures lend themselves to 
the budget process because workload measures, in combination with cost- 
per-unit information, can help relate appropriation levels to a desired 
level of service. 

Effective Communication and Consultation between OMB and Congress Could 
Increase the Role That the PART Plays in Congressional Deliberations: 

Despite its efforts, OMB has had limited success in engaging Congress 
in the PART process. For example, in June an OMB senior official 
testified that the PART had some effect on congressional 
authorizations, appropriations, or the oversight, but that OMB could 
clearly do a better job convincing Congress of the usefulness of 
performance information generated by the PART. Many majority and 
minority staff of House and Senate committees we talked with said that 
OMB should communicate the PART results in a way that meets individual 
committee needs. Most congressional committee staff said they would be 
more likely to use the PART results relevant to their committee 
responsibilities if OMB consulted with them early in the PART process 
and made PART information more useful for their work. They said it is 
important that such discussions also address performance information 
congressional committees find most useful. According to some staff, 
consulting them about congressional program priorities for PART 
assessments could be useful for linking these assessments to the 
authorization and appropriations processes by informing OMB about the 
committees' planned legislative agenda and informing Congress about 
programs OMB plans to assess in the near future. In discussing options 
for increasing congressional staff's access to performance information, 
we have previously noted that improved communication could go a long 
way to ensuring that congressional needs are understood and, where 
feasible, met.[Footnote 46] 

While some House and Senate committee staff stated that it would be 
difficult to conveniently time these consultations for both OMB and 
congressional staff, most agreed that they were a necessary step if 
Congress were to be able to use the PART to inform its deliberations. 
However, several majority and minority staff questioned how OMB could 
provide policy-neutral assessments given its institutional role. A 
couple of congressional subcommittee staff suggested that for any 
assessment to be considered credible it would have to be conducted or 
reviewed by an independent entity, such as a commission or a 
nonpartisan organization. 

OMB has sometimes been able to engage Congress when it has communicated 
selected PART results through traditional means of signaling executive 
branch priorities, such as legislative proposals. For example, as 
discussed previously, the administration recently proposed to 
consolidate 18 federal CED programs, including the Community 
Development Block Grant (CDBG), into a single block grant, citing as 
one factor the low PART scores received in a crosscutting review of CED 
programs. The proposal led to hearings by several committees, involving 
administration officials, programs' stakeholders, and experts. Although 
the full House and Senate Appropriations Committee rejected the 
President's proposal to transfer the CDBG program to the Department of 
Commerce and instead kept the program at the Department of Housing and 
Urban Development, the House and Senate reduced the funding level for 
the CDBG formula grants by $250 million and $347 million, respectively, 
from last year's level. 

Congress has initiated other hearings in which the PART has been a 
central subject of discussion. For example, OMB proposed funding cuts 
for the Environmental Protection Agency's science research grant 
programs (STAR) for the fiscal year 2005 budget because, according to a 
PART assessment, parts of STAR did not have adequate outcome measures 
and therefore could not demonstrate results. The Subcommittee on 
Environment, Technology, and Standards, House Committee on Science, 
held a hearing to discuss competing claims about whether these programs 
were contributing to their stated goals. 

The fact that Congress has held such hearings indicates that certain 
PART reviews have captured congressional attention and contributed to 
the policy debate. As we have previously noted, success in performance 
budgeting should not be defined only by its effect on funding decisions 
but by the extent to which it changes the kinds of questions raised in 
Congress and executive agencies.[Footnote 47] That is, performance 
budgeting helps shift the focus of congressional debates and oversight 
activities by changing the agenda of questions asked. 

Congress has a number of opportunities to provide its perspective on 
specific performance issues and performance goals--when it establishes 
or reauthorizes a new program, during the annual appropriations 
process, and in its oversight of federal operations. Opportunities also 
exist for Congress to enhance its institutional focus to enable a more 
systematic assessment of key programs and performance goals. For 
example, identifying the key oversight and performance goals that 
Congress wishes to set for its own committees and for the government as 
a whole, perhaps for major missions such as budget functions could be 
useful. Collecting the "views and estimates" of authorization and 
appropriations committees on priority performance issues for programs 
under their jurisdiction and working with such crosscutting committees 
as the House Committee on Governmental Reform and the House Committee 
on Rules could be an initial step. Such a process might not only inform 
and better focus congressional deliberations, but could allow for more 
timely input into the PART. 

It is important that Congress take full advantage of the benefits 
arising from the reform agenda under way in the executive branch. As we 
have suggested in the past, one approach to achieving the objective of 
enhancing congressional oversight is to develop a congressional 
performance resolution by modifying the current congressional budget 
resolution, which is already organized by budget function. Ultimately, 
what is important is not the specific approach or process, but rather 
the intended result of helping Congress better promote improved fiscal, 
management, and program performance through broad and comprehensive 
oversight and deliberation. 

Conclusions and General Observations: 

The federal government is in a period of profound transition and faces 
an array of challenges and opportunities to enhance performance, ensure 
accountability, and position the nation for the future. A number of 
overarching trends--including the nation's long-term fiscal imbalance-
-drive the need to reexamine what the federal government does, how it 
does it, who does it, and how it gets financed. Performance budgeting 
holds promise as a means for facilitating a reexamination effort and 
bringing the panoply of federal activities in line with the demands of 
today's world. It can help enhance the government's capacity to assess 
competing claims for federal dollars and has the potential to better 
inform the budget debate. 

PMA and its related initiatives, including the PART, demonstrate the 
administration's commitment to improving federal management and 
performance. Calling attention to successes and needed improvements is 
certainly a step in the right direction. The PART has helped perpetuate 
and sustain the performance culture ushered in by the management 
reforms of the 1990s. The PART has lent support to internal agency 
initiatives and--whatever criticism may be made regarding the value of 
scorecards and bottom-line ratings--has highlighted the need for 
improvements and motivated agencies to do more. 

There is no doubt that creating a closer link between the resources 
expended on programs and the results we expect from them is an 
important goal. The PART made a significant contribution by 
demonstrating one way to make a direct connection between performance 
and resource considerations. However, without truly integrating the 
PART and GPRA in a way that considers the differing needs of the budget 
formulation and strategic planning processes and their various 
stakeholders, OMB's ability to strengthen and further the performance- 
resource linkages for which GPRA laid the groundwork will continue to 
be hampered. 

Successful integration of the inherently separate but interrelated GPRA 
strategic planning and the PART performance budgeting processes is 
predicated on (1) ensuring that the growing supply of performance 
information is credible, useful, reliable, and used; (2) increasing the 
demand for this information by developing goals and measures relevant 
to the large and diverse community of stakeholders in the federal 
budget and planning processes; and (3) taking a comprehensive and 
crosscutting approach. 

By linking performance information to the budget process OMB has 
provided agencies with a powerful incentive for improving data quality 
and availability and has increased the potential for using performance 
information to inform the resource allocation process. To be effective, 
however, this information must not only be timely--to measure and 
affect performance--and reliable--to ensure consistent and comparable 
trend analysis over time and to facilitate better performance 
measurement and decision making--but also useful and used in order to 
make more informed operational and investing decisions. 

Improvements in the quality of performance data and the capacity of 
federal agencies to perform program evaluations will require sustained 
commitment and investment of resources. However, evaluations can be 
very costly; opportunities exist to carefully target federal evaluation 
resources such that the American people receive the most benefit from 
each evaluation dollar spent. Moreover, the question of investment in 
improved evaluation capacity is one that must be considered in budget 
deliberations both within the executive branch and in Congress. 
Importantly, it is critical that budgetary investments in this area be 
viewed as part of a broader initiative to improve the accountability 
and management capacity of federal agencies and programs. 

Some program improvements related to the PART's success--such as 
improving program outcomes, taking steps to address PART findings, 
improving program management, and becoming more efficient--can often 
come solely through executive branch action, but for the PART to meet 
its full potential the assessments it generates must also be meaningful 
to and used by Congress and other stakeholders. For the PART to result 
in congressional action on the PART's funding and policy 
recommendations as OMB desires, the PART must hold appeal beyond the 
executive branch. The PART was designed for and is used in the 
executive branch budget preparation and review process; as such, the 
goals and measures used in the PART must meet OMB's needs. Because OMB 
has not developed an effective strategy for connecting the PART process 
to congressional needs, Congress generally does not use the PART in its 
deliberations. Without developing an effective strategy for obtaining 
and acting on congressional views on what to measure, how to measure 
it, and how to best present this information to a congressional 
audience, it is more likely that PART will remain an executive branch 
exercise largely ignored in the authorization, appropriations, and 
oversight processes. Infusing a performance perspective into budget 
decisions may only be achieved when we discover ways to reflect both 
the broader planning perspective that can add value to budget 
deliberations and foster accountability in ways that Congress considers 
appropriate for meeting its roles, responsibilities, and interests. 
Congress also can facilitate the use of performance information by 
enhancing its focus on performance in budget, authorizing, 
appropriations, and oversight processes. 

Looking forward, opportunities exist to develop a more strategic 
approach to selecting and prioritizing areas for assessment under the 
PART process. Targeting PART assessments based on such factors as the 
relative priorities, costs, and risks associated with related clusters 
of programs and activities addressing related strategic and performance 
goals not only could help ration scarce analytic resources but also 
could focus decision makers' attention on the most pressing policy and 
program issues. Moreover, key outcomes in areas ranging from low income 
housing to food safety to counterterrorism are addressed by a wide 
range of discretionary, entitlement, tax, and regulatory approaches 
that cut across a number of agencies. Some tax expenditures amount to 
hundreds of billions of dollars of annual expenditures--the same order 
of magnitude as total discretionary spending, yet relatively little is 
known about the effectiveness of tax incentives in achieving the 
objectives intended by Congress. Broadening the PART to assess complete 
portfolios of tools used to achieve key federal outcomes is absolutely 
critical. A crosscutting approach could also facilitate the use of the 
PART assessments to review the relative contributions of similar 
programs to common or crosscutting goals and outcomes established 
through the GPRA process. 

As we have previously reported, effective congressional oversight can 
help improve federal performance by examining the program structures 
agencies use to deliver products and services to ensure that the best, 
most cost-effective mix of strategies is in place to meet agency and 
national goals. While Congress has a number of opportunities to provide 
its perspective on performance issues and performance goals, such as 
when it establishes or reauthorizes a program, during the annual 
appropriations process, and in its oversight of federal operations, a 
more systematic approach could allow Congress to better articulate 
performance goals and outcomes for key programs of major concern. Such 
an approach could also facilitate OMB's understanding of congressional 
priorities and concerns and, as a result, increase the usefulness of 
the PART in budget deliberations. 

Matter for Congressional Consideration: 

To facilitate an understanding of congressional priorities and 
concerns, Congress should consider the need for a strategy that 
includes (1) establishing a vehicle for communicating performance goals 
and measures for key congressional priorities and concerns; (2) 
developing a more structured oversight agenda to permit a more 
coordinated congressional perspective on crosscutting programs and 
policies; and (3) using such an agenda to inform its authorization, 
oversight, and appropriations processes. 

Recommendations for Executive Action: 

We make three recommendations to OMB. We recommend that the Director of 
OMB take the following actions: 

* Ensure that congressional leadership and key committees are given an 
opportunity to provide input early in the PART process on the 
performance issues and program areas they consider to be the most 
important and in need of review. 

* Seek input from congressional committees on the performance 
information they find useful and how that information could best be 
presented to them. 

* Target individual programs to be reassessed based on factors such as 
the relative priorities, costs, and risks associated with clusters of 
related programs, and in a way that reflects the congressional input 
described above. 

Agency Comments and Our Evaluation: 

In commenting on a draft of this report, OMB generally agreed with our 
findings, conclusions, and recommendations. OMB outlined several 
actions it is taking to address some of the issues raised in the 
report, including implementing information technology solutions to make 
application of the PART less burdensome and more collaborative. OMB 
also suggested some technical changes throughout the report that we 
have incorporated as appropriate. OMB's comments appear in appendix IV. 
We also received technical comments on excerpts of the draft from the 
Departments of Labor and Health and Human Services, which are 
incorporated as appropriate. 

We are sending copies of this report to the Director of the Office of 
Management and Budget and other interested parties. We will also make 
copies available to others upon request. In addition, the report will 
be available at no charge on GAO's Web site at [Hyperlink, 
http://www.gao.gov]. 

If you or your staff have any questions about this report please 
contact Susan Irving at (202) 512-9142 or [Hyperlink, irvings@gao.gov]. 
Contact points for our Offices of Congressional Relations and Public 
Affairs may be found on the last page of this report. GAO staff making 
key contributions to this report are listed in appendix V. 

Sincerely yours, 

Signed by: 

David M. Walker: 
Comptroller General of the United States: 

[End of section] 

Appendixes: 

Appendix I: Scope and Methodology: 

To address the first two objectives, we reviewed the Office of 
Management and Budget's (OMB) materials on the implementation, 
application, and revision of the Program Assessment Rating Tool (PART) 
for calendar years 2002 through 2004.[Footnote 48] We also interviewed 
OMB branch chiefs and OMB staff on the Performance Evaluation Team 
(PET). The PET's role is to provide guidance to budget examiners and 
help ensure consistent application of the PART across OMB offices. To 
better understand OMB's experience with crosscutting reviews, we 
interviewed OMB staff responsible for coordinating the Community and 
Economic Development and Rural Water crosscutting reviews conducted for 
the fiscal year 2006 President's budget request. To obtain agency 
perspectives on the relationship between the PART and the Government 
Performance and Results Act of 1993 (GPRA) and their interactions with 
OMB concerning that relationship, we interviewed department and agency 
officials, including senior managers, and program, planning, and budget 
staffs at (1) the Department of Health and Human Services (HHS), (2) 
the Department of Energy (DOE), (3) the Department of Labor (DOL), and 
(4) the Small Business Administration (SBA). We also interviewed 
officials from these departments and agencies concerning their 
perspectives and activities in response to the PART recommendations and 
the effects of implementing those recommendations on operations and 
results. 

We selected these three departments and one independent agency for a 
number of reasons. Collectively, they offered examples of all seven 
PART program types (e.g., block/formula grants, competitive grants, 
direct federal, and research and development) for review. These 
examples covered about a fifth of all the programs subject to the PART 
as of 2004 and thus could provide a broad-based perspective on how the 
PART was applied. We also chose to return to HHS and DOE--two of the 
departments included in our previous study on the PART.[Footnote 49] To 
broaden our coverage of agency perspectives we selected DOL and SBA 
because they had received a "green" score on their President's 
Management Agenda Executive Branch Management Scorecard for the budget 
and performance integration initiative and were considered good 
candidates for showing progress. Our selection of these four agencies 
was also influenced by our intent to integrate this work with our 
related work examining progress in addressing the PART program 
evaluation recommendations. Approximately half of the evaluation 
recommendations in the 2002 PART were encompassed in our four case 
selections. 

As part of our work on the second objective, we also performed various 
analyses of the PART recommendations made in all 3 years to discern 
possible changes or trends in recommendations over time and 
relationships between the type of recommendations made, type of 
program, overall rating, total PART score, and answers to selected PART 
questions. To do these analyses, we classified the recommendations OMB 
made into the same four categories we used in our prior report, i.e., 
program assessment, program design, program management, and funding. We 
employed a slightly modified classification procedure from our previous 
review, which included the addition of an "other" category for 
recommendations that did not fit within any of the four 
categories.[Footnote 50] We then combined the results of our 
recommendation classifications with selected data we downloaded from 
PART summaries and worksheets posted on OMB's PART Web sites, data 
developed for our previous report of the 2002 PART, and a data set 
provided by OMB of programs covered in the 2004 PART.[Footnote 51] In 
addition, we also examined relevant OMB and agency documents to help 
determine how recommendations are tracked and their impact evaluated by 
OMB and the selected agencies. 

To address our third objective of examining the steps OMB has taken to 
involve Congress in the PART process, we interviewed OMB and agency 
officials and asked questions about the steps OMB and agencies have 
taken to involve Congress in the PART process or in using the results 
of the PART. To obtain documented instances of Congress' uses and views 
of the PART, we interviewed House and Senate committee staff (minority 
and majority) for the authorizing and appropriations subcommittees with 
jurisdiction over our selected agencies as well as OMB and officials 
from the four selected agencies. Finally, we reviewed fiscal years 2005 
and 2006 House and Senate congressional hearing records and reports as 
well as conference reports for mentions of the PART. In addition, where 
possible, we corroborated testimonial evidence with documentary 
evidence of OMB's and agencies' strategies for involving Congress as 
well as evidence of collaboration and coordination, such as planning 
documents, briefing material, or other evidence of contact with 
Congress. 

We did not independently verify the PART assessments as posted on OMB's 
Web sites; however, we did take several steps to ensure that we 
reliably downloaded and combined the various data files with our 
recommendation classifications. Our steps included (1) having the 
computer programs we used to create and process our consolidated 
dataset verified by a second programmer; (2) performing various edit 
checks on the data and (3) selecting computer-processed data elements 
checked back to source files for a random sample of programs and also 
for specific programs identified in our analyses or through edit 
checks. We determined that the data were reliably downloaded and 
combined, and sufficient for the purposes of this report. 

While our summary analyses include all or almost all programs subject 
to the PART for the years 2002 to 2004 or all or almost all programs 
within a specified subset of programs (e.g., program type, specific 
year, specific rating), the information obtained from OMB, 
congressional and agency officials, as well as documentary material 
from the selected agencies is not generalizable to the PART process for 
all years or all programs. 

We conducted our audit work from January 2005 through August 2005 in 
accordance with generally accepted government auditing standards. OMB 
provided written comments on this draft that are reprinted in appendix 
IV. 

[End of section] 

Appendix II: The 2004 PART Questionnaire: 

Section I: Program Purpose & Design (Yes, No, N/A): 

1. Is the program purpose clear? 

2. Does the program address a specific and existing problem, interest, 
or need? 

3. Is the program designed so that it is not redundant or duplicative 
of any other Federal, State, local or private effort? 

4. Is the program design free of major flaws that would limit the 
program's effectiveness or efficiency? 

5. Is the program design effectively targeted, so that resources will 
reach intended beneficiaries and/or otherwise address the program's 
purpose directly? 

Section II: Strategic Planning (Yes, No, N/A): 

1. Does the program have a limited number of specific long-term 
performance measures that focus on outcomes and meaningfully reflect 
the purpose of the program? 

2. Does the program have ambitious targets and timeframes for its long- 
term measures? 

3. Does the program have a limited number of specific annual 
performance measures that can demonstrate progress toward achieving the 
program's long-term goals? 

4. Does the program have baselines and ambitious targets for its annual 
measures? 

5. Do all partners (including grantees, sub-grantees, contractors, cost-
sharing partners, and other government partners) commit to and work 
toward the annual and/or long-term goals of the program? 

6. Are independent evaluations of sufficient scope and quality 
conducted on a regular basis or as needed to support program 
improvements and evaluate effectiveness and relevance to the problem, 
interest, or need? 

7. Are Budget requests explicitly tied to accomplishment of the annual 
and long-term performance goals, and are the resource needs presented 
in a complete and transparent manner in the program's budget? 

8. Has the program taken meaningful steps to correct its strategic 
planning deficiencies? 

Specific Strategic Planning Questions by Program Type: 

Regulatory-Based Programs: 

RG1. Are all regulations issued by the program/agency necessary to meet 
the stated goals of the program, and do all regulations clearly 
indicate how the rules contribute to achievement of the goals? 

Capital Assets and Service Acquisition Programs: 

CA1. Has the agency/program conducted a recent, meaningful, credible 
analysis of alternatives that includes trade-offs between cost, 
schedule, risk, and performance goals and used the results to guide the 
resulting activity? 

Research and Development Programs: 

RD1. If applicable, does the program assess and compare the potential 
benefits of efforts within the program and (if relevant) to other 
efforts in other programs that have similar goals? 

RD2. Does the program use a prioritization process to guide budget 
requests and funding decisions? 

Section III: Program Management (Yes, No, N/A): 

1. Does the agency regularly collect timely and credible performance 
information, including information from key program partners, and use 
it to manage the program and improve performance? 

2. Are Federal managers and program partners (including grantees, sub- 
grantees, contractors, cost-sharing partners, and other government 
partners) held accountable for cost, schedule and performance results? 

3. Are funds (Federal and partners') obligated in a timely manner and 
spent for the intended purpose? 

4. Does the program have procedures (e.g., competitive sourcing/cost 
comparisons, IT improvements, appropriate incentives) to measure and 
achieve efficiencies and cost effectiveness in program execution? 

5. Does the program collaborate and coordinate effectively with related 
programs? 

6. Does the program use strong financial management practices? 

7. Has the program taken meaningful steps to address its management 
deficiencies? 

Specific Program Management Questions by Program Type: 

Competitive Grant Programs: 

CO1. Are grants awarded based on a clear competitive process that 
includes a qualified assessment of merit? 

CO2. Does the program have oversight practices that provide sufficient 
knowledge of grantee activities? 

CO3. Does the program collect grantee performance data on an annual 
basis and make it available to the public in a transparent and 
meaningful manner? 

Block/Formula Grant Programs: 

BF1. Does the program have oversight practices that provide sufficient 
knowledge of grantee activities? 

BF2. Does the program collect grantee performance data on an annual 
basis and make it available to the public in a transparent and 
meaningful manner? 

Regulatory-Based Programs: 

RG1. Did the program seek and take into account the views of all 
affected parties (e.g., consumers; large and small businesses; State, 
local and tribal governments; beneficiaries; and the general public) 
when developing significant regulations? 

RG2. Did the program prepare adequate regulatory impact analyses if 
required by Executive Order 12866, regulatory flexibility analyses if 
required by the Regulatory Flexibility Act and SBREFA, and cost-benefit 
analyses if required under the Unfunded Mandates Reform Act; and did 
those analyses comply with OMB guidelines? 

RG3. Does the program systematically review its current regulations to 
ensure consistency among all regulations in accomplishing program 
goals? 

RG4. Are the regulations designed to achieve program goals, to the 
extent practicable, by maximizing the net benefits of its regulatory 
activity? 

Capital Assets and Service Acquisition Programs: 

CA1. Is the program managed by maintaining clearly defined 
deliverables, capability/performance characteristics, and appropriate, 
credible cost and schedule goals? 

Credit Programs: 

CR1. Is the program managed on an ongoing basis to assure credit 
quality remains sound, collections and disbursements are timely, and 
reporting requirements are fulfilled? 

CR2. Do the program's credit models adequately provide reliable, 
consistent, accurate and transparent estimates of costs and the risk to 
the Government? 

Research and Development Programs: 

RD1. For R&D programs other than competitive grants programs, does the 
program allocate funds and use management processes that maintain 
program quality? 

Section IV: Program Results/Accountability (Yes, Large Extent, Small 
Extent, No): 

1. Has the program demonstrated adequate progress in achieving its long-
term performance goals? 

2. Does the program (including program partners) achieve its annual 
performance goals? 

3. Does the program demonstrate improved efficiencies or cost 
effectiveness in achieving program goals each year? 

4. Does the performance of this program compare favorably to other 
programs, including government, private, etc., with similar purpose and 
goals? 

5. Do independent evaluations of sufficient scope and quality indicate 
that the program is effective and achieving results? 

Specific Results Questions by Program Type: 

Regulatory-Based Programs: 

RG1. Were programmatic goals (and benefits) achieved at the least 
incremental societal cost and did the program maximize net benefits? 

Capital Assets and Service Acquisition Programs: 

CA1. Were program goals achieved within budgeted costs and established 
schedules? 

[End of section] 

Appendix III: Sample PART Summary Worksheets: 

[See PDF for image] 

[End of figure] 

[End of section] 

Appendix IV: Comments from the Executive Office of the President: 

EXECUTIVE OFFICE OF THE PRESIDENT: 
OFFICE OF MANAGEMENT AND BUDGET: 
WASHINGTON, D.C. 20503: 

DEPUTY DIRECTOR FOR MANAGEMENT: 

October 17, 2005: 

Ms. Susan J. Irving: 
Director for Federal Budget Analysis: 
Government Accountability Office: 
441 G Street, NW: 
Washington, DC 20548: 

Dear Ms. Irving: 

Thank you for the opportunity to comment on the draft GAO report on 
program evaluation (Performance Budgeting: PART Focuses Attention on 
Program Performance, But More Can Be Done to Engage Congress, GAO-06- 
28). 

We appreciate GAO's continued interest in the Program Assessment Rating 
Tool (PART) and our determination to assess federal programs in a 
consistent fashion through it. As is acknowledged in your conclusion, 
"There is no doubt that creating a closer link between the resources 
expended on programs and the results we expect from those is an 
important goal." [NOTE 1] We fervently believe that the PART has helped 
do just that, and we are grateful for any guidance you can provide that 
will help us achieve even better results. 

In this same spirit, OMB and agencies continue to search for ways to 
make PART assessments more rigorous and consistent. Additionally, we 
are implementing information technology solutions to make application 
of the PART less burdensome and more collaborative. Moreover, we 
reviewed each newly completed PART this year to ensure the answers were 
consistent with PART guidance. These steps and others will make the 
PART more reliable, less of a burden, and hopefully, more focused on 
identifying what steps programs need to take to become more effective. 

In many cases, it takes only administrative actions to address 
weaknesses in program efficiency and effectiveness and the PART process 
has helped do just that. But where Congressional action is required to 
ameliorate a program flaw, GAO correctly points out that PART has been 
less successful. OMB and agencies are grateful for any specific 
suggestions GAO may have to obtain greater Congressional support for 
our initiative to improve the performance of all programs. 

NOTE: 

[1] See, draft report "Conclusions and General Observations," p. 58. 

OMB notes the particular interest that GAO has taken in the 
Administration's standards for measuring performance. Thank you for 
your continued enthusiasm about the PART, as well as for your 
willingness to take our oral and written comments into consideration in 
the final draft. I look forward to working with you to improve the ways 
in which we are making the Federal Government more results-oriented. 

Sincerely, 

Signed by: 

Clay Johnson III: 

[End of section] 

Appendix V: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Susan Irving (202) 512-9142: 

Acknowledgments: 

In addition to the contact named above, Denise Fantone (Assistant 
Director), Thomas Beall, Kylie Gensimore, Joseph Leggero, Patrick 
Mullen, Jacqueline Nowicki, Stephanie Shipman, Katherine Wulff, and 
James Whitcomb made significant contributions to this report. Dianne 
Blank and Amy Rosewarne also provided key assistance. 

[End of section] 

Related GAO Products: 

21st Century Challenges: Performance Budgeting Could Help Promote 
Necessary Reexamination. 
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-05-709T] 
Washington, D.C.: June 14, 2005. 

Management Reform: Assessing the President's Management Agenda. 
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-05-574T] 
Washington, D.C.: April 21, 2005. 

Performance Budgeting: States' Experiences Can Inform Federal Efforts. 
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-05-215] 
Washington, D.C.: February 28, 2005. 

21st Century Challenges: Reexamining the Base of the Federal 
Government. 
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-05- 352T] 
Washington, D.C.: February 16, 2005. 

Long-Term Fiscal Issues: Increasing Transparency and Reexamining the 
Base of the Federal Budget. 
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-05-317T] 
Washington, D.C.: February 8, 2005. 

21st Century Challenges: Reexamining the Base of the Federal 
Government. 
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-05- 325SP] 
Washington, D.C.: February 2005. 

Performance Budgeting: Efforts to Restructure Budgets to Better Align 
Resources with Performance. 
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-05-117SP] 
Washington, D.C.: February 2005. 

Performance Budgeting: OMB's Performance Rating Tool Presents 
Opportunities and Challenges for Evaluating Program Performance. 
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-04-550T] 
Washington, D.C.: March 11, 2004. 

Results-Oriented Government: GPRA Has Established a Solid Foundation 
for Achieving Greater Results. 
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-04-38] 
Washington, D.C.: March 10, 2004. 

Performance Budgeting: OMB's Program Assessment Rating Tool Presents 
Opportunities and Challenges for Budget and Performance Integration. 
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-04-439T] 
Washington, D.C.: February 4, 2004. 

Performance Budgeting: Observations on the Use of OMB's Program 
Assessment Rating Tool for the Fiscal Year 2004 Budget. 
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-04-174] 
Washington, D.C.: January 30, 2004. 

Results-Oriented Government: Using GPRA to Address 21ST Century 
Challenges. 
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-03- 1166T] 
Washington, D.C.: September 18, 2003. 

Performance Budgeting: Current Developments and Future Prospects. 
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-03-595T] 
Washington, D.C.: April 1, 2003. 

Performance Budgeting: Opportunities and Challenges. 
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-02-1106T] 
Washington, D.C.: September 19, 2002. 

Managing for Results: Views on Ensuring the Usefulness of Agency 
Performance Information to Congress. 
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO/GGD-00-35] 
Washington, D.C.: January 26, 2000. 

Performance Budgeting: Fiscal Year 2000 Progress in Linking Plans With 
Budgets. 
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO/AIMD-99- 239R] 
Washington, D.C.: July 30, 1999. 

Performance Budgeting: Initial Experiences Under the Results Act in 
Linking Plans With Budgets. 
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO/AIMD/GGD-99-67] 
Washington, D.C.: April 12, 1999. 

Managing for Results: Measuring Program Results That Are Under Limited 
Federal Control. 
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO/GGD-99-16] 
Washington, D.C.: December 11, 1998. 

The Government Performance and Results Act: 1997 Governmentwide 
Implementation Will Be Uneven. 
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO/GGD-97-109] 
Washington, D.C.: June 2, 1997. 

Managing for Results: Analytic Challenges in Measuring Performance. 
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO/HEHS/GGD-97-138] 
Washington, D.C.: May 30, 1997. 

Performance Budgeting: Past Initiatives Offer Insights for GPRA 
Implementation. 
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO/AIMD-97-46] 
Washington, D.C.: March 27, 1997. 

Executive Guide: Effectively Implementing the Government Performance 
and Results Act. 
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO/GGD-96-118] 
Washington, D.C.: June 1996. 

Managing for Results: Achieving GPRA's Objectives Requires Strong 
Congressional Role. 
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO/T-GGD-96-79] 
Washington, D.C.: March 6, 1996. 

(450364): 

FOOTNOTES 

[1] For more information on reexamination of federal programs, see GAO, 
21ST Century Challenges: Reexamining the Base of the Federal 
Government, GAO-05-325SP (Washington, D.C.: February 2005). 

[2] Pub. L. No. 103-62 (1993). 

[3] In addition to budget and performance integration, the other four 
priorities under the PMA are strategic management of human capital, 
expanded electronic government, improved financial performance, and 
competitive sourcing. 

[4] GAO, Performance Budgeting: Observations on the Use of OMB's 
Program Assessment Rating Tool for the Fiscal Year 2004 Budget, GAO-04- 
174 (Washington, D.C.: Jan. 30, 2004). 

[5] GAO, Program Evaluation: OMB's PART Reviews Increased Agencies' 
Attention to Improving Evidence of Program Results, GAO-06-67 
(Washington, D.C.: Oct. 28, 2005). 

[6] This period covers fiscal years 2004-2006. 

[7] For more information on tax expenditures and their role in 
achieving federal objectives, see GAO, Government Performance and 
Accountability: Tax Expenditures Represent a Substantial Federal 
Commitment and Need to be Reexamined, GAO-05-690 (Washington, D.C.: 
Sept. 23, 2005). 

[8] For a detailed examination of previous federal performance 
budgeting initiatives, see GAO, Performance Budgeting: Past Initiatives 
Offer Insights for GPRA Implementation, GAO/AIMD-97-46 (Washington, 
D.C.: Mar. 27, 1997). 

[9] 5 U.S.C. § 306 (strategic plans); 31 U.S.C. § 1115 (performance 
plans). For more information on federal efforts to implement GPRA, see 
GAO, Results-Oriented Government: GPRA Has Established a Solid 
Foundation for Achieving Greater Results, GAO-04-38 (Washington, D.C.: 
Mar. 10, 2004). 

[10] For more information on using performance management systems as a 
strategic tool to drive internal change and achieve desired results, 
see GAO, Results-Oriented Cultures: Creating a Clear Linkage between 
Individual Performance and Organizational Success, GAO-03-488 
(Washington, D.C.: Mar. 14, 2003) and Managing for Results: Enhancing 
Agency Use of Performance Information for Managerial Decision Making, 
GAO-05-927 (Washington, D.C.: Sept. 9, 2005). 

[11] The administration is considering alternative methods and 
timelines for assessing programs with limited impact and large 
activities where it is difficult to determine an appropriate unit of 
analysis (generally programs within the Department of Defense). 

[12] GAO-06-67. 

[13] Although DOL received funding for this evaluation, it has been 
postponed until after the program is reauthorized because 
reauthorization is expected to result in an increased focus on out-of- 
school youths and a significant change in program activities. 

[14] The nine agencies are the Departments of Energy, Labor, State, and 
Transportation; U.S. Agency for International Development; National 
Aeronautics and Space Administration; National Science Foundation; 
Small Business Administration; and Social Security Administration. 

[15] http://www.whitehouse.gov/omb/part/index.html. 

[16] Follow-on actions made for programs assessed in 2004 would not be 
expected to have status updates. 

[17] Sixty-four percent have been partially implemented and the 
remaining 6 percent have not yet been acted on. 

[18] IDEA (Individuals with Disabilities Education Act) was 
reauthorized by the Individuals with Disabilities Education Act of 
2004, P. L. No. 108-446 (Dec. 3, 2004). 

[19] GAO-04-38. 

[20] GAO-06-67. 

[21] GAO-06-67. 

[22] For further discussion see Peter H. Rossi, Howard E. Freeman, and 
Mark W. Lipsey, Evaluation: A Systematic Approach, 6th ed. (Thousand 
Oaks, California: Sage Publications, 1999). For additional examples of 
alternative evaluation designs, see GAO, Program Evaluation: Strategies 
for Assessing How Information Dissemination Contributes to Agency 
Goals, GAO-02-923 (Washington, D.C.: Sept. 30, 2002). 

[23] The entire evaluation dialogue presentation is at 
http://www.epa.gov/evaluate/part.htm (Aug. 8, 2005). 

[24] OMB, Guidance for Completing the Program Assessment Rating Tool 
(PART) (Washington, D.C.: March 2005) is at 
http://www.whitehouse.gov/omb/part. 

[25] GAO, Tax Policy: Tax Expenditures Represent a Substantial 
Commitment of Federal Support and Need to Be Reexamined, GAO-05-690 
(Washington, D.C.: Sept. 19, 2005). 

[26] ICCED participants included the Community Development Financial 
Institutional Fund, the Economic Development Administration within the 
Department of Commerce, the Department of Housing and Urban 
Development, and the Department of Agriculture. The Appalachian 
Regional Commission and the Neighborhood Reinvestment Corporation also 
participated. The ICCED mission is to "coordinate the federal 
government's economic development, housing and community improvement 
policies, activities and initiatives in distressed communities and 
targeted populations for maximum economic stimulus, efficiency, and 
impact." 

[27] Such coordination is consistent with practices we identified in 
our recent report on agency coordination. See GAO, Results-Oriented 
Government: Practices That Can Help Enhance and Sustain Collaboration 
among Federal Agencies, GAO-06-15 (Washington, D.C.: Oct. 21, 2005). 

[28] According to OMB, these 35 programs include grant and loan 
programs and tax incentives spread across 11 federal agencies. 

[29] While sufficiently reliable as a gauge of general magnitude, the 
sum of the individual revenue loss estimates has important limitations 
in that any interactions between tax expenditures will not be reflected 
in the sum. In addition, tax expenditure revenue loss estimates for 
specific provisions do not take into account potential behavioral 
responses to changes in these provisions on the part of taxpayers, and, 
in turn, no potential behavioral response would be reflected in the sum 
of the estimates. Thus, the revenue loss from all or several tax 
expenditures together might be greater or less than the sum of the 
estimated revenue losses from the individual tax expenditures, and no 
measure of the size or the magnitude of these potential interactions or 
behavioral responses to all or several tax expenditures is available. 

[30] GAO, The Government Performance and Results Act: 1997 
Governmentwide Implementation Will Be Uneven, GAO/GGD-97-109 
(Washington, D.C.: June 2, 1997). 

[31] DOL agency officials described graduates as those who complete a 
vocation and/or attain a Graduate Equivalent Degree or High School 
diploma, stay in the program for 60 or more days, and do not violate 
the zero-tolerance policy. 

[32] For programs not yet assessed by PART, OMB expects the measures 
and targets to meet the standards set in the PART guidance. 

[33] OMB's November 15 deadline for the PAR submission is earlier than 
the statutory deadline. OMB's recent memorandum to agency heads on this 
subject made this accelerated deadline permanent, and notes that the 
purpose of accelerated reporting is to better ensure that timely and 
accurate financial and performance information is made available to 
federal agency managers as soon as possible after the end of the fiscal 
year and throughout the year. 

[34] Accountability and Results in Federal Budgeting, Hearing before 
the Senate Comm. On Homeland Sec. & Governmental Affairs, Subcomm. On 
Fed. Fin. Mgmt, Gov't Info. & Int'l Sec., 109th Cong. (June 14, 2005) 
(Testimony of Clay S. Johnson, III, Deputy Dir., OMB). See also 
Performance-Based Budgeting, Hearing before the House Budget Comm., 
109th Cong. (July 20, 2005) (Testimony of Clay S. Johnson, III, Deputy 
Dir., OMB). 

[35] Depts. of Trans., Treasury, & Housing and Urban Dev., the 
Judiciary, Dist. of Columbia, & Indep. Agencies Appropriations Bill 
2006, H.R. Rep. 109-153, at 137-138 (2005). 

[36] GAO, 21ST Century Challenges: Performance Budgeting Could Help 
Promote Necessary Reexamination, GAO-05-709T (Washington, D.C.: June 
14, 2005). 

[37] An OMB official recently testified that OMB recognizes that its 
Web site must more clearly communicate the PART information and that 
OMB is currently working on improvements to its Web site and the PART 
presentations. 

[38] SBA officials said although on the surface the two programs appear 
to overlap, internal assessments show the differences between the 
programs. However, they said that OMB is constructively challenging SBA 
to articulate these differences using an independent, outside 
evaluation. SBA is seeking resources to pursue these independent 
evaluations. 

[39] Fiscal Year 2005 EPA Budget, Hearing before the House Comm. on 
Science, Subcomm. on Environment, Technology & Standards, 108TH Cong. 
(March 14, 2004). 

[40] Performance-Based Budgeting, Hearing before the House Budget 
Comm., 109TH Cong. (July 20, 2005) (Testimony of Clay S. Johnson, III, 
Deputy Dir., OMB). 

[41] Science, State, Justice, Commerce, & Related Agencies 
Appropriations Bill, Fiscal Year 2006, H.R. Rep. No. 109-118, at 100 
(2005). 

[42] Ibid, p. 125. 

[43] Energy and Water Dev. Appropriations Bill, 2006, H.R. Rep. No. 109-
86, at 112 (2005). 

[44] GAO, Managing for Results: Views on Ensuring the Usefulness of 
Agency Performance Information to Congress, GAO/GGD-00-35 (Washington, 
D.C.: Jan. 26, 2000). 

[45] GAO, Performance Budgeting: States' Experiences Can Inform Federal 
Efforts, GAO-05-215 (Washington, D.C.: Feb. 28, 2005); and Performance 
Budgeting: Efforts to Restructure Budgets to Better Align Resources 
with Performance, GAO-05-117SP (Washington, D.C.: February 2005). 

[46] GAO/GGD-00-35. 

[47] GAO-05-709T. 

[48] This period covers fiscal budget years 2004-2006. 

[49] See GAO-04-174. 

[50] As a result, the percentage distribution of recommendation 
categories for 2002 differs somewhat from our prior report. See GAO-04- 
174. 

[51] While this combined data set includes all programs covered in the 
2004 PART, it does not include those programs subject to the PART in 
2002 or 2003 and subsequently merged or incorporated into other 
programs or dropped. These programs were excluded to help ensure that 
our analyses did not include programs that, as defined in either 2002 
or 2003, no longer exist. 

GAO's Mission: 

The Government Accountability Office, the investigative arm of 
Congress, exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability 
of the federal government for the American people. GAO examines the use 
of public funds; evaluates federal programs and policies; and provides 
analyses, recommendations, and other assistance to help Congress make 
informed oversight, policy, and funding decisions. GAO's commitment to 
good government is reflected in its core values of accountability, 
integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains 
abstracts and full-text files of current reports and testimony and an 
expanding archive of older products. The Web site features a search 
engine to help you locate documents using key words and phrases. You 
can print these documents in their entirety, including charts and other 
graphics. 

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as "Today's Reports," on its 
Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order 
GAO Products" heading. 

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to: 

U.S. Government Accountability Office 

441 G Street NW, Room LM 

Washington, D.C. 20548: 

To order by Phone: 

Voice: (202) 512-6000: 

TDD: (202) 512-2537: 

Fax: (202) 512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm 

E-mail: fraudnet@gao.gov 

Automated answering system: (800) 424-5454 or (202) 512-7470: 

Public Affairs: 

Jeff Nelligan, managing director, 

NelliganJ@gao.gov 

(202) 512-4800 

U.S. Government Accountability Office, 

441 G Street NW, Room 7149 

Washington, D.C. 20548: