HUD and Treasury Programs
More Information on Leverage Measures' Accuracy and Linkage to Program Goals Is Needed in Assessing Performance
GAO-08-136, Feb 19, 2008
Additional Materials:
Contact:
This is the second of two reports on the leveraging of federal funds in housing and community and economic development programs. Leveraging involves using a source of funds to attract other funds or combining multiple sources of funds. This report examines (1) the leverage measures and the transparency of the data and methods used to calculate them, and (2) the relevance of such measures in assessing performance that the Department of Housing and Urban Development (HUD) and the Department of the Treasury (Treasury) reported for six selected programs. To complete this work, GAO reviewed agency policies and reports, interviewed officials, and analyzed agency data.
The leverage measures (such as ratios) HUD and Treasury reported for the selected programs in performance, budget, and other documents lacked transparency because the agencies generally did not disclose the limitations of the data or the methods used to calculate them. Based on its review of available leveraging data and interviews with HUD and Treasury officials, GAO found that the leverage measures the agencies reported for the selected programs were based on incomplete data and thus did not capture the actual extent of leveraging in the programs. GAO also found that while the agencies generally reported measures that described the ratio of all other funds (federal, state, local, and private funds) to program funds, alternative measures that described the total federal investment or total private investment in a program provided considerably different results--also potentially of value to decision makers--about the extent of leveraging in a program. GAO regularly has reported that clearly communicating data limitations and their potential impact may foster appropriate use of data; however, no agency-specific or governmentwide guidance directs what agencies should disclose about the leverage measures they report for the selected programs. Consequently, absent specific information on how these measures were calculated and their limitations, decision makers would not have sufficient information to understand their meaning and determine how they could and should be used in performance assessment, budgeting, and other contexts. Leverage measures can provide basic information about the programs GAO reviewed; however, their relevance in assessing the performance of these programs varies considerably. For all of the programs GAO reviewed, leverage measures can describe inputs, or the resources used to support program activities, and may be useful for conveying basic financial information. To the extent that leveraging is a goal or expected activity of a program (as in the three Treasury programs), leverage measures generally can describe program outputs, or the products or services delivered (such as total leveraged funds), and may be used along with other performance indicators to assess the efficiency and effectiveness of a program in meeting its goals. In cases where leveraging is not clearly and appropriately linked to program goals and activities (as in the three HUD programs), use of such measures to describe program outputs could be misleading and result in adverse consequences. Although leveraging had limited relevance to the goals and activities of the selected HUD programs, GAO found that the Office of Management and Budget (OMB) and the agency often cited leverage measures for the programs in performance- and budget-related reviews and documents. Their continued use of leverage measures in these contexts could unnecessarily encourage HUD to place more importance on leveraging than meeting the stated goals of the selected programs.
Status Legend:
Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
- In Process
- Open
- Closed - implemented
- Closed - not implemented
Recommendations for Executive Action
Recommendation: To ensure that leverage measures provide accurate, useful, and relevant information to Congress and others, the Secretaries of HUD and the Treasury should consider disclosing the following when they publish such measures for the programs included in our review: Presentation of leverage measures should be accompanied by information about the completeness and accuracy of the data and the method(s) used to calculate the measures (for example, with leverage ratios, information on what sources of funds were compared, such as private funds to public funds or nonfederal funds to federal funds).
Agency Affected: Department of the Treasury
Status: Open
Comments: A May 12, 2008, letter from Treasury to Represantative Waxman, Chairman, Committee on Oversight and Government Reform notes that in response to the recommendations in our report, "going forward, the CDFI Fund will ensure that, in all publications in which it provides discussion of leverage ratios, such discussions include calculation methodologires and limitations to the underlying data." No recent reports serve as evidence that the recommendation has been implemented, therefore it remains open.
Recommendation: To ensure that leverage measures provide accurate, useful, and relevant information to Congress and others, the Secretaries of HUD and the Treasury should consider disclosing the following when they publish such measures for the programs included in our review: Presentation of leverage measures should be accompanied by information about the completeness and accuracy of the data and the method(s) used to calculate the measures (for example, with leverage ratios, information on what sources of funds were compared, such as private funds to public funds or nonfederal funds to federal funds).
Agency Affected: Executive Office of the President: Office of Management and Budget
Status: Open
Comments: HUD's 60-day letter dated April 17, 2008, stated that when reporting leveraging measures for the CDBG and HOME programs, HUD would provide information on the completeness and accuracy of the data used to calculate the measures and information on how the measures were calculated. HUD's letter did not address how the agency planned to address the recommendations for the HOPE VI program. Since the release of this GAO report, HUD has not issued any reports related to leveraging in the HOPE VI, CDBG, and HOME programs-- in September 2009, the team reviewed HUD's website for relevant report and found none and contacted HUD for an update on the status of the recommendation, but did not receive a response. Therefore, this recommendation remains open.
Recommendation: To ensure that leverage measures provide accurate, useful, and relevant information to Congress and others, the Secretaries of HUD and the Treasury should consider disclosing the following when they publish such measures for the programs included in our review: Presentation of leverage measures should be accompanied by a discussion of the relevance of the measure in assessing the program's performance. For example, the agencies should discuss the extent to which leverage measures are linked to program goals and core activities.
Agency Affected: Department of Housing and Urban Development
Status: Open
Comments: HUD's 60-day letter dated April 17, 2008, stated that the agency would issue guidance by Decemeber 31, 2008, encouraging CDBG and HOME grantees to improve their reporting on leveraging at the local level. HUD's letter did not address the HOPE VI program. In September 2009, the team contacted HUD for an update on the status of the recommendation, but did not receive a response. The team also also searched HUD's website to see if the agency had issued an reports related to leveraging in the CDBG, HOME, and HOPE VI programs, but found none. Therefore, this recommendation remains open.
Recommendation: To ensure that leverage measures provide accurate, useful, and relevant information to Congress and others, the Secretaries of HUD and the Treasury should consider disclosing the following when they publish such measures for the programs included in our review: Presentation of leverage measures should be accompanied by a discussion of the relevance of the measure in assessing the program's performance. For example, the agencies should discuss the extent to which leverage measures are linked to program goals and core activities.
Agency Affected: Department of Housing and Urban Development
Status: Open
Comments: In a May 12, 2008, letter to Represenative Waxman, Chairman, Committee on Oversight and Government Reform, Treasury stated that it agreed "that leverage measures are directly relevant to [CDFI prpgram and NMTC program] outcomes and therefore will make every effort to communicate the extent to which levergae measures are linked to program goals and core activities." As of September 2009, Treasury had not issued any performance-related reports for the CDFI or the NMTC programs.
Recommendation: The Director of OMB should provide guidance to help agencies determine how to calculate, describe, and use leverage measures in a manner consistent with the programs' design.
Agency Affected: Department of Housing and Urban Development
Status: Open
Comments: OMB's 60-day letter to GAO states "To improve program performance measures, OMB is currently working with agencies to develop additional verification and validation guidance, including that related to leveraging measures. This guidance is scheduled to be released in A-11 by July of this year. OMB and agencies have also undertaken a review of agency performance measures agencies and programs have arrived at over the last six years. The Summary of [A-11] Changes section 230.2 states "Requires agencies to have in place or begin to develop verification and validation techniques that will ensure the completeness and reliability of all performance measurement data contained in annual performance plans and reports and other documents that include performance measurement data." (p.2) The change to section 230.2 of A-11 states "A Diagnostic Analyses of 2008 Program Performance Results - For PARTed programs, agencies/programs should perform a diagnostic analysis that identifies the root causes of their 2008 performance results as indicated by their PART measures as reported in PARTWeb. The analysis should address and is informed (as appropriate) by the results of relevant, credible evaluation studies, investigations, and audits. Subsequent to the diagnostic analysis, the agency/program drafts improvements actions aimed to improve or ensure future program performance." (p.2) OMB's response does not address the intent of our recommendation. The team has attempted several times to contact OMB to discuss the actions OMB outlined in its 60-day letter would address this recommendation. To date, OMB has not responded to these inquiries. Therefore, this recommendation remains open.
Recommendation: The Director of OMB should re-evaluate the use of leverage measures and disclose their relevance to program goals and activities in future Program Assessment Rating Tool or other performance reviews of the selected programs.
Agency Affected: Department of the Treasury
Status: Open
Comments: As of September, 2009, OMB had not conducted PART reviews (or any other performance reviews) of the programs included in our report (CDFI, NMTC, CDBG, HOME). As a result, we were unable to assess whether OMB implemented this recommendation. Therefore, the recommendation remains open.