Moving to Work Demonstration:

Opportunities Exist to Improve Information and Monitoring

GAO-12-490: Published: Apr 19, 2012. Publicly Released: May 21, 2012.

Additional Materials:

Contact:

Mathew J. Scire
(202) 512-8678
sciremj@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

What GAO Found

Public housing agencies (PHA) that participate in the Moving to Work (MTW) program report annually on the performance of their activities, which include efforts to reduce administrative costs and encourage residents to work. But this performance information varies, and the Department of Housing and Urban Development’s (HUD) guidance does not specify that it be quantifiable and outcome oriented. Further, HUD has not identified the performance data that would be needed to assess the results of similar MTW activities or the program as a whole and has not established performance indicators for the program. The shortage of such analyses and indicators has hindered comprehensive evaluation efforts, although such evaluations are key to determining the success of any demonstration program. Further, while HUD has identified some lessons learned from the program, it has no systematic process to identify them and thus has relied primarily on ad hoc information. The absence of a systematic process for identifying lessons learned limits HUD’s ability to promote useful practices that could be more broadly implemented to address the purposes of the program.

HUD generally follows its MTW monitoring policies and procedures, but they could be strengthened. HUD staff review and approve each MTW agency’s annual plan to ensure that planned activities are linked to program purposes and visit each MTW agency annually to provide technical assistance. But HUD has not taken key monitoring steps set out in internal control standards, such as issuing guidance that defines program terms or assessing compliance with all of the requirements. Without clarifying key terms and establishing a process for assessing compliance with statutory requirements, HUD lacks assurance that agencies are actually complying with the statute. Additionally, HUD has not done an annual assessment of program risks despite its own requirement to do so and has not developed risk-based monitoring procedures. Without taking these steps, HUD lacks assurance that it has identified all risks to the program. Finally, HUD does not have policies or procedures in place to verify the accuracy of key information that agencies self-report. For example, HUD staff do not verify self-reported performance information during their reviews of annual reports or annual site visits. Without verifying at least some information, HUD cannot be sure that self-reported information is accurate.

Expanding the MTW program may offer benefits but also raises questions. According to HUD, affordable housing advocates, and MTW agencies, expanding MTW to additional PHAs would allow agencies to develop more activities tailored to local conditions and result in more lessons learned. However, data limitations and monitoring weaknesses raise questions about expansion. HUD recently reported that expansion should occur only if newly admitted PHAs structured their programs to permit high-quality evaluations and ensure that lessons learned could be generalized. Until more complete information on the program’s effectiveness and the extent to which agencies are adhering to program requirements is available, it will be difficult for Congress to know whether an expanded MTW would benefit additional agencies and the residents they serve. Some researchers and MTW agencies suggested alternatives to expansion, including implementing a program that was more limited in scope.

Why GAO Did This Study

HUD’s MTW demonstration program gives participating PHAs the flexibility to create innovative housing strategies through their fiscal year 2018. MTW agencies must create activities linked to three statutory purposes—reducing costs, providing incentives for self-sufficiency, and increasing housing choices—and meet five statutory requirements. Congress is considering expanding MTW and has asked GAO to examine what is known about (1) the program’s success in addressing the three purposes, (2) HUD’s monitoring efforts, and (3) the potential benefits of and concerns about expansion. GAO analyzed the most current annual reports for 30 MTW agencies; compared HUD’s monitoring efforts with internal control standards; and interviewed agency officials, researchers, and industry officials.

What GAO Recommends

GAO makes eight recommendations to HUD: that HUD improve its guidance on reporting performance information, develop a plan for identifying and analyzing standard performance data, establish performance indicators, systematically identify lessons learned, clarify key terms, implement a process for assessing compliance with statutory requirements, do annual assessments of program risks, and verify the accuracy of self-reported data. HUD generally or in part agreed with seven of them. HUD disagreed with our recommendation that it create overall performance indicators. GAO believes, however, that they are critical to demonstrating program results and thus maintains its recommendation.

For more information, contact Mathew J. Scirè at (202) 512-8678 or sciremj@gao.gov.

Recommendations for Executive Action

  1. Status: Closed - Implemented

    Comments: HUD revised the Standard MTW Agreement and the current Reporting Requirements for MTW Agencies (HUD-50900) in May 2013, requiring MTW agencies to report quantifiable and outcome-oriented information on MTW activities. Additionally, HUD issued webcast training on the revised HUD-50900. The revised HUD-50900 includes standard tables for MTW agencies to provide operating information, financial information, and metrics for all MTW activities, and directs MTW agencies to provide quantifiable and outcome-oriented data. Taking these steps should help HUD collect data that reflect outcomes of MTW activities.

    Recommendation: To improve what is known about the effectiveness of the MTW program, the Secretary of HUD should improve HUD's guidance to MTW agencies on providing performance information in their annual reports by requiring that such information be quantifiable and outcome-oriented to the extent possible.

    Agency Affected: Department of Housing and Urban Development

  2. Status: Open

    Comments: As of March 2015, HUD was still developing documentation describing how it will systematically identify lessons learned from the MTW demonstration program. According to HUD staff, the agency plans to use data collected through an updated HUD Form 50900 to identify lessons learned. HUD plans to provide documentation to GAO on these efforts by April 30, 2015.

    Recommendation: To enhance the ability to identify MTW practices that could be applied more broadly, the Secretary of HUD should create a process to systematically identify lessons learned.

    Agency Affected: Department of Housing and Urban Development

  3. Status: Closed - Implemented

    Comments: HUD adopted a revised MTW Standard Agreement (HUD-50900) in May 2013, which provides detailed clarifications on the meaning of the three statutory purposes of the program and relevant metrics. Additionally, the revised HUD-50900 includes standardized tables for agencies to report data related to the requirement to serve a comparable mix of families and provides definitions related to other requirements for categories of households served. By taking these steps, HUD can more effectively determine whether MTW agencies are addressing the program's purposes and meeting requirements and improve its oversight of the MTW program

    Recommendation: To improve HUD's oversight of the MTW program, the Secretary of HUD should issue guidance that clarifies key program terms, such as the three statutory purposes of the program and the five statutory requirements that MTW agencies must meet.

    Agency Affected: Department of Housing and Urban Development

  4. Status: Closed - Implemented

    Comments: HUD developed standard metrics, or measures, that correlate with each of the three statutory purposes, or objectives, of the MTW program. For example, for the statutory objective related to cost-effectiveness, HUD established five measures, including whether there was a decrease in the total cost of a task (in dollars), a decrease in the total time to complete a task (in hours), and an increase in the amount of funds leveraged (in dollars). For the statutory objective related to self-sufficiency, HUD created eight measures; and for the statutory objective related to housing choice, HUD created seven measures. For each measure, HUD will be able to calculate quantitative results and assess whether the result for the MTW program as a whole has been positive, neutral, or negative. Further, HUD will be able to total the results for each individual statutory objective, so that it can assess whether the MTW program as a whole is having a positive, neutral, or negative outcome for each objective.

    Recommendation: To improve what is known about the effectiveness of the MTW program, the Secretary of HUD should develop and implement a plan for quantitatively assessing the effectiveness of similar activities and the program as a whole including the identification of standard performance data needed.

    Agency Affected: Department of Housing and Urban Development

  5. Status: Closed - Implemented

    Comments: HUD developed a five-step plan for quantitatively assessing the effectiveness of similar activities and the program as a whole. According to this plan, HUD will assign standard metrics for all of the implemented and proposed activities of each MTW agency, and HUD will ensure that agencies conducting similar activities report annually on the same set of metrics. HUD's plan states that the agency will take the data and enter it into a centralized location, which will allow for aggregation across the MTW program. HUD has taken steps to implement this plan. First, HUD developed standard metrics that correlate with the statutory objectives of the MTW program. Second, HUD issued guidance to MTW agencies on how to assign the published standard metrics to proposed and implemented activities. Third, HUD has begun to compile the results reported by MTW agencies into a centralized location, enabling the aggregation of each metric across the program.

    Recommendation: To improve what is known about the effectiveness of the MTW program, the Secretary of HUD should establish performance indicators for the MTW program as a whole.

    Agency Affected: Department of Housing and Urban Development

  6. Status: Closed - Implemented

    Comments: Beginning with data for 2013, HUD has begun to systematically track MTW agencies compliance with each of the program's five statutory requirements. For the three requirements that agencies could previously self-certify, HUD has assessed compliance using available data. For example, to assess compliance with the requirement that at least 75 percent of households are very low income, HUD has used income-data for each participating MTW agency. Also, to assess compliance with the requirement for MTW agencies to maintain a comparable mix of families, HUD has begun to use available data to compare the mix of families that MTW agency originally assisted (by family size) to the mix assisted in 2013. According to HUD staff, MTW agencies' compliance with each statutory requirement will be updated throughout each year. As a result of continuing to implement these actions, HUD will be able to determine the extent to which MTW agencies are meeting the program's statutory requirements.

    Recommendation: To improve HUD's oversight of the MTW program, the Secretary of HUD should develop and implement a systematic process for assessing compliance with statutory requirements.

    Agency Affected: Department of Housing and Urban Development

  7. Status: Closed - Implemented

    Comments: HUD's Real Estate Assessment Center's (REAC) Quality Assurance Subsystem (QASS) staff performed an initial risk assessment of the MTW program. QASS staff completed this initial assessment in accordance with the HUD CFO Handbook 1840.1, and issued a report in February 2014. According to HUD, for subsequent years, the agency will assess the risks associated with the MTW program using a "risk register." HUD provided a copy of its 2015 risk register, dated December 2014. This register identified and prioritized risks to the MTW program. Additionally, in March 2015, HUD provided GAO with a document noting the factors that HUD considers when determining whether to conduct a site visit every year, or every other year. HUD considers agency size, whether the agency is new to the MTW program, whether the local HUD field office has raised issues, and the complexity of the MTW program being implemented, among other things. While HUD considers these factors when planning site visits, the agency's ability to conduct site visits is dependent on funding and staffing resources. As a result of conducting annual risk assessments of the MTW program as a whole and conducting risk-based site visits, HUD is taking steps to better assure that it is using its limited monitoring resources efficiently.

    Recommendation: To improve HUD's oversight of the MTW program, the Secretary of HUD should conduct an annual risk assessment for the MTW program and implement risk-based monitoring policies and procedures such as those currently being considered for site visits.

    Agency Affected: Department of Housing and Urban Development

  8. Status: Closed - Implemented

    Comments: HUD has taken steps to help ensure that the information that individual MTW agencies report about their MTW activities is assessed against program compliance requirements. In March 2014, HUD published a compliance supplement for financial auditors to use as a part of their annual audits of MTW agencies. This supplement, which became effective in May 2014, provides auditors with guidance on allowable MTW activities, costs, and performance reporting, among other things. In 2015, HUD provided procedures explaining how it will use the results of the annual audits. According to these procedures, auditors are to include their assessment of MTW agencies' compliance with the supplement in their completed audits, and HUD field offices are to review the audits of MTW agencies to determine appropriate action relative to any findings and follow up with MTW agencies to ensure findings are closed. Additionally, according to HUD, if there are audit findings related to the MTW program itself, the HUD field office will inform HUD's MTW office, and these offices will work together to agree upon the recommendations that will be made to the MTW agency to resolve the findings. By implementing these activities, HUD can be better assured of the reliability of the performance information that MTW agencies report.

    Recommendation: To improve HUD's oversight of the MTW program, the Secretary of HUD should implement control activities designed to verify the accuracy of a sample of the performance information that MTW agencies self-report.

    Agency Affected: Department of Housing and Urban Development

 

Explore the full database of GAO's Open Recommendations »

May 19, 2015

Apr 16, 2015

Apr 9, 2015

Mar 16, 2015

Feb 4, 2015

Oct 7, 2014

Aug 1, 2014

Mar 31, 2014

Mar 27, 2014

Mar 18, 2014

Looking for more? Browse all our products here