Recommendation: To ensure that the performance measures used to evaluate the TAA for Firms program demonstrate program results and to help ensure that EDA can comprehensively evaluate the effectiveness of the program, the Secretary of Commerce should broaden the program's evaluation approach, for instance, by developing additional quantifiable outcome-oriented performance goals and measures for key program areas and conducting further analysis of the data to isolate the impact of the TAA for Firms program from other influences, such as economic trends.
Agency: Department of Commerce
Status: Open
Priority recommendation
Comments: Commerce concurred with our findings and recommendations. As of August 2020, EDA is taking actions to implement our recommendation. In 2012, EDA commissioned a study to inform the development of performance metrics and evaluation methods for the TAA for Firms program. Issued in October 2014, the study provided recommendations to EDA on potential performance metrics, tested the metrics by conducting a statistical analysis of TAA for Firms clients against a control group as well as a survey, and suggested data sources for conducting future evaluations of the TAA for Firms program. According to EDA officials, the study would help provide a foundation for more robust longitudinal performance measurement and enhanced policy analysis, thereby enabling EDA to more comprehensively evaluate the program. These officials noted that, in April 2017, the authors of the study finalized an operational toolkit containing recommended metrics and protocols for data collection and impact evaluation methods. In June 2017, EDA began to pilot test these recommended metrics and evaluation methods to improve program evaluation. EDA formed a working group in December 2017 to further refine the new evaluation system and metrics based on the results of the pilot tests. In May 2018, EDA's working group concluded the refinement of the metrics. In August 2018, EDA continued to collect data on these metrics from its grantees using a pilot survey and refined the metrics as EDA received additional responses and feedback about the pilot survey from grantees. As of August 2020, EDA had developed new data collection instruments that will gather data and information on the TAA for Firms program's outputs and outcomes. EDA intends to use the information collected from the new instruments to conduct program evaluations to better understand how program performance compares with performance goals and the impact of the TAA for Firms program, according to EDA officials. EDA is in the process of developing software to administer the instruments and an internal working group is guiding the implementation of the new data collection system. EDA expects to implement the new system in fiscal year 2020, according to EDA officials. We will continue to monitor Commerce's efforts to implement our recommendation.
Recommendation: To improve the data available to manage and evaluate the TAA for Firms program, the Secretary of Commerce should develop a data system to consistently collect, maintain, and analyze sufficiently reliable and up-to-date data on program operations and participant firms.
Agency: Department of Commerce
Status: Open
Priority recommendation
Comments: Commerce concurred with our findings and recommendations. As of August 2020, EDA is taking actions to implement our recommendation. According to EDA officials, in fiscal year 2017, EDA had developed an improvement plan for its agency-wide data system for collecting data on program operations. As of August 2020, EDA is working with a contractor to implement a new data system to assist in administering and managing the TAA for Firms program. For example, EDA officials expect the new data system to allow TAA Centers to submit participant firms' petitions for certification and adjustment proposals to EDA electronically for review and approval. In addition, these officials noted that the new data system will facilitate EDA's analysis of program performance. EDA anticipates that the new data system for the TAA for Firms program will be operational in fiscal year 2021. We will continue to monitor Commerce's efforts to implement our recommendation.
