Employment and Training Administration:
More Actions Needed to Improve Transparency and Accountability of Its Research Program
GAO-11-285, Mar 15, 2011
To help guide the nation's workforce development system, the Department of Labor's (Labor) Employment and Training Administration (ETA) conducts research in areas related to job training and employment. Building upon our earlier work, GAO examined the following: (1) To what extent do ETA's research priorities reflect key national employment and training issues and how useful were the studies funded under them? (2) What steps has ETA taken to improve its research program? (3) How has ETA improved the availability of its research since our last review in January 2010? To answer these questions, GAO reviewed ETA's research reports disseminated between January 2008 and March 2010 costing $1 million or more, as well as ongoing studies costing $2 million or more. GAO also convened a virtual expert panel, interviewed Labor and ETA officials, and reviewed relevant documents.
ETA's 2007 to 2012 research plan generally addressed key employment and training issues, but some studies were limited in their usefulness. Most experts on our panel reported that the areas in ETA's plan reflected key national employment and training issues at least to a moderate extent. ETA invested most of its research and evaluation resources in the areas of Unemployment Insurance and increasing labor market participation of underutilized groups. Of the $96 million that ETA invested in the 58 research reports we reviewed, more than half--or about $56 million--funded studies in these two areas. The methodological approaches and statistical procedures researchers used in the studies we reviewed were generally consistent with the questions posed, but the studies were not always useful for informing policy and practice. For example, in one study, shortcomings in the data collection phase limited the strength of the findings. Experts suggested that ETA include more varied and rigorous methodologies in its future research projects. They also reported that future research should address additional areas, including a focus on employment and training approaches that work and for whom. Labor and ETA have taken steps to improve the way research is conducted, but additional actions are needed. In acknowledging the need for more rigorous evaluations to inform its policies, Labor recently established the Chief Evaluation Office to oversee departmentwide research and evaluation efforts. In addition, ETA made changes to some of its research practices. For example, ETA has begun involving outside experts in developing its research plan. Despite these improvements, ETA's process lacks critical elements needed to ensure that current improvements become part of its routine practices. For example, ETA's process lacks a formal provision to consult with the newly established Chief Evaluation Officer at important points in the research process. Moreover, ETA's current research practices fall short of ensuring research transparency and accountability--essential elements of a sound research program. For example, its research and evaluation center lacks safeguards to protect it from undue outside influence. ETA has recently begun efforts to increase the rigor of its research designs, but has faced design and implementation challenges. For example, some of ETA's ongoing research studies face challenges in recruiting large enough sample sizes to meet the studies' objectives. ETA has improved the availability of its research findings, but it lacks a plan for assessing the usability of its Web-based search page--the primary tool for making ETA's research publicly available. ETA recently improved the timeliness with which it disseminates its research reports, decreasing the average number of days to release its reports to the public from 804 days in 2008 to 76 days in 2009. ETA has taken steps to update its online, Web-based search page. However, the agency has not developed a formal plan for assessing the overall effectiveness of its Web-based search page, including user satisfaction. In addition to its research database, ETA uses a variety of other methods to disseminate its research, including providing its research reports at conferences and internal briefings. Experts suggested that ETA consider other effective dissemination methods, such as publishing a one-page summary of research findings. GAO recommends that ETA formally include the Chief Evaluation Officer in its research process, create a mechanism to enhance the transparency and accountability of its research program, and develop a plan to ensure that research reports are accessible through its Web-based search page. Labor agreed with our recommendations and noted its ongoing efforts to improve its research program. While these efforts are important, GAO stresses the need for additional actions to fully address the recommendations.
- Closed - implemented
- Closed - not implemented
Recommendations for Executive Action
Recommendation: To improve ETA's research program, the Secretary of Labor should require ETA to formally incorporate into its research process the routine involvement of the Chief Evaluation Officer at key milestones, including at the development of ETA's annual research agenda and spending priorities, as well as at the early stages of developing specific research projects.
Agency Affected: Department of Labor
Comments: The Department of Labor concurred with our recommendation and stated that ETA is formally involving the Chief Evaluation Officer (CEO) in various aspects of its research process, including at critical key research milestones such as the agenda-setting phase and the conceptualization and design of research studies. ETA plans to update its formal research process documentation to reflect this collaboration accordingly. In FY 2012, ETA reported that it has incorporated the CEO, and that this collaboration will be reflected in its formal research process documentation. Labor did not provide an update in FY13. When ETA's formal research process has been updated, GAO will be able to consider this recommendation implemented.
Recommendation: To improve ETA's research program, the Secretary of Labor should require ETA to develop a mechanism to enhance the transparency and accountability of ETA's research program. For example, such a mechanism might include involving advisory bodies or other entities outside ETA, in efforts to develop ETA's research policies and processes.
Agency Affected: Department of Labor
Comments: The Department of Labor concurred with our recommendation and ETA has begun to engage outside experts, such as research and policy experts, stakeholders, and other Federal agencies, in developing research priorities. ETA plans to formalize the involvement of these outside experts in its formal research process. ETA also plans to continue to reexamine components of its research process to ensure that current practices are accurately reflected. For example, ETA plans to include peer reviews for all major evaluations at key points, such as the design and analysis phases. In FY 2012, ETA reported taking a number of steps, including (1) using outside experts; (2) forming advisory and peer review groups; (3) working with Coalition for Evidence Based Policy; and (4) consulting Labor's Chief Evaluation Officer. Furthermore, ETA stated that these efforts would be reflected in its formal research process. Labor did not provide an update in FY 2013. When ETA's formal research process has been updated, GAO will consider this recommendation implemented.
Recommendation: To improve ETA's research program, the Secretary of Labor should require ETA to develop a formal plan for ensuring that ETA's research products are easily accessible to stakeholders and to the general public through its searchable database. Such a plan could involve requiring Labor to assess the overall effectiveness of its Web-based search page, including user satisfaction with search features.
Agency Affected: Department of Labor
Status: Closed - Implemented
Comments: ETA began making improvements to its Web-based database for disseminating research and evaluation findings. Improvements include an interface that produces more accurate keyword search returns and a link to a customer satisfaction survey on all departmental Web pages. ETA will continue to collect feedback on its database and outline further enhancements in an improvement plan. ETA will also update its formal research process to clarify the steps necessary to disseminate research reports to the public. In FY 2012, Labor provided documentation of its performance standards for dissemination activities, which included developing one-pagers, report summaries, and other materials to promote the release of reports and/or share key findings.