This is the accessible text file for GAO report number GAO-14-8 entitled 'Education Research: Further Improvements Needed to Ensure Relevance and Assess Dissemination Efforts' which was released on December 5, 2013. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. United States Government Accountability Office: GAO: Report to the Committee on Education and the Workforce, House of Representatives: December 2013: Education Research: Further Improvements Needed to Ensure Relevance and Assess Dissemination Efforts: GAO-14-8: GAO Highlights: Highlights of GAO-14-8, a report to the Committee on Education and the Workforce, House of Representatives. Why GAO Did This Study: The federal government has a longstanding role in conducting education research and collecting education data, and IES has a broad mission to provide this information to a wide variety of stakeholders. With a fiscal year 2013 budget of just under $600 million, IES is Education’s primary research and evaluation arm. GAO was asked to review IES’s performance. This report examines: (1) the extent to which IES has demonstrated its ability to support high-quality research and fulfill its mission, (2) the extent to which selected Education research and technical assistance groups disseminate relevant products to the education field, and (3) how IES coordinates its activities with other relevant federal research agencies and within Education. GAO reviewed relevant federal laws and regulations, agency documents and data, interviewed agency officials and stakeholders, and analyzed information from selected research and technical assistance groups. GAO also compared IES’s practices to federal internal control standards and leading practices for performance management and collaboration. What GAO Found: The Department of Education’s (Education) Institute of Education Sciences (IES) supports high-quality research, but lacks certain key procedures needed to fulfill other aspects of its mission. According to stakeholders, IES has substantially improved the quality of education research since its inception over a decade ago. However, GAO identified concerns with IES’s ability to produce timely and relevant research. For example, IES’s efforts are slow to respond to stakeholders’ needs, in part, because the time IES’s products have spent in peer review has substantially increased in recent years—-from an average of 117 days in fiscal year 2011 to 150 days in fiscal year 2013-—and IES does not monitor some aspects of these timeframes. In addition, IES does not have a structured process for incorporating stakeholder input into its research agenda, which previous GAO work has shown to be key to sound federal research programs. Lastly, IES’s performance measures do not fully reflect its current programs, which is not consistent with leading practices GAO has identified for performance management. For example, IES does not publicly report on the overall performance of the Regional Educational Laboratories (REL) program, which constitutes one of the agency’s largest investments. IES officials said that they have begun to develop new performance measures for all of their programs, but these will not be in place until fiscal year 2015. Although Education’s research and technical assistance groups have taken steps to disseminate relevant research to the education field, IES does not always assess these efforts. Some stakeholders raised concerns about the dissemination of relevant products from the RELs and Research and Development Centers (R & D Center). For example, they told GAO that these groups do not always adapt their products for use by both policymaker and practitioner audiences. Further, IES has not fully assessed REL and R & D Center dissemination efforts. As a result, IES does not know if these efforts are effective in meeting their mandated goal of providing usable research and information to stakeholders. GAO’s prior work on information dissemination suggests that further assessment could help inform IES’s oversight of the RELs and R & D Centers to improve these groups’ dissemination to key audiences. IES coordinates with other relevant federal research agencies to increase the use of research evidence in federal decision-making. For example, IES and the National Science Foundation recently released guidelines to help improve the quality of evidence resulting from federally-funded education research, which stakeholders said will benefit the education field. Within the department, IES plans evaluations of Education programs through coordination with various other offices. However, Education lacks statutory authority to combine evaluation funds for Elementary and Secondary Education Act (ESEA) programs, which officials said limits the department’s ability to conduct high-quality evaluations of programs it considers most important. GAO’s prior work has shown that many Education programs, especially smaller programs, have not been evaluated, limiting the ability of Congress to make informed policy decisions. For example, in 2009, GAO reported that 11 of Education’s teacher quality programs had not been evaluated in more than 7 years. Officials said that the ability to combine evaluation funds would allow the department to conduct needed evaluations of ESEA programs. What GAO Recommends: GAO recommends that Congress consider granting Education authority to combine funds authorized for evaluation of ESEA programs, and IES use available data to manage its peer review process, develop a structured process to gather stakeholder input, develop performance measures that reflect all key agency activities, and assess REL and R & D Center dissemination strategies. Education agreed with our recommendations. View [hyperlink, http://www.gao.gov/products/GAO-14-8]. For more information, contact George Scott at (202) 512-7215 or scottg@gao.gov. [End of section] Contents: Letter: Background: IES Supports High-Quality Research, but Lacks Key Processes and Performance Measures in Some Areas: Research and Technical Assistance Groups Take Steps to Disseminate Relevant Research, but IES Has Not Fully Assessed These Efforts: IES Coordinates with Other Federal Agencies, but Fragmented Evaluation Funding Poses Challenges for Education's Evaluation Efforts: Conclusions: Matter for Congressional Consideration: Recommendations for Executive Action: Agency Comments and Our Evaluation: Appendix I: Objectives, Scope, and Methodology: Appendix II: Comments from the Department of Education: Appendix III: GAO Contact and Staff Acknowledgments: Related GAO Products: Table: Table 1: The Institute of Education Science's Centers and Their Key Activities: Figure: Figure 1: Selected Education Research and Technical Assistance Groups: Abbreviations: AERA: American Educational Research Association: Education: Department of Education: ESEA: Elementary and Secondary Education Act: ESRA: Education Sciences Reform Act of 2002: IES: Institute of Education Sciences: NBES: National Board for Education Sciences: NICHD: Eunice Kennedy Shriver National Institute of Child Health and Human Development: NSF: National Science Foundation: NCEERA: National Center for Education Evaluation and Regional Assistance: NCER: National Center for Education Research: NCSER: National Center for Special Education Research: NCES: National Center for Education Statistics: OESE: Office of Elementary and Secondary Education: OPEPD: Office of Planning, Evaluation, and Policy Development: OMB: Office of Management and Budget: REL: Regional Educational Laboratory: R & D Center: Research & Development Center: SEA: state educational agency: [End of section] GAO: United States Government Accountability Office: 441 G St. N.W. Washington, DC 20548: December 5, 2013: The Honorable John Kline: Chairman: Committee on Education and the Workforce: House of Representatives: The Honorable George Miller: Ranking Member: Committee on Education and the Workforce: House of Representatives: For more than 100 years, the federal government has invested in conducting education research and collecting education data. More recently, the Education Sciences Reform Act of 2002 (ESRA) established the Institute of Education Sciences (IES) and outlines a broad mission for IES to expand fundamental knowledge and understanding of education and to provide this information to a wide variety of stakeholders, including parents, educators, researchers, policymakers, and the general public.[Footnote 1] As the primary research and evaluation arm of the Department of Education (Education), IES is charged with providing information about educational policies, programs, and practices that improve academic achievement and access to educational opportunities for all students. In fiscal year 2013, IES had a budget of just under $600 million, which it used to support a range of research, data collection, and evaluation activities.[Footnote 2] Specifically, IES funds projects on topics that range from studying the effectiveness of basic reading or math initiatives, to evaluating the impact of federal grant programs, to collecting student performance data through the National Assessment of Educational Progress, a nationally representative measure of student academic performance. You requested that we review IES's performance in addressing its critical and wide-ranging mission. This report examines: (1) the extent to which IES has demonstrated its ability to support high-quality research and fulfill its mission; (2) the extent to which selected Education research and technical assistance groups disseminate relevant products to the education field; and (3) how IES coordinates its activities with other relevant federal research entities and within Education. To assess the extent to which IES has demonstrated its ability to support high-quality research and fulfill its mission, we reviewed relevant federal laws and regulations as well as agency documents describing IES's performance measures and key agency processes. We conducted interviews with officials from IES and a range of stakeholder groups representing researchers, policymakers, and practitioners, as well as several members of IES's advisory board--the National Board for Education Sciences (NBES). We obtained IES data for the most recent four years, fiscal years 2010 through 2013, on the average number of work days from the submission of the initial manuscript to final approval for reports sponsored by IES that underwent external peer review before being publicly released. We assessed the reliability of IES data by obtaining written responses from agency officials knowledgeable about the data. We determined that the data were sufficiently reliable for the purposes of this report. To address the extent to which selected Education research and technical assistance groups disseminate relevant products, we obtained documents from each of these groups: the Regional Educational Laboratories (REL), the Comprehensive Technical Assistance Centers (Comprehensive Centers), and the Research and Development Centers (R & D Center). We analyzed these documents to identify what is known about the relevance,[Footnote 3] dissemination,[Footnote 4] and utilization of the research and products these groups produce. We also met with the directors of the RELs and Comprehensive Centers as well as program staff at IES and Education's Office of Elementary and Secondary Education (OESE) who administer these programs. We administered a short survey via e-mail to the directors of IES's R & D Centers to obtain additional information. We fielded this survey from April 2013 to May 2013 and achieved a 94 percent (17 of 18) response rate. We selected these three research and technical assistance groups because they are all authorized by the same law that established IES. To describe how IES coordinates with other relevant federal research agencies and within the department, we reviewed relevant federal laws and regulations and other relevant documents. We also interviewed Education officials responsible for planning evaluations of departmental programs. Finally, we met with officials from two federal research organizations that conduct education-related research--the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD) and the National Science Foundation (NSF)--to learn about their processes and coordination with IES. For all three objectives, we compared agency documents and procedures to GAO's criteria on internal controls, performance measurement and reporting, and collaboration, as well as a framework GAO developed to identify key elements of sound federal research and evaluation programs. [Footnote 5] Appendix I describes our scope and methodology in more detail. We conducted this performance audit from July 2012 to December 2013 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Background: Prior to the establishment of IES, federal education research was carried out by the Office of Educational Research and Improvement, a federal agency that--according to observers both within and outside the federal government--was challenged by frequent leadership changes, competing interests, and lack of a focused mission. The 2002 reauthorization of the Elementary and Secondary Education Act (ESEA) made numerous references to the use of research-based evidence in educational decision making and federal education programs. With its enactment later in 2002, ESRA established IES, an arm of the Department of Education dedicated to the improvement of federal education research, statistics, evaluation, information, and dissemination. IES's mission is "to provide national leadership in expanding fundamental knowledge and understanding of education from early childhood through postsecondary study, in order to provide parents, educators, students, researchers, policymakers, and the general public with reliable information about (1) the condition and progress of education in the United States, including early childhood education; (2) educational practices that support learning and improve academic achievement and access to educational opportunities for all students; and (3) the effectiveness of Federal and other education programs." Furthermore, in carrying out its mission, ESRA calls upon IES to "compile statistics, develop products, and conduct research, evaluations, and wide dissemination activities in areas of demonstrated national need."[Footnote 6] ESRA authorizes IES to conduct and support many different types of research and evaluations in support of its mission.[Footnote 7] Specifically, ESRA contains several key provisions related to the management, core functions, and processes of IES: * all research conducted by IES is to use scientifically based research standards that include, where appropriate, making claims of causal relationships only in random assignment experiments; * education evaluations conducted by IES are to employ experimental designs using random assignment, when feasible; * all research, statistics, and evaluation reports conducted by or supported through IES must be subjected to rigorous peer review before being published or otherwise made available to the public; * the Director of IES will be appointed by the President, by and with consent of the Senate, for a 6-year term and will propose the Institute's overall research priorities and report biennially to Congress and others on the Institute's activities, among other duties; and: * a board of directors is established--the NBES--whose duties include, among other things, (1) advising and consulting with the Director of IES regarding its policies and approving the Director's overall research priorities, (2) reviewing and approving procedures for peer review, and (3) reviewing the work of IES to ensure the consistency of scientifically valid research. All of IES's work is carried out by four Centers, created by law, that exist within IES. See table 1 for more information on these Centers and their key activities. Table 1: The Institute of Education Science's Centers and Their Key Activities: Center: National Center for Education Evaluation and Regional Assistance (NCEERA); Key activities: Through contracts with private research organizations, NCEERA conducts evaluations of federal education programs; provides technical assistance, conducts research, and disseminates research findings through the Regional Educational Laboratory program; and disseminates information about education through the Education Resources Information Center and the National Library of Education. Center: National Center for Education Research (NCER); Key activities: Through investigator-initiated research grants, NCER sponsors sustained research that will lead to the accumulation of knowledge and understanding of education and provide the basis for improving academic instruction and lifelong learning. Center: National Center for Education Statistics (NCES); Key activities: Through contractors and its own staff, NCES collects and analyzes data, and produces reports containing representative education statistics about the condition of the U.S. education system and the academic achievement of our nation's students over time. Center: National Center for Special Education Research (NCSER); Key activities: Through grants to researchers, NCSER sponsors research to expand knowledge and understanding of the needs of infants, toddlers, and children with disabilities in order to improve the developmental, educational, and transitional results of such individuals. Source: GAO analysis of IES documents. [End of table] Education administers programs that support education research and technical assistance through grants and contracts involving several research groups--the RELs and R & D Centers within IES and the Comprehensive Centers within OESE (see figure 1). Figure 1: Selected Education Research and Technical Assistance Groups: [Refer to PDF for image: illustration] Top level: Secretary of Education. Second level: Institute of Educational Sciences; Deputy Secretary of Education. Third level, reporting to Institute of Educational Sciences: National Center for Education Evaluation and Regional Assistance: * 10 Regional Educational Laboratories: Activities: Research and technical assistance on the use of data and research in classroom practice; Client: State and local education agencies. National Center for Education Research: * 12 Research and Development Centers: Activities: Develop, test, and disseminate new approaches to teaching and learning; Client: not defined. National Center for Education Statistics: National Center for Special Education Research: * 5 Research and Development Centers: Activities: Develop, test, and disseminate new approaches to special education teaching and learning; Client: not defined. Third level, reporting to Deputy Secretary of Education: Office of Elementary and Secondary Education: Comprehensive Centers: * 15 Regional Centers: Activities: Technical assistance related to carrying out Elementary and Secondary Education Act and other responsibilities; Client: State education agencies; * 7 Content Centers: Activities: Develop research-based information and products for use in schools; Client: Regional Comprehensive Centers. Source: GAO analysis of Education documents. [End of figure] The National Center for Education Evaluation and Regional Assistance (NCEERA) administers the REL program, a network of 10 regional entities that conduct applied research; develop and disseminate research and products; and conduct technical assistance and other activities to support the needs of state and local educational agencies in their region. The RELs were first established in 1965 under Title IV of ESEA with the broad goal of supporting general educational improvement efforts in their regions. Since IES was created and began to administer the REL program, the RELs have completed two contract cycles, and the current RELs were launched in 2012 under new 5-year contracts. ESRA also includes specific requirements for the administration of National R & D Centers, which are designed to address areas of national need and each of which addresses at least one of the broad research topics outlined in the law.[Footnote 8] R & D Centers are also responsible for the production of rigorous evidence and dissemination of products that provide practical solutions to important education problems in the United States. Each R & D Center is funded for no more than 5 years, and in fiscal year 2013, the National Center for Education Research (NCER) and the National Center for Special Education Research (NCSER) administered 18 R & D Centers in total. The Comprehensive Centers Program is a network of 22 technical assistance grantees that help to increase the capacity of state educational agencies (SEA) to assist districts and schools in meeting student achievement goals. Established in 1994, Comprehensive Centers provide services primarily to SEAs to enable them to assist school districts and schools, especially low-performing schools. Comprehensive Centers provide training and technical assistance in (1) the implementation and administration of programs authorized under ESEA and (2) the use of research-based information and strategies. Selected Comprehensive Centers focus on specific content areas and produce research-based information and products for use by SEAs. In addition to IES, other entities conduct education-related research and evaluations, and ESRA includes general requirements for the Director of IES to coordinate its research and evaluation work with these entities, both within Education and across the rest of the federal government.[Footnote 9] Within Education, the Office of Planning, Evaluation, and Policy Development (OPEPD) conducts analyses and program evaluations on behalf of the department. Other federal agencies, such as NSF and NICHD--part of the National Institutes of Health--also support education-related research and the directors of these agencies serve as nonvoting ex officio members on the NBES. [Footnote 10] In addition, many nongovernmental organizations conduct education-related research and evaluations that inform general knowledge and understanding of educational policies, programs, and practice. For example, foundations such as the Bill & Melinda Gates Foundation, The William T. Grant Foundation, and The Spencer Foundation, conduct education research or program evaluation in specific topic areas to advance the public interest or their organization's mission or goals. Additionally, other research organizations such as Mathematica Policy Research, MDRC, and SRI International, contract with the federal government or other clients to conduct agreed upon research or program evaluations on their behalf. IES Supports High-Quality Research, but Lacks Key Processes and Performance Measures in Some Areas: IES Supports High-Quality Research, but Does Not Monitor the Timeliness of Some Aspects of Its Peer Review Process: IES has substantially improved the education research field. In 2007, the Office of Management and Budget (OMB) assessed IES's research and concluded that IES had transformed the quality and rigor of research within Education and increased the demand for scientifically based evidence of effectiveness in the education field as a whole.[Footnote 11] Likewise, many stakeholders told us that IES's research standards had improved Education's research and had a positive influence on education research generally. More specifically, several stakeholders told us that IES products, such as its publications of education statistics reports, were useful for their work. In addition, one regional comprehensive center director told us that a Practice Guide IES released on dropout prevention has become the framework that one of the states in its region is using for dropout prevention efforts statewide. While IES's research grants and evaluations have resulted in many randomized controlled studies since the agency was established over 10 years ago, its research standards also include guidelines for the implementation of other rigorous research methodologies, and it has recently funded studies using those methodologies, such as regression discontinuity or single-case designs.[Footnote 12] Further, IES officials also described a variety of other types of research that the agency supports, such as data analyses and correlational analyses. IES's support of these multiple types of methodologies allows it to better meet its various stakeholders' needs. At the same time, IES's research can be of limited usefulness to policymakers and practitioners if it is not released in a timely manner.[Footnote 13] Some stakeholders told us that research and evaluations supported by IES may not be completed soon enough to inform the decision making of policymakers and practitioners on important questions, which is a key component of its mission.[Footnote 14] For example, officials from one constituency-based organization for policymakers told us that IES's evaluation of Education's Race to the Top and School Improvement Grant Programs are of great interest to states, but by the time the results of these evaluations are released in 2014, states will not have much time to implement lessons learned from these studies before their program's funding expires.[Footnote 15] IES's peer review process may also exacerbate timeliness concerns. In order to ensure the high quality of IES's work, ESRA requires IES- supported research reports to be peer reviewed before being published.[Footnote 16] In recent years however, the time it takes to complete this review process substantially increased, from an average of 117 days in fiscal year 2011 to 175 days in fiscal year 2012 and 150 days in fiscal year 2013.[Footnote 17] As long ago as 2008, peer review timeliness was a concern of the NBES and at that time, they recommended that IES establish procedures to ensure the timely receipt of reports and revisions from its contractors. When asked for explanations for the recent increase in its peer review timeframes, senior IES officials told us that the timeliness of IES contractor responses to peer review comments may still be a factor. They also cited factors such as the complexity of the reports reviewed recently and the time it took IES's Centers to work with its contractors on suitable responses to peer review comments. However, IES officials told us that while the peer review office within IES monitors the time its own staff take to review reports, the peer review office does not monitor the time IES's Centers or contractors take to respond to peer review comments. Such monitoring would allow IES to take steps to mitigate delays, such as by holding contractors more accountable. In accordance with federal internal control standards, program managers should have access to and use operational data to determine whether they are meeting their agencies' goals for accountability for effective and efficient use of resources.[Footnote 18] IES Has Taken Steps to Meet Stakeholders' Needs, but Lacks a Formal Process to Gather Input on Its Research Agenda: To ensure that its research addresses the needs of a range of stakeholders and to address concerns about how relevant its research is to these stakeholders, IES is soliciting feedback from practitioners and redesigning some existing programs. Current IES officials said that IES has in the past 10 years established the quality of IES-supported research, and that they are continuing to prioritize engagement with policymakers and practitioners. Several stakeholders with whom we spoke also noted that IES has recently devoted more attention to policymaker and practitioner outreach to improve its relevance to the education field. For example, IES recently convened a group of 17 state and local education officials and other stakeholders to discuss the strengths and weaknesses of products from the RELs and the What Works Clearinghouse, a program that IES administers to evaluate the merits of and disseminate education research evidence.[Footnote 19] According to IES, at this meeting participants discussed their opinions on the usability, relevance, and accessibility of current REL and What Works Clearinghouse products. In addition, in June 2013, IES officials told us they were planning to convene a small group of researchers to assess potential gaps in the research conducted by two of its Centers: NCER and NCSER. IES also recently held a discussion with its board of directors about how to make its research more relevant to policymakers and practitioners. IES has also reorganized several of its existing grant programs and initiated a new grant program to support research that will better target and understand the needs of the education field. For example, one of these grant programs focuses on collaborations between researchers and state or local educational agencies, and in July 2013, six new grants were awarded for collaborative research projects, such as the design of a randomized postsecondary developmental education experiment in Texas and the development of a system to monitor students' social and emotional learning in a school district in Nevada.[Footnote 20] Additionally, IES is planning to fund a new R & D Center in fiscal year 2014 on knowledge utilization to study how education researchers can make their work more relevant and useful to practitioners and how practitioners can make productive decisions based on research evidence. Despite IES's recent efforts to increase the relevance of its research, IES does not have a structured process for incorporating feedback from policymakers and practitioners into its research agenda. ESRA requires that IES's board of directors be composed of education researchers and other stakeholders, including educators, policymakers, parents, or school administrators.[Footnote 21] However, according to IES officials, IES has had difficulty with the nomination and approval process for appointing board members and, according to one long time board member, the NBES has, at times, had difficulty retaining members from these stakeholder groups. Further, according to IES officials, IES does not have in place any other ongoing, structured outreach to stakeholders for input to its research agenda, though the senior leaders of IES's Centers and the Director of IES periodically engage with policymakers and practitioners on specific topics or projects. Inconsistent outreach by IES may have contributed to gaps in its research. For example, stakeholders said that there is a shortage of research using mixed methodologies that could allow for shorter turnaround times among IES-funded research projects. While obtaining and integrating policymaker and practitioner feedback into a research agenda can be difficult, other research agencies have external groups of practitioner-stakeholders to which they routinely turn for perspectives on the overall direction of their research. Officials at NICHD, for example, told us that, in addition to its statutorily required National Advisory Council, they consult with an independent group called the Friends of NICHD, comprised of organizations that use NICHD research, to gather input on NICHD research plans and agendas. Similarly, in addition to the National Science Board, NSF officials told us that several advisory committees, composed of outside experts and stakeholders, regularly provide recommendations regarding the direction of the agency's research. Though there is no single or ideal way for government agencies to conduct research, we previously developed a framework to identify key elements that promote a sound federal research program, using guidelines from several leading national research organizations. Within that framework, we found that agencies should establish a structured process for developing their research and evaluation priorities that considers key stakeholders' input.[Footnote 22] For example, we reported on one federal research agency that has procedures that call for routinely consulting with stakeholders-- including policymakers and key institutions that influence public policy--about past research accomplishments and program effectiveness and impact.[Footnote 23] IES's Performance Measures Do Not Reflect Its Current Programs: IES cannot demonstrate the impact of its efforts to improve the quality and relevance of its research in some areas because its performance measures have not been updated to reflect its current programs. Since 1993, all federal executive branch agencies have been required to set strategic goals, measure performance, and report on the degree to which goals were met in an effort to ensure government accountability and enhance public awareness about agencies' accomplishments.[Footnote 24] In addition, according to federal internal control standards and leading practices on performance management, agencies should establish performance measures for their activities and continually compare actual performance data against these goals.[Footnote 25] However, IES officials told us that IES's current performance measures, which were developed after the agency was created by law in 2002, no longer capture the scope of IES's current research and priorities. Furthermore, in some cases, senior IES officials told us these performance measures are no longer relevant to managing the agency's operations. For example, one performance measure relies on the results of a survey of potential users of the What Works Clearinghouse, but IES officials told us they have never conducted this survey because they do not feel it would yield enough useful information to be worth the investment. To measure the effectiveness of its research grants, IES currently reports on a measure that counts the number of IES-supported interventions that have been determined to be effective in improving student outcomes in particular areas. While a new performance measure was reported in Education's fiscal year 2014 budget request for IES that better reflects the results of its recent research, this measure still does not include certain areas, such as research on the organization and management of schools and education systems. In addition, IES does not publicly report on the overall performance of the REL program, which constitutes one of the agency's largest investments. As we have reported, without performance measures that include targets or goals to demonstrate results, agencies may be at risk for failing to achieve their goals.[Footnote 26] IES officials told us they have begun work on revising their agency's overall performance measures. Officials told us that they plan to include revised performance measures in Education's fiscal year 2015 budget request for IES, and that they have begun discussions with OMB to establish these new measures. As of August 2013, IES officials told us they intend for the new performance measures they are developing to include all programs, including the REL program and its new grant programs for researcher-practitioner partnerships. Research and Technical Assistance Groups Take Steps to Disseminate Relevant Research, but IES Has Not Fully Assessed These Efforts: Groups Have Produced Research and Products for Use by Policymakers and Practitioners, but Relevance and Dissemination Could Be Improved: RELs, Comprehensive Centers, and R & D Centers have taken various steps to provide relevant research to the education field. According to statute, RELs and Comprehensive Centers are required disseminate information that can be used by practitioners and policymakers to improve academic achievement.[Footnote 27] Similarly, IES requires R & D Centers to disseminate research to policymakers and practitioners to improve teaching and learning, and ultimately, student achievement. To identify topics of relevance to the education field, all three groups have engaged policymakers and practitioners in planning research activities. For example, beginning in 2012, RELs were required to conduct their work through new or existing partnerships of practitioners, policymakers, and others--called research alliances, which would work together to use data and research to address specific problems in education. According to IES officials, REL projects must be based on needs identified and agreed to by the research alliances. For example, the REL that includes the Silicon Valley area works with a research alliance consisting of local school districts, county education officials, foundation leaders, university faculty, and intervention specialists to boost math achievement. This alliance began because local districts were concerned about low student achievement in math, given its importance for the local technology industry. With input from the alliance, this REL is developing and analyzing strategies to ensure adequate preparation in math with the goal of preparing students for postsecondary education. Despite efforts by these groups to ensure the relevance of their work, stakeholders, including teachers and policymakers, have raised concerns about the applicability of some of the research and products these groups have produced, as well as their timeliness.[Footnote 28] For example, a stakeholder group we spoke with that represents local school districts, as well as two superintendents we spoke with, said they did not find REL research as relevant or as timely as other sources they turn to for research information. Additionally, teachers we spoke with told us it would be helpful if IES-supported groups produced more products that synthesize research findings so that they are more applicable to classroom practice, similar to research-based products they use from professional associations or other intermediary organizations. Stakeholders also noted that the timeliness of REL products has been a concern. REL research findings cannot be released to the public until they have cleared the IES peer review process, which as we noted earlier, has taken longer in recent years.[Footnote 29] Some stakeholders expressed concern that in the absence of timely and applicable information from IES, other entities may provide practitioners and policymakers with research-based information that may not be conducted with comparable research standards or be as objective and unbiased as IES-supported research. In addition, the research topics and the products produced by the R & D Centers primarily reflect the priorities of researchers, according to many of the researchers we spoke with, even though the Centers have multiple audiences, including policymakers and practitioners. Stakeholder groups representing policymakers and practitioners also said that the R & D Centers could do more to adapt their research findings to formats readily accessible by these audiences, such as by producing nontechnical reports and shorter research summaries. All three groups use a range of methods to disseminate their research evidence and products, such as publications and conferences. For example, in our survey of R & D Center directors, we found the common methods they used to disseminate their research were academic journal publications and presentations at conferences, as well as hosting conferences. Comprehensive Centers disseminate products and tools they produce by sharing them on their websites, via e-mail distribution lists, and with other Comprehensive Centers. In addition, Comprehensive Centers create summaries of current research for their state clients, and also develop other technical assistance resources specifically for teachers, such as professional development courses. Although largely successful in reaching academicians, REL and R & D Center dissemination efforts do not always ensure that research reaches policymakers and practitioners. Regarding RELs, researchers and several groups representing policymakers and practitioners told us that REL work was not reaching these audiences, and teachers we spoke with were generally unfamiliar with RELs. A few stakeholders also noted that REL productivity varied widely, and that some have produced very few reports.[Footnote 30] Stakeholders gave several reasons why RELs may struggle to reach practitioner audiences. For example, some intermediaries--such as industry associations--we spoke with said their organizations help to disseminate research information to policymakers and practitioners. However, some noted that further efforts are needed to leverage intermediary groups to better market REL and R & D Center work to reach IES's target audiences. IES Has Not Fully Assessed Groups' Efforts to Disseminate Relevant Research: IES requires RELs to report some information about relevance and dissemination that could be used to evaluate their efforts, but this information has not been collected in a consistent manner and is therefore difficult for IES to use to improve program management. For example, RELs use a stakeholder feedback survey instrument to capture feedback from participants on relevance and other aspects of the products, activities, and events they sponsor. This survey includes a set of questions for five different types of REL activities and products, such as technical assistance workshops or research alliance participation. However, as administered in 2012--the first year of the current RELs' operation--RELs were not consistent in how they administered the survey. For example, some RELs administered the complete survey for their activity or product type, while other RELs selected specific questions for respondents. The results, therefore, are not comparable across all REL activities and products. Further, the data collected only applies to participants who elected to provide feedback, rather than all participants in REL programs or potential users. In addition to the feedback survey, RELs are required to annually report to IES on a series of indicators that they developed collaboratively with IES in March 2012. However, this preliminary information is not sufficient for IES's management of RELs' efforts to produce and disseminate relevant research. The REL program has six expected outcomes, one of which is to build a body of knowledge in topics that address regional needs. To assess performance on this and the other five program outcomes, RELs must report to IES on 24 indicators. For example, regarding dissemination, RELs must report the number of events held and the total number of attendees at these events. However, IES has not established performance targets or goals for these indicators to help RELs prioritize their activities or assess their performance in these areas. Were IES to establish targets or goals, the RELs would likely have more incentive to perform at the agency's desired level, and IES would be better positioned to determine if the RELs are meeting its expectations and hold them more accountable for performance. Our previous work has indicated agencies successful in measuring performance had performance measures including targets or goals that (1) demonstrate results; (2) are limited to the vital few; (3) cover multiple priorities; and (4) provide useful information for decision making.[Footnote 31] Regarding the R & D Centers, IES has few requirements for tracking relevance and dissemination, although some Centers collect additional information that could be helpful for assessing performance in these areas. In combination with other factors, IES assesses the relevance of the R & D Centers' proposed research projects when making funding decisions, but according to officials, this is the only formal assessment of the relevance of R & D Center work. However, although they are not required to do so, most R & D Center directors responding to our survey said they use quantitative data to gauge their relevance to policymakers and practitioners. For example, one R & D Center has conducted national surveys of certain constituent groups, such as state policymakers, asking about the relevance of the Center's findings and products for their use. In their annual performance reports, IES asks R & D Centers to report whether they disseminated research information, but does not specify what information they should provide, such as how they disseminated information or the effectiveness of these efforts. Though it can be difficult to demonstrate how research programs with broad dissemination goals--such as the R & D Centers--are able to inform others and contribute to outcome-oriented goals, there are several evaluation strategies to assess how programs with great breadth and flexibility contribute to agency goals.[Footnote 32] IES has not, however, set utilization or dissemination goals and objectives, or investigated whether dissemination strategies were effective in improving the efforts of these groups. According to officials, IES has no plans to conduct formal evaluations for the current group of RELs and R & D Centers to comprehensively assess their relevance and dissemination activities. IES is still in the process of conducting a mandated evaluation of the prior group of RELs, whose contracts ended in 2011.[Footnote 33] According to study design documents and IES officials, this IES-contracted evaluation will include information from a customer satisfaction survey of a representative sample of state and district staff regarding their views of the relevance and usefulness of REL work. The evaluation will also include an assessment by expert panelists of the quality and relevance of selected REL reports. However, it will not include an assessment of REL dissemination activities. This evaluation began in 2009 and an interim report was initially expected to be released in spring 2012; however, the interim report was delayed and was released in late September 2013.[Footnote 34] IES has no further plans to evaluate the REL program beyond the forthcoming evaluation of the prior group. Officials told us although there was a clear requirement in ESRA for this evaluation, they did not believe that the law required any subsequent evaluation of the REL program. In addition, as of August 2013, IES had no plans to conduct a formal evaluation of the R & D Center program, and there is no requirement to do so. According to IES officials, they do not plan to evaluate them because they do not view them as a program, but rather as one vehicle for supporting IES's programs of research, as with its education research grants. However, research on dissemination practices funded by other federal research programs has shown that dissemination efforts should be assessed periodically to improve further efforts to communicate key messages to target audiences.[Footnote 35] As required by ESRA, IES has previously evaluated the Comprehensive Centers.[Footnote 36] A 2011 IES-contracted evaluation reviewed how well the Comprehensive Centers addressed state client needs, as well as the quality, relevance, and usefulness of their assistance, among other things.[Footnote 37] The evaluation included a survey of SEA administrators and a review of Comprehensive Center products. Evaluation data were collected annually in 3 of the 5 program years in which the Centers operated. Comprehensive Centers were rated higher in each successive year, and in 2008-2009, 56 percent of state managers surveyed said that technical assistance from the Comprehensive Centers served the state's purposes completely. For the state managers who said the Comprehensive Centers did not completely meet their needs, the main reason cited was that Center staff had limited time to work with their state. The evaluation also included a quality assessment of a sample of Comprehensive Center projects by expert panelists and project participants. Each sampled project was independently rated by a panel of experts using a 5-point scale. SEA staff who participated in these projects (e.g., by serving on a work group associated with the project) were asked via survey to rate the relevance and usefulness of each project. For the sample of projects assessed, the evaluation showed that the Comprehensive Centers' technical assistance was rated higher on each measure in each successive year, from 2006- 2007 to 2008-2009. On average, across each of the 3 years, expert panels rated sampled project materials from "moderate" to "high" for quality, and project participants rated the sampled projects "high" for relevance and usefulness. While these results indicate a positive trend in Comprehensive Centers' performance, this evaluation covered the prior group of Comprehensive Centers and IES recently released a new Request for Proposals to evaluate the current group. IES Coordinates with Other Federal Agencies, but Fragmented Evaluation Funding Poses Challenges for Education's Evaluation Efforts: IES Coordinates with Other Agencies to Increase the Use of Evidence in Federal Decision Making and Research Planning: IES coordinates with relevant federal research agencies on projects to increase federal agencies' use of research evidence in guiding funding decisions. According to federal agency officials and stakeholders we spoke with, NICHD and NSF are the two other primary federal agencies conducting education-related research. IES officials meet regularly with their NICHD and NSF counterparts to jointly sponsor projects and ensure their efforts are complementary. For example, IES co-led a joint Education-NSF working group to develop common evidence guidelines for education research and development.[Footnote 38] These guidelines are intended to help guide NSF's and IES's respective decisions about investments in education research and clarify for potential grantees and peer reviewers the evidence expected from each type of study. Several researchers we interviewed, as well as officials from three federal agencies we spoke with, said these guidelines will benefit the education field. In addition, developing common evidence guidelines is consistent with our key practices for enhancing and sustaining collaboration among federal agencies. [Footnote 39] In particular, we have found that establishing compatible policies, procedures, and other means helps to align the partner agencies' activities, core processes, and resources to accomplish a common outcome.[Footnote 40] Consistent with this key practice, IES also coordinates with other federal agencies to leverage its expertise in planning federal research agendas on topics such as early learning and prisoner re- entry, as well as increasing the quality of and access to data on education outcomes. For example, IES sits on the Early Learning Policy Board, a joint Education-Health and Human Services body, to discuss ways to coordinate federally-funded research on early childhood topics. IES's statistics Center leads the Interagency Working Group on Expanded Measures of Enrollment and Attainment, to improve national data on education, training, and credentials for work. This effort includes OMB, the Council of Economic Advisors, and other federal statistical agencies, such as the Census Bureau and the Bureau of Labor Statistics at the Department of Labor. In addition, IES has worked with the Defense Advanced Research Projects Agency at the Department of Defense on its Small Business Innovation Research Program, and in December 2012 they released a joint solicitation to support the development and evaluation of education technology games. IES's efforts to help develop a systematic, transparent evidence review process and incorporate a tiered evidence framework for awarding grants for Education's Investing in Innovation Fund grant program have had an impact on the use of evidence in federal decision making both within and outside the department. According to an official from the Administration of Children and Families at the Department of Health and Human Services, for some of its grant programs, the Administration of Children and Families has recently used tiered evidence standards similar to those IES helped to develop for Education's Investing in Innovation Fund, reserving the majority of funds for applicants proposing to implement service models with evidence of effectiveness. IES is also working with Education program offices to incorporate tiered evidence standards in awarding their grants, such as the Strengthening Institutions Program in the Office of Postsecondary Education. Coordination between the RELs and Comprehensive Centers Has Improved: In addition to working with other federal agencies, IES coordinates within Education to facilitate coordination between the REL and Comprehensive Center programs. There is a statutory requirement for each Comprehensive Center to coordinate its activities and collaborate with the RELs in its region, as well as with certain other entities. [Footnote 41] Prior to the current award cycle, IES and OESE restructured REL and Comprehensive Center regions so that the number of Comprehensive Centers in each REL region was reduced and aligned more closely with REL regions. According to OESE officials, its Comprehensive Center grantees and states found the lack of alignment between the regions in the prior contract cycle to be a barrier to coordination. Further, IES has provided guidance regarding its expectations for coordination between RELs and Comprehensive Centers. For example, IES has encouraged RELs to partner with Comprehensive Centers and has allowed RELs to budget resources for these collaborative efforts. According to directors of these groups, coordination has improved with the current group of RELs and Comprehensive Centers, which began in 2012.[Footnote 42] One REL Director we spoke with told us that the realignment of the regions has facilitated improved coordination between the groups. In addition, some directors said that the RELs' new structure of research alliances has improved coordination. For example, some Comprehensive Centers are members of REL research alliances and have partnered with RELs to host "bridge" events, where state officials and researchers meet to discuss research findings relevant to issues of policy and practice. REL directors told us that although there can still be confusion among some SEA officials about the appropriate role and tasks performed by each, RELs and Comprehensive Centers have used several strategies to reduce confusion and improve coordination. For example, some groups have conducted joint visits with SEA officials so that they can describe their respective roles and responsibilities and specifically discuss how each group can address the state's various technical assistance needs. In addition, some directors described their groups' efforts to sequence their work for their state clients to better meet their needs. In some instances, for example, the Comprehensive Center may conduct initial planning work for a project, and the REL would later assist with any aspects of the project requiring data analysis or original research. For example, one Comprehensive Center director told us that, through joint efforts with the REL in their region, it developed a framework for measuring teacher effectiveness. For this project, the REL formed a research alliance with other Comprehensive Centers and SEAs interested in piloting the framework. The Comprehensive Center is helping states implement the pilot, and the REL will conduct an evaluation of the pilot projects. This Center Director told us that it will help the states adjust their strategies based on the findings from the pilot evaluation. Education's Inability to Combine Evaluation Funding Poses Challenges for Evaluation Planning Efforts: IES coordinates within the department to plan evaluations of Education programs. Beginning in 2010, OPEPD and IES have jointly led an annual department-wide evaluation planning process to identify evaluation projects to conduct.[Footnote 43] Although there is no official guidance delineating these roles, Education officials said that the annual evaluation planning process is used to help determine which of these offices will conduct each project. According to Education, this planning process has helped to improve the quality of evaluations conducted, and made evaluations more responsive to the information needs of the department. For example, IES is conducting an implementation and quasi-experimental evaluation of the Race to the Top Program. An IES official reported that input from the Education program office running this program helped ensure that the evaluation's data collection instruments focused on implementation issues most important to Education and state education leaders. IES is also working with this program office to select topics of greatest immediate interest to Education for short implementation briefs to be released as the study progresses. However, efforts to coordinate internally and prioritize evaluation research projects through this annual process are challenged in part by statutory requirements related to ESEA evaluation funding. For these programs, evaluation funds are typically set aside as a percentage of program funding or as national activities funds. [Footnote 44] With some exceptions, ESEA authorizes Education to reserve up to 0.5 percent of the amount appropriated to carry out each program or project authorized under that Act to conduct evaluations. [Footnote 45] However, according to Education officials, due to the statutory limitation, the funds available for evaluation are often insufficient to conduct high-quality evaluations of these programs. [Footnote 46] This is particularly true with respect to certain discretionary grant programs where overall program funding is itself relatively small. As a result, OPEPD and IES officials said that some evaluations, including high-priority evaluations, may not occur. For example, IES officials told us that the department is interested in conducting evaluations of the Mathematics and Science Partnerships and Promise Neighborhoods programs, but there is not sufficient evaluation funding under these programs due to the statutory limitation and Education does not have the authority to combine evaluation funds across the department and use them to conduct evaluations of these or other high-priority programs.[Footnote 47] We have previously reported that many Education programs, especially smaller programs, have not been evaluated, which can limit the ability of Congress to make informed decisions about which programs to continue, expand, modify, consolidate, or eliminate.[Footnote 48] For example, in 2009, we reported that 11 of Education's programs focused on teacher quality had been operating for over seven years and had never been evaluated.[Footnote 49] In addition, according to OPEPD officials, ideas for some high-priority evaluations that department officials have discussed during the annual evaluation planning process are never developed into plans because officials know that evaluation funds are not available to carry them out. However, according to officials from OPEPD, beginning in fiscal year 2015, Education will maintain a list of all of the department's evaluation needs should funds for conducting these evaluations become available. According to a senior IES official, if the department were able to combine funds authorized to be used for evaluation it would have more flexibility to conduct appropriate evaluations of any of ESEA programs that need evaluation. The President's Fiscal Year 2014 Budget request for Education contained a proposal to increase Education's flexibility to conduct program evaluations by allowing the department to use funds from certain programs across the department for this purpose.[Footnote 50] An OMB staff member told us that the ability to combine evaluation funds and prioritize evaluations is critical in the current environment of constrained funding to help make the best use of limited resources and to fill gaps in topic areas where research is needed. In fiscal year 2012, Congress granted the Department of Labor temporary authority to combine program-specific funds to support evaluations of its programs.[Footnote 51] The Department of Labor is required to provide a plan to the congressional appropriations committees before combining its funds in this manner. This plan is to describe the specific evaluations it plans to carry out using these funds and helps ensure transparency and accountability in the department's use of this authority. In the Department of Labor's fiscal year 2014 budget justification, it reported that the availability of flexible evaluation funds has allowed the department to evaluate large and high-priority programs, as well as programs that may not have been fully evaluated previously. Conclusions: Since its creation more than a decade ago, IES has made significant contributions to strengthening the rigor of the education research field and has promoted the use of scientifically based research in our nation's education system. However, IES's ability to release timely information to policymakers and practitioners is exacerbated by its peer review process which, while promoting rigorous research, can take a substantial amount of time. Without better use of available data to manage this peer review process, it will remain difficult for IES officials to identify potential causes of these delays and develop strategies to mitigate these challenges. More recently, the agency has begun working to make its products more relevant to a range of stakeholders, particularly policymakers and practitioners. Producing education research that is relevant for these stakeholders is critical to inform decision making in Congress and in the field about key federal investments in education. Despite progress IES has made in ensuring that its research is relevant to its stakeholders, IES does not have a formal, structured process for engaging with these groups in the research-agenda setting process. Until such a process is in place, IES limits its ability to ensure the relevance, and ultimately, usefulness of its research to the field. IES's relevance to the field is unclear in other ways as well. For example, without publicly reported performance measures for key investments--such as the RELs or new high-priority grant programs--stakeholders, such as Congress and the public, are unable to determine the effectiveness of IES's activities or evaluate the merits of the federal investment in these programs. IES's research and technical assistance grantees and contractors take a number of steps to help ensure they produce and disseminate relevant research and products. However, without assessing the effectiveness of these groups' dissemination efforts, IES may miss opportunities to think more strategically about dissemination and ensure that these groups are using the most effective strategies to reach their target audiences. Providing Education with the authority to combine funds authorized for evaluations under ESEA would allow the department more flexibility to prioritize and conduct effective evaluations of programs that represent key federal investments and critical subject areas. Without this authority, it will be difficult for Education to make the best use of its limited resources and to support Congress in making informed decisions about which programs to continue, expand, modify, consolidate, or eliminate. Matter for Congressional Consideration: To ensure that Education can conduct critical program evaluation work, Congress should consider granting Education authority to combine funds authorized for evaluations of ESEA programs and target them to high- priority evaluations, with appropriate measures to ensure transparency and accountability for how the funds will be used. Recommendations for Executive Action: To improve the management and accountability of IES's research and evaluation efforts, we are making the following recommendations: 1. IES should use available data to routinely monitor all stages of its peer review process and identify opportunities to improve the timeliness of its reviews. 2. IES should develop a structured process to systematically gather input from policymakers and practitioners and use this input when developing its research agenda. 3. IES should develop performance measures, including targets and goals that clearly demonstrate results, and that reflect its current programs and all key agency activities, such as the performance of the RELs and its new grant programs supporting researcher-practitioner partnerships. 4. In order to identify leading practices and target areas for improvement, IES should assess the effectiveness of REL and R & D Center dissemination strategies by, for example, collecting consistent data and lessons learned from these groups to inform future dissemination efforts. Agency Comments and Our Evaluation: We provided a draft of this report to the Department of Education for review and comment. Their comments are reproduced in appendix II. Education officials also provided technical comments, which we incorporated into the final report, as appropriate. Education generally agreed with our recommendations and has several plans in place to address them. Specifically, IES noted that it continues to be committed to producing high-quality reports while shortening the time period for its peer review process, and stated that it will develop a system to more comprehensively monitor all of the stages in its report production process so that it is able to identify and address potential problems more easily and quickly. To further emphasize its commitment to obtaining practitioner and policymaker input, IES also said it will begin an annual solicitation for practitioner and policymaker input on its research agenda by posting an invitation on its web site and widely advertising the opportunity through its IES Newsflash email notices. Additionally, as we noted in our report, IES commented that it is currently working to align its performance measures with its current investments, including developing measures for the REL program. Finally, IES agreed that it is important to measure and assess the effectiveness of IES-supported dissemination activities. IES noted it will build knowledge in this area through its fiscal year 2014 request for applications for a new R & D Center focused on knowledge utilization to develop measures for how practitioners use research evidence to guide what they do in schools and classrooms. We support IES's continued focus and emphasis on advancing knowledge and providing usable information to improve education policy and practice. We are sending copies of this report to relevant congressional committees and the Secretary of Education. In addition, the report will be available at no charge on GAO's website at [hyperlink, http://www.gao.gov]. If you or our staffs have any questions about this report, please contact me at (202) 512-7215 or scottg@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III. Signed by: George A. Scott: Director, Education, Workforce, and Income Security Issues: [End of section] Appendix I: Objectives, Scope, and Methodology: Our review examined (1) the extent to which the Institute for Education Sciences (IES) demonstrates its ability to support high- quality research and fulfill its mission; (2) the extent to which selected Department of Education (Education) research and technical assistance groups disseminate relevant products to the field; and (3) how IES coordinates its activities with other relevant federal research entities and within Education. For all three objectives, we reviewed the Education Sciences Reform Act of 2002 (ESRA) to identify statutory requirements related to IES management, processes, and procedures. We obtained and analyzed agency documents detailing IES's processes and performance. We also conducted interviews with senior IES officials, including Commissioners at all four IES Centers [Footnote 52] as well as selected program staff, several current and former members of IES's advisory board, the National Board for Education Sciences, and selected former senior IES officials to gather additional perspectives on IES's management and performance over time. Specifically, to assess the extent to which IES has demonstrated its ability to support high-quality research and fulfill its mission, we reviewed documents and data from IES's Standards and Review Office, which administers the peer review of IES reports and grant applications. In part to evaluate IES's timeliness in producing relevant research, we obtained IES data for the most recent 4 years, fiscal years 2010 through 2013, on the average number of work days from the submission of the initial manuscript to final approval for reports sponsored by IES that underwent external peer review before being publicly released. We assessed the reliability of IES data by obtaining written responses from agency officials knowledgeable about the data. We determined that the data were sufficiently reliable for the purposes of this report. To further assess IES's overall processes and operations, we applied a framework GAO previously developed to evaluate the soundness of federal research and evaluation programs. This framework includes five key phases of the research and evaluation process: (1) agenda setting, (2) selecting research, (3) designing research, (4) conducting research, and (5) disseminating research results.[Footnote 53] While we assessed IES's activities in each of the five phases, we primarily focused on the first and fifth phases of this process: agenda setting and disseminating research, as these two phases most directly addressed our research questions. Additionally, we reviewed publicly reported performance information contained in Education's most recent budget request to Congress for IES to compare performance trends over time and to assess the completeness of IES's performance measures against GAO's leading practices for performance management and federal standards for internal control in the areas of information and communications, and performance reporting and monitoring.[Footnote 54] We also interviewed officials at the research and evaluation arm of the Administration for Children and Families at the Department of Health and Human Services, and at the Office of Management and Budget that were recommended to us as knowledgeable of federal research and performance reporting to obtain their insight on IES's research, evaluation, and performance measurement. Lastly, we observed two public meetings of the National Board for Education Sciences that took place during the course of our review on October 5, 2012, and February 22, 2013. We also conducted more than 40 interviews with several categories of IES stakeholders: (1) education researchers; (2) organizations representing education policymakers and state-level administrators; and (3) practitioner organizations, individual teachers, and district leaders.[Footnote 55] To identify appropriate stakeholders in all three categories, we obtained recommendations on individuals or groups to contact from key organizations, contacted those individuals or groups, and obtained further recommendations from them on additional individuals or groups to contact.[Footnote 56] We continued to ask for names from the subsequent individuals and organizations until we began getting duplicate referrals. We also identified individuals and groups to contact through our prior work on education research and internal expertise. For the education researchers, we sought to identify education researchers that were considered to be knowledgeable about education research and the activities of IES and its predecessor organizations.[Footnote 57] For the education policymaker organizations, we ensured representation from groups that represent the wide range of topical areas in which IES funds research and evaluation, such as early childhood education, special education, and postsecondary education. We interviewed teachers and superintendents because they are part of IES's target population of end-users, though they had various levels of familiarity with IES prior to our interview. We conducted two discussion groups with experienced education researchers at the 2013 Annual Conference of the American Educational Research Association (AERA). We assembled senior researchers for these discussion groups with the help of AERA staff. Invitations to participate in these discussion groups were provided to selected members of the AERA Fellows program that were culled by AERA staff to ensure representation from a balanced range of subject matter areas. [Footnote 58] During these discussion groups, we met with a total of 16 senior education researchers and discussed issues such as IES's research standards, peer review process, and the relevance and dissemination practices of IES overall, as well as those of selected IES research and technical assistance groups, specifically--the Regional Educational Laboratories (REL) and Research & Development Centers (R & D Center). To address our second research objective, we focused on three types of research and technical assistance groups. Two are administered by IES- -the RELs and R & D Centers--and one is administered by the Office of Elementary and Secondary Education (OESE)--the Comprehensive Technical Assistance Centers (Comprehensive Centers). We selected these three research and technical assistance groups because they are all authorized by the same law that established IES. Moreover, there is a statutory requirement for the Comprehensive Centers to coordinate their activities and to collaborate with the RELs, among others. Specifically, to assess the extent to which these groups disseminate relevant products to the education field, we obtained and analyzed the most recent annual performance report submitted to IES and OESE, respectively, from each of these groups. We analyzed these documents to identify what quantitative and qualitative information IES collects about the relevance, dissemination, and utilization of the research and products these groups produce. We also interviewed Directors of the RELs and the Comprehensive Centers as well as program staff at IES and OESE that administer the programs to discuss relevance, dissemination, and coordination activities. To further understand the Comprehensive Centers activities related to the production and dissemination of relevant research and products to the field, we reviewed a contractor-led evaluation of the Comprehensive Center Program. In particular, we reviewed findings related to relevance of the products these groups produced and the levels of satisfaction reported by their customers. We also reviewed early study design documents and an interim report for an ongoing evaluation of the REL program.[Footnote 59] We reviewed the scope and methodology for each of these evaluations and determined they were sufficiently reliable for our reporting purposes. We also obtained information from IES staff responsible for overseeing these evaluations to obtain additional information. We administered a short survey to the Directors of IES's 18 R & D Centers that were operational in the most recent full program year (2011-2012) at the time of our review. We asked questions about activities the R & D Centers undertake to disseminate relevant research to the field and what qualitative and quantitative data they collect on their activities. We conducted pretests to check that (1) the questions were clear and unambiguous, (2) terminology was used correctly, (3) the survey did not place an undue burden on respondents, (4) the information could be feasibly obtained, and (5) the survey was comprehensive and unbiased. We conducted two pretests with R & D Centers, one grantee of the National Center for Education Research and one grantee of the National Center for Special Education Research. We made changes to the survey after both pretests based on the feedback we received. We sent the survey by e-mail in an attached Microsoft Word form on April 2, 2013. Surveys were completed by 17 of 18 R & D Center Directors, for a 94 percent response rate, by May 10, 2013. Because this was not a sample survey, it has no sampling errors. However, the practical difficulties of conducting any survey may introduce errors, commonly referred to as nonsampling errors. For example, difficulties in interpreting a particular question, sources of information available to respondents, or entering data into a database or analyzing them can introduce unwanted variability into the survey results. We took steps in developing the questionnaire, collecting the data, and analyzing them to minimize such nonsampling error. To describe how IES coordinates with other relevant federal research agencies and within the department, we reviewed pertinent documents and statutory requirements for coordination contained in ESRA and compared agency efforts to criteria we previously developed for practices agencies can use to help enhance and sustain interagency collaboration.[Footnote 60] To identify relevant federal research agencies, we obtained recommendations on federal research agencies that conduct education-related research during interviews with knowledgeable stakeholders. We continued to ask for recommendations during stakeholder interviews until we established that the vast majority of recommendations identified two other federal research agencies: the education directorate at the National Science Foundation (NSF) and the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD), an arm of the National Institutes of Health. While stakeholders and officials at IES, NSF, and NICHD noted some key differences between these agencies, such as the types of research funded,[Footnote 61] there was general consensus that there were clear commonalities in the research missions and some of the topics of research funded by all three agencies. As such, we interviewed officials at NSF and NICHD about their processes and procedures as well as their coordination with IES, asking for examples of coordination activities and any potential areas for improvement. Additionally, during our interviews with IES officials, we asked about their coordination activities with NSF and NICHD. To describe IES's coordination within the department, we reviewed agency documents such as those describing the annual evaluation planning process at Education and conducted interviews with IES officials and officials responsible for evaluation and research within the department's Office of Program Evaluation and Policy Development. Finally, we also asked about coordination activities between the REL and Comprehensive Centers programs during our interviews with program staff administering these programs, as well as during our interviews with Directors of these groups, to assess the extent and nature of the coordination between these technical assistance programs. We conducted this performance audit from July 2012 to December 2013 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. [End of section] Appendix II: Comments from the Department of Education: United States Department of Education: The Director: Institute of Education Sciences: 555 New Jersey Ave., NW: Washington, D.C. 20208: November 20, 2013: Mr. George A. Scott: Director: Education, Workforce. and Income Security Issues: Government Accountability Office: 441 G Street, NW: Washington, DC 20548: Dear Mr. Scott: Thank you for the opportunity to comment on the U.S. Government Accountability Office (GAO) draft report, "Education Research: Further Improvements Needed to Ensure Relevance and Assess Dissemination Efforts." The Department of Education (Department) appreciates the work that went into this review and the generally positive assessment of the quality of the work of the Institute of Education Sciences and its effect on the field of education research. Our response to your specific recommendations to the Department follows, and our technical comments on the draft report are enclosed. Recommendation 1: IES should use available data to routinely monitor all stages of its peer review process and identify opportunities to improve the timeliness of its reviews. IES generally concurs in this draft recommendation. We agree that timely reporting of research findings is very important, and we have taken steps to monitor timeliness at key stages of the review process. Indeed, since its creation, IES has worked to shorten the period between the end of data collection and the reporting of results. We are equally committed to accurate and high-quality reporting. Timeliness cannot come at the expense of quality. The time frames reported for the "peer review process" include not only the time required for expert peer review of a document, but also the time required for contractors and project officers to address any weaknesses or shortcomings identified by reviewers. The latter often requires more than one round of revision. We have recorded, and we monitor, detailed information about the time required for peer review (approximately 30 days), but we do not have a comprehensive system for monitoring the time required for every other step in the report production process. We intend to develop such a system, which we believe will help us identify and address problems more easily and quickly. Our goal is not just timely reviews and reports, but also high-quality reports, and we will continue to address any timeliness problems related to the various stages of producing high-quality reports. Recommendation 2: IES should develop a structured process to systematically gather input from policymakers and practitioners and use this input when developing its research agenda. IES generally concurs in this draft recommendation. While IES provides opportunities for appropriate input, we agree that opportunities for practitioner and policymaker input can be expanded further. Historically, our primary vehicle for obtaining such input has been the National Board for Education Sciences. More recently, we have engaged practitioners and policymakers, along with researchers, in technical working groups that we have convened to provide advice and feedback on particular programs or topics of research. In addition, in determining the foci for new research and development centers, we have consulted with other offices of the Department on the research they felt was needed to help meet the goals of the programs and activities they arc implementing. To add to these activities, we will annually solicit practitioner and policymaker input for our research agenda through an invitation posted on our Web site. To ensure that the invitation is widely viewed and to encourage significant feedback, we will issue an IES Newsflash to announce the invitation. Recommendation 3: IES should develop performance measures, including targets and goals that clearly demonstrate results, and that reflect its current programs and all key agency activities, such as the performance of the RELs and its new grant programs supporting researcher-practitioner partnerships. IES generally concurs in this draft recommendation. We take our responsibility to measure and analyze the performance of our in vestments seriously. As part of the Department's strategic planning process, we have developed new performance measures for our knowledge utilization activities, including the What Works Clearinghouse and the Education Resources Information Center, and for the Statewide Longitudinal Data Systems that we support. We are also in the process of reexamining our many existing research, statistics, and assessment measures, on which we have been regularly reporting for over a decade, to ensure that the measures arc aligned with current investments (including, for example, researcher-practitioner partnerships) and provide meaningful information that can be used 10 assess the effectiveness of our activities. We will revise those measures as needed. We arc developing new measures for assessing the activities of the Regional Educational Laboratories. Recommendation 4: In order to identify leading practices and target areas for improvement, IES should assess the effectiveness of REI, and R&D Center dissemination strategies by, for example, collecting consistent data and lessons learned from these groups to inform future dissemination efforts. IES generally concurs in this draft recommendation. We agree that it is important to measure and assess the effectiveness of IES-supported dissemination activities. As we examine our performance measures, we will be looking at how to collect better data on dissemination and giving attention to new measures that help us assess IES-supported dissemination, including that conducted by Regional Educational Laboratories and Research and Development Centers. We also agree with GAO's observation on the difficulty of demonstrating how programs with broad dissemination goals are able to inform others and contribute to meaningful outcomes. In our fiscal year 2014 Request for Applications. IES solicited applications for a Knowledge Utilization Research and Development Center to develop measures of how practitioners use research evidence to guide what they do in schools and classrooms. We anticipate that this center will document "lessons learned" about effective dissemination by the Regional Educational Laboratories and the other Research and Development Centers. It will seek to understand what factors seem to promote or inhibit research use by educators and policymakers, as well as examine what researchers can do to communicate their research in ways that arc more useful to practitioners. Thank you again for the opportunity to comment on the draft report. The Institute remains committed to advancing knowledge and providing usable information to improve education policy and practice. Please let us know if you have any questions or would like to discuss our comments. We look forward to receiving the final report. Sincerely, Signed by: John Q. Easton: Enclosure: [End of section] Appendix III: GAO Contact and Staff Acknowledgments: GAO Contact: George A. Scott, (202) 512-7215 or scottg@gao.gov: Staff Acknowledgments: In addition to the contact named above, Scott Spicer, Assistant Director; Lucas Alvarez, Nora Boretti, and Dana Hopings made significant contributions to this report. Also contributing to this report were Sandra Baxter, Deborah Bland, David Chrisinger, Elizabeth Curda, Helen Desaulniers, Sheila McCoy, Jean McSween, Mimi Nguyen, James Rebbe, and Sarah Veale. [End of section] Related GAO Products: Education Research: Preliminary Observations on the Institute of Education Sciences' Research and Evaluation Efforts. [hyperlink, http://www.gao.gov/products/GAO-13-852T]. Washington, D.C.: September 10, 2013. Managing for Results: Executive Branch Should More Fully Implement the GPRA Modernization Act to Address Pressing Government Challenges. [hyperlink, http://www.gao.gov/products/GAO-13-518]. Washington, D.C.: June 26, 2013. Employment and Training Administration: More Actions Needed to Improve Transparency and Accountability of Its Research Program. [hyperlink, http://www.gao.gov/products/GAO-11-285]. Washington, D.C.: March 15, 2011. Program Evaluation: Experienced Agencies Follow a Similar Model for Prioritizing Research. [hyperlink, http://www.gao.gov/products/GAO-11-176]. Washington, D.C.: January 14, 2011. Department of Education: Improved Dissemination and Timely Product Release Would Enhance the Usefulness of the What Works Clearinghouse. [hyperlink, http://www.gao.gov/products/GAO-10-644]. Washington, D.C.: July 23, 2010. Employment and Training Administration: Increased Authority and Accountability Could Improve Research Program. [hyperlink, http://www.gao.gov/products/GAO-10-243]. Washington, D.C.: January 29, 2010. Program Evaluation: Strategies for Assessing How Information Dissemination Contributes to Agency Goals. [hyperlink, http://www.gao.gov/products/GAO-02-923]. Washington, D.C.: September 30, 2002. Education Research: Education Should Improve Assessments of R&D Centers, Regional Labs, and Comprehensive Centers. [hyperlink, http://www.gao.gov/products/GAO-02-190]. Washington, D.C.: January 24, 2002. [End of section] Footnotes: [1] Pub. L. No. 107-279, Title I, Part A, § 111, 116 Stat. 1940, 1944 (codified at 20 U.S.C. § 9511). [2] According to IES, this amount does not include funds that support certain program evaluations conducted by IES, for example those required and funded by other statutes. [3] For the purposes of this report, we define relevance as the degree to which work (1) provides information that could be used to guide decisions about policies, programs, or practices, (2) contributes new information to the research topic, and (3) builds on key literature or previous research. [4] ESRA defines dissemination as "the communication and transfer of results of scientifically valid research, statistics, and evaluations, in forms that are understandable, easily accessible, and usable, or adaptable for use in, the improvement of educational practice by teachers, administrators, librarians, other practitioners, researchers, parents, policymakers, and the public, through technical assistance, publications, electronic transfer, and other means." Pub. L. No. 107-279, § 102(10) (codified at 20 U.S.C. § 9501(10)). For the purposes of this report, we will use this definition. [5] GAO, Managing for Results: Executive Branch Should More Fully Implement the GPRA Modernization Act to Address Pressing Governance Challenges, [hyperlink, http://www.gao.gov/products/GAO-13-518] (Washington, D.C.: June 26, 2013), Employment and Training Administration: More Actions Needed to Improve Transparency and Accountability of Its Research Program, [hyperlink, http://www.gao.gov/products/GAO-11-285] (Washington, D.C.: Mar. 15, 2011), Results-Oriented Government: Practices That Can Help Enhance and Sustain Collaboration among Federal Agencies, [hyperlink, http://www.gao.gov/products/GAO-06-15] (Washington, D.C.: Oct. 21, 2005), and Standards for Internal Control in the Federal Government, [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1] (Washington, D.C.: November 1999). [6] Pub. L. No. 107-279, § 111 (b) (codified at 20 U.S.C. § 9511(b)). [7] In this report, we may use the term IES to refer to one or more of its four Centers: the National Center for Education Evaluation and Regional Assistance (NCEERA), the National Center for Education Research (NCER), the National Center for Education Statistics (NCES), and the National Center for Special Education Research (NCSER). See table 1 for more information on these Centers and their key activities. [8] Pub. L. No. 107-279, § 133(c)(1) (codified at 20 U.S.C. § 9533(c)(1)). R & D Centers were first established in the 1960s. According to statute, NCER is required to support the following topics of research through national R & D Centers: adult literacy; assessment, standards, and accountability research; early childhood development and education; English language learners research; improving low achieving schools; innovation in education reform; state and local policy; postsecondary education and training; rural education; teacher quality; and reading and literacy. 20 U.S.C. § 9533(c)(2). According to officials, NSCER's authorizing legislation does not detail specific topics for NCSER's R & D Centers, but Centers are required to contribute significantly to the solution of special education problems in the United States. [9] Pub. L. No. 107-279, § 114(f)(3) (codified at 20 U.S.C. § 9514(f)(3)). [10] The Director of the Census and the Commissioner of Labor Statistics are also nonvoting ex officio members of the NBES. [11] Office of Management and Budget, ExpectMore.gov: Institute of Education Sciences Research, accessed August 1, 2013, [hyperlink, http://georgewbush- whitehouse.archives.gov/omb/expectmore/summary/10009008.2007.html]. [12] Single-case designs are studies that involve repeated measurement of a single subject (e.g., a student or a classroom) and regression discontinuity designs compare outcomes for a treatment and control group that are formed based on the results of a preintervention measure. [13] In 2010, we reported that some IES stakeholders said its research standards limit the usefulness of information for practitioners. See GAO, Department of Education: Improved Dissemination and Timely Product Release Would Enhance the Usefulness of the What Works Clearinghouse, [hyperlink, http://www.gao.gov/products/GAO-10-644] (Washington, D.C.: July 23, 2010). [14] Other factors influence IES's ability to initiate and complete research quickly. For example, under the Paperwork Reduction Act, federal agencies that wish to collect data from the public must submit their proposed information collection instrument to OMB for approval. According to officials we spoke to at IES and other federal agencies, this process typically adds months to the timeframes for initiating a government-run study. In 2005, we recommended that Congress consider amending the Paperwork Reduction Act to streamline this review process by eliminating one of its two public comment periods. This statutory change has not been made. See GAO, Paperwork Reduction Act: New Approach May Be Needed to Reduce Government Burden on Public, [hyperlink, http://www.gao.gov/products/GAO-05-424] (Washington, D.C.: May 20, 2005). [15] IES officials noted that, regardless of when funding for these specific programs ends, findings from these evaluations could help to inform education reform activities more broadly. [16] Pub. L. No. 107-279, § 186(c) (codified at 20 U.S.C. § 9576(c)). Scientific peer review is the responsibility of IES's Standards and Review Office. Generally, Standards and Review Office staff recruit independent researchers to serve as external peer reviewers. These individuals (1) have published research in top research journals in the content domains relevant to the study presented in the report, (2) have used similar methodological approaches, or (3) have substantial experience conducting studies similar to the one presented in the report. IES products with limited data analyses or that present new analyses of data limited to cross-tabulations are not required to go through an external peer review panel before IES approves publication. However, these products are still reviewed internally by Standards and Review Office staff before being published. [17] This time period includes time that the report is with IES's Standards and Review Office for review of the original submission as well as for review of revisions and time that the report is with IES's Centers or contractors for revision. [18] [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]. [19] The new contract for the What Works Clearinghouse, which began in February 2013, emphasizes the usability and relevance of the Clearinghouse, for example, by calling for new types of educator- friendly products, continued improvement of the website based on educator feedback, and optimization of the website for tablets and smartphones. [20] According to IES, 14 additional grants for supporting collaboration between researchers and policymakers and practitioners have been awarded since 2009 under the Evaluation of State and Local Education Programs and Policies grant program. [21] ESRA requires that the board have 15 presidentially-appointed members, of which no fewer than 8 must be researchers. Pub. L. No. 107- 279, § 116(c) (codified at 20 U.S.C. § 9516(c)). [22] This framework includes five key elements: (1) agenda setting, (2) selecting research, (3) designing research, (4) conducting research, and (5) disseminating research results. See [hyperlink, http://www.gao.gov/products/GAO-11-285]. For more information on leading practices for research and evaluation planning, see also National Research Council, Rebuilding the Research Capacity at HUD (Washington, D.C.: National Academies Press, 2008) and American Evaluation Association, An Evaluation Roadmap for a More Effective Government (September 2010), available at [hyperlink, http://www.eval.org/d/do/107]. [23] GAO, Employment and Training Administration: Increased Authority and Accountability Could Improve Research Program, [hyperlink, http://www.gao.gov/products/GAO-10-243] (Washington, D.C.: Jan. 29, 2010). [24] See, Government Performance and Results Act of 1993, Pub. L. No. 103-62, 107 Stat. 285 and GPRA Modernization Act of 2010, Pub. L. No. 111-352, 124 Stat. 3866 (2011). See also [hyperlink, http://www.gao.gov/products/GAO-13-518]. [25] See [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1] and GAO, Internal Control Management and Evaluation Tool, [hyperlink, http://www.gao.gov/products/GAO-01-1008G] (Washington, D.C.: August 2001). For additional information about government performance management requirements and our related work, see our web page on leading practices for results-oriented management at [hyperlink, http://www.gao.gov/key_issues/managing_for_results_in_government]. [26] [hyperlink, http://www.gao.gov/products/GAO-13-518]. [27] Pub. L. No. 107-279, §§ 174(f)(2) and 203(f)(1)(B) (codified at 20 U.S.C. §§ 9564(f)(2) and 9602(f)(1)(B)). [28] Comprehensive Centers conduct projects primarily for SEA clients and disseminate their work to these clients and to other Comprehensive Centers for their use. In 2011, IES issued an evaluation of the most recently completed group of Comprehensive Centers, including an assessment of the quality and relevance of their work for their state clients; therefore, we did not evaluate the relevance of the Comprehensive Centers' products. [29] IES officials told us, however, that under the most recent REL contract, interim REL research findings can be released to the members of the research alliances to inform discussion among the members. Officials said they hope this will help IES contribute to practice in a timelier manner, though this preliminary information is not available to those outside of the research alliance. [30] According to IES, among the 10 RELs operating during the contract period between 2006 and 2011, 2 had not produced any impact studies after about 5 years of operation. However, officials noted that by the end of the contract period, all RELs had produced at least one impact study and many descriptive reports. [31] GAO, Tax Administration: IRS Needs to Further Refine Its Tax Filing Season Performance Measures, [hyperlink, http://www.gao.gov/products/GAO-03-143] (Washington, D.C.: Nov. 22, 2002). [32] GAO, Program Evaluation: Strategies for Assessing How Information Dissemination Contributes to Agency Goals, [hyperlink, http://www.gao.gov/products/GAO-02-923] (Washington, D.C.: Sept. 30, 2002). [33] This evaluation was mandated by ESRA and was required to be carried out in the third year of the RELs' contracts. See, Pub. L. No. 107-279, § 174(j) (codified at 20 U.S.C. § 9564(j)). [34] According to IES, the customer satisfaction data will be included in the final evaluation report which is estimated to be released in the spring of 2014. [35] U.S. Department of Health and Human Services, Administration for Children and Families, Office of Planning, Research and Evaluation, The Value-Added Research Dissemination Framework (Washington, D.C.: February 2013). Deborah Carpenter, Veronica Nieva, Tarek Albaghal, and Joann Sorra, "Development of a Planning Tool to Guide Research Dissemination", in Advances in Patient Safety: From Research to Implementation (Volume 4: Programs, Tools, and Products), ed. Kerm Henriksen et al. (Rockville, Md.: Agency for Healthcare Research and Quality, 2005). [36] Pub. L. No. 107-279, § 204 (codified at 20 U.S.C. § 9603). [37] U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, National Evaluation of the Comprehensive Technical Assistance Centers: Final Report (Washington, D.C.: August 2011). The objectives of the evaluation were to (1) describe Center operations; (2) assess the performance of Comprehensive Centers in addressing state needs and priorities; and (3) assess the quality, relevance, and usefulness of assistance provided by the Comprehensive Centers. [38] Institute for Education Sciences, Department of Education and National Science Foundation, Common Guidelines for Education Research and Development (Washington, D.C.: September 2013). [39] In this report we use the term "coordination" broadly to include interagency activities that others have variously defined as "collaboration," "cooperation," "integration," or "networking." [40] [hyperlink, http://www.gao.gov/products/GAO-06-15]. [41] 20 U.S.C. § 9602(f)(2). [42] The current RELs began operating in January 2012, while the current Comprehensive Centers began in fall 2012. [43] See GAO, Program Evaluation: Experienced Agencies Follow a Similar Model for Prioritizing Research, [hyperlink, http://www.gao.gov/products/GAO-11-176] (Washington, D.C.: Jan. 14, 2011). Specifically, the Policy and Programs Studies Service within OPEPD is responsible for program and policy evaluation. Within IES, NCEERA is primarily responsible for conducting evaluations and coordinating with OPEPD. [44] According to Education, national activities funds are program- specific, but multipurpose; they can be used on a variety of activities, including evaluations or technical assistance, but only for the designated program. [45] Education may reserve up to 0.5 percent of program appropriations to conduct comprehensive program evaluations, evaluate the effects and cost efficiencies across federal education programs, and increase the usefulness of evaluations of grant recipients. 20 U.S.C. § 7941(a). ESEA also provides that where funds are authorized to be used for program evaluation under a specific provision, Education may not use the general authority described above to reserve additional funds for evaluation of that program. 20 U.S.C. § 7941(c). [46] Where the amount available for evaluation is insufficient, Education officials told us that the funds are redistributed to program implementation activities. [47] In addition to the limitation contained in ESEA, under a generally applicable provision of federal law, appropriations may only be applied to the purposes for which the appropriations were made, unless otherwise authorized. 31 U.S.C. § 1301(a). [48] GAO, Opportunities to Reduce Potential Duplication in Federal Teacher Quality Programs, [hyperlink, http://www.gao.gov/products/GAO-11-510T] (Washington, D.C.: Apr. 13, 2011). [49] GAO, Teacher Quality: Sustained Coordination among Key Federal Programs Could Enhance State Efforts to Improve Teacher Quality, [hyperlink, http://www.gao.gov/products/GAO-09-593] (Washington, D.C.: July 6, 2009). [50] Specifically, the proposal would authorize the department to reserve not more than 0.5 percent from each appropriation for specified accounts in order to carry out evaluations of any of the programs or activities that are funded under such accounts. Office of Management and Budget, Budget of the U.S Government, Fiscal Year 2014, (Washington, D.C.: Apr. 10, 2013). [51] The Department of Labor Appropriations Act of 2012 allowed Labor to use up to 0.5 percent of funds appropriated under certain major program accounts in order to carry out evaluations through fiscal year 2013. Further, the department was permitted to transfer these funds to its Office of the Chief Evaluation Officer. See Pub. L. No. 112-74, 125 Stat. 786, 1063-64. [52] IES's four Centers are: (1) the National Center for Education Evaluation and Regional Assistance; (2) the National Center for Education Research; (3) the National Center for Education Statistics; and (4) the National Center for Special Education Research. [53] GAO, Employment and Training Administration: More Actions Needed to Improve Transparency and Accountability of Its Research Program, [hyperlink, http://www.gao.gov/products/GAO-11-285] (Washington, D.C.: Mar. 15, 2011). [54] GAO, Managing for Results: Executive Branch Should More Fully Implement the GPRA Modernization Act to Address Pressing Governance Challenges, [hyperlink, http://www.gao.gov/products/GAO-13-518] (Washington, D.C.: June 26, 2013), Internal Control Management and Evaluation Tool, [hyperlink, http://www.gao.gov/products/GAO-01-1008G] (Washington, D.C.: August 2001), and Standards for Internal Control in the Federal Government, [hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1] (Washington, D.C.: November 1999). For additional information about government performance management requirements and our related work, see our web page on leading practices for results-oriented management at [hyperlink, http://www.gao.gov/key_issues/managing_for_results_in_government]. [55] We met with a total of 22 researchers (including individual researchers and research organizations), 7 policymaker organizations, 6 national organizations representing practitioners, and 11 individual teachers and superintendents (some teachers and superintendents were not available to interview but provided written responses to our questions) for a total of 46 interviews. In some cases, stakeholders could fall into multiple categories, but we have only counted each once. Stakeholder interviews generally covered topics related to both our first and second research objectives, though the emphasis differed slightly based on the subject's realm of knowledge. [56] For the interviews with teachers, we were assisted by the National Council of Teachers of English and the National Council of Teachers of Mathematics who contacted individual teachers on our behalf. To identify superintendents for interviews, we were assisted by the American Association of School Administrators. [57] In some cases, we interviewed research organizations that may have been current or former IES grantees or contractors. In these instances, we asked these organizations to disclose their current or former IES grants or contracts and sought their input because of their expertise and substantial knowledge of the field. [58] The AERA Fellows program honors individuals with substantial research accomplishments and exceptional contributions to education research. Fellows are nominated by their peers, selected and recommended by the Fellows Committee, and approved by the AERA Council, the Association's elected governing group. [59] U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, National Evaluation of the Comprehensive Technical Assistance Centers: Final Report (Washington, D.C.: August 2011) and U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Evaluation of the Regional Educational Laboratories: Interim Report (Washington, D.C.: September 2013). [60] GAO, Results-Oriented Government: Practices that Can Help Enhance and Sustain Collaboration among Federal Agencies, [hyperlink, http://www.gao.gov/products/GAO-06-15] (Washington, D.C.: Oct. 21, 2005). [61] According to officials, IES typically funds applied science research while NSF and NICHD typically fund basic science research. [End of section] GAO’s Mission: The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO’s website [hyperlink, http://www.gao.gov]. Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, go to [hyperlink, http://www.gao.gov] and select “E-mail Updates.” Order by Phone: The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s website, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO: Connect with GAO on facebook, flickr, twitter, and YouTube. Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts. Visit GAO on the web at [hyperlink, http://www.gao.gov]. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; E-mail: fraudnet@gao.gov; Automated answering system: (800) 424-5454 or (202) 512-7470. Congressional Relations: Katherine Siggerud, Managing Director, siggerudk@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, DC 20548. Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, DC 20548. [End of document]