This is the accessible text file for GAO report number GAO-12-373 entitled 'School Improvement Grants: Education Should Take Additional Steps to Enhance Accountability for Schools and Contractors' which was released on April 11, 2012. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. United States Government Accountability Office: GAO: Report to Congressional Requesters: April 2012: School Improvement Grants: Education Should Take Additional Steps to Enhance Accountability for Schools and Contractors: GAO-12-373: GAO Highlights: Highlights of GAO-12-373, a report to congressional requesters. Why GAO Did This Study: The School Improvement Grant (SIG) program funds reforms in low performing schools. Congress provided $3.5 billion for SIG in fiscal year 2009, and a total of about $1.6 billion was appropriated in fiscal years 2010-2012. SIG requirements changed significantly in 2010. Many schools receiving SIG funds must now use the funding for specific interventions, such as turning over certain school operations to an outside organization (contractor). GAO examined (1) what, if any, aspects of SIG pose challenges for successful implementation; (2) how Education and state guidance and procedures for screening potential contractors and reviewing contractor performance compare with leading practices; and (3) to what extent Education’s technical assistance and oversight activities are effectively supporting SIG implementation. GAO surveyed SIG directors in all 50 states and the District of Columbia; analyzed Education and state documents; and interviewed officials from 8 states and school districts in those states, SIG contractors, and education experts. What GAO Found: Successful SIG implementation posed a number of challenges. Specifically, state and district officials were challenged to build staff capacity and commitment for reform, facing difficulties such as recruiting and retaining strong staff members. In addition, the SIG requirements to develop teacher evaluations and increase student learning time were difficult to implement quickly and effectively because they required extensive planning and coordination. Furthermore, states sometimes had limited evidence about the performance of SIG schools when making grant renewal decisions. For example, although Education’s guidance identifies meeting annual student achievement goals as a key criterion for making renewal decisions, some states did not receive student achievement data by the time decisions had to be made. States also made decisions through qualitative assessments of schools’ implementation efforts, but such determinations were not always based on extensive interaction with schools or systematic monitoring. Education did not provide written guidance to states about making evidence-based grant renewal decisions after they encountered these challenges. Districts used a significant portion of their SIG funds to hire contractors for a range of services, such as managing school operations and conducting teacher professional development. Leading practices show that screening potential contractors and then reviewing their performance are important for ensuring accountability and quality of results. Education required screening of contractors before contract awards were made. However, Education did not require review of contractors during contract performance, and states varied in whether they ensured that contractors were reviewed during the course of contract performance. Education’s assistance and oversight activities are generally supporting SIG implementation. In our survey, nearly all states reported they were satisfied with Education’s technical assistance, particularly the agency’s SIG guidance and conferences. In addition, many states reported that Education’s guidance was timely. With respect to oversight, Education monitored 12 states in school year (SY) 2010-2011 and found deficiencies in 11 of the 12 states. Education is working with states to correct these deficiencies. For SY 2011-2012, the agency plans to use a risk-based approach to conduct on- site monitoring in 14 additional states. To maximize its oversight resources, Education also plans to conduct some limited monitoring in five additional states in SY 2011-2012. Education officials told us that they plan to monitor the remaining states in SY 2012-2013 and that these states represent a small percentage of SIG funds. What GAO Recommends: GAO recommends that Education (1) provide additional support to states about making evidence-based grant renewal decisions and (2) ensure that contractor performance is reviewed. Education generally supported our first recommendation but disagreed with the second. We modified our recommendation to address some of Education’s concerns. View [hyperlink, http://www.gao.gov/products/GAO-12-373]. To view an e- supplement with more data see GAO-12-370SP. For more information, contact George Scott at (202) 512-7215 or scottg@gao.gov. [End of section] Contents: Letter: Background: Successful SIG Implementation Has Posed a Number of Challenges: States and Districts Screen Contractors Before Selecting Them to Receive SIG Funding, but Inconsistent Approach to Reviewing Performance Reduces Contractor Accountability: Education's Assistance and Oversight Are Supporting SIG Implementation: Conclusion: Recommendations: Agency Comments and Our Evaluation: Appendix I: Objectives, Scope, and Methodology: Appendix II: Comments from the Department of Education: Appendix III: GAO Contact and Staff Acknowledgments: Figures: Figure 1: Percentage of SY 2010-2011 and 2011-2012 School Grantees Using Each Reform Model: Figure 2: State Responses on Whether Inadequate Action by SIG Schools or Districts Was a Reason Schools Were Not Operating Very Differently: Figure 3: Survey Results Regarding the Helpfulness of Education's Technical Assistance Offerings: Abbreviations: Education: U.S. Department of Education: SIG: School Improvement Grant Program: SY: school year: View GAO-12-373 key component: School Improvement Grants: Survey of State Educational Agencies Regarding Grant Implementation [hyperlink, http://www.gao.gov/products/GAO-12-270SP], an e-supplement to GAO-12-373. [End of section] United States Government Accountability Office: Washington, DC 20548: April 11, 2012: The Honorable Thad Cochran: Vice Chairman: Committee on Appropriations: United States Senate: The Honorable Tom Harkin: Chairman: The Honorable Richard C. Shelby: Ranking Member: Subcommittee on Labor, Health and Human Services, Education, and Related Agencies: Committee on Appropriations: United States Senate: The School Improvement Grant (SIG) program is designed to fund significant reforms in low performing schools. Schools that receive SIG funding (SIG schools) receive up to $2 million annually for 3 years to improve student outcomes, such as standardized test scores and graduation rates. While the program--which provides funds to states on a formula basis--was first authorized in fiscal year 2002, it changed significantly in fiscal year 2009. Specifically, Congress greatly increased funding for the program from $125 million in fiscal year 2007 to $3.5 billion in fiscal year 2009. From this amount, $3 billion was provided under the American Recovery and Reinvestment Act of 2009, and schools began spending these funds during school year (SY) 2010-2011.[Footnote 1] Additionally, a total of approximately $1.6 billion was appropriated in fiscal years 2010-2012. In addition to funding increases, the U.S. Department of Education (Education) made major changes to SIG program requirements in 2010. [Footnote 2] As a result of these changes, many schools receiving SIG funding must now use the funding to implement one of four intervention models, each with specific requirements, such as replacing principals or turning over school management or certain school operations to an outside organization (contractor). Also, states are now required to award SIG funds to school districts competitively, rather than by formula. States evaluate grant applications using several criteria, including the proposed intervention model for each school and the district's budget and reform implementation plan, as well as their capacity and commitment to effectively implement the reforms. After grant awards are made, states are also responsible for deciding whether to renew schools' SIG funding for additional years. State renewal decisions are to be based on criteria that can include whether school assessment results meet annual student achievement goals, and whether the school has implemented its chosen intervention model. Education oversees the program by reviewing state grant applications, monitoring implementation and providing technical assistance. The recent changes to the SIG program have raised questions about how grants are being implemented. The Senate Appropriations Committee and its Subcommittee on Labor, Health and Human Services, Education, and Related Agencies requested that we conduct a broad review of the SIG program, and in July 2011 we reported on early implementation issues in six states during the first year of the expanded SIG program. [Footnote 3] That report found that the program's implementation time frames in some cases did not allow schools sufficient time to plan and fully enact reforms, and we recommended that the Secretary of Education consider options to award SIG grants to school districts earlier in the school year. In this review, we addressed the following questions: (1) What, if any, aspects of SIG pose challenges for successful implementation? (2) How do Education and state guidance and procedures for screening potential contractors and reviewing contractor performance compare with leading practices? (3) To what extent are Education's technical assistance and oversight activities effectively supporting SIG implementation? To identify aspects of SIG that pose challenges to successful school turnaround, we administered a survey to the 50 states and the District of Columbia, and received a 100 percent response rate.[Footnote 4] We also reviewed Education documents and interviewed Education officials. We gathered in-depth information from eight states that were selected to represent a range of size, geographic diversity, and the intervention models being used. These states were California, Delaware, Nebraska, Nevada, Ohio, Rhode Island, Texas, and Virginia. In each of the states, we also interviewed school district and school officials from one to three districts that had SIG schools. To collect information about policies and procedures for screening potential contractors and reviewing contractor performance, we examined Education guidance and state monitoring policies. We also interviewed Education officials and state and local officials from our site visit states. We identified leading practices for screening and reviewing contractors based on our past work. To evaluate Education's technical assistance and oversight efforts, we reviewed relevant documents, such as SIG monitoring protocols, and interviewed relevant Education officials. We also gathered information from states through our survey and state site visits. We interviewed several Education-funded technical assistance providers that served our site visit states and various stakeholders, such as teachers' union officials. For more information about our scope and methodology, see appendix I. We conducted this performance audit from January 2011 to April 2012 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Background: Education's changes to SIG requirements in 2010 have led to new responsibilities for the agency, states, and school districts. These entities all play key roles in the SIG award and implementation process, with Education supporting and overseeing state SIG efforts. Before awarding formula grants to states, Education reviews each state's application and approves the state's proposed process for competitively awarding SIG grants and monitoring implementation. As part of the state application process, states identify and prioritize eligible schools into three tiers: * Tier I schools. Receive priority for SIG funding and are the state's lowest-achieving 5 percent of Title I schools (or 5 lowest-achieving schools, whichever number is greater) in improvement status.[Footnote 5] * Tier II schools. Secondary schools eligible for, but not receiving, Title I funds with equivalently poor performance as Tier I schools. * Tier III schools. Title I schools in improvement status that are not Tier I or Tier II schools. After states receive SIG funding, school districts submit applications to states describing their SIG reform plans for eligible schools. Education has required that districts base their plans on an analysis of each school's needs, called a needs assessment. After reviewing district applications, states distribute their SIG dollars using their approved competitive grant award process, giving priority to districts seeking funding for Tier I and Tier II schools. Education's regulations and guidance require districts receiving SIG awards for Tier I or Tier II schools to implement one of four intervention models in each school. Select aspects of each model are as follows: * Transformation. Transformation schools must replace the principal, implement a transparent and equitable teacher and principal evaluation system that incorporates student academic growth, identify and reward staff who are increasing student outcomes, and provide increased student learning time, among other requirements. * Turnaround. In addition to implementing many requirements of the transformation model, turnaround schools must use locally adopted: * competencies to screen existing staff and rehire no more than 50 percent of the existing staff. * Restart. The district must reopen the school under the management of a contractor, such as a charter school operator, charter management organization, or education management organization. * Closure. The district must close the school and enroll its students in a higher achieving school within a reasonable proximity. Districts may choose to use contractors to implement aspects of their reform plans. Schools enacting a restart model are required to contract with an organization that will assume many of the decision- making and leadership functions in that school. Districts employing other models may also contract with external organizations for services that could include data analysis, teacher professional development, and efforts to create safe school environments. Our work notes the importance of screening potential contractors before awarding contracts, as well as regular evaluation in order to ensure contractors are providing timely and quality services with government funds.[Footnote 6] In addition to reviewing district applications, states are also responsible for monitoring grant implementation. States make decisions about whether to renew funding for each SIG school for an additional year, based on factors such as whether schools meet annual student achievement goals that districts set for the schools. Pursuant to Education's guidance, if a school meets its annual goals, the state must renew the school's SIG grant. If a school does not meet one or more annual goals, Education's guidance gives states the flexibility to consider other factors such as the "fidelity with which the school is implementing" its chosen intervention model.[Footnote 7] Education provides states with technical assistance and oversight regarding SIG implementation. For example, Education funds 21 Comprehensive Centers that help build states' capacity to assist school districts and schools. Sixteen of these organizations serve states in designated regions, and 5 provide technical assistance on specific issues, such as teacher quality. In addition, Education funds Regional Educational Laboratories, a network of 10 laboratories that serve designated regions by providing access to applied education research and development projects, studies, and other related technical assistance activities. Education also monitors states' implementation of SIG. This monitoring process consists of visits to selected states and several SIG districts and schools within the monitored states, followed by reports documenting any findings. States have an opportunity to respond to any findings before the release of Education's monitoring reports. States have awarded funding to two cohorts of schools since the program was modified and expanded in 2010. In the first cohort, 867 schools received SIG funding to implement one of the four intervention models in SY 2010-2011, and in the second cohort 488 schools received funding to implement one of the intervention models in SY 2011-2012. Seven states[Footnote 8] have received waivers from Education to delay awarding funding to their second cohort of schools until SY 2012-2013 because of various issues, such as turnover of key staff in state educational agencies. The proportion of schools choosing each model was similar in both cohorts and, as shown in figure 1, most schools chose to implement the transformation model. Figure 1: Percentage of SY 2010-2011 and 2011-2012 School Grantees Using Each Reform Model: [Refer to PDF for image: pie-chart] Transformation: 74%; Turnaround: 20%; Restart: 5%; Closure: 1%. Source: GAO survey of states. [End of figure] Successful SIG Implementation Has Posed a Number of Challenges: Staff Capacity and Commitment to Reform Posed Implementation Challenges: Although most states have increased the amount of staff time devoted to SIG since the program was expanded, some states have struggled to develop the necessary staff capacity to successfully support and oversee SIG implementation because of budget constraints. In our survey, 29 states told us that they have increased the staff time devoted to SIG since they first applied for SIG funds in the expanded SIG program. However, officials from four of the eight states we visited--California, Nebraska, Rhode Island, and Texas--told us that because of budgetary constraints, the time staff could devote to administering the SIG program and monitoring district implementation was significantly limited. For example, officials in California said that as a result of the state budget crisis, the state legislature reduced the amount of SIG funds available for state administration from the allowable 5 percent[Footnote 9] to 0.5 percent, limiting the number of staff available to administer the program and monitor districts. Several state officials we spoke with also reported that their existing workloads made it difficult to focus on SIG. For example, in several states the program was administered by officials who also had responsibilities for other major education programs, such as Race to the Top.[Footnote 10] In addition, state officials sometimes did not have expertise in supporting school turnaround efforts. Officials from Education and several states and research groups told us that SIG required states to support local reform efforts to a much greater extent than they had in the past, and staff in some states had not yet developed the knowledge base to fulfill these responsibilities. Even when states were able to develop expertise in school reform and hire necessary staff, officials from Education told us that personnel turnover in many states made it difficult to retain such knowledge. For example, Rhode Island officials said they encountered difficulties filling vacated positions because many nearby states were also recruiting from the same small pool of qualified applicants. Several states increased their capacity through actions such as contracting with nationally recognized experts to help them run their grant competitions, establishing school turnaround offices, or hiring turnaround specialists that regularly handled an individualized caseload of SIG schools. For example, 18 states created new turnaround offices to help districts implement SIG, according to our survey. In addition to these state level issues, many districts also struggled to develop the necessary staff capacity to implement successful school reforms. It was particularly difficult for schools to recruit and retain qualified staff members, according to many stakeholders, including officials from several states and districts we visited. They told us that SIG schools were sometimes in rural areas or needed staff to have expertise that was in short supply, such as experience with reform or specialized academic subjects. Among the 12 states that Education monitored during SY 2010-2011, they found that 4 states did not ensure that turnaround schools met requirements to remove at least half of the schools' staff and hire new staff for those positions based on staff effectiveness.[Footnote 11] For example, Education found that one monitored district in Minnesota did not base hiring decisions on prospective teachers' instructional effectiveness. In addition, Education found that three of the monitored states did not ensure appropriate replacement of the principal in turnaround or transformation schools. Moreover, some districts did not have staff with expertise in using performance and evaluation data--such as data on student performance-- to inform plans for reforming schools and ongoing instructional improvements. Education officials said that, in many cases, school district staff were able to collect data, but did not have experience linking data to needed interventions. In addition, our review of the needs assessments districts were required to develop when planning SIG interventions showed that some were more extensive than others. Also, in one district we visited, the new teacher evaluation process did not include state assessment data on student achievement as one of the evaluation criteria, as required by Education. Several states, districts, and researchers identified promising practices for recruiting and retaining staff or improving data usage, such as developing "grow-your-own" leadership programs, conducting priority hiring for SIG schools, or hiring data coaches to help teachers collect and analyze student data. Districts also varied in their commitment to use SIG funds to enact major reforms. According to our survey, 35 of 51 states awarded grants to all or most Tier I applicants who applied for grants starting in SY 2010-2011, but several officials from states we visited and research organizations reported that some districts receiving SIG grants were not prepared to make significant reforms. For example, officials in one large school district we visited told us they followed turnaround model requirements to rehire no more than 50 percent of teachers at a SIG school. However, the district officials said they relocated the released teachers to other SIG schools in their district because those schools had almost all of the vacancies. Similarly, in two states we visited, district officials moved a school's previous principal into another leadership position on site so that person could continue to work in the school even after a new principal was assigned. State and district officials also cited instances where districts chose their SIG model for reasons other than its likelihood of improving student success. For example, the superintendent in one district told us they chose the restart model because they considered it less restrictive than other models. Although many states responding to our survey told us that all or most of their transformation model schools were operating very differently after the first year of SIG, 33 states said that at least some of these schools choosing the transformation model were not.[Footnote 12] Figure 2 shows responses from these 33 states about whether inadequate action by SIG schools or districts was a reason the schools were not operating very differently. Figure 2: State Responses on Whether Inadequate Action by SIG Schools or Districts Was a Reason Schools Were Not Operating Very Differently: [Refer to PDF for image: pie-chart] Reason in none of schools: 1; Too early to tell or missing: 2; Reason in all schools: 4; Reason in around half of schools: 4; Reason in most schools: 9; Reason in few schools: 13. Source: GAO survey of states. [End of figure] Key SIG Requirements Are Difficult to Implement, Especially Under Short Time Frames: SIG requirements to increase student learning time and develop new teacher evaluation systems were particularly challenging for some districts to implement fully, especially given the significant time pressure that we identified as a concern in our previous report on SIG.[Footnote 13] Indeed, Education found districts that were not appropriately implementing increased learning time requirements in about half of the states that it monitored during SY 2010-2011 and in both states for which it had completed SY 2011-2012 monitoring reports by February 2012. According to district officials, at least half the districts we spoke with will not have fully implemented new teacher evaluation systems by the end of their second year of SIG. In addition, during its SY 2011-2012 monitoring visit to Iowa, Education found that student growth was not always incorporated in new teacher evaluation systems, as required. Our analysis showed that increased learning time and teacher evaluation requirements were challenging because the planning needed to implement them was complex and time- consuming, and stakeholders, such as unions and parents, were sometimes reluctant to embrace the changes. Some districts struggled to develop increased learning time initiatives that would be sustainable after their 3-year SIG grant ended. More specifically: * Interventions required extensive planning. Effectively implementing increased learning time and teacher evaluations required extensive planning. Several stakeholders stressed the importance of carefully designing increased learning time schedules because, for the intervention to be successful, it must provide quality instruction rather than simply increasing the amount of poor instruction. Officials from several districts said they were unable to fully implement their plans for increased learning time at the beginning of the first year of increased SIG funding because, for example, they first needed to fully analyze their existing schedules and curricula and adapt them to meet SIG requirements. Officials from states and districts we visited often stressed that developing a teacher evaluation system is time-consuming because it requires districts to accurately and comprehensively identify, collect, and analyze information about teachers' performance and students' academic growth. In response to the challenges involved in planning and implementing teacher evaluation systems, Education allowed states to apply for a waiver to extend the planning period for this requirement, and 27 states applied for and received the waiver as of February 2012. In its final SIG requirements, Education required schools implementing the transformation model to implement new teacher evaluation systems within the first year of the grant. Districts in states receiving these waivers must develop their evaluation systems during SY 2011- 2012; pilot or fully implement them by SY 2012-2013; and use them to make decisions about retention, promotion, and compensation by SY 2013- 2014. The timeline is the same regardless of whether the SIG schools in the district are from the first or second SIG cohort. * Stakeholders sometimes reluctant to embrace required changes. Implementation was also delayed or otherwise challenged by concerns from various stakeholder groups. Teachers and teachers' unions were sometimes concerned about increasing student learning time or implementing new teacher evaluations in SIG schools, according to Education, state, and district officials. For example, these officials said unions were concerned about whether teacher evaluation systems that incorporated student academic growth could do so in a manner that would not penalize teachers working with the most challenging students. Such concerns sometimes led to delays in finalizing evaluation systems. In a few cases, officials told us that other stakeholder groups such as parents and school board members were also resistant to SIG requirements. For example, an official in Virginia said that some schools trying to increase learning time had met resistance from parents because students often had jobs or responsibilities at home once the traditional school day was over. * Difficulty designing sustainable approaches for increasing learning time. State and district officials also questioned whether increased learning time initiatives would be sustainable after SIG funds were exhausted. For example, survey respondents from 26 states said the costs of increased learning time were unlikely or very unlikely to be sustainable after the SIG grant ends, compared with 10 states that reported it was likely or very likely to be sustainable.[Footnote 14] Rhode Island officials noted that increased learning time benefits students enrolled in SIG schools during the grant cycle, but state and local financial constraints will make it difficult to sustain the increased learning time for future students. Due in part to these concerns, one of the two districts in the state with SIG schools limited the amount of learning time it added in order to avoid significant cuts in this time after grant funding ends. While many officials stressed the complexity of effectively implementing these requirements, some states and districts that we visited found ways to address the challenges they posed. This was particularly true in districts that had started to plan for and implement similar school reforms prior to applying for SIG funds. Many officials from Education, state, and districts stressed the importance of stakeholder involvement while designing and implementing SIG reforms in order to enhance buy-in and strengthen reform initiatives. In order to increase students' learning time without increasing teacher workloads and salaries, Education officials and researchers told us that a few districts were working with community partners to fund or staff additional learning time or were staggering teachers' schedules so that students would be in class longer but teachers would not. In addition, a few states developed sample teacher evaluation system that met SIG requirements so that districts could use it as a framework for developing their own systems. Some States Had Limited Evidence about SIG Grant Performance When Making Grant Renewal Decisions: States often had limited evidence for making decisions about whether to renew schools' SIG funding. For example, in our survey, officials from 10 states told us that they did not use schools' achievement of annual goals to make grant renewal decisions after SY 2010-2011. According to state officials, at least half the states we interviewed did not have the annual student achievement data available at the time they had to make renewal decisions because assessment results only became available at the end of the summer. Officials from two of these states told us that timely access to annual achievement data will continue to be a problem in future years. In addition, even when these data were available, states frequently chose not to base their decisions on schools' achievement of annual goals. Twenty-three of 44 states responding to our survey question said that, among schools that had their funding renewed, all or most did not meet their annual goals. Several officials from our site visits questioned the usefulness of annual goal data in determining whether progress was made, particularly because districts set their own performance targets. For example, California officials said they did not find annual goals data useful because districts often included generic annual goals in their applications for SIG funding instead of proposing goals based on schools' unique circumstances. Regardless of whether annual goals information was available, states almost always considered "fidelity of implementation"--the extent to which the school is implementing the requirements of its intervention model--when making grant renewal decisions. However, states did not always base decisions about this criterion on extensive information. In our survey, 48 of 51 states identified fidelity of implementation as an important factor in their decision-making process, more than any other factor. Several states we spoke with said that qualitative information about implementation was important for assessing grant progress because the first steps of school reform, such as efforts to change school culture, do not always result in measurable student achievement gains. However, making this assessment can involve a high degree of subjectivity and states' determinations were not always developed based on extensive interaction with schools or systematic monitoring of their implementation efforts. For example, officials in California told us they used fidelity of implementation as their key criterion for making grant renewal decisions, and that the primary method for evaluating this criterion was one telephone conversation with each district at the end of the year. Prior to those conversations, the state had limited interaction with most districts for the purpose of assessing their implementation and was unable to conduct SIG monitoring visits for budgetary reasons. In addition, a Virginia official told us the state used fidelity of implementation for making renewal decisions but would benefit from guidance on how to define and measure it. States were in some cases reluctant to discontinue SIG funding even when information they collected showed that schools were not implementing key requirements with fidelity. Several officials from states we visited said they renewed all schools' SIG funding even if the schools were struggling to fulfill key SIG requirements because tight implementation timeframes made the officials reluctant to eliminate funding after the first year of the grant. In our survey, 21 states reported that half or fewer of their Tier I and Tier II schools were able to implement major aspects of their plan by the beginning of SY 2010-2011, such as extending the school day or having new staff in place. In the 19 cases where these states had made renewal decisions, the state renewed all or most grants. Furthermore, officials in several states we visited identified instances where they chose to renew schools' funding despite significant problems at the district or school level, such as having administrators who were not committed to enacting major reforms or were not ensuring that planned reforms were fully implemented. For example, officials from Nevada said they renewed such grants after the first year because they did not want to negatively impact students and teachers when significant district- level problems were outside their control. Although Education reviewed states' proposed grant renewal procedures through the state SIG application process, the agency did not provide written guidance after grant renewal challenges arose. Education required states to submit their renewal processes for review as part of their SIG applications. Nonetheless, in several state applications we reviewed, descriptions of renewal processes and criteria did not align with the practices the state actually implemented. For example, states that told us they were unable to use annual goals data to make renewal decisions had originally identified these goals as a key renewal criterion in their applications to Education. In its work with states, Education officials told us they found some had difficulty using annual goals data or fidelity of implementation and that the agency provided technical assistance to several states that asked for help. However, agency officials were not aware of how states ultimately addressed these issues, and said the agency has not provided any additional technical assistance on grant renewal. [Footnote 15] States renewed almost all SIG grants at the end of SY 2010-2011, and in some cases imposed conditions on schools for renewal. According to Education, 39 states chose to renew funding to every SIG school in their state. Eleven states and the District of Columbia chose not to renew funding to one or more schools, for a total of 16 nonrenewed schools overall.[Footnote 16] Of these 16 schools, about two thirds were not renewed due to problems with fidelity of implementation. Several states we spoke with chose to renew grants with conditions or required changes. For example, officials in Ohio told us that struggling schools were required to take corrective actions in the second year of the 3-year grant and that their level of success in taking such actions will be a key criterion in future renewal decisions. In addition, New York officials renewed all grants after SY 2010-2011 under the condition that transformation and restart schools would implement state and federal SIG teacher evaluation requirements by December 30, 2011. Once that deadline passed, state officials determined that no districts had met the requirements and suspended all SIG funding until they were met. In February 2012, the state commissioner reinstated funds to half of the SIG districts after determining that the districts had made the necessary changes. In our survey, 23 states reported that at least a few of their SIG schools were required to make major changes to their SIG plans as a condition of having funding renewed. States and Districts Screen Contractors Before Selecting Them to Receive SIG Funding, but Inconsistent Approach to Reviewing Performance Reduces Contractor Accountability: Contractors Provide a Wide Range of Services, and Many Receive Significant Amounts of SIG Funding: Contractors provide a wide range of services with SIG funds, and school districts have often given contractors major roles in schools using the restart, turnaround, and transformation models. Education's guidance identifies a clear role for contractors in schools using the restart model. Specifically, districts must hire a contractor to take over school operations. For example, in the Los Angeles Unified School District, the Partnership for Los Angeles Schools has been given full management authority over five restart schools. In contrast, Education allows districts with schools using the turnaround and transformation models--which include more than 90 percent of schools receiving SIG funds--to use contractors, but does not identify a specific role for them. Most turnaround and transformation schools we visited were working with contractors. Although in some cases turnaround and transformation schools used these contractors for minor tasks, in other cases the contractors played a major role in school operations. For example, in Virginia, the state required schools implementing the turnaround and transformation intervention models to use a contractor for a range of services that could include improving teacher performance, principal and management leadership, or changing school culture. Among the school districts we visited, several planned to spend significant amounts of their SIG grants on hiring contractors. These included districts using the restart, turnaround, and transformation models. For example, a district with one SIG school using the transformation model planned to spend about $450,000 for contractors in one school year. In addition, a district that we visited with three SIG schools planned to spend approximately $1.5 million on contractors over the 3-year period for services that included data analysis and curriculum planning. Our prior work and reports regarding services acquisition have shown the importance of building safeguards into acquisition processes to ensure accountability. These leading practices include screening potential contractors prior to award using a thorough selection process that evaluates their ability to achieve results and the contractors' past performance.[Footnote 17] Once a contractor has been selected, officials should routinely review contractors' work to help ensure they are providing timely and quality services and to help mitigate any contractor performance problems.[Footnote 18] Education and States Required Contractors to Be Screened, but Screening Procedures Varied: Education required and states reported requiring that potential contractors be selected after a thorough screening process. Education required that either states or districts screen contractors prior to contract award to ensure their quality.[Footnote 19] Although Education's guidance does not provide specific criteria for approval, Education requires each state to describe in its state application how it will ensure that school districts screen contractors. Each of the eight states we reviewed required districts to describe their plans to screen contractors in their applications for SIG funding. In addition, states varied in how they approached contractor screening at the state level, either taking an active role in the process or delegating screening responsibilities to districts. According to our survey, 17 of the 51 developed approved lists of contractors from which districts could choose. For example, Virginia officials told us they enacted statewide contracts with four organizations, and strongly encouraged districts to choose one of those four organizations. Ohio officials said they developed a list of approximately 100 state- screened organizations from which districts could choose, but districts were free to use other contractors, provided that they screened those organizations. States that we visited that did not develop a list of approved contractors reported requiring districts to screen contractors. For example, Texas officials told us they required all districts to use a formal competitive process in selecting contractors, which included a process to evaluate contractor proposals, in order to be approved by the state. Education Does Not Require that All Contractors' Performance Be Reviewed, and State Review Approaches Vary: Education's monitoring protocols for the SIG program require the review of contractors in schools using the restart model, but they do not require review of contractors during contract performance for the other school improvement models. Education's protocol for monitoring states' SIG implementation asks whether districts have included accountability measures in the contract for restart schools and also asks for the district's current assessment of the contractor. The protocol does not include a similar question for turnaround and transformation schools.[Footnote 20] States varied in their approaches to the review of contractors, and in some cases reported that they did not require that districts review contractors during contract performance. Among the eight states we spoke with, none assessed districts' plans to review contractors in their SIG applications. In addition, several states reported not having any state-level review requirements. For example, Nebraska state officials said their districts conduct informal reviews of the contractors, but the state does not require reviews or provide districts with a formal process or metrics to assess performance. Similarly, in follow up calls for our state survey, state officials in several states said they do not require districts to review contractor performance and were unaware of whether districts conducted any reviews. In contrast, Nevada officials told us they require districts to add accountability steps for contractors in each phase of work. Inconsistent review of contractors during contract performance reduces states' and districts' ability to ensure that they are receiving the services they have paid for. In our work, one stakeholder told us that in the absence of stronger guidance or oversight, the extent to which contracts include accountability measures is largely dependent on the knowledge and experience of the individual contract manager. Although some district officials in our site visits described efforts to include accountability measures or regular review in the contracts, others indicated that contractors are reviewed informally, if at all. Education's Assistance and Oversight Are Supporting SIG Implementation: Education's Technical Assistance Was Well Received by States: Education's guidance and technical assistance on SIG implementation was well received by nearly all states. In our survey, nearly all states responded favorably about Education's guidance and various technical assistance offerings for SY 2011-2012. Most states reported that Education's guidance and technical assistance were helpful and many reported they were very helpful (see fig. 3). In our survey, we also inquired about the amount and timeliness of guidance provided by Education. Forty-one states reported that Education provided about the right amount of guidance for the second year of SIG. In addition, 33 states responded that in SY 2011-2012, Education's guidance was timely, allowing the state to meet its needs, while 14 states commented that the guidance was not timely. Figure 3: Survey Results Regarding the Helpfulness of Education's Technical Assistance Offerings: [Refer to PDF for image: horizontal bar graph] Technical assistance offering: Guidance documents; Very helpful: 21; Helpful: 26; Neither helpful nor unhelpful: 1; Unhelpful: 1; Very helpful: 1. Technical assistance offering: Conferences; Very helpful: 24; Helpful: 19; Neither helpful nor unhelpful: 4; Unhelpful: 2; Very helpful: 1. Technical assistance offering: Webinars or teleconferences; Very helpful: 12; Helpful: 29; Neither helpful nor unhelpful: 4; Unhelpful: 3; Very helpful: 1. Technical assistance offering: 1-on-1 technical assistance; Very helpful: 20; Helpful: 19; Neither helpful nor unhelpful: 5; Unhelpful: 1; Very helpful: 1. Technical assistance offering: Comprehensive centers; Very helpful: 18; Helpful: 19; Neither helpful nor unhelpful: 6; Unhelpful: 0; Very helpful: 0. Technical assistance offering: Newsletters/emails; Very helpful: 5; Helpful: 30; Neither helpful nor unhelpful: 7; Unhelpful: 2; Very helpful: 1. Technical assistance offering: Website; Very helpful: 3; Helpful: 30; Neither helpful nor unhelpful: 11; Unhelpful: 2; Very helpful: 1. Technical assistance offering: Regional education laboratories. Very helpful: 9; Helpful: 20; Neither helpful nor unhelpful: 6; Unhelpful: 0; Very helpful: 1. Source: GAO survey of states. Note: The data for this graphic is from our survey, in which we asked states about the helpfulness of specific technical assistance offerings. In addition to the answers presented, we also included "do not know" as a potential response. We also asked states about the usefulness of Education's communities of practice, the implementation initiative, and the on-site monitoring visit. For these three items, the number of states responding "do not know" was 23, 30, and 30, respectively. Because the "do not know" responses were numerous, we did not include them in our graphic. [End of figure] Although most states told us that Education's guidance was helpful, some identified additional technical assistance that would assist with SIG implementation. In an open-ended question on our survey that asked about the types of additional guidance that Education could provide, 15 states indicated they wanted additional information about other states' SIG implementation efforts that are working well. Several states that we met with also mentioned wanting more information on successful and sustainable implementation strategies, proven contractors, increased learning time strategies, and teacher/principal evaluation systems. To provide additional support and enhance information sharing among the states, Education has recently begun three new assistance efforts. First, Education selected nine states to participate in the SIG "implementation support initiative" as an optional technical assistance resource.[Footnote 21] Under this initiative, each participating state receives a visit from an Education representative as well as officials from the eight other participating states. These site visits have two purposes--first, to provide technical assistance to the states, and second, to enable states to engage in peer-to-peer information sharing. Education reported that they have used information from these site visits to produce targeted technical assistance reports. Second, in December 2011, Education began conducting monthly check-in calls with state officials to better manage SIG implementation. Each state was assigned an Education program officer responsible for providing oversight and technical assistance support, including outreach and monthly check-in calls. Lastly, Education launched the School Turnaround Learning Community-- an on-line forum to provide states and districts with access to resources and to facilitate networking. According to Education, this initiative offers research-based practices and practical examples from states, districts and schools for developing and implementing SIG. Education is Now Using a Risk-Based Approach to Monitor All States within 3 Years: Education's oversight strategy is to monitor all states during the 3- year period[Footnote 22]--starting with SY 2010-2011--in which the first cohort of schools will receive SIG funding. In selecting states for on-site monitoring for SY 2010-2011, Education did not use a SIG- specific risk-based approach and instead used the existing Title I monitoring schedule.[Footnote 23] However, due to resource constraints, Education suspended its Title I monitoring and instead focused exclusively on SIG monitoring. Education also delayed SY 2010- 2011 monitoring to allow states and districts time to implement SIG before beginning monitoring in February 2011. For SY 2010-2011, Education conducted on-site monitoring in 12 states[Footnote 24] uncovering 28 deficiencies. At least one deficiency was identified in 11 of the 12 monitored states, with California and Pennsylvania having the most deficiencies with seven and five, respectively. Half of the monitored states had deficiencies in ensuring appropriate district implementation of the increased learning time requirement. In addition, two states did not ensure that all SIG funds were used consistent with the SIG requirements.[Footnote 25] In SY 2011-2012, Education selected states with a risk based approach tailored for SIG based on factors such as the size of a state's SIG grant. For SY 2011-2012, Education officials initially selected 12 states to conduct on-site monitoring.[Footnote 26] As of February 2012, Education had issued SY 2011-2012 monitoring reports for Iowa and Florida, containing seven and two deficiencies respectively. For example, in Iowa, Education found that funds were not used consistently with SIG grant requirements nor was the state monitoring SIG as written in its approved SIG application. Education also set aside a portion of its oversight resources so that additional states could be selected for monitoring as more information became available. As of February 2012, Colorado and South Carolina were also selected to receive an on-site review. To maximize its oversight resources, Education plans to conduct some limited "desk monitoring" in five additional states in SY 2011-2012.[Footnote 27] According to Education, the desk monitoring protocol is similar to the on-site visit protocol, but--unlike the on-site monitoring--does not include interviews with school officials. Finally, Education officials told us that they plan to monitor the remaining states in SY 2012-2013, and that these states represent a small percentage of SIG funds. Conclusion: Dramatic funding increases in a short period of time--such as those made to SIG--can subject federal programs to considerable financial risk. While states and school districts carry a large share of the responsibility for planning and implementing successful SIG reforms, Education also plays a critical role in supporting these efforts and mitigating risk through strong oversight and accountability. For example, it is important that Education have rigorous processes for reviewing state SIG applications, conducting oversight, and providing technical assistance when needed. The ability to successfully carry out these functions is vital to ensuring the long-term success of the SIG program and protecting taxpayer funds from waste and abuse. Although SIG has been challenging to implement, in part due to the short implementation timeframes we highlighted in our July 2011 report, Education has reviewed state SIG applications, distributed funds to states, begun its monitoring activities, and provided technical assistance. However, the agency's guidance in some cases has not been sufficient to ensure that schools and contractors are fully accountable. For example, given the implementation issues we and Education's monitoring have found, it is critical that states have rigorous SIG grant renewal procedures in place to identify schools that are not making progress. Education has provided limited guidance to states about how to make renewal decisions. Some states are using highly subjective review processes to renew nearly all grants, often without key information on SIG schools' performance. Until Education provides additional support about how states should make evidence- based renewal decisions when, for example, state assessment results are received too late to be factored into these decisions, schools that are not making progress may continue receiving SIG funds. In addition, although contractors are receiving large amounts of many schools' SIG funds, Education has not ensured that states or districts review contractor performance during the terms of their contracts. Unless Education takes action to ensure that states or districts review contractor performance, districts may not receive an appropriate level of contractor services for their SIG funds and funds may not be well spent. Recommendations for Executive Action: To ensure that SIG grant renewal decisions serve to hold districts and schools accountable, we recommend that the Secretary of Education provide additional support to states about how to make evidence-based grant renewal decisions, particularly when states do not have annual student achievement goal information available at the time renewal decisions are made. To ensure that contractors hired with SIG funding are accountable for their performance, we recommend that the Secretary of Education take steps to ensure that the performance of SIG funded contractors, including those in turnaround and transformation schools, is reviewed during contract performance. In developing such requirements and to ensure that those reviews are targeted to contractors receiving large amounts of SIG funding, Education could consider setting a dollar threshold amount for contracts, above which contractor performance should be reviewed. Agency Comments and Our Evaluation: We provided a draft copy of this report to the Department of Education for review and comment. Education's comments are reproduced in appendix II. Education generally supported our recommendation about SIG grant renewal and outlined how the agency is planning to address this recommendation. Education did not agree with our draft recommendation that it should require states to ensure that the performance of all SIG-funded contractors be reviewed, including contractors in turnaround and transformation schools. In its comments, Education said that it believed that existing provisions and requirements address this issue appropriately. For example, Education cited a federal regulation that requires districts to follow their existing procurement procedures, and noted that districts and states have their own requirements for evaluating contractors to ensure accountability. Education also said that the type of evaluation process needed for a contractor should depend on the contractor's role, and that contractors used by schools implementing the turnaround or transformation models may be working on small, discrete projects and may require less provider-specific reviews than contractors in schools implementing the restart model. We agree with Education that the need for performance reviews should be dependent on the specific role of the contractor, and we modified our recommendation to address some of Education's concerns. Specifically, Education may wish to create a dollar threshold above which performance reviews are required. We continue to believe, however, that the current monitoring framework is inadequate. As noted in our report, schools implementing the turnaround and transformation models account for the overwhelming majority of SIG schools, and contractors operating in these schools are performing a range of functions, including some that are large or complex. In our view, there is a need for additional steps to ensure adequate review of contractor performance. Furthermore, our work shows that as a practical matter, states varied in their approaches to contractor review, with some imposing no requirements on districts. Education says that it will clarify in existing guidance the requirement for SIG recipients to follow state and local procurement procedures. Education could use this opportunity to implement our recommendation through additional guidance on contractor performance reviews. In addition, Education implied that our report was based only on the first year of SIG implementation. This is inaccurate. We also conducted interviews with all eight states, reviewed SIG documents, received finalized survey responses, and interviewed Education officials several times during the second year of implementation, thereby enabling us to reflect activities beyond the first year. Based on the number and significance of deficiencies identified in Education's SIG monitoring reports--including some completed during the SY 2011-2012--as well as our own findings, we continue to believe that Education should take additional steps to increase program accountability. Education also provided technical comments that we have incorporated into the report as appropriate. We are sending copies of this report to relevant congressional committees, the Secretary of Education, and other interested parties. In addition, this report will be available at no charge on GAO's website at [hyperlink, http://www.gao.gov]. If you or your staff have any questions about this report, please contact me at (202) 512-7215 or scottg@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. Signed by: George A. Scott, Director: Education, Workforce, and Income Security Issues: [End of section] Appendix I: Objectives, Scope, and Methodology: This study's objectives were to answer the following questions: (1) What, if any, aspects of the School Improvement Grant (SIG) program pose challenges to successful implementation? (2) How do U.S. Department of Education (Education) and state guidance and procedures for screening potential contractors and reviewing contractor performance compare with leading practices? (3) To what extent are Education's oversight and technical assistance activities effectively supporting SIG implementation? To meet these objectives, we used a variety of methods, including document reviews of Education and state documents; a web-based survey of the 50 states and the District of Columbia; interviews with Education officials and stakeholders; site visits to and teleconferences with 8 states; and a review of the relevant federal laws, regulations, and guidance. The survey we used was reviewed by Education and several external reviewers, and we incorporated their comments as appropriate. We conducted this performance audit from January 2011 through April 2012 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Providing Information on Challenges to Successful SIG Implementation: To identify aspects of the SIG program that pose challenges for successful SIG implementation, we analyzed responses to our survey of state educational agency officials with responsibility for SIG in the 50 states and the District of Columbia. The wed-based survey was in the field from August to October 2011. In the survey, we asked states to provide information on challenges they faced in implementing the SIG program and on other aspects of the program, such as SIG grant renewal. We received responses from all 50 states and the District of Columbia, for a 100 percent response rate. We reviewed state responses and followed up by telephone and e-mail with select states for additional clarification and context. Nonsampling error could affect data quality. Nonsampling error includes variations in how respondents interpret questions, respondents' willingness to offer accurate responses, and data collection and processing errors. We included steps in developing the survey, and collecting and analyzing survey data to minimize such nonsampling error. In developing the web survey, we pretested draft versions of the instrument with state officials in various states to check the clarity of the questions and the flow and layout of the survey. Education officials also reviewed the draft survey and provided comments. On the basis of the pretests and reviews, we made minor revisions of the survey. Using a web-based survey also helped remove error in our data collection effort. By allowing state SIG directors to enter their responses directly into an electronic instrument, this method automatically created a record for each SIG director in a data file and eliminated the errors (and costs) associated with a manual data entry process. In addition, the program used to analyze the survey data was independently verified to ensure the accuracy of this work. Detailed survey results are available at GAO-12-370SP. We also conducted site visits to and teleconferences with eight states--California, Delaware, Nebraska, Nevada, Ohio, Rhode Island, Texas, and Virginia--that reflect a range of population size, number of SIG schools, and use of the four SIG intervention models. In each state, we interviewed state officials, as well as district or school officials from one to three districts that had Tier I or Tier II SIG schools. Districts were selected in consultation with state officials to cover heavily and sparsely populated areas, and a variety of SIG intervention models. We also reviewed documents, such as state and district applications for SIG funds, and the relevant federal laws, regulations and guidance.[Footnote 28] We interviewed Education officials and stakeholders, such as teachers' union officials from the national and local levels. Identifying How Education and State Guidance on Contractors Compare with Leading Practices: To gather information about state policies and procedures for selecting and overseeing contractors, we analyzed state survey results. Our survey questions included whether states had developed a list of approved contractors, the SIG turnaround models for which they required that districts work with contractors, and whether states reviewed contractor performance. We reviewed Education documents, including SIG guidance, the state application template, and monitoring protocols, and interviewed Education officials responsible for reviewing state applications and providing oversight of states. Further, we reviewed state and district SIG applications from the eight states to identify their selection and review processes for contractors, and proposed contract expenditures. We also spoke with state and local officials about their procedures for selecting and overseeing contractors, as well as with several contractors working with districts we visited. We compared Education and state requirements for selecting and overseeing SIG contractors to leading contracting practices that were identified through collaboration with our contracting experts and review of GAO-09-374[Footnote 29] and GAO- 05-274.[Footnote 30] Describing the Extent to Which Education Has Supported SIG Implementation: To address the extent of Education's support and oversight of SIG implementation, we reviewed Education guidance, summaries of Education assistance, monitoring time frames, monitoring protocols, and monitoring reports from SY 2010-2011. In addition, we analyzed survey results. We asked states to provide information on the federal role in SIG, including their perspectives on technical assistance offered by Education and Education's monitoring process. We also talked with officials from Comprehensive Centers and Regional Educational Laboratories serving several of the eight states we worked with. The technical assistance providers were selected to include those working with large, medium, and small rural states. In addition, we interviewed Education officials in charge of the Comprehensive Centers Program and in charge of SIG monitoring efforts. [End of section] Appendix II: Comments from the Department of Education: United States Department of Education: 400 Maryland Ave., SW: Washington, DC 20202: [hyperlink, http://www.ed.gov] March 23, 2012: Mr. George Scott: Director, Education, Workforce, and Income Security Issues: U.S. Government Accountability Office: 441 G Street, NW: Washington, DC 20548: Dear Mr. Scott: Thank you for the opportunity to comment on the Government Accountability Office's (GAO's) draft report, "School Improvement Grants: Education Should Take Additional Steps to Enhance Accountability for States, Schools and Contractors" (12-373). We appreciate the time that your office has committed to this study and your efforts to gather grantee perspectives on implementation. The School Improvement Grants (SIG) program focuses on tuning around the country 's persistently lowest-achieving schools. It requires States and districts to target rigorous interventions, and funding to support those interventions, at those schools that have continued to fail students year after year. Given the difficulty and complexity of some of these very important tasks, we anticipated that States and districts would encounter various challenges, and we developed extensive guidance and provided technical assistance to help them meet those challenges. While we agree that tackling specific aspects of SIG requirements may not be easy for some districts and schools, it is nonetheless important work that is already showing great promise and early positive results. States and districts have demonstrated courageous leadership in implementing reforms that require them to do things in a different way than they have done before. And districts and their school leaders and teachers across the country are taking on their duties under SIG responsibly and effectively. We are encouraged that most of your survey respondents reported that most or all of their schools implementing the transformation model are operating very differently after the first year of SIG. That means that hundreds of the lowest- performing schools across the country are making dramatic changes under the SIG program. While initially we were not expecting significant increases in achievement gains after the first year of implementation, there are preliminary data from 43 States that show meaningful improvement. To provide some further background on the program, SIG seeks to accelerate achievement in our nation's lowest-performing five percent of schools through far-reaching interventions. Federal grants support school leaders, teachers, unions, and local partners in the community to undertake this challenging work. We are seeing some very encouraging signs of progress in the first year of data from SIG schools, which few experts anticipated. At the national level, the No Child Left Behind law required persistently low-performing schools to take action to improve student learning. But in reality, chronically underperforming schools were required to do little—and for years the Federal government provided little or no money to help support change in these schools. Almost no high schools, for example, were included in the earlier Federal school improvement efforts—even though just 15 percent of our nation's high schools produce half of the nation's 1.2 million dropouts. SIG has impacted over 1,200 schools, each of which received a three- year grant of up to $2 million a school. In the first cohort of schools, 45 percent were high schools. Some of the nation's high schools present some of the toughest challenges. We are still getting in the results of the first year of the SIG program. But preliminary data show that after just one year, the commitment to change is producing dramatic gains in learning in a significant number of schools. There are about 850 schools in the first SIG cohort. We now have preliminary achievement data from 43 States, covering about 700 of those schools in their first year of the program. In year one, roughly one in four schools saw double-digit increases in mathematics proficiency. About one in five schools had double-digit increases in reading proficiency. In roughly 60 percent of SIG schools, the percentage of students who were proficient in mathematics or reading went up in the first year of the program. As encouraging as these increases in academic achievement are, they arc still preliminary. We will need several years of data to confirm a lasting improvement in academic achievement. We are also continuing to gather data on other critical outcomes that matter to assessing student progress, such as changes in graduation rates, dropout rates, discipline, attendance, and other indices. States are taking very seriously their responsibility to award grants and provide effective oversight and support to their districts and schools. We are encouraged that, according to your survey, 29 States increased the amount of staff dedicated to the SIG program since 2009 and 50 States linked each SIG school or district with a designated State support person or team. Moreover, 31 States have visited SIG districts at least once a month for monitoring or technical assistance, and all have been providing technical assistance to their districts through webinars, conferences, and telephone calls. We also agree that States have some varying levels of capacity to implement the SIG program, especially because they have been required to provide new levels of support and oversight for their lowest- performing schools. For this reason, the Department has focused on providing technical assistance to States to help build capacity. In addition to our guidance and application support, we launched the online School Turnaround Learning Community, which now has over 3,000 members, to provide webinars and tools to support implementation of challenging interventions like increased learning time and teacher evaluations. We also initiated a peer-to-peer technical assistance initiative that brings State SIG leads to other States to observe and learn from promising practices in those other States. We are pleased that, as the draft report noted, the Department's technical assistance has been well received by nearly all the States: 44 States reported that the Department's guidance and resources are helpful or very helpful and all 11 States in the peer-to-peer initiative found it helpful or very helpful. In the fall of 2011, the Department also created the new Office of School Turnaround (OST) in the Office of Elementary and Secondary Education to administer the SIG program. OST has focused on increasing formal monitoring, as well as regular communications with SIG grantees through frequent check-in calls. As a result, the Department has more information to address implementation challenges and identify opportunities to target technical assistance and oversight to where they are most needed. Because we understand that States want even more opportunities to learn from each other, we continue to provide those peer-to-peer efforts though additional conferences, the online School Turnaround Learning Community, and a second round of the peer- to-peer technical assistance initiative. Our responses to your specific recommendations for the Department are as follows: Recommendation #1: To ensure that SIG grant renewal decisions serve to hold districts and schools accountable, we recommend that the Secretary of Education provide additional support to states about how to make evidence-based grant renewal decisions, particularly when states do not have annual student achievement goal information available at the time renewal decisions are made. We have no significant objection to Recommendation 1, since the Department is already providing such additional support to help States with these issues. It should be noted, however, that two different types of evidence may inform a State educational agency's (SEA'S) evidence-based decision regarding whether to renew a local educational agency's (LEA's) SIG grant with respect to a particular school or schools: (1) "results-based" evidence, or evidence related to achievement and progress on the SIG leading indicators; and (2) "process-based" evidence, or evidence related to how a school or LEA is implementing a SIG intervention model and whether that implementation is in accordance with the LEA's approved application and the requirements of the SIG program. With respect to "results-based" determinations, the Department recognizes that turning around the nation's lowest-achieving schools, which by definition have experienced year after year of failure, is challenging work that does not always bring immediate results. The Department also recognizes that improvement in academic achievement often may not be evident in one academic year, even though we have seen some early successes in the SIG program. With this in mind, the Department issued final requirements for the SIG program that require SEAs to renew an LEA's SIG grant with respect to all schools that are meeting their achievement goals and making progress on the SIG leading indicators, and that permit SEAs to renew the grants even with respect to schools that are not making their goals. The SIG requirements also permit SEAs to set their own criteria for making renewal decisions with respect to grants for schools that are not meeting their achievement goals and making progress on the leading indicators. Consistent with the Department's view about the time it may take to improve achievement in persistently lowest-achieving schools, many SEAs were reluctant to terminate SIG grants at the end of the first year of implementation for a school's failure to meet its achievement goals and to demonstrate progress on the leading indicators, as the SEAs felt there had not been sufficient time for schools to demonstrate progress in improving student achievement. Additionally, as noted in the GAO draft report, student achievement data were not available in many States at the time renewal decisions were made at the end of the first year, but became available during the second year of implementation. Given State timelines in reporting achievement data, we agree that SEAs would benefit from some additional guidance on how to make "results-based-renewal decisions when they do not have annual achievement data available at the time renewal decisions are made. With respect to "process-based" determinations, under section 440 of the General Education Provisions Act, 20 U.S.C. § l232c, an SEA has authority to take certain enforcement action, including suspending or withholding payments under a Federal program, if an LEA has failed to comply substantially with the Federal requirements applicable to that program. OST has confirmed that, consistent with this authority, at least eight SEAs made "process-based" determinations to decline to renew SIG grants for twelve schools in nine LEAs because the schools were not implementing their SIG interventions with fidelity to the SIG final requirements. OST maintains regular communication with States regarding renewal decisions through monthly check-in calls, reviews of SIG applications, and on-site and desk monitoring. SEAs were not yet making renewal decisions during the first monitoring cycle. However, OST is monitoring renewal decisions during the second monitoring cycle. OST will examine the criteria used by States that have declined to renew one or more SIG awards to determine whether such criteria were appropriate, and will provide additional guidance and/or technical assistance to all States based on those findings. Additionally, OST will update guidance to clarify that SEAs may terminate a grant if a district fails to comply with program requirements, even in the absence of data on annual goals, and provide guidance on using other data, such as data on SIG leading indicators, as part of renewal decisions. Finally, the requirement that fiscal year (FY) 201 I applicants provide a signed assurance that they will use the renewal process identified in the FY 2010 application to determine whether to renew an LEA's grant illustrates the importance of the renewal process. Recommendation #2: To ensure that contractors hired with SIG funding are accountable for their performance, we recommend the Secretary of Education require states to ensure that the performance of all SIG funded contractors, including those in turnaround and transformation schools, is reviewed during contract performance. For the reasons outlined below, we do not agree with this recommendation. This issue is already addressed appropriately and adequately through existing provisions and requirements, and we will continue to monitor these matters and provide further assistance when appropriate to drive continuous improvement. The Department requires that LEAs screen external providers to ensure their quality and has emphasized the importance of this screening process through guidance, a Dear Colleague Letter, and recurring technical assistance efforts. SEAs and LEAs are also required to monitor implementation of the SIG grants and school progress, which includes the performance of SIG contractors. The Department believes, however, that the type of evaluation process necessary for contractors paid with SIG funds depends on the contractors' role in supporting SIG models. For example, the Department has required that LEAs hold external providers under the restart model accountable for 'fleeting the SIG requirements because they are engaged in whole-school reform and have considerable flexibility with respect to the school improvement activities they will undertake. External providers supporting the implementation of a turnaround or transformation model may require less provider-specific monitoring, as they generally work on specific, discrete activities to improve student academic achievement. Further, under 34 C.F.R. § 80.36(a), when procuring services under a SIG grant, an LEA must follow the same policies and procedures it uses for procurements from non-Federal funds. As part of their policies and procedures for procurements from non-Federal funds, districts and States have their own requirements and practices for screening and evaluating external partners and for ensuring appropriate controls and accountability in this context. While we continue to review and monitor this area carefully, it is not always necessary to establish new Federal requirements on top of this type of State and local requirements. The Department does not believe it is necessary or appropriate to review State procurement laws as part of the SIG application process, but it is important for States and districts to follow these laws. OST will clarify through our guidance that States and districts are required, when using SIG funds, to follow their State and local procurement laws and evaluate external providers where required under those laws. Overall, we believe that in a number of aspects, the draft report presents an incomplete and somewhat misleading view of how well SIG is being implemented. It is important to note that the draft report examines the first year of a redesigned program that dramatically alters how States and districts intervene in their lowest-performing schools. While there is always room for some improvements in the early stages of implementation, and the Department is committed to continuous improvement in implementing its programs such as SIG, the draft report presents some information that should be modified and clarified, including the title of the report. As noted in the enclosed technical comments, we believe a more appropriate title would be "Education Should Provide Some Additional Guidance and Assistance to Further Improve School Improvement Grants Implementation by States, Schools and Contractors." Please see the enclosure for a more detailed discussion of this issue and other suggested edits to the report. Thank you again for the opportunity to comment on the draft report. Please let us know if you have any questions or would like to discuss our comments and suggestions. We look forward to receiving the final report. Sincerely, Signed by: Michael Yudin: Acting Assistant Secretary for Elementary and Secondary Education: Enclosure: [End of section] Appendix III: GAO Contact and Staff Acknowledgments: GAO Contact: George Scott at (202) 512-7215 or scottg@gao.gov: Staff Acknowledgments: In addition to the contact named above, the following staff members made important contributions to this report: Elizabeth Sirois, Assistant Director; Scott Spicer, Analyst-in-Charge; Jacques Arsenault; Melissa King; Salvatore Sorbello; and Barbara Steel-Lowney. In addition, Jean McSween, James Rebbe, Tom James, William Woods, and Kathleen Van Gelder provided guidance on the study. [End of section] Footnotes: [1] Pub. L. No. 111-5, 123 Stat. 115, 181. [2] 75 Fed. Reg. 66,363 (Oct. 28, 2010). [3] For more information, see GAO, School Improvement Grants: Early Implementation Under Way, but Reforms Affected by Short Time Frames, [hyperlink, http://www.gao.gov/products/GAO-11-741] (Washington, D.C., July 25, 2011). The six states were: Delaware, Nebraska, Nevada, Rhode Island, Ohio, and Virginia. [4] This report does not contain all the results from the survey. The survey and a more complete tabulation of the results are included in [hyperlink, http://www.gao.gov/products/GAO-12-370SP]. When discussing survey results in this report, the term "states" also includes the District of Columbia. See appendix I for more information about our survey design and implementation. [5] Title I, Part A of the Elementary and Secondary Education Act of 1965, as amended, provides funding to school districts and schools with high concentrations of students from low-income families. Under Title I, states set academic targets and measure schools' progress in meeting them. Schools in improvement status have missed academic targets for at least two consecutive years. The definitions of Tier I and Tier II schools also include high schools that have a graduation rate of less than 60 percent over a number of years. [6] GAO, Contract Management: Opportunities to Improve Surveillance on Department of Defense Service Contracts, [hyperlink, http://www.gao.gov/products/GAO-05-274] (Washington, D.C.: Mar. 17, 2005). [7] We refer to this as fidelity of implementation. Education's guidance does not further define the process or factors states should use when making this determination. [8] The seven states are Alabama, California, Hawaii, Missouri, Rhode Island, Tennessee, and Vermont. In addition, the Bureau of Indian Affairs was granted a waiver. [9] Under federal law, states are allowed to reserve not more than 5 percent of SIG grant funds for administrative costs. 20 U.S.C. § 6303(g)(8). [10] Race to the Top awards education reform grants to select states through a competitive grant process. Funding for the program was first authorized through the American Recovery and Reinvestment Act of 2009. [11] Using locally adopted competencies to measure the effectiveness of staff, schools implementing the turnaround model must screen all existing staff and rehire no more than 50 percent, and select new staff. [12] For each model, our survey asked whether all, most, around half, few, or none of the schools in a state were operating very differently after the first year of SIG implementation. Thirty-three states told us that some (most, around half, or few) transformation schools were operating very differently after the first year, while 13 states told us that all transformation schools were operating very differently and 5 did not provide a response. For more information, about the extent to which schools were operating very differently after the first year of SIG and reasons cited by state officials, see our e-supplement [hyperlink, http://www.gao.gov/products/GAO-12-370SP]. [13] [hyperlink, http://www.gao.gov/products/GAO-11-741]. Both the turnaround model and the transformation model require school districts to provide increased learning time. Increased learning time is defined by Education's guidance as significantly increasing the total number of school hours to include: (1) instruction in core academic subjects; (2) instruction in other subjects and enrichment activities that contribute to a well-rounded education; and (3) teachers to collaborate, plan, and engage in professional development. The transformation model also requires school districts to use "rigorous, transparent, and equitable evaluation systems for teachers and principals that take into account data on student growth as a significant factor." [14] The remaining 15 respondents said that either it was as likely as unlikely that these costs would be sustainable; they did not know; the question was not applicable, or it was too early to tell. [15] Education officials told us they will review state renewal processes through monitoring starting in SY 2011-2012, but that their SY 2010-2011 monitoring took place prior to states making renewal decisions. [16] The Bureau of Indian Affairs also chose not to renew two schools. [17] GAO, Federal Contractors: Better Performance Information Needed to Support Agency Contract Award Decisions. [hyperlink, http://www.gao.gov/products/GAO-09-374], (Washington, D.C.: Apr. 23, 2009). [18] For the purposes of this report, we define review during the course of contract performance as monitoring or quality assurance surveillance, as described in [hyperlink, http://www.gao.gov/products/GAO-05-274]. [19] Education's guidance requires that contractors implementing the restart model must be selected through a "rigorous review process," which permits a district to examine a prospective contractor's plans and strategies, and to ensure that the contractor has the capacity to implement its strategies. The guidance also requires districts to screen all other contractors for quality, stating that this screening serves a similar purpose as the rigorous review process used for restart contractors. [20] Education officials said they included this requirement for restart schools because their regulation requires districts to contract with external management organizations and that the contractors are selected through a rigorous review process. Education officials also said that they include review requirements for restart contractors because those contractors are an essential part of the restart model, and that they have more flexibility than the contractors used in a school implementing a different model. [21] According to Education officials, 17 states volunteered for this initiative, and the 9 states selected to participate were Alabama, Arizona, Arkansas, Connecticut, Louisiana, Michigan, Oklahoma, South Carolina, and Virginia. States that had received an Education SIG monitoring visit or were awarded a Race to the Top grant were not selected to participate. [22] Education officials also told us that nine of these states received technical assistance visits through the implementation initiative. [23] Education monitors how states receiving Title I grants implement and administer the Title I program to determine whether states comply with program requirements. [24] In SY 2010-2011, Education conducted monitoring site-visits to the following states: California, Indiana, Maine, Michigan, Minnesota, Mississippi, Montana, Nebraska, Nevada, Pennsylvania, South Dakota, and Tennessee. Education officials told us that they also conducted pilot monitoring site-visits in Maryland and New Mexico, but these visits were limited and used to test monitoring questions [25] Education required states to take corrective actions to address identified deficiencies. Education officials commented that states must submit evidence to prove that the identified deficiencies have been addressed. Education would then evaluate the strength of the evidence to determine if it adequately addressed the weakness. Education began issuing its monitoring reports in the spring of 2011 and as of March 2012, seven states--Indiana, Maine, Michigan, Mississippi, Nevada, Pennsylvania, and South Dakota--had adequately addressed all of the weaknesses identified from their monitoring report. In addition, two states, Minnesota and Nebraska, have addressed some of the identified weaknesses, but still have other weaknesses to resolve. [26] For SY 2011-2012, Education initially planned to conduct monitoring site visits to the following 12 states: Florida, Georgia, Hawaii, Illinois, Iowa, Missouri, New Mexico, New York, Oregon, Rhode Island, Texas, and Wisconsin. [27] As of February 2012, Education had identified Idaho and Wyoming as states that would receive desk monitoring. Education has completed the desk monitoring for Idaho. [28] This includes Department of Education, Office of Inspector General, School Improvement Grants: Selected States Generally Awarded Funds Only to Eligible Schools, ED-OIG/A05L002 (Washington D.C., March 2012). [29] [hyperlink, http://www.gao.gov/products/GAO-09-374]. [30] [hyperlink, http://www.gao.gov/products/GAO-05-274]. [End of section] GAO’s Mission: The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO’s website [hyperlink, http://www.gao.gov]. Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, go to [hyperlink, http://www.gao.gov] and select “E- mail Updates.” Order by Phone: The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s website, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO: Connect with GAO on facebook, flickr, twitter, and YouTube. Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts. Visit GAO on the web at [hyperlink, http://www.gao.gov]. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; E-mail: fraudnet@gao.gov; Automated answering system: (800) 424-5454 or (202) 512-7470. Congressional Relations: Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-4400 U.S. Government Accountability Office, 441 G Street NW, Room 7125 Washington, DC 20548. Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800 U.S. Government Accountability Office, 441 G Street NW, Room 7149 Washington, DC 20548.