This is the accessible text file for GAO report number GAO-02-393 entitled 'Title I: Education Needs to Monitor States’ Scoring of Assessments' which was released on April 1, 2002. This text file was formatted by the U.S. General Accounting Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. United States General Accounting Office: GAO: Report to the Secretary of Education: April 2002: Title I: Education Needs to Monitor States’ Scoring of Assessments: GAO-02-393: Contents: Letter: Results in Brief: Background: Most States Are Not in Compliance with the 1994 Title I Accountability and Assessment Requirements: States Cited Support and Coordination as Furthering Compliance: Most States are Taking Some Action To Ensure Accurate Scoring, Justification for Exemptions, and Appropriate Accommodations, but Actions are Limited: Conclusions: Recommendation: Agency Comments: Appendix I: Scope and Methodology: Appendix II: Comments from the Department of Education: Tables: Table 1: Accountability and Assessment Requirements under the 1994 and 2001 Reauthorizations of Title I: Table 2: Status of States’ Compliance with 1994 Title I Assessment Requirements as of March 2002: Table 3: Timeline Waiver Status for 30 States in Meeting the 1994 Title I Assessment Requirements as of March 2002: Abbreviations: ESEA: Elementary and Secondary Education Act: GAO: General Accounting Office: LEA: local education agency: [End of section] United States General Accounting Office: Washington, DC 20548: April 1, 2002: The Honorable Roderick R. Paige: The Secretary of Education: Dear Mr. Secretary: Title I of the Elementary and Secondary Education Act (ESEA), the largest source of federal funding for education, will provide states with $10.3 billion in fiscal year 2002 to improve the educational achievement of children at risk. Title I serves about 12.5 million children in all 50 states, the District of Columbia, and Puerto Rico. In response to concerns that Title I funding was not significantly improving the educational achievement of at-risk students, in 1994 Congress mandated major changes to Title I, including changes to how states measure student performance. As part of the 1994 ESEA reauthorization, states were required to adopt or develop challenging curriculum content and performance standards, assessments aligned with content standards, and accountability systems to assess schools’ and districts’ progress in raising student achievement. In exchange for meeting these requirements, states were given increased flexibility in the use of Title I and other federal funds. New legislation that reauthorizes ESEA – The No Child Left Behind Act of 2001 – has since been passed. The new legislation does not reduce or eliminate any of the 1994 requirements. Instead, it augments the assessment and accountability requirements that states must implement and increases the stakes for schools that fail to make adequate progress. This report provides a snapshot of how close states had come to meeting the 1994 requirements when the 2001 requirements were signed into law. In light of the increasingly important role played by the required assessment and accountability measures, we have collaborated with other audit organizations to review aspects of how states are implementing these measures and ensuring that data used to assess schools’ progress in raising student achievement are complete and accurate. This report is one of several on this topic to be issued separately by the various collaborating audit organizations, which include the U.S. Department of Education’s Office of the Inspector General (OIG), the Texas State Auditor’s Office, the Pennsylvania State Auditor’s Office, and the City of Philadelphia Controller’s Office. In this report, we provide GAO’s findings regarding (1) the status of states’ compliance with key 1994 Title I requirements, (2) factors that have helped or hindered states in meeting the requirements, and (3) the actions states are taking to ensure that Title I assessments are scored accurately, exemptions for students with limited English proficiency are justified and students with disabilities are appropriately accommodated during testing according to Title I regulations. To assess states’ progress in implementing important assessment and accountability requirements introduced by Title I, we (1) interviewed Education officials and reviewed Education memoranda and reports regarding states’ compliance[Footnote 1] with the 1994 requirements, (2) surveyed State Education Agency (SEA) Title I directors regarding the challenges faced by states with and without approved assessment systems, and (3) conducted telephone interviews with Title I and other state officials in states that are in compliance with Title I to develop detailed information on strategies for overcoming key barriers. We also conducted similar interviews in some states still working to attain compliance with the 1994 requirements to determine factors that have hindered their progress. We sent surveys to the 50 states, the District of Columbia, and Puerto Rico and obtained 50 responses (96 percent). We conducted our work in accordance with generally accepted government auditing standards between June 2001 and March 2002. (See app. I for specifics on our scope and methodology.) Results in Brief: As of March 2002, 17 states were in compliance with the 1994 Title I assessment requirements; however, 35 states and instrumentalities were not.[Footnote 2] The 1994 legislation required states to be in full compliance with the requirements by January 2001 but allowed the Department of Education to extend that deadline. For states that have not met the requirements, Education distinguishes between states that are near compliance and those that still have a significant amount of work remaining, granting timeline waivers to the former and compliance agreements to the latter. Education has granted timeline waivers to 30 states to give them more time to meet all requirements. According to Education, by January 31, 2004, all of these states will be in compliance and meet all 1994 Title I requirements, none of which are reduced or eliminated by the 2001 legislation. If states fail to meet extended timelines for implementing the 1994 requirements, they are subject to the withholding of some Title I administrative funds. Education has asked the remaining five states to enter into compliance agreements that will establish the final date by which they must meet all requirements. Education has held public hearings in each of these states. Under the law, without a compliance agreement, these states may lose funding. Noncompliant states most frequently have not met the specific requirements to assess all students and to report the data by subgroups of students, but some also have more intractable problems, such as assessments that are not aligned with state standards. Because the majority of states have not met the requirements of the 1994 law, many states may not be well positioned to meet the deadlines for implementing the additional requirements in the 2001 legislation. Title I directors indicated that a state’s ability to meet the 1994 requirements improved when the necessary players—both state leaders and state agency staff—made compliance a priority and coordinated with one another to achieve it; most directors indicated that state compliance was hindered by inadequate funding. Each compliant state that we interviewed said that the state’s governor, legislature, department of education leaders, or business leaders first prompted compliance initiatives, such as establishing a blue ribbon committee to address the issue. According to Title I directors, state leader initiatives that endorsed compliance as a high-priority, the backing of requirements by state and local staff involved in implementation, and technical expertise at the state level were among the types of support that contributed most to states’ compliance. Coordination between staff in different offices and levels of government was another factor identified as important to compliance. Inadequate funding was the most often cited factor that hindered state compliance with requirements, according to our survey of Title I directors. In interviews, the directors said that their investment of time and money in systems of assessment that predated and conflicted with the requirements of the 1994 legislation was an obstacle to compliance. Most of the states reported taking some action to ensure that Title I assessments were scored accurately, that any exemptions for students with limited English proficiency were justified, and students with disabilities were receiving appropriate testing accommodations. Almost all states hire a contractor to score the Title I assessments; however, 16 of these states reported that they did not monitor the scoring done by the contractor. Most of those who did monitor the scoring reported they did so by selecting a sample of answer sheets to compare with the contractor’s results to ensure their accuracy. Others compared school and district test results with the results from previous years to identify large discrepancies that might suggest a problem. Several states have experienced errors in scoring done by contractors. For some states, these errors resulted in the incorrect identification of schools in need of improvement and students in need of additional services. Thirty-three states also reported taking some actions to ensure that any exemptions for students with limited English proficiency were justified and 41 reported actions to ensure students with disabilities received appropriate accommodations during testing. For example, states reported they had developed standards for districts to follow in accommodating these students so that assessments can yield valid measures of their performance. However, states reported few actions that would ensure that these guidelines were being followed. Many states are still developing procedures to ensure that any exemptions are justified and accommodations are appropriate. Education performs compliance reviews of grantee programs and is in the process of redesigning this review process. However, the redesigned reviews do not specifically include the monitoring of states’ actions with regard to contractors’ scoring of assessments. To reduce the potential for undetected errors in test scoring that could have material effects on educational decisions or damage confidence in the test results, we are recommending that Education specifically include monitoring of state actions regarding contractors and scoring provisions in its state compliance reviews. Education agreed to our recommendation in its official agency comments. A copy of the comments is printed in Appendix II. Background: The original Title I legislation was passed in 1965, but the 1994 reauthorization of ESEA mandated fundamental changes to the Title I program. One of the key changes involved the development of state systems of standards and assessments to ensure that students served by Title I were held to the same standards of achievement as all other children. Prior to 1994, some states had already implemented assessment systems, but these tended to be norm-referenced—students’ performance was judged in relation to the performance of other students. The 1994 legislation required assessments that were criterion-based—students’ performance was to be judged against an objective standard.[Footnote 3] Every state applying for Title I funds since 1994 agreed to implement the changes described in the 1994 law and to bring its assessment systems into compliance. States are also required to develop a definition of adequate yearly progress based on the assessments to hold schools accountable for educational progress. To help states that could not meet the proposed 2001 timeline, Education had authority to grant timeline waivers and compliance agreements to states under certain conditions. In its 2001 ESEA reauthorization, Congress increased testing requirements for states as well as the consequences for not improving test scores in schools and did not eliminate any of the requirements of the 1994 legislation. As shown in table 1, the 1994 and 2001 legislative requirements for assessment and accountability concern developing standards for content and performance; measuring improvement; implementing and administering assessments, including assessing students with limited English proficiency; reporting assessment data; and applying consequences for not meeting performance goals. Table 1: Accountability and Assessment Requirements under the 1994 and 2001 Reauthorizations of Title I: Developing standards for content and performance: 1994 requirements: Develop challenging standards for what students should know in math and reading or language arts. In addition, for each of these standards, states should develop performance standards representing three levels: partially proficient, proficient, and advanced. The standards must be the same for all children. If the state does not have standards for all children, it must develop standards for Title I children that incorporate the same skills, knowledge, and performance expected of other children. 2001 requirements: In addition, develop standards for science content by 2005-06. The same standards must be used for all children. Implementing and administering assessments: 1994 requirements: * Develop and implement assessments aligned with the content and performance standards in at least math and reading or language arts. 2001 requirements: * Add assessments aligned with the content and performance standards in science by the 2007-08 school year. These science assessments must be administered at some time in each of the following grade ranges: from grades 3 through 5, 6 through 9, and 10 through 12. 1994 requirements: * Use the same assessment system to measure Title I students as the state uses to measure the performance of all other students. In the absence of a state system, a system that meets Title I requirements must be developed for use in all Title I schools. 2001 requirements: * Use the same assessment system to measure Title I students as the state uses to measure the performance of all other students. If the state provides evidence to the secretary that it lacks authority to adopt a statewide system, it may meet the Title I requirement by adopting an assessment system on a statewide basis and limiting its applicability to Title I students or by ensuring that the Title I local educational agency (LEA) adopts standards and aligned assessments. 1994 requirements: * Include in the assessment system multiple measures of student performance, including measures that assess higher-order thinking skills and understanding. 2001 requirements: * Unchanged. 1994 requirements: * Administer assessments for math and reading in each of the following grade spans: from grades 3 through 5, 6 through 9, and 10 through 12. 2001 requirements: * Administer reading and math tests annually in grades 3 through 8, starting in the 2005-06 school year (in addition to the assessments previously required sometime within grades 10 through 12). States do not have to administer math and reading or language arts tests annually in grades 3 through 8 if Congress does not provide specified amounts of funds to do so, but states have to continue to work on the development of the standards and assessments for those grades. Have students in grades 4 and 8 take the National Assessment for Educational Performance (NAEP) exams in reading and math every other year beginning in 2002-03, as long as the federal government pays for it. 1994 requirements: * Implement controls to ensure the quality of the data collected from the assessments. 2001 requirements: * Unchanged. Including students with limited English proficiency and with disabilities in assessments: 1994 requirements: * Assess students with disabilities and limited English proficiency according to standards for all other students. Provide reasonable adaptations and accommodations for students with disabilities or limited English proficiency, to include testing in the language and form most likely to yield accurate and reliable information on what they know and can do. 2001 requirements: * By 2002-03, annually assess the language proficiency of students with limited English proficiency. Students who have attended a U.S. school for 3 consecutive years must be tested in English unless an individual assessment by the district shows testing in a native language will be more reliable. Reporting data: 1994 requirements: * Report assessment results according to the following: by state, LEA, school, gender, major racial and ethnic groups, English proficiency, migrant status, disability, and economic disadvantage. 2001 requirements: * Unchanged. 1994 requirements: * LEAs must produce for each Title I school a performance profile with disaggregated results and must publicize and disseminate these to teachers, parents, students, and the community. LEAs must also provide individual student reports, including test scores and other information on the attainment of student performance standards. 2001 requirements: * Provide annual information on the test performance of individual students and other indicators included in the state accountability system by 2002-03. Make this annual information available to parents and the public and include data on teacher qualifications. Compare high- and low-poverty schools with respect to the percentage of classes taught by teachers who are “highly qualified,” as defined in the law, and conduct similar analyses for subgroups listed in previous law. Measuring improvement: 1994 requirements: * Use performance standards to establish a benchmark for improvement referred to as “adequate yearly progress.” All LEAs and schools must meet the state’s adequate yearly progress standard, for example, having 90 percent of their students performing at the proficient level in math. LEAs and schools must show continuous progress toward meeting the adequate yearly progress standard. The state defines the level of progress a school or LEA must show. Schools that do not make the required advancement toward the adequate yearly progress standard can face consequences, such as the replacement of the existing staff. 2001 requirements: In addition to showing gains in the academic achievement of the overall school population, schools and districts must show that the following subcategories of students have made gains in their academic achievement: pupils who are economically disadvantaged, have limited English proficiency, are disabled, or belong to a major racial and ethnic group. To demonstrate gains among these subcategories of students, school districts measure their progress against the state’s definition of adequate yearly progress. States have 12 years for all students to perform at the proficient level. Consequences for not meeting the adequate yearly progress standard: 1994 requirements: * LEAs are required to identify for improvement any schools that fail to make adequate yearly progress for 2 consecutive years and to provide technical assistance to help failing schools develop and implement required improvement plans. After a school has failed to meet the adequate yearly progress standard for 3 consecutive years, LEAs must take corrective action to improve the school. 2001 requirements: * New requirements are more specific as to what actions an LEA must take to improve failing schools. Actions are defined for each year the school continues to fail leading up to the 5th year of failure when a school must be restructured by changing to a charter school, replacing school staff, or state takeover of the school administration. The new law also provides that LEAs offer options to children in failing schools. Depending on the number of years a school has been designated for improvement, these options may include going to another public school with transportation paid by the LEA or using Title I funds to pay for a private tutor. Source: P.L. 103-382 and P.L. 107-110. [End of table] Almost all states employ contractors to perform services to help them meet these requirements. Among states that we interviewed, contractors included private companies, universities, nonprofit organizations, and individual consultants. These entities were hired to provide services that may include assessment development, administration, scoring, analysis, and reporting of results. Some of these entities can provide combinations of services to states, such as test development and test scoring. States are responsible for monitoring contractor performance. Congress allowed states to phase in the 1994 ESEA requirements over time, giving states until the beginning of the 2000-01 school year to fully implement them with the possibility of limited extensions. Education is responsible for determining whether or not a state is in compliance with these requirements and is authorized under ESEA, to give states more time to implement the requirements as long as states are making adequate progress toward this goal. States submit evidence to Education showing that their system for assessing students and holding schools accountable meets Title I requirements. Education has contracted with individuals with expertise in assessments and Title I to review this evidence. The experts provide Education with a report on the status of each state regarding the degree to which a state’s system for assessing students meets the requirements and deserves approval. Using this and other information, the Secretary sends each state a decision letter that summarizes the experts’ review and communicates whether a state is in full compliance, in need of a timeline waiver, or more seriously, a compliance agreement. Education may withhold funds if a state does not meet the terms of its compliance agreement. The 1994 legislation was not specific in the amount of administrative funds that could be withheld from states failing to meet negotiated timelines, but the 2001 legislation states that Education must withhold 25 percent of state administrative funds until the state meets the 1994 requirements including the terms of any timeline waivers or compliance agreements. In June 2000, we issued a report on states’ efforts to ensure compliance with key Title I requirements.[Footnote 4] At that time, we expressed concern about the number of states that were not positioned to meet the deadlines in the 1994 law. To increase compliance, we made two recommendations. We recommended that the Department of Education should (1) facilitate among states the exchange of information and best practices regarding the implementation of Title I requirements and (2) implement additional measures to improve research on the effectiveness of different services provided through Title I to improve student outcomes. Education continues to work on the implementation of these recommendations. In addition, we said that Congress should consider requiring that states’ definitions of adequate yearly progress apply to disadvantaged children, as well as to the overall student population. The 2001 legislation does require that states apply adequate yearly progress requirements and report on the results by subgroups, including students in poverty, with disabilities, and with limited English proficiency. Most States Are Not in Compliance with the 1994 Title I Accountability and Assessment Requirements: As of March 2002, 17 states were in compliance with the 1994 Title I assessment requirements; however, 35 were not. (See table 2.) Departmental approval of timeline waivers to give states more time to reach compliance has been granted for 30 states. Education has asked five states to enter into compliance agreements that will establish the final date by which they must be in compliance before losing Title I funding. Among other requirements, states that are not in compliance have most frequently not met the specific requirements to assess all students and break out assessment data by subcategories of students. The 2001 legislation requires states to implement additional assessments through 2008, thus substantially augmenting current assessment requirements. Education has published a notice of proposed negotiated rulemaking in the Federal Register and has solicited comments from outside parties in preparation for establishing state compliance standards for the 2001 legislation. Table 2: Status of States’ Compliance with 1994 Title I Assessment Requirements as of March 2002: Compliant (17): Colorado: Delaware: Indiana: Kansas: Louisiana: Maine: Maryland: Massachusetts: Missouri: North Carolina: Oregon: Pennsylvania: Rhode Island: Texas: Vermont: Virginia: Wyoming. Noncompliant (35): Alabama: Alaska: Arizona: Arkansas: California: Connecticut: District of Columbia: Florida: Georgia: Hawaii: Idaho: Illinois: Iowa: Kentucky: Michigan: Minnesota: Mississippi: Montana: Nebraska: Nevada: New Hampshire: New Jersey: New Mexico: New York: North Dakota: Ohio: Oklahoma: Puerto Rico: South Carolina: South Dakota: Tennessee: Utah: Washington: West Virginia: Wisconsin. Source: Department of Education. [End of table] When Education determines that a state is not in compliance with the 1994 Title I assessment requirements, it may grant the state a timeline waiver for meeting those requirements. A waiver may not exceed 3 years. Education officials indicate that the agency grants waivers to states that have a history of success in implementing significant portions of their assessment systems, have a clear plan with a definite timeline for complying with the Title I requirements, and have the capacity to carry out the plan and thus meet those requirements. When a state requests a waiver, it must provide Education with a plan that includes a timeline for addressing deficiencies in the state’s assessment system. Education reviews this information to decide whether the waiver should be granted and its duration. So far, Education has granted timeline waivers to 30 states. (see table 3.) Table 3: Timeline Waiver Status for 30 States in Meeting the 1994 Title I Assessment Requirements as of March 2002: 1-year (9): Arkansas: Connecticut: Florida: Illinois: Kentucky: New Hampshire: New York: Utah: Washington. 2-year (18): Alaska: Arizona: California: Georgia: Hawaii: Iowa: Michigan: Mississippi: Nebraska: Nevada: New Jersey: New Mexico: North Dakota: Oklahoma: South Carolina: South Dakota: Tennessee: Wisconsin. 3-year (3): Minnesota: Ohio: Puerto Rico. Note: Each timeline waiver has an exact ending date. 1 year = to be complete December 31, 2002 or sooner. 2 year = to be complete December 12, 2003 or sooner. 3 year = to be complete January 31, 2004 or sooner. Source: Department of Education. [End of table] A compliance agreement is deemed necessary when Education determines that a state will not complete the implementation of its assessment system in a timely manner. According to Education officials, a state requiring a compliance agreement generally does not have a history of successful implementation, has not met a significant number of Title I requirements, and does not have a plan in place for meeting those requirements. Education recommends a compliance agreement so that a state may continue to receive Title I funds. Before Education may enter into a compliance agreement, a public hearing must be held in which the state has the burden of persuading Education that full compliance with the Title I requirements is not feasible until a future date. The state must be able to attain compliance within 3 years of the signing of the compliance agreement by the state and Education. The state then negotiates the terms of the agreement with Education. Education’s written findings and the terms of the compliance agreement are published in the Federal Register. A state that enters into a compliance agreement to address requirements of the 1994 Title I law and subsequently fails to meet the requirements of the agreement can be subject to loss of some state Title I administrative funds. Education is presently working on five compliance agreements (Alabama, Idaho, Montana, West Virginia, and the District of Columbia) and has held public hearings for each. The 2001 reauthorization of ESEA was signed into law on January 8, 2002. The act provides states not in compliance with the 1994 Title I requirements at the time of the signing of the 2001 legislation with a 90-day period that started on January 8, 2002 to negotiate changes in the dates by which they must be in compliance with the 1994 requirements. After the conclusion of this 90-day period, the legislation prohibits further extensions for compliance with the 1994 requirements. States failing to meet these negotiated timelines will be subject to loss of some of their Title I administrative funds. According to senior Education officials, this loss could be significant to states, as many use federal program administrative funds to pay the salaries of state department of education staff. A review of documents from Education shows that noncompliant states have most commonly not met two Title I requirements—assessing all students and breaking out assessment data by subcategories of students. Title I does not permit states to exempt any student subgroup from their assessments and Education’s guidance states that individual exemptions may be permitted by the states in extraordinary circumstances. Nonetheless, many states allow substantial exemptions for students with disabilities and limited English proficiency. Several states reported that they have only recently amended laws that prohibited testing of some students with limited English proficiency. Title I also requires states, local districts, and schools to report the performance of students overall and in a variety of subcategories. These categories are gender, race, ethnicity, English proficiency status, migrant status, disability status, and economic disadvantage. Many states disaggregated data for some but not all of these categories. Documents from Education show that data for the disabled, migrant, and economically disadvantaged subcategories are the most common subgroups excluded from state, district, and school reports. In addition, many states lag in other areas, such as aligning assessments to state content standards. To achieve compliance with the 2001 legislation, states will need to add new standards and increase assessment efforts, as detailed in table 1. In responding to our survey, 48 states reported that they have developed content standards in science, but only 16 reported having annual assessments for math and 18 reported annual assessments for reading in all grades 3 through 8. In addition, states will not have the 2 to 3 year timeline waivers available to them as they had when they worked to meet the 1994 requirements. New 2001 requirements listed in table 1 have deadlines that vary according to the requirement, and the Secretary of Education can give states 1 additional year from those deadlines to meet the new requirements, but only in case of a “natural disaster or a precipitous and unforeseen decline in the financial resources of the state.” Since the majority of states have not met the requirements of the 1994 law, it appears that many states may not be well-situated as they work to meet the schedule for implementing new requirements that build upon the 1994 requirements. States Cited Support and Coordination as Furthering Compliance: States successful in meeting key Title I requirements attributed their success primarily to four factors. These factors were (1) the efforts of state leaders to make Title I compliance a priority; (2) coordination between staff of different agencies and levels of government; (3) obtaining buy-in from local administrators, educators, and parents; and (4) the availability of state-level expertise. Survey respondents identified inadequate funding as an obstacle to compliance. The state Title I officials we interviewed said that their states’ commitment of resources to norm-referenced assessments that conflicted with the 1994 Title I requirements contributed to this obstacle. Almost 80 percent of the respondents identified state leaders’ efforts as a factor that facilitated their meeting the 1994 Title I requirements. In every state that had attained compliance with the Title I requirements, the officials that we interviewed said that the governor, legislators, or business leaders made compliance with the Title I requirements a high-priority. States described the development of high-level committees, new state legislation, and other measures to raise the visibility and priority of this issue. For example, one governor spearheaded a plan that used commissions to develop content standards and assessments aligned with those standards. Some state officials we interviewed reported that efforts by state department of education leaders resulted in major organizational changes in the state education department. For example, according to one Title I Director, the state changed the organizational structure and reporting relationships of state offices to organize them by function rather than by funding streams and to enhance coordination; according to another Title I Director, state leaders who did not support changes necessary to achieve compliance with Title I were replaced with staff that did support the changes. In responding to our survey, over 80 percent of the Title I officials identified the ability of staff or agencies to coordinate their efforts with one another as a factor that helped them meet requirements. In our interviews, state officials cited the necessity of coordination between state and local staff working in the areas of assessment, instruction, and procurement. Two of the states we interviewed specifically noted that when the assessment office shared a physical location with the Title I office, coordination was easier and the ability to achieve compliance with Title I was enhanced. Title I and other officials we interviewed in those states that had met the 1994 Title I assessment requirements noted that they had made great efforts to obtain buy-in from other state officials, local administrators, educators, and the public. They said that efforts to ensure buy-in paved the way for changes meant to ensure compliance with the assessment and accountability requirements of the1994 legislation. Several officials we interviewed reported holding public meetings and focus groups to obtain input from parents, teachers and local administrators regarding how the state should implement Title I requirements. They also reported conducting public relations campaigns to educate the public about the importance of complying with Title I requirements for standards-based assessment. One state, for example, conducted 6 years of focus groups and hearings and conducts a conference annually to allow local education officials to gain advice from experts regarding any concerns or problems they are having in implementing Title I requirements. In responding to our survey, over 80 percent of state Title I directors identified the availability of state level expertise as a factor that facilitated their efforts to meet Title I requirements. State officials we interviewed reported that training for teachers and district personnel was often needed to apply new content standards in the classroom and to administer assessments correctly. Two states, for example, used regional centers to educate local staff on assessments and standards. Fifty percent of survey respondents identified inadequate funding as an obstacle in moving toward compliance and noncompliant states cited this problem more often than compliant ones. In our interviews, Title I and assessment officials from noncompliant states reported that progress toward compliance with Title I requirements was stalled because of investments they had made in assessment systems that predated and conflicted with the requirements of the 1994 Title I reauthorization. Respondents said that they had made substantial investments of time and money in systems of assessment that often relied upon norm-referenced assessments and did not meet the 1994 requirement for criterion-based tests. They noted that it took their states several years to change from the old system of assessment to one meeting the requirements specified by the 1994 reauthorizing legislation. According to the officials we interviewed, building support to start again on another system and obtaining the funding made it more difficult to make the necessary changes in a timely manner. In addition, one survey respondent from a very small state noted that due to the state’s size it has a small number of staff and does not have the technical expertise needed to develop a new system, thus hampering the state’s ability to meet the requirements. Most States are Taking Some Action To Ensure Accurate Scoring, Justification for Exemptions, and Appropriate Accommodations, but Actions are Limited: Most states are taking some action to ensure that Title I assessments are scored accurately, that any exemptions for students with limited English proficiency are justified, and that students are receiving appropriate accommodations when these are needed to gather an accurate assessment of their abilities. Most states hire a contractor to score Title I assessments and about two-thirds of these states monitored the scoring performed by the contractor. Some states that hire contractors have found errors in the scoring the contractors did, and in some cases, these errors have had serious negative consequences for schools and students. Most states reported taking some actions to ensure that students with limited English proficiency and disabilities received appropriate accommodations during testing. Education is redesigning its current compliance and monitoring program to better monitor states’ implementation of Title I. Problems Found in Contractor Scoring: According to our survey results, most states (44) hire a contractor for test scoring, but 16 of these states identified no monitoring mechanism to ensure the accuracy of their contractor’s scoring and reporting. Among those states that did report one or more monitoring mechanisms, 15 reported that they monitored the contractor’s scoring by comparing a sample of original student test results to the contractor’s results. A few states also reported, in interviews with us, that they compared their most recent test scores with those from previous years and looked for significant variations that suggested potential errors in scoring. However, in our interviews, some assessment officials indicated that they use this type of monitoring rather informally. The problems identified in assessment scoring suggest that these approaches do not always provide adequate assurance of complete and accurate results. Indeed, several of the states that use contractors to score tests reported that they have had problems with errors in scoring whether or not they had monitoring measures in place. In some cases, contractors marked correct answers as incorrect and in other cases the contractors calculated the scores incorrectly. The errors were discovered by a number of individuals, including local district officials, parents, and state agency staff. These scoring errors had impacts on students, families, and school and district resources. Based on erroneous scores calculated by a contractor, one state sent thousands of children to summer school in the mistaken belief that their performance was poor enough to meet the criterion for summer intervention. In addition to disrupting families’ summer plans and potentially preventing student promotions, this may have drawn resources away from other necessary activities. In another case, based on a contractor’s erroneous scoring, a state incorrectly identified several schools as “in need of improvement,” a designation that carries with it both bad publicity and extra expense, for example, districts may have to fund the needed improvements. A few state officials that we interviewed told us that they have begun instituting processes to check the accuracy of scoring. For example, three states said that they had hired individuals who were experts in test scoring or they hired other third parties to conduct independent audits. States that were in compliance with 1994 Title I assessment requirements generally had more complete monitoring systems, including measures such as technical advisory committees to review results, conduct site visits, and follow a sample of tests through the scoring and reporting process. In contrast, several states indicated they are still relying on contractor self-monitoring to ensure accurate scoring. Although Education is obligated under the Federal Managers’ Financial Integrity Act of 1982[Footnote 5] and the Single Audit Act[Footnote 6] to ensure that states that receive federal funds comply with statutory and regulatory requirements to monitor contractors, it currently takes limited action regarding states’ monitoring of assessment contractors. Education’s inspector general has reported deficiencies in an important vehicle for such oversight - -Education’s compliance reviews of state programs.[Footnote 7] The compliance reviews are conducted on a 4-year cycle and include an on-site visit that lasts 1 week. Specifically, the OIG cited insufficient time to conduct the reviews, lack of knowledge among Education staff about areas they were reviewing, and a lack of consistency in how the reviews were conducted. Senior Education officials told us the department is redesigning the current compliance and monitoring program used for its on-site visits to better focus on outcomes and accountability in Title I and that it is addressing the OIG’s recommendations. However, a senior Education official who is working on the redesign of the compliance reviews told us that the current draft plans did not include specific checks on state monitoring of assessment scoring. Confidence in the accuracy of test scoring is critical to acceptance of the test results’ use in assessing school performance. Some Actions Taken To Ensure Exemptions Justified and Accommodations Appropriate: According to our surveys and interviews, 33 states have taken at least minimal actions to ensure any exemptions for students with limited English proficiency are justified and 41 states take actions to ensure accommodations for students with disabilities are appropriate.[Footnote 8] Most states reported that they had developed standards for districts to follow in accommodating these students so that assessments could yield accurate measures of their performance. However, states reported few actions that would ensure that these guidelines were being followed. For example, 17 states reported that they compare the number of students with limited English proficiency tested within a given year against the number for the previous year. They used this comparison as their means of verifying that the numbers of students receiving exemptions were reasonable. As the pool of students in a particular school can change substantially from year to year, this comparison has obvious limitations. Moreover, students’ status, for example with respect to English proficiency, can change from year to year. Similarly, 37 states reported using an annual comparison of the number of students with disabilities being tested as a check for appropriate accommodations. However, it is not evident how such comparisons would allow states to ascertain the appropriateness of the accommodations. Survey results and interviews did indicate that more states are taking actions to monitor accommodations for students with disabilities than for students with limited English proficiency. For example, while 25 states reported that they had standards for accommodating students with limited English proficiency, 36 had standards for accommodating students with disabilities. The state officials that we interviewed told us that this was because districts built upon steps they had taken under the Individuals with Disabilities Education Act (IDEA)[Footnote 9] to document the accommodations needed by students with disabilities. In general, states said that the districts have more experience and technical expertise for assessing and supporting students with disabilities because of working under IDEA for many years. In contrast, some states lacked consistent standards for identifying students with limited English proficiency and more states were still working to develop alternate assessments or accommodations for these students. Augmenting the 1994 requirements, the new 2001 legislation requires that states annually assess the language proficiency of students with limited English proficiency by the 2002-03 school year. States do conduct cyclical monitoring of the implementation of all their programs that might be used to assess the appropriateness of district policy and practice with regard to testing accommodations. However, in a recent review,[Footnote 10] we found that states varied dramatically in the frequency of their on-site visits. The average time between visits to districts ranged from 2 years or less (6 states) to more than 7 years (17 states). Conclusions: This snapshot of the states’ status with respect to the 1994 Title I requirements suggests that many states may not be well-positioned to meet the requirements added in 2001. Only 17 states were in compliance with the assessment requirements of the 1994 law in March of 2002; therefore, the majority of states will still be working on meeting the 1994 requirements as they begin work toward meeting the new requirements. In addition, despite the enhanced emphasis on assessment results, states still appear to be struggling with ensuring that assessment data are complete and correct. The 1994 and 2001 ESEA reauthorizations raised student assessments to a new level of importance. The assessments are intended to help ensure that all students, including those who have disabilities and those who have limited English proficiency are meeting challenging standards. In addition, assessment results are a key part of the mechanism for holding both schools and states accountable for improving educational performance. Thus, ensuring the completeness and accuracy of assessment data is central to measuring students’ progress and ensuring accountability. Without adequate oversight of assessment scoring, efforts to identify and improve low-performing schools could be hindered by lack of confidence in assessment results or uncertainty regarding whether particular schools have been appropriately identified for improvement. Education’s current monitoring does not include specific oversight of how states ensure the quality of scoring contractors’ work, but Education’s revision of its monitoring process provides the agency with an opportunity to help states ensure that scoring done by contractors is accurate. Recommendation: To enhance confidence in state assessment results, we recommend that when the Department of Education monitors state compliance with federal programs, it include checks for contractor monitoring related to Title I, Part A. Specifically, Education should include in its new compliance reviews a check on the controls states have in place to ensure proper test scoring and the effective implementation of these controls by states. Agency Comments: We provided Education with a draft of this report for review. The Department’s official comments are printed in appendix II. In its comments, Education agreed with our recommendation. Education also provided us with technical comments that we incorporated in the report as appropriate. We are sending copies of this report to appropriate congressional committees and other interested parties. If you have any questions about this report, please contact me on (202) 512-7215 or Betty Ward- Zukerman on (202) 512-2732. Key contributors to this report were Mary Roy, G. Paul Chapman II, Laura Pan Luo, Corinna Nicolaou, and Patrick DiBattista. Sincerely yours, Signed by: Marnie S. Shaul, Director: Education, Workforce, and Income Security Issues: [End of section] Appendix I: Scope and Methodology: We conducted this review in conjunction with our partners in the Domestic Working Group. The Domestic Working Group’s objective is to allow officials in the federal, state, and local governmental audit communities to interact on a personal and informal basis on various topics of mutual concern. The group consists of 18 (6 federal, 6 state, and 6 local) top officials and is intended to complement the work of the intergovernmental audit forums and other professional associations. For this review, the Texas State Auditor's Office conducted a detailed assessment of data quality at the state and local levels in Texas, while the Department of Education's Office of Inspector General did so at the state and local levels in California and conducted additional work on control processes at the Department of Education. In Pennsylvania, the Pennsylvania Department of Auditor General conducted work at the state level and the Philadelphia Controller's Office pursued the same goal within the city of Philadelphia. To complement these efforts, GAO surveyed all states and conducted detailed interviews with several regarding their experiences in implementing major provisions of Title I. Specifically, we reviewed three key questions: (1) the status of states’ compliance with key 1994 Title I assessment requirements; (2) factors that have hindered or helped states move toward meeting the requirements; and (3) the actions states are taking to ensure that Title I assessments are scored accurately, exemptions for students with limited English proficiency are justified, and students with disabilities are accommodated during testing according to federal regulations. We obtained information on the first objective from the Department of Education. We met with Education officials and obtained updated listings of compliance throughout the audit. In addition, we reviewed state decision letters, peer reviews of state assessment systems, and reports completed or commissioned by Education’s Planning and Evaluation Service. To address the second and third questions, we used both a state survey of Title I directors and detailed interviews with state Title I officials and other state officials who played a key role in Title I compliance - often assessment officials and sometimes Special Education, program evaluation, and information technology officials. We sent the survey to all 50 state directors and to the District of Columbia and Puerto Rico. We received 50 completed surveys. We followed up with 19 states to clarify and expand on questions in the interview related to contracting for the scoring of tests. We interviewed officials from 5 states that had assessment systems approved by Education and 3 states that were still trying to attain compliance. We also interviewed two expert reviewers, Education officials with responsibility for Title I and program review, three officials at Education’s regional assistance centers, and officials at the Council of Chief of State School Officers. We coordinated our work and findings with our audit partners, who provided us with information relative to their states’ activities. [End of section] Appendix II: Comments from the Department of Education: United States Department Of Education: Office Of Elementary And Secondary Education: The Assistant Secretary: 600 Independence Ave., S.W. Washington, D.C. 20202-6100: "Our mission is to ensure equal access to education and to promote educational excellence throughout the Nation." Mamie S. Shaul: Director: Education, Workforce and Income Security Issues: U.S. General Accounting Office: Washington, DC 20548: Dear Ms. Shaul: This is in response to your letter to Secretary Paige requesting Department of Education comments on the draft report Title I: Education Needs to Monitor States' Scoring of Assessments (GAO/02-393). Thank you for the opportunity to comment. The draft document recommends that the Department monitor States' actions with respect to their contractors' scoring of assessments. As a matter of clarification, scoring of assessments is not typically a State function. Often, the State enters into a contract with a test developer who, as a part of the contractual agreement, is responsible for scoring. States have the primary responsibility to oversee the work of the test developer and to verify the accuracy of results. Typically, States include scoring in their contracts with test developers, as well as other safeguards related to administration, analysis, and reporting procedures. The Department, as recommended, will include in its monitoring activities a review of a State's compliance in monitoring the technical quality of products delivered by its contractor, and its procedures to ensure accuracy of assessment data prior to dissemination. If I can be of further assistance, please do not hesitate to contact me at (202) 401-0113. Sincerely, Signed by: Susan B. Neuman, Ed.D. [End of section] Footnotes: [1] In this report, we refer to a state as compliant when Education has fully approved its assessment system for meeting the 1994 final assessment requirements. We do not intend to imply that these states are necessarily in compliance with all requirements of the 1994 law. [2] In this report, when we refer to states we will be including the instrumentalities of the District of Columbia and Puerto Rico. [3] A norm-referenced test is an objective test that is standardized on a group of individuals whose performance is evaluated in relation to the performance of others. Criterion-referenced tests are assessments that measure the mastery of specific skills or subject content and focus on the performance of an individual as measured against a standard or criterion rather than the performance of others taking the test. [4] General Accounting Office, “Title I Program: Stronger Accountability Needed for Performance of Disadvantaged Students” [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO/HEHS-00-89], (Washington, D.C.: June 1, 2000). [5] U.S. General Accounting Office, “Standards for Internal Control in the Federal Government” [hyperlink, http://www.gao.gov/cgi- bin/getrpt?GAO/AIMD-00-21.3.1], (Washington, D.C.: Nov. 1999). [6] P.L. 98-502. [7] See “Review of the Office of Elementary and Secondary Education’s Monitoring of Formula Grants: Final Audit Report” (ED-OIG/A04-A0013), November 2001. [8] Accommodations for testing modify the circumstances for the student while taking a test. For example, students with learning disabilities may be allowed more time to take a test. [9] IDEA was enacted in 1970 and, among other things, requires schools to provide assessments for students with disabilities and provide whatever accommodations they may need, including specific accommodation for testing, such as more time to take tests, to ensure that the tests provide valid reflections of their abilities. [10] [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO/HEHS-00-89], June 1, 2000. [End of section] GAO’s Mission: The General Accounting Office, the investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through the Internet. GAO’s Web site [hyperlink, http://www.gao.gov] contains abstracts and full text files of current reports and testimony and an expanding archive of older products. The Web site features a search engine to help you locate documents using key words and phrases. You can print these documents in their entirety, including charts and other graphics. Each day, GAO issues a list of newly released reports, testimony, and correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site daily. The list contains links to the full-text document files. To have GAO e-mail this list to you every afternoon, go to [hyperlink, http://www.gao.gov] and select “Subscribe to daily E-mail alert for newly released products” under the GAO Reports heading. Order by Mail or Phone: The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. General Accounting Office: 441 G Street NW, Room LM: Washington, D.C. 20548: To order by Phone: Voice: (202) 512-6000: TDD: (202) 512-2537: Fax: (202) 512-6061: To Report Fraud, Waste, and Abuse in Federal Programs Contact: Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Public Affairs: Jeff Nelligan, managing director, NelliganJ@gao.gov: (202) 512-4800: U.S. General Accounting Office: 441 G Street NW, Room 7149: Washington, D.C. 20548: