This is the accessible text file for GAO report number GAO-08-9 
entitled 'District of Columbia Opportunity Scholarship Program: 
Additional Policies and Procedures Would Improve Internal Controls and 
Program Operations' which was released on November 2, 2007. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

GAO: 

United States Government Accountability Office: 

Report to Congressional Requesters: 

November 2007: 

District Of Columbia Opportunity Scholarship Program: 

Additional Policies and Procedures Would Improve Internal Controls and 
Program Operations: 

District of Columbia Opportunity Scholarship Program: 

GAO-08-9: 

GAO Highlights: 

Highlights of GAO-08-9, a report to congressional requesters. 

Why GAO Did This Study: 

The D.C. School Choice Incentive Act created the first private 
kindergarten-through-grade-12 school-choice program supported by 
federal funds. The program was named the D.C. Opportunity Scholarship 
Program (OSP). GAO was asked to assess the (1) accountability 
mechanisms governing the use of funds, (2) results of the grantee’s 
efforts to meet the program’s recruiting priorities and eligibility 
requirements and inform parents of their choices, and (3) extent that 
the evaluation of OSP reflects statutory requirements and the 
implementation of the program supported the detection of useful and 
generalizable findings. To assess the program, GAO analyzed financial, 
program, and evaluation data. GAO did not assess the performance of 
participating private schools nor did GAO evaluate the impact of the 
program. 

What GAO Found: 

Following the Department of Education’s selection of the Washington 
Scholarship Fund (WSF) to operate the District of Columbia Opportunity 
Scholarship Program, WSF greatly expanded its operations from $150,000 
in federal and foundation grants in fiscal year 2004 to $12.9 million 
in 2006 without sufficient accountability mechanisms to govern the use 
of the funds. With such rapid expansion, WSF had little time to design 
and implement the needed systems, procedures, and internal controls for 
managing such a major increase in its operations. WSF’s accountability 
was further weakened by high staff turnover, a lack of detailed fiscal 
policies, and nonintegrated accounting functions. We found that WSF did 
not adhere to its own procedures for making scholarship payments, and 
WSF’s use of OSP funds to pay tuition for students attending schools 
that do not normally charge students tuition is not in accordance with 
the Act. Additionally, on the basis of a lack of documentation about 
whether before-and-after care included academic support, GAO was unable 
to determine if use of OSP funds to pay these fees was in accordance 
with the Act. 

Despite recruitment efforts and efforts to inform parents of their 
choices, WSF faced challenges recruiting students from schools 
designated as in need of improvement, ensuring private school quality 
and placement opportunities, and providing parents with accurate 
information regarding private schools. Students who were offered 
scholarships generally reported income consistent with OSP’s financial 
eligibility standards, but, among students offered scholarships, 
students from schools in need of improvement were underrepresented 
relative to their presence in the population of District students. 
Although most private schools in the District officially participated 
in the program, the schools varied widely in the number of openings 
available to scholarship students, and few openings were available at 
the secondary level. The characteristics of participating schools 
varied, and some schools did not meet basic requirements to operate in 
the District, but the information WSF provided to parents to help them 
choose schools for children was not always complete and correct. 

The evaluation contractor developed a strong evaluation design that 
reflected the statutory requirements and used random assignment to 
strengthen comparisons between students offered and not offered 
scholarships. However, factors related to program implementation 
limited the ability to perform comparisons directed by the Act and the 
usefulness and generalizability of findings. For example, the Act 
directed the evaluation contractor to use the same test as the District 
used to measure achievement and to compare the achievement of students 
offered scholarships with students in District public schools. The 
contractor did use the test used by the District, but District 
officials adopted a new testing program the second year of the 
evaluation, making it infeasible to compare students offered 
scholarships with students in District public schools. 

What GAO Recommends: 

The Secretary of Education should direct the grantee to improve 
internal controls, continue to integrate its financial systems, improve 
monitoring, and provide accurate information to parents. The Secretary 
and the Mayor of the District of Columbia should ensure participating 
schools are in compliance with District requirements. The Secretary 
should ensure that programs are implemented in accordance with any 
required evaluation. The entities providing comments on this report 
generally agreed with our recommendations but disagreed with many 
findings. 

To view the full product, including the scope and methodology, click on 
[hyperlink, http://www.GAO-08-9]. For more information, contact 
Cornelia M. Ashby at 202-512-7215 or ashbyc@gao.gov. 

[End of section] 

Contents: 

Letter: 

Results in Brief: 

Background: 

WSF's Accountability and Internal Control Were Inadequate for OSP, and 
Some Payments Were Not Made in Accordance with the Act While Others 
Raise Questions about Compliance: 

While OSP Students Generally Met the Program's Eligibility 
Requirements, WSF Had Limited Success Meeting Recruiting Priorities and 
Did Not Provide Parents Complete Information: 

Evaluation Contractor Adopted a Strong Methodology, but Program 
Implementation and Other Factors Limited the Usefulness and 
Generalizability of Findings: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Scope and Methodology: 

Appendix II: Laws Authorizing Related Voucher Programs: 

Appendix III: Comments from the Office of the Mayor of the District of 
Columbia: 

Appendix IV: Comments from the Department of Education: 

Appendix V: Comments from the Washington Scholarship Fund: 

Appendix VI: Contact and Staff Acknowledgment: 

Related GAO Products: 

Tables: 

Table 1: Key Events in Implementing and Operating the OSP: 

Table 2: Opportunity Scholarship Program Federal Funding for Fiscal 
Years 2004-2006: 

Table 3: Growth in WSF from Fiscal Years 2003--2006: 

Table 4: Types of Outreach Methods Used to Reach Potentially Eligible 
Families: 

Table 5: Summary of Tuition Rates by Grade Level for Participating 
Private Schools, School Year 2006-07: 

Table 6: Unweighted Percentage of Students with Missing Achievement 
Test Scores at Baseline and after 1 Year of Program Participation: 

Table 7: Number of Total Private School Placements for OSP Students: 

Table 8: Eligible Applicants Offered and Not Offered Scholarships and 
Numbers Included in and Excluded from the Evaluation: 

Table 9: Analysis of Laws Authorizing Voucher Programs in the District 
of Columbia and the State of Ohio: 

Table 10: Analysis of Laws Authorizing Voucher Programs in Cleveland, 
Ohio, and Milwaukee, Wisconsin: 

Figures: 

Figure 1: WSF Staff Turnover between May 2006 and June 2007 Based on 
Positions in Place in August 2006: 

Figure 2: Reporting Structure Among WSF Staff as of August 2007: 

Figure 3: Examples of Outreach Materials Grantee Used To Recruit Low- 
Income Families: 

Figure 4: Percentages of Students Offered OSP Scholarships Who Attended 
Schools Designated as in Need of Improvement Were Consistently Below 
the Percentages of Students in District of Columbia Schools Who 
Attended Such Schools: 

Figure 5: Examples of Schools That Operated without a Certificate of 
Occupancy Specifying Use as a Private Day School: 

Abbreviations: 

OSP: Opportunity Scholarship Program: 

MOU: memorandum of understanding: 

MPCP: Milwaukee Parental Choice Program: 

SSP: Signature Scholarship Program: 

WSF: Washington Scholarship Fund: 

United States Government Accountability Office: 

Washington, DC 20548: 

November 1, 2007: 

The Honorable Edward M. Kennedy: 
Chairman, Committee on Health, Education, Labor, and Pensions: 
United States Senate: 

The Honorable Richard J. Durbin: 
Chairman, Subcommittee on Financial Services and General Government: 
Committee on Appropriations: 
United States Senate: 

The Honorable Eleanor Holmes Norton: 
House of Representatives: 

In an effort to enhance educational opportunity for children from low- 
income families in the District of Columbia (the District), Congress 
passed the D.C. School Choice Incentive Act in January 2004.[Footnote 
1] The Act created a private school scholarship program, subsequently 
called the D.C. Opportunity Scholarship Program (OSP), for low-income 
students living in the District of Columbia, setting a precedent for 
the use of federal funds to provide scholarships to finance private 
education for students in kindergarten through grade 12. The purpose of 
the Act is to provide low-income parents of students in the District, 
particularly parents of students who attend public schools identified 
as in need of improvement under the No Child Left Behind Act,[Footnote 
2] "with expanded opportunities for enrolling their children in higher 
performing schools in the District" by providing annual scholarships of 
up to $7,500 per child. The D.C. School Choice Incentive Act directed 
the Secretary of Education to award a grant for up to 5 years to an 
eligible entity or entities to operate the program, giving priority to 
grant applicants who would most effectively give priority to students 
from schools in need of improvement, target resources to students and 
families that lack financial resources to take advantage of available 
educational options, and provide students and families with the widest 
range of educational options. The Act also directed the Secretary and 
the Mayor of the District of Columbia to select an independent 
evaluator to compare outcomes, including academic achievement, for 
several groups of children, such as participating eligible students and 
eligible students who were not offered scholarships and participating 
eligible students and students in District Public Schools. The 
evaluation was also to examine the effect of the program on District 
public schools. 

You asked us to assess the implementation of the program, including the 
(1) accountability mechanisms in place governing the use of funds, (2) 
results of the grantee's efforts to meet the program's recruiting 
priorities and eligibility requirements and inform parents about their 
choices, and (3) extent to which the evaluation reflects statutory 
requirements and the implementation of the program supported the 
detection of useful and generalizable findings. 

To address these issues, we used multiple strategies. To assess the 
accountability mechanisms governing the use of funds, we interviewed 
officials of the Washington Scholarship Fund (WSF), the grantee 
operating the program, of the Department of Education's Office of 
Innovation and Improvement and of the District of Columbia; reviewed 
documents describing the grantee's processes for implementing the 
program; analyzed WSF's financial statements; obtained and reviewed 
detailed financial data from WSF; and tested specific internal controls 
over payments. To assess the results of the grantee's efforts to meet 
the program's recruiting priorities and eligibility requirements and 
inform parents about their choices, we interviewed officials at the 
Office of Innovation and Improvement and the grantee; examined 
pertinent documentation; and analyzed students' demographic data, 
characteristics of participating schools, and information provided to 
parents. We reviewed information on private schools from WSF and public 
sources, but we did not assess specific private schools or their 
performance. We did not evaluate the impact of the program, which was 
the subject of Education's evaluation. We reviewed WSF's process for 
collecting financial eligibility information, the records maintained by 
WSF on household income data reported by parents of scholarship 
recipients, and the findings on eligibility made by the grantee's 
auditor. We did not perform independent testing of eligibility. To 
assess the extent to which the evaluation reflects statutory 
requirements and program implementation supported the detection of 
useful and generalizable findings, we interviewed officials at 
Education's Institute of Education Sciences and at Westat--the 
evaluation contractor--reviewed Westat's and its affiliates'[Footnote 
3] reports for year one and year two of the OSP that described the 
participation of students and schools in the program, analysis plan, 
and evaluation of the results of the program after 1 year of 
participation. In assessing data reliability, we found problems with 
WSF's and the evaluation contractor's program data systems, which we 
discuss in appendix 1, that limited the analyses that we could perform. 
We brought these issues to the attention of Education, WSF, and the 
evaluation contractor. However, we were able to perform sufficient 
analyses to answer the study's questions. We conducted our work between 
September 2006 and September 2007 in accordance with generally accepted 
government auditing standards. (For more detail on our scope and 
methodology, see app. 1.) 

Results in Brief: 

Following the Department of Education's selection of WSF to operate the 
District of Columbia Opportunity Scholarship Program, WSF greatly 
expanded its operations from $150,000 in federal and foundation grants 
in fiscal year 2004 to $12.9 million in 2006 without sufficient 
accountability mechanisms to govern the use of the OSP funds. Rapid 
expansion and new demands put at risk WSF's accountability for its use 
and safeguarding of federal funds. This accountability risk was 
increased by high turnover in key staff positions. WSF did not develop 
the comprehensive and detailed fiscal policies and procedures needed 
for internal control over its financial and programmatic operations. 
WSF's draft policies and procedures lacked key control activities and 
did not detail procedures for approving scholarship payments or require 
documentation that procedures were carried out as required. WSF's 
automated financial systems were not interfaced, and reliance on manual 
processes between automated systems increased the risk of error. 
Serious weaknesses in the implementation of internal controls affected 
the processing of OSP scholarship payments, reconciliations between 
financial records and bank statements, and assurance that participating 
schools were financially responsible. For example, while WSF 
established policies to help ensure that OSP funds are used to pay for 
services in accordance with the Act establishing the program, WSF's use 
of OSP funds to pay tuition for students attending schools that do not 
normally charge students tuition is not in accordance with the Act. 
However, WSF maintains that students attending these schools are 
supported by scholarship dollars just as those who receive OSP funds. 
Furthermore, our review questions certain other payments, but we were 
unable to determine if they were allowable because these payments 
lacked documentation. 

Despite efforts to recruit eligible applicants and private schools and 
inform parents, WSF faced challenges recruiting students from schools 
in need of improvement, ensuring private school placement opportunities 
and quality, and providing parents with accurate information regarding 
private schools. WSF designed procedures to ensure that scholarship 
recipients met program financial eligibility standards and income data 
reported by parents of participants were broadly consistent with income 
guidelines. While students from schools in need of improvement had 
priority for receiving scholarships, for each year the program has 
operated, the percentage of students from schools in need of 
improvement who were offered scholarships has been smaller than the 
percentage of students in District public schools who attended such 
schools. Although most private schools in the District officially 
participated in the program, the schools varied widely in the number of 
openings available to scholarship students, and very few openings were 
available at the secondary level. In school year 2006-2007, more than 
half the scholarship students were concentrated in 19 participating 
private schools, and 16 private schools had enrolled five or fewer OSP 
students. The extent to which private schools had characteristics 
associated with high-quality educational programs has varied. For 
example, some schools reported that some of their teachers lacked at 
least a bachelor's degree. Furthermore, some participating schools did 
not meet basic requirements to operate in the District. For example, a 
few had no certificate of occupancy on file with the District or had 
certificates that did not specify educational use. Despite important 
variation among schools, the grantee did not always provide parents 
with complete and accurate information on their characteristics. For 
example, WSF provided parents inaccurate information on teachers' 
qualifications and tuition for some schools. 

The evaluation contractor developed a strong methodology that used 
random assignment to strengthen comparisons between students offered 
and not offered scholarships, but factors related to program 
implementation limited the ability to perform the comparisons directed 
by the Act and the usefulness and generalizability of evaluation 
findings. For example, although the Act directed the evaluators to 
compare the achievement of participating eligible students who used 
scholarships with students in the same grade in District public 
schools, the evaluation contractor could not reasonably or cost- 
effectively make these comparisons. Although the evaluation contractor 
administered the same tests to measure achievement as were given by the 
District in the first year, the District adopted a new achievement 
testing program during the second year of the program for students in 
grades 3-8 and grade 10. This meant that a common test was no longer 
available for the program's 5 years of operation and made it necessary 
to pay the evaluation contractor to test more students than planned 
because, in addition to testing students with OSP scholarships in 
private schools, the contractor also had to test eligible applicants in 
District public schools who were not offered scholarships. Also, the 
Act called for comparisons of dropout rates, graduation rates, and 
college admission rates of scholarship program participants and 
nonparticipants, but the limited placements offered by high schools 
restricted the number of high school students and graduates in the 
evaluation and eliminated the evaluator's ability to make these 
comparisons. Because the WSF awarded many scholarships nonrandomly and 
to some students who already attended private schools, the number of 
students who could be included in the evaluation was diminished, 
reducing the ability of the evaluation's statistical tests to identify 
differences between groups and the strength of possible findings. Other 
factors arising from the operation of the program, including the 
inability of some students who were offered scholarships to find 
placements and the provision of school or community based supplemental 
services to some scholarship recipients might have made the 
interpretation of findings difficult and also might have limited the 
usefulness of the evaluation to programs in other jurisdictions. 

To improve OSP, we are making several recommendations to the Secretary 
of Education and the Mayor of the District of Columbia. To better focus 
resources on achieving the program's goals of providing accountability, 
quality educational experiences, and allowing parents to make an 
informed school choice for their child, we recommend that the Secretary 
direct the grantee to establish policies and procedures that improve 
certain internal controls, continue efforts to integrate its financial 
systems, improve monitoring, and provide accurate and complete 
information to parents. We are also recommending that the Secretary and 
the Mayor of the District of Columbia take specific steps to improve 
private school compliance with applicable District requirements and 
improve oversight of the program. Finally, we recommend that, in 
planning for future programs for which Congress has required an 
evaluation, Education should oversee the program to ensure consistency 
between program implementation and the evaluation design. 

We sent this report to the Office of the Mayor of the District of 
Columbia, Education and WSF for review and comment. (Comments are 
reprinted in appendixes III, IV, and V, respectively.) These entities 
generally agreed with our recommendations, but Education and WSF 
disagreed with many of our findings. Education officials raised 
concerns related to scope of the assignment and stated that the draft 
report did not present a complete and balanced picture of issues such 
as the extent to which OSP students previously attended schools in need 
of improvement. In addition, Education provided comments indicating 
that GAO should have focused on the percentage of students whose former 
schools were identified for improvement by the end of the students' 
first year in the OSP. We disagree. Scholarship lottery organizers 
could not be expected to give priority to students from schools that 
had not yet been identified. Moreover, regardless of which year is 
used, the percentage of students from schools in need of improvement 
was smaller among scholarship recipients than among the general 
population of District students from whom scholarship recipients could 
have been recruited. Education also indicated we should have provided 
more information on the speed with which the program was implemented, 
satisfaction among parents, demand for scholarships, and participating 
schools' traditional independence in areas such as hiring and 
establishing teacher qualifications. As discussed further in our 
response beginning on page 46, we believe that these subjects were 
addressed in a complete and balanced fashion to the extent that they 
were included in the scope of our review. Moreover, we found that lack 
of reliable data on applicants limited the analysis we could perform to 
determine the demand for scholarships. We agree that participating 
schools had independence in areas such as hiring and establishing 
teacher qualifications, but note that participating schools are also 
required to be in compliance with District education and safety 
requirements. In its comments, WSF also acknowledged that some of GAO's 
recommendations, findings, and observations were valid and useful and 
that it had taken or will take action to address some of them but 
disagreed with many of our observations, interpretations, and findings. 
While WSF implies that the deficiencies noted in this report are minor 
in nature and disagreed with our conclusion that the Act prohibited 
payments to schools that customarily charged no tuition, we stress that 
the issues we identified are indicative of potential problems and if 
not addressed could have a material, detrimental effect on WSF's 
accountability over federal funds, especially when combined with the 
accounting systems and cash reconciliation weaknesses that existed. As 
we noted above in our response to Education, we believe the findings 
and interpretations are accurate as stated in the draft and therefore 
have not made changes other than technical corrections. 

Background: 

The D.C. School Choice Incentive Act: 

The D.C. School Choice Incentive Act of 2003 directs the Secretary of 
Education to award a grant on a competitive basis for up to 5 years to 
an entity or entities to use to make scholarship payments to parents of 
eligible students to be used for private school tuition, fees, and 
transportation expenses. The Act directs the grantee to ensure that the 
amount of tuition or fees that a participating private school charges a 
scholarship student does not exceed the tuition or fees that the school 
customarily charges to students not participating in the scholarship 
program. Under the Act, Education could not approve a request for a 
grant unless the entity's application detailed how it would, among 
other things: 

* address the priorities identified in the Act--that is, giving 
priority to students who attended schools identified as in need of 
improvement,[Footnote 4] targeting resources to families that lacked 
resources to take advantage of available educational options, and 
providing students and families with the widest range of educational 
choice; 

* ensure that if more eligible students seek admission in the program 
than the program can accommodate, eligible students are selected for 
admission through a random selection process that gives weight to the 
program priorities; 

* notify parents of eligible students of expanded choice opportunities 
and ensure that parents receive sufficient information about their 
options to allow the parents to make informed decisions; 

* determine the amount that would be provided to parents for the 
tuition, fees, and transportation payments, if any; 

* seek private elementary and secondary schools in the District to 
participate in the program, and ensure that participating schools would 
meet the application and reporting requirements of this Act; 

* ensure that participating schools were financially responsible and 
would use funds effectively; and: 

* address the renewal of scholarships to participating eligible 
students, including continued eligibility. 

The Act limits payments for each scholarship student to $7,500 per 
year. If tuition exceeds the $7,500 cap, schools may supplement the 
$7,500 with private scholarships or donated funds, absorb the 
difference, or work out payment plans with parents or guardians. 
According to the program requirements established by WSF, participating 
private schools may apply their own admission standards and decide 
whether to admit students who have received an OSP scholarship. The 
program's funding allowed scholarships for about 2,000 District 
students out of a public school population of about 72,000. (App. II 
presents a comparison of the District of Columbia school choice program 
and programs in several states and cities.) 

The Department of Education's Office of Innovation and Improvement 
awarded a grant to operate the program to WSF, a nonprofit organization 
in the District of Columbia that had experience providing privately 
funded scholarships to low-income students. WSF offered its first 
scholarships under OSP for the 2004-2005 school year, and the program 
recently completed its third year of operation with the conclusion of 
the 2006-07 school year. The Act required that scholarships be awarded 
through lotteries if the number of eligible applicants exceeded the 
number of private school placements available or the number of 
scholarships that the program could accommodate. 

In these lotteries, students were assigned different priorities for 
receiving a scholarship. Students attending schools in need of 
improvement received the highest priority, followed by students who had 
attended a public school not in need of improvement, followed by 
students from private schools. The number of District schools assigned 
the designation of needing improvement has increased from 2003 to 2006, 
and the designations are updated following the school year. 

The Act also required that the Secretary award a contract to an entity 
jointly selected by the Secretary and the Mayor of the District of 
Columbia for an independent evaluation that would use "the strongest 
possible research design" to determine the effectiveness of the 
program. The evaluators were to provide (1) a comparison of the 
academic achievement of participating eligible applicants who were 
offered scholarships and those who were not; (2) a comparison of the 
academic achievement of applicants who were offered scholarships and 
children attending District of Columbia Public Schools; (3) comparisons 
of drop-out rates, graduation rates, and college admission rates of 
students who participate in the program with students in District 
schools who do not participate; (4) a comparison of the safety of 
schools attended by participants in the program and the schools 
attended by students who do not participate in the program; (5) 
measures of the effect of the program on District public schools and 
their students; and (6) the success of the program in expanding choice 
for parents. The Secretary awarded the evaluation contract to Westat 
and its affiliates. 

Additional Laws and Agreements Governing Administration of the OSP: 

The Act required that Education and the District enter into a 
memorandum of understanding (MOU) regarding the implementation of the 
D. C. School Choice Incentive Program. It directed that the MOU conform 
to requirements contained in the conference report accompanying the 
Act. In that report, the conferees stated their expectation that the 
MOU would include (1) strong accountability measures and specifications 
for program performance evaluations; (2) specifications for a lottery 
system providing for fair and unbiased acceptance of students into the 
program; (3) provisions for joint oversight of the program's operations 
by the Mayor of the District of Columbia and the Secretary of 
Education; (4) the methodology for the evaluation and selection of 
participating schools that have met the District of Columbia's current 
licensure requirements; (5) the methodology for determining the tuition 
and fees of participating schools, including the actual cost; (6) 
requirements for the development of appropriate oversight and 
accountability measures; and (7) teacher quality criteria. 

The MOU reflected this language in part. It identified Education's 
Deputy Under Secretary of Innovation and Improvement as having lead 
responsibility for implementation of the OSP, Education's Institute of 
Education Sciences as having lead responsibility for implementation of 
the evaluation, and the Mayor or his designee as having lead 
responsibility for carrying out cooperative activities between the 
District and Education. It also specified that Education and the Mayor 
would jointly oversee the program and ensure it was carried out in 
accordance with statutory requirements, the approved grant application, 
and sound management and educational principles. Also, the MOU 
conditioned schools' participation, in part, on "operating lawfully 
within the District of Columbia," -meaning that participating schools 
meet all the District of Columbia's current requirements. For example, 
the Code of District of Columbia Municipal Regulations requires that 
for purposes of the District's compulsory attendance law, private 
schools must provide evidence that they are currently accredited by or 
undergoing the accreditation process from one of seven named 
organizations or any other accrediting body approved by the Board of 
Education, or that they submit evidence of their educational soundness 
acceptable to the Board.[Footnote 5] Accreditation provides assurance 
that a school's course credits can be counted toward a District diploma 
or accepted by other schools. Operating lawfully in the District of 
Columbia also entails meeting additional District requirements, such as 
securing a certificate of occupancy to demonstrate compliance with 
health, safety, and fire code requirements. WSF's grant application 
indicated that it would require participating private schools to 
"attach their Certificate of Occupancy [to their letter of intent to 
participate] and document that they are in compliance with all relevant 
D.C. health, safety, and fire code requirements." Further, the MOU 
stated that the grantee was to provide information on each 
participating private school to parents of students offered 
scholarships including, but not limited to, teacher qualifications, the 
academic achievement of the school's students, and the safety and 
environment of the school. However, the MOU did not include a 
methodology for determining actual tuition and fees, or specify 
oversight and accountability measures and teacher quality criteria. 

Initiation and Funding of the Opportunity Scholarship Program: 

Soon after the D.C. School Choice Incentive Act was passed, Education 
awarded the grant to operate the OSP to the Washington Scholarship 
Fund, which was the only entity that submitted a complete proposal. 
Table 1 shows the timing of key events in the implementation and 
operation of OSP. 

Table 1: Key Events in Implementing and Operating the OSP: 

Date: January 23, 2004; 
Event: Congress enacted the D.C. School Choice Incentive Act of 2003. 

Date: February 3, 2004; 
Event: The District's Mayor selected Fight for Children to serve as the 
interim, nonprofit entity charged with conducting early outreach and 
publicity efforts. 

Date: March 24, 2004; 
Event: Education awarded the grant to Washington Scholarship Fund to 
operate the DC School Choice Program. 

Date: March 26, 2004; 
Event: Education awarded a contract to provide technical assistance for 
placement lotteries. 

Date: Late March 2004; 
Event: Outreach efforts and preliminary data on interested families 
were transferred from Fight for Children to WSF. 

Date: Late March-late August 2004; 
Event: WSF developed application and eligibility determination 
processes, and program management systems and continued multifaceted 
recruitment and outreach efforts. 

Date: April 16, 2004; 
Event: The evaluation contractor and its affiliates submitted a 
proposal to conduct the evaluation. 

Date: Mid-April -Mid-May 2004; 
Event: WSF collected its first applications from potential scholarship 
families and held orientation meetings at the Washington Convention 
Center. 

Date: April-May 2004; 
Event: Families submitted applications, and WSF and its contractor 
processed applications and conducted eligibility determination. 

Date: May 11, 2004; 
Event: Deadline for private schools to report available number of 
openings by grade for first school year of program. 

Date: June 17, 2004; 
Event: WSF conducted lottery for awarding year one scholarships. 

Date: July 21, 2004; 
Event: Education awarded a contract for the evaluation of the program 
to the firm contracted to provide technical assistance on placement 
lotteries. 

Date: August/September 2004; 
Event: First school year of Opportunity Scholarship program began with 
1,027 OSP students enrolled in private schools. 

Date: May 2005; 
Event: WSF held lotteries for awarding scholarships. 

Date: August/September 2005; 
Event: Second school year began; 
1,661 OSP students were enrolled in private schools. 

Date: March 2006; 
Event: Deadline for parents to apply for a scholarship for the 2006-07 
school year; only children entering grades kindergarten through 5th 
grade could apply for new scholarships. 

Date: August/September 2006; 
Event: Third school year began, with about 1,743 OSP students enrolled 
in private schools. 

Date: December 20, 2006; 
Event: Congress passed legislation raising the eligible income level to 
300 percent of the poverty line for students who received scholarships 
during the first two years of the program. 

Date: June 2007; 
Event: Education released Evaluation of the DC Opportunity Scholarship 
Program: Impacts After One Year. 

Source: GAO analysis of information from Education and WSF. 

[End of table] 

As shown in table 2, $14 million was initially appropriated to 
Education in each of fiscal years 2004 through 2006. Each year's 
appropriation was reduced through subsequent rescissions. Table 2 also 
shows the flow of funds from Education to the program. Specifically, of 
the funds for fiscal year 2006, Education received $13.86 million. Of 
this amount, about $12.45 million was awarded to WSF in the form of a 
federal grant to operate the OSP. The amount of funding retained by 
Education, $1.4 million, was provided to Education's Office of 
Innovation and Improvement for program administration and oversight 
($0.1 million) and to Education's Institute of Education Sciences for 
an evaluation of the program ($1.3 million) as mandated by the Act. 
Funds disbursed to the grantee, WSF, are to be expended for tuition, 
fees, transportation, and program administration. According to WSF, the 
majority of funds donated by private sources are used for eligibility 
determination, infrastructure, and other technical systems development 
and maintenance, and student placement. 

Table 2: Opportunity Scholarship Program Federal Funding for Fiscal 
Years 2004-2006: 

Sources and disbursement of funds: Appropriation to Education: Federal 
appropriation; 
2004: $14,000,000; 
2005: $14,000,000; 
2006: $14,000,000. 

Sources and disbursement of funds: Appropriation to Education: 
Rescission[A]; 
2004: (82,600); 
2005: (112,000); 
2006: (140,000). 

Sources and disbursement of funds: Appropriation to Education: Net 
appropriations to the U.S. Department of Education; 
2004: 13,917,400; 
2005: 13,888,000; 
2006: 13,860,000. 

Sources and disbursement of funds: Disbursement of grant funds: 
Department of Education: Institute of Education Sciences; 
2004: 1,344,822; 
2005: 1,308,640; 
2006: 1,305,877. 

Sources and disbursement of funds: Disbursement of grant funds: 
Department of Education: Office of Innovation and Improvement (OII); 
2004: 66,800; 
2005: 100,000; 
2006: 99,923. 

Sources and disbursement of funds: Disbursement of grant funds: 
Department of Education: Grant provided to WSF; 
2004: 12,505,778; 
2005: 12,479,360; 
2006: 12,454,200. 

Sources and disbursement of funds: Disbursement of grant funds: 
Department of Education: Total funds disbursed to Education and WSF; 
2004: 13,917,400; 
2005: 13,888,000; 
2006: 13,860,000. 

Source: GAO analysis of appropriations laws and allotments data. 

[A] Congress enacted across-the-board rescissions for appropriations 
made for fiscal years 2004, 2005, and 2006. The rescissions were 0.59 
percent for fiscal year 2004, 0.80 percent for fiscal year 2005, and 1 
percent for fiscal year 2006. 

[End of table] 

The Act provided that the grantee may not use more than 3 percent of 
the total amount provided under the grant each year for the 
administrative expenses of carrying out the program during the year, 
including (1) determining the eligibility of students to participate, 
(2) providing information about the program and the schools involved to 
parents of eligible students, (3) selecting students to receive 
scholarships, (4) determining the amount of scholarships and issuing 
the scholarships to eligible students, (5) compiling and maintaining 
financial and programmatic records, and (6) providing funds to assist 
parents in meeting expenses that might otherwise preclude the 
participation of their child in the program. WSF officials defined 
administrative expenses as any program expenses other than tuition, 
fees, and transportation. These administrative expenses generally 
included direct expenses, such as student eligibility reviews and 
payroll, and indirect expenses, such as rent and utilities. 

WSF's Accountability and Internal Control Were Inadequate for OSP, and 
Some Payments Were Not Made in Accordance with the Act While Others 
Raise Questions about Compliance: 

After being selected as the OSP grantee, WSF struggled with developing 
and implementing accountability mechanisms and internal controls in the 
management of federal and foundation grants that increased from 
$150,000 in fiscal year 2004 to $12.9 million in fiscal year 2006. 
Further, with the addition of OSP to its operations, WSF was challenged 
to find systems and approaches for community outreach, school 
placement, family and school support, and financial management while 
implementing the program. Due to its need to quickly implement the 
program, WSF had little time to develop internal control policies and 
procedures, systems, and internal control activities needed to provide 
accountability over the use of grant funds for this new 
program.[Footnote 6] WSF had to rely on the knowledge of its staff to 
perform the daily operations for OSP, and its ability to rely on its 
key staff was greatly affected by high staff turnover. The discovery of 
fraud allegedly perpetrated by a WSF employee affecting one of its 
private scholarship programs highlighted the need for strengthened 
internal control activities. The lack of comprehensive policies and 
procedures, the use of nonintegrated systems, and the outsourcing of 
basic bookkeeping functions increased risk and weakened WSF's 
accountability over OSP. Moreover, we found that WSF did not 
consistently adhere to the basic accountability and internal control 
procedures it had established for scholarship payments. In reviewing 
OSP scholarship payments, we found that WSF's practice of paying for 
tuition scholarships to schools that do not normally charge students 
tuition is a violation of the Act because the Act provides that the 
tuition charged OSP students cannot be more than the tuition charged 
for similarly situated non-OSP students. Furthermore, we could not 
determine whether payments for before-and-after care programs were 
appropriate because the documentation provided by WSF did not indicate 
that it had verified that the care programs met the criteria for 
determining whether a fee is allowable. Based on the documentation 
provided by WSF, we could not determine that WSF verifies that before- 
and-after school care programs are tied to the student's academic 
program and part of customary fees charged by the school, which are 
Education and WSF's key criteria under the Act for determining whether 
a fee is allowable. 

WSF's Rapid Growth and High Staff Turnover Created Accountability Risk 
for the Opportunity Scholarship Program: 

Following its selection as a grantee in March 2004, WSF rapidly 
expanded its scholarship operations to provide OSP scholarships to 
students for the upcoming 2004-2005 school year.[Footnote 7] This rapid 
expansion can be seen in the increase of WSF's annual federal and 
foundation grants from $150,000 in fiscal year 2004 to $8.6 million in 
fiscal year 2005, and to $12.9 million in fiscal year 2006. (See table 
3.) WSF, charged with running OSP, had to create systems for community 
outreach, school placement, and family and school support while 
implementing the program, and WSF faced the challenge of accomplishing 
school search and enrollment activities before the beginning of the 
2004-2005 school year. WSF was also challenged to find ways in which 
thousands of OSP checks could be correctly generated, accounted for, 
and monitored for compliance with statutory requirements. With this 
rapid expansion, WSF had little time to design and implement the needed 
systems, policies, procedures, and internal controls for managing such 
a major increase in its operations. While WSF was able to process the 
OSP scholarship payments for school years 2004-2005 and 2005-2006, it 
used systems that were not integrated, resulting in data being manually 
transferred, and thereby increasing the risk of error. Also, WSF's 
basic bookkeeping and accounting functions, including the printing of 
OSP scholarship checks and preparation of bank reconciliations, were 
not integrated, contributing to the fragmentation of WSF's OSP 
operations and weakened accountability. 

Table 3: Growth in WSF from Fiscal Years 2003--2006: 

Financial statement line item: Federal and foundation grants[A]; 
Fiscal year ending June 30: 2003: --[B]; 
Fiscal year ending June 30: 2004: $150,000; 
Fiscal year ending June 30: 2005: $8,563,995; 
Fiscal year ending June 30: 2006: $12,893,898. 

Financial statement line item: Contributions and grants; 
Fiscal year ending June 30: 2003: $1,801,251; 
Fiscal year ending June 30: 2004: $1,929,707; 
Fiscal year ending June 30: 2005: $1,576,706; 
Fiscal year ending June 30: 2006: $1,793,409. 

Total; 
Fiscal year ending June 30: 2003: 1,801,251; 
Fiscal year ending June 30: 2004: 2,079,707; 
Fiscal year ending June 30: 2005: 10,140,701; 
Fiscal year ending June 30: 2006: 14,687,307. 

Financial statement line item: Scholarships expense[C]; 
Fiscal year ending June 30: 2003: 1,967,480; 
Fiscal year ending June 30: 2004: 2,268,942; 
Fiscal year ending June 30: 2005: 9,043,594; 
Fiscal year ending June 30: 2006: 14,108,731. 

Financial statement line item: Expenditures of federal awards; 
Fiscal year ending June 30: 2003: --[D]; 
Fiscal year ending June 30: 2004: -- [D]; 
Fiscal year ending June 30: 2005: 6,570,407; 
Fiscal year ending June 30: 2006: 11,664,898. 

Source: WSF audited financial statements for fiscal years ending June 
30, 2003, 2005, and 2006. 

[A] While OSP is a major federal program for WSF, WSF also has received 
a 2-year "Next Generation" grant from the Corporation for National and 
Community Service. This 2-year grant is designed to help organizations 
build programs in community service and volunteerism. 

[B] Beginning with fiscal year 2004, WSF reclassified contributions and 
grants for comparative purposes to show the federal and foundation 
grants separately from contributions. 

[C] This expense includes scholarships paid by WSF under both its 
publicly funded OSP and its Signature Scholarship Program, which is 
privately funded. 

[D] Single audits, which include the schedule of expenditures of 
federal awards, are required for grant recipients who expend $500,000 
or more in federal awards in each year. These amounts are not available 
because WSF expended less than $500,000 in federal grants and therefore 
was not required to submit a single audit report for fiscal years 2003 
and 2004. 

[End of table] 

WSF's rapid growth, the changing nature of its systems and processes 
for OSP scholarship payments, and the lack of comprehensive and 
detailed procedures meant that WSF had to rely on the knowledge of key 
staff to perform daily operations. However, the ability of WSF to rely 
on institutional knowledge was greatly affected by staff turnover in 
key positions. (See fig.1.) The Chief Financial and Operations Officer 
resigned in May 2006. WSF's Accounting and Grants Manager resigned in 
January 2007 and the Director, Finance and Operations resigned in 
February 2007. The Senior Director Education and Outreach left WSF 
during that time. The President and Chief Executive Officer then left 
in May 2007. The Senior Director for Scholarship Operations--a key 
person who understood WSF's scholarship payment processes for the OSP-
-resigned in June 2007. Staff turnover at WSF, combined with the lack 
of established accountability procedures and systems, has significantly 
increased WSF's overall fiscal and program operations risk. 

Figure 1: WSF Staff Turnover between May 2006 and June 2007 Based on 
Positions in Place in August 2006: 

This figure is a chart showing WSF staff turnover between May 2006 and 
June 2007. 

[See PDF for image] 

Source: Washington Scholarship Fund. 

[End of figure] 

WSF has recently increased its staff significantly. A comparison of 
figure 1 and figure 2 shows this increase in staff resources from 11 
staff in May 2006 to 17 staff in August 2007. Staffing is supplemented 
by interns who work with the full-time staff on both of WSF's 
scholarship programs. 

Figure 2: Reporting Structure Among WSF Staff as of August 2007: 

This figure is a chart showing reporting structure among WSF staff as 
of August 2007. 

[See PDF for image] 

Source: Washington Scholarship Fund. 

[End of figure] 

WSF's Accountability Was Further Weakened by a Lack of Comprehensive 
Fiscal Policies and Procedures and Use of Nonintegrated Systems for 
Basic Bookkeeping and Accounting Functions: 

WSF's accountability over OSP scholarship payments was further weakened 
by a lack of comprehensive and detailed policies and procedures to 
guide its processing of receipts and payment of scholarships and 
administrative expenses. For the fiscal year ended June 30, 2005, WSF's 
external auditor reported that WSF was in the process of developing a 
formal accounting manual documenting its procedures. The auditor stated 
in a letter to WSF management that this type of accounting manual is 
necessary to ensure that transactions are treated in a standardized 
manner. In its letter, the auditor also noted that written procedures, 
instructions, and assignments of duties will prevent or reduce 
misunderstandings, errors, wasted effort, duplicated or omitted 
procedures, as well as other situations that can result in inaccurate 
or untimely accounting records. The auditor recommended that WSF 
include in its accounting manual descriptions of each fiscal procedure-
-including payment of invoices, maintenance of accounts receivable 
subsidiary ledgers, and payroll procedures. 

In February 2007, WSF provided us with a copy of its Internal Systems 
Review, which was still in draft form and provided only general process 
descriptions. The document was created in response to the 
recommendation from its external auditor. WSF officials told us that 
its operations team would be continually writing and editing this 
document to further encourage organizational and financial integrity in 
its daily operations. The officials added that they expected the fiscal 
policies and procedures document to be final once WSF's new nonprofit 
financial management software is operational. As of late June 2007, WSF 
officials stated that this new software would be operational for 
processing the 2007-2008 scholarship payments. The February 2007 draft 
provides a high-level description of the processes for receipts, OSP 
and other scholarship payments, administrative expenses, human 
resources, and budgeting/ bank reconciliations. However, without 
extensive on-the-job training, the descriptions in this document are 
not adequately detailed for use in carrying out the procedures or for 
use in holding individuals responsible for key internal controls and 
accountability. 

The policies described in the draft Internal Systems Review are not 
specific and do not include controls over key activities for handling 
WSF's funds and payments or the specific internal controls needed to 
help ensure that key control activities are being carried out. For 
example, the procedures state that bills submitted by WSF's vendors are 
to be approved by the financial team, but the procedures do not include 
how the approval is to be documented and how the payment of the invoice 
is to be recorded in WSF's accounting software. Also, in the section on 
procedures for sending the OSP scholarship payments data to the 
contractor for printing checks, there is no mention of controls and 
procedures to verify the accuracy of the payment amounts and the 
printed checks. While segregation of duties[Footnote 8] is mentioned in 
the Internal Systems Review document for processing receipts and for 
issuing checks to WSF's vendors, these procedures do not clearly 
outline responsibilities for (1) authorizing transactions, (2) 
processing and recording them, (3) reviewing the transactions, (4) 
handling any related access to cash, and (5) how these duties are to be 
separated. The control achieved through segregation of duties is 
important for small as well as large organizations. Where the staff is 
not large enough to segregate all duties, a compensating control, such 
as supervisory review, can be established. Segregation of duties is 
among the controls designed to prevent such fraudulent acts as those 
allegedly committed by a former WSF staff member in processing WSF's 
private scholarship funds (no OSP funds were involved, according to WSF 
officials) between October 2006 and January 2007. The manager diverted 
funds from WSF's Signature Scholarship Program (SSP) bank account and 
made personal charges to a WSF credit card. A WSF official subsequently 
identified the fraudulent activity through reviews of bank 
reconciliations and credit card charges. WSF management told us that it 
has implemented additional controls to help ensure that access to any 
accounts or funds is secure and that these controls will be tested as a 
part of WSF's next financial statement audit. 

WSF processed OSP scholarship payments using systems that were not 
interfaced, resulting in data being manually transferred, a procedure 
that increases the risk of error. For school years 2004-2005 and 2005- 
2006, due to limitations in WSF's accounting system, WSF used an Excel 
spreadsheet to track information about individual OSP scholarship 
payments. However, the detailed payment and check information in the 
Excel spreadsheet could not be uploaded or interfaced with the 
accounting software used by WSF. As a result, only the lump sum totals 
for batches of payments were recorded in the WSF accounting software, 
not the individual scholarship check information. This system was used 
as the basis for WSF's financial reports. 

WSF enhanced its scholarship payment process for 2005-2006 through the 
implementation of a Web-based, online billing system that enabled 
participating schools to electronically enter the tuition and fee 
charges and submit invoices for all OSP students attending that 
school.[Footnote 9] WSF also created a student database that included 
key information about each student. However, because the online billing 
system was not integrated with the student database, key information 
about each OSP student had to be uploaded electronically into the 
online billing system to enable participating schools to submit 
invoices. 

WSF's overall process for key bookkeeping and accounting functions, 
such as check printing and bank reconciliation, was not well defined or 
integrated. Further, these processes were outsourced, requiring 
monitoring and oversight of contractor's work to ensure against errors. 
For example, the bank reconciliation process--a key cash management 
control--was made more difficult by the tracking of individual payments 
in the Excel spreadsheet or student database, neither of which was 
integrated with the accounting system. The lump sum recording of 
payments also made it difficult to track the separate manual entries in 
its accounting software for voided checks, reissued checks, and other 
adjustments, because individual OSP scholarship checks were not 
directly matched with corresponding entries. Therefore, the preparation 
of bank reconciliations was difficult. In 2007, WSF officials noted 
errors in recent bank reconciliations. These errors, combined with 
WSF's detection of fraud allegedly perpetrated by the former WSF 
employee in its nonfederal program, have prompted WSF to begin a 
process of redoing all bank reconciliations, starting with 2004. As of 
the end of our field work, this effort had not been completed and 
therefore the outcome is unknown. 

WSF officials have contracted to purchase a financial management 
software package designed for nonprofit organizations that they believe 
will facilitate the recording of the individual scholarship checks. 
According to WSF, the system should be in place for the 2007-2008 
school year. As WSF continues to implement integrated systems and 
processes, the need for manual data transfers will be reduced. 

WSF Did Not Consistently Adhere to Basic Accountability and Internal 
Control Procedures: 

WSF did not consistently adhere to its established OSP scholarship 
payment procedures and made payments without the required documentation 
or approvals, and it did not fulfill its oversight role for ensuring 
that participating schools are financially responsible. Lack of 
appropriate accountability and internal control increases the risk that 
program funds will not be used in accordance with program requirements. 

On the basis of our examination of supporting documentation for 
payments for a random sample of 50 students receiving scholarships at 
25 schools in school year 2005-2006, we found the following problems 
with scholarship payments made for 46 of the 50 students. Payments for 
some students had more than one of these problems. 

* For 39 of the 50 student files we reviewed, WSF had paid fees based 
on incomplete fee approval forms from the participating schools, which 
lacked authorizing signatures from the participating school, dates, or 
both. Participating schools submit these forms to obtain WSF's approval 
of fees that are not included in the school's published list of tuition 
and fees. According to WSF's procedures, these forms must be reviewed 
and approved by WSF officials before payment can be made. Without 
having properly approved fee information, WSF is at risk of paying 
unapproved fees. 

* For 9 of the 50 students, we found related student or school files 
were incomplete. Some of the items that were missing included 
disbursement and payment detail reports as well as other documents such 
as school placement letters. Disbursement reports provide a summary of 
the tuition and fees that have been billed to the student's scholarship 
account for the year. Payment detail reports list all tuition and fees 
for each OSP student attending the participating school, and the 
correctness of the information on these reports is to be certified by 
officials of the participating schools. Without this information, we 
could not assess whether the controls were operating as intended. 

* In 23 of the 50 student files we reviewed, disbursement reports did 
not include the WSF student identification number next to the student's 
name or the student identification or name was incorrect. Although WSF 
maintains that its student identification numbers are not used as key 
identifiers, these student numbers are the only numbers that link the 
disbursement reports and the scholarship checks. Without a consistent 
student identifier to link students in the program with the actual 
payments being made, WSF is at risk of losing accountability for 
specific payments made for individual students in the program. 

* We found a scholarship payment that WSF paid for a fee that was not 
included on the payment detail report and a fee payment for another 
student that was based on a payment detail report that had not been 
certified by a school official. Without this certification, WSF 
officials did not have assurance from the participating school that the 
payment was made based on correct and complete information. 

Some Payments Are Not in Compliance with the Act, and Others Raise 
Questions about Compliance: 

Under the Act, the grantee is to use the funds to provide eligible 
students with scholarships to pay tuition, fees, and transportation to 
enable students to attend the D.C. private school of their choice. 
Neither the Act nor its legislative history defines the terms 
"tuition," "transportation," or "fees." According to Education 
officials, the Department left the determination of what fees are to be 
included in a scholarship payment to WSF. WSF established the policy 
that any fee for an activity, service, or product that contributes to 
the academic success of a student is allowable if two key criteria are 
met. One criterion is that fees paid must be for services offered to 
all similarly situated students, not just OSP students. The other 
criterion was that the rates of all fees for OSP students must be the 
same as rates charged for similarly situated, non-OSP students. WSF 
commonly uses OSP scholarship funds to pay fees for school uniforms, 
books, cafeteria services, tutoring, before-and-after care, physical 
education, and enrichment (music, dance, and art). 

During the 2006-2007 school year, WSF offered scholarships up to $7,500 
to students enrolled in schools that do not normally charge students 
tuition. For school year 2006-2007, WSF offered such scholarships to 
about 30 students. At these schools, families of non-OSP students 
typically pay a small monthly fee as a sign of commitment to the 
school, but tuition and other expenses are supported by private 
donations to the school. OSP is providing scholarship payments of 
$7,500 per year to pay tuition for OSP scholarship children to attend 
these schools while the families of non-OSP students pay no tuition. 
Section 307(a)(1) of the Act requires that the grantee ensure that the 
amount of any tuition or fees that the school customarily charges OSP 
students not exceed the amount of tuition or fees that the school 
customarily charges non-OSP students. In reviewing OSP scholarship 
payments, we found that WSF's practice of paying for tuition 
scholarships to schools that do not normally charge students tuition is 
a violation of the Act because the Act provides that the tuition 
charged OSP students cannot be more than the tuition charged for 
similarly situated non-OSP students. The small monthly fee, because it 
is charged to all students, would be an allowable fee under the OSP. 
WSF maintains that there is no violation of the Act because students 
attending these schools tuition-free are supported by scholarship 
dollars in the same way as those who receive OSP funds. 

During school year 2005-2006, WSF was billed for before-and-after 
school care for 31 of the 50 students whose payments we reviewed. On 
the basis of the documentation that WSF provided to us for the 25 
schools in our sample that offered before-and-after care, in 22 cases 
we could not determine whether these services included any academic 
support activities. Absent this characteristic, before-and-after care 
would not meet the requirement that a fee must contribute to a 
student's academic success. 

While OSP Students Generally Met the Program's Eligibility 
Requirements, WSF Had Limited Success Meeting Recruiting Priorities and 
Did Not Provide Parents Complete Information: 

WSF had limited success in recruiting students from schools in need of 
improvement, ensuring private school placement opportunities and 
quality, and providing parents accurate information on private schools. 
Students who were offered OSP scholarships generally met the program's 
income eligibility requirements, and applicants from schools in need of 
improvement had priority for receiving scholarships, but low numbers of 
scholarship recipients came from such schools. While a large number of 
schools were recruited to participate, they varied widely in the 
numbers of placements they made available and characteristics 
associated with quality, such as teachers' qualifications, and 
information WSF provided parents was not always complete and accurate. 

Grantee Recruited Eligible Students, but Low Numbers Were from Schools 
in Need of Improvement: 

Data reported to the grantee generally indicated students who were 
offered scholarships met the financial eligibility criteria. Despite 
considerable outreach efforts and priority given to students from 
schools in need of improvement, they were underrepresented among 
recipients of scholarship offers relative to their presence in the 
population of District students, owing to both challenges in recruiting 
applicants and school openings. 

Under the terms of the memorandum of understanding between Education 
and the District of Columbia regarding the operation of the program, 
the District, in consultation with Education, agreed to identify a 
community-based nonprofit organization to conduct activities, including 
outreach to parents and private schools, prior to the award of the 
grant in order to promote a successful launch of the program for the 
2004-2005 school year. This entity--Fight for Children--met with 
private school officials, collected preliminary information about these 
schools, and initiated community outreach activities. After the grant 
was awarded, WSF assumed responsibility for these efforts and built 
upon them by engaging in such activities as holding community meetings, 
running radio advertisements, conducting mail campaigns, and reaching 
out to minority communities. However, WSF encountered obstacles in 
recruiting students from schools in need of improvement, and less than 
a quarter of all students who were offered scholarships by 2006 came 
from these schools. In fact, 4 percent of students who were offered 
scholarships the first year were from schools in need of improvement, 
while about 11 percent of District students attended regular public and 
charter schools in need of improvement that year. Table 4 describes 
some of WSF's outreach activities, and figure 3 shows examples of some 
outreach materials used. 

Table 4: Types of Outreach Methods Used to Reach Potentially Eligible 
Families: 

Method: Paid advertising; 
Examples: 
* Direct mail; 
* Newspapers; 
* Public transportation ads; 
* Movie theater advertisements; 
* Radio. 

Method: Neighborhood meetings at a variety of locations; 
Examples: 
* District of Columbia Armory and convention center; 
* Community centers; 
* Churches; 
* Housing developments; 
* Libraries; 
* Public housing; 
* School-sponsored events; 
* Homes. 

Method: Direct mail to targeted audiences; 
Examples: 
* Temporary Assistance for Needy Families recipients; 
* Families with students attending District of Columbia public schools 
designated as in need of improvement; 
* Commercial list of 27,000 low-income families; 
* District of Columbia Public School and charter school mailings. 

Method: Posters at a variety of locations; 
Examples: 
* Community centers; 
* Clinics; 
* Churches; 
* Homeless shelters; 
* Government services intake offices; 
* Outreach centers targeting minority populations; 
* Retail outlets (grocery stores, beauty salons). 

Method: Flier distribution at a variety of locations; 
Examples: 
* Schools in need of improvement; 
* Public transportation stops; 
* Targeted neighborhoods; 
* Retail outlets (grocery stores, beauty salons). 

Method: Office accessibility; 
Examples: 
* Evening and weekend hours; 
* Satellite offices; 
* Bilingual interpreters, primarily Spanish; 
* Volunteers; 
* School and application fairs. 

Source: GAO summary of WSF information. 

[End of table] 

Figure 3: Examples of Outreach Materials Grantee Used To Recruit Low- 
Income Families: 

This figure shows a combination of different outreach materials used to 
recruit low-income families.

[See PDF for image] 

Source: Washington Scholarship Fund. 

[End of figure] 

Various factors hindered WSF's efforts to recruit more students from 
schools in need of improvement for the first year, according to WSF 
officials. Officials said that many families were skeptical about the 
program--stating that it was "too good to be true" or that private 
schools would be reluctant to accept their children-- and had heard 
misinformation, such as that receiving a scholarship could reduce 
social service benefits. In addition, WSF officials said they only had 
a short time after receiving the grant to recruit a large number of 
potentially eligible families and help them through the complicated 
application and school admissions process before they had to conduct 
the first lottery. 

By the second school year, 2005-2006, the percentage of students from 
schools in need of improvement who were offered scholarships increased 
to 38 percent. According to WSF officials, several factors contributed 
to this growth: More families knew about the program and were less 
skeptical than they had been in the beginning, and WSF had more time to 
recruit eligible families. In addition, the percentage of children who 
attended District public and charter schools designated as in need of 
improvement increased, therefore enlarging the pool of potentially 
eligible applicants from these schools. 

WSF decided to reduce the extent of its outreach efforts for the third 
year, 2006-2007, because of the limited availability of new 
scholarships for students.[Footnote 10] Most of the grant funds were 
needed to support students awarded scholarships in years one and two. 
According to WSF officials, conducting extensive outreach efforts when 
only a limited number of scholarships were available would have 
unrealistically raised some families' hopes of receiving scholarships. 
WSF cut back on media advertising, but mailed flyers to families with 
children in schools in need of improvement and held community meetings 
and events, such as school fairs.[Footnote 11] About 24 percent of 
students who were offered new scholarships for the third year were from 
schools in need of improvement. (See fig. 4.) 

Lotteries for scholarship awards incorporated a mechanism to afford 
required priority to students from schools in need of improvement, but 
an applicant's probability of selection was also influenced by the 
number of private school openings offered at his or her grade level. 
This is significant because private school openings were most plentiful 
at the elementary level, while improvement designations had been more 
common among middle and secondary schools. In order to give priority to 
students attending schools in need of improvement, the evaluator 
assigned highest probabilities to those students that attended District 
schools with this designation, lower priority to students from public 
schools without this designation, and, for the first year of operation, 
lowest priority to students from private schools. Because the lottery 
also awarded scholarship offers in proportion to the number of possible 
openings reported by participating schools at particular grade levels, 
students in kindergarten through grade 8 had considerably better 
chances of receiving scholarship offers than high school students 
because many more openings were available for them.[Footnote 12] While 
District middle and secondary schools were more likely to be in 
improvement status than elementary schools, private school openings at 
the secondary level were so scarce that WSF restricted application in 
later years of the program to students in the elementary grades. 

Figure 4: Percentages of Students Offered OSP Scholarships Who Attended 
Schools Designated as in Need of Improvement Were Consistently Below 
the Percentages of Students in District of Columbia Schools Who 
Attended Such Schools: 

This figure is a bar chart showing percentages of students offered OSP 
scholarships who attended schools designated as in need of improvement 
were consistently below the percentages of students in District of 
Columbia schools who attended such schools. 

The X axis represents the school/program year, and the Y axis 
represents the percentage. In 2004-2005, 4% of students offered OSP 
scholarships who were from schools in need of improvement. That same 
school year, 11% of District public and charter school students 
enrolled in schools in need of improvement. 

In 2005-2006, 38% of students of students offered OSP scholarships who 
were from schools in need of improvement. That same school year, 51% of 
District public and charter school students enrolled in schools in need 
of improvement. 

In 2006-2007. 24% of students of students offered OSP scholarships who 
were from schools in need of improvement. That same school year, 52% of 
District public and charter school students enrolled in schools in need 
of improvement. 

Source: GAO analysis of data from WSF and the District of Columbia 
Public Schools. 

Note: The District of Columbia issued its list of schools in need of 
improvement each August for the prior school year. Because WSF and the 
evaluation contractor had to conduct scholarship lotteries in the 
spring before the District issued the lists, they had to use prior year 
designations. For example, they conducted the scholarship lotteries for 
the 2004-2005 school year in spring 2004 but had to use the list of 
schools designated as in need of improvement in 2002-2003 because the 
District did not issue the 2003-2004 school list before the end of the 
school year. Accordingly, in the figure above, both bars in each year 
represent students from schools that were designated as in need of 
improvement in the previous August. 

[End of figure] 

Of the 2,845 students who were offered scholarships during the first 3 
years of the program, most were African-American, but some were 
Hispanic, Asian, and white students,[Footnote 13] and almost all met 
the income eligibility requirements.[Footnote 14] About 77 percent had 
attended District public schools, with the remainder split among 
students who had attended private schools, students who had not 
previously attended school, and students from public and private 
schools outside the District. Most students--about 80 percent--were 
enrolled in elementary and middle schools,[Footnote 15] while about 17 
percent were enrolled in grades 9-12. 

Although most students used their scholarships when they received them, 
many did not or did not continue to use them after the first year, 
potentially freeing up scholarship money for some additional 
scholarship offers, but students who had been offered scholarships 
could decide to use the scholarships at any time during the program's 
operation by enrolling or re-enrolling in a participating private 
school. About 75 percent of all students who received scholarships 
during the first 2 years used them at some point during the time they 
were available to them. Among students offered scholarships in the 
first year the program operated (cohort 1), 68 percent initially used 
their scholarship, and of those who used them initially, 69 percent 
used their scholarships all 3 years that the scholarships were 
available to them and for which data had been collected. Among students 
who were first offered scholarships in the program's second year of 
operation (cohort 2), about 71 percent initially used their 
scholarship, and of those who used them initially, 78 percent continued 
to use their scholarships for the next 2 school years. One hundred 
eleven students in cohort 1 and 5 students in cohort 2 did not use 
their scholarships when first available, but did so eventually. 

WSF had little data that described reasons students did not use their 
scholarships when they first received them or continue to use them for 
subsequent school years, but anecdotal information was available. The 
reasons reported by some parents related to family issues, such as 
personal problems, moving, and special education needs, while others 
were more logistical, such as lack of transportation, before-or after- 
care, or a convenient school. Parents also chose to enroll their 
children in District charter schools, neighborhood schools, or other 
private schools that did not participate in the program. According to 
the Parent and Student Voices May 2007 report,[Footnote 16] some 
parents also reported several factors that hindered their children's 
ability to use the scholarship, such as losing their scholarship 
because the family's income exceeded the previous 200-percent poverty 
threshold for renewing scholarships,[Footnote 17] a dearth of openings 
at the high school level, and receipt of a scholarship for only one 
child in a sibling group. 

Scholarship usage rates also varied according to the type of school the 
student previously attended and students' needs for special educational 
services. The program evaluation found that fewer students from schools 
in need of improvement and from high schools in general used their 
scholarships than other students. In addition, fewer students that 
reported having a learning or physical disability used scholarships 
than other students. 

Participating Schools Ranged Widely in the Number of Openings Available 
to Scholarship Recipients, and Few Openings Were Available at the 
Secondary Level: 

The 70 District of Columbia private schools that have participated in 
the program varied in terms of their characteristics and the number of 
OSP students they enrolled.[Footnote 18] Students who received 
scholarships ranged from none to 67 percent of the enrollment of 
participating private schools. For example, in 2006-2007, more than 
half the scholarship students were concentrated in 19 schools. Sixteen 
schools had accepted between 0 and 5 scholarship students. Four schools 
accepted only students who they had previously accepted. 

Some students who received scholarships had limited choices, 
particularly students in the upper grades and those who wanted to 
attend a secular school. For example, according to WSF data for school 
year 2005-2006, only about 70 openings were available at the high 
school level (compared to about 650 for students in kindergarten 
through grade 5 and about 200 for students in grades 6-8). The majority 
of scholarship students attending high schools went to one religious 
school, and WSF raised private funding to pay for the tuition above the 
scholarship cap. In addition, students who desired a secular school had 
a limited number to choose from, since most of the participating 
private schools were Catholic or Protestant, and these schools offered 
most of the openings. The remaining schools included some that were 
Afro-centric or Muslim, or offered only early childhood education. 

About 88 percent of all scholarship users attended schools with 
tuitions below the $7,500 cap. Although tuition rates varied, only 3 
percent attended the most expensive schools that charged $20,000 or 
more, perhaps because these schools or private donations could not make 
up the difference between the tuition and the $7,500 cap or because 
these schools had financial aid programs of their own on which they 
could draw that were not subject to OSP financial and residential 
eligibility requirements. In general, schools offering high school 
placements had higher tuitions than other schools. (See table 5.) 

Table 5: Summary of Tuition Rates by Grade Level for Participating 
Private Schools, School Year 2006-07: 

Tuition: $2,100-5,000; 
Elementary: Number of schools: 4; 
Elementary: Number of OSP students: 76; 
Elementary and middle: Number of schools: 18; 
Elementary and middle: Number of OSP students: 999; 
Elementary through high school: Number of schools: 1; 
Elementary through high school: Number of OSP students: 67. 

Tuition: $5,001-7,500; 
Elementary: Number of schools: 4; 
Elementary: Number of OSP students: 95; 
Elementary and middle: Number of schools: 7; 
Elementary and middle: Number of OSP students: 190; 
Elementary through high school: Number of schools: 2; 
Elementary through high school: Number of OSP students: 97. 

Tuition: $7,501-10,000; 
Elementary: Number of schools: ; 
Elementary: Number of OSP students: ; 
Elementary and middle: Number of schools: 2; 
Elementary and middle: Number of OSP students: 15; 
Elementary through high school: Number of schools: ; 
Elementary through high school: Number of OSP students: . 

Tuition: $10,001-15,000; 
Elementary: Number of schools: ; 
Elementary: Number of OSP students: ; 
Elementary and middle: Number of schools: 1; 
Elementary and middle: Number of OSP students: 5; 
Elementary through high school: Number of schools: ; 
Elementary through high school: Number of OSP students: . 

Tuition: $15,001-20,000; 
Elementary: Number of schools: 2; 
Elementary: Number of OSP students: 4; 
Elementary and middle: Number of schools: 1; 
Elementary and middle: Number of OSP students: 25; 
Elementary through high school: Number of schools: ; 
Elementary through high school: Number of OSP students: . 

Tuition: $20,000 +; 
Elementary: Number of schools: 1; 
Elementary: Number of OSP students: 1; 
Elementary and middle: Number of schools: 1; 
Elementary and middle: Number of OSP students: 3; 
Elementary through high school: Number of schools: 3; 
Elementary through high school: Number of OSP students: 38. 

Total; 
Elementary: Number of schools: 11; 
Elementary: Number of OSP students: 176; 
Elementary and middle: Number of schools: 30; 
Elementary and middle: Number of OSP students: 1,237; 
Elementary through high school: Number of schools: 6; 
Elementary through high school: Number of OSP students: 202. 

Middle or junior high: Number of schools: [Empty]; 
Middle or junior high: Number of OSP students: [Empty]; 
Middle and high school or high school only: Number of schools: [Empty]; 
Middle and high school or high school only: Number of OSP students: 
[Empty]; 
All schools: Number of schools: 23; 
All schools: Number of OSP students: 1,142. 

Middle or junior high: Number of schools: [Empty]; 
Middle or junior high: Number of OSP students: [Empty]; 
Middle and high school or high school only: Number of schools: [Empty]; 
Middle and high school or high school only: [Empty]; 
All schools: Number of schools: 13; 
All schools: Number of OSP students: 382. 

Middle or junior high: Number of schools: 1; 
Middle or junior high: Number of OSP students: 15; 
Middle and high school or high school only: Number of schools: 1; 
Middle and high school or high school only: Number of OSP students: 72; 
All schools: Number of schools: 4; 
All schools: Number of OSP students: 102. 

Middle or junior high: Number of schools: 1; 
Middle or junior high: Number of OSP students: 12; 
Middle and high school or high school only: Number of schools: 2; 
Middle and high school or high school only: Number of OSP students: 14; 
All schools: Number of schools: 4; 
All schools: Number of OSP students: 31. 

Middle or junior high: Number of schools: 1; 
Middle or junior high: Number of OSP students: 3; 
Middle and high school or high school only: Number of schools: 4; 
Middle and high school or high school only: Number of OSP students: 5; 
All schools: Number of schools: 8; 
All schools: Number of OSP students: 37. 

Middle or junior high: Number of schools: [Empty]; 
Middle or junior high: Number of OSP students: [Empty]; 
Middle and high school or high school only: Number of schools: 1; 
Middle and high school or high school only: Number of OSP students: 2; 
All schools: Number of schools: 6; 
All schools: Number of OSP students: 44. 

Middle or junior high: Number of schools: 3; 
Middle or junior high: Number of OSP students: 30; 
Middle and high school or high school only: Number of schools: 8; 
Middle and high school or high school only: Number of OSP students: 93; 
All schools: Number of schools: 48; 
All schools: Number of OSP students: 1,738. 

Source: GAO analysis of WSF data. 

Note: In addition, three students attended preschools with tuition 
between $7,501 and $10,000, and two students attended a preschool with 
tuition between $5,001 and $7,500. 

[End of table] 

While the Act requires WSF, as the grantee, to ensure that 
participating schools are financially responsible and are using the 
received funds effectively, WSF did not fulfill its oversight role of 
ensuring that participating schools are financially stable. WSF 
designed letters of agreement that detailed the requirements for 
participating schools, including the need to show evidence of financial 
stability on an annual basis. However, for all 25 schools attended by 
the 50 students in our random sample, we did not find certain documents 
that should have been submitted by the schools showing financial 
stability or that a financial audit had been completed. 

Proof of financial stability is critical for providing assurance that a 
participating school can maintain operations as an ongoing entity and 
that OSP students can continue enrollment in those schools. During 
school year 2006-2007, one school that enrolled 25 scholarship users 
announced closure due to bankruptcy. Two schools that were members of 
the Center City Consortium, a group of Catholic schools run by the 
Catholic Archdiocese of Washington, closed in school year 2007-2008 due 
to financial problems, but the 134 scholarship users affected by the 
closures were offered placements in other consortium schools. 
Presently, the archdiocese is exploring the possibility of converting 8 
of the 12 schools in the consortium to charter schools by school year 
2008-2009. 

The extent to which private schools reported characteristics associated 
with high-quality educational programs varied. We found that, as 
reported in the School Directory, D.C. Scholarship Program: 2004- 
05,[Footnote 19] at least 3 of 52 schools that participated that year 
indicated that at least half of their teachers did not have at least a 
bachelor's degree, and 6 schools indicated that about 10 to 20 percent 
of their teachers lacked at least a bachelor's degree. Further, many of 
the schools were not accredited, and there is no evidence that they 
submitted evidence of educational soundness acceptable to the Board. 

The MOU and the Act limit participation to private schools operating 
lawfully in the District. Yet, neither Education, the District of 
Columbia, nor WSF ensure that all participating private schools meet 
basic requirements for operating legally in the District of Columbia, 
and some schools have not met these requirements. WSF relies on school 
officials to self-certify that they are operating lawfully, and the 
District had not reviewed any schools to determine whether they had met 
the District's requirements. We selected a non-probability sample of 18 
schools using various criteria, including whether or not schools had 
registered with the District's Department of Consumer and Regulatory 
Affairs and whether or not schools were accredited by an agency 
recognized by the District and asked the District to supply copies of 
their occupancy certificates to us. District officials provided 
documentation indicating that 3 of 18 schools we selected for review 
for certificates of occupancy lacked them; 6 had permits that did not 
specify use as a private school, child development center, or before 
and after school care center; and 7 of the 18 appeared to have 
occupancy permits that designated use as child development centers with 
before and after school care for school-age children, but did not 
reflect the operation of a private school. One school where OSP 
students constituted over 60 percent of enrollment applied for an 
occupancy permit for operating as a private school in March 2007 after 
it had participated in OSP for 3 years. Because District officials 
could not find any reports on file, they concluded several schools may 
not have submitted required annual reports of operation providing basic 
information on their curriculum, teachers' education, accreditation, 
and school facilities. These officials also reported that the District 
government had not previously played an active role in overseeing the 
operations of WSF. Figure 5 shows examples of schools participating in 
OSP that District officials reported did not have certificates of 
occupancy specifying use as an educational facility on file with the 
Department of Consumer and Regulatory Affairs. 

Figure 5: Examples of Schools That Operated without a Certificate of 
Occupancy Specifying Use as a Private Day School: 

This figure is a combination of two photographs showing schools that 
operated without a certificate of occupancy specifying use as a private 
day school. 

[See PDF for image] 

Source: District of Columbia government. 

[End of figure] 

According to WSF officials, they had conducted school visits at 42 of 
the 58 schools with OSP students. However, we found only one written 
report documenting a visit to one of these schools. Selection of 
schools to visit was based partly on certain triggers, such as parent 
complaints. WSF officials told us that they had several versions of a 
school review form used to document visits and provided us a copy of 
the form used at the beginning of the 2005-2006 school year. However, 
they informed us that the use of this form had been discontinued 
shortly afterward. Site visits such as WSF's school visits are an 
important control, especially given that schools self-certify that they 
meet all regulations established for private schools under District 
law. Self-certification without review to verify that the 
certifications are factual increases the risk that federal funds 
intended to allow children from low-income families to attend private 
schools will result in some students attending schools that are not in 
compliance with District law. 

Information WSF Provided to Help Parents Select Private Schools Was 
Incomplete and Contained Inaccuracies: 

The Act required the grantee to provide parents with information that 
would help them make informed decisions about selecting a school for 
their child. The MOU between Education and the District of Columbia 
provided that the grantee was to provide certain specific information 
about the school to parents, including the qualifications of the 
school's teachers, the achievement of its students, and its safety and 
environment. The provision of information is important since many 
students who were offered OSP scholarships had little experience with 
private schools. Parents also said they would like to have information 
that would help them evaluate private schools.[Footnote 20] 

Although WSF compiled an annual directory to help parents during the 
selection process, it did not collect or omitted or incorrectly 
reported some information that would have helped parents evaluate the 
quality of participating schools.[Footnote 21] WSF did not provide 
information about the achievement of each school's students in order to 
help parents select appropriate schools for their children. However, it 
did require schools to certify that they had given parents of enrolled 
OSP students information on their children's academic progress and 
overall school safety.[Footnote 22] 

WSF also requested some information from the schools that it did not 
provide that could have helped parents during their decision-making 
process, most notably information on a school's accreditation status. 
In addition, some information WSF did provide to parents may have been 
misleading. Some schools reported to WSF that they had such facilities 
as a library or gymnasium, although the information they reported to 
the District indicated no such facilities. 

Finally, WSF incorrectly reported information on some schools that 
could have significantly affected parents' choice of schools, primarily 
the percentage of teachers who had at least a bachelor's degree and 
tuition rates. For example, the 2007-2008 school directory reported 
that less than half of teachers in five schools and between 51 and 75 
percent of teachers in another five schools had at least a bachelor's 
degree, although the 2004-2005 directory reported that 100 percent of 
teachers in these schools had at least a bachelor' degree. In addition, 
the school directory for 2007-2008 reported that the tuition for at 
least three private schools was between $10,000 and $15,000. However, 
in verifying tuitions listed in the directory by comparing them to 
tuitions cited in other sources, we learned these schools do not charge 
non-OSP students tuition and typically charge only a modest fee, such 
as $25 per month.[Footnote 23],[Footnote 24] The tuitions listed for 
these schools may have motivated parents to obtain scholarships in the 
belief that they could not otherwise afford them. 

Evaluation Contractor Adopted a Strong Methodology, but Program 
Implementation and Other Factors Limited the Usefulness and 
Generalizability of Findings: 

The evaluation contractor adopted an evaluation design that used random 
assignment to determine which applicants were offered scholarships and 
which were not, but program implementation decisions and other factors 
limited the usefulness and generalizability of the evaluation's 
findings regarding the effects of scholarship offers on the achievement 
of low-income, District public school students. The contractor adopted 
a random assignment evaluation design that enabled comparisons between 
students from District public schools who were offered scholarships 
(the treatment group) and those that were not (the control group) and 
was designed to measure the effect of scholarship offers on applicants 
who had attended public schools. The contractor used statistical 
techniques to estimate the effects of using such a scholarship. The 
methodology adopted was a strong experimental methodology that could 
have produced an evaluation that reflected statutory requirements. 

Many factors have limited the evaluator's ability to make the 
achievement comparisons described in the statute and have complicated 
interpretation of the evaluation's findings or their usefulness and 
generalizability. For example, changes in the District's achievement 
testing program and the scarcity of placements offered by participating 
high schools limited the evaluators' ability to make comparisons that 
were of interest to Congress between students offered scholarships and 
their peers in District schools, including comparisons with regard to 
graduation rates, dropout rates, and college placement. In addition, 
although the Act directed the evaluator to determine the effects of the 
program on District regular public schools, the opportunities for 
choice within the public school sector grew over the same time frame, 
making it very difficult, if not impossible, to disentangle the effects 
of this program on regular public schools from the effects of charter 
schools, special schools, and magnet and special programs that serve 
far more students. Along with changes in the District's testing 
program, limited number of scholarship placements at the high school 
level, and simultaneous growth of other choice options, other factors 
may have limited the evaluator's ability to make requested comparisons 
and to detect statistically significantly differences and generalizable 
results. These factors include the inability to include some 
scholarship recipients in the evaluation, crossover between treatment 
and control groups, concentration of scholarship students in a subset 
of schools, and the combination of academic and social supports with 
scholarship offers. Given these differences, some comparisons relied on 
limited data. 

Changes in the District's testing program. Changes in the District's 
testing program made it impractical for the contractor to perform some 
comparisons that Congress had requested, greatly increased the cost of 
testing, and will likely continue to decrease the quantity and 
sufficiency of the follow-up achievement test data. At Education's 
urging, the District changed its standardized achievement testing 
program in spring 2006 from a norm-referenced test to a criterion- 
referenced test to better comply with the requirements of the No Child 
Left Behind Act, but this had major effects on the evaluation. The norm-
referenced test, like all norm-referenced tests, had compared students' 
knowledge with respect to the knowledge of other students and sorted 
and ranked students according to how much they know in comparison to 
other students. Because students are ranked against each other, and 
normalized on a bell curve, norm-referenced tests are constructed so 
that scores at different grade levels are vertically equivalent and 
scores of children at various grade levels can be combined to draw 
comparisons. For example, a second and an eleventh grader who received 
a percentile score of 75 would have equivalent scores because both 
surpassed the scores of 75 percent of their grade peers. In contrast, 
the new criterion-referenced tests, like all tests of this type, 
measure whether students have mastered certain skills and concepts 
and/or what they have learned in certain grades or classes. Scores on 
criterion referenced tests do not yield equivalent scores that can be 
readily combined to draw comparisons across grade levels. 

Although the Act directed the evaluator to compare the achievement of 
students offered scholarships with students of the same grade in public 
schools and the achievement of scholarship users with students of the 
same grade in public schools, the evaluation contractor could not 
reasonably or cost-effectively make these comparisons. The District's 
new testing program meant a common test that yielded comparable scores 
for all grades, from the year eligible applicants entered the program 
through the duration of the program, was no longer available, and 
administering two testing programs to all students in the District 
would not be economically or educationally feasible. 

In addition, the change in the District's testing program made it 
necessary for the evaluation contractor to expand the number of 
students it planned on testing from only those in the treatment group 
to also include those in the control group in order to make comparisons 
between these groups because students in the control group would no 
longer be participating in the norm-referenced testing as part of their 
public school experience. According to Education officials, this 
increased the cost of testing and outreach efforts associated with the 
testing to a total of $800,000 to $1 million per year. 

The change in tests will also make it difficult to make certain other 
potentially useful achievement comparisons, particularly with respect 
to students who do not use their scholarships and students in the 
control group. Although threats to withhold scholarship funds have been 
generally effective in securing the participation of scholarship users 
in testing sessions, no punitive measures are linked to a failure to 
participate for the other groups. The rewards offered--cash gifts, free 
tax preparation, and an opportunity to enter a lottery to receive a 
second chance for a scholarship--have not been sufficient to motivate 
many nonusers to attend Saturday testing sessions or to engage in 
appropriate test-taking behavior such as completing subtests and 
remaining for the duration of the test. Missing test data, as well as 
the number of students who fail to participate in testing, will likely 
increase over time, reducing the comparability of the treatment and 
control group. Although statistical techniques, such as weighting and 
imputation, can be used to adjust for missing scores, these techniques 
cannot guarantee findings would not be biased when the unweighted 
response rate--that is, the number of students for whom scores are 
available--falls to very low levels. (See table 6.) 

Table 6: Unweighted Percentage of Students with Missing Achievement 
Test Scores at Baseline and after 1 Year of Program Participation: 

Rates in percent. 

Cohort 1; 
Reading: Treatment: Missing achievement scores at baseline: 23.3; 
Reading: Control: Missing achievement scores at baseline: 14.4; 
Math: Treatment: Missing achievement scores at baseline: 13.0; 
Math: Control: Missing achievement scores at baseline: 23.8; 
Both reading and math: Treatment: Missing achievement scores at 
baseline: 13.0; 
Both reading and math: Control: Missing achievement scores at baseline: 
23.3. 

Cohort 2; 
Reading: Treatment: Missing achievement scores at baseline: 25.1; 
Reading: Control: Missing achievement scores at baseline: 31.0; 
Math: Treatment: Missing achievement scores at baseline: 13.6; 
Math: Control: Missing achievement scores at baseline: 22.7; 
Both reading and math: Treatment: Missing achievement scores at 
baseline: 12.8; 
Both reading and math: Control: Missing achievement scores at baseline: 
22.4. 

Cohort 1; 
Reading: Treatment: Missing achievement scores after 1 year of program 
participation: 21.7; 
Reading: Control: Missing achievement scores after 1 year of program 
participation: 42.0; 
Math: Treatment: Missing achievement scores after 1 year of program 
participation: 21.7; 
Math: Control: Missing achievement scores after 1 year of program 
participation: 42.0; 
Both reading and math: Treatment: Missing achievement scores after 1 
year of program participation: 21.7; 
Both reading and math: Control: Missing achievement scores after 1 year 
of program participation: 42.0. 

Cohort 2; 
Reading: Treatment: Missing achievement scores after 1 year of program 
participation: 23.6; 
Reading: Control: Missing achievement scores after 1 year of program 
participation: 35.2; 
Math: Treatment: Missing achievement scores after 1 year of program 
participation: 20.8; 
Math: Control: Missing achievement scores after 1 year of program 
participation: 30.4; 
Both reading and math: Treatment: Missing achievement scores after 1 
year of program participation: 20.3; 
Both reading and math: Control: Missing achievement scores after 1 year 
of program participation: 29.5. 

Source: GAO analysis of Westat data. 

[End of table] 

Scarcity of high school placements. The high schools participating in 
the program offered very few placements, thus limiting the number of 
high school students and graduates in the evaluation and hampering the 
ability of the evaluator to make statistically meaningful comparisons 
of dropout, graduation, and college admission rates of scholarship 
recipients and nonrecipients. (See table 7.) For example, according to 
data provided by WSF, the program graduated only 7 students from high 
school in its first 2 years of operation, too small a number to make 
sound comparisons. In addition, the low numbers of secondary openings 
could indirectly exacerbate attrition from the evaluation if, as 
scholarship users advance through the grade levels, they find 
placements in higher grades are not available and become less motivated 
to participate in data collection necessary for the evaluation. 

Table 7: Number of Total Private School Placements for OSP Students: 

Year: 2004-2005; 
Kindergarten: 296; 
Grades 1-5: 977; 
Grades 6-8: 397; 
Grades 9-12: 68; 
Total: 1,738. 

Year: 2005-2006; 
Kindergarten: 169; 
Grades 1-5: 474; 
Grades 6-8: 192; 
Grades 9-12: 72; 
Total: 907. 

Source: GAO analysis of Washington Scholarship Fund data. 

[End of table] 

Simultaneous Growth of Other Choice Options. The Act directs the 
evaluator to determine the effects of the program on District public 
schools, but the opportunities for choice within the public school 
sector, such as charter schools and magnet programs, are far more 
numerous than those offered by OSP and have grown over the same time 
period, making the determination of OSP's independent effect on public 
schools, if any, very difficult, if not impossible, to assess. The 
evaluation design was based largely on an assumption that the award of 
an opportunity scholarship would affect student achievement and public 
schools by expanding school choice to include higher performing 
schools. This assumption, however, does not account for the large 
number of public charter schools, magnet schools, and other specialized 
programs available through the District of Columbia as well as 
scholarships offered by individual private schools or other private 
organizations. Because these opportunities are far more numerous than 
those offered by OSP and expanded in a similar time frame, it would be 
difficult to attribute changes in the behavior of traditional schools 
uniquely to OSP. Nor were participating private schools necessarily 
limited to those that might be higher performing. Moreover, the large 
portion of some schools' enrollment composed of OSP students suggests 
that, in some instances, program effects, if any, may be more profound 
for the participating schools that the scholarship users attend than 
for the public schools from which the students departed. 

Inability to include some scholarship recipients in the evaluation. In 
the first year of the program, Education and WSF decided to process 
applications in May and hold the placement lottery in August. At that 
time, however, the program was undersubscribed at the kindergarten 
through grade 5 level; that is, the number of eligible applicants from 
public schools who attended kindergarten through grade 5 was less than 
the number of private school openings available. Consequently, 
Education and WSF gave scholarships to approximately all 800 eligible 
public school applicants in these grades, thus eliminating them from 
the evaluation. In addition, only 55 eligible students from schools in 
need of improvement at the middle and high school levels applied that 
year, and Education and WSF decided to offer these students 
scholarships on a non-random basis, thereby eliminating these 55 
students from the evaluation. Education and WSF further reduced the 
number of students in the cohort 1 evaluation pool by awarding 
scholarships to about 200 students who were already attending private 
schools. Although the Act did not specifically prohibit scholarship 
offers to students in private schools, the evaluation was designed to 
make comparisons among students who had attended only District public 
schools.[Footnote 25] Therefore, the contractor excluded them from the 
evaluation so as not to reduce the usefulness of the evaluation 
findings. As a result of these decisions, of the 2,454 students who 
were offered scholarships in the first 2 years of the program, only 
1,387--less than 60 percent---could be included in the evaluation. (See 
table 8.) 

Table 8: Eligible Applicants Offered and Not Offered Scholarships and 
Numbers Included in and Excluded from the Evaluation: 

Cohort: 1; 
Eligible applicants: Offered scholarship: 1,366; 
Eligible applicants: Not offered scholarship: 482; 
Eligible applicants: Total: 1,848; 
Number included in evaluation: Offered scholarship (treatment): 299; 
Number included in evaluation: Not offered scholarship (control): 193; 
Number included in evaluation: Total: 492; 
Number excluded from evaluation: Offered scholarship: 1,067; 
Number excluded from evaluation: Not offered scholarship: 289; 
Number excluded from evaluation: Total: 1,356. 

Cohort: 2[A]; 
Eligible applicants: Offered scholarship: 1,088; 
Eligible applicants: Not offered scholarship: 728; 
Eligible applicants: Total: 1,816; 
Number included in evaluation: Offered scholarship (treatment): 1,088; 
Number included in evaluation: Not offered scholarship (control): 728; 
Number included in evaluation: Total: 1,816; 
Number excluded from evaluation: Offered scholarship: 0; 
Number excluded from evaluation: Not offered scholarship: 0; 
Number excluded from evaluation: Total: 0. 

Cohort: Total; 
Eligible applicants: Offered scholarship: 2,454; 
Eligible applicants: Not offered scholarship: 1,210; 
Eligible applicants: Total: 3,664; 
Number included in evaluation: Offered scholarship (treatment): 1,387; 
Number included in evaluation: Not offered scholarship (control): 921; 
Number included in evaluation: Total: 2,308; 
Number excluded from evaluation: Offered scholarship: 1,067; 
Number excluded from evaluation: Not offered scholarship: 289; 
Number excluded from evaluation: Total: 1,356. 

Source: GAO analysis of Westat data. 

[A] Numbers for cohort 2 exclude eligible private school students 
because the evaluation contractor assigned them a zero percent 
probability of receiving a scholarship offer. 

[End of table] 

Because of the limited size of the treatment and control groups, the 
evaluation contractor combined the cohort 1 and cohort 2 treatment and 
control groups to make comparisons after year one. Future evaluations 
will compare outcomes for students in these cohorts after they have 
participated in the program for 2 and for 3 years. 

Crossover. Crossover between treatment and control groups can reduce 
the effectiveness of randomization and can make comparisons between 
treatment and control groups difficult to interpret and generalize. 
While scholarship offers made to applicants in the treatment group were 
not withdrawn even if they failed to find a private school placement, 
applicants assigned to the control group sometimes found other means to 
attend private school. In fact, 15 percent of students in the control 
group who provided math scores were enrolled in private schools. The 
evaluation contractor used statistical adjustments to bound its 
estimates of treatment effects for both control group students who 
attended private schools and those who could not or did not use their 
scholarships. However, the contractor did not consider potential 
selection effects as a result of the inability of students to find an 
acceptable placement as opposed to declining the use of a scholarship 
for some other reason. While some students in the treatment group did 
not use their scholarships or did not do so consistently and some 
students in the treatment group who wanted to use their scholarships 
could not because no school, or no school of their choice had openings 
or would accept them, the number who did not use scholarships 
specifically because they were unable to find acceptable placements is 
unknown. 

Concentration of scholarship students in a subset of schools. Because 
private schools--which differ greatly in motivation, capacity, and 
ability to accept OSP students--determine the number of OSP students to 
accept, large numbers of scholarship users have been clustered within a 
small subset of private schools. Additionally, OSP students constituted 
60 percent or more of total reported enrollment in three participating 
schools in school year 2006-2007. The evaluation contractor did 
consider the potential influence of clustering of students within 
families and in baseline schools prior to OSP participation on variance 
estimates, which are used in determining the effects of scholarship 
offers and scholarship use. However, the evaluation does not discuss 
the potential impact of clustering large percentages of OSP students 
within relatively few schools. Such clustering of evaluation 
participants within a small number of schools has the potential to 
confound program and school-level effects and may distort differences 
between the comparison groups unless appropriate statistical 
methodologies are employed to disentangle these effects when analyzing 
the effect of receiving a treatment. 

Combining scholarship offers with other supports. In addition to 
offering scholarships, the program provides scholarship students with 
additional individual supports, which may include case management and 
parent empowerment services from the grantee, summer school, tutoring, 
remedial classes, before-and-after school care, and mentoring. 
Throughout their participation in the program, OSP students receive 
different types and combinations of these services that were not 
readily available to students in the control group, bringing into 
question the comparability of students in the treatment and control 
groups and raising problems in generalizing the study's findings to 
circumstances in which a scholarship is the only treatment offered. 

Some Required Comparisons Relied on Limited Data: 

The Act required the contractor to compare the safety of schools 
attended by participants in the program and the schools attended by 
students who do not participate in the program and to evaluate the 
success of the program in expanding choice options for parents. In its 
evaluation of effects after year one, the contractor relied exclusively 
on self-reported data to compare the safety of schools attended by the 
two groups, using survey responses to compare perceptions of school 
safety among parents of students offered scholarships and students 
offered scholarships with the perceptions of those who were not and 
their parents. While perceptions may be a valid measure, they do not 
necessarily reflect the safety risks students face at a given school, 
and parent and student perceptions sometimes diverged. While not 
specifically required by the Act, the contractor also used surveys to 
compare school satisfaction of (1) parents of students offered 
scholarships and (2) students offered scholarships, with (3) school 
satisfaction of parents of students not offered scholarships and (4) 
students not offered scholarships. In the evaluation report, the 
contractor cast satisfaction as "an indicator of the success of the 
program in expanding options for parents." 

Conclusions: 

The scholarship program administered by WSF has provided low-income 
families in the District an additional option to enroll their children 
in private schools. However, to maintain a program capable of using 
public funds for their intended purpose--that of providing increased 
opportunities to low-income parents to send their children, 
particularly those attending schools designated as in need of 
improvement, to private schools--the agency or organization with 
responsibility for operating a school choice program such as OSP needs 
a strong financial accountability infrastructure that incorporates a 
system of internal control. 

During the period of our review, through year three of the program, WSF 
struggled to build an accountability infrastructure that could assure 
Congress, families of participating students, and the public that its 
funds were used effectively and in compliance with laws and 
regulations. Strong accountability and internal controls over a 
scholarship program, such as OSP, are critical for ensuring that funds 
are used for their intended purposes, and to maintain fiscal viability 
and credibility. Overall financial management policies and procedures 
and the underlying systems need to provide assurance that federal funds 
are being used for the purposes intended and that funds are safeguarded 
against loss from error, abuse, and fraud. 

Without complete and accurate information about schools' performance 
and other quality indicators, parents cannot make well-informed choices 
among schools for their child. The grantee did not provide parents 
information about the achievement levels of all students in 
participating private schools and other indicators of school quality 
and, for some schools, provided inaccurate information about teacher 
qualifications and tuition levels. As a result, parents might have used 
opportunity scholarships to place their children in private schools 
that were less successful in raising achievement levels than the public 
schools their children previously attended. In other cases, parents 
might have rejected some private schools because they were given 
inaccurate information about the schools or, in the case of the few 
schools that do not customarily charge tuition, applied for a 
scholarship in the belief that their children needed scholarships to 
attend these schools. 

In the absence of an accountability mechanism to ensure that 
participating schools are operating lawfully in the District, students 
were placed at risk of attending facilities that did not meet basic 
health and safety requirements. Because OSP is a federal program that 
had the support of the District of Columbia's government, parents may 
have incorrectly assumed that Education and the city were overseeing 
the program and making sure participating schools met such standards. 
As a consequence, parents may have been less inclined to investigate 
the status of these schools. 

Finally, implementation of educational evaluations involving random 
assignment often requires substantial oversight and structure. Without 
such oversight and structure, the ability to draw valid conclusions 
about the program to improve educational policy can be compromised by 
the zeal to ensure program participants' success or to maximize access 
to a treatment that program operators and applicants may believe is 
beneficial even in the absence of empirical evidence. Indeed, 
throughout the short history of OSP, decisions made to advance program 
goals, such as maximizing the number of scholarships awarded in the 
initial year and providing a wide range of academic and social supports 
to scholarship users, have undermined the goals of the evaluation to 
produce meaningful findings and the ability of Congress to use these 
findings for decisions about other programs. 

Recommendations for Executive Action: 

Given the importance of using funds appropriately to further program 
objectives and help ensure that schools are safe and provide sound 
educational experiences, parents are given accurate information about 
schools their children may attend, and program oversight is sufficient, 
we are making the following recommendations. Specifically, we recommend 
that the Secretary of Education direct WSF, the grantee, to: 

* establish and implement detailed policies and procedures to improve 
financial controls over OSP grant funds, including specific 
requirements for the process of approving scholarship payments and 
documentation of the process; 

* establish compensating controls, such as supervisory review, to 
reduce the risk of fraud in situations where segregation of duties is 
not possible due to the size limitations of OSP's staff so that no one 
employee can authorize, process, review, and have access to the funds 
relating to OSP; 

* continue its efforts to implement an integrated financial management 
system to facilitate processing and recording of scholarship payments 
and overall financial reporting; 

* develop and implement procedures for conducting site visits, 
including that site visit reports be prepared and contain information 
on the overall financial stability of the school; and: 

* develop procedures to ensure that accurate information is provided to 
parents before a school is chosen about the summary achievement data of 
students, teacher qualifications, and tuition levels, and that schools 
make such information available on an annual basis to parents of 
enrolled students. 

We also recommend that the Secretary of Education collaborate with the 
Mayor of the District of Columbia to ensure participating schools are 
in compliance with all relevant District of Columbia education and 
safety requirements, including school accreditation and health, safety, 
and fire code requirements, and receive approval to operate in the 
District. 

Further, we recommend that the Mayor direct the Office of the State 
Superintendent of Schools to collaborate with the Board of Education to 
develop and implement procedures for ensuring that private schools in 
the District meet applicable District requirements and to actively 
participate in the oversight of OSP. Finally, we recommend that, in 
planning for future programs for which Congress has required an 
evaluation, the Secretary of Education should take steps to make 
certain the program to be evaluated is overseen to ensure it is 
implemented in a manner consistent with the evaluation design. 

Agency Comments and Our Evaluation: 

At our invitation, the Office of the Mayor of the District of Columbia 
(the District), Education, and WSF provided written comments on a draft 
of this report, which are reproduced in appendixes III, IV, and V, 
respectively, and summarized below. WSF also provided technical 
comments, which we incorporated when appropriate. In general, all three 
entities concurred with our recommendations, but Education and WSF 
disagreed with many of our findings and interpretations. 

The District agreed with our findings and recommendations. It said the 
report's findings have been extremely helpful to Mayor Fenty's 
administration as it developed its plan for moving forward to fulfill 
its responsibilities under the MOU with Education, particularly on the 
steps the District should take to ensure that parents receive accurate 
and complete information and that schools receiving funds under the 
program meet certain basic health, safety, and instructional 
requirements. The District's comments include specific ongoing or 
planned steps to ensure that WSF complies with the requirements of 
federal law and applicable local statutes and regulations. In addition, 
the Deputy Mayor for Education has directed the District's Department 
of Consumer and Regulatory Affairs to conduct inspections at the 
schools identified as having missing or inapplicable certificates of 
occupancy and is developing an accelerated schedule for reviewing the 
regulatory status of all schools receiving funds under OSP. In its 
comments, WSF supplied additional information about one of the several 
schools DCRA identified as lacking a certificate of occupancy 
reflecting operation of a private school. After verifying this 
information with DCRA, we changed the draft to reflect it. 

With one exception, Education generally agreed with our recommendations 
and stated that it would use the information in our report to continue 
to improve its oversight of the program and its coordination with the 
District. With respect to our recommendation that, in planning for 
future programs for which Congress has required an evaluation, 
Education ensure that the program is implemented in a manner consistent 
with the evaluation design, Education did not express agreement or 
disagreement. Rather, Education's comments focus on the OSP evaluation, 
indicating that we overemphasized the evaluation's challenges resulting 
from program implementation decisions and other factors and therefore 
underestimated the evaluation's utility. Specifically, Education stated 
that the inability to compare OSP students to all students does not 
undermine the more important analysis of program impact. While 
Education adopted a strong methodology for its analysis of impact, as 
required by statute, Education's comment is not responsive to our point 
that its evaluation was unable to provide a comparison between students 
offered scholarships and those in the same grades in District schools, 
as the statute mandated. Also, Education wrote that the inability to 
include some early scholarship recipients in the evaluation is not a 
major issue but also noted that the Department rejected the notion of 
excluding these early recipients altogether, "so that there would be 
sufficient samples of students to allow impacts to be estimated for 
subgroups of students." We maintain excluding about half of students 
offered scholarships from the evaluation will affect the ability of the 
evaluation to make strong comparisons. Education also wrote that other 
supports provided by WSF were not organized and were available to both 
the treatment and control groups. We disagree. According to WSF, it 
offered case management services, conducted parent empowerment 
sessions, found and worked with community organizations that provide 
mentoring and tutoring, and, on a case by case basis, funded private 
schools to provide enrichment and remedial services to support OSP 
students. We continue to believe that the challenges we discuss have 
complicated the interpretation of evaluation findings, limit 
generalizability, and merit close examination when planning for the 
evaluation of future programs. 

Education's comments also expressed disagreement with many of our 
findings. Education believes we mistakenly concluded that WSF's 
practice of paying for tuition scholarships to schools that normally 
receive donations to cover tuition violates the Act and WSF's policies. 
We disagree. As discussed in the report and as expressly confirmed in 
WSF's comments on the draft report, children who attend the three 
scholarship-only schools pay no tuition. Section 307(a) (1) of the D.C. 
School Choice Incentive Act states that, to be reimbursed under OSP, 
the tuition charged by schools may not "exceed the amount of tuition or 
fees that the school customarily charges to students who do not 
participate in the program." (emphasis added) As section 307(a) (1) 
makes clear, the amount of allowable tuition reimbursements depends 
upon the amount of tuition the schools charge to students, not what 
costs schools incur or what other funding mechanisms, such as donated 
scholarships, they use to cover those costs. 

In addition, Education stated its belief that the report does not 
present a complete and balanced picture with respect to (1) the extent 
to which OSP students previously attended schools in need of 
improvement; (2) the establishment of OSP in time for the 2004-2005 
school year; (3) responses from parents and demand for scholarships; 
and (4) participating schools' legal independence in areas such as 
hiring and establishing teacher qualifications. With respect to the 
extent to which OSP students previously attended schools in need of 
improvement, Education wrote that it would have been more accurate to 
focus on the percentage of students from schools that received an 
improvement designation during the student's first year in OSP. We 
disagree. Scholarship lottery organizers could not be expected to give 
priority to students from schools that had not yet been designated as 
in need of improvement.[Footnote 26] Moreover, regardless of which year 
is considered, the percentage of students from schools in need of 
improvement was consistently smaller among scholarship recipients than 
among the general population of District students from whom scholarship 
recipients would have been recruited. With respect to the establishment 
of OSP in time for the 2004-2005 school year, we included a timeline 
delineating the key events in implementing this program and discussed 
the short time available to implement the program in the draft we 
provided Education to review and in this report. Education indicates 
that GAO should have included information about positive responses from 
parents of scholarship students enrolled in private schools. A balanced 
picture of parental satisfaction would also include views of parents 
whose children no longer participate or comparisons to school 
satisfaction among parents who elected not to participate, but 
collecting such data would have been tangential to our objectives 
regarding mechanisms to account for program funds, results of efforts 
to meet recruiting targets, and review of the evaluation. With respect 
to demand for scholarships, as discussed in appendix I, we attempted to 
work with WSF's data on program applicants and found these data 
unreliable. Finally, Education commented that participating schools 
were allowed to use their traditional independence and that the intent 
of the law was to maximize participation of private schools in the 
program. We agree but, as noted above, maintain that participating 
schools are required to be in compliance with District education and 
safety requirements. We believe we appropriately addressed each of 
these areas and, therefore, have not made any changes based on 
Education's comments. 

Education also expressed dissatisfaction with our exit briefing and 
asserts that GAO refused to brief the Department on its proposed 
findings and recommendations prior to sending the Department the draft 
report. However, at a July 10, 2007, exit conference, the meeting that 
Education mentions in its comments, we presented our findings. The 
findings we present at exit conferences are preliminary in that they 
could change on the basis of information we receive at the exit 
conference as well as information contained in official agency 
comments. We also held exit conferences with WSF and the Office of the 
Mayor. We considered all information provided by Education, WSF, and 
the Office of the Mayor at and subsequent to the exit conferences. 
Education also noted its concern about the inappropriate disclosure to 
the news media of the report draft we had provided to Education, WSF, 
and the Office of the Mayor for comment. We share this concern. While 
we have policies and procedures designed to help prevent premature 
disclosure, we cannot ensure that drafts will not be prematurely 
released once they leave GAO's control. 

Education provided further comments: 

* Education wrote that the concentration of scholarship recipients in a 
subset of participating private schools does not interfere with 
accurate estimation of the program impact. GAO believes it is essential 
to investigate whether OSP's effects are generalizable or limited to a 
small subset of participating schools, and we note that the evaluation 
contractor's report did not explicitly discuss the potential effects of 
clustering in private schools.