Both The Wall Street Journal and The New York Times in July
jumped on the release of a new study by the National Center for Education
Statistics. The Wall Street Journal’s headline was particularly dramatic:
"Long-Delayed Education Study Casts Doubt on Value of Vouchers."
No, it doesn’t.
And it is a failure on my part, as well as a failure of the
school choice movement as a whole, that the media don’t understand why. More on
the wisdom of this below.
Taking the study entirely at face value what the report says is
this: Private school students consistently score better in math and reading on
the National Assessment of Educational Progress than public school students, but
this advantage essentially goes away if you apply a particular set of controls
for the differing student characteristics between the two sectors (things like
wealth, race, and so on).
Okay, you say, but if private schools don’t significantly
outscore public schools, what’s the point of school voucher programs or other
reforms that would give all parents access to the public or private school of
their choice? Why, in other words, is The Journal’s headline wrong?
It is wrong because the point of voucher programs is to create a
competitive education industry, and the existing population of U.S. private
schools does not constitute such an industry.
A vigorous free market in education requires that all families
have easy access to the schools of their choice (whether public or private);
that schools are not burdened with extensive regulations on what they can teach,
whom they can hire and what they can charge, etc.; that consumers directly pay
at least some of the cost of the service; that private schools not be
discriminated against financially by the state in the distribution of education
funding; and that at least a substantial minority of private schools be operated
This set of conditions does not exist in any state in the
nation. Instead, American education is dominated by a 90 percent government
monopoly that is funded entirely through taxation. The private sector occupies
the remaining 10 percent niche, is almost exclusively operated on a nonprofit
basis and is forced to charge thousands of dollars in tuition in the face of the
"free" monopoly schools that spend an average of $10,000 per pupil per year.
This is not a market. No study was necessary to point this out.
Competitive markets are characterized by innovation, inexorable
improvements in cost effectiveness and the quality of goods and services, and
the rapid growth of the most successful providers. None of this has occurred in
the U.S. private education sector, precisely because that sector does not
constitute a competitive market.
The last great innovation to transform classroom instruction
occurred during the presidency of Thomas Jefferson (the invention of the
chalkboard, around 1801). Since that time, the pace of innovation has been so
slow that a student from the mid-1800s would immediately recognize a typical
modern classroom. The most sought-after private schools enroll only about a
thousand more students today than they did a century ago. This degree of
stagnation is unheard of outside of the education sector, because it is only in
the education sector (at least in liberal democracies) that market activity has
been so thoroughly extinguished by government monopoly provision.
Hence, this study of our small, nonmarket niche of private
schools does not allow any generalization to the sort of outcomes to be expected
from a true free market in education — and the creation of such a market is the
primary justification for vouchers and other school choice policies.
If I were better at my job, and if the school choice movement as
a whole had a more effective media machine, this fact would be widely understood
and we wouldn’t see fallacious headlines like the one cited above.
That major point having been made, let’s take a look at the
study’s findings on their own merits, as an examination of the current crop of
public versus private schools.
The first problem with the study is that it collects no data on
per-pupil spending in public versus private schools. Private school tuition,
according to the NCES itself, is about half of the average public school
expenditure per pupil. While private schools have some other sources of revenue,
they still spend thousands of dollars less per pupil than public schools, even
after taking these other revenues into account. Private schools may be
dramatically more efficient even if their absolute achievement levels are
comparable to those in public schools.
Hence it is possible that if spending were equalized, private
schools would raise student learning substantially compared to current levels.
While it has been shown that spending and achievement are largely unrelated in
the public sector, this has not been demonstrated in the private sector. In
fact, evidence from developing countries suggests that higher spending in
private schools DOES increase student achievement.
Next, it is worth observing the specifics of the study’s
findings. It reports that there is a small advantage to public schools in 4th
grade math, but that this advantage is not present at the 8th grade. It further
says that at the 8th grade, private school students have a small advantage over
public school students in reading. One possible interpretation of these findings
is that public school students fall behind their peers in private schools the
longer they spend in the classroom.
That, of course, is only one possibility. At any rate, it is
clear that parents are most concerned with what their children know and are able
to do at the end of their K-12 education, so if, by the later grades, private
schools confer a significant advantage, this would definitely seem to favor
Methodological and Data Problems
All of the above discussion takes the study’s findings at face
value. This may be ill-advised, since a preliminary review suggests that there
may be real methodological problems and potentially serious data problems.
Several of the control variables used in the model seem problematic, including
The rate of student absenteeism
It is entirely possible that sectoral differences in the feeling
of community or level of personal attention, ability of school staff to motivate
students, etc., could affect student absenteeism. So it is erroneous to treat
this as exogenous (i.e., as independent of school sector) and to control for it.
This variable is clearly endogenous (i.e., affected by school
sector). Parents tend to prefer schools in which teachers know all the students
by name and which create a friendly, community atmosphere. This is much easier
in smaller schools, and hence there is a competitive pressure not to get too
large in the private education sector. No such pressure exists in the public
sector, where contrary bureaucratic incentives encourage large school size. As a
result, the average public school is roughly three times the size of the average
private school: 521 students versus 182 students. It is thus unjustifiable to
pretend that school size is independent of school sector.
The percentage of students in the Title I program
A report by the Congressional Budget Office notes that "About 97
percent of public schools and 45 percent of private schools participate in the
school lunch program." This vast difference in level of participation by schools
may have a significant effect on the share of eligible students who are in fact
being served by the program.
Sample specification problems
Between a fifth and a quarter of the private schools selected
for the study did not participate. The authors make no serious attempt to
analyze non-participants to determine how and to what extent they might differ
from participating schools in ways related to student performance. This could
bias their results in unknown ways.
It seems likely that public and private sector schools apply the
federal Specific Learning Disability label differentially. This label states
that children are disabled if they perform at a level below what would be
expected for students of their age and intelligence. It does account for the
possibility that poor performance may be the result of poor instruction. Roughly
6 percent of all public school students are placed in this category, making up
nearly half (43 percent) of all students classified as disabled in the public
sector. Among private schools participating in this study, a total of 3 to 4
percent of students are classified as suffering from ANY disability, mental or
physical. Because students classified as SLD can be excluded from the test
taking pool or given extra time or other accommodations, differential SLD
classification rates between the sectors may affect sectoral mean scores
(because these students, by definition, perform below the average of their
Tom Loveless has pointed out in a paper for the Brookings
Institution that the NAEP mathematics test does a poor job of measuring the
skills that it is purported to measure. Calculator use is allowed throughout, so
the exam does not measure basic arithmetic ability. More advanced topics, such
as algebra with fractions, are all but absent, making it a poor test of these
more advanced skills. If the two school sectors differ in either of these
important areas, the NAEP will not detect it. It is natural for scholars to want
to analyze the data they have, but readers should be aware of the shortcomings
of those data as a measure of both basic and advanced mathematical ability.
Taking all of the above analysis together, this study’s findings
would have little bearing on market-based reforms like vouchers and tax credits
even if the research were methodologically flawless. Even as a comparison of
public schools and the existing (nonmarket) crop of private schools, this study
leaves much to be desired because it neglects to consider the substantially
higher per-pupil spending of public schools.
But the study, as noted above, is not methodologically flawless.
Several of its control variables appear to be misspecified, and so its adjusted
test score averages may be significantly biased. It makes no attempt to assess
the impact of the non-participation by between a fifth and a quarter of all the
private schools selected for participation in the study — another probable
source of bias. And the study uses a mathematics test (the NAEP) that has been
shown to do a poor job of assessing both basic arithmetic and more advanced
mathematical skills, thus obfuscating possible differences in performance in
these (rather important) areas among the students tested.
In a nutshell: This study does not say what some reporters think
it says, and it may not even say what its own authors think it says.
Andrew J. Coulson
is director of the Center for Educational Freedom at the Cato Institute in
Washington and an adjunct fellow for the Mackinac Center for Public Policy, an
independent educational and research institute headquartered in Midland, Mich.