Why preschool critics are wrong

In a recent blog on his Brookings Institution site, Grover Whitehurst claims that the preponderance of scientific evidence does not indicate lasting positive benefits from preschool.  Others disagree, he says, because they are not as “picky” about the evidence. As there is no disputing taste, I assume he means that the research he prefers is more rigorous and relevant, not just better aligned with his personal preferences.  Hence, we would be looking for a valid and reliable process when he arrays the research and grades each study.  In fact, his list of studies and his analyses are appallingly inaccurate for someone who claims to be an expert.  As shown in detail below, Whitehurst omits much of the relevant research, and he misrepresents the studies that he includes. 

Whitehurst claims that “Not one of the studies that has suggested long-term positive impacts of center-based early childhood programs has been based on a well-implemented and appropriately analyzed randomized trial.”   This claim is false based even on the studies he does cite.  His own statements in the blog regarding the Perry Preschool study and its re-analyses by Jim Heckman contradict this claim, as do older analyses demonstrating that minor departures from random assignment in the Perry study had no substantive effects on the results.[1] No study is perfect, so it is seems odd that Perry receives an A- for an inconsequential fault when other less than perfect studies get an A. Then there is the Infant Health and Development program (IHDP) study, which Whitehurst assigns higher grades than Perry, but which he seems to forget when making his “not one” study claim.

Yet Whitehurst’s credibility problem is far more serious when one turns to the studies that are missing from his analysis.

In fact, a number of other well-implemented and appropriately analyzed randomized trials find lasting effects from preschool education.  For example, a study of long-term effects conducted by the Institute for Developmental Studies (IDS) included 402 children randomly assigned to a public school pre-K program or to a control group at age 4 for one year.[2]  A teacher and an aide staffed each preschool classroom of 17 children. Positive effects were found through at least third grade.  Even longer term follow-up indicates adult gains in achievement, educational attainment, and employment, but suffers from severe attrition. So while we can have strong confidence in the results through third grade, we have less confidence in the very long-term results.  However, the findings for adults are consistent with the earlier results in the elementary grades and with findings in Perry and other studies.  Another randomized trial of preschool education is noteworthy because it was conducted with relatively advantaged children, and it also found evidence of lasting effects on achievement into the early elementary grades.[3]

Inexplicably, Whitehurst fails to recognize a large number of studies (once again including well-implemented randomized trials) that compare one form of preschool education to another to study the effects of curriculum, length of day, and other features.  When such studies find lasting differences due to the type of preschool program, from the end of kindergarten to the end of high school, they add to the evidence that high-quality preschool education per se has long-term effects.  This literature includes studies (here, here, and here) over many years, some begun decades ago with very long-term follow-ups and some very recent with much shorter follow-ups.[4]  These studies also add to the evidence for successful scale up in large-scale public programs.

As preschool research is conducted in other countries, not just the United States, there is a broad range of research Whitehurst omits that finds lasting benefits from quality preschool education, including rigorous studies in countries with universal programs and additional well-implemented, appropriately analyzed randomized trials.[5]  When similar outcomes from quality pre-K are found with different populations in different contexts, such studies are confirmatory—not irrelevant.  Taken together, they indicate that the relationships between quality preschool education and long-term outcomes are quite robust with respect to variations in the children and families served.

Returning to the matter of how Whitehurst represents the few studies he prefers to include, the ongoing Tennessee evaluation of pre-K effectiveness is one of only two to receive double A grades.  He calls this study a well-implemented and analyzed Random Control Trial (RCT) and reports that it finds no differences later in elementary school.

This description of the Tennessee study and its findings is shockingly inaccurate.

The Tennessee pre-K evaluation includes both a large randomized trial of pre-K that follows children using the data routinely collected by schools, and a smaller intensive substudy (ISS) in which randomization failed.  The results Whitehurst cites come from that substudy, which is not analyzed as a randomized trial.  Let me quote directly from the study authors: “The nonconsent rates for the two cohorts in the ISS sample mean that we do not have data on the main ISS outcomes for many of the children who were initially randomized, so analysis on the basis of that randomization is not possible.”[6]  To be perfectly clear, because so many fewer parents in the control group agreed to have their children tested in the substudy, randomization was not used to analyze the data.  Instead, the substudy used a quasi-experimental approach of the type that Whitehurst otherwise gives lower grades.

Yet it is not just the Tennessee study’s methods that are misrepresented, but also its results.  While the full randomized trial will not provide test score results until children reach the state’s third grade tests, it does provide results for grade retention.  Let me again quote the study’s authors directly:

For the Intensive Substudy sample, there was a statistically significant difference between the 4.1% of the TN‐VPK participants who were retained in kindergarten compared to the 6.2% retention rate for the nonparticipants. This effect was confirmed in Cohort 1 of the full randomized sample, with retention data still unavailable for Cohort 2 of that sample. In Cohort 1, 4.0% of the TN‐VPK participants were retained in kindergarten compared to 8.0% of the nonparticipants, also a statistically significant difference.[7]

In other words, analyses of both the compromised Tennessee substudy and the full randomized trial find that the pre-K program significantly reduced grade retention in kindergarten.  The only finding from this adequately randomized part of the Tennessee study is this persistent positive effect, and this effect is twice as large as that found in the quasi-experimental substudy (which still finds a significant positive effect of pre-K).  Yet, Whitehurst reported the study found “no differences later.”

I could go on to list additional problems with Whitehurst’s review, but surely these suffice to demonstrate that his summary of the evidence is just plain wrong–not picky.

– W. Steven Barnett, Director, NIEER and CEELO

This entry originally appeared in Valerie Strauss’ education blog “The Answer Sheet” at the Washington Post.