One Swallow Does Not a Summer Make

Centuries ago the migration of swallows was known to be seasonal and their arrival a good predictor of the changing of seasons. The proverb about one swallow was a caution against generalizing from a single instance.  Hundreds of years after its common use, the overgeneralization of Vanderbilt’s Tennessee pre-K study suggests it should be brought back into common use.

First, let me quickly review the study and its important findings. When this study began in 2009, statistical studies of the state assessment data in Tennessee (e.g., Comptroller, 2008) had already revealed the Tennessee pre-K (TPK) program’s unique pattern of long-term outcomes. Children who attended that pre-K program performed better on achievement tests initially but were more likely to be placed in special education starting in kindergarten. As they moved through school, those who had attended TPK fell behind on achievement, and this negative outcome became worse as they progressed through the elementary grades. The Vanderbilt randomized trial confirms these results and adds additional measures including attendance and disciplinary incidents for which sixth grade outcomes also favor nonattenders.

The primary value of the Vanderbilt study is that it confirms the results of the broader, earlier statistical study in Tennessee. I see no reason to doubt them. Something has gone seriously wrong in Tennessee after exit from the pre-K program.  The results are surprising and disturbing. However, no other study—randomized or not—has found this unique pattern of outcomes from a preschool program (see a review here). Many other studies have found that initially positive pre-K impacts disappeared over time as nonattenders caught up. Some have found that low quality preschool (typically in long hours of child care) produced negative impacts that persisted. Other studies find that high quality early care and education in and outside public schools can produce long-term positive gains.  In the context of this larger body of evidence that outcomes vary considerably across programs, the Vanderbilt study makes another contribution: It adds to the evidence that preschool programs cannot simply be assumed to produce the desired long-term results even if there are positive immediate impacts.

Unfortunately, the Vanderbilt study has been generalized on its own far beyond the Tennessee program studied. To begin with, Tennessee has strengthened its policies for continuous improvement in ways that could have changed the pre-K program’s long-term outcomes. Furthermore, it is not clear that Tennessee’s specific problem lies in the pre-K program rather than in some systematic adverse treatment of the pre-K children in the elementary schools. Children exited TPK with clear advantages in learning but nevertheless were more likely to be placed in special education in kindergarten. That suggests the problem could be in the elementary schools’ response to the TPK children and not the pre-K program per se.  For these reasons, policymakers should avoid jumping to premature conclusions about the pre-K program and launch appropriate studies to further understand and solve the problem.

The most worrying overgeneralization has been to assert that other state-funded pre-K programs (past, present, and future) have the same results as Tennessee. This is based on two unfounded assertions. The first is that only randomized trials yield accurate estimates of pre-K program outcomes so other studies cannot be trusted. The Vanderbilt study is itself evidence that this is false as it replicates findings of an earlier nonexperimental study, and it is not always true that randomized trials are more accurate. The second is that differences in pre-K program design and implementation, population served, and context including subsequent schooling don’t matter for the outcomes.  A wealth of theory and evidence contradicts this assertion. Person-process-context interactions are the norm, and differences in outcomes should be expected.

Yet, as one study considered in the context of the literature as a whole (see examples here, and here), there are appropriate takeaways from the Vanderbilt study for Tennessee and for the field more generally.

Policymakers, educators, and researchers in Tennessee should be focused on finding out whether the pattern of negative effects still occurs and, if so, what can be done to turn outcomes around. This can be started immediately with data already available. Analyses of kindergarten data can reveal whether the higher special education rates continue, and state assessment data from first grade and beyond can be analyzed to see if gains from TPK disappear and reverse. If these patterns are found again, qualitative and quantitative studies should be launched to investigate why. Such studies should consider possible impacts on parents as well as on children. Were parents of those who went to pre-K lulled into an expectation that they had less need to support learning, while parents of nonattenders felt a need to make up for a lost opportunity? Did long-term employment increase for parents of those in pre-K and reduce parent involvement? In addition to investigating experiences during the pre-K year, later experiences should be studied as well. Are children who went to TPK disappointed or frustrated with subsequent schooling that is more didactic and less interesting than their preschools? Do the schools route more of the former preschoolers not just into special education but into lower ability groups, and why? These are just a few of the possibilities.

The Vanderbilt study is far from alone in suggesting that public preschool programs can fail to produce their desired long-term outcomes, much less the kinds of remarkable results produced by the Perry Preschool or Abecedarian programs. It is just as wrong to generalize from those two programs to public programs that do not resemble them in program features, population served, or context, as it is to generalize from Tennessee to every other state program. That some public programs (e.g., Boston, New Jersey, North Carolina) have  long-term positive outcomes does not mean that this is assured in most other programs.

Research consistently finds that long-term effects of interventions vary considerably in part because of what happens to children later. The Vanderbilt study adds to the evidence that we cannot be lulled into thinking long-term benefits will always follow because a pre-K program gets some or even many things right, as does TPK. Public investments in preschool should include substantial support for ongoing evaluation of both implementation and long-term outcomes so that a continuous improvement process guides them toward effectiveness. Everyone should be on guard for unexpected long-term results, though these will not always be negative. Evaluations must look beyond immediate processes and outcomes to also consider interactions with experiences beyond preschool including in the home and school in subsequent years. Rigorous quasi-experimental as well as randomized trials can help us to learn more and inform progress in each program. Sharing information across programs will have value but does not remove the need for every program to collect information on its own performance and long-term outcomes.