Lessons learned from Vanderbilt’s study of Tennessee Pre-K

Newly released findings from Vanderbilt’s rigorous study of Tennessee’s state-funded pre-K program are a needed tonic for overly optimistic views. No study stands alone, but in the context of the larger literature the Tennessee study is a clear warning against complacency, wishful thinking, and easy promises. Much hard work is required if high quality preschool programs are to be the norm rather than the exception, and substantive long-term gains will not be produced if programs are not overwhelmingly good to excellent. However, the Vanderbilt study also leaves researchers with a number of puzzles and a similar warning that researchers must not become complacent and have some hard work ahead.

Let’s review the study’s findings regarding child outcomes. Moderate advantages in literacy and math achievement were found for the pre-K group at the end of the pre-K year and on teacher ratings of behavior at the beginning of kindergarten. However, by the end of kindergarten these were no longer evident and on one measure the no-pre-K group had already surpassed those who had attended pre-K. The pre-K children were less likely to have been retained in kindergarten (4% v. 6%) but were much more likely to receive special education services in kindergarten than the no-pre-K group (12% v. 6%). The pre-K group’s advantage in grade repetition did not continue, but it did continue to have a higher rate of special education services (14% v. 9%) in first grade.

By the end of second grade, the no-pre-K group was significantly ahead of the pre-K group in literacy and math achievement. The most recent report shows essentially the same results, though fewer are statistically significant. Teacher ratings of behavior essentially show no differences between groups in grades 2 and 3. Oddly, special education is not even mentioned in the third grade report. This is puzzling since prior reports emphasized that it would be important to determine whether the higher rate of special education services for the pre-K group persisted. It is also odd that no results are reported for grade retention.

If we are to really understand the Tennessee results, we need to know more than simply what the outcomes were. We need information on the quality of the pre-K program, subsequent educational experiences, and the study itself. It has been widely noted that Tennessee’s program met 9 of 10 benchmarks for quality standards in our annual State of Preschool report, but this should not be taken as evidence that Tennessee had a high quality program. Anyone who has read the State of Preschool knows better. It (p.10) specifies that the benchmarks “are not, in themselves, guarantees of quality. Arguably some of them are quite low (e.g., hours of professional development), even though many states do not meet them. Moreover, they are primarily indicators of the resources available to programs, not whether these resources are used well. In addition to high standards, effective pre-K programs require adequate funding and the continuous improvement of strong practices.

The State of Preschool reported that Tennessee’s state funding was nearly $2300 per child short of the per child amount needed to implement the benchmarks. More importantly, the Vanderbilt researchers found that only 15% of the classrooms rated good or better on the ECERS-R. They also found that only 9% of time was spent in small groups; the vast majority was spent in transitions, meals, and whole group. This contrasts sharply with the high quality and focus on intentional teaching in small groups and one-on-one for programs found to have long-term gains (Camilli et al and Barnett 2011). The Tennessee program was evaluated just after a major expansion, and it is possible that quality was lowered as a result.

Less seems to be known about subsequent educational experiences. Tennessee is among the lowest ranking states for K-12 expenditures (cite Quality Counts), which is suggestive but far from definitive regarding experiences in K-3. We can speculate that kindergarten and first grade catch up those who don’t go to pre-K, perhaps at the expense of those who did, and to fail to build on early advantages. However, these are hypotheses that need rigorous investigation. Vanderbilt did find that the pre-K group was more likely to receive special education. Perhaps this lowered expectations for achievement and the level of the instruction for enough of the pre-K group to tilt results in favor of the no-pre-K group. Such an iatrogenic effect of pre-K would be unprecedented, but it is not impossible. There are, however, other potential explanations.

Much has been made of this study being a randomized trial, but that point is not as important as might be thought. One reason is that across the whole literature, randomized trials do not yield findings that are particularly different from strong quasi-experimental studies. The Head Start National Impact Study and rigorous evaluations of Head Start nationally using ECLS-K yield nearly identical estimates of impacts in the first years of school. Another reason is that the new Vanderbilt study has more in common with rigorous quasi-experimental studies than “gold standard” randomized trials. Two waves were randomly assigned. In the first wave, just 46% of families assigned to pre-K and 32% assigned to the control group agreed to be in the study. In the second wave, the researchers were able to increase these figures to 74% and 68%, respectively. These low rates of participation that differ between pre-K and no-pre-K groups raise the same selection bias threat faced by quasi-experimental studies. And, uncorrected selection bias is the simplest explanation for both the higher special education rate for the pre-K group and the very small later achievement advantage of the no-pre-K group. I don’t think the bias could be nearly strong enough to have overturned large persistent gains for the pre-K group.

Even a “perfect” randomized trial has weaknesses. Compensatory rivalry has long been recognized as a threat to the validity of randomized trials. In Tennessee one group got pre-K; the other sought it but was refused. It appears that some went away angry. Families who agreed to stay in the study could have worked very hard to help their children catch up and eventually surpass their peers who had the advantage of pre-K. Alternatively, families who received the advantage of pre-K could have relaxed their efforts to support their children’s learning. Similar behavior has been suggested by other studies, including a preschool randomized trial I conducted years ago for children with language delays. Such behaviors also could occur even without a randomized trial, but it seems less likely.

Randomized trials of individual children also create artificial situations for subsequent schooling. If only some eligible children receive the program, do kindergarten teachers spend more time to help those who did not attend catch and “neglect” those who had preschool? Would kindergarten teachers change their practices to build on pre-K if the vast majority of their children had attended pre-K and not just some; perhaps they would only change with support and professional development?

Clearly, the Vanderbilt study has given the early childhood field much to think about. I am reminded of Don Campbell’s admonition not to evaluate a program until it is proud. However, programs may also be in the habit of becoming proud a bit too easily. We have a great deal of hard work in front of us to produce more programs that might be expected to produce long-term results and are therefore worth evaluating. Researchers also would do well to design studies that would illuminate the features of subsequent education that best build upon gains from preschool.

What we should not do is despair of progress. The media tend to focus on just the latest study, especially if it seems to give bad news. They present a distorted view of the world. Early childhood has a large evidence base that is on balance more positive than negative. There is a consensus that programs can be effective and that high quality is a key to success. Research does help us move forward. Head Start responded to the National Impact study with reforms that produced major improvements. Some states and cities have developed even stronger programs. Tennessee can learn much from those that could turn its program around. If it integrates change with evaluation in a continuous improvement system, Tennessee’s program could in turn become a model for others over the next 5 to 10 years.

–Steve Barnett, Director, NIEER