Why Curriculum Decision Makers Should Look at All the Evidence

Developing guidance on what works in early education is always challenging and that certainly applies to the difficult business of evaluating and selecting a curriculum. Whenever specific early education curricula are evaluated, judgment calls have to be made on the strengths and weaknesses of the evaluation including such issues as the duration and quality of training in the curriculum prior to the evaluation, how well the measures used actually measure children’s learning and development (are they broad enough and deep enough?), and how well any given curriculum is implemented in the classroom at the time the research was conducted.

When the final results are published, these become the go-to issues when people scratch their heads about why a curriculum did or didn’t do well in the review. Recent efforts to summarize evidence regarding the effects of various curricula have brought these issues front and center. The 2008 federal Preschool Curriculum Evaluation Research (PCER) report found most curricula in the study provided little or no advantage over existing practice. On an ongoing basis, the Institute for Educational Sciences’ What Works Clearinghouse (WWC) provides reviews that yield lackluster ratings for a number of programs.

Among the findings WWC reports is that the Tools of the Mind Curriculum had no significant effects. What may not be clear to readers is that the study WWC reviewed was designed to determine whether Tools of the Mind could produce equivalent academic results while improving results for self-regulation and social behavior compared to a more traditional curriculum that was also expected to produce strong academic gains. Indeed, this is exactly what was found—strong reductions in behavior problems and improvements in self-regulation with academic gains at least as strong as from the other curriculum. However, WWC does not take into account any effects of early childhood curriculum on executive function, self-regulation, or any aspect of social and emotional development.

The latest effort in this arena comes by way of Johns Hopkins University where Bette Chambers and a team of colleagues have taken another look at many of the same preschool programs appearing in the federal efforts. The team conducting the review says 40 studies evaluating 28 different programs met its research standards. Without getting bogged down in details, suffice to say that they made different judgments than did WWC about what studies to include and how to evaluate their results. It should come as no surprise that the results of the Johns Hopkins review differ substantially from those of the federal efforts. The Johns Hopkins team review awards favorable ratings to 11 of the 28 programs and reports that six show “strong” evidence of effectiveness.

Many will be asking whether the Johns Hopkins review provides a better basis for choosing a curriculum than the previous reviews. Not really. Their criterion for “strong evidence of effectiveness” is two findings of effect sizes larger than .20 on any measure regardless of the importance of the outcome. They ignore the nature of the comparison made in the study (was it to another strong curriculum or poor practices?), and they also ignore non-academic outcomes. I respectfully suggest that neither the Johns Hopkins review nor the federal efforts provides an adequate basis for choosing a curriculum. Together they provide useful information but remain incredibly narrow. They still do not completely survey the relevant studies of curricula (for example, rigorous studies of the High/Scope approach), consider many of the outcomes we in early education are trying to produce, or take into account much of the field’s knowledge about what works. Much of this comes from basic and applied research on the education and development of young children that does not follow the “horse race” model for curriculum comparison studies.

On a practical level, the question for curriculum decision makers becomes what resources to consult in deciding on a curriculum for their programs. The best advice is to take a very broad look at the evidence that includes wide-ranging analyses such as NIEER’s meta-analysis of 120 studies; even if this does not point to specific curricula it does help identify characteristics of more effective early education from the broadest set of comparative studies collected to date.

Readers interested in more detailed, concrete advice on how to choose a highly effective curriculum should consult NIEER’s policy brief Preschool Curriculum Decision-Making: Dimensions to Consider.

Steve Barnett
Co-director, NIEER