Stephen Raudenbush: How NCLB Testing Can Leave Some Schools Behind

Stephen Raudenbush, professor of education and statistics and research professor at the Institute for Social Research at the University of Michigan, is one of the nation's leading authorities on school testing. In a lecture at the Educational Testing Service in Princeton, N.J., Raudenbush offered an in-depth critique of current methods for measuring the school effectiveness. Raudenbush suggests the current testing required by the Federal 'No Child Left Behind' Act not only misses the mark, but is 'scientifically indefensible' as a measure of a school's impact. Since such testing has already had broad impact on the field of early education, we invited him to share some of his findings here. The full text of his ETS lecture can be found at www.ets.org/research/pic/angoff9.pdf.

Q. What caused you to question the tests required by the No Child Left Behind Act?
A. Researchers have known for many years that the average test score of a school is strongly correlated to the social and ethnic background of the community where it is located, and to the social and ethnic background of the families of the children. So that score tells you almost nothing about how good a school is. We've known that for many years.

Q. So the current tests are not a good way to judge how good a job a school is doing?
A. It's scientifically indefensible to use the average achievement test scores of a school for that purpose. We need to know how much kids are learning, not just how much they know. That's a separate issue. As I said, how well a child scores on an achievement test is highly correlated to that child's social and ethnic background--and relative privilege or lack of it. So schools serving higher numbers of poor children tend to have lower scores. But those scores do not always reflect the rate of learning going on at the school. That's the part we have to be very careful about.

Q. Why is that? It just sounds so logical to link children's scores on achievement tests with whether a school is doing a good job.
A. Yes, it sounds logical, until you consider that achievement tests were never meant to measure what goes on in a school. If you want to measure what goes on in a school, you have to develop measures that look at the educational process and practices, not just at children's relative achievement. Many schools have reasonably good rates of growth in learning, but these schools might still have high percentages of children with low test scores. That's because they are dealing with large numbers of poor children, who may not score as well on achievement tests. In such cases, we don't want to disrupt those schools. They are good schools, with sound educational practices that promote learning, but still have lower achievement scores because of the large number of poor children attending that school. We don't want to judge them as failing and in need of improvement. In other schools, children have not only low average achievement but also poor rates of learning. These are the schools we need to worry about.

Q. Is that happening under the No Child Left Behind Act?
A. Well, we do know that large numbers of schools in poor, urban neighborhoods have been judged to need improvement. In fact, the overall pattern emerging is that schools serving high poverty communities are more likely to be judged as ineffective.

Q. Still, if achievement scores are low, shouldn't we hold the schools accountable? Aren't schools responsible for helping all children to learn?
A. Of course, no child should go to a bad school. None of us want that. But the point is that the tests do not measure how good the educational practices are at a given school and the relative rates of learning. If the goal is to make better schools, then we need to have systems of accountability that help schools do a better job. The current law does not do that because the tests don't measure either the educational practices or the rate of learning.

Q. If that's the case, how did achievement tests become part of federal law?
A. Well, everyone agrees that children shouldn't go to schools that are doing a bad job, and it's easy to make a leap between the idea that how students do on achievement tests reflects practices at a particular school. In fact, the idea of using student achievement tests in this way has been popular since at least the mid-1980's, in different states, different localities, even different countries.

Q. Many researchers agree with your analysis. Haven't they protested?
A. Many have objected to this use of the tests. Some have suggested alternatives. There are some accountability systems that use a "value-added" approach, that is, they try to hold schools accountable for how much children learn while under the care of a particular school--the value that school adds to learning. That approach has been around for at least ten years, and most recently, interest in this approach has exploded, because of the dissatisfaction with accountability based on average achievements.

Q. Does that mean we should switch to value-added testing?
A. If we use both pieces of information, we'd certainly do better. If we could identify schools with low test scores and low rates of learning, then we could separate these schools from those with higher rates of learning. We'd be a lot better off. There are some states and districts that do use both kinds of information now.

Q. So is that a good enough system for rating schools?
A. We need a system that follows the same children over time, or groups within a school. Usually all we have are average test scores from schools. We don't even know if it's the same kids year after year. We may also be missing information about minority groups within a school. Subgroups within a school must reach a certain size before their test scores are visible. So a school may do well on average, but students within a subgroup may not be doing well. That could be missed, especially at schools that have a more diverse student body. They are not held accountable for how all the different subgroups of kids are doing.

Q. So how can we judge how good a school really is?
A. We need to do more to identify effective practices in schools, and learn how to promote those practices, to improve the rate of learning among all students. There's tremendous interest in this right now, and that's actually a very positive feature of the current situation. The upset over the current testing is motivating people to think more about this, about what effective practices are and what incentives we can create so that educators will adopt those practices.

Q. Is all of this relevant to preschool education?
A. Absolutely. In some ways, you have the best examples in preschool education. Certain kinds of intervention have proved to be very effective, and they are tied to specific practices. The High/Scope Foundation's research, for example, shows that certain practices make a difference. Teaching language in certain ways can make a critical difference in children's learning. Those practices can be used in many settings to encourage learning.

Q. So you think we need to shift the spotlight from achievement to classroom practice?
A. Educational practices have to be at the heart of it. That's critical. And we need the resources to do more research to find out more about what practices work. We have to really know what to do to create the settings to help children learn. Then we can understand what is essential to reorganizing the delivery of instruction so that children will learn. We can't leave children in dysfunctional settings, but it would be an error to disrupt those where learning rates are high. That's the central point. We need to measure practice and combine that knowledge with measurement of how rapidly children are learning.