CSAP Season

This time of year, Colorado learns the results of the state's standardized achievement tests caled the CSAPs (Colorado Student Assessment Program) taken by students in grades 3 to 10 throughout the state, which are used to rate schools. These ratings are quite simple minded. Schools that have students that score well on average on the tests get a positive rating, while those that have students that score poorly on average on the tests get a negative rating. As a result, the bottom line is a familiar one:

An achievement gap of about 30 percentage points persists between the scores of white and Asian students and their black and Latino peers. In the fifth-grade science test, for example, 14 percent of Latino students were proficient in science, compared with 50 percent of white students. . . . .

40 elementary schools . . . have consistently ranked in the top quarter of schools for percentage of students at or above grade level. On average, only 5.5 percent of students in those schools last fall qualified for free or reduced-price lunches, an indicator of poverty.

By contrast, an average of 86 percent of students qualified for free or reduced-price lunches at 71 schools that consistently ranked in the bottom quarter[.]

The overwhelming evidence from ten years of CSAP tests, and a mountain of other data on academic testing, is that the best predictor of a schools aggregate test score performance is the socioeconomic status of the students. Garbage in, garbage out.

The effect of the way the state uses CSAP test results is that truly abysmal teaching resulting in much lower CSAP scores than the demographic makeup of the school would lead one to expect is almost certain not to result in any sanction from the state in an affluent district. But, schools that are doing amazing things to help children who arrive at school already lagging far behind may be punished, rather than rewarded, unless they can meet almost impossible goals. The socioeconomic gaps are already pervasive by the time kids are taking their third grade CSAP tests, even though school has, at that point, played only a small part in their lives.

Averages are not destiny. There are 450,000 students or so taking CSAPs at more than a thousand schools. Not every school is going to be right on the line due to random variation, especially in smaller schools. But, the effects are overwhelming, compared to other predictors of performance based on things like teacher experience or student to teacher ratios. Even an approach based on tracking schools from year to year, if not cabined by requiring same student comparisons from year to year, is likely to be a better gauge of changing demographics in a school, than it is to be a gauge of better teacher and administrator performance.

Even more importantly, while the law of averages cloaks the individual differences in kids from the same background at a school wide level, the differences are real and educationally important for the kids. While perhaps 80%-90% of kids living in poverty don't obtain proficient or beter scores on the CSAPs, the rest do. Similarly, there are kids who have comfortable middle class lives who fail and need extra help. But, while the CSAPs are high stakes for the schools, they have little or no direct impact on the lives of individual students.

In the same way, even if most schools have some teachers who are stellar, and others who are only going through the motions, a school level CSAP assessment provides little incentive to any individual teacher. The impact of any one teacher is overwhelmed by the fact that most teachers are in the average range, and that good teachers are balanced out by less than stellar teachers, in any decent sized school. For every Jennifer Wilson (a Denver Public Schools teacher who is the best elementary school teacher in the nation according to Disney), there is a grown up Billy Madison who has chosen to pursue teaching about whom you can say "Everyone in this room is now dumber for having listened to" you.

Moreover, given the realities of tenure, an administrator's potential to impact teaching quality in the school or schools with which an administrator is entrusted, is, at best, incremental -- primarily taking place through the process of attrition and hiring to replace departed teachers.

One would expect some improvement in scores overall, over time, simply as a result of being better able to game the system. For example, districts are severely punished for failing to get students to show up and take the test. A few students showing up and receiving even mediocre scores helps a district more than improving the scores of students who would show up in any case.

Likewise, making sure that the schools curriculum is in synch with the test is also crucial to test scores, but not to education. A school is severely penalized If it don't teach fractions and multiplication and equilateral triangles in the right order. Why? Because its third graders may not have the math that they will be tested on when they need it, even if they ultimately learn it all by the time they make it to higher grades. Schools get no credit for teaching something sooner than it appears on the test, to balance out what students don't yet know that is on the test. Similarly, simply figuring out how subtle skills like writing are graded in a standardized manner, can make all the difference in helping students to know what is important.

But, until state policy makers understand that the CSAP is a much better tool for preparing individual educational plans for particular students, than it is for rating school quality or teacher quality, the immense effort that goes into administering 1.5 million tests a year to 450,000 students at more than a thousand schools will be wasted.

Cross Posted at Colorado Confidential.