L&S: Stupid Study on Smart States

During the evening lecture, I read this article over James’s shoulder:

The smartest state in the union for the second consecutive year is Massachusetts.

The dumbest, for the third year in a row, is New Mexico.

These are the findings of the Education State Rankings, a survey by Morgan Quitno Press of hundreds of public school systems in all 50 states. States were graded on a variety of factors based on how they compare to the national average. These included such positive attributes as per-pupil expenditures, public high school graduation rates, average class size, student reading and math proficiency, and pupil-teacher ratios. States received negative points for high drop-out rates and physical violence. [emphasis added]

This method of calculating educational performance is, for lack of a better word, crapola. Some of the factors listed – graduation rates, reading proficiency, math proficiency – are plausible measures of educational output. But the other items – expenditures per student, class size, pupil-teacher ratios, and physical violence – are putative inputs into education.

By including these factors in a measure of a state’s “smartness,” the study designers have assumed they actually make a difference. In fact, there’s evidence that some of these things (I’m thinking particularly of expenditures per student) are not that important. But even if the designers' assumptions are correct, the study design is still flawed, because they are double-counting. If higher spending does improve education, then the results of higher spending should show up in the real output measures, like reading and math proficiency. Likewise for class size, pupil-teacher ratios, and physical violence. There’s no need to include them separately.

I eagerly await the news reports noting that the states’ “smartness” as measured in this study is correlated with higher educational expenditures and lower pupil-to-teacher ratios (which, of course, is true by design), and offering this as evidence of the need to expand education funding and hire more teachers.