The most direct proof of the model's accuracy comes from comparing the scores that USN&WR assigned to each of the 100 law schools it its top two tiers and the scores that the model generated. The following chart shows how closely the two sets of numbers correspond.
As you can see, the model's scores generally came within a few percentage points of those assigned by USN&WR.
You might fruitfully compare that measure of the model's accuracy with the same measure I applied to last year's model of the USN&WR rankings. Back then, the differences between the score that USN&WR gave to the University of Pennsylvania School of Law and the score that my model gave to it presented the most striking puzzle. This year, while Penn's score in the rankings still varies a bit from its score in the model (see the first dip in the red line on the left end of the above chart), it does not stand out nearly so much as it did last year. To make a long story short, I credit that improved convergence to an effect of the change that USN&WR made in the way it calculates expenditures/student.
Herewith the law schools that got scores notably higher (hence the "+") in USN&WR's 2007 rankings than they did in my model, measured as the difference between the two scores divided by the USN&WR score.
+6.29%: Baylor U.
+3.74%: U. of Florida (Levin)
+2.90%: Arizona State U.
+2.73%: U. of Pennsylvania
+2.63%: U. of Georgia
+2.44%: U. of Richmond (VA)
+2.19%: U. of Missouri–Columbia
+2.08%: Seattle U.
The divergences at the other end of the scale do not jump out quite as much. Herewith the law schools that got scores notably lower in USN&WR's 2007 rankings than they did in my model:
-2.01%: U. of Toledo (OH)
-2.16%: Rutgers State U.–Newark (NJ)
-2.17%: DePaul U. (IL)
-2.22%: Santa Clara U. (CA)
-2.32%: George Mason U. (VA)
-2.38%: Seton Hall U. (NJ)
-2.39%: Georgetown U. (DC)
-2.55%: U. of Iowa
-2.62%: U. of Maryland
Why does my model generate different scores for some schools than USN&WR does? I have come up with a few tentative explanations. The foremost puts the blame squarely on me, an all-too-likely if uncomfortable possibility. Another pins the blame on dissembling law schools, a possibility that it makes me still more uncomfortable to admit. Other explanations involve likely but rather inconsequential matters.
First and foremost, my model's methodology may fail to match USN&WR's. I've studied USN&WR's methodology pretty carefully and my results evidently match those of USN&WR in most cases. I might add that my third and fourth tiers divide exactly where USN&WR has told me they should: at 27 points. But I remain as human as anybody else, and at least as prone to make mistakes. So I must, in all frankness, admit that the failings of my model may lie entirely with me.
It may also or alternatively turn out that the USN&WR rankings used different data than my model did. How could that happen? Both USN&WR and I relied primarily on ABA data, after all. In isolated instances, the model's data may vary from USN&WR's due to scanning errors, typos, and the like. Mistakes of that sort do happen, of course. More significantly, though, three sorts of systemic problems may have caused the two sets of data to diverge.
First, I remain a bit unsure exactly how USN&WR calculates a proxy figure for law schools that decline to report the percentage of their graduates employed at graduation. I have it from USN&WR that they deduct 30% from the figure that such schools reported for employment nine months after graduation. A fellow law prof and USN&WR rankings geek relates that USN&WR told him something a bit different. I suspect that USN&WR in fact uses a discount rate specially calculated each year, and that the "30%" figure they've disclosed represents a rounded figure. But even if the proxy my model uses errs, it cannot err by much, and no such slight error in the "Emp0" indicator, worth only 4% of any school's score in the rankings, could account for the divergences documented above. [Update: I've now resolved that question, and am confident I'm using the same proxy calculations that USN&WR does.]
Second, as described earlier, USN&WR applies a proprietary cost-of-living (CoL) index to one addend in each school's Overhead/Student indicator. The model uses a proxy CoL index, one I worked up using publicly available data and common sense. I know the rough limits of USN&WR's CoL index, though, which at any rate only marginally affects part of a measure that counts for only 9.75% of each school's score, so I don't think that possible cause can account for the differences between the actual and simulated scores.
Third, and most disturbingly, some law schools might not have told USN&WR quite the same thing that they told the ABA. They might have fudged the financial numbers, for instance. A law school might have reasoned, "How would anybody know if we characterized our finances one way on the ABA questionnaire and another, more helpful way, on the USN&WR questionnaire? What we tell the ABA is kept a secret from U.S. News, and U.S. News does not publicly disclose what we tell it." [Update: Subsequent to this post, I learned that USN&WR's data might vary from the ABA's due to errors on the former's part. That looks like a plausible explanation of the variation between Baylor's scores, for instance.]
I'm loath to impute dishonesty to anyone involved in the rankings process. I hope that there is some better explanation for the differences between the USN&WR rankings and my model. I've aired my concerns to USN&WR, but it has yet to put my mind at ease by pointing out another, more likely cause. Admittedly, though, USN&WR has shown nothing like my interest in solving the problem. Considering the matter dispassionately, it occurs to me that USN&WR might not be particularly eager to pursue an investigation that threatens to impugn the accuracy of its law school rankings. More likely, of course, USN&WR simply regards me as a pest tripped up by his own errors. At any rate, and as I plan to discuss more fully in a future post, USN&WR could by publicizing all the data that goes into its rankings both guarantee the accuracy of that data and reassure rankings-obsessed worrywarts such as myself.
[Please note that I've edited this post to reflect corrections that I made once and again to my model of the USN&WR rankings.]
Earlier posts about the 2007 USN&WR law school rankings:
- Change to U.S. News Law School Rankings Methodology
- "Financial Aid" Revised in U.S. News Methodology
- How USN&WR Counts Faculty for Rankings
- Whence Come the LSATs and GPAs Used in the Rankings?
- Gains and Losses Due to USN&WR's Use of Reported Median LSATs and GPAs
- How to Model USN&WR's Law School Rankings
- Why to Model USN&WR's Law School Rankings
- The ABA and USN&WR's Law School Rankings
No comments:
Post a Comment