I earlier discussed U.S. News and World Report's plan to change the way it measures "Employment at 9 Months" for its law school rankings, a change that will make it harder for schools to game their Emp9 numbers. An anonymous commentator on that post asked, "Anyone want to recompute the 2008 rankings using this 'more accurate' methodology based on the figures conveniently made available by the ABA in excel format?" I had to admit that the project would generate interesting results. And since I've created just such a model of the 2008 rankings, I figured I was the guy for the (tedious and nonpaying) job. That labor of love generated interesting results sooner than I'd expected, however.
USN&WR's published rankings include each law school's Emp9 score. The magazine calculates those scores using data from questionnaires it sends to the law schools it ranks—questionnaires that ask each school to repeat what it earlier said in reply to the American Bar Association's annual questionnaire. Happily for ranking geeks, the ABA recently began publishing that data in a downloadable Excel file. But when I plugged that ABA data into USN&WR's Emp9 formula, I discovered that it did not always generate the Emp9 scores that USN&WR used in its 2008 rankings. Herewith the problematic cases:
I see four explanations for these divergences: 1) Errors in the ABA data; 2) Errors in the USN&WR data; 3) Errors in my calculations; or 4) Differences between what a law school told the ABA and USN&WR. The first two explanations seem most likely to me, but I of course cannot rule out the third. As for the last, it bears noting that a school's Emp9 data comes from the prior February—seven or eight months before a school fills out its ABA and USN&WR questionnaires. It's thus hard to imagine how a school could dig up new data, after so long a remove, in the relatively short span between when it fills out the former questionnaire and the latter.
With luck, I'll have firmer answers, soon. I've emailed administrators at the four law schools that stake out the extremes on the above list, asking if they could please help me understand this phenomenon. That out of the way, I plan soon to return to my original goal: recalculating the 2008 USN&WR rankings using the Emp9 formula that will officially take effect next year.
[Crossposted to MoneyLaw.]
Earlier posts about the 2008 USN&WR law school rankings:
Earlier posts about Emp9 measure: