Voter Fraud in U.S. News Surveys?

In ranking law schools, U.S. News and World Report weights peer reputation more heavily than any other measure of quality. A school's reputation among its peers counts for 25% of its overall score in the rankings (the next-most important measure, in contrast, counts for only 15%). How does USN&WR calculate a school's peer reputation? It says that it relies on surveys sent to "law school deans, deans of academic affairs, the chair of faculty appointments, and the most recently tenured faculty members" of each of the ABA-accredited law schools that it ranks. In truth, however, other people also get the chance to vote on USN&WR's reputation surveys.

Two people recently and independently told me that they had received USN&WR reputational surveys even though they do not fit any of the criteria—law school dean, dean of academic affairs, etc.—that USN&WR has published. Both people work at law schools. One of my informants told me that he/she got the forms both at his/her present employer and at a law school she worked at earlier. My other informant told me that he/she knows of a similarly situated person who likewise got an apparently unauthorized USN&WR reputation survey. Both informants asked that I not identify them—hence my coy phrasing—but their claims strike me as completely credible.

Those few anecdotes do not, of course, establish how often USN&WR sends reputation surveys to people other than those it (says it) intends to poll. Notably, however, the reports I've received came to me unbidden, simply because I have a reputation as a rankings geek. Query how many more such cases a comprehensive investigation would uncover; a lot, I'd guess.

Query, too, whether USN&WR really means to send surveys to people such as those who contacted me. Perhaps it has a "secret list" of reputation survey recipients, people whose opinions it holds in high regard but whom it wants to safeguard from the taint of law school public relations campaigns designed to influence USN&WR voters. Yet another caveat: Perhaps USN&WR manages to screen out reputation surveys that get filled out and returned by unqualified parties.

We thus have, as yet, no solid proof that voter fraud materially affects the way that USN&WR ranks law schools. We do, however, have reason to wonder whether the most important part of USN&WR's rankings really works as advertised.


[Crossposted at Agoraphilia, MoneyLaw, and College Life O.C.]