Sunday, May 01, 2005

Rank This Job

Imagine that you hold a job where you control the fates of powerful people. The decisions that you make have a major impact on the status of the institutions where those people work and, consequently, the status of those people. Yet your subjects have absolutely not say over what you do. They do not pay your salary, do not elect you, and can do nothing but complain if they don't care for how you exercise your discretion.

Sound good? Not so fast. The people affected by your decisions do complain—loudly, repeatedly, and publicly. They're good at it, too, as their own work qualifies them as something very much like professional complainers. And though the parties that your decisions disadvantage naturally head that unpleasant chorus, even those who enjoy vast benefits from your decisions disparage the legitimacy of what you do.

It gets worse. Few people really understand how you make your decisions. Far from acting arbitrarily, you apply statistical tools to a wide range of data when you compile your rankings (for that is what you do at your job). That means you have relatively little discretion in how you rank universities and graduate schools (for that is what you rank). You thus suffer the responsibilities of power without enjoying many of the perks.

That, I think, pretty well summarizes the job of the fellow at U.S. News and World Reports who heads that magazine's efforts to rank educational programs. I've been spending a lot of time poring over the law school rankings of late, trying to figure out just what how the U.S. News & World Report ends up with the rankings it does. After much befuddlement, I decided to go straight to the source. It proved surprisingly easy. I simply called the magazine, asked to get put through to Mr. X (as I will call him, so as to protect him from further harassment), and picked up my jaw when he picked up his phone.

My jaw fell not only because Mr. X actually took my call, but because he had one of the saddest voices I have ever heard. We exchanged a couple of calls and many emails after that, and so far as I could peg the guy's mood it didn't change much. Granted, he may have been suffering from some recent and unfortunate news. Some of the things he said about his critics led me to conclude, however, that he must feel down quite often. And, granted, he might just have a disposition to feel that way. But the more I thought about his job, the more I began to sympathize with him.

Would you want a job ranking law schools? I don't think you should. You wouldn't really enjoy a great deal of power, given your task of objectively applying dispassionate formulae, and even those you rate highly would disparage your efforts. So pity Mr. X, a fellow whose job cannot rank among the happiest.


Mr. Bugman said...

Of course if he truly was as sad as his voice sounded, he can alway find new employment. Wood's Rule = "at will" employment (yes, we are studying diligently), unless you could hear chains dangling in the background. Did you ask him how long he had been working on the List? I think it would be interesting to know just how long his self-imposed torture has been going on. Then again, maybe that's his "phone voice" so compassionate university representatives don't tear him a new one every time the phone rings. Hopefully you were still able to get enough info to continue CUs climb in the rankings. I think these formulae sound about as objective as the BCS rankings in college football and both should be investigated by Congress. Ok, maybe it's not THAT important.

Tom W. Bell said...
This comment has been removed by a blog administrator.
Tom W. Bell said...

[The prior one had a typo. I can't stand those.]

Ah, but no, Mr. Bugman! Wood's Rule means only that he can *lose* his job at the drop of the hat. It says nothing about whether or not he can get a *new* one!

Ted said...

Tom, you were at Chicago when Harvard complained that they were ranked beneath Chicago. Presto chango, the weights are tweaked, the category of "library volumes" is added, and Harvard is now #2/#3 to stay by virtue of tens of thousands of unopened books.

A few years later, schools complained about the use of employment data, and USNWR added a bar-passage component that had the effect of overvaluing NY and CA law schools, bumping Chicago below Stanford, NYU and Columbia to sixth, which means that Chicago Law no longer competes for the tippy-top students that it used to compete for when you and I went there. Mr. X has quite a bit of discretion.

Dave said...

You pore over rankings. You don't pour over them.

Sorry, a pet peeve of mine.

Tom W. Bell said...

Ted: Heya! Congrats on the new blog--and the (upcoming) new job!

I dunno' about the Harvard/USN&WR conspiracy. I think we would need to know a lot more--such as whether other changes *helped* Chicago--before seizing on such a simple explanation. I do agree, of course, that the U of C get under-rated. I'm just not convinced of the reasons for that. And, at all events, I rather doubt that Mr. X bears the blame; those sorts of decisions probably came from higher-up.

Dave: Ooo, you are *so* right. Thanks. I fixed it.

Jeff Stake said...

Mr. X's discretion over weights might make greater differences in lower rankings because the scores are more closely spaced than at the top. The USNEWS choices on criteria and weights also have some nasty effects on the operation of law schools, effects for which USNEWS has not been willing to take responsibility.