Friday, July 27, 2007

ABA v. USN&WR on "Employment at 9 Months" Data

I earlier discussed U.S. News and World Report's plan to change the way it measures "Employment at 9 Months" for its law school rankings, a change that will make it harder for schools to game their Emp9 numbers. An anonymous commentator on that post asked, "Anyone want to recompute the 2008 rankings using this 'more accurate' methodology based on the figures conveniently made available by the ABA in excel format?" I had to admit that the project would generate interesting results. And since I've created just such a model of the 2008 rankings, I figured I was the guy for the (tedious and nonpaying) job. That labor of love generated interesting results sooner than I'd expected, however.

USN&WR's published rankings include each law school's Emp9 score. The magazine calculates those scores using data from questionnaires it sends to the law schools it ranks—questionnaires that ask each school to repeat what it earlier said in reply to the American Bar Association's annual questionnaire. Happily for ranking geeks, the ABA recently began publishing that data in a downloadable Excel file. But when I plugged that ABA data into USN&WR's Emp9 formula, I discovered that it did not always generate the Emp9 scores that USN&WR used in its 2008 rankings. Herewith the problematic cases:

Table Contrasting USN&WR Emp9 Scores with Emp9 Scores Calculated Using ABA Data

I see four explanations for these divergences: 1) Errors in the ABA data; 2) Errors in the USN&WR data; 3) Errors in my calculations; or 4) Differences between what a law school told the ABA and USN&WR. The first two explanations seem most likely to me, but I of course cannot rule out the third. As for the last, it bears noting that a school's Emp9 data comes from the prior February—seven or eight months before a school fills out its ABA and USN&WR questionnaires. It's thus hard to imagine how a school could dig up new data, after so long a remove, in the relatively short span between when it fills out the former questionnaire and the latter.

With luck, I'll have firmer answers, soon. I've emailed administrators at the four law schools that stake out the extremes on the above list, asking if they could please help me understand this phenomenon. That out of the way, I plan soon to return to my original goal: recalculating the 2008 USN&WR rankings using the Emp9 formula that will officially take effect next year.

[Crossposted to MoneyLaw.]

Earlier posts about the 2008 USN&WR law school rankings:

Earlier posts about Emp9 measure:


Anonymous said...

Could I propose that this analysis distracts from the real problem?

The national employment rate is around 95%. Anyone with a highschool education, really with a pulse, can get *a job* in this country. Who cares if some school manipulated their emp@9 number from 91% to 94%?

What you guys need to look at is the quality of the jobs. At my law school it's the pits. It's awful. I am in utter shock at the terrible market value of a top 25% Loyola Law School degree. Most of my class work at small firms for $15/hour or so over the Summer, and then about $25/hour if they are lucky enough to get an offer.

Below are three recent entries on our job board. These are not cherry picked jobs either, they are the market.

Employer Name:
Contact Name: xx
Address: xxx
City: Woodland Hills, CA
Telephone: 818-xx
Facsimile: 818-xx
E-Mail: xx
Description: HOURS: Part-time. DURATION: Contract. SALARY: $11.00/hour. STUDENT LEVEL: 1L, 2L JOB DESCRIPTION: File Clerk - help organize files, organize storage boxes, some legal research and writing. EMPLOYER PRACTICE/DESCRIPTION: Commercial litigation, real estate, personal injury. FIRM SIZE: 3 attorneys. NOTES: Temporary position, but could possible become permanent. HOW TO APPLY: Submit resume and cover letter via email to xx
Date Entered: 07/25/07
Job ID: 413784
Contact Name: xx
Address: xx Wilshire Boulevard
City: Beverly Hills, CA xx
Telephone: xx
Facsimile: xx
E-Mail: xx
Description: HOURS: Part-time. DURATION: Permanent. SALARY: $20/hr negotiable. BAR ADMISSION: Required. WILLING TO CONSIDER: Recent Grads Awaiting Results, New Admits. YEARS OF EXPERIENCE: 1 QUALIFICATIONS: Some prior criminal defense and complex civil litigation. NOTES: Great opportunity to expand with firm. HOW TO APPLY: Submit resume and cover letter via email to xx
Date Entered: 07/25/07
Job ID: 413788
Employer Name:
Contact Name: xx
Address: xx
City: xx, CA xxx
Telephone: 310-xx
Facsimile: 626-xx
E-Mail: xxx
Description: HOURS: Part-time. DURATION: Contract. SALARY: $12/hour STUDENT LEVEL: 1L, 2L, 3L, grads awaiting results, grads admitted to bar. QUALIFICATIONS: Attentive, able to go to courthouse to check on files, not afraid to use the phone. JOB DESCRIPTION: Administrative/paralegal help needed to close out about 10 different types of cases at all stages of settlement and litigation. Possible legal research involved. EMPLOYER PRACTICE/DESCRIPTION: Personal injury, civil litigation. FIRM SIZE: one attorney NOTES: Closing law practice after 30 years and need administrative help closing out last 10 cases. Good learning experience from an experienced attorney. HOW TO APPLY: Submit resume via email to xx
Date Entered: 07/25/07
Job ID: 413810

Tom W. Bell said...


While I understand your concern about what jobs pay, the Emp9 data matters insofar as it has a big impact on how schools get ranked by USN&WR. If schools game their Emp9 numbers, students get fooled. So the analysis here goes hand-in-hand with your concern about fraud.

Speaking more directly to your concern, you might be interested to know that I've been trying to collect data on each school's mean and median graduate salaries. I've been chagrined to discover that NALP collects that data but does not publish it. Ultimately, I'd like to offer would-be students financial data that would let them run cost/benefit calculations on their JDs. Too many student have been burned.

Anonymous said...


The problem with mean and median numbers is that they on their own do not take into account issues such as cost of living or accurately reflect the difference between private and public sector jobs (and how many students choose either option). Of course, law school loans do not take COL into account either.

Tom W. Bell said...

Agreed, Daniel. We'd need quite a lot of data to give students a very good idea of whether going to law school is a good investment. But--golly!--it wouldn't take much for us to do better than we're doing now.