Wednesday, May 31, 2006

How to Model USN&WR's Law School Rankings

In a just-completed series of posts, I described recent changes to the methodology that U.S. News and World Report uses to rank law schools. You can find links to those posts below. I now take up a related topic: My model of the USN&WR law school rankings. I'll start, here, by describing how I "reverse engineered" the rankings and by offering some observations about the challenges of that task. In future posts, I'll discuss why I tried to model USN&WR's law school rankings and how well I did.

In brief, I modeled the USN&WR rankings by trying to do the same things that USN&WR did to the same data that it used. That verb—"trying"—merits emphasis. With regard both to USN&WR's methodology and data, I cannot be absolutely sure that I've copied the original. With regard to the data, in particular, I've at best only come pretty close. Allow me to explain.

So far as methodology goes, I of course made reference to the explanation that USN&WR publishes on its website. That proved insufficiently detailed for my purposes, however. I gleaned further details from various other sources, most notably from Robert J. Morse and Samuel Flanigan, the Director and Deputy Director of Data Research, respectfully, at U.S. News. Those busy gentlemen proved very patient and helpful. Though there may remain some minor variations between the methodologies we use, I am reasonably confident that USN&WR does not calculate law school scores in a materially different way from my model.

Copying the data USN&WR uses proved more difficult. I sometimes had the same data that USN&WR did use, sometimes had the data that USN&WR should have used, and sometimes had to guess what data it might have used. Insofar as possible, I used the same data that USN&WR publishes along with its ranking of law schools. In many categories, however, USN&WR does not disclose what law schools told it. In those cases, I made reference to the ABA data that law schools were supposed to have copied onto their USN&WR questionnaires. Only with regard to one datum—the cost of living index that USN&WR applies to each school's reported total expenditures on instruction and administration, one of the components of the Overhead/Student indicator—did I have to generate my own numbers.

The following table offers some details about the methodology used by, and the sources of the data used in, the USN&WR rankings and my model thereof:

DataWeight
in
Ranking
USN&WR's
Source
Published
by
USN&WR?
Model's Source
Peer Rep 25.00% USN&WR survey yes USN&WR rankings
Bar Rep 15.00% USN&WR survey yes USN&WR rankings
Emp at 9 Mos 14.00% ABA data
schools reported
to USN&WR
yes USN&WR rankings
LSAT Median 12.50%ABA data
schools reported
to USN&WR
no ABA data
GPA Median 10.00% ABA data
schools reported
to USN&WR
no ABA data
Overhead Exp/Stu 9.75%ABA data
schools reported
to USN&WR
no ABA data
CoL Index for
Overhead Exp/Stu
N.A. Bought by USN&WR
from 3rd party
no estimated
Emp at 0 Mos 4.00% Reported by schools
to USN&WR
yes USN&WR rankings
Stu/Fac Ratio 3.00%ABA data
schools reported
to USN&WR
yes USN&WR rankings
Acceptance 2.50%ABA data
schools reported
to USN&WR
yes USN&WR rankings
Bar Pass Rate 2.00% ABA data
schools reported
to USN&WR
yes USN&WR rankings
Fin Aid
Exp/Stu
1.50%ABA data
schools reported
to USN&WR
no ABA data
Library
Resources
.75%ABA data
schools reported
to USN&WR
no ABA data


Although figuring out all the details of the methodology used by USN&WR took some time and effort, it did not prove unnecessarily difficult. I wish I could say the same about figuring out the data used by USN&WR. There, the hurdles proved not only higher, but inexplicably so.

As the above table indicates, my search for the data used in the rankings sometimes called on me to do no more than refer to the figures published by USN&WR. In many cases, however—and for no apparently sound reason—USN&WR declined to disclose the data it used. Why does USN&WR publish schools' 25th and 75 percentile LSAT and GPA scores but not publish the medians it actually uses in calculating its rankings? Interested parties could perhaps use the median LSATs and GPAs published by the Law School Admissions Council, but that data is hardly easy to download. More crucially, why doesn't USN&WR publish the data about expenditures and financial aid that schools report to it and the ABA? That data, worth over 10% of each school's score, appears in no public source. As I'll discuss in more detail later, USN&WR could greatly improve its rankings simply by letting all of us know more about the data that it uses. In other words, every entry in the above table's "Published by USN&WR?" should read, "yes."

Earlier posts about the 2007 USN&WR law school rankings:

No comments: