As in every year since 2005, I’ve again built a model of the U.S. News & World Report ("USN&WR") law school rankings. This latest effort generated a record-high r-squared coefficient: .998673. More about what that means—and more about the one law school that doesn’t fit—below. First, here’s a snapshot comparison of the scores of the most recent (USN&WR calls them “2013”) law school rankings and the model:
As that graphical comparison indicates, the model replicated USN&WR’s scores very closely. Indeed, the chart arguably overstates the differences between the two sets of scores because it shows precise scores for the model but scores rounded to the nearest one for USN&WR.
As I mentioned above, comparing the two data sets generates an r-squared coefficient of .998673. That comes very close to an r-squared of 1, which would show perfect correlation between the two sets of scores. Plainly, the model tracks the USN&WR law school rankings very closely.
In most cases, rounding to the nearest one, the model generated the same scores as those published by USN&WR. In four cases, the scores varied by 1 point. That’s not enough of a difference to fuss over, given that small variations inevitably arise from comparing the generated scores with the published, rounded ones. Consider, for instance, that USN&WR might have generated a score of 87.444 for the University of Virginia School of Law and published it as “87.” The model calculates Virginia’s score in the 2013 rankings as 88.009. The rounded and calculated scores differ by 1.009. But if we could compare the original USN&WR score with the model’s score would get difference of only .565 points. I won’t worry over so small a difference.
You know what does worry me, though? Look at the far right side of the chart above. That red “V” marks the 4.48 difference between the 34 points USN&WR gave to the University of Idaho School of Law and the score that the model generated. Idaho showed a similar anomaly in last year’s model, though then it was not alone. This year, only Idaho does much better in the published rankings than in the model.
[Crossposted at Agoraphilia and MoneyLaw.]
Tuesday, June 12, 2012
Accuracy of Model of the 2013 USN&WR Law School Rankings
Tuesday, May 22, 2012
U.S. News & World Report Improves Transparency of Law School Rankings
Huzzah for U.S. News and World Report! The most recent edition of its law school rankings includes the median LSAT and GPA of each school’s entering class. Finally. I have long argued that USN&WR should publish all of the data that it uses in its rankings. How else can the rest of us (read: rankings geeks) understand how—and, indeed, whether—the rankings work? Though USN&WR remains short of that ideal, disclosing median LSATs and GPAs represents a major step towards making the rankings more transparent and, thus, trustworthy.
USN&WR started the trend towards transparency last year, when it began publishing the “volume and volume equivalents” measures that it uses in its law school rankings. That input counts for only .75% of a school’s score, however. Median LSATs and GPAs together count for 22.5% of a school’s score, in contrast, making their disclosure by USN&WR all the more helpful.
There remain only two categories of data that USN&WR still uses in its law school rankings but does not disclose: overhead expenditures/student (worth 9.75% of a school’s score in the rankings) and financial aid expenditures/student (worth 1.5%). It isn’t evident why USN&WR declines to publish those inputs, too, though perhaps the financial nature of the data raises special concerns. If USN&WR cannot bring itself to publish overhead expenditures/student and financial aid expenditures/student, however, it should abandon those measures. They serve as poor proxies for the quality of a school’s legal education and if we cannot double-check the figures we cannot trust their accuracy.
[Crossposted at Agoraphilia and MoneyLaw.]
Friday, August 12, 2011
Volokh on the Constitution's Plain, Present, Public Meaning
My friend, Eugene Volokh, has posted some thoughtful comments on my consent-based approach to reading the Constitution. His excellent questions show one benefit of looking for the plain, present, public meaning of the Constitution: It gives us a fresh look at a document we risk treating as putty for judges or as a dead historical artifact.
I won't pretend that a consent-based approach to reading the Constitution solves all our interpretive problems--especially hard problems like those Eugene raises--but every theory struggles with the text. Looking for the plain, present, public meaning of the Constitution offers at least another way to tackle the problem. I'd argue, further, that it offers us the best way to maximize the consent of the governed.
Eugene cites a "number of terms that either don’t have a 'plain, present, public meaning' apart from either their originalist or precedential meaning," such as those in the First Amendment or other, well-known parts of the Constitution. I appreciate that observation--though again I emphasize that precedent-based "living" constitutionalists and originalists struggle with the same passages. I won't bother, here, to try to answer each puzzle that Eugene offers, as I am more interested in describing a methodology than I am in dictating particular results. I have opinions about such things, to be sure, but I'm no judge.
How should real-world judges interpret constitutional words or phrases that the public understands to have a specialized meaning (e.g., "establishment of religion") or no discernible meaning at all (e.g., "writ of habeas corpus")? The same way they would interpret and construct a contract in similar circumstances: interpreting legal terms as such, taking heed of their context, and constructing uncertainties against the drafting party (the U.S. federal government, here) and in favor of the adhering party (citizens and residents of the U.S.).
That does not lead to simply following Supreme Court precedents. I doubt that Kelo would come out the same way under that approach, for instance, nor that "Property" would end up with a different meaning each time it appears in the Constitution. Note, too, that the approach I describe leaves room for considering original meaning, which we should treat as something akin to "course of dealing" in the contract context. By borrowing the methodologies of contract law, a court can come as close as possible to finding the plain, present, public meaning of popularized constitutional phrases and uncommon terms such as "writ of habeas corpus." Perfection remains elusive, here as with other approaches to the Constitution. But this approach offers the best guarantee of preserving the consent of the governed.
Monday, August 08, 2011
The Constitutional Meaning of “Property” and the Rule of Law
Although the U.S. Constitution uses “Property” four times, it nowhere defines the term. What does it mean? I’ve been grappling with that question this summer, and been surprised to find commentators and cases arguing that “Property” counts as different things in different places. On some accounts, for instance, the word means only land in Article IV, § 3, general common law property in the Takings Clause, and all that plus welfare entitlements in the Fourteenth Amendment’s Due Process Clause.
Giving constitutional property so many different definitions threatens the rule of law. An average citizen, using ordinary English, would not likely read “Property” to stand for something different each time the word appears in the Constitution. Yet both leading theories of constitutional interpretation—originalism and “living” constitutionalism—invite that sort of confusion. Both theories avow that the meaning of "Property" or another constitutional term can change from place to place because of historical accidents (in the case of the former) or Supreme Court decisions (in the case of the latter).
I thank my friend and fellow law prof, Eugene Volokh, for discussing some of these matters in an extended email exchange. He recently quoted some of his thoughts about how constitutional words—he uses “Place” and “Law” as examples—can change depending on context. He makes sound observations. In fact, I'll go Eugene’s examples one better. Consider the word, "it," which means "House" in Art. § 5, "Bill" in Art. I, § 7, cl. 2, "Writ of Habeas Corpus" in Art. I, § 9, cl. 2, "State" in Art. I, § 10, cl. 2, and so forth.
We should expect such semantic flux if, as Eugene rightly says, the Constitution is written in ordinary English. Query, though, whether the four appearances of "Property" in the Constitution appear in contexts that give it four (or even, on one accounting, five!) distinct meanings. I think not, though I remain open to persuasion otherwise.
Eugene's observations do nothing to save originalism or living constitutionalism from the charge that they offend the rule of law, however. The rule of law does not suffer if we read words in their constitutional context because ordinary speakers of ordinary English can figure out that “it” means "House" in one place and "Writ of Habeas Corpus" in another. But originalism and living constitutionalism raise a different problem, given that they base constitutional meaning on historical usage (in the first instance) or Supreme Court precedents (in the second). Only specialists in constitutional law—and not even all of them—have the expertise to engage in that sort of decoding process. Did you know, for instance, that “Property” includes government entitlements in the Fourteenth Amendment but not in the Fifth? If so, I doubt you figured it out from reading the Constitution, alone.
How do we fix this problem with both originalism and living constitutionalism? By rejecting those theories for one that gives the Constitution’s text its plain, present, public meaning. For more, see my paper, Graduated Consent in Contract and Tort Law: Toward a Theory of Justification, 61 Case Western L. Rev. 17 (2010), a slightly revised version of the download-ready, Graduated Consent Theory, Explained and Applied, Chapman University School of Law, Legal Studies Research Paper Series, Paper No. 09-13 (March 2009) [PDF format].
Ratification may be necessary to make a Constitutional term effective in the first place, but it is not sufficient to make an original meaning binding on subsequent generations. People today, using ordinary English, almost certainly regard public flogging as unconstitutionally cruel and unusual punishment. In contrast, Justice Scalia has argued that an originalist (such as himself) should (and does) regard public flogging not as unconstitutional but simply, "stupid." That he can offer so tepid a criticism of something almost any citizen would regard as beyond the pale demonstrates the salient gap between an originalist approach and a consent-based one. (Living constitutionalism's abuse of ordinary English makes it susceptible to a similar critique.)
Saturday, April 30, 2011
U.S. Supreme Court Prediction Market
Recently posted to SSRN: FantasySCOTUS: Crowdsourcing a Prediction Market for the Supreme Court, a draft paper by Josh Blackman, Adam Aft, & Corey Carpenter assessing the accuracy of the Harlan Institute's U.S. Supreme Court prediction market, FantasySCOTUS.org. The paper compares and contrasts the accuracy of FantasySCOTUS, which relied on a "wisdom of the crowd" approach, with the Supreme Court Forecasting Project, which relied on a computer model of Supreme Court decision making. From the paper's abstract:
During the October 2009 Supreme Court term, the 5,000 members made over 11,000 predictions for all 81 cases decided. Based on this data, FantasySCOTUS accurately predicted a majority of the cases, and the top-ranked experts predicted over 75% of the cases correctly. With this combined knowledge, we can now have a method to determine with a degree of certainty how the Justices will decide cases before they do. . . . During the October 2002 Term, the [FantasySCOTUS] Project’s model predicted 75% of the cases correctly, which was more accurate than the [Supreme Court] Forecasting Project’s experts, who only predicted 59.1% of the cases correctly. The FantasySCOTUS experts predicted 64.7% of the cases correctly, surpassing the Forecasting Project’s Experts, though the difference was not statistically significant. The Gold, Silver, and Bronze medalists in FantasySCOTUS scored staggering accuracy rates of 80%, 75% and 72% respectively (an average of 75.7%). The FantasySCOTUS top three experts not only outperformed the Forecasting Project’s experts, but they also slightly outperformed the Project’s model - 75.7% compared with 75%.
You can download a copy of the draft paper here.
[Crossposted at Agoraphilia, Midas Oracle, and MoneyLaw.]
Saturday, April 23, 2011
The Power of Property Rights
Why have property rights? This brief video, which I wrote and narrated for LearnLiberty, explains. Set in my hometown of San Clemente, the video co-stars some of the most lovely scenery that SoCal can offer.
Wednesday, April 20, 2011
Martin Interviews Bell for Kosmos Online
Chris Martin recently interviewed me for Kosmos Online, a project sponsored by the friendly folks at the Institute for Humane Studies. Our podcast covered the Socratic method, measuring class participation, and other teaching techniques used in law school and other settings. You can listen here or read a rough transcript here.
(Note that this semester, at the suggestion of my students, I decided to try an approach to grading classroom participation different from the one I described in my interview with Chris. So far, so good. The experiment has yet to finish, though, so I’m not sure what technique I’ll use next year.)
Wednesday, February 09, 2011
Hoffman Interviews Bell on Kosmos Online
Jeanne Hoffman, of Kosmos Online, a project sponsored by the friendly folks at the Institute for Humane Studies, recently recorded a podcast interview with me. Our discussion covered such things as career engineering, how IHS helped me win freedom, and the current state of intellectual property scholarship. You can listen here or read a rough transcript here.
Tuesday, February 01, 2011
FedSoc Presentation on Libertarian—But Not Originalist!—Consitutionalism
Should libertarians interpret the Constitution according to its original meaning? I argue not. Instead, they should read the Consitution according to its plain, present, public meaning. I gave a very brief presentation on the topic at the 13th Annual Federalist Society Annual Conference, held in San Francisco on Friday, January 2011:
Among other things, this “consensualist” approach to constitutional theory implies that we should:
- Interpret the Constitution according to its plain, present, public meaning;
- Construct the Constitution as we would a standard form agreement, thus favoring individual rights over federal powers; and
- Decide cases involving Constitutional rights by Citizen Courts structured akin to the panels that help to guarantee the fairness of commercial arbitration procedures.
For an earlier but more detailed account of my views, see this post about my presentation at the 2010 Students for Liberty Southern California Regional Conference.
Thursday, December 16, 2010
Z-Scores in Model of 2011 USN&WR Law School Rankings
As I have for each of the past several years, I this year again built a model of the most recent U.S. News and World Report ("USN&WR") law school rankings. This year's model matched the publishing rankings very nicely; comparing the model's scores with the published ones generated an r-squared of .997 (where 1 would indicate perfect correspondence). At the request of my readers, I here offer the weighted z-scores of the top-tier schools from last spring's (the "2011") USN&WR law school rankings:
Why do my fellow rankings geeks care about z-scores? In brief, these z-scores measure how well each school performed relative to its peers, thereby establishing its rank. (See here for a fuller explanation.) Because USN&WR uses z-scores to rank law schools, so too must any model of its rankings.
I weighted these z-scores simply by multiplying the z-score for each school, in each category of data, by the percentage that that category influences a school's overall score in USN&WR's rankings. That method of presenting z-scores has the virtue of highlighting which scores matter the most. You will thus generally find the largest weighted z-scores in the upper, left-hand corner of the chart, for instance, where lie both the most important categories of data and the law schools that scored the highest the rankings.
Consider, for instance, the weighted z-scores of .68 enjoyed by both Yale and Harvard under the "PeerRep" category. Numbers that large (comparatively speaking) overwhelm the effect of other measures of those schools' performances—the schools' BarRep scores, at .39 each, come in a distant second—and have twice the impact of the peer reputation scores of schools ranked as close as 20th from the top.
Using weighted z-scores also has the virtue of showing how very little influence many of the things that USN&WR measures have on its rankings. The weighted z-scores for Bar pass rates among top-tier schools, for instance, vary between only .07 and -.02.. Bar pass rates, however important to students, evidently do not matter much in USN&WR rankings.
Why did it take me so long to finish this year's model? In large part, you can blame my prepping two new classes (Property and a Law & Economics seminar) and serving on Chapman's Dean Search Committee (an effort that should soon conclude with our announcment of a fantastic new leader for our law school). Notably, though, some of the delay stems from how the ABA manages its statistical take-offs. The ABA recently abandoned its former practice of routinely sending electronic copies of its statistical take-offs at the request of any subscribing school. Allegedly, some Deans had complained that to make the data available electronically would make modeling the USN&WR rankings too easy. Nice try, Deans! Also, the ABA this year neglected to send several subscribing schools, including my own, even hardcopies of the statistical takeoffs. We got a prompt response from the ABA when we finally figured out that we we were not to blame for the missing take-offs, but the mix up still impeded my efforts. Again, though, geekery finally prevailed.
Interested in prior years' z-scores? Here are the ones from the 2010 rankings, the 2008 rankings, the 2007 rankings, the 2006 rankings, and the 2005 rankings.
[Crossposted at Agoraphilia, MoneyLaw.]
Wednesday, October 27, 2010
Three Kinds of Diversity
Diversity comes in many flavors. I here compare three types—diversity of skin color and sex, cultural diversity, and ideological diversity—and offer some observations about the distinctive costs and benefits of each. I conclude that, holding all else equal, a group of people having diverse colors and sexes will enjoy modest institutional gains at low cost, while a group touting ideological diversity runs the risk of high transaction costs but wins a shot at great intellectual gains. Groups with high cultural diversity fall in between those two extremes.
Diversity of skin color and sex appears on the face of a group, thus offering ready proof that its selection, such as through hiring and promotion, was not tainted with invidious discrimination. Holding all else equal—assuming, specifically, that the racially and sexually diverse group does not possess above-average cultural and ideological diversity—the costs of intra-group transactions remain low. Thus, for instance, might a facially diverse group of culturally and ideologically similar people get along very smoothly. Think, here, of an elite law school where every professor has absorbed Ivy League norms and all lean moderately left. They might bicker, of course; law professors specialize in that. But such a culturally and ideologically uniform group is not likely to host nasty public fights about ballot initiatives or the like.
Are there downsides to pursuing diversity of skin color and sex in hiring and promotion? Not if you can find enough well-qualified candidates, and not if you avoid discriminating against candidates for blameless having an uninteresting color or sex. Happily, it is not too hard to satisfy both conditions, these days.
Cultural diversity proves harder to document, and runs some risk of increasing intra-group transaction costs. Someone brought up solely within the confines of respectable East Coast institutions will have to work a bit to understand a peer raised Mormon, in Utah's backcountry. So, too, might differences of sexual orientation (which like cultural differences generally do not appear on a person's face) sometimes lead to innocent misunderstandings. Holding equal for other sorts of diversity, however, cultural differences offer many charms and few serious costs. Most of us, and especially those of us in academia, enjoy meeting friendly people with exotic backgrounds. When we share ideologies, moreover, meeting fellow travelers who differ from us suggests that our most heartfelt values transcend race, sex, and culture—a comforting, if somewhat smug, idea.
Ideological diversity, standing alone, proves at least as hard to document as cultural diversity—it does not appear on a person's face nor even, typically, in a person's dress or hairstyle—and much more likely to raise intra-group transaction costs. Religious differences prove largely intractable, though in polite society we tend to keep them private. Political differences, at least in American institutions, threaten to burst out into loud and public disagreements, however. Such frank exchanges can help each side to hone its arguments, of course, and thus offers the prospect of considerable gains both to the disputants and the group that harbors them both. But if local norms do not temper the tone and proper boundaries of ideological debate, transactions costs can easily soar, making it hard for a group to manage even run-of-the-mill functions efficiently.
In sum: diversity of skin color and sex offers few costs and modest benefits; cultural diversity creates slightly higher transaction costs but compensates with intriguing charms; and ideological diversity presents a high risk/high return strategy for institutions devoted to generating new and useful ideas.
[Crossposted at Agoraphilia, and MoneyLaw.]
Thursday, May 20, 2010
U.S. News: Less Transparency = More Fairness
Robert Morse today announced that, in response to evidence that law schools had been gaming its rankings, U.S. News would change the way it estimates the "Employment at 9 Months" measure for schools that decline to report that figure. Paul Caron offers some background here. Said Morse: "U.S. News is planning to significantly change its estimate for the at-graduation rate employment for nonresponding schools in order to create an incentive for more law schools to report their actual at-graduation employment rate data. This new estimating procedure will not be released publicly before we publish the rankings."
I understand that U.S. News generated the formula it formerly used to estimate the Emp9 figure for non-reporting schools by running a regression comparing the Emp0 and Emp9 data from reporting schools. It used to puzzle me that U.S. News did not evidently re-run the regression each year, but rather stuck with the original estimate. In retrospect, though, I see that sticking to the same formula might have partially helped U.S. News offset the gaming it so dislikes. After all, as more and more schools with low numbers refused to report Emp9 data, opting to rely instead on the publicized formula, the correlation between Emp0 and Emp9 scores would change so as to favor non-reporting schools. Better to stick with the old formula, dated though it might be, than to increase the incentive to opt out of reporting.
U.S. News thus avoided a vicious cycle, but only at the cost of signaling to schools exactly when hiding Emp9 data would help their rankings. Will its new reticence work? Schools can now only guess at how U.S. News will turn Emp0 numbers into Emp9 estimates, and will rightly worry that they might misjudge the new cutoff. Even if big-E ethics does not counsel reporting Emp9 numbers, therefore, small-c conservatism will. Granted, a school might reason, "U.S. News will still try to find a reasonably accurate way to turn Emp0 data into Emp9 estimates, and it has always helped us to not report in the past, so it remains a gamble worth taking." But such schools should also rightly worry that U.S. News might throw a punitive little kick into its new formula, to encourage schools to worry more about accuracy than about rankings.
[Crossposted at Agoraphilia and MoneyLaw.]
Tuesday, April 27, 2010
The Hand Rule
Judge Learned Hand famously opined that if the burdens of preventing an accident outweigh its cost multiplied by its probability, it does not constitute carelessness to avoid those burdens. Doesn't that little gem make you want to break out in song? I've got just the thing: The Hand Rule, a little ditty I recently composed and played for some students at Chapman Law School.
Though I've yet to record The Hand Rule, I can offer you a .pdf of the lyrics and chords as well as a PowerPoint, complete with pictures of Learned Hand, to accompany the performance (both uncopyrighted). Here's a sample of a verse and the refrain:
In the case of Carrol Towing Co., Learned Hand set forth to showSilly? Yes, but it gets students to pay attention and remember what they learn. So goes the modus operandi of the Law and Fun school.
The meaning of "reasonability."
Defendant failed to leave in charge, a man to watch its unmoored barge.
And plaintiff's cargo met calamity.
"Negligence!" plaintiff complained and on appeal, Judge Hand explained,
The proper scope of liability.
Learned, learned, Learned. Learned in the law was he.
Learned Judge Hand, Learned, he judged so learnedly!
So learn what the Hand Rule teaches: "There's no liability,
If the burden of the cost exceeds the loss times the probability."
[Crossposted at Agoraphilia and MoneyLaw.]
Tuesday, January 05, 2010
My Favorite Motions
Faculty meetings may have their charms, but efficiency does not rank among them. Many a time I have looked around a room full of my colleagues, long minutes into a winding discussion of what was supposed to take only a few moments to resolve, considered the full agenda still stretching before us, and bemoaned the deadweight social costs of law school governance. Allow me, then, to share a couple of partial cures—one an old favorite and the other a new find—from Robert's Rules of Order.
I've long been a fan of "calling the question," as we casually style the motion at my school. Full-on Robert's geeks know it as the "Previous Question" motion. Call it what you like, you have to love its effect: It takes precedence over every debatable question and, if the motion carries, forces a vote on the issue under debate.
Suppose, for instance, that a handful of faculty members have been arguing back and forth about some relatively inconsequential motion for 20 minutes or so, as everyone else's attention wanders and more important business goes untended. You get the Chair to recognize you and simply say, "I move to call the question." Once the motion carries—and often with sighs of relief—you and your colleagues can vote on the trifling motion and move on to other topics. (Section 20 of the Rules offers caveats and details, but most law school faculties seem to manage, surprisingly enough, with less than the full panoply of formalities.) Try calling a question the next time a faculty meeting starts spinning its wheels. You—and most your colleagues—will enjoy the ride.
Calling the question does not cure all the inefficiencies that afflict faculty meetings, however. Because we law profs so love to hear ourselves speak, for instance, we sometimes run on (and on and on) a bit. Polite coughs, finger drumming, and the like usually suffices to keep our monopolizing tendencies in control, happily. In fact, it was only very recently that I found myself wondering what a fellow could do when those informal measures failed. Here, too, Robert's Rules offers a remedy: a Question of Order pertaining to decorum.
Roberts Rule's provides, in § 34, that "no member shall speak more than twice to the same question . . . nor longer than ten minutes at one time, without leave of the assembly, and the question upon granting the leave shall be decided by a two-thirds vote [§ 39] without debate." Upon encountering an infraction of that rule, you have the right to interrupt the speaker. As section 14 says, one who so objects "shall rise from his seat, and say, 'Mr. Chairman, I rise to a point of order.'" The Chair must then decide the issue immediately, without debate. If the Chair finds the challenged speaker out of order, and if anyone objects to the speaker continuing, he or she must cede the floor unless the assembly votes to grant leave.
That sounds like strong medicine, granted, and would doubtless ruffle some feathers. But faculty meetings pose a classic tragedy of the commons, one where just a few overly-talkative people risk consuming far more than their fair share of everyone else's time and attention. Raising a Question of Order can help you save you—and thus your school—from the perils of a grossly inefficient faculty meeting.
[Crossposted at Agoraphilia, MoneyLaw.]
Sunday, August 30, 2009
How Top-Ranked Law Schools Got That Way, Pt. 3
Part one and part two of this series focused on the top law schools in U.S. News and World Report's 2010 rankings, offering graphs and analysis to explain why those schools did so well. This part rounds out the series by way of contrast. Here, we focus on the law schools that ranked 41-51 in the most recent USN&WR rankings, those that ranked 94-100, and the eight schools that filled out the bottom of the rankings.
The above chart shows the weighted and itemized z-scores of law schools about 1/3rd of the way from the top of the 2010 USN&WR rankings. Note the sharp downward jog at Over$/Stu—a residual effect, perhaps, of the stupendously large Over$/Stu numbers we earlier saw among the very top schools. Note, too, that three schools here—GMU, BYU, and American U.—buck the prevailing trend by earning lower scores under PeerRep than under BarRep (GMU's line hides behind BYU's). As you work down from the top of the rankings, GMU offers the first instance of that sort of inversion; all of the more highly ranked schools have larger itemized z-scores for PeerRep than for BarRep. It raises an interesting question; Why did lawyers and judges rank those schools so much more highly than fellow academics did?
The above chart shows the weighted, itemized z-scores of the law schools ranked 94-100 in the 2010 USN&WR rankings—about the middle of all of the 182 schools in the rankings. As we might have expected, the lines bounce around more wildly on the left, where they trace the impact of the more heavily weighted z-scores, than on the right, where z-scores matter relatively little, pro or con. Beyond that, however, no one pattern characterizes schools in this range.
The above chart shows the weighted and itemized z-scores of law schools that probably did the worst in the 2010 USN&WR rankings. I say, "probably," because USN&WR does not reveal the scores of schools in the bottom two tiers of its rankings; these eight schools did the worst in my model of the rankings. Given that uncertainty, as well as for reasons explained elsewhere, I decline to name these schools.
Here, as with the schools at the very top of the rankings, we see a relatively uniform set of lines. All of the lines trend upward, of course. These schools did badly in the rankings exactly because they earned strongly negative z-scores in the most heavily weighted categories, displayed to the left. Several of these schools did very badly on the Emp9 measure, and one had a materially poor BarPass score. Another of them did surprisingly well on Over$/Stu, perhaps demonstrating that, while the very top schools boasted very high Over$/Stu scores, no amount of expenditures-per-student can salvage otherwise dismal z-scores.
[Crossposted at Agoraphilia, MoneyLaw.]
Sunday, August 23, 2009
How Top-Ranked Law Schools Got That Way, Pt. 2
In the first post in this series, I discussed the mysterious distribution of maximum z-scores in the top two tiers of law schools in U.S. News & World Report's 2010 rankings, and focused on the top-12 schools to solve that mystery. In brief, among the very top schools, employment nine months after graduation" ("Emp9") varies too little to make much of a difference in the schools' overall scores, whereas overhead expenditures/student ("Over$/Stu") varies so greatly as to almost swamp the impact of the other factors that USN&WR uses in its rankings. Here, in part two, I focus on the top 22 law schools in USN&WR's 2010 rankings. In addition to the Emp9 and Over$/Stu effects observed earlier, this wider study uncovers some other interesting patterns.
The above graph, "Weighted & Itemized Z-Scores, 2010 Model, Top-22 Schools," offers a snapshot comparison of how a wide swath of the top schools performed in the most recent USN&WR rankings. It reveals that the same effects we observed earlier, among just the top-12 schools, reach at least another ten schools down in the rankings. With the exception of Emory and Georgetown, Emp9 scores (indicated by the dark blue band) barely change from one top-22 school to another. Over$/Stu scores, in contrast (indicated by the middle green hue), vary widely; compare Yale's extraordinary performance on that measure with, for instance, Boston University's.
This graph also reveals some other interesting effects. Like the Emp9 measure, the Emp0 measure (for "Employment at Graduation," indicated in yellow-green) varies little from school to school. Indeed, it varies even less than the Emp9 measure does. Why so? Because all of these top schools reported such high employment rates. All but Minnesota reported Emp0 rates above 90%, and all but Georgetown, USC, and Washington U. reported rates above 95%.
These top 22 schools also reported very similar LSATs. Their weighted z-scores for that measure, indicated here in light blue, range from only.20 to .15. The weighed z-scores for GPA, in contrast, marked in dark green, range from .24 to .06.
As the graph indicates, the measures worth 3% or less of a school's overall score—student/faculty ratio, acceptance rate, Bar exam pass rate, financial aid expenditures/student, and library volumes and equivalents—in general make very little difference in the ranking of these schools. One exception to that rule pops up in the BarPass scores (in dark orange) of the California schools, which benefit from a quirk in the way that USN&WR measures Bar Pass rates. Another interesting exception appears in Harvard's Lib score (in white)—only thanks to its vastly larger law library does Harvard edge out Stanford in this ranking.
To best understand how a few law schools made it to the top of USN&WR's rankings, we should contrast their performances with those of the many schools that did not do as well. I'll thus sample the statistics of the law schools that ranked 41-51 in the most recent USN&WR rankings, those that ranked 94-100, and the eight schools that filled out the bottom of the rankings. Please look for that in the next post.
[Crossposted at Agoraphilia, MoneyLaw.]
Thursday, August 20, 2009
How Top-Ranked Law Schools Got That Way, Pt. 1
How do law schools make it to the top of the U.S. News & World Report rankings? USN&WR ranks law schools based on 12 factors, each of which counts for a certain percentage of a school's total score. Peer Reputation counts for 25% of each law school's overall score, for instance, whereas Bar Passage Rate counts for only 2%. More precisely, USN&WR calculates z-scores (dimensionless statistical measures of relative performance) for each of the 12 factors for each school, multiplies those z-scores by various percentages, and sums each school's weighted, itemized z-scores to generate an overall score the school. USN&WR then rescales the scores to run from 100 to zero and ranks law schools accordingly.
In earlier posts I described my model of the most recent U.S. News & World Report law school rankings (the "2010 Rankings"), quantified its accuracy, and published itemized z-scores for the top two tiers of schools. (Separately, I also suggested some reforms that might improve the rankings.) Studying those z-scores reveals a great deal about how the top-ranked law schools got that way. The lessons hardly jump out from the table of numbers, though, so allow me to here offer some illustrative graphs.
The above graph, "Weighted & Itemized Z-Scores of Top 100 Law Schools in Model of 2010 USN&WR Rankings," reveals an interesting phenomenon. The items on the left of the graph count for more of each school's overall score, whereas the items on right count for less. We would thus expect the line tracing the maximum weighted z-scores for each item to drop from a high, at PeerRep (a measure of a school's reputation, worth 25% of its overall score), to a low, at Lib (a measure of library volumes and equivalents, worth only .75%). Instead, however, the maximum line droops at Emp9 (employment nine months after graduation) and soars at Over$/Stu (overhead expenditures per student). The next graph helps to explain that mystery.
The above graph, "Weighted & Itemized Z-Scores, 2010 Model, Top-12 Schools," reveals two notable phenomena. First, the Emp9 z-scores, despite potentially counting for 14% of each school's overall score, lie so close together that they do little to distinguish one school from another. In practice, then, the Emp9 factor does not really affect 14% of these law schools' overall scores in the USN&WR rankings. (Much the same holds true of top schools outside of these 12, too.)
Second, the Over$/Stu z-scores range quite widely, with Yale having more than double the score of all but two schools, Harvard and Stanford, which themselves manage less than two-thirds Yale's Over$/Stu score. That wide spread gives the Over$/Stu score an especially powerful influence on Yale's overall score, making it almost as important as Yale's PeerRep score and much more important than any of the school's remaining 10 z-scores. In effect, Yale's extraordinary expenditures per student buy it a tenured slot at number one. (I observed a similar effect in last year's rankings.)
Other interesting patterns appear in "Weighted & Itemized Z-Scores, 2010 Model, Top-12 Schools." Note, for instance, that Virginia manages to remain in the top-12 despite an unusually low Over$/Stu score. The school's strong performance in other areas makes up the difference. Though it is not easy to discern from the graph, Virginia's reputation and GPA scores fall in the middle of these top-12 schools' scores. Northwestern offers something of a mirror image on that count, as it remains close to the bottom of the top-12 despite a disproportionately strong Over$/Stu score. The school's comparatively low PeerRep and BarRep scores (the lowest of those in the top-12) and GPA (nearly tied for the lowest) score pull it down; Northwestern's Over$/Stu score saves it.
[Since I find I'm running on a bit, I'll offer some other graphs and commentary in a later post or posts.]
[Crossposted at Agoraphilia, MoneyLaw.]
Tuesday, August 04, 2009
Reforms Suggested by Modeling the Law School Rankings
As I recently observed, the close fit between law schools' scores in U.S. News & World Report's rankings and the scores of those same schools in my model of the ranking "suggests that law schools did not try game the rankings by telling USN&WR one thing and the ABA . . . another." Since both Robert Morse, Director of Data Research for USN&WR, and the ABA Journal saw fit to comment on that observation, perhaps I should clarify a few points.
First, I have no way of knowing whether or not law schools misstated the facts, by accident or otherwise, to both the ABA and USN&WR. The fit between USN&WR's scores and my model's scores indicates only that law schools reported, or misreported, the same facts to each party.
Second, this sort of consistency test speaks only to those measures USN&WR uses in its rankings, that it does not publish with its rankings, and that the ABA collects from law schools: median LSAT, median GPA, overhead expenditures/student, financial aid/student, and library size. Measures that USN&WR uses and publishes—reputation among peers and at the Bar, employment nine months after graduation, employment at graduation, student/faculty ratio, acceptance rate, and Bar exam performance—go straight into my model, so I do not have occasion to test their consistency against ABA data. In some cases—the reputation scores and the employment at graduation measure, the ABA does not collect the data at all. This proves especially troubling with regard to the latter. We have little assurance that USN&WR double-checks what schools report under the heading of "Employment at Graduation," and no easy way to double-check that data ourselves.
Third, and consequently, USN&WR could improve the reliability of its rankings by implementing some simple reforms. I suggested three such reforms some time ago. USN&WR has largely implemented two of them by making its questionnaire more closely mirror the ABA's and by publishing corrections and explanations when it discovers errors in its rankings. (I claim no credit for that development, however; I assume that USN&WR acted of its own volition and in its own interest.)
Another of my suggested reforms remains as yet unrealized, however, so allow me to repeat it, here: USN&WR should publish all of the data that it uses in ranking law schools. It could easily make that data available on its website, if not in the print edition of its rankings. Doing so would both provide law students with useful information and allow others to help USN&WR double-check its figures.
To that, I now add this proposed reform: USN&WR should either convince the ABA to collect data on law school graduates' employment rates at graduation or discontinue using that data in its law school rankings. That data largely duplicates the more trustworthy (but still notoriously suspect) "Employment at Nine Months" data collected by the ABA and used by USN&WR in its rankings. And, unlike that data, law schools do not report "Employment at Graduation" numbers under the threat of ABA sanctions. We cannot trust the employment at graduation figures and USN&WR does not need them.
Among the reforms I suggested some two years ago I also included one directed at the ABA, calling on it to publish online, in an easily accessible format, all of the data that it collects from law schools and that USN&WR uses in its rankings. I fear that, in contrast to USN&WR, the ABA moved retrograde on that front. I leave that cause for another day, however; here I wanted to focus on what my model can tell us about USN&WR's rankings.
[Crossposted at Agoraphilia, MoneyLaw.]
Thursday, July 23, 2009
Z-Scores in Model of 2010 USN&WR Law School Rankings
If you want to know how U.S. News & World Report's law school rankings work, you'll want to know about z-scores. In very brief, z-scores measure how well each school performed relative to its peers, thereby establishing its rank. (See here for a fuller explanation.) My model of the rankings aims to recreate those z-scores, and thus the rankings themselves, by duplicating both the data and the methodology that USN&WR uses. Here are the results for the law schools most recently ranked in the top 100:
For cross-year comparisons, please see the similar reports I offered in 2005, 2006, 2007, and 2008. This year, in response to a reader's request, I've added various diagnostic measures, such as the mean, median, and standard deviation of each itemized category of data. As I did last year, I again provided weighted z-scores, meaning simply that I've multiplied the z-scores in each category of data by the percentage that category influences a school's overall score. That method of presenting z-scores has the virtue of highlighting which scores matter the most.
Unsurprisingly, you'll generally find the largest numbers in the upper, left-hand corner of the chart. There lie the most heavily-weighted z-scores of the law schools that scored the highest in USN&WR's rankings. Consider, for instance, the .71 weighted z-scores enjoyed by Yale and Harvard under the "PeerRep" category; those numbers nearly swamp the effect of other measures of those schools' performances, and have twice the impact of the peer reputation scores of schools ranked as close as 20th from the top.
This presentation of the data also shows how very little influence many of the things that USN&WR measures have on its rankings. The weighted z-scores for Bar pass rates, for instance, vary between only .07 and -.02, with a whole lot of zeros filling that span. Bar passage rates evidently do not matter much to any school's USN&WR score.
Rankings geeks will doubtless find close study of this table rewarding. I'm especially interested in the surprising impact of the top schools' overhead expenditures/student—a phenomenon that I discussed in some detail last year. Perhaps I'll return to that topic, and raise some new ones, in later posts. In the meantime, I welcome your own observations.
[Crossposted at Agoraphilia, MoneyLaw.]
Wednesday, July 22, 2009
Accuracy of the Model of the 2010 USN&WR Law School Rankings
I earlier offered a snapshot comparison of the scores generated by my model of the 2010 U.S. News & World Report law school rankings and the original. After Robert Morse, director of data research for USN&WR, asked me if I could quantify the fit between the two data sets, I realized that others might share his curiosity. Here, then, are the r-squared measures (more precisely, the squares of the Pearson product moment correlation coefficients) for each of the models I've done over the past few years:
2010 rankings: 0.999
2009 rankings: 0.999
2008 rankings: 0.999
2007 rankings: 0.997
2006 rankings: 0.995
What do those numbers mean? In brief, an r-squared closer to 1 (or –1) shows a closer fit between the two data sets. It might seem a bit absurd to report these results out to three decimals, but I wanted to make clear that the model has yet to obtain results absolutely identical to those reported by USN&WR. I daresay, though, that any r-squared above .99 shows a pretty strong correlation.
[Crossposted at Agoraphilia and MoneyLaw.]
