Tuesday, June 27, 2006

Baylor Explains the Data it Reported for the USN&WR Rankings

In earlier posts about U.S. News & World Report's law school rankings, I described a puzzle about the score given to Baylor University School of Law and offered some tentative answers. The available evidence indicates that Baylor got ranked based on median LSAT and GPA figures different from—and rather higher than—the figures that USN&WR asked for. How did that happen? I here relate the explanation that Baylor's Associate Dean, Leah Jackson, recently sent to me.

In very brief, she offers credible proof that Baylor answered USN&WR's online questionnaire accurately. How, then, did USN&WR end up attributing the wrong median LSAT and GPA figures to Baylor? Associate Dean Jackson's account strongly suggests that USN&WR mistook a follow-up fax from Baylor as a correction to that school's online form, rather than as the supplementary explanation that Baylor intended.

Associate Dean Jackson's six-page letter and supporting documents say a great deal that merits consideration. I've encouraged her to post the material online, and promised to link to it if she does. I here quote only some of the highlights.
The first item that needs to be clarified is the nature of our admissions process. . . . Since the beginning, Baylor Law School truly has operated on a traditional quarter system, with classes matriculating at least three of the four terms of each calendar year. . . . . Admission decisions are made independently for each class and any differences between classes in credentials are solely a result of the differences in each applicant pool. . . . We do not move applicants admitted to one entering class to another entering class.

. . .

We have been using this system of totally separate entering classes throughout the entire modern history of Baylor Law School, starting in the 1940’s. . . . It is an arduous system for students and faculty and it was adopted for the benefits it confers upon our educational program and the efficiencies of operation it offers -- certainly not for the purpose of statistical manipulation of admissions data.

. . .

Our uncommon system of three separate entering classes has often created confusion with regard to statistical reporting. For many, many years, the fall entering data was reported to the ABA in the annual questionnaire by putting the fall numbers in the questionnaire answer blanks for the admissions section and then providing the same data for the spring and summer classes in the comments section at the end of the admissions section.

In early 2000’s, the ABA notified us that we had to begin reporting the data in an aggregate for all three entering classes. . . .

. . . [P]er ABA’s request, we report to the ABA the aggregate as well as the separate data for each entering class. . . .

Because of our concern for fair and accurate reporting, when asked to report statistical information about our entering classes, we have adopted the approach of answering the specific question asked but also providing the pertinent information for all our entering classes. . . .

. . .

Turning now to the specific issue of our reporting information to U.S. News, we proceeded consistent with our reporting to the ABA. For years, U.S. News asked specifically for information on our Fall entering class and we provided it. Additionally, we sent them the rest of the information on our other entering classes. Back when the survey was completed by hand and faxed to U.S. News, Baylor completed the U.S. News survey by supplying the fall numbers in the answer blanks, as we did with the ABA, but qualifyied [sic] the entry with an asterisk and a note to see an attached addendum, since there was no space for comments as there is on the ABA questionnaire. . . .

. . .

Once the U.S. News converted their survey to one completed and submitted online, we had to change the method of reporting the data for each entering class. Again, with no comment section built into the online survey, we continued to create an addendum; however, it had to be faxed in separately.

. . .

This past spring, we completed the 2006 survey by inputting the aggregate [LSAT and GPA] numbers . . . and submitted it online January 3, 2006. On January 4, 2006 we faxed to U.S. News some changes that we could not make online and we faxed our addendum with a notation that . . . we [had] inserted the aggregate numbers based upon the prior year’s discussion with Mr. Morse. We added, “If you are going to publish the data for the Fall class, as you were still doing last year, please publish the correct numbers below.” [Jackson's emphasis] . . .

Interestingly, when we received our fax copy of our survey from U.S. News on 1/11/2006, someone, working for the U.S. News survey, changed our entry to use our fall numbers. I noted on our copy that they changed our entry from the aggregate numbers to the fall numbers. We assumed that this meant that they were going to continue to publish numbers based on fall entering classes even though they were using the aggregate numbers for their ranking purposes. . . .
. . .

In conclusion, we would like to emphasize several specific points. First, . . . . [w]e fully comply with our responsibility to accurately and thoroughly report the data that we submit to them; however, if they have in fact not used the data correctly pursuant to their proprietary formula, then we are not at fault. Second, all admissions data reported to U.S. News is taken directly from our admissions data reported to the ABA on our annual questionnaire. No changing or manipulating of admission data occurs between the time we report to the ABA and the time we complete the U.S. News survey. Third, we do not know how U.S. News uses the information it receives. Your approximation of their formula indicates what might have happened but we do not know that for sure. . . . Finally, and most importantly, our system of three entering classes is a long-standing practice and is necessary for educational and programmatic purposes, not for admissions statistics purposes.


Absent some new and material information, I offer that as the last word on the Baylor puzzle. I leave readers to draw their own conclusions about whether USN&WR ranked the school based on the wrong median LSAT and GPA, how that happened, and who bears the blame. I care more about how to prevent and cure similar problems. More about that, anon.

Earlier posts about the 2007 USN&WR law school rankings:

6 comments:

Anonymous said...

Dean Jackson writes that "admission decisions [at Baylor] are made independently for each class and any differences between classes in credentials are solely a result of the differences in each applicant pool. . . . [Baylor does] not move applicants admitted to one entering class to another entering class."

While it may be true that an applicant who is admitted to one entering class is not subsequently switched to a different entering term after an admission, Baylor may still switch students to different terms within the formal offer of admission: http://lawschoolnumbers.com/display.php?user=jesska812

This lawschoolnumbers.com profile shows a student who was initially waitlisted after applying for Fall term at Baylor, but subsequently she was admitted to the Spring class: a switch from the applicant pool in which she initially applied.

It may not be an incorrect to say that decisions are made "independently for each class" but that statement might be a bit empty if Baylor is artificially creating the Spring class, in part, with applicants who they would not have admitted for the Fall.

Tom W. Bell said...

A nicely close reading, Anon! I suppose, to follow in that vein, we could affirm the truth of "[Baylor does] not move applicants admitted to one entering class to another entering class," while still supposing that they move applicants to one class who *have not yet been admitted to that class* to different entering class.

All that said, I don't much care why Baylor runs its school as it does so, so long as nobody suffers from material misdescriptions of what it's doing. Further, I think it's great that they offer something different from the typical law school. Would that more law schools experimented with programs that might better suit law students' needs!

Anonymous said...

Did jesska812 ask to be considered for spring admission after she was waitlisted for the fall? If so, I would argue that she -- not Baylor -- is responsible for any "class shifting" that occurred.

Anonymous said...

Jesska812's particular case aside, Baylor does allow application to multiple quarters simultaneously, therefore one need not be rejected entrance from the fall quarter before application to the summer or spring.

Anonymous said...

Baylor's Fall application asks students to check a box for whether or not they would like to be considered for Spring (maybe Summer too) admission if they are not admitted to the Fall.

I know these are technicalities, but the students do request Spring consideration in their Fall application. I do not, however, know for sure that Baylor does not make Spring offers to students who don't check the box to be considered for the Spring.

Personally, I think this issue goes both ways. I think Baylor suffers a bit statistically by taking a significant portion of entrants in the Spring and Summer, when the pool is small and has much lower scores. On the other hand, Baylor benefits from having a smaller Fall class and is able to be more selective than they otherwise would.

Ultimately, I think Baylor will suffer considerably as a result of USNews using the year-round aggregate for rankings. Their full entering class statistics would be much higher if they admitted only in the Fall, when applicants are much stronger (and Baylor's incoming classes are always very small anyway) than they will be admitting students year-round.

Anonymous said...

We appreciate the opportunity to respond to your inquiry about what you incorrectly assumed might be a discrepancy between the information reported by Baylor to the U.S. News and the ABA. As our response letter, with supporting documentation, clearly shows, we do fully and accurately report our LSAT and GPA data to both organizations.

We believe the confusion results from a lack of understanding about the uniqueness of our true quarter system, with three completely separate entering classes (and three separate applicant pools), which has been in place for programmatic and educational reasons since the 1940’s. Operating on a quarter system allows us to more effectively and efficiently prepare our students to face the challenges that await them as practicing lawyers. See our website at http://law.baylor.edu/ProspectiveStudents/more/PS_more_quarterSystem.html for a more complete explanation for why we choose to keep this system.

Any suggestion that our system of three entering classes per year is designed to allow us to adjust our admissions statistics, or that we manipulate the data, is patently false. We gladly will send the complete response (including our submissions to the ABA and U.S. News) to anyone needing more information about this matter. Please contact me at Leah_Jackson@baylor.edu or 254.710.1911.