The Durability of Law School Reputation

Journal of Legal Education

Volumn 48, Issue 4, 1998, Pages 568-590

  Schmalbeck, Richard ^a  

^a DUKE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0032236809     PISSN: 00222208     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Article

References (35)

1. The Reputations of American Professional Schools, Change, Winter 1974-75, at 42. The authors had published a preliminary version a year earlier, America's Leading Professional Schools, Change, Nov. 1973, at 21. Since the second survey was somewhat better designed and enjoyed a higher response rate, it seemed the better point of departure for this article.
2. To say that these overall rankings are controversial is an understatement. Most law deans believe they are so badly done, and so counterproductive, that they cannot be saved and should instead be simply taken out behind the barn and humanely destroyed. See, e.g., Carl C. Monk, The AALS Role in Opposing Rankings of Law Schools, AALS Newsl., Apr. 1997, at 4; AALS Releases Study Criticizing Law School Rankings, AALS Newsl., Apr. 1998, at 7; James P. White, Law School Rankings, Syllabus, Spring 1997, at 2.
3. To say that these overall rankings are controversial is an understatement. Most law deans believe they are so badly done, and so counterproductive, that they cannot be saved and should instead be simply taken out behind the barn and humanely destroyed. See, e.g., Carl C. Monk, The AALS Role in Opposing Rankings of Law Schools, AALS Newsl., Apr. 1997, at 4; AALS Releases Study Criticizing Law School Rankings, AALS Newsl., Apr. 1998, at 7; James P. White, Law School Rankings, Syllabus, Spring 1997, at 2.
4. To say that these overall rankings are controversial is an understatement. Most law deans believe they are so badly done, and so counterproductive, that they cannot be saved and should instead be simply taken out behind the barn and humanely destroyed. See, e.g., Carl C. Monk, The AALS Role in Opposing Rankings of Law Schools, AALS Newsl., Apr. 1997, at 4; AALS Releases Study Criticizing Law School Rankings, AALS Newsl., Apr. 1998, at 7; James P. White, Law School Rankings, Syllabus, Spring 1997, at 2.
5. Except as otherwise noted, I am referring throughout this article to the surveys conducted among academics, which I will call "academic reputation." U.S. News has also, since 1990, annually conducted surveys among practitioners and judges. I must admit to a bias in thinking that the survey among academics is entitled to a greater (albeit still weak) presumption of accuracy. It seems to me reasonably self-evident that legal academics have more information about and insight into their own particular subpart of the profession than others are likely to, just as practitioners would have a comparative advantage in reaching judgments about the quality of the law firms in their regions. Some readers may disagree. On the other hand, everyone would agree that no one person could possibly have enough information about all accredited law schools to make the sort of judgments that a universal survey of their reputations would require to be entitled to any very strong presumption of accuracy.
6. The structure of, and the weight assigned to, the academics' survey has varied a bit over the years. As is described in the appendix (A Note on Methodology), the 1987 U.S. News report on law schools consisted only of a reputational survey conducted among law school deans. From 1990 to 1994 the academics' survey - which by that time was broadened to include faculty besides deans - was assigned a weight of 20 percent in the overall report. Since 1995 that weight has been 25 percent. Note also that, in terms of the influence on the overall ranks, the explicit weights are only part of the story, since the overall ranks will also be affected by the variability within each of the several measures on which overall rank is based. This statistical truth will be obvious to many. Any to whom it is not might contemplate the following example. A professor tells his students that their grades will be determined by weighting their midterm and final examinations equally. But it turns out that all students receive a grade of B on the midterm, while the final examination grades are more normally dispersed. In such a case, the differences on the final examination will obviously determine whatever differences there turn out to be among the students' total grades, regardless of the promised equal weighting of the midterm exam. Comments on this phenomenon, and other criticisms of the U.S. News ethodology, are contained in a report commissioned by the AALS, completed in February 1998, by Stephen P. Klein and Laura Hamilton, and reported at the AALS web site .
7. Throughout this article, I have followed a convention of recording tied ranks as the sum of the tied ranks divided by the number of tied schools at that rank. Thus, if four schools are tied for the seventh through tenth places in a survey, each is shown as having a rank of 8.5, which is 7+8+9+10 / 4.
8. The differences are largely that the 1998 survey produced more tied ranks. Harvard and Yale are tied for first in 1998, while Harvard enjoyed a slight lead in 1974. Similarly, Columbia, Chicago, and Stanford are tied for the next tier in 1998, while they were ranked in the order just named in the 1974 survey. Berkeley and NYU are shown in 1998 as tied for seventh and eighth places, while the 1974 survey had Berkeley slightly ahead. Finally, Pennsylvania and Virginia are now shown as tied for ninth and tenth places, while the 1974 survey gave the ninth and last spot to Pennsylvania alone. The greater number of ties is a predictable result of the U.S. News methodology: many respondents could very plausibly have given the top score, 5 points, to each of the schools listed in Table 2, which represent, after all, the top 10 percent of all American law schools. The 1974 methodology, which asked for a list of only the best five schools, forced harder choices on the respondents, leading to somewhat finer distinctions at the top.
9. Not much rests on this point, so I have not made much of an effort to verify the precise accuracy of these assertions. I did, however, review the current lists of faculty at two top law schools that I know well to inform slightly my guess as to the impact of turnover and expansion of faculties over the last quarter century: the University of Chicago, where I was a student at the time of the Blau-Margulies study, and Duke University. At Chicago, it appears that only 17.2 percent of the 1997-98 tenure-track law faculty were among such in 1974; at Duke, 21.6 percent of the 1997-98 tenure-track law faculty had that status in 1974. During the years in question, Chicago has been served by five deans (Neal, Morris, Casper, Stone, and Baird), while Duke has been served by four (Pye, Dellinger, Carrington, and Gann). Indeed, some perspective on the amount of time that has passed between the two surveys can be gained by noting that Dean Baird, who is about to step down from his deanship, had not even entered law school when the Blau-Margulies study was published.
10. To quantify this, we looked at the coefficients of correlation between the various schools' ranks in the 1974 study and their corresponding ranks in the 1998 surveys. We found that the correlation between the 1974 and 1998 lists for law schools was .904. For schools of medicine it was .676; for schools of engineering .764; for schools of business only .359. There were also surveys of nursing schools reported in both the 1974 and 1998 studies, which appeared to be very poorly correlated. But the 1998 study of nursing schools aggregated the survey of academics with its survey of practitioners, making the results less comparable than in the case of the other fields. We should add that we faced in every field except law the statistical problem raised by the appearance of a school among the top schools in 1998, when it had not appeared there in 1974. This meant that no 1974 rank could be definitively assigned for purposes of computing the correlation coefficients. In those cases, we assigned to that school the highest rank it could have had in 1974. For example, since Penn's medical school was ranked in a tie for 7th in 1998, but did not appear among the top 11 schools listed in the 1974 study, we assigned it a rank of 12 in 1974, implicitly assuming that it was the highest unlisted medical school in that survey. Thus, the correlation coefficients listed for all but law schools above are in fact biased on the high side; the actual correlation coefficient could not have exceeded the numbers shown, but it could have been somewhat lower.
11. The methodologies of the various surveys are discussed in the appendix at the end of this article. Readers who are interested in these details might benefit from referring to those descriptions at this point.
12. There is one possible, though improbable, exception. The 1990 survey shows Georgetown as ranking 16th, with Illinois and North Carolina tied at 18th. No school is shown as ranking 17th. The missing school is probably the University of Wisconsin, whose academic reputation rank was not shown, because it was not in the top 25 that year on an overall basis, and fully detailed rankings were only provided for the top 25 overall. Wisconsin probably ranked exactly 17th in 1990, since in no other survey did it rank higher than a tie for 17th place. But it is possible that it tied Georgetown for 16th place in 1990, since the report did not indicate when ranks shown were shared.
13. The U.S. News methodology, which until 1998 only asked respondents to designate a series of unranked lists of schools by quartile, was poorly designed to discriminate among the best schools, all of which should reasonably have appeared on everyone's ballot. Indeed, in the 1992 and 1994 surveys, all six of these schools were shown as tied for the top score in academic reputation, giving each a score that year of 3.5. (That is, 1+2+3+4+5+6/6.) See further discussion of this point, and the implications of the 1998 changes, in the appendix.
14. For example, only twice did one of the top six schools in the average rank fall out of the top six in any single year: Stanford was ranked at 7.5 (tied with Berkeley, NYU, and Virginia for ranks 6 through 9) in 1997, and Michigan was ranked at 8 (tied with Berkeley, NYU, Penn, and Virginia for ranks 6 through 10) in 1996. And only twice did a school in the Texas-to- Duke cluster fail to fall within the ranks from 11 to 14: Duke ranked 14.5 (tied with Georgetown for 14th and 15th place) in 1997, and Texas ranked 9.5 (tied with NYU, Penn, and Virginia for ranks 8 through 11) in 1993.
15. In the 1987 survey, only the top 20 schools were listed. In the 1990 and 1991 surveys, 25 schools were listed on the basis of their overall rank, but some of the schools in ranks 17 through 24 in terms of academic reputation were not among the 25. Only since 1992 has U.S. News listed the academic reputation results for all law schools, so only those years provide fodder for analysis of schools below the top 16.
16. Hastings was tied with Boston College, Boston University, Emory, George Washington, and the University of Washington for 24th place in both the 1996 and 1998 surveys; in 1997 it was tied with all of the above listed schools except for Emory, which was tied for 29th place in that year.
17. Until 1998 the academic reputation surveys showed only ranks, without reporting any absolute scores on the surveys. In 1998, however, U.S. News did begin reporting an absolute academic reputation score, which permits in at least that year some assessment of the density of the distribution in absolute terms. That density, for that year, can be summarized as follows: 16 schools scored from 4.0 to 4.9; 29 schools scored from 3.0 to 3.9; 93 schools scored from 2.0 to 2.9; and the remaining 36 schools scored from 1.3 (the lowest score) to 1.9. And, in fact, the two schools tied for 7th place (=7.5) had absolute academic reputation scores that were only one-tenth of a point higher than the two that tied for 9th place (=9.5), just as the eight schools tied for 28th place (=31.5) were only one-tenth of a point higher than the two schools that tied for 36th place (=36.5). So the same movement in absolute terms would indeed have produced a greater movement in rank as one moves toward the middle of the distribution, of approximately the magnitude estimated in the text
18. Again, the 1998 U.S. Anus methodology in the academic survey, by which schools are assigned absolute scores rather than just ranks, permits us to note that the density of the law school distribution for the lower three-quarters of that distribution was about three times what it was for the top quarter: 45 schools scored between 3.0 and 4.9, inclusive, while 129 schools scored between 1.3 and 2.9. Thus, a movement of 12 ranks within the lower group generally represents about as much absolute score change as a movement of about 4 ranks in the top quarter would. Of course, I am extrapolating the 1998 data backward to the earlier years, assuming that the same sort of absolute distribution of scores would have prevailed before 1998, an assumption as to which there can be no evidence.
19. Similarly, the lawyer-judge reputational rank of George Washington remained steady in the face of its dramatic drop in overall rank in 1994.
20. Using 3-rank movements in the top 16 and 5-rank movements in the next group of 8 is consistent with our observation about the density of the distribution within those respective ranges. Because there are only 16 schools in the 1998 ranks that achieved academic reputation scores from 4.0 to 4.9, but 29 schools that achieved scores from 3.0 to 3.9, it appears that a movement of 3 ranks in the top group is likely to represent about the same movement in absolute perceived quality as a 5-rank movement in the next range of the distribution would.
21. The annual U.S. News rankings are published in the late winter of each year, which means that students have the (dubious) benefit of those rankings as they receive and weigh their offers of admission. It is also possible that prospective students applying for admission the following fall would use the most recent rankings available in deciding which schools to apply to. This would suggest that some of the possible effect on admissions might show up in the second fall following a major change in ranking. Unfortunately, in many cases, a change in rankings in one year was followed by an offsetting change the following year. For example, Virginia's drop from 8th to 14th place in the overall ranks in 1994 was followed by a 7th place showing in 1995. In such cases, which constituted a significant part of the database, the one- year lagged effect of rank change and the two-year lagged effect of the previous year's rank change went in opposite directions. In a rich database, the one-year and the two-year effects could be teased out; in this database, they could not, so we looked only at the one-year effect.
22. This apparent paradox of schools suffering overall rank declines followed by higher median LSAT ranks the following year may well be explained by simple random variation. But another possibility is that schools that suffered precipitous declines in their rank may have decided to alter their admissions policies the following year in such a way as to address the perceived problem of their declining U.S. News overall rank. A wise admissions policy includes review of the life experiences of applicants, their written personal statements, the rigor of their undergraduate program and course selection, and the quality and grading policies of the undergraduate institution. But, to the extent that these factors lead - as they inevitably will - to the admission of worthwhile candidates with lower GPAs and LSAT scores than those who would be admitted by a simple formula combining those two numbers, those policies will also lead to a lower score on the U.S. News selectivity index. This is perhaps the single most pernicious effect of the U.S. News rankings, and it is an effect that might present a special temptation to schools whose administrations were particularly worried about slippage in rank.
23. In particular, we noticed in going through the data a number of schools whose academic reputation scores were consistently much higher than their overall U.S. News ranks. To mention a few outstanding examples, Arizona State, Chicago-Kent, and Southern Methodist have frequently had overall ranks in the undifferentiated third tier of law schools at or a bit below the middle of the distribution, even though their academic reputation ranks have generally been in the 40s, which would place them at about the boundary between the first and second quartiles. They may derive some reassurance from their continued high rank among academics; it is clear that, in this area as in all the others examined in this article, reputation seems to be impervious to influence by low overall ratings. But at the same time it seems likely that the inappropriately low ranks suffered by these schools must be a constant source of irritation to them as they try to make their cases to their alumni, central campus administrations, prospective students, and the like.
24. It is reasonably clear that the variability from year to year is greater in the overall U.S. News rankings than in the academic reputation component of those rankings. (A table detailing these variations is available from the author on request.) We have not tried to prove this systematically, though we did make a quick comparison between the 1990 and 1998 lists of the top 16 schools (chosen because 1990 was the first U.S. News effort to go beyond a mere academic reputation survey, while 1998 is of course the most recent) and found that the correlation between the academic reputation ranks in the two years was an amazing .936. The correlation between the overall rankings in the two years was also fairly high, at .815, but much less so than in the case of academic reputation.
25. The number of schools included in the U.S. News rankings has varied slightly from year to year. The 1998 report describes the effort as providing "rankings for 174 of the nation's accredited law schools," without elaboration on the criteria by which they determined which schools were accredited. Indeed, the absence of a definite article ("the 174 accredited law schools") suggests that they are not purporting to provide a comprehensive list of accredited law schools. For purposes of this article, the precise definition of an accredited law school is not important, and I do not mean to adopt one.
26. Entire books could be written - and some have been - on the best way to characterize the impact of law and economics and/or critical legal studies on the legal academy. See, e.g., Arthur Austin, The Empire Strikes Back: Outsiders and the Struggle over Legal Education (New York, 1998), especially the discussion of Kuhnian paradigm shifts at 164-66. I don't really mean to express a viewpoint on most of the controversies involved in those characterizations. My point in this paragraph depends on only two fairly obvious notions: that neither movement has been wholly successful in overthrowing traditional legal analysis, and that it is nevertheless nearly impossible to read any issue of any major law review without hearing the echoes of both movements.
27. I would strongly suggest, if this were to be undertaken, that respondents be allowed to show as many ties for ranks as they think they see in the phenomenon they are reporting. A typical ballot might show: Harvard/Yale; Stanford/Chicago; Columbia/Michigan/Berkeley; etc. My sense is that at least some part of the resistance of law deans to ranking is the perceived necessity to differentiate between schools they view as equivalent
28. It might make sense as well to rank schools below the top 40 or so on a regional basis, with only the deans in the region casting ballots with respect to those schools. This would respond to the common complaint among reputation survey respondents that they have very little information about most law schools that are geographically distant from their own, unless those law schools have a broad national reputation.
29. I have already made this suggestion more or less formally in a letter to the executive director of the AALS. His response was polite but unencouraging. At this point, a major obstacle to AALS participation in the proposed venture is that the AALS has invested a great deal in promoting the idea that ranking law schools is an irredeemably foolish enterprise. Even if a majority of deans could be persuaded that the proposed approach would be less foolish and less destructive than the approach of U.S. News, there would remain a sense that internal, wholly volitional efforts in pursuit of a foolish goal should not be countenanced.
30. This example is counterfactual in a number of respects, among them the anachronistic treatment of the critical legal studies movement, which did not reach its apogee for at least a decade after the Blau and Margulies study.
31. Let me say that I doubt, based on all the surveys I have examined, that this is in fact an accurate depiction of what happened in the 1974 study; I simply mean to say that the possibility cannot be conclusively denied on the basis of the methods that Blau and Margulies used in that study.
32. The others included have varied somewhat from year to year, as has the detail with which U.S. News describes its methods. But, typically, the senior academic associate dean or the equivalent was included in the survey, as was the chair of the faculty appointments committee. In some years, the most recently tenured faculty member also received a survey.
33. This was the implication of the instructions to the survey; but no adjustment was presumably made for respondents who, for example, listed only 25 schools in their top quartile.
34. Any reader who thinks this method is acceptable is invited to imagine what might happen to the readership of the weekly Associated Press polls of college football and basketball teams if the voters on those surveys were asked simply to divide the Division I programs into quartiles, with AP then ranking the schools according to the percentage of ballots in which they appeared in the lop quartile.
35. Indeed, this has been a major problem with the overall rankings produced by U.S. News: it has tinkered each year with the basic methodology, and with the weights assigned to different elements of the studies, to a degree that makes time-series comparisons very treacherous.