Judicial evaluations and information forcing: Ranking state high courts and their judges

Duke Law Journal

Volumn 58, Issue 7, 2009, Pages 1313-1381

  Choi, Stephen J ^a   Gulati, Mitu ^b   Posner, Eric A ^c  

^a NEW YORK UNIVERSITY SCHOOL OF LAW   (United States)

^b DUKE UNIVERSITY   (United States)

^c UNIVERSITY OF CHICAGO LAW SCHOOL   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 68049095997     PISSN: 00127086     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Article

References (74)

1. Inst. for Legal Reform, Lawsuit Climate 2008, http://www.instituteforlegalreform.com/index.php?option=com_ilr_harris_Poll&year=2008 (last visited Feb. 2, 2009).
2. One example is the advertisement on the Delaware state courts website. See First State Judiciary, Superior Court in the News!, http://courts.delaware.gov/Courts/SuperiorCourt/AboutUs/?press_99_05.htm (last visited Feb. 2, 2009) ("The Judicial Branch of Delaware government is extremely pleased and gratified that our Courts rank number one in the nation in the quality of our litigation system." (quoting Chief Justice Norm Veasey of the Delaware Supreme Court).
3. see also official Site of the Governor of Virginia, http://www.governor.virginia.gov/MediaRelations/newsReleases/viewRelease.cfm?id=213 (last visited Feb. 2, 2009) (talking about Virginia's high ranking on the Chamber of Commerce survey).
4. For discussions of the rankings and the need for reforms to improve rankings, see, for example, Chamber Fights to Improve Legal Climate, Bus. Advoc. (Kan.), Apr. 21, 2005, http://www.kansaschamber.org/forms/advo3/V3Num14.htm (noting Kansas's drop from fourth to sixteenth).
5. On the use of rankings to argue for salary increases, see generally Letter from Robert D. Ray, Iowa Judicial Comp. Task Force, to Nicholas Critelli, President, Iowa State Bar Ass'n (Apr. 18, 2005), http://www.iowabar.org/miscdocuments.nsf/2b85a4ea12f4bfac8625669d006e27ab/f0fb92e322a9987d86256ff20049a0bb/$FILE/Judicial%20compensation.pdf.
6. See, e.g., Am. Bar Ass'n, Ratings of Article III Judicial Nominees: 110th Congress (2008), available at http://www.abanet.org/scfedjud/ratings/ratings110.pdf.
7. See The Robe Probe, http://robeprobe.com (last visited Feb. 6, 2009).
8. The Robing Room, http://www.therobingroom.com (last visited Feb. 6, 2009).
9. See, e.g., William M. Landes, Lawrence Lessig & Michael E. Solimine, Judicial Influence: A Citation Analysis of Federal Courts of Appeal Judges, 27 J. Legal Stud. 271, 271 (1998).
10. Stephen J. Choi, G. Mitu Gulati & Eric A. Posner, Professionals or Politicians: The Uncertain Empirical Case for an Elected Rather than Appointed Judiciary, J.L. Econ. & Org. (forthcoming 2009) (manuscript at 3), available at http://ssrn.com/abstract_id=1008989.
11. Stephen J. Choi, G. Mitu Gulati & Eric A. Posner, Are Judges Overpaid?: A Skeptical Response to the Judicial Salary Debate, 1 J. Legal Analysis 47, 50 (2009), https://ojs.hup.harvard.edu/index.php/jla/article/view/3/18.
12. See Steven Kerr, On the Folly of Rewarding A, While Hoping for B, 18 Acad. Mgmt. J. 769, 778 (1975).
13. Wendy Nelson Espeland & Michael Sauder, Rankings and Reactivity: How Public Measures Recreate Social Worlds, 113 Amer. J. Soc. 1, 1 (2007).
14. E.g., William P. Marshall, Be Careful What You Wish for: The Problems with Using Empirical Rankings to Select Supreme Court Justices, 78 S. Cal. L. Rev. 119, 122-29 (2004).
15. For further discussion of methodological issues involved in ranking judges, see generally Stephen J. Choi & G. Mitu Gulati, Choosing the Next Supreme Court Justice: An Empirical Ranking of Judge Performance, 78 S. Cal. L. Rev. 23 (2004).
16. See Choi et al., supra note 6 (manuscript at 26). Controlling for these institutional differences does not have a meaningful effect on our regression results in this Article.
17. See Alex Kozinski & Stephen Reinhardt, Please Don't Cite This!: Why We Don't Allow Citation to Unpublished Dispositions, Cal. Law., June 2000, at 43-44, 81.
18. See, e.g., Landes et al., supra note 5, at 271.
19. Richard A. Posner, How Judges Think 149 (2008).
20. See Daniel A. Farber, Supreme Court Selection and Measures of Past Judicial Performance, 32 Fla. St. U. L. Rev. 1175, 1183 (2005).
21. For a description of our independence measure, see Choi et al., supra note 6 (manuscript at 17). In another paper, Choi and Gulati treat a 0 independence score as highest on the theory that zero independence means that party affiliation makes no difference to case outcomes.
22. See Choi & Gulati, supra note 10, at 66. For purposes of this Article, we treat a judge who votes against partisan affiliation as likely to be more independent, as it shows that she feels strongly about the outcome. It is possible that the judge switched ideologies while sitting, but prior scholarship indicates that this is unusual.
23. See Jeffrey A. Segal & Harold J. Spaeth, The Supreme Court and the Attitudinal Model Revisited 180 (2002).
24. In many countries, the concern about judges taking bribes is a real one. And in those contexts, the relevant measure of independence should include some measure of judicial corruption. But in the United States, where high court judges have the option of earning higher sums in private practice, this concern may be misplaced. It is unlikely that these judges would engage in corrupt Behavior and risk criminal sanctions when they could instead simply move into the private sector.
25. E.g., Joanna M. Shepherd, The Influence of Retention Politics on Judges' Voting, 38 J. Legal Stud. (forthcoming 2009) (manuscript at 14), available at http://ssrn.com/abstract=997491.
26. Related are reports of so-called judicial hellholes put out by organizations like the American Tort Reform AsSociation (that the U.S. Chamber of Commerce applauded). See Press Release, U.S. Chamber of Commerce, Report Bolsters Harris Poll Finding on Abusive Legal Climates (Dec. 15, 2004), http://www.uschamber.com/press/releases/2004/december/04-163.htm.
27. Inst. for Legal Reform, State Resources Center: Executive Summary, http://www.instituteforlegalreform.com/index.php?option=com_ilr_harris_Poll&id=1&view=lawsuit_climate&Itemid=60 (last visited Feb. 11, 2009).
28. Inst. for Legal Reform, State Resources Center: Methodology, http://www.instituteforlegalreform.com/index.php?option=com_ilr_harris_Poll&id=7&view=lawsuit_climate&Itemid=60 (last visited Feb. 11, 2009).
29. See infra Part II.C.5.
30. See supra note 2.
31. See Press Release, Tom Donohue, President & CEO, U.S. Chamber of Commerce, Release of the 2006 Harris Poll State Liability Rankings (Mar. 27, 2006), http://www.uschamber.com/press/speeches/2006/060327_ilr_rankings_remarks.htm ("But there's still quite a ways to go before we can rid our courts of lawsuit abuse and correct the deep flaws in our legal system. One of the key weapons in our arsenal is the annual State Liability Systems Ranking Study. Since the inception of the study, it has become the benchmark against which businesses, elected officials, the media and other opinion leaders measure their state's legal climate. They want to see how they stack up against other states, and also how well-or poorly-the system is serving employers, workers and consumers.").
32. See ABA Div. for Bar Servs., Bar AsSociations' Response to Chamber of Commerce Ad Campaign, http://www.abanet.org/barserv/tortreform.html (last visited Feb. 2, 2009) (reporting on advertisements run in both national newspapers like the New York Times, the Washington Post, and the Wall Street Journal and in local newspapers in Illinois, West Virginia, California, and Mississippi).
33. See Arthur B. Laffer & Stephen Moore, Rich States, Poor States 40 (2007) (using the 2002 Chamber of Commerce rankings as one of sixteen factors in their state competitiveness rankings).
34. E.g., Russell S. Sobel & Joshua C. Hall, The Effect of Judicial Selection Processes on Judicial Quality: The Role of Partisan Politics, 27 Cato J. 69, 71 (2007).
35. Russell S. Sobel, Matt E. Ryan & Joshua C. Hall, Electoral Pressures and the Legal System: Friends or Foes?, in law Without Romance: Public Choice and Legal Institutions (E. Lopez ed.) (forthcoming) (manuscript at 5), available at http://joshua.c.hall.googlepages.com/electoralpressures.pdf.
36. Daniel Berkowitz & Karen Clay, Initial Conditions, Institutional Dynamics and Economic Performance: Evidence from the American States 7-8 (William Davidson Inst., Working Paper No. 615, 2004), available at http://ssrn.com/abstract=485003 (using the 2002 Chamber of Commerce rankings to measure court quality in the different states).
37. see also Michael J. Hicks, Reduce the Cost of Civil Litigation and DePoliticize the Courts, in Unleashing Capitalism 185, 189 (Russell S. Sobel ed., 2007) (referring to a study by Professors Sobel and Hall that uses the Chamber of Commerce rankings). Neither of the latter two studies mentions that the Chamber and Commerce data reflect the views of only lawyers at corporations with annual Revenues of at least $100 million.
38. See Press Release, Pub. Citizen, New U.S. Chamber of Commerce Poll Ranking States' Liability Systems Is Part of a Disinformation Campaign to Restrict Consumer Rights (Mar. 9, 2005), http://www.commondreams.org/news2005/0309-10.htm.
39. see also Pub. Citizen, Class Action "Judicial Hellholes": Empirical Evidence is Lacking 2 (2005), available at http://www.citizen.org/documents/OutlierReport.pdf (featuring a complaint regarding the judicial hellholes reports).
40. Letter from Dennis W. Archer, President, Am. Bar Ass'n, to Thomas J. Donohue, U.S. Chamber of Commerce 1 (Mar. 11, 2004), http://www.abanet.org/media/statementsletters/chamberopenletter.pdf (accusing the Chamber of Commerce of mounting a campaign against members of the legal system to avoid taking responsibility for the nation's financial problems).
41. Lawrence M. Friedman et al., State Supreme Courts: A Century of Style and Citation, 33 Stan. L. Rev. 773, 774 (1981).
42. Rodney L. Mott, Judicial Influence, 30 Am. Pol. Sci. Rev. 295, 295-96 (1936).
43. Gregory A. Caldeira, On the Reputation of State Supreme Courts, 5 Pol. Behav. 83, 87-93 (1983).
44. Scott A. Comparato, On the Reputation of State Supreme Courts Revisited (Apr. 2002) (unpublished manuscript, on file with the Duke Law Journal).
45. Jake Dear & Edward W. Jessen, "Followed Rates" and Leading State Cases, 1940-2005, 41 U.C. Davis L. Rev. 683, 690-93 (2007).
46. For 1940-2005, the top ten states, in order, are California, Washington, Colorado, Iowa, Minnesota, Kansas, Massachusetts, Wisconsin, Oregon, and New York. Id. at 694.
47. For the latest rankings and asSociate advertising, see Inst. for Legal Reform, http://www.instituteforlegalreform.com (last visited Feb. 2, 2009).
48. Less directly on point are a handful of other studies that could also be read to contain rankings of the states but that we do not discuss because they are tangential to our inquiry. For example, a 1981 study by Professors Canon and Baum compared the states' innovativeness in terms of being willing to adopt a set of twenty-three plaintiff-friendly tort law doctrines. Bradley C. Canon & Lawrence Baum, Patterns of Adoption of Tort Law Innovations: An Application of Diffusion Theory to Judicial Doctrines, 75 Am. Pol. Sci. Rev. 975, 975 (1981).
49. Also, a number of other studies examine the citation patterns of individual state courts. Professor Merryman, for example, in two studies twenty years apart, looked at the citation practices of the California Supreme Court, which could be read to be California's ranking of the rest of the state high courts. John Henry Merryman, The Authority of Authority: What the California Supreme Court Cited in 1950, 6 Stan. L. Rev. 613, 613-14 (1954).
50. John Henry Merryman, Toward a Theory of Citations: An Empirical Study of the Citation Practice of the California Supreme Court in 1950, 1960, and 1970, 50 S. Cal. L. Rev. 381, 381 (1977).
51. Nat'l Ctr. for State Courts, Survey of Judicial Salaries 1 (2007), available at http://www.ncsconline.org/WC/Publications/KIS_JudComJudSal070107Pub.pdf.
52. See Choi et al., supra note 6 (manuscript at 41).
53. A ready objection to using the number of published opinions is that some states have norms of producing and publishing short opinions and others use longer and more detailed opinions. If one assumes that the shorter opinions involve less effort (a questionable, but plausible assumption), then the better measure of effort might be the number of published pages. Alternatively, one could look at the number of Westlaw KeyCites, which would provide a sense of the number of issues that opinion tackled (shorter and more routine opinions would have fewer Westlaw KeyCites). Unreported here, we calculated state rankings on each of these measures as well. The rankings do change. On the Westlaw KeyCite measure, for example, the top five states are South Carolina, Montana, Pennsylvania, New Jersey and Georgia. On the number of pages measure, the top states are Montana, Pennsylvania, California, Mississippi and Maryland.
54. We define "Common and Commercial Law Cases" to include cases in the following subject matter areas: Contracts; Insurance; Private arbitration; Creditor v. Debtor; Lessor-Lessee; Usury Laws; Franchise v. Franchisor; Employment Contractual Disputes; Corporate Law; Piercing the Corporate Veil; Tax; Bankruptcy; Enforcement of mechanics lien; Implied warrant of merchantability; Takings claims; Zoning issues; Property rights; Property Licensing-Related or Permit-Related; Landlord-Tenant-Related; Federal Tort Related Act; Medical Malpractice; Products Liability; Wrongful Death; Libel; and other tort cases.
55. Appendix A also provides a ranking of judges based on common and commercial law productivity. Due to the relatively small number of opposing opinions available to calculate the independence ranking, we do not compute a separate common and commercial law independence ranking.
56. Inst. for Legal Reform, ILR Advertisement, http://www.instituteforlegalreform.com/images/stories/images/ads/files/wrongwayri.mp3 (last visited Feb. 2, 2009).
57. See David B. Offer, Why So Much Political Corruption in Rhode Island? But Not in Maine?, Morning Sentinel (Waterville, Me.), Mar. 13, 2007, at 5A, available at http://morningsentinel.mainetoday.com/view/columns/3708052.html.
58. After a wave of scandals, the judicial appointments process was reformed from having the legislature choose supreme court justices to having the governor choose justices from a list of names provided by a nonpartisan commission (subject to legislative approval). See Barton P. Jenks, III, Rhode Island's New Judicial Merit Selection Law, 1 Roger Williams U. L. Rev. 63, 66-67 (1996).
59. Michael J. Yelnosky, Rhode Island's Judicial Nominating Commission: Can "Reform" Become Reality?, 1 Roger Williams U. L. Rev. 87, 88 (1996).
60. See U.S. Chamber of Commerce, States Liability Systems Ranking Study, Final Report 25 (2002). We also calculate the Spearman Rank Correlation Coefficient = -0.1576 (p-value of two-sided test of null hypothesis that the two are independent = 0.2745).
61. Weighting choices matter more for the independence measure, which is uncorrelated with influence and productivity; influence and productivity have a correlation coefficient of 0.3.
62. Cf. Choi et al., supra note 6 (manuscript at 41).
63. The data are from William D. Berry et al., Measuring Citizen and Government Ideology in the American States, 1960-93, 42 Am. J. Pol. Sci. 327, 330-31 (1998). Updated data are available at State Citizen and Government Ideology, http://www.uky.edu/%7Erford/Home_files/page0005.htm (last visited Mar. 26, 2009).
64. For related criticisms of U.S. Chamber of Commerce studies, see Elizabeth G. Thornburg, Judicial Hellholes, Lawsuit Climates and Bad Social Science: Lessons from West Virginia, 110 W. Va. L. Rev. 1097, 1100-07 (2008).
65. See, e.g., Nina Totenberg, Report: Spending on Judicial Elections Soaring, Nat'l Pub. Radio, May 18, 2008, http://www.npr.org/templates/story/story.php?storyId=10253213.
66. Democratic legislatures do worse compared with the base category of split legislatures in Models 1 and 3. The coefficient on Legis. Democrat loses significance, however, once state court-level controls are added. In addition, the difference between Legis. Democrat and Legis. Republican is significant at the 10 percent level for Model 1 (although the difference is insignificant in the other models).
67. A study by Professors Russell Sobel and Joshua Hall, Sobel & Hall, supra note 27, at 75, runs a similar regression but finds that states with electoral systems have the lowest Chamber of Commerce ratings. We suspect that our inclusion of a number of control variables that are correlated with the type of selection system explains the difference (they also used Chamber of Commerce ratings from 2004, whereas we used 2002 ratings). They also found that Republicancontrolled state supreme courts during their period had a higher judicial quality rating than Democratic-controlled state supreme courts. Id. at 77.
68. We also checked whether the partisan composition of the bench affected Chamber of Commerce scores. A variable equal to the fraction of Republican judges is not significantly different from zero for all four of the models in Table 13: Chamber of Commerce Models.
69. E.g., Landes et al., supra note 5, at 271.
70. See Paul Brace, Laura Langer & Melinda Gann Hall, Measuring the Preferences of State Supreme Court Judges, 62 J. Pol. 387, 393-98 (2000).
71. Putting aside the findings with respect to gender effects, our results overall are similar to those of the Landes et al. study on federal circuit judges, although they have some different variables in their regressions. They find that judges who graduate from Harvard and Yale Law Schools are cited more often than other judges, but these results are weak and do not hold for top-twenty law schools, and that race, sex, measures of academic achievement, prior experience, ideology, and ABA ratings are mostly insignificant. Landes et al., supra note 5, at 324-25.