Pitfalls of empirical studies that attempt to understand the factors affecting appellate decisionmaking

Duke Law Journal

Volumn 58, Issue 8, 2009, Pages 1895-1989

  Edwards, Harry T ^a   Livermore, Michael A ^a  

^a NEW YORK UNIVERSITY SCHOOL OF LAW   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 68049122884     PISSN: 00127086     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Article

References (270)

1. Allegheny Gen. Hosp. v. NLRB, 608 F.2d 965, 969-70 (3d Cir. 1979).
2. Benjamin N. Cardozo, The Nature of The Judicial Process 20 (Yale Univ. Press 1964) (1921).
3. I first discussed these ideas in The Role of a Judge in Modern Society: Some Reflections on Current Practice in Federal Appellate Adjudication. Harry T. Edwards, The Role of a Judge in Modern Society: Some Reflections on Current Practice in Federal Appellate Adjudication, 32 Clev. St. L. Rev. 385 (1983-84). My general assessment of the breakdown of "easy," "hard," and "very hard" cases heard by court of appeals judges has remained the same over my twenty-nine years on the bench.
4. See Harry T. Edwards, The Effects of Collegiality on Judicial Decision Making, 151 U. Pa. L. Rev. 1639, 1685 (2003) [hereinafter Edwards, Collegiality]
5. Harry T. Edwards, The Judicial Function and the Elusive Goal of Principled Decisionmaking, 1991 Wis. L. Rev. 837, 854.
6. See infra note 69.
7. Frank B. Cross, Decision Making in the U.S. Courts of Appeals 202 (2007).
8. Cardozo, supra note 2, at 20
9. In this Article, we use a restricted view of "empirical legal analysis." For our purposes, we are referring to the body of literature that applies social science techniques of data analysis to legal opinions. Lee Epstein and Gary King, in a wide-ranging critique of empirical work in the legal academy, provide a much more expansive definition. See Lee Epstein & Gary King, The Rules of Inference, 69 U. Chi. L. Rev. 1, 2-3 (2002). Epstein and King invoke a broader understanding of "[t]he word 'empirical' [to] denote[] evidence about the world based on observation or experience. That evidence can be numerical (quantitative) or nonnumerical (qualitative). ... What makes research empirical is that it is based on observations of the world-in other words, data, which is just a term for facts about the world." Id. at 2-3.
10. See, e.g., Frank B. Cross & Emerson H. Tiller, Judicial Partisanship and Obedience to Legal Doctrine: Whistleblowing on the Federal Courts of Appeals, 107 Yale L.J. 2155, 2157 (1998)
11. Harry T. Edwards, Collegiality and Decision Making on the D.C. Circuit, 84 Va. L. Rev. 1335, 1336 (1998)
12. Edwards, Collegiality, supra note 3, at 1641.
13. Epstein & King, supra note 7, at 2.
14. William S. Jordon, III, Judges, Ideology, and Policy in the Administrative State: Lessons from a Decade of Hard Look Remands of EPA Rules, 53 Admin. L. Rev. 45, 52 (2001)
15. Richard L. Revesz, A Defense of Empirical Legal Scholarship, 69 U. Chi. L. Rev. 169, 188 (2002) [hereinafter Revesz, A Defense of Empirical Legal Scholarship].
16. Environmental Regulation, Ideology, and the D.C. Circuit, 83 Va. L. Rev. 1717, 1771 (1997) [hereinafter Revesz, Environmental Regulation].
17. Richard L. Revesz, Ideology, Collegiality, and the D.C. Circuit: A Reply to Chief Judge Harry T. Edwards, 85 Va. L. Rev. 805, 851 (1999)
18. Gregory C. Sisk & Michael Heise, Judges and Ideology: Public and Academic Debates About Statistical Measures, 99 Nw. U. L. Rev. 743, 746 (2005)
19. Patricia M. Wald, A Response to Tiller and Cross, 99 Colum. L. Rev. 235, 239 (1999)
20. Gregory C. Sisk, The Quantitative Moment and the Qualitative Opportunity: Legal Studies of Judicial Decision Making, 93 Cornell L. Rev. 873, 877 (2008) (book review)
21. Dan M. Kahan, "Ideology in" vs. "Cultural Cognition of" Law: What Difference Does it Make? 1 (Harvard Law Sch. Program on Risk Regulation, Research Paper No. 08-22, 2008), available at http://ssrn.com/abstract=1111865. For a discussion of cultural cognition, see infra Part V.
22. See, e.g., CROSS, supra note 5, at 201-27.
23. See, e.g., Sisk, supra note 8, at 876.
24. Id. at 884.
25. Scholars in the field typically refer to variables relating to the appointment process of judges, like the party of appointing president, as "ideology" or "ideological variables"-a convention we reject. We instead refer to party of appointing president and other such measures as "appointment variables."
26. Sisk, supra note 8, at 886 n.72.
27. Cross, supra note 5, at 4.
28. Epstein & King, supra note 7, at 6-7. But see Revesz, A Defense of Empirical Legal Scholarship, supra note 8, at 188-89 ("[E]mpirical legal scholarship has a great deal to contribute to the understanding of law and legal institutions, and social scientists would benefit from paying close attention to the methodological innovations performed by legal scholars. Because of their flawed methodology and unwarranted criticisms and exaggerations, Epstein and King have missed an important opportunity to examine what legal and social science empirical scholarship can learn from one another.").
29. For a description of the attitudinal model, see, for example, Jeffrey A. Segal & Harold J. Spaeth, The Supreme Court and the Attitudinal Model Revisited 86-97 (2002).
30. Lee Epstein, Nancy Staudt & Peter Wiedenbeck, Judging Statutes: Thoughts on Statutory Interpretation and Notes for a Project on the Internal Revenue Code, 13 Wash. U. J.L. & Pol'y 305, 320 (2003).
31. Carolyn Shapiro, Coding Complexity: Bringing Law to the Empirical Analysis of the Supreme Court, 60 Hastings L.J. 477, 486-87 (2009). Professor Shapiro offers the following example of a "mind-boggling" analysis: [D]escribing the "apparently unidimensional nature of Supreme Court decisionmaking," two political scientists (including [Harold] Spaeth's frequent co-author Jeffrey Segal) assert in a 2005 article: "The vote on the merits in any given case is as straightforward as a majority rule process gets. Justices essentially make a binary, reverse or affirm decision." Id. at 486 n.41 (quoting Jeffrey A. Segal & Chad Westerland, The Supreme Court, Congress, and Judicial Review, 83 N.C.L. Rev. 1323, 1324 (2005)).
32. Id. at 486-87 (footnotes omitted) (quoting Parents Involved in Cmty. Sch. v. Seattle Sch. Dist. No. 1, 127 S. Ct. 2738 (2007) (Kennedy, J., concurring in part and concurring in the judgment)).
33. For example, a major problem in many studies premised on the attitudinal model is that overly simplistic proxies for "ideology," such as the party of the president who appointed a judge, are often used. Scholars in this area have attempted to develop more sophisticated proxies. For example, Cross uses Giles scores to "assign[] relative levels of ideological preferences to particular presidents and to the judges that those presidents appointed." CROSS, supra note 5, at 19. However, the use of Giles scores results in only a small, marginal improvement over the party-of-the-appointing-president proxy. And naturally, this change does not address the issue of the binary nature of outcomes in his or other studies.
34. Id. at 20.
35. Charles Grove Haines, General Observations on the Effects of Personal, Political, and Economic Influences in the Decisions of Judges, 17 Ill. L. Rev. 96, 114 (1922).
36. Id. at 105 (citing George Everson, The Human Element in Justice, 10 J. Crim. L. & Criminology 90, 98 (1919)).
37. Nancy Maveety, The Study of Judicial Behavior and the Discipline of Political Science, in The Pioneers Of Judicial Behavior 1, 8 (Nancy Maveety ed., 2005).
38. See David W. Rohde & Harold J. Spaeth, Supreme Court Decision Making 134-57 (1976)
39. Jeffrey A. Segal & Harold J. Spaeth, the Supreme Court and the Attitudinal model 65 (1993)
40. Segal & Spaeth, supra note 17, at 86; Harold J. Spaeth, The Attitudinal Model, in Contemplating Courts 296, 296 (Lee Epstein ed., 1995)
41. Frank B. Cross, Political Science and the New Legal Realism: A Case of Unfortunate Interdisciplinary Ignorance, 92 Nw. U. L. Rev. 251, 265-79 (1997).
42. Glendon Schubert, The Judicial Mind: The Attitudes and Ideologies of Supreme Court Justices, 1946-1963, at 37 (1965)
43. Id. at 42.
44. Id. at 37.
45. Maveety, supra note 27, at 25-26.
46. Id. at 22.
47. See Cornell W. Clayton, The Supreme Court and Political Jurisprudence: New and Old Institutionalisms, in Supreme Court Decision-Making: New Institutionalist Approaches 15, 27 (Cornell W. Clayton & Howard Gillman eds., 1999) (discussing how attitudinalists simply assume that judges "vote their policy preferences and use legal principles to mask their true motives").
48. Haines referred to legal realists' ideas as a conception of "free legal decision." Haines, supra note 23, at 96-102.
49. Cross, supra note 5, at 12, 14.
50. Haines's research agenda was deeply influenced by his views concerning how judges make decisions. Haines contrasted two theories of decisionmaking, which he termed the "mechanical theory" and the "theory of free legal decision." Haines, supra note 23, at 96-102. The mechanical theory-which today we might refer to as legal formalism-"postulates absolute legal principles, existing prior to and independent of all judicial decisions, and merely discovered and applied by courts." Id. at 97. The theory of free legal decision, which Haines embraced, postulated "that judicial decisions are affected by the judge's views of public policy and by the personality of the particular judge rendering the decision." Id. at 102. Haines's theory of free legal decision was deeply influenced by legal realists of his time-Justice Holmes and other legal realists are quoted extensively by Haines to support his views of how judges make decisions.
51. Id. at 105.
52. Schubert, supra note 29.
53. Maveety, supra note 27, at 14.
54. Anthony D. Castberg, Book Review, 19 W. Pol. Q. 583, 585 (1966) (reviewing Schubert, supra note 29)
55. see also Joseph Tanenhaus, The Cumulative Scaling of Judicial Decisions, 79 Harv. L. Rev. 1583, 1591 (1966) ("[T]he application of Schubert's rules can lead to odd results.").
56. Frederick Schauer & Virginia J. Wise, Legal Positivism as Legal Information, 82 Cornell L. Rev. 1080, 1096-97 (1997).
57. Richard A. Posner, Pragmatic Adjudication, 18 Cardozo L. Rev. 1, 20 (1996). Judge Posner rejects the attitudinal model as an overly simplistic model of judicial behavior.
58. Richard A. Posner, The Problematics Of Moral And Legal Theory 242 (1999).
59. Brian Z. Tamanaha, The Realism of Judges Past and Present, 56 Clev. St. L. Rev. (forthcoming 2009) (manuscript at 13-14), available at http://ssrn.com/abstract=1024747.
60. Edwards, Collegiality, supra note 3, at 1644.
61. see also Alex Kozinski, Circuit Judge, U.S. Court of Appeals for the Ninth Circuit, What I Ate for Breakfast and Other Mysteries of Judicial Decision Making, Address at the Symposium on the California Judiciary, Loyola Law School, Los Angeles (Mar. 19, 1993), in 26 LOY. L.A. L. REV. 993, 994 (1993) ("The larger reality... is that judges exercise their powers subject to very significant constraints.")
62. Wald, supra note 8, at 237 ("At this point I must reveal [the D.C. Circuit's] 'dirty little secret': A large portion of our cases (particularly administrative law cases) have no apparent ideology to support or reject at all-the judges are tasked simply with plowing through volumes of complex data and reams of statistical evidence to see if the agency has substantial evidence to back its findings or has acted in an arbitrary and capricious way.").
63. See Kozinski, supra note 47, at 995; Sisk & Heise, supra note 8, at 793-94.
64. Howard Gillman & Cornell W. Clayton, Beyond Judicial Attitudes: Institutional Approaches to Supreme Court Decision-Making, in Supreme Court Decision-Making, supra note 35, at 1, 4-5.
65. See Edwards, Collegiality, supra note 3, at 1656-57.
66. Deanell Reece Tacha, The "C" Word: On Collegiality, 56 Ohio St. L.J. 585, 586 (1995) ("I urge that we go beyond the matrix of computerized decisionmaking to consider the qualitative aspects of judicial interaction. .. ."); Wald, supra note 8, at 255 (noting that the "formal labeling of judges" by political party "is the antithesis of collegial decisionmaking").
67. See Edwards, Collegiality, supra note 3, at 1648-52. For a compilation of additional sources discussing the influence of collegiality on decisionmaking, see id. at 1641 n.10.
68. Edwards, Collegiality, supra note 3, at 1661; see also Richard L. Revesz, Congressional Influence on Judicial Behavior? An Empirical Examination of Challenges to Agency Action in the D.C. Circuit, 76 N.Y.U. L. Rev. 1100, 1112 (2001) (stating that "panel effects" can be explained by either a "deliberation hypothesis," pursuant to which judges modify their views because they take seriously the views of their colleagues, or a "dissent hypothesis," under which a judge who sits with two colleagues from a different political party moderates his or her views in order to avoid having to write a dissent).
69. Alternative approaches have been proposed, but have not yet been widely adopted by the empirical legal community. See, e.g., Joshua B. Fischman & David S. Law, What Is Judicial Ideology, and How Should We Measure It?, 29 Wash. U. J.L. & Pol'y (forthcoming 2009) (manuscript at 3-4), available at http://works.bepress.com/david_law/14 (suggesting that empiricists have failed meaningfully to define "ideology" for purposes of assessing appellate decisionmaking and proposing a new methodology for characterizing judges' votes based on past voting behavior).
70. Epstein & King, supra note 7, at 88.
71. Id. at 89.
72. Id. at 89 n.280.
73. See infra notes 149-66 and accompanying text.
74. William M. Landes & Richard A. Posner, Rational Judicial Behavior: A Statistical Study 3 (Univ. of Chi. Law & Econ., Olin Working Paper No. 404, 2008), available at http://papers.ssrn.com/abstract=1126403.
75. As discussed in Part I, that the party of the president and Senate at the time of a judge's confirmation correlates with some measure of case outcomes does not confirm that extralegal factors are in fact determining those outcomes. Debates and differences of perspective fully within the law could equally account for such differences.
76. Posner, The Role of the Judge, supra note 44, at 1058-59.
77. Id. at 1060.
78. Id. (footnote omitted).
79. Sisk, supra note 8, at 892 (footnote omitted).
80. Sisk & Heise, supra note 8, at 794.
81. See The Judicial Research Initiative, Appeals Court Data, http://www.cas.sc.edu/poli/juri/appctdata.htm (last visited Jan. 21, 2008). The database was originally compiled by Donald R. Songer and then updated by Ashlyn K. Kuersten and Susan B. Haire.
82. See infra text accompanying notes 74-77.
83. Donald R. Songer, The United States Courts Of Appeals Data Base Documentation For Phase 1, at 8 (n.d.), available at http://www.cas.sc.edu/poli/juri/cta_codebook.pdf. See generally Melissa M. Serfass & Jessie Wallace Cranford, Federal and State Court Rules Governing Publication and Citation of Opinions: An Update, 6 J. APP. PRAC. & PROCESS 349, 351-57 tbl.1 (2004) (setting forth publication standards in different circuits).
84. 57,973 appeals were filed in the courts of appeals for the year ending December 31, 2007. Admin. Office Of The U.S. Courts, Statistical Tables For The Federal judiciary tbl.B (2007), available at http://www.uscourts.gov/stats/dec07. During this same period, courts of appeals terminated 31,340 cases on the merits. Id. at tbl.B-5. Less than 17 percent of those dispositions were published. Admin. Office of the u.s. Courts, Judicial Business of The U.S. Courts 48 tbl.S-3 (2007), available at http://www.uscourts.gov/judbus2007/JudicialBusinespdfversion.pdf.
85. Epstein & King, supra note 7, at 85 tbl.5 (citing The Judicial Research Initiative, supra note 65).
86. Id. (noting that a researcher seeking to replicate or update a study based on a binary coding system for case outcome will have to make "judgment call[s], which may or may not be the same one[s] [the author] made.. .. [thereby] detract[ing] from the reliability of [the] measure").
87. Schubert, supra note 29, at 97.
88. See, e.g., Songer, supra note 68, at 5-6 ("[In the Songer database], the directionality of the court's decision was recorded, using conventional definitions of directionality that are closely analogous to those in the Spaeth Supreme Court data base [sic]. For most, but not all issue categories, these will correspond to notions of 'liberal' (coded as '3') and 'conservative' (coded as '1') that are commonly used in the public law literature.").
89. Chevron U.S.A. Inc. v. Natural Res. Def. Council, Inc., 467 U.S. 837 (1984).
90. Landes & Posner, supra note 58, at 3.
91. See Shapiro, supra note 19, at 481-82 (providing in-depth analysis of coding problems in a widely used database of Supreme Court cases).
92. For an example of an attempt to add legal issues to the coding process, see id.
93. See, e.g., Owen M. Fiss, Against Settlement, 93 Yale L.J. 1073, 1085 (1984).
94. Kenneth E. Scott, Two Models of the Civil Process, 27 STAN. L. REV. 937, 938 (1975).
95. See, e.g., Richard Posner, Economic Analysis Of Law passim (7th ed. 2007).
96. Cass Sunstein has written extensively on "judicial minimalism," the theory that courts should address only the case at hand, hew closely to precedent, and avoid overly disruptive changes in the law. See, e.g., Cass R. Sunstein, Radicals In Robes: Why Extreme Rightwing Courts Are Wrong For America 27-30 (2005). The simple measure of case outcome fails to distinguish between a minimalist and its opposite-clearly a distinction that Sunstein would find important.
97. Sisk, supra note 8, at 885.
98. See Mark A. Hall & Ronald F. Wright, Systematic Content Analysis of Judicial Opinions, 96 Cal. L. Rev. 63, 64 (2008).
99. Id. at 121. Unlike more traditional subjects of statistical study-like physical phenomena-"many of the civil justice phenomena that need study are not [well] suited to current quantitative analytic techniques." Deborah R. Hensler, Researching Civil Justice: Problems and Pitfalls, 51 Law & Contemp. Probs. 55, 63 (Summer 1988).
100. Stefanie A. Lindquist & Frank B. Cross, Empirically Testing Dworkin's Chain Novel Theory: Studying the Path of Precedent, 80 N.Y.U. L. Rev. 1156, 1180-81 (2005).
101. Id. at 1184.
102. In another part of the study, Lindquist and Cross found that, in the context of interpreting the phrase "under color of" state law from 42 U.S.C. § 1983, the power of precedent to reduce the predictive power of the almanac variable does not uniformly increase over time. Id. at 1198-200.
103. Cross, supra note 5, at 54.
104. Id. at 200.
105. See, e.g., Cass R. Sunstein et al., Are Judges Political?: An Empirical Analysis of The Federal Judiciary 150 (2006) ("The most difficult issues are resolved, [and] the principal empirical findings are clear. In many domains, Republican appointees vote very differently from Democratic appointees, and ideological tendencies are both dampened and amplified by the composition of the panel.").
106. See, e.g., Jordan, supra note 8 passim; Revesz, Environmental Regulation, supra note 8 passim.
107. See, e.g., Cross & Tiller, supra note 8 passim; Orin S. Kerr, Shedding Light on Chevron: An Empirical Study of the Chevron Doctrine in the U.S. Courts of Appeals, 15 Yale J. On Reg. 1, 38-39 (1998)
108. see also Richard J. Pierce, Jr., Is Standing Law or Politics?, 77 N.C. L. Rev. 1741, 1758-63 (1999) (correlating appointment variables and outcomes in standing cases).
109. See, e.g., Virginia A. Hettinger, Stefanie A. Lindquist & Wendy L. Martinek, Judging on a Collegial Court: Influences on Federal Appellate Decision Making passim (2006).
110. See, e.g., Revesz, supra note 52 passim.
111. See, e.g., Sara C. Benesh, The Contribution of "Extra" Judges, 48 ARIZ. L. REV. 301 passim (2006).
112. Thomas J. Miles & Cass R. Sunstein, The New Legal Realism, 75 U. Chi. L. REV. 831, 831 (2008) (quoting Karl N. Llewellyn, Some Realism About Realism: Responding to Dean Pound, 44 Harv. L. Rev. 1222, 1244 (1931)).
113. Sisk, supra note 8, at 876.
114. Frank B. Cross, Decision Making in The U.S. Courts of Appeals (2007).
115. Sunstein et al., supra note 91.
116. Landes & Posner, supra note 58.
117. Cross, supra note 5, at 3 (noting the use of two databases of published opinions).
118. Landes & Posner, supra note 58, at 2 (same).
119. Miles & Sunstein, supra note 97, at 18 ("Most of the relevant studies are limited to published judicial opinions.").
120. Miles & Sunstein, supra note 97, at 835.
121. Sisk, supra note 8, at 877.
122. Sunstein et al., supra note 91, at 4.
123. Id. at 8.
124. Id. at 19.
125. Id. at 6, 153 nn.4-7 (noting that the party of the appointing president is a "crude" proxy for judicial ideology).
126. Id. at 163.
127. Id. at 8-9.
128. Id. at 117.
129. Id. at 157-63 nn.2-25.
130. Id. at 161 n.20.
131. Id. at 19.
132. Id. at 20-21 tbl.2-1.
133. Id. at 22.
134. Id. at 23, 163 n.30.
135. Id. at 20-21 tbl.2-1.
136. Id. at 25.
137. Id. at 20-21 tbl.2-1.
138. Chevron U.S.A. Inc. v. Natural Res. Def. Council, Inc., 467 U.S. 837 (1984).
139. Sunstein et al., supra note 91, at 80.
140. Id. at 24.
141. Id. at 48.
142. Id. at 54-57.
143. Id. at 5.
144. Id. at 150.
145. Id. at 138.
146. Id. at 141-42.
147. Cross, supra note 5, at 3.
148. Sunstein also uses logistic regression to determine statistical significance. See Sunstein et al., supra note 91, at 163 n.30. However, the main analyses that Sunstein reports are not based on logistic regressions. Many people are familiar with ordinary least squares (OLS) regression, where a line of "best-fit" is developed for a given data set. Logistic regression is a form of regression used when the dependent variable is binary. While logistic regression differs in important ways from OLS regression, the idea is similar and each technique attempts to model the relationship between variables.
149. Cross, supra note 5, at 19.
150. Id. at 20.
151. Cross found a logit coefficient of .061. Id. at 24.
152. Id. This finding is based on the coefficient of .061. A logit coefficient must be interpreted with care. Unlike in the case of ordinary least square regressions, the slope coefficient in a logistic regression is not (an estimation of) the rate of change in the dependent variable as the independent variable changes. Rather, the slope coefficient is the rate of change in the log odds as the independent variable changes. Fred C. Pampel, Logistic Regression: A Primer 18-39 (2000). Where logistic regression is used only for the purposes of computing statistical significance (as in Sunstein's work), the interpretation of the logit coefficient does not pose a problem. However, where conclusions are drawn directly from the logit coefficients, there must be adequate explanation of how those conclusions follow.
153. Cross, supra note 5, at 27 tbl.1.3.
154. Id. at 27.
155. Id. at tbl.1.3.
156. Id. at 28 tbl.1.4.
157. Id. at 54 tbl.2.2. The logit coefficients were .040 for Giles score and .149 for the new lower court variable-indicating that the later variable had more explanatory power. Id.
158. Id. at 165.
159. Id. at 165 tbl.6.2. The Giles scores of the other judges (referred to by Cross as "OtherIdeology") had a coefficient of .086 and a p-score of .000, while the Giles score of the judge in question had a coefficient of .020 and a p-score of .106. Id.
160. Id. at 165.
161. Id. at 166 tbl.6.3. The median Giles score (referred to by Cross as "MedianIdeology") had a coefficient of .04 and a p-score of .816. The total Giles score (referred to by Cross as "TotalIdeology") had a coefficient of .087 and a p-score of .003. Id.
162. Id. at 166.
163. Id. at 103. Segal-Cover scores were developed by Jeffrey A. Segal and Albert D. Cover and attempt to identify ideological leanings of Supreme Court Justices using newspaper articles about the Justice prior to confirmation. See generally Jeffrey A. Segal & Albert D. Cover, Ideological Values and the Votes of U.S. Supreme Court Justices, 83 AM. POL. SCI. REV. 557 (1989) (using content-analytic techniques to create a predictive attitudinal model for judges' voting).
164. CROSS, supra note 5, at 229.
165. Id. at 228.
166. Id. at 168.
167. Landes & Posner, supra note 58, at 4.
168. Id. at 5.
169. Id. at 56 tbl.12. In Landes and Posner's analysis, whether PAP was Republican was the independent variable. The coefficient for fraction conservative in civil cases was .023; the coefficient for fraction liberal in civil cases was -.028; the coefficient for fraction conservative in criminal cases was .059; the coefficient for fraction liberal in criminal cases was -.051. Id.
170. See id. (noting that the coefficients were .192 for fraction conservative, and -.193 for fraction liberal in civil cases).
171. Id. at 30.
172. Id. at 32.
173. Id. at 33.
174. Id. at 33, 34.
175. Id. at 34.
176. Id. at 34-35.
177. Cross, supra note 5, at 229.
178. Id. at 4-5.
179. Sunstein et al., supra note 91, at 5.
180. Tamanaha, supra note 46 (manuscript at 11).
181. Daniel R. Pinello, Linking Party to Judicial Ideology in American Courts: A Meta-analysis, 20 Just. Sys. J. 219, 237 (1999). Administrative Office of the Courts data on the D.C. Circuit show that dissent rates hover from below 5 percent to 10 percent of cases for which opinions were written. When all dispositions (including unreported judgments) are taken into account, the dissent rate is even lower.
182. See supra note 69.
183. Judges rarely dissent in "unpublished" decisions.
184. Tamanaha, supra note 46 (manuscript at 12).
185. See, e.g., Ronald Dworkin, Law's Empire 130-50 (1986). Compare Japan Whaling Ass'n v. Am. Cetacean Soc'y, 478 U.S. 221, 229-30 (1986) ("[N]ot every matter touching on politics is a political question, and. .. it is 'error to suppose that every case or controversy which touches foreign relations lies beyond judicial cognizance.'" (quoting Baker v. Carr, 369 U.S. 186, 211 (1962))), with United States v. Pink, 315 U.S. 203, 229 (1942) ("What government is to be regarded here as representative of a foreign sovereign state is a political rather than a judicial question, and is to be determined by the political department of the government." (quoting Guar. Trust Co. v. United States, 304 U.S. 126, 137 (1938))).
186. N.Y. Times Co. v. Sullivan, 376 U.S. 254 (1964); see also Geoffrey R. Stone et al., The First Amendment 138 (2d ed. 2003) (discussing New York Times v. Sullivan and invoking Meiklejohn)
187. Harry Kalven, Jr., The New York Times Case: A Note on "The Central Meaning of the First Amendment," 1964 SUP. CT. REV. 191, 208, 221 n.125 (noting, in discussing New York Times v. Sullivan, that "[t]he Amendment has a 'central meaning'-a core of protection of speech without which democracy cannot function" and that Professor Mieklejohn had described the case as "an occasion for dancing in the streets").
188. See, e.g., Leegin Creative Leather Prods., Inc. v. PSKS, Inc., 127 S. Ct. 2705, 2710, 2714 (2007) (overruling the holding of Dr. Miles Medical Co. v. John D. Park & Sons Co., 220 U.S. 373 (1911), that it is a per se violation of the Sherman Act for a manufacturer to agree with its distributor to set a minimum price for the good at issue and citing Bork regarding the "procompetitve" effects of resale price maintenance (citing Robert H. Bork, The Antitrust Paradox 288-91 (1978))); State Oil Co. v. Khan, 522 U.S. 3, 7, 16 (1997) (overruling the holding of Albrecht v. Herald Co., 390 U.S. 145 (1968), that vertical maximum price fixing is a per se violation of the Sherman Act and quoting Bork for the proposition that such price fixing had "no anticonsumer effect" (quoting BORK, supra, at 281-82)).
189. In Textile Workers Union of America v. Lincoln Mills of Alabama, 353 U.S. 448 (1957), the Court held that a grievance-arbitration provision of a collective bargaining agreement could be enforced against unions and employers under section 301 of the Labor Management Relations Act ("LMRA"), 29 U.S.C. § 185, id. at 451. The Court found that congressional policy, as embodied in section 203(d) of the LMRA, 29 U.S.C. § 173(d), was to promote industrial peace and that the grievance-arbitration provision of a collective agreement was a major factor in achieving this goal. Id. at 453-55. In the seminal "Steelworkers Trilogy," the Court amplified this policy by declaring that the parties' agreement to arbitrate their disputes under a collective bargaining agreement would be enforced unless it could "be said with positive assurance that the arbitration clause is not susceptible of an interpretation that covers the asserted dispute." United Steelworkers of Am. v. Warrior & Gulf Navigation Co., 363 U.S. 574, 582-83 (1960)
190. see also United Steelworkers of Am. v. Enter. Wheel & Car Corp., 363 U.S. 593, 596-97 (1960) (part of the Steelworkers trilogy); United Steelworkers of Am. v. Am. Mfg. Co., 363 U.S. 564, 567-68 (1960) (same); cf. United Paperworkers Int'l Union, AFL-CIO v. Misco, Inc., 484 U.S. 29, 43-44 (1987) (emphasizing that a court may not set aside an arbitrator's award on grounds of public policy unless the policy clearly can be "ascertained 'by reference to laws and legal precedents and not from general considerations of supposed public interests'" (quoting Muschany v. United States, 324 U.S. 49, 66 (1945)); Alexander v. Gardner-Denver Co., 415 U.S. 36, 60 (1974) (holding that an employee's statutory right to trial de novo under Title VII of the Civil Rights Act of 1964, 42 U.S.C. § 2000e, is not foreclosed by prior submission of claim to final arbitration under the nondiscrimination clause of a collective bargaining agreement); Boys Mkts., Inc. v. Retail Clerk's Union, Local 770, 398 U.S. 235, 237-38 (1970) (holding that the anti-injunction provisions of the Norris-LaGuardia Act, 29 U.S.C. § 104, do not preclude a federal district court from enjoining a strike in breach of a no-strike obligation under a collective bargaining agreement when the agreement contains provisions enforceable under section 301(a) of the LMRA for binding arbitration of the grievance dispute concerning which the strike was called).
191. Indeed, omitting variables also increases the standard errors of included variables, and hence the omission of precedent and deliberation may reduce the likelihood of finding a significant effect from ideology. See Jacob Cohen, Patricia Cohen, Stephen G. West & Leona S. Aiken, Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences 143-44 (3d ed. 2003).
192. See, e.g., Cross, supra note 5, at 202 ("The close study of precedents and their impact [on appellate decisionmaking] is impossible with currently available or readily foreseeable empirical tools."); Sisk, supra note 8, at 884 ("More sophisticated statistical models that include legal factors and legal reasoning as variables are perhaps the greatest priority in continued quantitative examination of the federal judiciary. A fully specified legal model will prove eternally elusive [however] because legal reasoning is not formulaic in nature: the reasonable parameters for debate on the determinate nature of text and doctrine cannot be described by number." (footnote omitted)).
193. This Part has been written solely by Judge Edwards.
194. Edwards, Collegiality, supra note 3, at 1648.
195. Id. at 1644-45 (footnotes omitted).
196. Judge Posner has suggested that judges "don't deliberate very much" and that "Judicial deliberation is overrated." Posner, The Role of the Judge, supra note 44, at 1051. I could not disagree more. It may be that his experience on the Seventh Circuit has been different from mine on the D.C. Circuit.
197. Revesz, supra note 52, at 1112.
198. Landes & Posner, supra note 58, at 34-35.
199. "Easy" cases can be decided easily, with minimal judicial deliberations, so long as every judge is diligent in reviewing the case record. This is routine.
200. Cross and Lindquist and Cross, himself, have tried in a limited way to measure the effects of the law and controlling precedent on judicial decisionmaking. See CROSS, supra note 5; Lindquist & Cross, supra note 86, at 1157-58. But a number of empirical scholars have acknowledged that it is very difficult to capture the effects of law and precedent in a quantitative study. See supra Part I.C.3.
201. I include this information more as a qualitative point of interest than a formalized analytic finding
202. The data cover cases decided between September 2000 and July 2008. In the database compiled by the Clerk's Office for the U.S. Court of Appeals for the D.C. Circuit, cases involving review of administrative agency actions include matters in which a federal agency is a party and there is either a direct petition for review, an application for enforcement, a crossapplication for enforcement, or a direct petition for mandamus. The Clerk's Office review of administrative agency actions database does not include agency actions that are first reviewed in the district court and then appealed to the court of appeals. Nor does it include cases that are resolved by special panels (that is, three-judge panels that dispose of motions for summary affirmance, summary reversal, dismissals for want of jurisdiction, and dismissals on grounds of ripeness). Special panels almost always act without hearing oral argument, and they usually issue unpublished judgments, orders, and memoranda disposing of the matters under review. Judges almost never dissent from a special panel decision; and special panels rarely reverse agency actions.
203. Between February 2004 and March 2005, Judge Silberman served as the cochair of the Commission on the Intelligence Capabilities of the United States Regarding Weapons of Mass Destruction
204. Data compiled by and on file with the Clerk's Office of the U.S. Court of Appeals for the D.C. Circuit, Washington, D.C.
205. "Lead cases" include all cases that have been consolidated with the principal case under review. Even though the court's decision, judgment, or order disposes of more than one case, only the lead case is counted. For example, a disposition may include a lead case and seven separate consolidated appeals or petitions for review, but the disposition counts as only one case.
206. For a definition of cases involving review of administrative agency actions, see supra note 203. This total number includes both published and unpublished lead case dispositions, but excludes unpublished decisions issued by special panels.
207. Special panel orders, judgments, memoranda, and opinions almost never draw a dissent. For an explanation of special panels, see supra note 203. The Clerk's Office estimates that, from 2000-2001 through 2007-2008, special panels issued an additional 345 lead-case dispositions in cases involving administrative agency actions.
208. The figures were largely the same when I roughly scanned the agency actions that were first reviewed in the district court and then appealed to the court of appeals
209. See, e.g., Clayton, supra note 35, at 27 (discussing how attitudinalists simply assume that judges "vote their policy preferences and use legal principles to mask their true motives").
210. See, e.g., LaShawn A. v. Barry, 87 F.3d 1389, 1396-97 (D.C. Cir. 1996) (en banc) (holding that one panel of the court may not reconsider a prior panel's decision in the same case).
211. Data compiled by and on file with the Clerk's Office of the U.S. Court of Appeals for the D.C. Circuit, Washington, D.C.
212. Data compiled by and on file with the Clerk's Office of the U.S. Court of Appeals for the D.C. Circuit, Washington, D.C.
213. See Douglas H. Ginsburg & Brian M. Boynton, The Court En Banc: 1991-2002, 70 Geo. Wash. L. Rev. 259, 259-60 (2002) (noting that the number of en banc cases heard by the D.C. Circuit declined from sixty-three in the 1980s to thirty-three in the 1990s); see also id. at 260 (arguing that the decrease in the number of cases reheard en banc in the D.C. Circuit could be partially attributed to the judges becoming "more collegial, in the sense that the judges, notwithstanding their different views, had more confidence in each other's good faith and competence, and so deferred more to judgments of panels on which they did not sit").
214. Kahan, supra note 9, at 1
215. Yale Law School, The Cultural Cognition Project, http://culturalcognition.net/ ("The Cultural Cognition Project is a group of scholars from Yale and other universities interested in studying how cultural values shape the public's risk perceptions and related policy beliefs. Cultural cognition refers to the tendency of individuals to conform their beliefs about disputed matters of fact (for example, whether global warming is a serious threat; whether the death penalty deters murder; whether gun control makes society more safe or less) to values that define their cultural identities. Project members are using the methods of various disciplines- including social psychology, anthropology, communications, and political science-to chart the impact of this phenomenon and to identify the mechanisms through which it operates. The Project also has an explicit normative objective: to identify processes of democratic decisionmaking by which society can resolve culturally grounded differences in belief in a manner that is both congenial to persons of diverse cultural outlooks and consistent with sound public policymaking.").
216. Kahan, supra note 9, at 3.
217. Id. at 3 (footnote omitted).
218. Id. at 5.
219. Posner, The Role of the Judge, supra note 44, at 1065.
220. Kahan, supra note 9, at 5.
221. Dan M. Kahan & Donald Braman, Cultural Cognition and Public Policy, 24 Yale L. & Pol'y Rev. 149, 152 (2006).
222. Dan M. Kahan, David A. Hoffman & Donald Braman, Whose Eyes Are You Going to Believe? Scott v. Harris and the Perils of Cognitive Illiberalism, 122 Harv. L. Rev. 837, 852 (2009).
223. Kahan, supra note 9, at 5-6 (footnotes omitted).
224. See Posner, The Role of the Judge, supra note 44, at 1059.
225. Id. at 1060.
226. Schauer & Wise, supra note 43, at 1096-97.
227. Kahan, supra note 9, at 8-9.
228. Id. at 9.
229. This point is well documented in Kahan et al., supra note 224, at 879.
230. Id. at 879-80.
231. Edwards, Collegiality, supra note 3, at 1656.
232. See Hensler, supra note 85, at 63 ("Researchers simply do not have available very good quantitative approaches to studying large social organizations [like courts] or interaction processes [within the courts].").
233. Deliberation and its hallmarks are at the core of a collegial court: The deliberately cultivated attitude among judges of equal status and sometimes widely differing views working in intimate, continuing, open, and noncompetitive relationship with each other, which manifests respect for the strengths of the others, restrains one's pride of authorship, while respecting one's own deepest convictions, values patience in understanding and compromise in nonessentials, and seeks as much excellence in the court's decision as the combined talents, experience, insight, and energy of the judges permit. Frank M. Coffin, on Appeal: Courts, Lawyering And Judging 215 (1994).
234. Edwards, Collegiality, supra note 3, at 1648.
235. Judicial Conference oF the U.S., Long Range Plan for the Federal Courts 44 (1995).
236. See Kahan et al., supra note 224, at 900-01.
237. Edwards, Collegiality, supra note 3, at 1656, 1657.
238. Walter F. Murphy, Courts as Small Groups, 79 Harv. L. Rev. 1565, 1566-67 (1966).
239. Sisk, supra note 8, at 894.
240. Theodore Eisenberg, Why Do Empirical Legal Scholarship?, 41 San Diego L. Rev. 1741, 1741 (2004).
241. Cases compiled by the Clerk's Office of the U.S. Court of Appeals for the D.C. Circuit, Washington, D.C.
242. "MPS" denotes decisions involving mixed panels, that is, a three-judge panel consisting of judges appointed by presidents of both political parties in which the two judges appointed by presidents of the same party disagree. In an en banc decision, it means that judges appointed by presidents of the same party joined both the majority and dissent.
243. "PS" denotes decisions in which all of the judges on the panel were appointed by presidents of the same political party, and one of the judges dissented.