Coding complexity: Bringing law to the empirical analysis of the supreme court

Hastings Law Journal

Volumn 60, Issue 3, 2009, Pages 477-540

  Shapiro, Carolyn ^a  

^a CHICAGO KENT COLLEGE OF LAW   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 64949182215     PISSN: 00178322     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Article

References (382)

1. See generally, e.g., Jennifer Arlen & (Geoffry P. Miller, Second Annual Conference on Empirical Legal Studies (CELS): Announcement and Call for Papers (Feb. 6, 2007), available at http://papers.ssrn.com/s0l3/ papers.cfm?abstract-id=96i62o;
2. CELS 2009: Announcement and Call for Papers, http://law.usc.edu/cels/ (last visited Feb. 14, 2009); Cornell Law School: CELS 2008, http://www.lawschool.cornell.edu/cels2008/index.cfm (last visited Feb. 14, 2009);
3. Empirical Legal Studies, www.elsblog.org (last visited Feb. 14, 2009); First Annual Conference on Empirical Legal Studies: Announcement and Call for Papers, http://www.utexas.edu/law/news/2005/112805-black.html (last visited Feb. 14, 2009);
4. The Journal of Empirical Legal Studies-Journal Information, http://www.wUey.com/bw/society.asp?ref=i740-i453&site=i (last visited Feb. 14, 2009).
5. See, e.g., James J. Brudney & Corey Ditslear, Canons of Construction and the Elusive Quest for Neutral Reasoning, 58 VAND. L. REV. I, I (2005);
6. Cornell W. Clayton & J. Mitchell Pickeril, The Politics of Criminal Justice: How the New Right Regime Shaped the Rehnquist Court's Criminal Justice Jurisprudence, 94 GEO. L.J. 1385, 1385 (2006);
7. Frank B. Cross & Stefanie Lindquist, The Decisional Significance of the Chief Justice, 154 U. PA. L. REV. 1665, 1665 (2006);
8. Lee Epstein et al., The Political (Science) Context of Judging, 47 ST. LOUIS U. L.J. 783, 783, 812 (2003);
9. Lee Epstein et al. The Role of Qualifications in the Confirmation of Nominees to the U.S. Supreme Court, 32 FLA. ST. U. L. REV. 1145, 1145 (2005);
10. Lee Epstein & Gary King, The Rules of Inference, 69 U. CHI. L. REV. I. I (2002);
11. Mark David Hall, Jeffersonian Walls and Madisonian Lines: The Supreme Court's Use of History in the Religion Clause Cases, 85 OR. L. REV. 563, 563 (2006);
12. Thomas Hensley & Scott P. Johnson, Unanimity on the Rehnquist Court, 31 AKRON L. REV. 387, 387 (1998);
13. Andrew D. Martin et al., The Median Justice on the United States Supreme Court, 83 N.C. L. REV. 1275, 1275, 1284 n.35 (2005);
14. Theodore W. Ruger et al., The Supreme Court Forecasting Project: Legal and Political Science Approaches to Predicting Supreme Court Decisionmaking, 104 COLUM. L. REV. 1150, 1150 (2004);
15. Jeffrey A. Segal & Chad Westerland, The Supreme Court, Congress, and Judicial Review, 83 N.C. L. REV. 1323, 1323 (2005);
16. Keith E. Whittington, Taking What They Give Us: Explaining the Court's Federalism Offensive, 51 DUKE L.J. 477,477 (2001).
17. For discussions chronicling this development, see Frank Cross et al. Above the Rules: A Response to Epstein and King, 69 U. CHI. L. REV. 135, 135 n.2 (2002) (citing Symposium, Empirical and Experimental Methods in Law, 2002 U. III. L. REV. 789 (2002)): Richard L. Revesz. A Defense of Empirical Legal Scholarship, 69 U. CHI. L. REV. 169, 170 (2002);
18. Gregory C. Sisk & Michael Heise. Judges and Ideology: Public and Academic Debates About Statistical Measures, 99 Nw. U. L. REV. 743, 744-45 (2005); and infra text accompanying notes 7-8.
19. The databases are described in detail below.
20. See infra Part I.A. They are all multi-user databases that are publicly available.
21. See Supreme Court Data, http://www.cas.sc.edu/poli/ juri/sctdata.htm (last visited Feb. 14, 2009).
22. Harold Spaeth is an emeritus professor of political science at Michigan State University. In addition to having a Ph.D. in political science and a long and distinguished career in that field, he holds a J.D. from the University of Michigan. Harold J. Spaeth, Curriculum Vitae (Jan. 31, 2009) (on file with The Hastings Law Journal).
23. Epstein et al., The Political (Science) Context of Judging, supra note 2, at 812 (footnote omitted);
24. see also Michael Heise, The Past, Present, and Future of Empirical Legal Scholarship: Judicial Decision Making and the New Empiricism, 2002 U. ILL. L. REV. 819, 848 (2002) ('"There is little doubt that today [Spaeth's] U.S. Supreme Court Judicial Data Base is the greatest single resource of data on the Court; there are virtually no social-scientific projects on the Court that fail to draw on it.'" (quoting Lee Epstein, Social Science, the Courts, and the Law, 83 JUDICATURE 224. 225 (2000))).
25. Barry Friedman, Taking Law Seriously, 4 PERSP. ON POL. 261, 272 n.io (2006).
26. The search was in the JLR database and was for "te(Spaeth) and da(aft 12/31/2005)." It was conducted on June 25, 2007. A subsequent search for "Spaeth and da(aft 12/31/2005)" was run to double-check for any articles that the first search might have missed. The searches resulted in 117 documents, only eighty-five of which referred to Harold Spaeth's work.
27. Examples of law review articles relying on data from the Spaeth databases include Brudney & Ditslear, supra note 2, at 20;
28. Clayton & Pickeril, supra note 2, at 1411;
29. Cross & Lindquist, supra note 2, at 1685;
30. Frank B. Cross & Blake J. Nelson, Strategic Institutional Effects on Supreme Court Decisionmaking, 95 NW. U. L. REV. 1437,1483 (2001);
31. Michael Richard Dimino, Counter-Majoritarian Power and Judges' Political Speech, 58 FLA. L. REV. 53,96 (2006);
32. Epstein et al., The Political (Science) Context of Judging, supra note 2;
33. Ward Farnsworth, Signatures of Ideology: The Case of the Supreme Court's Criminal Docket, 104 MICH. L. REV. 67, 67 (2005);
34. Hensley & Johnson, supra note 2; Ruger et al., supra note 2, at 1153;
35. Christopher E. Smith, The Impact of New Justices: The U.S. Supreme Court and Criminal Justice Policy, 30 AKRON L. REV. 55, 55 (1996);
36. Michael E. Solimine & Rafeal Gely. The Supreme Court and the DIG: An Empirical and Institutional Analysis, 2005 WIS. L. REV. 1421, 1425 (2005); antrWhittington, supra note 2, at 481.
37. Because the other databases are largely derivative of the Supreme Court Database, much of the critique of the Database applies to the others as well. In addition to the Supreme Court databases, the website also houses a widely-used Appeals Court Database that has some similar coding protocols.
38. See Appeals Court Data, http://www.cas.sc.edu/poli/juri/appctdata. htm (last visited Feb. 14,2009).
39. 519 U.S. 357 (1997).
40. Id. at 366-67,371.
41. Id. at 379-85.
42. All of the database coding reported in this Article can be found in the Database.
43. See supra note 4; infra note 28.
44. The Database's influence is extensive and surprising. For example, a popular website that archives Supreme Court oral arguments and opinion announcements (as well as the opinions), organizes the cases "by issue.
45. See Oyez: U.S. Supreme Court Case Summaries, Oral Arguments, & Multimedia, www.oyez.org (last visited Feb. 14, 2009). In so doing, it uses Spaeth's issue coding, but without explanation.
46. See id. Many scholars-including many non-empirical scholars-might take advantage of the website, and are likely to find its organization to be anything from mysterious to misleading.
47. Ironically, the Database's ubiquity is sometimes presented as an argument for its continued uncritical use.
48. See, e.g., Ruger et al., supra note 2, at 1175 (acknowledging that Spaeth's "issue area categories may seem awkward or even arbitrary from a legal perspective, as they do not neatly track traditional doctrinal categories," but defending their use in part because the "coding protocol is well- defined, and his issue area labels have been widely used by political scientists").
49. A word about terminology may be helpful here. In statistics, reliability has a very precise meaning. A statistical measure of something is reliable if it produces the same results repeatedly. EDWARD G. CARMINES & RICHARD A. ZELLER, Reliability and Validity Assessment ii (I979). So a clock that is always precisely 9 minutes slow is reliable. Of course, this clock is inaccurate-it does not correctly measure what it purports to measure (the time). In the parlance of statistics, the clock's measurements of time therefore are not valid.
50. Id. at 12. Although it would certainly be possible to discuss the Database and its limitations in these and other technical terms, that would, I believe, make this Article somewhat less accessible to the less-technically minded readers I hope to reach. Therefore, I use the term "reliable" in its colloquial, descriptive sense, not in its technical, statistical sense. Of course, the two uses of the terms sometimes overlap.
51. Spaeth is a leader in this field. See generally, e.g., JEFFREY A. SEGAL & HAROLD J. SPAETH, The Supreme Court and the Attitudinal Model (1993);
52. JEFFREY A. SEGAL & HAROLD J. SPAETH, The Supreme Court and the Attttudinal Model Revisited (2002) [hereinafter SEGAL & sPAETH Revisited]. There are, of course,political scientists who are critical approach.
53. See generally, e.g., Cornell W. Clayton, The Supreme Court and Political Jurisprudence: New and Old Instititutionalisms, in Supreme Court Decision-making: New Institutionalist Approaches 15, 15-41 (Cornell W. Clayton &Howard Gillman eds., 1999);
54. Howard Gillman, What's Law Got to Do with It? Judicial Behavioralists Test the 'Legal Model' of Judicial Decision Making, 26 LAW & SOC. Inquiry 465 (2001). Legal academics as well have been critical, particularly of the attitudinal model (described in more detail in Parts I.A and I.B.2.a.ii), which posits that the case outcomes and Justices' votes can be explained by reference to the Justices' policy preferences and ideology.
55. See, e.g., Ward Farnsworth, The Use and Limits of Martin-Quinn Scores to Assess Supreme Court Justices, with Special Attention to the Problem of Ideological Drift, 101 NW. U. L. REV. COLLOQUY 143, 145-48 (2007);
56. John CP. Goldberg, What Nobody Knows, 104 MICH. L. REV. 1461, 1482-84 (2006) (describing the reductionism of the attitudinal model's understanding of "attitudes"); Heise, supra note 6, at 836 (noting that empirical studies may overstate the role of ideology because they do not account for the particularly hard cases that the Supreme Court decides);
57. Revesz, supra note 3, at 177-78 (criticizing positive scholars for over-reliance on case outcome and lack of understanding of the practical workings of the justice system);
58. David Stras, The Incentives Approach to Judicial Retirement, 90 MINN. L. REV. 1417, 1422-25 (2006) (describing and echoing many critiques of the attitudinal model); id. at 1428-30
59. (describing and adding to critique of rational choice model); Emerson H. Tiller & Frank B. Cross, What Is Legal Doctrine?, 100 NW. U. L. REV. 517, 522-25 (2006) (criticizing attitudinal model for ignoring law and legal doctrine);
60. Ernest A. Young, Judicial Activism and Conservative Politics, 73 U. COLO. L. REV. 1139, 1189 (2002). Nor have all examinations of political scientists' quantitative approaches resulted in criticism by legal scholars.
61. See, e.g., Sisk & Heise, supra note 3, at 788-90 (finding judicial common space scores a useful measure of ideological difference in religious freedom cases).
62. Friedman, supra note 7, at 262. Other political scientists indeed do explore the way that legal doctrine operates.
63. See generally Herbert M. Kritzer & Mark J. Richards, Jurisprudential Regimes and Supreme Court Decisionmaking: The Lemon Regime and Establishment Clause Cases, 37 LAW & SOC'Y REV. 827 (2003);
64. Stefanie A. Lindquist & David E. Klein, The Influence of Jurisprudential Considerations on Supreme Court Decisionmaking: A Study of Conflict Cases, 40 LAW &- SOC'Y REV. 135 (2006);
65. Mark J. Richards & Herbert M. Kritzer, Jurisprudential Regimes in Supreme Court Decision Making, 96 AM. POL. SCI. REV. 305 (2002).
66. A Natural Court is one during which there are no personnel changes. Saul Brenner, The New Certiorari Game, 41 J. POL. 649, 653 (1979). The last Rehnquist Natural Court lasted from 1994 to 2005. The Justices who served on it were Rehnquist, Stevens, O'Connor, Scalia, Kennedy, Souter, Thomas, Ginsburg, and Breyer.
67. This critique is not a condemnation of every use of the Database. To the contrary, the Database can serve-and has served-many useful purposes, if used properly.
68. See infra notes 28-37 and accompanying text.
69. Friedman, supra note 7, at 262.
70. It should be emphasized that most of the limitations I identify are not latent. To the contrary, most of them reflect deliberate choices Spaeth made in constructing the Database and about which he is explicit. Nonetheless, some scholars appear to overlook, misunderstand, or insufficiently appreciate them. "That [the databases] are widely used does not necessarily mean they are widely understood." Sara C. Benesh, Becoming an Intelligent User of the Spaeth Supreme Court Databases i (unpublished paper, prepared for the Southwestern Political Science Association, Mar. 2002), available at http:// www.cas.sc.edu/poli/juri/benesh-handout.pdf.
71. Heise, supra note 6.
72. See The Judicial Research Initiative, http://www.cas.sc.edu/poli/ juri/ (last visited Feb. 14, 2009).
73. Supreme Court Data, supra note 4.
74. Id.
75. Benesh, supra note 22.
76. The coding and accompanying decision rules are set out in detail in a publicly-available codebook. Harold J. Spaeth. The Original United States Supreme Court Judicial Database 1953-2007 Terms: Documentation (2008) [hereinafter Codebook 2008], available at http://www.cas.sc.edu/ poli/juri/allcourt-codebook.pdf. There are different codebooks for the different databases, all publicly available.
77. See Supreme Court Data, supra note 4.
78. Codebook 2008, supra note 28, at 9-16, 28.
79. Id. at 66-73.
80. See Epstein et al., The Political (Science) Context of Judging, supra note 2, at 790 (explaining the usefulness of having a standard determination of when each Natural Court begins and ends). Of course, as previously discussed, this advantage becomes a disadvantage when the Database coding is problematic.
81. See supra text accompanying note 15.
82. Lori A. Ringhand, Judicial Activism: An Empirical Examination of Voting Behavior on the Rehnquist Natural Court, 24 CONST. COMMENT. 43,46 (2007).
83. Id. at 43-63.
84. Ryan Black & Lee Epstein, Recusals and the "Problem" of an Equally Divided Supreme Court, 7 J. ApP. PRAC. & PROCESS 75,85 (2005).
85. Hensley & Johnson, supra note 2, at 388.
86. Kevin M. Scott, Understanding Judicial Hierarchy: Reversals and the Behavior of Intermediate Appellate Judges, 40 LAW & SOC'Y REV. 163,174 (2006).
87. Cross & Nelson, supra note 9, at 1484.
88. Codebook 2008, supra note 28, at 53-55. The ideological coding in particular areas has recently been challenged.
89. See Nancy Staudt et al., The Ideological Component of Judging in the Taxation Context, 84 WASH. U. L. REV. 1797, 1808 (2006) (noting that Spaeth's liberal/conservative coding does not have explanatory power in the context of economic cases, and proposing a different approach for coding the outcome of tax cases);
90. see also Michael S. Greve & Jonathan Klick, Preemption in the Rehnquist Court: A Preliminary Empirical Assessment, 14 SUP. CT. ECON. REV. 43, 79-80 (2006) (noting difficulties in identifying liberal and conservative outcomes in preemption cases);
91. Matthew J. Sag et al., The Effect of Judicial Ideology in Intellectual Property Cases 8 (July 2, 2007) (unpublished manuscript), available at http://papers.ssrn.com/sol3/papers.cfm?abstract-id=997963 (same with respect to intellectual property cases).
92. Harold Spaeth, Jurimetrics and Professor Mendelson: A Troubled Relationship, 27 J. POL. 875, 879(1965).
93. Lee Epstein et al., Judging Statutes: Thoughts on Statutory Interpretation and Notes for a Project on the Internal Revenue Code, 13 WASH. U. J.L. & POL'Y 305,320 (2003).
94. Id. This focus on outcome to the exclusion of law can reach extremes that, to a lawyer, are mind-boggling: describing the "apparently unidimensional nature of Supreme Court decisionmaking," two political scientists (including Spaeth's frequent coauthor Jeffrey Segal) assert in a 2005 article: "The vote on the merits in any given case is as straightforward as a majority rule process gets. Justices essentially make a binary, reverse or affirm decision." Segal & Westerland, supra note 2, at 1351.1324.
95. See, e.g., SEGAL & SPAETH, REVISITED, supra note 17, at 86. The attitudinal model and its related theory, the strategic model,
96. see LEE EPSTEIN & JACK KNIGHT, THE CHOICES JUSTICES MAKE (1998), are not the only ways to think about the role of politics or political preferences in Supreme Court decision making.
97. See, e.g., Farnsworth, supra note 9 at 74-88, 93 (discussing the relationship between judicial preferences or "priors" and case facts, outcomes and reasoning);
98. McNollgast, Politics and the Courts: A Positive Theory of Judicial Doctrine and the Rule of Law, 68 S. CAL. L. REV. 1631, 1635 (1995) (arguing that electoral stability leads to doctrinal stability and that electoral and political changes are likely to lead to doctrinal changes).
99. See Cross & Lindquist, supra note 2, at 1688 ("The data on case outcomes may simply be too unrefined to identify an ideological effect. They show nothing about the content of the opinion");
100. Frank B. Cross et al., Warren Court Precedents in the Rehnquist Court, 24 CONST. COMMENT. 3,4 (2007)
101. ("[T]he binary outcome coding cannot measure whether a particular opinion is moderately liberal (or conservative) or more extremely ideological." (citations omitted));
102. Michael J. Gerhardt. Attitudes About Attitudes, 101 MICH. L. REV. 1733,1733-34 (2003) (reviewing objections to the attitudinal model and Spaeth and Segal's responses).
103. Parents Involved in Cmty. Schs. v. Seattle Sch. Dist. No. 1,127 S. Ct. 2738 (2007).
104. Id. at 2768;
105. id. at 2788 (Kennedy, J., concurring in part and concurring in the judgment).
106. Id. at 2791 (Kennedy, J., concurring in part and concurring in the judgment).
107. Tiller & Cross, supra note 17, at 523;
108. see also CASS SUNSTEIN eT aL., ARE JUDGES POLITICAL? AN EMPIRICAL ANALYSIS OF THE FEDERAL JUDICIARY 65 (2006) ("For the actual development of the law, the opinion matters a great deal."); Revesz, supra note 3, at 177 n.39.
109. See Barry Friedman, The Politics of Judicial Review, 84 TEX. L. REV. 257, 260 (2005).
110. Building resources and conducting scholarship that take account of law and doctrine does not necessarily mean, of course, that scholars will find that the Justices' votes and opinions can be explained or predicted in whole or in part on the basis of "law," and I make no such claim here. More legally nuanced coding and analysis might weaken the claims of those who believe that the Justices are in fact motivated largely by ideology. But it also might strengthen those claims by showing that the Justices treat similar legal issues differently in different "public policy contexts."
111. Joshua B. Fischman & David S. Law, What Is Judicial Ideology, and How Do We Measure It?, 29 WASH. U. J.L. POL'Y (forthcoming 2009) (manuscript at 19), available at http://papers.ssrn.com/s0l3/ papers.cfm?abstract-id=i 121228).
112. Greve & Klick, supra note 38, at 79 (emphasis added) (footnotes omitted);
113. see also Epstein et al., supra note 40, at 305-07;
114. Tiller & Cross, supra note 17, at 523;
115. Young, supra note 17.
116. See infra note 81 and accompanying text.
117. See, e.g., Epstein et al., The Political (Science) Context of Judging, supra note 2, at 808 (praising the "richness" of the Database's coding and encouraging legal scholars to rely on it);
118. see also supra text accompanying notes 6-8.
119. I do not discuss here certain other codes, such as authority for decision or type of party, that also may be problematic. This Article's discussion is thus more illustrative than comprehensive.
120. Codebook 2008, supra note 28, at 42 (emphasis added).
121. All of the cases discussed in this Part were in the random sample of ninety-five cases drawn for the Recoding Project, described in Part III.
122. The cases discussed are Rousey v. Jacoway, 544 U.S. 320 (2005);
123. Almendarez-Torres v. United States, 523 U.S. 224 (1998);
124. Clinton v. Jones, 520 U.S. 681 (1997);
125. Schenck v. Pro-Choice Network of Western New York, 519 U.S. 357 (1997);
126. and Markman v. Westview Instruments, Inc., 517 U.S. 370 (1996).
127. Codebook 2008, supra note 28, at 42-52.
128. Abortion is issue 533.
129. Id. at 48. Confrontation is issue no.110.
130. Id. at 45. Union arbitration is issue 553.
131. Id. at 48. Consumer protection is issue 656.
132. Id. at 49.
133. Id. at 42-52-
134. Id. at 52.
135. Id. at 44-47, 50-51.
136. Id. at 31.
137. See Supreme Court Data, supra note 4.
138. Codebook 2008, supra note 28, at 42.
139. Dimino, supra note 9, at 97 (emphasis added).
140. Martin et al., supra note 2 (emphasis added).
141. Timothy R. Johnson et al., Passing and Strategic Voting on the U.S. Supreme Court, 39 LAW & SOC'Y REV. 349,363 (2005) (emphasis added).
142. Youngsik Lim, An Empirical Analysis of Supreme Court Justices' Decision Making, 29 J. LEGAL STUD. 721,734 (2000) (emphasis added).
143. Lee Epstein & Jeffrey A. Segal, Trumping the First Amendment?, 21 WASH. U. J.L. & POL'Y 81,93 (2006) (emphasis added).
144. Codebook 2008, supra note 28, at 42.
145. Epstein et al., The Political (Science) Context of Judging, supra note 2, at 808.
146. The authors actually significantly overstate the number of codes in the criminal procedure area, saying that there are "well over hundreds of distinct issues." Id. In fact, the criminal procedure area contains sixty distinct issues. Codebook 2008, supra note 28, at 44-45.
147. Codebook 2008, supra note 28, at 50.
148. Id. at 42 ("Of the many thousand records in the database, few have a legal basis for decision that applies to a second issue.").
149. To calculate this percentage, I first identified the total number of orally argued cases decided between October 1953 and December 2006 by relying on case citations (analu = o) and decision types (1, 6, and 7), for a total of 6138 cases. I identified the total number of orally argued multi-issue cases (528) by identifying cases assigned more than one issue (analu = 2 and 5), keeping the relevant decision types (1,6, and 7) constant, and deleting duplicate case citations.
150. Id. at 31.
151. Id. at 42 ("[E]ach legal provision should not generally have more than a single issue applied to it.").
152. Id.
153. See id. at 42 ("Although criteria for the identification of issues are hard to articulate, the focus here is on the subject matter of the controversy rather than its legal basis.");
154. id. ("I have attempted to identify issues on the basis of the Court's own statements as to what the case is about.");
155. id. ("The objective is to categorize the case from a public policy standpoint, a perspective that the legal basis for decision. commonly disregards.").
156. See id. at 53-54 (noting that although some exceptions exist, "each issue in each case will either indicate a liberal or conservative outcome");
157. see also id. at 53 (listing the exceptions, and noting that for a very small number of specific issues in the Interstate Relations and Miscellaneous issue, "[a]n '8' has been entered in the DIR variable of these cases either because the issue does not lend itself to a pro or con description (e.g., a boundary dispute between two states), or because no convention exists as to which is the pro side and which is the con side").
158. Artificial unidimensionality is not the only way to solve this problem. In fact, in those few cases that have more than one issue, Spaeth reportedly codes the political direction of the outcome with reference to the first issue coded-presumably the issue he concludes is dominant. Epstein &Segal, supra note 69, at 94 n.52.
159. Had Spaeth coded the case as involving a First Amendment issue, he might well have identified the votes in the opposite ideological direction, because vindications of First Amendment rights are generally coded as liberal. Codebook 2008, supra note 28, at 58. That Spaeth's identification of issue is crucially related to his determination of the ideology of a case raises questions about the circularity of his coding and of the attitudinal model itself.
160. See Paul H. Edelman & Jim Chen, The Most Dangerous Justice Rides into the Sunset, 24 CONST. COMMENT. 199, 207 (2007) ("[T]he choice to code Rosenberger as an establishment controversy," as opposed to a free speech controversy, "almost certainly led to the coding of this decision as conservative." (citing Rosenberger v. Rectors & Visitors of the Univ. of Va., 515 U.S. 819 (1995)));
161. Young, supra note 17, at 1190 ("[M]any cases can be classified as either 'liberal' or 'conservative,' depending on the salience of different factors to the classifier. Was United States v. Lopez a 'liberal' decision, because it was 'pro-person accused or convicted of crime,' or was it 'conservative,' because it was anti-federal power?" (footnote omitted)).
162. See Codebook 2008, supra note 28, at 44-52.
163. There are no codes that can plausibly be understood to relate to immunities. There is one code for sex discrimination in employment (284).
164. Id. at 44. There is another code (222) defined as: "employment discrimination: on basis of race, age, religion, national origin, or working conditions. Not alienage, which is 272, or gender, which is 284."
165. Id. This definition could conceivably include sexual harassment as being "on the basis of" working conditions, but working conditions and harassment are not bases of discrimination. They are a manifestation of discriminatory intent, actionable only if motivated by impermissible discrimination.
166. See Oncale v. Sundowner, 523 U.S. 75, 80 (1998) ("We have never held that workplace harassment, even harassment between men and women, is automatically discrimination because of sex merely because the words used have sexual content or connotations.").
167. Compare Codebook 2008, supra note 28, at 44 (search and seizure codes), with id. at 49 (securities code).
168. 517 U.S. 370,372 (1996).
169. Id. at 384-91.
170. Codebook 2008, supra note 28, at 45.
171. In fact, criminal and civil jury rights involve entirely different constitutional amendments and legal doctrines.
172. See U.S. CONST, amend. VI; U.S. CONST, amend. VII. Spaeth himself acknowledges that there are some such discrepancies. So, for example, Spaeth says that codes 111 through 119, which appear within the criminal procedure issue area and deal with "subconstitutional fair procedure need not necessarily pertain to a criminal action." Codebook 2008, supra note 28, at 45.
173. Codebook 2008, supra note 28, at 82.
174. See, e.g., Dimino, supra note 9, at 86 n.163 (relying on Spaeth's coding of First Amendment cases to discern the Justices' track records in upholding First Amendment challenges);
175. id. at 96-98 & n.214 (same for "judicial power" cases);
176. Smith, supra note 9, at 58-64 (relying on "criminal procedure" cases).
177. This problem may not matter for every research project. It is less likely to affect an analysis that, for example, lumps together a number of issue areas-civil rights, criminal procedure, due process, privacy, and First Amendment, for example, to study the Court's treatment of "civil liberties" cases in general.
178. See, e.g., Lee Epstein et al., The Supreme Court During Crisis: How War Affects Only Non-War Cases, 80 N.Y.U. L. Rev. 1, 3-10 (2005). Efforts to study more specific areas of the law or interactions between particular areas of law are much more likely to be unreliable if they use the Database uncritically.
179. 520 U.S. 681,684 (1997).
180. Id.
181. There is a decent argument that this case is miscoded even on Spaeth's own terms. Arguably, the case should have been coded as "391: liability, civil rights acts. tort actions involving liability that are based on a civil rights act," within the Civil Rights issue area. Codebook 2008, supra note 28, at 49.
182. Paula Jones sued Bill Clinton and a former Arkansas state police officer, Danny Ferguson. Clinton, 520 U.S. at 684. She brought two counts against Clinton alone: one under § 1983 for depriving her of her constitutional rights, and one state common law count for intentional infliction of emotional distress.
183. Id. at 684-86. She brought two counts against both defendants-one under § 1985 for conspiracy to deprive her of her constitutional rights and one state law cause of action for defamation.
184. Id. So the case involved both civil rights and non-civil rights torts claims, and the case was widely characterized as a sexual harassment case. But the bottom line is that Paula Jones' specific causes of actions are among the least important aspects of the case.
185. Clinton also raises questions about Spaeth's liberal/conservative designations. He codes the outcome in Clinton as liberal, presumably because it is a decision in favor of a sexual harassment plaintiff against a powerful defendant, a civil rights plaintiff against a government official. Codebook 2008, supra note 28, at 53. But in the political atmosphere of its day, Clinton certainly was understood to advance the cause of conservative interests. In contrast, Spaeth codes Bush v. Gore. 531 U.S. 98 (2000), as a conservative decision. Most people would undoubtedly agree, but here Spaeth appears to be using the opposite criteria from Clinton. After all, the victor in Bush v. Gore was the plaintiff, who was advancing a broad and novel theory of equal protection-generally a liberal view.
186. Id. at 111.
187. Codebook 2008, supra note 28, at 31.
188. There are a few exceptions. "5AMI" is the legal provision code for Miranda wgs.
189. Id. at 35. In addition, the Codebook identifies the following doctrines as legal provisions: abstention, retroactivity of a constitutional right, exclusionary rule, harmless error, res judicata, estoppel, and writ improvidently granted.
190. Id. at 38.
191. Chevron U.S.A. v. Natural Res. Def. Council, 467 U.S. 837 (1984).
192. Skidmore v. Swift & Co., 323 U.S. 124 (1944).
193. There is an issue code-in the judicial power issue area-for "judicial review of administrative agency's or administrative officer's actions and procedures" (721). Codebook 2008, supra note 28, at 50. This code is overinclusive, covering all kinds of judicial review of all kinds of administrative actions. Moreover, because of the presumption in favor of only one issue per case, this code may not be attached to many cases that in fact involve such review.
194. The Codebook explicitly acknowledges the possibility that some cases will not have a coded legal provision.
195. Id. at 32.
196. These limitations do not necessarily lead to problems with every study. A scholar interested in Chevron, for example, could certainly generate her own list of cases in which the Court relied on or addressed that case or the doctrine that bears its name. But not all studies lend themselves to such independent data collection, and not all researchers understand when and why they should try to undertake it.
197. Harold J. Spaeth, The Original United States Supreme Court Judicial Database 1953-2003 Terms: Documentation 33 (2005) [hereinafter Codebook 2005] (on file with The Hastings Law Journal). In September 2008, Spaeth revised his Codebook. The current Codebook has a slightly different explanation. According to the September 2008 Codebook, "[t]he basic criterion to determine the legal provision(s) [within a case] is the 'summary' in the Lawyers' Edition [a commercial reporting service provided by LexisNexis]. Supplementary is a reference to it in at least one of the numbered holdings in the 'Syllabus By Reporter Of Decisions.'" Codebook 2008, supra note 28, at 31. This change is unexplained in the Codebook, but Professor Spaeth tells me that he made the change after he read an earlier draft of this Article and realized that the Codebook inaccurately described his protocol for this variable. E-mail from Harold J. Spaeth, Research Professor of Law, Michigan State University College of Law, to author (Feb. 9, 2009, 07:52:00 CST) (on file with author). In this Article, my analysis of Spaeth's coding protocols for legal provision primarily addresses his reliance on the syllabus, but all of my criticisms apply with equal, if not greater, force to reliance on a commercial reporter's summary.
198. 104- Codebook 2005, supra note 103, at 33; Codebook 2008, supra note 28, at 31.
199. Of course, the Lawyers' Edition summaries also have no authority whatsoever. Moreover, they do not appear in the slip opinions or official reporters, and, unlike the syllabi, are not even drafted by Court personnel.
200. The Supreme Court itself makes all these limitations explicit in a note that appears before every syllabus: "The syllabus constitutes no part of the opinion of the Court but has been prepared by the Reporter of Decisions for the convenience of the reader."
201. See, e.g., Crawford v. Metro. Gov't. 129 S. Ct. 846, 847 (2009) (citing United States v. Detroit Timber & Lumber Co., 200 U.S. 321, 337 (1906) ("In the first place, the headnote is not the work of the court, nor does it state its decision-though a different rule, it is true, is prescribed by statute in some States. It is simply the work of the reporter, gives his understanding of the decision, and is prepared for the convenience of the profession in the examination of the reports.")).
202. The syllabi are written by the Reporter of Decisions, an office of career appointees who assist the Court in its work. ROBERT L. STERN eT aL., SUPREME COURT PRACTICE 30 (8th ed. 2002).
203. Id.
204. Codebook 2008, supra note 28, at 31.
205. But see id. ("Where this summary lacks numbered holdings, it is treated as though it has but one number.").
206. 523 U.S. 224 (1998).
207. Id. at 228-39.
208. Id. at 229-35.
209. Id. at 239.
210. Id.
211. Id. at 239-47.
212. Id. at 224-25.
213. Id. at 225. Oddly (in light of the September 2008 revisions to the Codebook), although the Lawyers' Edition summary explicitly mentions the Fifth Amendment Due Process Clause, Almendarez-Torres v. United States, 140 L.E.2d 350, 350 (1998), Spaeth does not code it as a legal provision.
214. See infra Parts I.B.2.b.iii and III.A.2.c. for more discussion of the Database's failure to code legal provisions even when they fall within its protocols.
215. The case would also not be picked up through its issue coding. Oddly, the issue coded is double jeopardy. Although double jeopardy is mentioned in the opinion, it is not in fact at issue.
216. Id. at 247.
217. The Database does not consistently report legal provisions referenced in the Lawyers' Edition summary either.
218. Codebook 2008, supra note 28, at 31.
219. 544 U.S. 320 (2005).
220. Brief for Petitioners at i, Rousey v. Jacoway, 544 U.S. 320 (2005) (No. 03-1407), 2004 WL 1900505.
221. 544 U.S. at 320.
222. Id. at 320-21. The Internal Revenue Code is likewise mentioned but not described or analyzed in the Lawyers' Edition summary. Rousey v. Jacoway, 161 L.Ed.2d 563, 563-64 (2005).
223. Rousey, 544 U.S. at 320-22.
224. Id. at 327-29,328 n.3.
225. Codebook 2008, supra note 28, at 31.
226. See discussion supra Part I.B.
227. Not every one of the Database's limitations affects every study, of course. Some scholars do their own additional coding to compensate for the limitations, and some studies rely on aspects of the coding that are not implicated by my critiques. In addition, randomly distributed errors will probably not affect a study's findings, at least in large scale studies. Here, however, I explore a few examples of studies that, because of the way they rely on the Database, produce unreliable findings. This discussion, like the discussion in Part I, is more illustrative than comprehensive.
228. See, e.g., Andrew D. Martin & Kevin M. Quinn, Dynamic Ideal Point Estimation via Markov Chain Monte Carlo for the U.S. Supreme Court, 1953-1999, 10 POL. ANALYSIS 134, 145 (2002) (noting that the assumption of unidimensionality is made in "nearly all statistical analyses of Supreme Court behavior").
229. Edelman & Chen, supra note 81, at 199. Edelman and Chen go on to criticize this assumption in detail.
230. Id. at 203-09. They note, for example, that a Justice's response to a case involving a single issue, such as abortion, may include competing preferences that are sometimes explicitly incorporated into the law in the form of a balancing test.
231. Id. at 205-06.
232. Martin et al., supra note 2, at 1279 & n.17.
233. Id. at 1279.
234. Id. at 1302-03.
235. Farnsworth, supra note 17, at 144.
236. Martin et al., supra note 2, at 1281.
237. Id.
238. Id. at 1284 n.35. My calculation of the proportion of multi-issue cases in the Database results in a somewhat higher, but still small, percentage (8.6%).
239. See supra note 74 and accompanying text. Edelman and Chen criticize the authors' methodology on a number of grounds.
240. See Edelman & Chen, supra note 81, at 203. Among other things, they point out that the authors' technical description of their own methodology does not make sense.
241. Id. at 204. They also argue that the Database's issue coding is unlikely to accurately identify cases involving single-dimension issue space, and they consider whether the number of legal provisions coded in a case might help in identifying the number of issue dimensions.
242. See id. at 204-06. They conclude, however, that more than one legal provision might relate to the same issue, while, on the other hand, it is possible that the same legal provision-for example, equal protection-could be relevant to multiple dimensions.
243. Id. at 207.
244. This criticism arguably holds even if the authors agree with Spaeth's decision to code public policy context rather than legal issue.
245. In fairness, the authors of The Median Justice on the United States Supreme Court do not rely solely on the Database for their claim that most Supreme Court cases are single issue cases. They also cite Bernard Grofman and Timothy J. Brazill, Identifying the Median Justice on the Supreme Court Through Multidimensional Scaling: Analysis of "Natural Courts" 1953-1991, 112 PUB. CHOICE 55 (2002). Grofman and Brazill find that, using a statistical technique called multidimensional scaling (MDS), a one-dimensional statistical model explains over 80% of the variation in the votes.
246. Id. at 58. Even in their model, however, close to 20% of the variation in votes goes unexplained.
247. Id. Moreover, their model operates on an assumption that each individual Justice operates along a single dimension, discounting the possibility that individual Justices evaluate cases and issues from more than one perspective.
248. Id. As noted, this assumption may be unwarranted.
249. See supra note 132. Even to the extent that Grofman and Brazill's conclusions about unidimensionality are warranted, that does not mean that the Spaeth Database accurately identifies that dimension with its issue coding.
250. For the most part, the authors' identification of each Term's median justice is unsurprising. But the mere identification of the median justice does not add a lot of new information to the study of the Supreme Court. Most observers of the Court know who the key swing voters are and how the Justices generally line up from left to right.
251. See Farnsworth, supra note 17, at 1894 (noting "everyone already knows" that "each Justice predictably votes with certain colleagues and not others"). It is the quantification that is new-the effort to put a precise number on the likelihood of a Justice to be the median in any given case during each Term. And it is this quantification, with the apparent precision of numbers-numbers with lots of digits after the decimal point-that the researchers' use of the Database renders unreliable.
252. See Edelman & Chen, supra note 81, at 218-19 (arguing that the median justice is not necessarily the most powerful Justice, and demonstrating an alternative methodology to identify Justices most likely to be in winning coalitions).
253. Scott P. Johnson, The Influence of Case Complexity on the Opinion Writing of the Rehnquist Court, 25 OHIO N.U. L. REV. 45, 47 (1999). Johnson is not the only one to erroneously equate case complexity wholly or in part with the number of issues coded in the Database.
254. See, e.g., Robert J. Hume, The Use of Rhetorical Sources by the U.S. Supreme Court, 40 LAW & SOC'Y REV. 817,826 (2006);
255. Johnson et al., supra note 67, at 361. Another example of reliance on the number of issues coded is Solimine and Gely, supra note 9.
256. Codebook 2008, supra note 28, at 42.
257. Lim, supra note 68, at 733-34.
258. Id. at 734.
259. Id.
260. Id.
261. 530 U.S. 133,149 (2000).
262. Id. at 149-51 (citing Anderson v. Liberty Lobby, Inc., 477 U.S. 242, 247-50 (1986)).
263. See Supreme Court Data, supra note 4.
264. Anderson was one of three cases known as the summary judgment trilogy. Arthur R. Miller, The Pretrial Rush to Judgment: Are the "Litigation Explosion," "Liability Crisis," and Efficiency Cliches Eroding Our Day in Court and Jury Trial Commitments?, 78 N.Y.U. L. REV. 982, 984 (2003). The other cases are Matsushita Electrical Industrial Co. v. Zenith Radio Corp., 475 U.S. 574, 587-86 (1986), and Celotex Corp. v. Catrett, 477 U.S. 317, 322-23 (1986). The trilogy is often credited with signaling the lower courts that they should grant summary judgment more willingly, and the cases are frequently discussed as a group.
265. See, e.g., Miller, supra, at 984-85 (blaming "an expansive reading of the trilogy" for lower courts' undue willingness to grant summary judgment). Yet the three cases do not share any issue area codes. Matsushita is coded as an economic activity case, Celotex as a judicial power case, and Anderson as a First Amendment case.
266. See Supreme Court Data, supra note 4.
267. Epstein & Segal, supra note 69.
268. Id. at 8I.
269. The specific issue codes allow the authors to weed out First Amendment cases coded as involving religion. Spaeth also has identified separate legal provision codes for the First Amendment religion clauses and the speech clause. Codebook 2008, supra note 28, at 37.
270. Id. at 47. Because legal provisions are themselves underreported, it is likely that even Epstein and Segal inadvertently exclude some First Amendment cases from their sample.
271. See discussion infra Part III.B.2. It is not obvious, however, that such underreporting likely has a systematic skew.
272. Epstein & Segal, supra note 69.
273. Id.
274. 491 U.S. 397, 419-20 (1989).
275. Epstein and Segal acknowledge that in some sense every case is a value-conflict case. Epstein & amp; Segal, supra note 69, at 94 n.49. Texas v. Johnson, for example, could be seen as involving a conflict between freedom of expression values on the one hand with respect for the symbol of the flag and all it represents. Although Epstein and Segal do not say so explicitly, it appears that they are specifically interested in cases in which a (liberal) vote in favor of First Amendment rights would be a vote against some other ideologically liberal value or position. It is worth noting here that I am able to undertake this dissection of Epstein and Spaeth's work only because they (1) explicitly set forth most of their coding protocols in their article, and (2) make their data available. Lee Epstein's Dataverse, http://dvn.iq. hanrvard.edu/dWdv/lepstein/faces/study/StudyPage.jsp;jsessionid= 90oc9f5o6555eo3ioci40 985dd33.dvnlnstancei?studyld=735 (last visited Feb. 14, 2009). These are norms that legal academics would do well to adopt.
276. See generally Epstein & amp; King, supra note 2.
277. 500 U.S. 173, 201 (1991).
278. Rust v. Sullivan is not the only case where the different values do not in fact conflict ideologically.
279. See e.g., In re Sawyer, 360 U.S. 622, 639-40 (1959) (reversing disciplinary action by state bar of lawyer representing defendants in anti-Communist activity trial; coded with First Amendment and Attorneys issue areas; identified as value conflict by Epstein and Segal).
280. See, e.g., Cramp v. Bd. of Pub. Instruction, 368 U.S. 278, 288 (1961) (finding statutorily-required loyalty oath void for vagueness in violation of due process);
281. Cafeteria & amp; Rest. Workers' Union, Local 473 v. McElroy. 367 U.S. 886, 899 (1961) (upholding against due process challenge military officer's decision not to allow civilian worker on base);
282. Deutch v. United States, 367 U.S. 456, 472 (1961) (reversing contempt of Congress conviction on due process grounds);
283. Konigsberg v. State Bar of Cal., 366 U.S. 36, 56 (1961) (upholding state bar decision to deny admission for applicant's refusal to answer question about Communist activities);
284. Nelson v. County of L.A., 362 U.S. 1, 9 (1960) (upholding dismissal of social workers for refusing to answer questions from a congressional committee);
285. Greene v. McElroy, 360 U.S. 474, 508 (1959) (reversing administrative decision, unsupported by congressional or presidential authority, to deprive government contractor of his position);
286. Kent v. Dulles, 357 U.S. 116, 130 (1958) (holding that the Secretary of State did not have authority to issue regulation denying passports to Communists);
287. Watkins v. United States, 354 U.S. 178, 216 (1957) (striking down contempt of Congress conviction on due process grounds);
288. Slochower v. Bd. of Educ, 350 U.S. 551, 559 (1956) (reversing summary termination of professor who invoked his right against self-incrimination);
289. Peters v. Hobby, 349 U.S. 331, 349 (1955) (striking down actions of Loyalty Board);
290. Barsky v. Bd. of Regents, 347 U.S. 442, 456 (1954) (upholding suspension of physician's license following conviction for refusing to produce subpoenaed documents).
291. Epstein and Segal would likely respond to this critique by pointing out that what they purport to be measuring is the likelihood that the Court rules in favor of "First Amendment values," not necessarily in favor of First Amendment rights themselves. Most readers of their article, however, are unlikely to make such a subtle distinction. For one thing, the authors themselves describe the cases they are looking at as involving "First Amendment claims" and as cases in which "the First Amendment guarantees of press, speech, assembly, or association were at stake." Epstein & amp; Segal, supra note 69, at 81, 92 (emphasis added). Lawyers reading such descriptions would certainly assume that the cases in fact involve the First Amendment, not simply related "values."
292. These Cold War cases suggest other interesting challenges for Epstein and Segal. The Warren Court decided this series of cases in part because there were such cases to decide during the Cold War. Theoretically, therefore, the Warren Court's apparent favoring of First Amendment values relative to subsequent Courts could reflect instead a contemporaneous antipathy towards First Amendment values by other government actors, leading to this series of cases.
293. See Barbara Graham, Explaining Supreme Court Policymaking in Civil Rights: The Influence of the Solicitor General, J953-2002, 31 Pol'y Stud. J. 253, 261-62 (2003) (explaining shift in Court's treatment in race discrimination cases in part due to a change in the nature of the cases the Court was handling-a change from identifying constitutional violations to remedying them);
294. see also Jonathan P. Kastellec & Jeffrey R. Lax, Can We Ignore Case Selection When We Study Judicial Politics? (Oct. 22, 2007) (unpublished paper prepared for the Second Annual Conference on Empirical Legal Studies), available at http://papers.ssrn.com/sol3/papers.cfm?abstract-id=951873 (arguing that the Supreme Court's case selection has substantial implications for empirical scholars trying to draw conclusions about the Court's decision making).
295. Similarly, and without explanation, Epstein and Segal identify cases with more than one issue as pure cases as long as any one of the issues is a First Amendment issue.
296. See, for example, Lorillard Tobacco Co. v. Reilly, 533 U.S. 525 (2001), which is coded by Spaeth as having both a First Amendment and a federalism issue, but is coded by Epstein and Segal as a "pure" First Amendment case. Lee Epstein's Dataverse, supra note 160.
297. See Supreme Court Data, supra note 4.
298. 536 U.S. 765 (2002).
299. See Supreme Court Data, supra note 4.
300. White, 536 U.S. at 788.
301. Id. at 797-803 (Stevens, J., dissenting);
302. id. at 803-21 (Ginsburg, J., dissenting).
303. Additionally, in part due to Spaeth's failure to consistently code legal provisions, discussed in detail in Part III, their sample entirely omits cases that should be included.
304. See, e.g., Cal. Bankers Ass'n v. Schultz, 416 U.S. 21, 56 (1974) (holding, inter alia, that First Amendment freedom of association claims of the ACLU were not yet justiciable).
305. Any of these outcomes are possible. Some cases wrongly coded as pure cases have a majority of conservative Justices who decide the case in favor of "First Amendment values" and a group of liberal Justices who dissent, emphasizing other values. Republican Party of Minnesota is just such a case. Were it correctly classified as a value-conflict case, it would therefore support Epstein and Segal's conclusions. And there are other such cases.
306. See, e.g., Lorillard Tobacco Co. v. Reilly, 533 U.S. 525, 571 (2001) (finding, by conservative majority, some tobacco advertising regulations unconstitutional and finding others preempted by federal law). On the other hand, there are some cases, likewise wrongly designated as pure, in which some or all of the liberal Justices reject the competing values and vote in favor of First Amendment values.
307. See, e.g., Hutchinson v. Proxmire, 443 U.S. III, I33 (I979) (holding that certain statements by U.S. Senator were not subject to immunity and that target of those statements was not a public figure for purposes of libel; Justice Brennan was the only dissenter). Such cases detract from Epstein and Segal's claims. These two types of cases have offsetting effects on Epstein and Segal's results, but, depending on how many of each there are. they may not completely offset each other. A wholesale recoding of Epstein and Segal's cases would also require identifying cases involving the First Amendment that are not included in their sample because they lack the necessary First Amendment issue and legal provision codes.
308. See Anna Harvey, What Makes a Judgment 'Liberal'? Coding Bias in the United States Supreme Court Database (unpublished paper prepared for the Third Annual Conference on Empirical Legal Studies), available at http://papers.ssrn.com/sol3/papers.cfm?abstract-id=II20970 (arguing that the identity of the Justices in the majority and the dissent may influence selection of the ideology and issue codes).
309. The sample was generated by Stata 9. Stata is a commercial statistical software package used to analyze, manage, and present data.
310. See generally Alan C. Acock, A Gentle Introduction to Stata (2d ed. 2008); Stata: Data Analysis and Statistical Software, www.stata.com (last visited Feb. 14, 2009) (providing information about Stata). I define an individual case as an opinion with a unique case citation. I included only cases with published opinions of the Court. I selected the last Rehnquist Natural Court for purposes of manageability. The findings I report here must therefore be understood within that context. For a list of all the cases used in the Recoding Project, see infra Appendix 2. The Recoding Project Data and Codebook can be found online.
311. See Carolyn Shapiro Dataverse, Replication Data for: Coding Complexity: Bringing Law to the Empirical Analysis of the Supreme Court, http://dvn.iq.harvard.edu/dvn/dv/shapiro/faces/study/StudyPage.jsp?studyId= 38o8o (last visited Feb. 14, 2009).
312. See infra Appendix 1.
313. 467 U.S. 837 (1984).
314. 424 U.S. 319(1976).
315. 542 U.S. 600 (2004).
316. 384 U.S. 436 (1966).
317. 470 U.S. 298 (1985).
318. Id. at 3I8.
319. 542 U.S. at 6I8. There were also other differences between the factual circumstances of the two cases, which the Seibert court noted.
320. Id. at 615-16.
321. See supra note 97 (noting that Spaeth himself recognizes the Miranda doctrine as a legal provision).
322. Hamdi v. Rumsfeld, 542 U.S. 507 (2004), was assigned seven legal provisions; Slack v. McDaniel, 529 U.S. 473 (I999), was assigned nine legal issues.
323. The 95% confidence intervals for these sample means do not overlap at all as between the Original Database coding and the Recoding Project. There is a 95% chance that the actual mean number of issues and issue areas for Spaeth's coding for the last Rehnquist Natural Court is between 0.990 and I.031. There is a 95% chance that were the Recoding Project's protocol applied to all of the cases from the last Rehnquist Natural Court, the mean number of issues coded per case would be between 3.3476 and 4.I05032. Likewise, there is a 95% chance that the mean number of issue areas coded per case would be between 2.197594 and 2.602406.
324. To calculate this statistic, I compared the number of unique case citations (analu=o)-958- with the number of cases in which the Original Database reports multiple issues (analu = 2 and analu=5). It is not possible to easily calculate the number of issues reported per case, nor is it possible to determine the number of cases with multiple issue areas.
325. For this calculation, if I coded a newly created issue area, I checked to see if that new area was primarily based on one of Spaeth's issue areas. If so, and if Spaeth coded that original area, I counted it as a match. In addition, there was one case that Spaeth coded as a privacy case that I also counted as a match because the case involved abortion rights, newly relocated in the due process issue area. In this recalculation, there were a total of nine additional cases that I concluded should be counted as a match. Five of the cases even had identical issue codes in the Original Database and in mine. Of the remaining nine unmatched cases, Spaeth coded two as privacy cases, three as criminal cases, two as civil rights cases, and two as economic activity cases.
326. For a discussion of the relative merits of a very detailed coding system versus a more general one, see infra Part IV.B.
327. My finding that the Database gives short shrift to structural issues is consistent with other research. In their 2006 article, Michael S. Greve and Jonathan Klick begin an empirical study of Rehnquist Court cases addressing preemption, a structural issue implicating the relationship between the states and the federal government. Finding that reliance on "[e]ven the most complete, up-to-date, and widely-used data set, the United States Supreme Court Judicial Data Base [sic].⋯ contains only a sample of 'preemption' cases-a good number of which do not conform to something a competent lawyer would recognize as preemption," the authors generated their own list of preemption cases. Greve & Klick, supra note 38, at 46. They used several techniques, including a LEXIS keyword search and "less systematic means, such as reviews of the pertinent legal literature."
328. Id. at 91-92. Out of their final list of I05 preemption cases, the Database failed to code thirty-four cases (32.38%) as preemption cases, underreporting this important structural issue.
329. Id. at 92 n.99. Of the seventy-six cases coded as preemption cases by the Database, eight were not in fact preemption cases.
330. Id. There are other preemption studies that rely at least in part on Spaeth's coding, thereby missing a significant number of cases Greve and Klick identified and counting a few that should not have been included.
331. See, e.g., Brady Baybeck & amp; William Lowry, Federalism Outcomes and Ideological Preferences: The U.S. Supreme Court and Preemption Cases, 30 PUBLIUS 73, 83-84 (2000);
332. James B. Staab, The Tenth Amendment and Justice Scalia's "Split Personality," I6 J.L. & POL. 231, 300 n.266 (2000);
333. see also Whittington, supra note 2, at 506 n. I39 (identifying the universe of "federalism" cases as those Spaeth coded with issue codes 930-49).
334. For her work on tax cases, Professor Staudt has constructed her own database of every Supreme Court tax case from I94I through 2004. Staudt et al., supra note 38, at 1815 n.63 (describing nature and content of LEXIS search). For those years that her database overlaps with the Original Database, Staudt compared the two and found that the Original Database accurately identified about 88% of the tax cases. E-mail from Nancy Staudt, Class of 1940 Research Professor of Law at Northwestern Univ. Sch. of Law, to author (Sept. I5, 2008, 15:52:00 CST) (on file with the author). This substantial overlap is not surprising. Federal taxation is a fairly discrete and easily identifiable area of law and policy. But these cases may well also involve structural and jurisprudential issues like federalism, federal government operations and powers, administrative law, and judicial procedure; all of which are likely to go uncoded as issues in the Database.
335. 530 U.S. 133 (2000). Reeves was in the Recoding Project sample.
336. See infra Appendix 2.
337. 507 U.S. 163(1993).
338. 534 U.S. 506 (2002).
339. See generally cases cited supra notes I92-94.
340. The Database does not even identify the Rules of Civil Procedure as a legal provision for Reeves, although it does for both Leatherman and Swierkiewicz. See Supreme Court Data, supra note 4. The problem of underreported legal provisions is discussed in the following subsection.
341. Cf. Edelman & Chen, supra note 81; Fischman & Law, supra note 50.
342. 545 U.S. 1 (2005).
343. Id. at 5.
344. For a particularly entertaining discussion of the Justices' dilemmas in this case, see Dahlia Lithwick, Dude, Where's My Integrity? Medical Marijuana Tests the Supreme Court's True Love of Federalism, Slate, Nov. 29, 2004, http://www.slate.com/id/2110204;
345. see also Michael C. Dorf, Whose Ox is Being Gored? When Attitudinalism Meets Federalism, 21 Sr. John's J. Legal Comment. 497,507-12 (2007);
346. Ernest A. Young, Just Blowing Smoke? Politics, Doctrine, and the Federalist Revival After Gonzales v. Raich, 2005 Sup. Ct. Rev. 1, 1-3 (2005).
347. Raich's only issue code is in the federalism issue area.
348. See Supreme Court Data, supra note 4.
349. To calculate this statistic, I compared the number of unique case citations (analu=o)-958- with the number of cases in which the Original Database reports multiple legal provisions (analu=3 and analu=5). It is not possible to easily calculate the number of legal provisions Spaeth coded per case.
350. For example, Spaeth codes the legal provision in Jerome B. Grubart, Inc. v. Great Lakes Dredge & Dock Co., 513 U.S. 527 (1995), as 28 U.S.C. §1331, which is the statute authorizing federal question jurisdiction in the federal courts. But Grubart is about maritime jurisdiction. 5I3 U.S. at 529. The correct legal provision is 28 U.S.C. § I333, the statute authorizing maritime jurisdiction. Likewise, in Cooper Industries, Inc. v. Leatherman Tool Group, Inc., 532 U.S. 424, 433-34 (2001), the Court addressed the substantive limits that the Fourteenth Amendment places on punitive damages. The only legal provision that Spaeth codes is the Fifth Amendment.
351. For example, Spaeth codes the legal provision in Solid Waste Agency of Northern Cook County v. U.S. Army Corps of Engineers, 531 U.S. 139 (2001), as 33 U.S.C. § 1334, when the correct legal provision is 33 U.S.C. § 1344. (33 U.S.C. § 1334 does not exist.) In addition, a number of legal provisions in which the statute is labeled with a five-digit section number contain only four of those digits. (Spaeth acknowledges that the Database sometimes so codes such five-digit statutes. Codebook 2008, supra note 28, at 33.)
352. See, e.g., Norfolk S. Ry. v. Shanklin, 529 U.S. 344 (2000) (addressing 49 U.S.C. § 20106; coded in Database as 49 U.S.C. § 2010); Norfolk & W. Ry. v. Hiles, 516 U.S. 400 (I996) (addressing 49 U.S.C. § 20302; coded in Database as 49 U.S.C. § 2030).
353. In calculating this match, I counted as legal provisions as matching if they identified the same statute, even if they used different notation. Some statutes can be identified by an abbreviation of their popular name, but statutes may also be identified by citation. Likewise, a citation might be to the first section of a statute or to a particular section within it. For some discussion on the challenges of coding statutes, see infra Part IV.B.
354. Readers familiar with the Database might wonder whether the STATE code is necessary to identify cases that address state laws. One of Spaeth's codes-authority for decision-identifies cases in which the Court is engaged in judicial review of state action or law for constitutionality. But this code is both underinclusive and overinclusive. It is underinclusive because not every case in which the Court considers a state law receives this code, even under Spaeth's coding protocols. Statutory preemption cases, for example, which may require the Court to construe a state statute to determine if it conflicts in some way with federal law, generally do not receive this particular code as they do not involve constitutional judicial review. In fact, of the eighteen cases I coded as involving a state law, only nine were coded by Spaeth as involving judicial review of state action or law. And the code is overinclusive in that it identifies cases in which the state action being reviewed is not a statute or ordinance but is, for example, some action taken by an executive or law enforcement official or a ruling of a state court.
355. Three were not mentioned at all, and one was mentioned but not in a numbered holding.
356. Additionally, identifying "seminal cases" appears to be particularly challenging. When I had twenty cases independently recoded, the recoders identified substantially more such cases than I did. See infra Appendix I.
357. Pub. L. No. 104-132, no Stat. 1214 (codified as amended in scattered sections of 8, 18, 22, 28, and 42 U.S.C).
358. Pub. L. No. I02-166, I05 Stat. 1071 (codified as amended in scattered sections of 2 U.S.C).
359. 517 U.S. 370, 372(1996).
360. Tiller and Cross have proposed some interesting ideas for getting at this kind of multilayered legal analysis. See generally Tiller & Cross, supra note 17;
361. see also Epstein et al., supra note 40, at 322 ("[While] social scientists code cases addressing the Internal Revenue Code as involving one issue: 'federal taxation[,]' [t]his description⋯ ignores all the subsidiary questions the Court may have addressed, including problems of statutory interpretation, concerns about the federal budget or the national economy, and general notions of equity and efficiency.").
362. One potential approach worth further investigation would be to turn to the West Key Number System. This System has the potential to provide at least some of the detail missing from the Database, as it is geared towards practicing lawyers who might be interested in any number of aspects of a given case, and so aims for more comprehensive coding than the Database. On the other hand, the level of detail provided appears to vary greatly from case to case, with some cases identified by many, often repetitive keys, and others identified by only the most general.
363. I drew the sample originally for a project that, due to the Database's limitations, I was unable to complete.
364. Codebook 2008, supra note 28, at 3.
365. See Carolyn Shapiro, Codebook for Coding Complexity: Bringing Law to the Empirical Analysis of the Supreme Court i (2009), available at http://dvn.iq.harvard.edu/dvn/dv/shapiro/faces/study/StudyPage.jsp?studyId= 38o8o&tab=files (follow "download" hyperlink located to the right of "Coding Complexity Codebook.pdf").
366. Id. at 3.
367. 542 U.S. 600 (2004).
368. 384 U.S. 436 (1966).
369. 470 U.S. 298 (1985).
370. Id. at 318.
371. 542 U.S. at 618. There were also other differences between the factual circumstances of the two cases, which the Seibert court noted.
372. Id. at 615-16.
373. See supra note 97 (noting that Spaeth himself recognizes the Miranda doctrine as a legal provision).
374. 424 U.S. 319(1976).
375. Id. at 335.
376. See Shapiro, supra note 216, at 3.
377. Codebook 2008, supra note 28, at 35, 38.
378. SHAPIRO, supra note 216, at 5.
379. Cohen's Kappa is a statistical comparison of the coding of the same items by different coders, taking into account the level of agreement that would occur purely by chance. "If Kappa equals o then the amount of agreement between the two coders is exactly what one would expect by chance. If Kappa equals I, then the coders agree perfectly." James F. Spriggs II & Thomas G. Hansford, Measuring Legal Change: The Reliability and Validity of Shepard's Citations, 53 POL. RES. q. 327, 334 n.n (2000) (citing Jacob Cohen, A Coefficient of Agreement for Nominal Scales, 20 EDUC. & PSYCHOL. MEASUREMENT 37 (1960)).
380. 2 Stata Base Reference Manual: Release 9, at 2 (2005) (citing J.R. Landis & G.G. Koch, The Measurement of Observer Agreement for Categorical Data, 33 Biometrics 159, 165 (1977)) (offering these interpretations).
381. In looking at agreement on legal provision, I looked for substantive agreement. So, for example, if the recoder identified two sections of one large statute, and my recoding identified that large statute, I counted that coding as a single match. To calculate Cohen's Kappa, I created one row per case for each unique legal provision identified in the case. Where both the recoder and I identified the same legal provision, both were identified in that row. Where the recoder identified a legal provision that I did not, I put a dummy code in the column for my coding, and vice versa. Note that this approach understates the intercoder agreement, as it fails to capture the myriad legal provisions that the recoder and I both agreed not to include.
382. To calculate this measure, I made one row per case for each issue area. For twenty recoded cases, and eighteen issue areas, there were 360 rows. For each row, I created two dichotomous variables-one each for my coding and for the recoder. I calculated Cohen's Kappa based on the match between these variables.