On the effective communication of the results of empirical studies, Part II

Vanderbilt Law Review

Volumn 60, Issue 3, 2007, Pages 801-846

  Epstein, Lee ^a,c   Martin, Andrew D ^b   Boyd, Christina L ^b  

^a NORTHWESTERN UNIVERSITY   (United States)

^b WASHINGTON UNIVERSITY   (United States)

^c NORTHWESTERN UNIVERSITY SCHOOL OF LAW   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 36249030999     PISSN: 00422533     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Review

References (118)

1. Daniel E. Ho, Scholarship Comment, Why Affirmative Action Does Not Cause Black Students to Fail the Bar, 114 YALE L.J. 1997 (2005).
2. Ho is responding to the controversial claims in Richard H. Sander, A Systemic Analysis of Affirmative Action in American Law Schools, 57 STAN. L. REV. 367 (2004).
3. These observable variables include race, gender, LSAT score, and undergraduate GPA. Ho, supra note 1, at 1999.
4. The figure appeared in Ho's work. Id. Ho utilizes data from Sander, supra note 1. s
5. See, e.g., Emily Bazelon, Sanding Down Sander, SLATE, Apr. 29, 2005, http://www.slate.com/id/2117745/ ("The forthcoming responses to Sander pounce on several of his moves (which they call causal inferences). To begin with, there is the problem of 'post-treatment bias,' which means that it's a bad idea to control for a factor that is itself a consequence of the cause you're studying. That no-no is explained by Daniel Ho . . .") ;
6. Vic Fleischer, On Changing One's Mind, A TAXING BLOG, May 9, 2005, http://vic.typepad.com/taxingblog/2005/05/ on_changing_one.html ("Perhaps my initial agreement with Sander was in part out of an urge to defend him. In any event, I've changed my mind. Dan Ho's presentation changed my mind. Ask yourself - when was the last time an empirical paper changed your mind about an issue like affirmative action?").
7. Ho, supra note 1, at 2002 n.25.
8. Lee Epstein, Andrew D. Martin, & Matthew M. Schneider, On the Effective Communication of the Results of Empirical Studies, Part I, 59 VAND. L. REV. 1811 (2006) [hereinafter Communication I].
9. To wit: Aim for Clarity and Impact Iterate Write Detailed Captions Id. at 1845.
10. Id. at 1814; see also Gary King, Michael Tomz, & Jason Wittenburg, Making the Most of Statistical Analyses: Improving Interpretation and Presentation, 44 AM. J. POL. SCI. 347, 360 (2000) (arguing that such attention to interpretation and presentation "could help bridge the acrimonious and regrettable chasm that often separates quantitative and nonquantitative scholars, and make the fruits of statistical research accessible to all who have a substantive interest in the issue under study"). Much of the inspiration for our series (especially infra Part III) comes from this article.
11. We also assume that readers of this piece have at least skimmed Communication I, supra note 6. Accordingly, we do not reiterate, e.g., the basics of good graphic construction, among other topics, here.
12. See generally Andrew Gelman et al., Let's Practice What We Preach: Turning Tables into Graphs, 56 AM. STATISTICIAN 121 (2002).
13. See also Communication I, supra note 6, at 1842-43 n.15.
14. For discussions of inference, see generally, e.g., Communication I, supra note 6; Lee Epstein & Gary King, The Rules of Inference, 69 U. CHI. L. REV. 1 (2002).
15. See Epstein & King, supra note 11, at 29 ("[D]escriptive inferences are different than data summaries. We do not make them by summarizing facts; we make them by using facts we know to learn about facts we do not observe.")
16. To be clear, causal inference is the difference between two descriptive inferences. More specifically, a causal inference is the difference in the dependent variable between the situation where the treatment is applied and the situation where the control is applied. Different statistical models approach causal inference using varying modeling assumptions. See, e.g., Epstein & King, supra note 11, at 36.
17. Daniel M. Schneider, Using the Social Background Model to Explain Who Wins Federal Appellate Tax Decisions: Do Less Traditional Judges Favor the Taxpayer?, 25 VA. TAX REV. 201 (2005).
18. Id. at 211, 221.
19. Id. at 221-22.
20. For more on Schneider's data, see infra Table 1.
21. For more on this point, see, e.g., Communication I, supra note 6, at 1819-21; EDWARD R. TUFTE, THE VISUAL DISPLAY OF QUANTITATIVE INFORMATION 168 (2nd ed. 2001).
22. Ho, supra note 1.
23. As we emphasize throughout this Article, there are different rules for describing data collected versus performing inference.
24. See Ho, supra note 1, at 2003 fig.1 (providing 95% confidence intervals for all estimated effects).
25. See Communication I, supra note 6, at 1838 n.72 (noting Gelman's apparent disagreement).
26. That is, whether presenting data or results, researchers must aim for clarity and impact, employ iterative efforts to improve visualization and craft detailed captions. Id. at 1811.
27. Schneider, supra note 14, at 213 n.36, 216 n.42.
28. Quantitative variables come in two varieties: those that can only take on a limited, or finite, number of values are discrete; and those that can be any possible number are continuous. See ALAN AGRESTI & BARBARA FINLAY, STATISTICAL METHODS FOR THE SOCIAL SCIENCES 16 (3d ed. 1997).
29. Categorical variables can be ranked (e.g. interval and ordinal variables) or unranked (e.g. nominal variables). See id.
30. As an illustration, if a dependent variable is quantitative, oftentimes a linear regression model is appropriate. If, however, a dependent variable is dichotomous, a logistic regression model would usually be appropriate. See J. SCOTT LONG, REGRESSION MODELS FOR CATEGORICAL AND LIMITED DEPENDENT VARIABLES (1997).
31. Michael Kao, Comment, Calculating Lawyers' Fees: Theory and Reality, 51 UCLA L. REV. 825, 838 (2004).
32. For recommendations on this point, see discussion infra Part IV.
33. See WILLIAM G. JACOBY, STATISTICAL GRAPHICS FOR UNIVARIATE AND BIVARIATE DATA 47 (1997) ("[R]esearchers often have difficulty seeing the forest (i.e., a variable's distribution) because of the trees that it contains (i.e., the individual observations).").
34. Illustrative is Guhan Subramanian's study, The Influence of Antitakeover Statutes on Incorporation Choice: Evidence on the "Race" Debate and Antitakeover Overreaching, 150 U. PA. L. REV. 1795 (2002), which sought to join the "race to the top/bottom" debate by exploring whether managers migrate to states with anti-takeover statutes in place at the time of their decision to incorporate. As part of his demonstration that "bottom" proponents have the better argument, he presents and labels the measurements of a single continuous variable: the number of companies incorporating in a number of states. Id. at 1815 fig.2. In the left panel of the figure below we reproduce his display. Id. Unlike Kao's inclination to provide information on and label each case in his study, Subramanian's strikes us as entirely reasonable: the observations are small in number, familiar, and of clear substantive interest to participants in the debate he seeks to engage. Moreover, for all the reasons we discussed in Communication I, supra note 6, Subramanian shows good sense in graphing the data rather than presenting it in tabular form, as did Kao. On the other hand, and again for the reasons we offered in the earlier paper, we would draw a line at his use of pie charts. These "pop" displays are never a good choice, and here the chart is particularly problematic. Subramanian's figure obscures the data, making visualization difficult - perhaps even more difficult than a tabular display. Subramanian, supra, at 1815 fig.2. Far better for purposes of decoding, as Cleveland demonstrates, is the dot chart, located in the right panel.
35. See WILLIAM S. CLEVELAND, THE ELEMENTS OF GRAPHING DATA 262-63 (2d ed. 1994) ("[With a dot plot we] can effortlessly see a number of properties of the data that are either not apparent at all in the pie chart or that are just barely noticeable");
36. see also William S. Cleveland & Robert McGill, Graphical Perception: Theory, Experimentation, and Application to the Development of Graphical Methods, 79 J. AM. STAT. ASS'N 531, 545 (1984) ("A pie chart can always be replaced by a bar chart . . . . [But] we prefer dot charts . . . ."). Indeed, we strongly recommend dot charts for those rather rare circumstances in which labeling the measurements of a quantitative variable is desirable - that is, those circumstances presented by the Subramanian's study: a manageable number of substantively interesting cases. For more on these circumstances, see JACOBY, supra note 30, at 50 ("[D]ot plots are particularly suitable for detection or for the ability to discern individual data points in the graph.").
37. First developed in J.M. CHAMBERS ET AL., GRAPHICAL METHODS FOR DATA ANALYSIS (1983) 20-21, jittering helps to separate points in a univariate scatterplot by adding (or subtracting) a small amount to their value in order to set them off from other data points and thus aid in visud inspection. See also CLEVELAND, supra note 31, at 158 (defining jittering as "adding a small amount of random uniform noise to the data before graphing"); JACOBY, supra note 30, at 31 (describing jittering as the process of "displacing the points somewhat in the direction perpendicular to the variable's scale line");
38. Richard A. Becker & William S. Cleveland, Brushing Scatterplots, 29 TECHNOMETRICS 127, 134 (1987) (explaining that jittering is used to "alleviate overlap).
39. Bivariate scatterplots are a different matter altogether. See infra Part II.C for more on the use of these plots.
40. See JACOBY, supra note 30, at 32 ("When the number of observations is large, there will still be quite a bit of overplotting despite the jittering. Therefore, unidimensional scatterplots remain primarily useful for small data sets.").
41. This metaphor is borrowed from Jacoby. Id. at 47, 50.
42. The mean is "the simple average," the median is "the middle of the distribution of cases," and the standard deviation is a measure of spread or dispersion of the data. Epstein & King, supra note 11, at 25-26; see also AGRESTI & FINLAY, supra note 25, at 45-58 (describing the mean and median).
43. The boxplot was developed decades ago by John W. Tukey, a giant in the field of scientific graphing. See JOHN W. TUKEY, EXPLORATORY DATA ANALYSIS 39-43 (1977).
44. As represented by the boxplot in Figure 4, the interquartile range covers the data points between the 25th and 75th percentiles. In other words, the box covers the middle 50% of the data. The minimum and maximum values are the first and last values of the data when the observations have been sorted from smallest to largest. Outliers, as represented by circles in the Kao boxplot, are data points that are located further than 1.5 interquartile range units from the upper or lower quartile; i.e., a great distance from center of the distribution. See WILLIAM S. CLEVELAND, VISUALIZING DATA 25-26 (1993) and TUKEY, supra note 37, at 39-43, for more information on the boxplot and its components.
45. See Communication I, supra note 6, where we make a series of suggestions for ensuring that the many things that can go wrong with graphing data go right. Because of the detail supplied there, we do not here dwell on design details (e.g., the appropriate plotting symbols, etc.) for this or any other plot.
46. A Lexis search of U.S. law reviews and journals (conducted January 15, 2007) uncovers no relevant results for a search of "violin" within the same paragraph as "plot" or "graph." A Lexis search of the same journals (conducted contemporaneously) turns up 10 articles utilizing kernel density plots in some fashion.
47. See J. L. Hintze & Ray D. Nelson, Violin Plots: A Box Plot-Density Trace Synergism, 52 AM. STATISTICIAN 181 (1998) (developing the violin plot);
48. see also Andrew G. Bunn & Scott J. Goetz, Trends in Satellite-Observed Circumpolar Photosynthetic Activity from 1982 to 2003: The Influence of Seasonality, Cover Type, and Vegetation Density, 10 EARTH INTERACTIONS 1, 10 (2006) (using violin plots to show the distribution of slopes for models of major forest types and categories of low growing vegetation);
49. M. Jorgensen & Dag I.K. Sjoberg, Impact of Experience on Maintenance Skills, 14 J. SOFTWARE MAINTENANCE & EVOLUTION 123, 131 (2002) (noting that "the violin plot highlights the peaks and valleys of a variable's distribution");
50. Thomas R. Steinheimer & Kenwood D. Scoggin, Fate and Movement of Atrazine, Cyanazine, Metolachlor, and Selected Degradation Products in Water Resources of the Deep Loess Hills of Southwestern Iowa, USA, 3 J. ENVTL. MONITORING 126, 128 (2001) (using a violin plot to reveal "an observed distribution of values above the minimum detection limit");
51. Kenneth L. Weiss et al., Clinical Brain MR Imaging Prescriptions in Talairach Space: Technologist and Computer-Driven Methods, 24 AM. J. NEURORADIOLOGY 922, 926 fig.6 (2003) (featuring a violin plot of prescription errors).
52. See Hintze & Nelson, supra note 41, at 181 ("The violin plot . . . synergistically combines the box plot and the density trace (or smoothed histogram) into a single display that reveals structure found within the data.").
53. Kao, supra note 28, at 843.
54. See CLEVELAND, supra note 38, at 8 (arguing that although the histogram is over a century old and is widely used, "maturity and ubiquity do not guarantee the efficacy of a tool"); JACOBY, supra note 30, at 13-17 (noting that the arbitrary designation of bins impacts that shape of the histogram, the assignment of the number of observations in each bin impacts the bumpiness of the distribution, and the very nature of assigning bins means that data are forced to be assigned to one group or another).
55. See, e.g., B. W. SILVERMAN, DENSITY ESTIMATION FOR STATISTICS AND DATA ANALYSIS (1986) (reviewing various approaches to kernel density estimation).
56. See, e.g., supra fig.4, where we emphasize the median of the data.
57. See, e.g., Marina Angel, Criminal Law and Women: Giving the Abused Woman Who Kills a Jury of Her Peers Who Appreciate Trifles, 33 AM. CRIM. L. REV. 229 (1996) (tracing historically the perspective of women in the political and judicial systems);
58. Devon W. Carbado, (E)Racing the Fourth Amendment, 100 MICH. L. REV. 946 (2002) (arguing courts must recognize certain "racial realities" for minorities to receive full Fourth Amendment protection);
59. R. A. Lenhardt, Understanding the Mark: Race, Stigma, and Equality in Context, 79 N.Y.U. L. REV. 803 (2004) (arguing courts should be sensitive to racial stigma);
60. James S. Liebman, Slow Dancing with Death: The Supreme Court and Capital Punishment, 1963-2006, 107 COLUM. L. REV. 1 (2007) (discussing capital punishment's constitutionality);
61. Erik Luna, Race, Crime, and Institutional Design, 66 LAW & CONTEMP. PROBS. 183 (2003) (surveying minority representation in the criminal process);
62. Carolyn B. Ramsey, Intimate Homicide: Gender and Crime Control, 1880-1920, 77 U. COLO. L. REV. 101 (2006) (contrasting the degree of punishment between men and women convicted for killing their significant others);
63. Laura E. Reece, Women's Defenses to Criminal Homicide and the Right to Effective Assistance of Counsel: The Need for Relocation of Difference, 1 UCLA WOMEN'S L.J. 53 (1991) (suggesting criminal defendants' differing perspectives be integrated into the substantive law);
64. Victor L. Streib, Gendering the Death Penalty: Countering Sex Bias in a Masculine Sanctuary, 63 OHIO ST. L.J. 443 (2002) (examining capital punishment through sexual bias analysis).
65. See, e.g., Stephen J. Choi & G. Mitu Gulati, Choosing the Next Supreme Court Justice: An Empirical Ranking of Judge Performance, 78 S. CAL. L. REV. 23 (2004) (creating a "tournament" judging the merits of potential Supreme Court nominees);
66. Thomas J. Miles & Cass R. Sunstein, Do Judges Make Regulatory Policy? An Empirical Investigation of Chevron, 73 U. CHI. L. REV. 823 (2006) (examining policy judgments made by courts post-Chevron);
67. Max Schanzenbach, Racial and Sex Disparities in Prison Sentences: The Effect of District-Level Judicial Demographics, 34 J. LEGAL STUD. 57 (2005) (analyzing the effect of jurisprudential characteristics on federal criminal sentencing);
68. Emerson H. Tiller & Frank B. Cross, A Modest Proposal for Improving American Justice, 99 COLUM. L. REV. 215 (1999) (asserting that acknowledging the partisan component of judging would improve the federal judicial system).
69. See, e.g., Orley Ashenfelter, Theodore Eisenberg & Stewart J. Schwab, Politics and the Judiciary: The Influence of Judicial Background on Case Outcomes, 24 J. LEGAL STUD. 257 (1995) (examining jurisprudential predilection's effect on case outcome);
70. James S. Kakalik et al., Just, Speedy, and Inexpensive? An Evaluation of Judicial Case Management Under the Civil Justice Reform Act, 49 ALA. L. REV. 17 (1997) (analyizing the CJRA's impact);
71. Daniel Kessler, Thomas Meites & Geoffrey Miller, Explaining Deviations from the Fifty-Percent Rule: A Multimodal Approach to the Selection of Cases for Litigation, 25 J. LEGAL STUD. 233 (1996) (arguing that more reliable results are provided by looking beyond the party's simple divergent expectations in the selection of a case for litigation);
72. Joel Waldfogel, Reconciling Asymmetric Information and Divergent Expectations Theories of Litigation, 41 J.L. & ECON. 451 (1998) (concluding that pretrial adjudication and settlement cause plaintiff win rates that tend toward central, rather than extreme, results).
73. Take Epstein and Segal's study of Senate confirmations of Supreme Court justices, in which they present a table of the relationship between a nominee's qualifications and ideology and the number of votes he received in the Senate. LEE EPSTEIN & JEFFREY A. SEGAL, ADVICE AND CONSENT: THE POLITICS OF JUDICIAL APPOINTMENTS 114 fig.4 (2005). The table, reproduced below, shows both the percentage of votes cast in favor of the nominee (the top number) and the total number of votes in that category (the lower number). Epstein and Segal use the table not to reach a causal inference about the effect of qualifications/ideology on votes - they realized that many other factors influence votes - but rather to communicate to readers the plausibility of such a relationship. With that demonstration in hand, they eventually moved toward a more sophisticated analysis containing a variable for qualifications, along with many others, designed to draw causal inferences. (Table Presented)
74. Samuel R. Gross & Katherine Y. Barnes, Road Work: Racial Profiling and Drug Interdiction on the Highway, 101 MICH. L. REV. 651 (2002).
75. Douglas Cumming & Jeffrey MacIntosh, Boom, Bust, and Litigation in Venture Capital Finance, 40 WILLAMETTE L. REV. 867 (2004).
76. These data appeared id. at 885 tbl.3. The table was reproduced by the authors from Tim Loughran & Jay R. Ritter, Why has IPO Underpricing Changed Over Time? (2003) (unpublished manuscript, on file with author).
77. Ann Juliano & Stewart J. Schwab, The Sweep of Sexual Harassment Cases, 86 CORNELL L. REV. 548 (2001).
78. Lynn M. LoPucki & Sara D. Kalin, The Failure of Public Company Bankruptcies in Delaware and New York: Empirical Evidence of a "Race to the Bottom", 54 VAND. L. REV. 231 (2001).
79. Gross & Barnes, supra note 51.
80. Id.
81. Mosaic plots were first developed in J.A. Hartigan & B. Kleiner, Mosaics for Contingency Tables, in COMPUTER SCIENCE AND STATISTICS: PROCEEDINGS OF THE 13TH SYMPOSIUM ON THE INTERFACE (W.F. Eddy ed., 1981)
82. and were further refined in Michael Friendly, Mosaic Displays for Multi-Way Contingency Tables, 89 J. AM. STAT. ASS'N 190 (1994).
83. In the physical, biological, and social sciences, the predominant graph is the scatterplot, appearing in its many variations; indeed, scholars have estimated that 75% of the graphs used in the sciences are scatterplots. See Ian Spence & Robert F. Garrison, A Remarkable Scatterplot, 47 AM. STATISTICIAN 12 (1993). Analysts often use simple scatterplots before analyzing their data, and the insights gained may stimulate the production of more complicated variations or may guide the choice of a model.
84. Like so many other visualization tools, bivariate scatterplots can go awry. To avoid unnecessary problems, Cleveland recommends the use of visually prominent plotting symbols, outward facing tick marks, and, where necessary, jittering, along with the avoidance of grid lines. CLEVELAND, supra note 31, at 158. See also JACOBY, supra note 30, at 52-56; Communication I, supra note 6, for a recommendation of similar approaches.
85. As the smoothness parameter α increases, so too increases the smoothness of the loess curve.
86. See e.g., CLEVELAND, supra note 31, at 170.
87. Also note our use of dot plots. As was the case for our earlier example regarding Subramanian's article, the number of measurements in Juliano and Schwab's study is small enough and the circuits well known enough that labeling each serves a substantive purpose.
88. Juliano & Schwab, supra note 54, at 574.
89. Harvard Law Review's practice of publishing Supreme Court statistics from the preceding term began in 1949. The Supreme Court, 2004 Term: The Statistics, 119 HARV. L. REV. 415, 415 (2005)
90. (citing The Supreme Court, 1948 Term - The Business of the Court, 63 HARV. L. REV. 119 (1949)).
91. Kao, supra note 28.
92. Gross & Barnes, supra note 51.
93. Lee Epstein, Jeffrey A. Segal, & Harold J. Spaeth, The Norm of Consensus on the U.S. Supreme Court, 45 AM. J. POL. SCI. 362 (2001);
94. Mark J. Roe, Political Preconditions to Separating Ownership from Corporate Control, 53 STAN. L. REV. 539 (2000);
95. Nancy C. Staudt, Modeling Standing, 79 VA. L. REV. 612 (2004).
96. This model follows from work by EPSTEIN & SEGAL, supra note 50; Charles M. Cameron, Albert D. Cover, & Jeffrey A. Segal, Senate Voting on Supreme Court Nominees: A Neoinstitutional Model, 84 AM. POL. SCI. REV. 525, 530 tbl.2 (1990);
97. Lee Epstein et al., The Changing Dynamics of Senate Voting on Supreme Court Nominees, 68 J. POL. 296 (2006).
98. Previous work has assessed this by analyzing the content of newspaper editorials written from the time of the nomination until the vote by the Senate and then deriving a qualifications score for each nominee. These Segal-Cover scores range from 0 (most qualified) to 1 (least qualified). See EPSTEIN & SEGAL, supra note 50, at 114 fig.4; Cameron, Cover & Segal, supra note 69, at 530 tbl.2..
99. Following Epstein et al., supra note 69, at 299, we measure the ideological distance between a senator and a nominee via the generation of Common Space scores for each nominee. More detail on the creation of this measure is available in Epstein et al., supra note 69, so suffice it to say here that these scores are generated by "bridging" candidates nominated by presidents whose party holds a majority of Senate seats. These "bridged" nominees receive the Segal-Cover scores, see discussion supra note 70, of their appointing president, and those scores, along with their Segal-Cover scores, permit a linear transformation. The result is that Common Space scores can be created for all nominees based on their Segal-Cover scores.
100. We could have also conducted our estimation with a logistic regression model. Like probit, logistic ("logit") regression is utilized when the dependent variable is dichotomous. The structural models of logit and probit are different, but they are related to each other in such a way that logit coefficients, when statistically significant, are approximately 1.7 times larger than probit coefficients, making the choice between the two models 'largely one of convenience and convention." LONG, supra note 27, at 47-49, 83.
101. The recent work by Gelman et al. and King et al. has been particularly influential in establishing the need to convey uncertainty and providing the equipment to do so. Gelman, supra note 10; King et al., supra note 8. We follow their lead here.
102. For example, for regression models, knowing the scale of the variable of interest allows the reader to contextualize the regression coefficient. It is necessary to know the scale of the dependent and independent variable for the reader to understand what β = -2.217 actually means. Additionally, outside of the linear regression context, such substantive interpretations of coefficients are extremely difficult because they require the reader to make complex calculations that depend on the values of the independent variables. Thus, why ask the reader to do these calculations for a multivariate statistical analysis when conveying results in an easy-to-consume manner with figures is straight forward. We emphasize this point in the text to follow.
103. Without the presence of a statistically significant relationship between the coefficient and the dependent variable, there is no reason to test the substantive effect of a variable.
104. The arbitrary setting of a p-value simply means that a researcher wants to have a certain level of confidence (known as the α-level) in the accuracy of the estimated relationship of the variables. Providing differing levels of p-values merely amounts to having different levels of confidence in the statistical relationship. It would certainly be incorrect to say that, in Table 3, the coefficient for the president controlling the Senate is less important of a finding than the others simply because it is only statistically significant at the .05 level as opposed to the .01 level. See AGRESTI & FINLAY, supra note 25, for more information on p-values, α-levels, and the proper use and interpretation of each.
105. Because some statistical techniques are inappropriate for studies with a small number of observations, reporting the N provides a check. For example, the properties of maximum likelihood estimation models, such as probit, do not hold when sample sizes are too small. See LONG, supra note 27, at 54 ("It is risky to use [maximum likelihood] with sample smaller than 100, while samples over 500 seem adequate. These values should be raised depending on characteristics of the model and the data.")
106. Reporting the proportional reduction in error from the newly estimated model provides information about the utility of the researcher's chosen model. With the assistance of statistical software, computing this reduction in error is also simple. For example, in Stata, after the model has been estimated, the user need simply install and use the "pre" command to yield this reduction in error. Within the software, this is computed by finding the errors when simple guessing is employed and then finding the number of errors after the model has been estimated. The final proportional reduction in error is computed by subtracting the number of errors in the model from the number of guessing errors and dividing that number by the number of guessing errors.
107. For a similar graphical display of statistical results, see supra Figure 1.
108. See Communication I, supra note 6, at 1831-31, where we make a similar point. See generally King et al., supra note 8 (making similar arguments); Gelman, supra note 10 (same). Our inspiration for this section follows from their work, especially King et al.
109. See, e.g., WERL, On Tournaments for Appointing Great Justices to the U.S. Supreme Court, 78 S. CAL. L. REV. 157, 157 (2005) (commenting on debate over "whether the current system [for appointing justices] does any better than [a] tournament in predicting skills that include the ability to compromise and negotiate, a talent for shaping national policy, and a gift for choosing among the thousands of petitions for certiorari filed with the Court").
110. This is the predicted probability of a senator casting a yea vote for a Supreme Court nomination when a nominee's (lack of) qualifications are held at their mean, when the president is weak (i.e., his party does not control the Senate), and when the voting senator and the president are of different political parties, changing the ideological distance between the nominee and the voting senator from distant to close.
111. King et al., supra note 8, at 359-60, makes this point, and we adopt it here.
112. CLARIFY can be found at Michael Tomz et al., CLARIFY: Software for Interpreting and Presenting Statistical Results, June 1, 2001, http://gking.harvard.edu/clarify/docs/clarify.html.
113. SPost can be found at J. Scott Long, SPost: Post-estimation with Stata, Mar. 1, 2007, http://www.indiana.edu/~jslsoc/spost.htm.
114. Zelig can be found at Kosuke Imai, Zelig: Everyone's Statistical Software, http://gking.harvard.edu/zelig/ (last visited Mar. 14, 2007).
115. Communication I, supra note 6, at
116. See Epstein & King, supra note 11, at 132 (recommending law reviews require documentation and archiving of empirical data that would enable replication).
117. Professor Gary King at Harvard offers the following, easy-to-implement replication policy for journals: Authors submitting quantitative papers to this journal for review must address the issue of data availability and replication in their first footnote. Authors are ordinarily expected to indicate in this footnote in which public archive they will deposit the information necessary to replicate their numerical results, and the date when it will be submitted. The information deposited should include items such as original data, specialized computer programs, lists of computer program recodes, extracts of existing data files, and an explanatory file that describes what is included and explains how to reproduce the exact numerical results in the published work. Authors may find the Publication-Related Archive of the Inter-university Consortium for Political and Social Research (ICPSR) a convenient place to deposit their data. Statements explaining the inappropriateness of sharing data for a specific work (or of the necessity for periods of embargo past the publication date) may fulfill the requirement . . . . Authors of works relying upon qualitative data should submit a comparable footnote that would facilitate replication where feasible. As always, authors are advised to remove information from their datasets that must remain confidential, such as the names of survey respondents. Gary King, An Example Replication Policy for Journals, http://gking.harvard.edu/repl.shtml (last visited Feb. 18, 2007). We should also note that a replication standard such as this is in the best interest of authors, as it requires them to give a little extra effort at the time of submission to centrally organize the data and statistical analysis (something that we often wish we would have done better for our own projects).
118. The ever-present "paper on file with the journal" or "with the author."