Meta-analysis: A primer for legal scholars

Temple Law Review

Volumn 80, Issue 1, 2007, Pages 201-244

  Blumenthal, Jeremy A ^a  

^a National Institute of Disability Rehabilitation and Research

Author keywords

[No Author keywords available]

Indexed keywords

EID: 34547990152     PISSN: 08998086     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Article

References (229)

1. An empirical approach to legal scholarship has been advocated at times throughout the twentieth century, with varying success. See Gregory Mitchell, Empirical Legal Scholarship as Scientific Dialogue, 83 N.C. L. REV. 167, 168-69 (2005) (noting legal scholars' calls for more empirical legal research).
2. See generally JOHN HENRY SCHLEGEL, AMERICAN LEGAL REALISM AND EMPIRICAL SOCIAL SCIENCE (1995) (providing detailed review of early efforts to develop empirical legal work);
3. Jeremy A. Blumenthal, Law and Social Science in the Twenty-First Century, 12 S. CAL. INTERDISC. L.J. 1, 7-22 (2002) (reviewing historical efforts by legal scholars and social scientists to empirically examine aspects of the legal system). Empirical legal scholars recently instituted a journal and a "blog" devoted to the publication and the discussion of Empirical Legal Studies.
4. See generally J. EMPIRICAL LEGAL STUD., available at http://www.blackwellpubushing.com/ journal.asp?ref=1740- 1453&site=1 (last visited Apr. 15, 2007) (journal); Empirical Legal Studies, http://www.elsblog.org/about.html (last visited Apr. 15, 2007) (blog).
5. See generally Lee Epstein & Gary King, The Rules of Inference, 69 U. CHI. L. REV. 1 (2002) (outlining numerous concerns with existing body of empirical legal literature).
6. See Mitchell, supra note 1, at 187-88 (identifying value of meta-analysis for legal scholarship). As meta-analyst experts John Hunter and Frank Schmidt have pointed out: For decades policymakers seeking factual foundations for policy have looked to psychological and social science research. Until recently, they have been disappointed to find research literatures that were conflicting and contradictory. As the number of studies on each particular question became larger and larger, this situation became increasingly frustrating and intolerable. These problems stemmed from reliance on defective procedures for achieving cumulative knowledge: the statistical significance test in individual primary studies in combination with the narrative subjective review of research literatures. Meta-analysis principles have now correctly diagnosed this problem and, more important, have provided the solution. John E. Hunter & Frank L. Schmidt, Cumulative Research Knowledge and Social Policy Formulation: The Critical Role of Meta-Analysis, 2 PSYCHOL. PUB. POL'Y & L. 324, 342-43 (1996).
7. Dan Orr & Chris Guthrie, Anchoring, Information, Expertise, and Negotiation: New Insights from Meta-Analysis, 21 OHIO ST. J. ON DISP. RESOL. 597, 612 (2006) ("Legal scholars routinely cite meta-analyses to support empirical claims they want to make about the legal world, but few law reviews have actually published original meta-analyses." (footnote omitted)).
8. In part, then, the Article helps address one of Lee Epstein and Gary King's concerns, the lack of articles in the legal literature "devoted exclusively to solving methodological problems unique to legal scholarship." Epstein & King, supra note 2, at 6 n.19. The methodological "problems" that metaanalysis addresses are hardly unique to empirical legal scholarship, but they are common enough in that scholarship that the procedure will be quite useful.
9. See generally, e.g., Louise F. Fitzgerald & Alayne J. Ormerod, Perceptions of Sexual Harassment: The Influence of Gender and Academic Context, 15 PSYCHOL. WOMEN Q. 281 (1991) (concluding that combination of severity or explicitness of incident and gender of perceiver had a bearing on whether perceiver would consider incident harassment);
10. Tricia S. Jones & Martin S. Remland, Sources of Variability in Perceptions of and Responses to Sexual Harassment, 27 SEX ROLES 121 (1992) (concluding that notion of harassing behavior varies based on nature of behavior, gender of observer, and gender of target of behavior);
11. Natalie J. Malovich & Jayne E. Stake, Sexual Harassment on Campus: Individual Differences in Attitudes and Beliefs, 14 PSYCHOL. WOMEN Q. 63 (1990) (concluding that self-esteem, sex-role attitudes, and gender all had bearing on perception of harassment).
12. See William J. Bowers et al., Death Sentencing in Black and White: An Empirical Analysis of the Role of Jurors' Race and Jury Racial Composition, 3 U. PA. J. CONST. L. 171 (2001) (examining whether jury's racial composition influences sentencing phase of death penalty cases and concluding that chance of defendant receiving death sentence is increased when jury is predominantly white and defendant is black).
13. See generally, e.g., Craig Haney et al., Deciding to Take a Life: Capital Juries, Sentencing Instructions, and the Jurisprudence of Death, 50 J. SOC. ISSUES 149 (1994) (concluding that jury death penalty instructions not only failed to guide juries' decisions but distorted decision-making process);
14. James Luginbuhl, Comprehension of Judges' Instructions in the Penalty Phase of a Capital Trial: Focus on Mitigating Circumstances, 16 LAW & HUM. BEHAV. 203 (1992) (concluding that old version of jury death penalty instructions in North Carolina resulted in confusion among jurors as to significance of mitigating circumstances).
15. Three reviews of this literature provide examples of the many studies examining this question: Ronald Mazzella & Alan Feingold, The Effects of Physical Attractiveness, Race, Socioeconomic Status, and Gender of Defendants and Victims on Judgments of Mock Jurors: A Meta-Analysis, 24 J. APPLIED SOC. PSYCHOL. 1315 (1994);
16. Tara L. Mitchell et al., Racial Bias in Mock Juror Decision-Making: A Meta-Analytic Review of Defendant Treatment, 29 LAW & HUM. BEHAV. 621 (2005);
17. Laura T. Sweeney & Craig Haney, The Influence of Race on Sentencing: A Meta-Analytic Review of Experimental Studies, 10 BEHAV. SCI. & L. 179 (1992).
18. See generally, e.g., Max M. Schanzenbach & Emerson H. Tiller, Strategic Judging Under the United States Sentencing Guidelines: Positive Political Theory and Evidence, 23 J.L. ECON. & ORG. 24 (2007) (concluding that political orientation impacted criminal sentencing in cases where judges departed from federal sentencing guidelines);
19. Cass R. Sunstein et al., Ideological Voting on Federal Courts of Appeals: A Preliminary Investigation, 90 VA. L. REV. 301 (2004) (concluding that judicial votes are influenced by individual judge's political affiliation as well as by judicial panel's political composition).
20. See David B. Wilson, Meta-Analytic Methods for Criminology, 578 ANNALS AM. ACAD. POL. & SOC. SCI. 71, 72 (2001) ("averaging across studies is analogous to averaging across individuals within a single study").
21. Hines v. Consol. Rail Corp., 926 F.2d 262, 273 n.7 (3d Cir. 1991) ("Meta-analysis involves pooling data from a number of different epidemiological studies (in order to enhance sample size) and comparing the results of those pooled data with the results produced by each study individually."); Lewis v. City of Chi., No. 98 C 5596, 2005 WL 693618, at *13 n.8 (N.D. III. Mar. 22, 2005) ("A meta-analysis is a statistical analysis of the results of a collection of individual studies to integrate and summarize their results.");
22. Black v. Rhone-Poulenc, Inc., 19 F. Supp. 2d 592, 604 (S.D.W. Va. 1998) ("In short, a meta-analysis simply pools all of the data from many studies and treats them as one mega-study.");
23. United States v. Nguyen, 793 F. Supp. 497, 512 n.23 (D.N.J. 1992) (quoting expert witness Professor Steven Penrod defining meta-analysis as "accepted method of analysis" that "combin[es] the results of independent studies in order to arrive at a general conclusion"); Gene V. Glass, Primary,
24. Secondary and Meta-Analysis of Research, 5 EDUC. RES. 3, 3 (1976) (stating meta-analysis is "the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings");
25. Raymond J.G.M. Florax et al., Meta-analysis: A Tool for Upgrading Inputs of Macroeconomic Policy Models 1 (Apr. 1, 2002) (unpublished manuscript, available at http://www.tinbergen.nl/discussionpapers/02041.pdf) (stating that meta-analysis is used to synthesize and summarize the results previously reported in literature on area of research). Of the cases' descriptions, the definitions in Nguyen and Lewis are, in fact, the more accurate - meta-analysis synthesizes the results of studies, not their raw data. Statistical problems may arise when the raw data of studies are pooled, rather than their results.
26. See, e.g., ROBERT ROSENTHAL, META-ANALYTIC PROCEDURES FOR SOCIAL RESEARCH 99-101 (rev. ed. 1991) (explaining that pooling data from various studies can yield results paradoxically outside the range of results of any individual study).
27. Blumenthal, supra note 1, at 40 ("This aggregation and averaging can lead to a more robust finding from which to infer policy than either an abbreviated selection of research findings or pure assumption or anecdote." (footnote omitted)).
28. See Frank L. Schmidt, Statistical Significance Testing and Cumulative Knowledge in Psychology: Implications for Training of Researchers, 1 PSYCHOL. METHODS 115, 123 (1996) ("Applications of meta-analysis to accumulated research literatures have generally shown that research findings are not nearly as conflicting as we had thought and that useful general conclusions can be drawn from past research.").
29. See id. at 122-23 for an explanation of the benefits of meta-analysis when confronted with seemingly conflicting findings based on moderator variables.
30. See Harris Cooper & Larry V. Hedges, Research Synthesis as a Scientific Enterprise, in THE HANDBOOK OF RESEARCH SYNTHESIS 3, 4-5 (Harris Cooper & Larry V. Hedges eds., 1994) (citations omitted) (noting common use of literature reviews as means of keeping up with developing empirical research);
31. Richard Lempert, "Between Cup and Lip": Social Science Influences on Law and Policy, 10 LAW & POL'Y 167, 175 (1988) (suggesting that sources synthesizing studies are more useful to policy makers than single studies because they provide better overview of that area of research);
32. Sarah H. Ramsey & Robert F. Kelly, Social Science Knowledge in Family Law Cases: Judicial Gate-Keeping in the Daubert Era, 59 U. MIAMI L. REV. 1, 79 (2004) (suggesting that reviews of literature on area of study will allow judges to be better informed when confronting and assessing specific findings of study presented in court). Unsurprisingly, literature reviews have thus long been among the most commonly cited articles, at least in the social sciences. Cooper & Hedges, supra, at 4.
33. Lempert, supra note 16, at 175;
34. Wilson, supra note 11, at 85.
35. Gregory Mitchell, Beyond Fireside Inductions, 32 FLA. ST. U. L. REV. 315, 318 (2005).
36. See, e.g., Lempert, supra note 16, at 176 (explaining that comparison of various studies, conducted differently, and sometimes reaching seemingly conflicting results, can produce explanations for inconsistencies and provide broader understanding of an area of research);
37. Thomas O. McGarity, Proposal for Linking Culpability and Causation to Ensure Corporate Accountability for Toxic Risks, 26 WM. & MARY ENVTL. L. & POL'Y REV. 1, 65 (2001) (citation omitted) (noting that meta-analysis can provide better understanding of a particular area of research as a whole than results from individual study).
38. See infra Part II.D for a discussion of the use of meta-analysis to examine moderator variables impacting individual studies.
39. E.g., JOHN E. HUNTER ET AL., META-ANALYSIS: CUMULATING RESEARCH FINDINGS ACROSS STUDDIES 26 (1982) (criticizing traditional review techniques for failing to integrate seemingly conflicting studies into review of area of research);
40. John P.A. Ioannidis & Joseph Lau, Systematic Review of Medical Evidence, 12 J.L. & POL'Y 509, 534 (2004) (suggesting that quantitative methods of integrating data from many studies in particular research topic are necessary for more total appraisal of that topic).
41. See GENE V. GLASS ET AL., META-ANALYSIS IN SOCIAL RESEARCH 13 (1981) (listing various problems that arise when reviewers selectively review studies but fail to explain or assess the methods of their selections);
42. FREDRIC M. WOLF, META-ANALYSIS: QUANTITATIVE METHODS FOR RESEARCH SYNTHESIS 10 (1986) (noting that uterature reviews are often subject to biases of reviewers).
43. See Hunter & Schmidt, supra note 3, at 330 ("Relying on various personal and subjective theories and beliefs about methodological quality, reviewers often exclude[] all but a small number of studies as methodologically inadequate and then base[] their reviews on only the remaining few studies." (internal quotation marks omitted) (emphasis added)).
44. GLASS ET AL., supra note 22, at 18.
45. Id. at 22.
46. Id. at 13.
47. Harris M. Cooper & Robert Rosenthal, Statisical Versus Traditional Procedures for Summarizing Research Findings, 87 PSYCHOL. BULL. 442 (1980).
48. Id. at 448.
49. Id. The two groups' estimates of the size and strength of the relationship differed as well. Id.
50. Compare Doris L. Redfield & Elaine W. Rousseau, A Meta-analysis of Experimental Research on Teacher Questioning Behavior, 51 REV. EDUC. RES. 237 (1981) (reporting meta-analytic study finding significant and "positive effect on student achievement" when higher cognitive questioning was used),
51. with Philip H. Winne, Experiments Relating Teachers' Use of Higher Cognitive Questions to Student Achievement, 49 REV. EDUC. RES. 13 (1979) (reporting narrative review finding little beneficial pedagogical effect of teachers' asking students "higher cognitive" questions as opposed to "fact" questions).
52. R. Rosenthal & M.R. DiMatteo, Meta-Analysis: Recent Developments in Quantitative Methods for Literature Reviews, 52 ANN. REV. PSYCHOL. 59, 63 (2001) ("Meta-analysis prevents our reliance on the significance test of any one finding as a measure of its value");
53. Robert Rosenthal, Writing Meta-Analytic Reviews, 118 PSYCHOL. BULL. 183, 185 (1995) ("Effect size estimates are the meta-analytic coin of the realm.").
54. In a well-known critical discussion of such testing, Jacob Cohen elaborated on the acronym saying, "I resisted the temptation to call it statistical hypothesis inference testing." Jacob Cohen, The Earth is Round (p < .05), 49 AM. PSYCHOL. 997, 997 (1994).
55. Jeremy A. Blumenthal, Does Mood Influence Moral Judgment? An Empirical Test with Legal and Policy Implications, 29 LAW & PSYCHOL. REV. 1, 9 n.54 (2005).
56. Id.; see also DAVID C. HOWELL, STATISTICAL METHODS FOR PSYCHOLOGY 88 (3d ed. 1992) (explaining significance level test, which discards null hypotheses with probabihties less than or equal to .05).
57. HUNTER ET AL., supra note 21, at 20.
58. E.g., ROSENTHAL, supra note 12, at 14 (explaining that test of significance is product of effect size - or strength of relationship between variables - and study size);
59. Rosenthal & DiMatteo, supra note 31, at 63 (stating that significance of effect size is determined by size of study).
60. The analogy is inexact, but the broader point should be clear.
61. Blumenthal, supra note 33, at 9 n.54 (emphasizing importance of not solely relying on .05 p-values);
62. see also Vuyanich v. Republic Nat'l Bank of Dallas, 505 F. Supp. 224, 272 (N.D. Tex. 1980) (noting problems with arbitrarily excluding results with p-value greater than .05 when such studies may still be useful), vacated on other grounds, 723 F.2d 1195 (5th Cir. 1984);
63. Robert Rosenthal & Donald B. Rubin, Comparing Significance Levels of Independent Studies, 86 PSYCHOL. BULL. 1165 (1979) (noting that in reality, there is little difference between p-value of .05 and .06);
64. cf. Robert Rosenthal & John Gaito, The Interpretation of Levels of Significance by Psychological Researchers, 55 J. PSYCHOL. 33 (1963) (empirically documenting overreliance among empirical scientists on .05 p-levels);
65. Robert Rosenthal & John Gaito, Further Evidence for the Cliff Effect in the Interpretation of Levels of Significance, 15 PSYCHOL. REP. 570 (1964) (documenting same).
66. See Allison v. McGhan Med. Corp., 184 F.3d 1300, 1315 (11th Cir. 1999) (upholding district court's exclusion of expert's meta-analysis where the court found the "study unreliable because it was a re-analysis of other studies that had found no statistical correlation"). But see In re Paoli R.R. Yard PCB Litig., 916 F.2d 829, 856-58 (3d Cir. 1990) (reversing district court admissibility ruling that synthesizing nonsignificant studies could not achieve statistical significance).
67. Rosenthal & DiMatteo, supra note 31, at 63.
68. Id.; see also GLASS ET AL., supra note 22, at 220-21 (explaining that many studies with different weaknesses, when pulled together, can yield strong results).
69. Robert Rosenthal & Peter David Blanek, Science and Ethics in Conducting, Analyzing, and Reporting Social Science Research: Implications for Social Scientists, Judges, and Lawyers, 68 IND. L.J. 1209, 1220 (1993).
70. See, e.g., DeLuca v. Merrell Dow Pharm., Inc., 911 F.2d 941, 947 (3d Cir. 1990) (citation omitted) (stating that "if P is greater than 5% the relationship is rejected as insignificant");
71. Coates v. Johnson & Johnson, 756 F.2d 524, 537 n.13 (7th Cir. 1985) ("A P value below .05 is generally considered to be statistically significant.");
72. Presseisen v. Swarthmore Coll., 442 F. Supp. 593, 617 (E.D. Pa. 1977) ("Since a small value of P, i.e., less than .05 (for example, .04, .03, etc.), indicates an effect is statistically significant; and since all the P values in [question] are not less than .05, the average differences mentioned above could be attributable to chance alone."); Bloomquist v. Wapello City, 500 N.W.2d 1, 5 (Iowa 1993) (quoting Deluca, 911 F.2d at 947) (suggesting that relationship is regarded as insignificant for p-values greater than .05);
73. Celestial S.D. Cassman & Lisa R. Pruitt, A Kinder, Gentler Law School? Race, Ethnicity, Gender, and Legal Education at King Hall, 38 U.C. DAVIS L. REV. 1209, 1242 & n.142 (dividing study results between those with statistically significant p-values of less than .05 and those with no detectable difference, or p-values of more than .05);
74. Fred O. Smith, Jr., Note, Gendered Justice: Do Male and Female Judges Rule Differently on Questions of Gay Rights?, 57 STAN. L. REV. 2087, 2113-14 (2005) (noting, without additional information, that p-value was "out of the bounds of statistical significance at a .05 level" and that there was "not necessarily" a difference in groups being examined).
75. Richard D. Friedman, The Death and Transfiguration of Frye, 34 JURIMETRICS J. 133, 148 (1994) ("As compared to traditional methods of significance testing, meta-analysis may more closely approach our evidentiary system's willingness to allow an inference to be drawn from various bits of information, none of which independently supports the inference.").
76. Blumenthal, supra note 1, at 42 (discussing different types of moderator variables that could be considered under meta-analytic approach).
77. Mark W. Lipsey, Those Confounded Moderators in Meta-Analysis: Good, Bad, and Ugly, 587 ANNALS AM. ACAD. POL. & SOC. SCI. 69, 69 (2003).
78. See generally Bowers et al., supra note 7, at 181-89 (collecting examples of such research).
79. E.g., Sweeney & Haney, supra note 9, at 179 (noting that methodological differences across numerous studies accounted for stronger results in some studies than others).
80. Id.
81. Id at 190.
82. Jeremy A. Blumenthal, The Reasonable Woman Standard: A Meta-Analytic Review of Gender Differences in Perceptions of Sexual Harassment, 22 LAW & HUM. BEHAV. 33, 44 tbl.7 (1998).
83. Id. at 46.
84. E.g., Blumenthal, supra note 1, at 43-44 (asserting that benefits of meta-analysis comport with goals of judges and social scientists in yielding practical legal applications);
85. Ioannidis & Lau, supra note 21, at 535 (noting that meta-analytical analysis can aid judges and lawyers in understanding expert testimony).
86. 476 U.S. 162 (1986).
87. 522 U.S. 136 (1997).
88. Lockhart, 476 U.S. at 169-73.
89. Id. Other courts have reacted similarly. See Free v. Peters, 12 F.3d 700, 705-06 (7th Cir. 1993) (criticizing studies on juror comprehension of sentencing instructions); State v. Deck, 994 S.W.2d 527, 542-43 (Mo. 1999) (en banc) (same), aff'd in part and rev'd in part, 68 S.W.3d 418 (Mo. 2002), rev'd on other grounds, 544 U.S. 622 (2005);
90. cf. Green v. United States, 126 S. Ct. 497, 497 (2005) (denying certiorari in federal death penalty case in which circuit court reversed trial judge's order, based on empirical evidence of death-qualified individuals' bias, to bifurcate guilt and sentencing jury). Then-Chief Justice Rehnquist's disparagement of the literature has since been criticized in turn by a number of scholars.
91. E.g., Phoebe C. Ellsworth, Unpleasant Facts: The Supreme Court's Response to Empirical Research on Capital Punishment, in CHALLENGING CAPITAL PUNISHMENT: LEGAL AND SOCIAL SCIENCE APPROACHES 177, 196-97 (Kenneth C. Haas & James A. Inciardi eds., 1988) (arguing that due to unanimity of results of various studies, Rehnquist's perceived flaws in methodology should not have been fatal flaws);
92. J. Alexander Tanford, The Limits of a Scientific Jurisprudence: The Supreme Court and Psychology, 66 IND. L.J. 137, 145-47 (1990) (suggesting that Rehnquist, with no training in social sciences, should have given more credence to findings of experimental psychologists);
93. William C. Thompson, Death Qualification After Wainwright v. Witt and Lockhart v. McCree, 13 LAW & HUM. BEHAV. 185, 195-98 (1989) (criticizing Rehnquist's dismissal of studies he saw as less than definitive, even though they could have been nevertheless informative).
94. Joseph W. Filkins et al., An Evaluation of the Biasing Effects of Death Qualification: A Meta-Analytic/Computer Simulation Approach, in THEORY AND RESEARCH ON SMALL GROUPS 153, 171 (R. Scott Tindale et al. eds., 1998). Filkins and colleagues qualified their statement by noting that to some extent the Lockhart Court was correct; based on their computer simulations of jury decision making, death-qualified juries (i.e., rather than jurors) are less likely to convict than had been thought, though still slightly more so than non-death-qualified juries. Id. at 171-72.
95. Joiner, 522 U.S. at 144-45.
96. Id. at 153 (Stevens, J., concurring in part and dissenting in part).
97. Cf. Blumenthal, supra note 1, at 41 (suggesting that Stevens's approach to Joiner studies might have been able to yield more useful conclusions than Rehnquist's approach). Or it might not. The majority and Justice Stevens both dismissed the underlying studies because they did not reach conventional statistical significance levels.
98. See Joiner, 522 U.S. at 154 & n.8 (Stevens, J., concurring in part and dissenting in part) (conceding that even broader meta-analysis approach yielded statistically insignificant results in this case). As noted above, however, this is misguided; a meta-analysis might have shown that overall, the findings across all studies did reach such levels.
99. E.g., Stephen J. Deery & Roderick D. Iverson, Labor-Management Cooperation: Antecedents and Impact on Organizational Performance, 58 INDUS. & LAB. REL. REV. 588, 605 (2005) ("It is important, however, not to overstate the practical implications of our findings. We could explain only a relatively small proportion of the variance in the performance measures.").
100. Cf. Jerome McCristal Culp, Jr., Toward a Black Legal Scholarship: Race and Original Understandings, 1991 DUKE L.J. 39, 87 n.142 ("Statistically, a model with a very small R-squared may be a better model than one which purports to explain all of the variance."). For further discussion about presenting meta-analytic and other findings in a way that demonstrates the practical importance of even small effects, see infra Part III.D.
101. E.g., Wilson, supra note 11, at 85 (noting that meta-analysis can be used to identify areas of study that are well researched and areas where more research is needed).
102. Blumenthal, supra note 1, at 45.
103. Lempert, supra note 16, at 176-77 (cautioning policy and legal decision makers against overreliance on single-study analyses).
104. See MORTON HUNT, HOW SCIENCE TAKES STOCK: THE STORY OF META-ANALYSIS 42 (1997) (criticizing meta-analysis for integrating studies of poor as well as sound design).
105. See Rosenthal & DiMatteo, supra note 31, at 66-67 (acknowledging '"garbage in and garbage out' issue" and suggesting weighting technique to accommodate quality variance among studies.
106. It may be too much to say that "bad" studies will be defined as those of our "enemies," see ROSENTHAL, supra note 12, at 130 (citation omitted), but subjectivity in such evaluation will be more common in narrative reviews.
107. Jessica Gurevitch & Larry V. Hedges, Statstical Issues in Ecological Meta-Analyses, 80 ECOLOGY 1142, 1146 (1999) (stating that developing methods for integrating poorly reported data is more desirable than ignoring those data altogether). The same is true for the synthesis of data of varying quahty. See Sarah H. Ramsey & Robert F. Kelly, Using Social Science Research in Family Law Analysis and Formation: Problems and Prospects, 3 S. CAL. INTERDISC. L.J. 631, 680 (1994) (noting that many advocates believe meta-analysis can make good use of methodologically poor studies and maintaining that those studies should not always be excluded from reviews).
108. E.g., WOLF, supra note 22, at 15 (explaining meta-analysis' ability to empirically handle quality of research design across various tests).
109. See also supra note 25 and accompanying text for a discussion of assessing study quality and incorporating those assessments into a meta-analysis.
110. ROSENTHAL, supra note 12, at 130.
111. Indeed, there is evidence that in the typical meta-analysis, there is little such relationship. See GLASS ET AL., supra note 22, at 226 (stating that in reviews using large enough number of cases, differences in effects of high-quality and low-quality studies were small). Thus, dropping studies based on perceived methodological deficiencies wastes data. Cf. Nancy G. Berman & Robert A. Parker, Meta-Analysis: Neither Quick Nor Easy, 2 BMC MED. RES. METHODOLOGY 10, 17-18 (2002) ("[G]iven the effort that goes into identifying and evaluating papers, ignoring or rejecting valuable information is wasteful.").
112. E.g., McNeil-P.C.C., Inc. v. Bristol-Myers Squibb Co., 938 F.2d 1544, 1547 (2d Cir. 1991) ("Moreover, evidence at trial showed that only seven out of the thirty studies incorporated into the meta-analysis concerned acetaminophen. [The district court judge] therefore accorded the metaanalysis little or no weight.").
113. Gene V. Glass, In Defense of Generalization, 1 BEHAV. & BRAIN SCI. 394, 395 (1978).
114. Wilson, supra note 11, at 73.
115. ROSENTHAL, supra note 12, at 129.
116. E.g., WOLF, supra note 22, at 15 (asserting that even problematic differences in methodology can be examined empirically).
117. See Robert S. Goldfarb & H.O. Stekler, Meta-Analysis, 16 J. ECON. PERSP. 225, 225 (2002) (noting such concerns).
118. E.g., ROSENTHAL, supra note 12, at 131 (noting that studies conducted by same laboratories or research groups may end up bearing similarities and may not be independent).
119. See Rosenthal & DiMatteo, supra note 31, at 67 ("It is possible and often valuable to block by laboratory or researcher and examine this as a moderator variable.").
120. T.D. Stanley, Response from T.D. Stanley, 16 J. ECON. PERSP. 227, 227 (2002).
121. Id. at 227-28.
122. Id. at 228.
123. E.g., Blumenthal, supra note 51, at 43 (noting three studies that made use of interviews from the same subjects).
124. See, e.g., ROSENTHAL, supra note 12, at 26 (suggesting that for study of effects of alcoholism treatment programs, separate analyses could be performed for dependent variables like sobriety, number of days of employment or arrests, general medical health, and personal and social adjustment).
125. Id. at 27.
126. See Robert Rosenthal & Donald B. Rubin, Meta-Analytic Procedures for Combining Studies with Multiple Effect Sizes, 99 PSYCHOL. BULL. 400, 401-02 (1986) (discussing procedures for combining multiple effect sizes from single studies).
127. E.g., WOLF, supra note 22, at 15 (advising that to combat bias in favor of significant results in published articles, researcher should review results in books, dissertations, and unpublished papers from professional meetings).
128. See, e.g., Robert Rosenthal, The "File Drawer Problem" and Tolerance for Null Results, 86 PSYCHOL. BULL. 638, 638 (1979) (describing "file draw problem" in which majority of studies did not have significant-for example, p > .05 - results and thus are not found within journals);
129. Robert Rosenthal & Donald B. Rubin, Comment: Assumptions and Procedures in the File Drawer Problem, 3 STAT. SCI. 120, 120 (1988) (observing that assumption underlying original file draw computations is that selection process results in studies with significant results being published and retrieved, while nonsignificant results are not published or retrieved).
130. A straightforward guide to such calculations is available in ROSENTHAL, supra note 12, at 104-05.
131. Haejung Paik & George Comstock, The Effects of Television Violence on Antisocial Behavior: A Meta-Analysis, 21 COMM. RES. 516, 530 tbl.4 (1994).
132. Additional discussion of potential limitations of meta-analysis and other research synthesis may be found in Harris Cooper & Larry V. Hedges, Potentials and Limitations of Research Synthesis, in THE HANDBOOK OF RESEARCH SYNTHESIS, supra note 16, at 521, 523-24,
133. and in Georg E. Matt & Thomas D. Cook, Threats to the Validity of Research Synthesis, in THE HANDBOOK OF RESEARCH SYNTHESIS, supra note 16, at 503, 503.
134. See Blumenthal, supra note 1, at 38-46 (identifying and discussing such concerns).
135. Patricia E. Lin, Note, Opening the Gates to Scientific Evidence in Toxic Exposure Cases: Medical Monitoring and Daubert, 17 REV. LITIG. 551, 581 (1998).
136. Id. at 581.
137. See In re Paoli R.R. Yard PCB Litig., 916 F.2d 829, 857 (3d Cir. 1990) ("[Hundreds of meta-analyses are done each year . . . .");
138. ROSENTHAL, supra note 12, at 10-11 (Usting early examples). Meta-analyses being "rarely published" in law journals is more likely due to their being rarely conducted and submitted.
139. E.g., In re Paoli R.R. Yard PCB Litig., 706 F. Supp. at 373 (accepting defense expert's contention that meta-analysis is novel scientific technique that is to be evaluated under United States v. Downing, 753 F.2d 1224 (3d Cir. 1985)).
140. ROSENTHAL, supra note 12, at 5-7 (reviewing early examples of meta-analysis);
141. Cooper & Hedges, supra note 16, at 5-6 (reviewing historical examples);
142. Ingram Olkin, History and Goals, in THE FUTURE OF META-ANALYSIS 3, 3-9 (Kenneth W. Wachter & Miron L. Straf eds., 1990) (noting examples throughout twentieth century).
143. Useful summaries of how to approach such literature searches in the social sciences appear in MaryLu C. Rosenthal, The Fugitive Literature, in THE HANDBOOK OF RESEARCH SYNTHESIS, supra note 16, at 85, 86-87,
144. and MaryLu C. Rosenthal, Bibliographic Retrieval for the Social and Behavioral Scientist, 22 RES. HIGHER EDUC. 315, 315-17 (1985).
145. A useful beginning point, which reviews some of the primary social science electronic resources, appears in JOHN MONAHAN & LAURENS WALKER, SOCIAL SCIENCE IN LAW app. at 652-54 (6th ed. 2006).
146. Legal researchers are becoming accustomed to viewing working papers through various electronic databases such as SSRN, http://www.ssrn.com, and bepress, http://www.bepress.com. Such databases, however, are less common in many domains of empirical social sciences - psychology, sociology, etc.
147. Most social science journals use parenthetical references, with bibliographies, rather than footnote references.
148. See supra Part II.F.1 for a discussion of this criticism.
149. See, e.g., Paul M. Wortman, Judging Research Quality, in THE HANDBOOK OF RESEARCH SYNTHESIS, supra note 16, at 97, 98-99, 101-06 (discussing methods to assess quality of research).
150. See Rosenthal, supra note 31, at 184 (warning that 1.0 weighting system where good studies are included and bad studies are excluded "is often suspect on grounds of weightier bias").
151. This approach has itself been criticized for subjectivity. See, e.g., Sander Greenland & Keith O'Rourke, On the Bias Produced by Quality Scores in Meta-Analysis, and a Hierarchical View of Proposed Solutions, 2 BIOSTATISTICS 463, 464, 466-67 (2001) (describing problems associated with quality scores as bias predictors). Specification of the evaluative criteria, however, and ratings by third parties whose reliability can then be statistically analyzed and reported, should eliminate such concern.
152. E.g., ROSENTHAL, supra note 12, 51-58 (discussing such evaluation and formal methods for assessing reliabihty).
153. Wortman, supra note 105, at 98 (recognizing that issue is not whether to eliminate studies, which is inevitable, but to choose which studies to include or exclude based on quality measurement).
154. See supra note 52 and accompanying text.
155. Mitchell et al., supra note 9, at 627.
156. Daniel R. Pinello, Linking Party to Judicial Ideology in American Courts: A Meta-Analysis, 20 JUST. SYS. J. 219, 237 (1999).
157. Id.
158. Nancy Mehrkens Steblay et al., The Effects of Pretrial Publicity on Juror Verdicts: A Meta-Analytic Review, 23 LAW & HUM. BEHAV. 219, 224 tbl.2, 224-25, 227 (1999). It seemed to; crimes of murder, sexual abuse, or drugs yielded larger effects of pretrial publicity than did other crimes. Id. at 227.
159. Paik & Comstock, supra note 92, at 528-29, 530 tbl.4. They seemed to; cartoons, for instance, had a higher effect than other sorts of shows. Id. at 529.
160. See THOMAS D. COOK & DONALD T. CAMPBELL, QUASI-EXPERIMENTATION: DESIGN & ANALYSIS ISSUES FOR FIELD SETTINGS 6 (1979) (explaining purpose and procedures involved in quasi-experiments).
161. See supra notes 31-44 and accompanying text for an explanation of the significance of and flaws of p-values.
162. For a useful discussion of one- versus two-tailed significance tests, see HOWELL, supra note 34, at 92-93,
163. and GEORGE W. SNEDECOR & WILLIAM G. COCHRAN, STATISTICAL METHODS 67-68 (8th ed. 1989).
164. ROSENTHAL, supra note 12, at 61.
165. See infra Appendix Z for a table converting p-values to their corresponding Z-values.
166. E.g., ROSENTHAL, supra note 12, at 85 (going through analysis for combining p-values for two example sets of data).
167. See supra Part III.C for issues regarding coding based on quality or various other factors.
168. Robert Rosenthal & Donald B. Rubin, Comparing Significance Levels of Independent Studies, 86 PSYCHOL. BULL. 1165, 1166-67 (1979).
169. See supra note 119 and accompanying text.
170. A table giving critical values for testing chi-squares is provided infra in Appendix X.
171. ROSENTHAL, supra note 12, at 14.
172. Blumenthal, supra note 51, at 39;
173. Orr & Guthrie, supra note 4, at 613.
174. 2.
175. To go from Cohen's d to r,
176. 2+4, so long as the sample sizes of the two groups being compared to calculate d are equal or about equal. If not, a more generalized form of the equation is available. ROSENTHAL, supra note 12, at 19.
177. I refer to this equation infra in the text following note 143 and the text accompanying note 145.
178. As a methodological matter, the degrees of freedom in the numerator is always one fewer than the number of groups being compared.
179. E.g., ROSENTHAL, supra note 12, at 13 (commenting that F-test with df greater than one in numerator results in quantitative answers that are "hopelessly imprecise"). This broader test is called a "diffuse" test; the test with one df in the numerator is a "focused" test.
180. ROSENTHAL, supra note 12, at 15.
181. E.g., MARK W. LIPSEY & DAVID B. WILSON, PRACTICAL META-ANALYSIS 63 (2001) (discussing need for this statistical transformation).
182. Note that this is the natural logarithm (i.e., with e as the base, rather than with 10 as the base).
183. Note that the summation is of the converted Fisher z's, rather than calculating the mean r and then converting that r into a Fisher z,.
184. E.g., LIPSEY & WILSON, supra note 132, at 64 (noting that retransformation);
185. ROSENTHAL, supra note 12, at 21 (suggesting that although z, makes a serviceable effect size estimate, r is more easily interpreted).
186. Some meta-analysts recommend further adjustments, in order to "correct" for a variety of potential sorts of unreliability in the data, such as variation in range in the dependent and independent variables, unreliability in those variables, differences in their sample size, etc. E.g., HUNTER ET AL., supra note 21, at 35-92 (describing several ways of correcting statistical bias). This approach certainly helps focus the data and demonstrate "what effect size we might expect to find in the best of all possible worlds." ROSENTHAL, supra note 12, at 24. Nevertheless, it is not clear that this is necessarily what a meta-analyst - or a judge or policy maker - in fact wishes to learn. Id. at 25. What is at issue in a synthetic review of an empirical literature is what we know about existing research, not what might be the case if all variables were "perfectly measured, perfectly valid, perfectly continuous, and perfectly unrestricted in range." Id. More specifically, focusing on unadjusted effect size estimates allows evaluation of the existing literature as it actually stands. When 'corrections' are not used, analysis can reflect what researchers actually found, rather than what they might have found. Moreover, it can help identify potential moderators of the effects at hand, examining what factors may have led to any biases observed. Blumenthal, supra note 51, at 39 (citations omitted).
187. ROSENTHAL, supra note 12, at 73.
188. 2.
189. See supra notes 132-38 and accompanying text for how effect sizes affect the comparison of those effect sizes.
190. See generally ROBERT ROSENTHAL & RALPH L. ROSNOW, CONTRAST ANALYSIS: FOCUSED COMPARISONS IN THE ANALYSIS OF VARIANCE (1985) (providing in-depth description of value of contrast analysis);
191. ROBERT ROSENTHAL ET AL., CONTRASTS AND EFFECT SIZES IN BEHAVIORAL RESEARCH: A CORRELATIONAL APPROACH (2000) (looking at wider and more useful application of contrast analysis by introducing correlation effect size estimates).
192. E.g., ROSENTHAL, supra note 12, at 79-81 (reviewing meta-analysis of contrast studies).
193. As Rosenthal and DiMatteo point out: The standardized beta from a multiple regression, as well as a partial correlation, can be used as effect size estimates, but it must be remembered that these represent the relationship between the independent and the dependent variable controlling for other factors (and the meta-analyst might want separately to combine r's and partial r's/standardized betas). Rosenthal & DiMatteo, supra note 31, at 72.
194. JOHN E. HUNTER & FRANK L. SCHMIDT, METHODS OF META-ANALYSIS: CORRECTING ERROR AND BIAS IN RESEARCH FINDINGS 192-95 (2d ed. 2004) [hereinafter HUNTER & SCHMIDT (2d ed.)].
195. The partial correlation derived from the standardized regression coefficient is more appropriate to use than the simple correlation, which does not take into account the other variables in the multiple regression equation. Chris Doucouliagos & Patrice Laroche, Unions and Productivity Growth: A Meta-Analytic Review, in THE DETERMINANTS OF THE INCIDENCE AND THE EFFECTS OF PARTICIPATORY ORGANIZATIONS 57,77 n.5 (Takao Kato & Jeffrey Pliskin eds., 2003).
196. As a third method, one might combine the covariance matrices of the different predictor variables. Francesca Dominici et al., Combining Information from Related Regressions, 2 J. AGRIC. BIOLOGICAL & ENVTL. STAT. 313, 316-19 (1997). That information is even more rarely provided in the empirical legal literature.
197. JOHN E. HUNTER & FRANK L. SCHMIDT, METHODS OF META-ANALYSIS: CORRECTING ERROR AND BIAS IN RESEARCH FINDINGS 502 (1990) ("[B]eta weights are relative to the set of predictors considered and will only replicate across studies if the exact set of predictors is considered in each. If any predictor is added or subtracted from one study to the next, then the beta weights for all variables may change."). In the second edition of their text, however, Professors Hunter and Schmidt imply that using standardized coefficients (i.e., beta weights) may ameliorate this concern, as suggested in the text above.
198. HUNTER & SCHMIDT (2d ed.), supra note 143, at 194. Their focus, however, is only on simple (i.e., bivariate) regression, not multiple regression. Id.
199. Cf., e.g., HOWELL, supra note 34, at 494 (noting that "a test on a variable is done in the context of all other variables in the equation," and thus, its contribution to predicting outcome may change depending on what other variables are included).
200. E.g., HUNTER & SCHMIDT (2d ed.), supra note 143, at 475 (suggesting that "regression weights are typically not suitable for cumulation").
201. But cf. Christos Doucouliagos & Patrice Laroche, What Do Unions Do to Productivity? A Meta-Analysis, 42 INDUS. REL. 650, 658 (2003) (explaining usefulness of meta-regression analysis and using it despite challenges).
202. Note too that even traditional narrative reviews will be subject to such criticism. See Doucouliagos & Laroche, supra note 148, at 654-55 (criticizing traditional qualitative reviews for being overly subjective and speculative and unable to scientifically assess specification differences).
203. Analogously, multiple regression studies may be better or worse at specifying the "correct" or consistent model or at including the "correct" or relevant or consistent predictors. Each, however, is testing the relationship between predictors and an outcome, and the studies are not per se incomparable.
204. Compare supra notes 83-84.
205. See supra Part III B for an explanation of this and other procedures that may be used to address the reliability or quality of the synthesized studies.
206. See supra Part III.C for an explanation of the possibilities for coding for various moderator variables.
207. E.g., HUNTER & SCHMIDT (2d ed.), supra note 143, at 475 (providing example of how to choose studies with common variables to compare). Professors Hunter and Schmidt would also consider synthesizing studies that might include the same predictor variables (alone or as a subset), so long as the study published the full set of intercorrelations among the predictors.
208. Id. This is rare, especially in legal literature.
209. There are at least three other alternatives for a researcher seeking to synthesize a multiple regression literature. First, one might use "meta-regression analysis," an approach expressly designed to summarize multiple regression studies in empirical economics. T.D. Stanley, Wheat from Chaff: Meta-Analysis As Quantitative Literature Review, 15 J. ECON. PERSP. 131, 131 (2001);
210. T.D. Stanley & Stephen B. Jarrell, Meta-Regression Analysis: A Quantitative Method of Literature Surveys, 3 J. ECON. SURVEYS 54 (1989),
211. reprinted in 19 J. ECON. SURVEYS 299, 301 (2005) ("Simply stated, to review empirical economic literature, one must summarize regression results."). Second, one might apply a variation of hierarchical modehng and data augmentation techniques.
212. E.g., Dominici et al., supra note 145, at 314 (proposing "combination of hierarchical modeling and data augmentation" to deal with combining multiple regressions). This approach is useful in addressing issues that arise when studies use different sets of predictors or have missing data. Nonetheless, substantially more information from the primary studies is necessary, often prohibitively so.
213. Id. at 331 ("[W]e assumed throughout that the study's means and covariance matrices are available for analysis. Meta-analysis of regression studies requires different approaches when more limited information, such as significance test results, is reported."). Finally, an alternative to explicit synthesis that nevertheless is useful in drawing causal inferences from the sorts of large databases that econometric analysis often utilizes is "propensity score analysis." This approach is a distant cousin-though arguably a more elegant and precise relative-of comparing studies with different sets of predictors. It is also quite similar to the "nonparametric matching" approach recently used by Epstein and colleagues.
214. See Lee Epstein et al., The Supreme Court During Crisis: How War Affects Only Non-War Cases, 80 N.Y.U. L. REV. 1, 65-69 (2005) (describing their procedure, "which uses the insights of random assignment to draw causal inferences in observational studies, while decreasing the role of onerous assumptions of conventional parametric estimates"). More specifically, propensity score analysis controls for different sets of naturally occurring background characteristics that are likely not controlled for in different studies, "reducing the entire collection of background characteristics to a single composite characteristic that appropriately summarizes the collection."
215. Donald B. Rubin, Estimating Causal Effects from Large Data Sets Using Propensity Scores, 127 ANNALS INTERNAL MED. 757, 757 (1997);
216. see also Ralph B. D'Agostino, Jr. & Donald B. Rubin, Estimating and Using Propensity Scores with Partially Mising Data, 95 J. AM. STAT. ASS'N 749, 749 (2000) (explaining that "[p]ropensity scores are a one-dimensional summary of multidimensional covariates, X, such that when the propensity scores are balanced across the treatment and control groups, the distribution of all the covariates, X, are balanced in expectation across the two groups");
217. Paul R. Rosenbaum & Donald B. Rubin, The Central Role of the Propensity Score in Observational Studies for Causal Effects, 70 BIOMETRIKA 41, 41 (1983) (explaining that "propensity score is the conditional probability of assignment to a particular treatment given a vector of observed covariates").
218. E.g., Blumenthal, supra note 51, at 40 tbls.2 & 3 (illustrating stem-and-leaf plots of effect sizes).
219. See generally Robert Rosenthal & Donald B. Rubin, A Simple, General Purpose Display of Magnitude of Experimental Effect, 74 J. EDUC. PSYCHOL. 166 (1982) (describing BESD, which displays change in success rate attributable to certain treatment procedures).
220. Table patterned after ROSENTHAL, supra note 12, at 134 tbl.7.2.
221. See, e.g., ROSENTHAL, supra note 12, at 134-36 & tbl.7.5 (describing propranolol study with effect accounting for one-fifth of one percent of variance).
222. E.g., Blumenthal, supra note 51, at 43 tbl.6 (summarizing moderator effects in study on reasonable woman standard for sexual harassment).
223. See Lipsey, supra note 46, at 69-70 (investigating hazards and complexities of interpreting moderator variables especially when they are related to one another).
224. E.g., Stephen W. Raudenbush, Random Effects Models, in THE HANDBOOK OF RESEARCH SYNTHESIS, supra note 16, at 301, 302 (Ulustrating random effects approach and weighted least squares regression approach to deal with issues surrounding moderators). Note that Raudenbush's chapter also highlights an important difference in types of meta-analyses, those using random effects models and those using fixed effects models. The difference is important, reflecting issues of generalizability and other inferences.
225. See, e.g., Blumenthal, supra note 51, at 47 (noting some differences).
226. A good discussion of the differences and reasons for choosing one approach over the other appears in Cooper and Hedges, supra note 93, at 526-27.
227. See Stanley, supra note 155, at 131-32 (discussing potential of meta-analysis to summarize, evaluate, and analyze empirical economic research).
228. Blumenthal, supra note 1, at 45.
229. Recall that the p-value must be one-tailed, either in the original study or as a result of halving the researcher's value. See supra notes 117-18 and accompanying text for a discussion of the requirement that the p-value must be one-tailed.