The stricter standard: An empirical assessment of daubert's effect on civil defendants

Catholic University Law Review

Volumn 62, Issue 3, 2013, Pages 675-731

  Jurs, Andrew ^a   De Vito, Scott ^b  

^a DRAKE UNIVERSITY   (United States)

^b Florida Coastal School of Law   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84882352593     PISSN: 00088390     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Article

References (115)

1. Daubert v. Merrell Dow Pharms., Inc., 509 U.S. 579 (1993).
2. See Erica Beecher-Monas, Blinded by Science: How Judges Avoid the Science in Scientific Evidence, 71 TEMP. L. REV. 55, 75-76 (1998) (indicating that the issue of whether Daubert imposes a stricter standard for the admissibility of scientific evidence is a recurring topic of debate);
3. see also Sophia I. Gatowski et al., Asking the Gatekeepers: A National Survey of Judges on Judging Experience in a fW-Daubert World, 25 LAW & HUM. BEHAV. 433, 443 (2001) (concluding that judges are split in their views on the effect Daubert has had in their analyses of whether to admit expert evidence).
4. See, e.g., DAVID H. KAYE ET AL., THE NEW WIGMORE, A TREATISE ON EVIDENCE: EXPERT EVIDENCE § 6.3.2, at 204 (2004) (concluding that, based on the available research, "Daubert is neither uniformly stricter nor invariably weaker that Frye");
5. Jennifer L. Groscup et al., The Effects of Daubert on the Admissibility of Expert Testimony in State and Federal Criminal Cases, 8 PSYCHOL. PUB. POL'Y & L. 339, 363 (2002) (finding that an analysis of appellate decisions demonstrated "no change in the overall rate of admission for all types of expert evidence").
6. See LLOYD DlXON & BRIAN GILL, CHANGES IN THE STANDARDS FOR ADMITTING EXPERT EVIDENCE IN FEDERAL CIVIL CASES SINCE THE DAUBERT DECISION XV (2001) ("Our analysis of district court opinions suggests that after Daubert, judges scrutinized reliability more carefully and applied stricter standards in deciding whether to admit expert evidence.");
7. see also Carol Krafka et al., Judge and Attorney Experiences, Practices, and Concerns Regarding Expert Testimony in Federal Civil Trials, 8 PSYCHOL. PUB. POL'Y & L. 309, 322 (2002) (noting that judges "reported that they were more likely to scrutinize expert testimony before trial and were less likely to admit it" following Daubert).
8. Frye v. United States, 293 F. 1013, 1014 (D.C. Cir. 1923) (holding that scientific evidence must have gained sufficient "standing and scientific recognition among" authorities in the field).
9. See Edward K. Cheng & Albert H. Yoon, Does Frye or Daubert Matter? A Study of Scientific Admissibility Standards, 91 VA. L. REV. 471, 503 (2005) (stating that "a state's adoption of Frye or Daubert makes no difference in practice").
10. Daubert v. Merrell Dow Pharms., Inc., 509 U.S. 579, 588 (1993) (internal quotation marks omitted) (citing Beech Aircraft Corp. v. Rainey, 488 U.S. 153, 169 (1988)). Four years after Daubert, the Court reiterated its understanding that Daubert lowered the admissibility standard. See Gen. Electric v. Joiner, 522 U.S. 136, 142 (1997) (acknowledging that "the Federal Rules of Evidence allow district courts to admit a somewhat broader range of scientific testimony than would have been admissible under Frye"); see also infra notes 46-52 and accompanying text (detailing the Court's perception of the role of the Daubert decision).
11. Michael J. Saks, The Aftermath of Daubert: An Evolving Jurisprudence of Expert Evidence, 40 JURIMETRICS J. 229, 239 (2000) ("Testimony from those fields that could, but do not, provide adequate data ⋯ should be excluded until they can provide adequate empirical support for what the experts are claiming for their field in general, or for themselves individually.").
12. The percentage of cases involving experts remains an area with surprisingly few definitive statistics. One oft-cited study concluded that experts are involved in as many as 86% of all civil jury cases. Samuel R. Gross, Expert Evidence, 1991 WISC. L. REV. 1113, 1119 (reporting data from all civil jury cases tried in California Superior Court between 1985 and 1986).
13. Other studies returned an expert witness rate between 63% and 71%. Anthony Champagne et al., An Empirical Examination of the Use of Expert Witnesses in American Courts, 31 JURIMETRICS J. 375, 380 (1991) (finding that 57 out of 90 civil cases tried in the Dallas County District Court in Texas in 1988 utilized expert witness testimony);
14. Daniel Shuman et al., An Empirical Examination of the Use of Expert Witnesses in the Courts-Part II: A Three City Study, 34 JURIMETRICS J. 193, 197 (1994) (reporting that 131 cases out of 183 included expert testimony, a rate of 73%). Regardless of the exact rate, experts are a permanent and enormous part of modern litigation.
15. The central premise allowing for this comparison is that evidence rules are generally considered to be procedural, and thus a state's evidentiary rules on expert admissibility will not transfer upon removal. Instead, the Federal Rules of Evidence will usually apply to the action, even if state substantive tort law remains. See Cheng & Yoon, supra note 6, at 483 (explaining that, although state substantive law is retained in tort cases because of diversity jurisdiction and the Erie doctrine, the procedural rules of the new forum are applied (citing Erie R.R. Co. v. Tompkins, 304 U.S. 64, 78 (1938))); see also, e.g., Robin Kundis Craig, When Daubert Gets Erie: Medical Certainty and Medical Expert Testimony in Federal Court, 77 DENV. U. L. REV. 69, 82 (1999) (asserting that "[m]ost federal courts apply the Federal Rules of Evidence rather than state rules to all evidentiary questions in diversity cases");
16. Abbe R. Gluck, Intersystemic Statutory Interpretation: Methodology as "Law" and the Erie Doctrine, 120 YALE L.J. 1898, 1979 & n.279 (2011) (explaining that the Federal Rules of Evidence are generally considered to be procedural and are applied in cases removed from state courts);
17. Alex Stein, Constitutional Evidence Law, 61 VAND. L. REV. 65, 98-99 (2008) (noting that all evidence rules, except burdens of proof, privileges, presumptions, and the competency of witnesses, are procedural).
18. See Krafka et al., supra note 4, at 328 (reporting that approximately 69% of judges surveyed changed their procedures for considering the admissibility of expert evidence following their jurisdictions' shift to Daubert and that, generally, judges view themselves as more active in the admissibility process following Daubert); see also The Changing Role of Judges in the Admissibility of Expert Evidence, Civ. ACTION (Nat'l Ctr. for State Courts), Spring 2006, at 3 [hereinafter Changing Role of Judges].
19. Frye v. United States, 293 F. 1013, 1014 (D.C. Cir. 1923).
20. KAYE ET AL., supra note 3, § 1.2.1, at 6-7 (noting that expert testimony must have been "particularly important to aiding the trier of fact" in order to be admitted); David L. Faigman et al., Check Your Crystal Ball at the Courthouse Door, Please: Exploring the Past, Understanding the Present, and Worrying About the Future of Scientific Evidence, 15 CARDOZO L. REV. 1799, 1803 (1994) (explaining that the relevant inquiry was whether the testimony was from an area beyond the knowledge of the average juror).
21. Faigman et al., supra note 29, at 1804 & n.13 (noting that judges would evaluate the qualifications and expertise of the expert through "the expert's success in an occupation or profession which embraced" the subject matter in question); Michael J. Saks, Judging Admissibility, 35 J. CORP. L. 135, 136 (2009) (explaining that judges often inferred expertise from the expert's commercial success).
22. Frye, 293 F. at 1013; see also Faigman et al., supra note 29, at 1805 (arguing that Frye was a response to the failure of the commercial marketplace test); Saks, supra note 30, at 137 (noting that "commercial value is not a measure of scientific or any other kind of validity"); Michael J. Saks, Merlin and Solomon: Lessons from the Law's Formative Encounters with Forensic Identification Science, 49 HASTINGS L.J. 1069, 1074 (1998) (explaining that the flaws of the commercial marketplace test necessitated the Frye standard).
23. Saks, supra note 30, at 139; see also Faigman et al., supra note 29, at 1808 (describing Frye as "barely noticed" and "of little importance" in the decades after it was decided). Professor David Bernstein noted that, despite the dearth of federal court citations to Frye, state courts may have used the standard more often in deciding cases. David E. Bernstein, Frye, Frye, Again: The Past, Present, and Future of the General Acceptance Test, 41 JURIMETRICS J. 385, 388-90 (2001). However, that theory cannot be tested because state trial court opinions generally remain unpublished. Id.
24. Bernstein, supra note 35, at 390 (asserting that Frye was applied only in criminal cases until 1988); Kenneth J. Chesebro, Galileo's Retort: Peter Huber's Junk Scholarship, 42 AM. U. L. REV. 1637, 1693-95 (1993) (indicating that 64 of 67 federal appellate decisions applying Frye were criminal decisions);
25. Susan Haack, An Epistemologist in the Bramble-Bush: At the Supreme Court with Mr. Joiner, 26 J. HEALTH POL. POL'Y & L. 217, 227 (2001) (noting that Frye was the standard for the admissibility of expert evidence first in criminal cases, but later became the standard in civil cases); cf. KAYE ET AL., supra note 3, § 5.3.2, at 159-60 (detailing the several types of evidence the Frye test has been used to exclude).
26. See, e.g., Bert Black, A Unified Theory of Scientific Evidence, 56 FORDHAM L. REV. 595, 628 (1988) (arguing that the Frye test is too incoherent to be applied effectively by the courts);
27. Paul C. Giannelli, The Admissibility of Novel Scientific Evidence: Frye v. United States, A Half-Century Later, 80 COLUM. L. REV. 1197, 1207-08, 1223 (1980) (arguing that the Frye test is too vague and leads to inconsistent results).
28. See Andre A. Moenssens, Admissibility of Scientific Evidence - An Alternative to the Frye Rule, 25 WM. & MARY L. REV. 545, 551-52 (1984) (asserting that judges often admitted evidence under Frye because of their failure to understand the relevant scientific principles).
29. See, e.g., CARL F. CRANOR, TOXIC TORTS 46-47 (2006) (arguing that expert testimony can be used to mislead judges and juries unfamiliar with the scientific evidence);
30. MICHAEL D. GREEN, BENDECTIN AND BIRTH DEFECTS: THE CHALLENGES OF MASS TOXIC SUBSTANCES LITIGATION 20 (1996) (articulating concern about the use of invalid scientific evidence); HUBER, supra note 40, at 14-17;
31. David E. Bernstein, Junk Science in the United States and the Commonwealth, 21 YALE J. INT'L L. 123, 123 (1996) (alleging misuse of scientific evidence in toxic tort cases);
32. Gary Edmond & David Mercer, Trashing "Junk Science", 1998 STAN. TECH. L. REV. 3, 111 52, 57 (noting tensions regarding the use of junk science);
33. Barry M. Epstein & Marc S. Klein, The Use and Abuse of Expert Testimony in Product Liability Actions, 17 SETON HALL L. REV. 656, 660-62 (1987) (acknowledging the concern over the use of bad science in product liability cases).
34. Daubert v. Merrell Dow Pharms., Inc., 43 F.3d 1311 (9th Cir. 1991), cert, granted, 506 U.S. 214 (1992).
35. Daubert v. Merrell Dow Pharms., Inc., 509 U.S. 579, 582, 589 (1993).
36. Gen. Electric Co. v. Joiner, 522 U.S. 136, 142 (1997).
37. Kumho Tire Co. v. Carmichael, 526 U.S. 137, 152 (1999).
38. KAYE ET AL., supra note 3, § 6.4.2, at 225-26 & n.17. This group of states includes: Pennsylvania, see Grady v. Frito-Lay, 839 A.2d 1038, 1039 (Pa. 2003) (retaining Frye);
39. Illinois, see Donaldson v. Cent. 111. Pub. Serv. Co., 767 N.E.2d 314, 323 (111. 2002) (same);
40. California, see People v. Leahy, 882 P.2d 321, 324 (Cal. 1994) (same);
41. New York, see People v. Wesley, 633 N.E.2d 451, 454 (N.Y. 1994) (same);
42. and Florida, see Flanagan v. State, 625 So. 2d 827, 828 (Fla. 1993) (same).
43. For a complete list of those states that retained the Frye standard after Daubert, see David E. Bernstein & Jeffrey D. Jackson, The Daubert Trilogy in the States, 44 JURIMETRICS J. 351, 355 n.25 (2004).
44. See Cheng & Yoon, supra note 6, at 508 (arguing that removal rate analysis "depends on defense counsel's judgment" of the "practical ramifications" of the admissibility standard in his or her jurisdiction); see also Henry G. Miller, The Daubert Debacle, N.Y. ST. B.A. J., Mar./Apr. 2005, at 24, 28 (asserting that "[t]hose representing corporate defendants accused of manufacturing dangerous products will, of course, try to go to federal court and make as many Daubert motions as they can," and that a defense attorney can subsequently win a case based on a Daubert motion alone)
45. cf. David Paul Horowitz, "Will the Gatekeeper Let Daubert In?", N.Y. ST. B.J., June 2006, at 18, 18 (observing that defense attorneys in New York not only prefer to apply Daubert but have also advocated for its adoption in New York, a Frye state).
46. We used the following Frye states: Alaska, see Contreras v. State, 718 P.2d 129, 134 (Alaska 1986)
47. (using the Frye test and noting that the Frye test was adopted by Pulakis v. Stale, 476 P.2d 474, 478 (Alaska 1970)
48. overruled by State v. Coon, 974 P.2d 386, 394 (Alaska 1999))
49. Arizona, see State v. Lehr, 38 P.3d 1172, 1178-79 (Ariz. 2002) (stating that "Arizona adheres to the Frye standard in ruling on the admissibility of novel scientific evidence")
50. Connecticut, see State v. Porter, 698 A.2d 739, 746 (Conn. 1997) (reconsidering Connecticut's test for admitting scientific evidence, which had previously been based on the Frye test)
51. Florida, see Williamson v. State, 994 So. 2d 1000, 1009 (Fla. 2008) (asserting that the Frye standard should be used before admitting expert testimony)
52. Indiana, see Sears Roebuck & Co. v. Manuilov, 742 N.E.2d 453, 460-61 (Ind. 2001) (noting that Indiana courts previously relied on the Frye test)
53. Kansas, see State v. Simmons, 254 P.3d 97, 101 (Kan. 2011) (asserting that opinions based on scientific methods or procedures must be scrutinized under the test articulated in Frye)
54. Michigan, see Craig v. Oakwood Hosp., 684 N.W.2d 296, 305-06 (Mich. 2004) (noting that the testimony in question was subject to the Frye test)
55. Minnesota, see State v. Nose, 649 N.W.2d 815, 818 (Minn. 2002) (using the Frye test)
56. Missouri, see State v. Taylor, 298 S.W.3d 482, 500 (Mo. 2009) (stating that the Frye test was used to determine admissibility of scientific evidence)
57. New York, see Giordano v. Market Am., Inc., 941 N.E.2d 727, 733 (N.Y. 2010) (stating that New York courts follow the Frye test)
58. North Carolina, see State v. Peoples, 319 S.E.2d 177, 187 (N.C. 1984) (noting that Frye is the general rule for admittance of expert testimony)
59. Tennessee, see McDaniel v. CSX Transp., Inc., 955 S.W.2d 257, 262-65 (Term. 1997) (noting Tennessee's prior adherence to Frye)
60. Washington, see State v. Roberts, 14 P.3d 713, 74(M1 (Wash. 2000) (applying Frye).
61. We excluded data for Alaska from 1999 onward, see State v. Coon, 974 P.2d 386, 394 (Alaska 1999) (holding that Alaska no longer applies the Frye test)
62. Connecticut from 1997 onward see State v. Porter, 698 A.2d 739, 746 (Conn. 1997) (adopting Dauben)
63. Indiana from 1995 onward, see Steward v. State, 652 N.E.2d 490, 498-99 (Ind. 1995) (assessing the admissibility of expert evidence based on the Indiana Rules of Evidence and the principles of Daubert)
64. Tennessee from 1997 onward, see McDaniel v. CSX Transp., Inc., 955 S.W.2d 257, 265 (Tenn. 1997) (adopting certain aspects of Daubert and allowing expert evidence only where it will "substantially assist the trier of fact" and does not "indicate a lack of trustworthiness"). Because Arizona, Florida, Kansas, Michigan, Minnesota, Missouri, New York, and Washington did not adopt Daubert in the relevant period, we used data from the entire relevant period for those states. See supra note 119 (detailing the standards for the states in question).
65. See Daubert v. Merrell Dow Pharms., Inc., 509 U.S. 579 (1993).
66. We used a logistic regression because this method is generally preferred when the independent variable is categorical/binary. See, e.g., ALAN AGRESTI & CHRISTINE FRANKLIN, STATISTICS 610 (2007)
67. DAMODAR N. GUJARATI & DAWN C. PORTER, ESSENTIALS OF ECONOMETRICS 387-89 (2010).
68. All measures of statistical significance discussed in this Article relate to thep-value of a statistical hypothesis. We will consider a result to be statistically significant if its corresponding p-value is less than or equal to 0.05. This means that there is no more than a 1 in 20 chance (5% chance) that our result is due to chance. DAVID HENSHER, JOHN M. ROSE & WILLIAM H. GREENE, APPLIED CHOICE ANALYSIS: A PRIMER 46-47 (2005).
69. In addition, a p-value of < 0.05 is consistent with general practice. See, e.g., id; SCOTT E. MAXWELL & HAROLD D. DELANY, DESIGNING EXPERIMENTS AND ANALYZING DATA: A MODEL COMPARISON PERSPECTIVE 47 (2d ed. 2004).
70. The five states are Alaska, Connecticut, Indiana, North Carolina, and Tennessee. See State v. Coon, 974 P.2d 386, 394-95 (Alaska 1999) (adopting Dauben and rejecting Frye as inconsistent with the Alaska Rules of Evidence)
71. see also State v. Porter, 698 A.2d 739, 746 (Conn. 1997) (holding that Daubert should be the new standard for accepting expert scientific evidence in the state)
72. Steward v. State, 652 N.E.2d 490, 498 (Ind. 1995) (holding that, although the Daubert standard is not binding, it is helpful in applying the Indiana Rules of Evidence)
73. State v. Goode, 461 S.E.2d 631, 639 (N.C. 1995) (holding that the test of whether to admit expert evidence requires analyzing the test adopted by the Supreme Court in Daubert)
74. McDaniel v. CSX Transp., Inc., 955 S.W.2d 257, 262-65 (Tenn. 1997) (noting Tennessee's prior adherence to Frye and holding that a partial acceptance of Daubert is appropriate).
75. We ignore Arizona, Florida, Kansas, Michigan, Minnesota, Missouri, New York and Washington because they did not adopt Daubert at any point during the duration of our study. See infra note 140. We also ignore Arkansas and Oregon because they adopted Daubert before the federal courts. See Prater v. State, 820 S.W.2d 429, 431 (Ark. 1991) (adopting a Daubert standard of admissibility)
76. see also State v. Brown, 687 P.2d 751, 759 (Or. 1984) (adopting a liberal test that allows the judge to be a gatekeeper when determining whether certain evidence should be admitted).
77. The tests adopted by both courts are similar to the test later adopted in Daubert. See Farm Bureau Mut. Ins. Co. v. Foote, 14 S.W.3d 512, 519 (Ark. 2000) (noting that Prater is "strikingly similar" to Daubert)
78. see also Moore v. State, 915 S.W.2d 284, 294 (Ark. 1996) (comparing the approach adopted in Prater to the approach in Daubert)
79. State v. O'Key, 899 P.2d 633, 680 (Or. 1995) (holding that Daubert is a further development of the process the court started in Brown).
80. Finally, we ignore New Mexico because it adopted Daubert in 1993-the same time as the federal courts. See State v. Alberico, 861 P.2d 192, 203-04 (N.M. 1993) (rejected Frye and citing Daubert as the controlling standard of admissibility in New Mexico).
81. The eight states that reject Daubert and continue to rely on Frye are Arizona, Florida, Kansas, Michigan, Minnesota, Missouri, New York, and Washington. See, e.g., State v. Lehr, 238 P.3d 1172, 1178-79 (Ariz. 2002)
82. Williamson v. State, 994 So. 2d 1000, 1009 (Fla. 2008)
83. State v. Simmons, 254 P.3d 97, 101 (Kan. 2011)
84. Craig v. Oakwood Hosp., 684 N.W.2d 296, 305-06 (Mich. 2004)
85. State v. Nose, 649 N.W.2d 815, 818 (Minn. 2002)
86. State v. Taylor, 298 S.W.3d 482, 500 (Mo. 2009)
87. Giordano v. Market Am., Inc., 941 N.E.2d 727, 733 (N.Y. 2010)
88. State v. Roberts, 14 P.3d 713, 740-41 (Wash. 2000).
89. Our suspicion is that the failure to find an effect is due to three factors. First, the number of torts filed in state court compared to the number of torts removed to federal courts is extremely small. See supra FIGURE 4 (noting that the rate of removal for torts filed in state courts is less than 0.53%, likely whether or not Daubert is adopted in federal courts). Second, the number of torts in which scientific evidence is involved is also very small. In this context, any change in the number of removals is likely to be very small compared to the total number of torts filed. Third, in this context, Indiana and North Carolina both adopted Daubert in 1995 and Alaska adopted Daubert in 1999. See State v. Coon, 974 P.2d 386, 394-95 (Alaska 1999)
90. see also Steward v. State, 652 N.E.2d 490, 498 (Ind. 1995)
91. State v. Goode, 461 S.E.2d 631, 639 (N.C. 1995). This means that for Indiana and North Carolina there is only one year where the trend can begin to be established before it is expected to reverse. Similarly, for Alaska there is only one year where the reversal would be measured. In such a context, it would almost be a miracle to find the effect.
92. See, e.g., Responsible Conduct in Data Management, U.S. DEP'T OF HEALTH & HUM. SERVS., http://ori.dhhs.gov/education/products/n-illinois-u/ datamanagement/dctopic.html (last visited Mar. 25, 2013).
93. The EDNY includes the counties of Kings (the Bronx), Nassau, Queens, Richmond (Staten Island), and Suffolk. See EDNY, http://www.nyed.uscourts.gov/ (last visited Mar. 25, 2013) (stating "[t]he district comprises the counties of Kings, Nassau, Queens, Richmond, and Suffolk and concurrently with the Southern District, the waters within the counties of Bronx and New York").
94. Id. at 485. Cheng and Yoon believed that such a comparison was appropriate for a number of reasons. First, New York and Connecticut are geographically close, economically connected to New York City, and "demographically similar." See id. at 486. This implies that removal rate (and any subsequent increase or decrease in removal rates) of both areas, all else being equal, should be similar. Second, at the time, New York state courts were Frye jurisdictions while Connecticut state courts transitioned from Frye to Daubert in 1997. See State v. Porter, 698 A.2d 737, 746 (Conn. 1997). Consequently, Cheng and Yoon believed that, if there is a statistically significant change in the removal rates between the two states after 1997, that would be evidence that the choice of admissibility standard for scientific evidence affects removal rates. Cheng & Yoon, supra note 6, at 486.
95. Codebooks were provided for various periods of time. See FED. JUDICIAL CTR., FEDERAL COURT CASES: INTEGRATED DATA BASE, 1970-2000 pt. 117 [hereinafter FED. JUDICIAL CTR. DATA BASE], available at http://www.icpsr.umich.edu/icpsrweb/ NACJD/studies/08429/ascii (covering 2000); see also id. at pt. 116 (covering 1999); id. at pt. 115 (covering 1998); id. at pt. 104 (covering 1997); id. at pt. 103 (covering 1996); id. at pt. 98 (covering 1995); id. at pt. 88 (covering 1994); id. at pt. 87 (covering 1993); id. at pt. 86 (covering 1992); id. at pt. 74 (covering 1991); id. at pt. 73 (covering 1990); id. at pt. 65 (covering 1989); id. at pt. 64 (covering 1988). All codebooks are on file with the authors.
96. See Cheng & Yoon, supra note 6, at 483-84 (discussing the importance of scientific evidence in products liability tort cases); see also Margaret A. Berger, Upsetting the Balance Between Adverse Interests: The Impact of the Supreme Court's Trilogy on Expert Testimony in Toxic Tort Litigation, 64 LAW & CONTEMP. PROBS. 289, 289-90 (2001) (explaining the contested nature of scientific evidence in toxic tort cases and how those cases led the fight for a reformed standard).
97. MYOUNG-JAE LEE, MICRO-ECONOMETRICS FOR POLICY, PROGRAM, AND TREATMENT EFFECTS 99 (2005) (describing the technique of difference-in-differences).
98. See James J. Heckman et al., Matching as an Econometric Estimator, 65 REV. ECON. STUD. 261, 261 (1998). Two populations are "comparable" if both populations "would have experienced the same outcomes ⋯ had they participated in the programme." Id. at 262. A central difficulty with matching is ensuring that the two groups are comparable. See id. (explaining that it is difficult to determine whether two groups, having their "tested variable" measured, are truly comparable).
99. See JOSHUA D. ANGRIST & JORN-STEFFEN PISCHKE, MOSTLY HARMLESS ECONOMETRICS: AN EMPIRICIST'S COMPANION 227, 229-30 (2009) (discussing the requirements of a fixed-effects difference-in-differences analysis); see also LEE, supra note 207, at 99 (noting that the subjects' response variables must be measured before and after treatment).
100. See ANGRIST & PlSCHKE, supra note 220, at 228-29 (explaining an algebraic model of difference-in-differences analysis); see also David Card & Alan B. Krueger, Minimum Wages and Employment: A Case Study of the Fast-Food Industry in New Jersey and Pennsylvania, 84 AM. ECON. REV. 772, 778-79 (1994) (applying the algebraic difference-in-differences method intuitively to study the effect of minimum wage changes on employment in the fast food industry)
101. Daniel S. Hamermesh, Comment, What a Wonderful World This Would Be, 48 INDUS. & LAB. REL. REV. 835, 835 (1995) (specifying the algebraic formula for the difference-in-differences methods and critiquing the work of Card and Krueger).
102. See ANGRIST & PlSCHKE, supra note 220, at 233-41 (discussing difference-in-differences as fixed effects); see also MARNO VERBEEK, A GUIDE TO MODERN ECONOMETRICS 345 (2d ed. 2004) (noting that "[t]he fixed effects model is simply a linear regression model in which the intercept terms vary over the individual units i⋯ We can write this in the usual regression framework by including a dummy variable for each unit; in the model")
103. Marianne Bertrand et al., How Much Should We Trust Differences-in- Differences Estimates?, 119 Q. J. ECON. 249, 250-51 (2004) (discussing difference-in-differences estimation and the fixed-effects regression formula); Card & Krueger, supra note 222, at 779-81 (utilizing a least-squares dummy variable model to discuss the effect of minimum wage changes on employment).
104. The treatment group is Connecticut because it adopted Daubert in 1997. See State v. Porter, 698 A.2d 739, 746 (Conn. 1997).
105. See Heckman et al., supra note 215, at 262 (explaining how difficult it is to tell whether two groups are homogeneous). Ideally, if comparing data from two different subjects, we would prefer to assume that the statistical properties of the data for each subject are overall the same for the other subject. When this occurs, we have homogeneity. When this does not occur, we have heterogeneity; there may be unobserved relevant variables that are correlated with the observed variables and the value of those variables may differ from subject to subject. See LEE, supra note 207, at 64-65, 99 (discussing how changes that take place over time make it difficult to separate the "treatment effect from the 'time effect'"). A heterogeneous population has unique or individual characteristics as compared to another population. See DAMODAR GUJARATI, ECONOMETRICS BY EXAMPLE 282 (2011).
106. See Siu L. CHOW, STATISTICAL SIGNIFICANCE: RATIONALE, VALIDITY AND UTILITY 8 (1996) (noting that "an important real-life effect may be ignored simply because it does not reach the arbitrarily chosen level of statistical significance"). Adam Narkiewicz brought this point to our attention.
107. Arkansas and Oregon adopted tests similar to Daubert as a state evidentiary standard before the federal courts adopted Daubert in 1993, and New Mexico adopted the standard in the same year. See State v. Alberico, 861 P.2d 193, 203-04 (N.M. 1993)
108. see also Prater v. State, 820S.W.2d429, 431 (Ark. 1991)
109. State v. Brown, 687 P.2d 751, 759 (Or. 1984); supra note 139.
110. We excluded 1993 because that is the year the federal courts adopted Daubert. Daubert v. Merrell Down Pharms., Inc., 509 U.S. 579 (1993).
111. See supra notes 140, 270 and accompanying text. We eliminated New Mexico, which adopted Daubert in 1993, although it is unlikely that the adoption of Daubert by the federal courts would have any effect on removal rates in New Mexico because, before 1993, the federal and state courts used the same evidentiary standard, and, in this case, even after 1993, they both used the same standard. See State v. Alberico, 861 P.3d 192, 203-04 (N.M. 1993). Thus, its inclusion in the analysis would only create the same error found in Cheng and Yoon's analysis.
112. See Stein, supra note 14, at 81 ("For civil trials, the default proof standard is a preponderance of the evidence. Under this standard, adjudicators must deem a factual allegation proven when it is more probable than not." (footnote omitted) (citing Grogan v. Gartner, 498 U.S. 279, 286(1991))).
113. See supra text accompanying notes 47-56; see also Daubert v. Merrell Dow Pharms., Inc., 509 U.S. 579, 588 (1993)
114. (quoting Beech Aircraft Corp. v. Rainey, 488 U.S. 153, 169 (1988)).
115. Cheng and Yoon cite one example of aggregate case data methodologies being applied to an area other than to Daubert. For example, they credit a 2002 Federal Judicial Center study on class action lawsuits for the origin of removals as an aggregate measure of procedural change. See Cheng & Yoon, supra note 6, at 482 n.32 (citing BOB NIEMIC & TOM WILLGING, FED. JUDICIAL CTR., EFFECTS OF AMCHEM/ORTIZ ON THE FILING OF FEDERAL CLASS ACTIONS: REPORT TO THE ADVISORY COMMITTEE ON CIVIL RULES 12-13 (2002)).