Fingerprint error rates and proficiency tests: What they are and why they matter

Hastings Law Journal

Volumn 59, Issue 5, 2008, Pages 1077-1100

  Koehler, Jonathan J ^a  

^a ARIZONA STATE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 49549083906     PISSN: 00178322     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Conference Paper

References (111)

1. A latent is a print of unknown source that is recovered from a crime scene.
2. See, e.g., State v. Rose, Case No. K06-0545, at 25 (Md. Cir. Ct. Oct. 19, 2007) (order granting motion to exclude testimony) ("Mr. Meagher has stated that the FBI testifies to 'a 100 percent certainty that we have an identification.' . . . Mr. Meagher claimed that there is no error rate for ACE-V [the fingerprint technique]."); OFFICE OF THE INSPECTOR GEN., U.S. DEP'T OF JUSTICE, A REVIEW OF THE FBI's HANDLING OF THE BRANDON MAYFIELD CASE 8 (2006) ("Latent fingerprint identifications are subject to a standard of 100 percent certainty.").
3. Tamara F. Lawson, Can Fingerprints Lie?: Re-Weighing Fingerprint Evidence in Criminal Jury Trials, 31 AM. J. CRIM. L. 1, 65 (2003) ("[F]ingerprint analysis is under attack because of the lack of study done on the accuracy of the examiners coupled with the unwillingness or inability of the forensic science community to detect, acknowledge, and correct mistakes.");
4. Michael J. Saks & Jonathan J. Koehler, The Coming Paradigm Shift in Forensic Identification Science, 309 SCIENCE 892, 894 (2005) [hereinafter Saks & Koehler, Paradigm Shift];
5. Michael J. Saks & Jonathan J. Koehler, The Individualization Fallacy in Forensic Science Evidence, 61 VAND. L. REV. 199, 214-15 (2008) [hereinafter Saks & Koehler, Individualization Fallacy];
6. Adina Schwartz, Commentary on Nichols, 52 J. FOREN. SCIS. 1414, 1414-15 (2007).
7. The evidence is best characterized as a "reported match" rather than a "match" because the former description captures the uncertainty associated with the possibility that an examiner mistakenly failed to notice differences between two prints that should have produced a conclusion that the two prints do not match. See Jonathan J. Koehler, On Conveying the Probative Value of DNA Evidence: Frequencies, Likelihood Ratios, and Error Rates, 67 U. COLO. L. REV. 859, 870 (1996) ("The major difference between treating DNA evidence as a certain match and treating DNA evidence as a reported match is that only the latter requires consideration and explicit incorporation of the possibility of error.").
8. Suzanne O. Kaasa et al., Statistical Inference and Forensic Evidence: Evaluating a Bullet Lead Match, 31 LAW & HUM. BEHAV. 433, 434 (2007).
9. Id.
10. In DNA testing, the random match probability (RMP) captures the diagnosticity of a match report. The RMP is the frequency with which a genetic profile (i.e., the relevant set of alleles) exists in a reference population. William C. Thompson & Simon A. Cole, Psychological Aspects of Forensic Identification Evidence, in EXPERT PSYCHOLOGICAL TESTIMONY FOR THE COURTS 31, 34 (Mark Costanzo, Daniel Krauss & Kathy Pezdek eds., 2007).
11. Kaasa et al., supra note 5.
12. Saks & Koehler, Paradigm Shift, supra note 3; Saks & Koehler, Individualization Fallacy, supra note 3.
13. Daubert v. Merrill Dow Pharm., Inc., 509 U.S. 579 (1993).
14. Kumho Tire Co. v. Carmichael, 526 U.S. 137 (1999).
15. Kumho Tire, 526 U.S. at 147 (interpreting Daubert to apply not only to "'scientific' testimony, but to all expert testimony"); Daubert, 509 U.S. at 594.
16. On the diagnosticity side, the chance of a coincidental DNA match is often extremely small (e.g., one in many millions, billions, or trillions). On the reliability side, the chance of a false match that arises from, say, a sample handling mistake is much larger. If fingerprint coincidental match probabilities also turn out to be quite remote, then the rate at which false matches due to lab errors will probably be much larger.
17. False positive errors are defined infra.
18. A similar point has been made many times in the context of reported DNA matches. See, e.g., DAVID J. BALDING, WEIGHT-OF-EVIDENCE FOR FORENSIC DNA PROFILES 35 (2005) ("If the false-match probability (ii) is judged to be much larger than the chance-match probability (i), then the latter probability is effectively irrelevant to evidential weight [and] it is not the absolute but the relative magnitude of the false-match to the chance-match probabilities that determines whether the former can be safely neglected.");
19. Jonathan J. Koehler et al., The Random Match Probability in DNA Evidence: Irrelevant and Prejudicial?, 35 JURIMETRICS J. 201, 201 (1995) ("[Random Match Probabilities (RMPs)] contribute little to an assessment of the diagnostic significance of a reported DNA match beyond that given by the false positive laboratory error rate when RMPs are several orders of magnitude smaller than this error rate.");
20. Richard Lempert, After the DNA Wars: Skirmishing with NRC II, 37 JURIMETRICS J. 439, 447 (1997) ("[T]he probative value of a DNA match is always limited by the chance of false positive error.");
21. William C. Thompson et al., How the Probability of a False Positive Affects the Value of DNA Evidence, 48 J. FORENSIC SCIS. 47, 47 (2003) ("[H]aving accurate estimates [of] the false positive probabilities can be crucial for assessing the value of DNA evidence.");
22. see also Steve Gutowski, Error Rates in the Identification Sciences, FORENSIC BULL., Summer 2005, at 23 ("An estimate of actual or potential error rate is crucial to the probative value of all evidence.").
23. For reviews of fingerprint errors, see SIMON A. COLE, SUSPECT IDENTITIES: A HISTORY OF FINGERPRINTING AND CRIMINAL IDENTIFICATION (2001), and Simon A. Cole, The Prevalence and Potential Causes of Wrongful Conviction by Fingerprint Evidence, 37 GOLDEN GATE U. L. REV. 39, 57-60 (2006).
24. For a review of sloppy laboratory procedures and fraud see, e.g., Paul Giannelli, The Abuse of Scientific Evidence in Criminal Cases: The Need for Independent Crime Laboratories, 4 VA. J. SOC. POL'Y & LAW 439 (1997).
25. IRVING B. WEINER ET AL., HANDBOOK OF FORENSIC PSYCHOLOGY 552 ("In scientific writing and research, error rate has no uniform definition.").
26. There is no widely agreed-upon terminology for describing this particular error rate. In using the term "false discovery rate," to describe P(-S | "S"), I follow the recent statistics literature. See Yoav Benjamin & Yosef Hochberg, Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing, 57 J. ROYAL STAT. SOC'Y 289, 291 (1995) (the false discovery rate is "the proportion of the rejected null hypotheses which are erroneously rejected).
27. The epidemiological literature, which spawned the terms "false positive" and "false negative," does not identify this error rate (which is computationally identical to "one minus the positive predictive value"). Others have referred to it as the "percent false positives." See JOHN MONAHAN, THE CLINICAL PREDICTION OF VIOLENT BEHAVIOR: AN ASSESSMENT OF CLINICAL TECHNIQUES 79 (1981);
28. Stephen D. Hart et al., A Note on Portraying the Accuracy of Violence Predictions, 17 LAW & HUM. BEHAV. 695, 697 (1993). However, "percent false positives" may be confused with its inverse, the "false positive error rate." Ther term "false alarm rate" would seem to describe this error, but this phrase is often used it as a synonym for the false positive error rate. Terminological ambiguities are yet another reason why there is confusion over the meaning of error rate.
29. Michael J. Saks & Jonathan J. Koehler, Questions About Forensic Science: Response, 311 SCIENCE 607, 609 (2006) (pointing out four different ways to compute the error rate in a hair comparison study).
30. Simon A. Cole, More Than Zero: Accounting for Error in Latent Fingerprint Identification, 95 J. CRIM. L. & CRIMINOLOGY 985, 1030 (2005) ("There are a number of different ways of reporting false positives. Often the false positive rate has been reported as the number of participants who committed at least one false positive divided by the total number of participants.");
31. William C. Thompson, Subjective Interpretation, Laboratory Error and the Value of Forensic DNA Evidence: Three Case Studies, 96 GENETICA 153, 155 (1995) ("The false positive rate of a test is most usefully stated as the ratio of false positives to the sum of true positives and false positives." (citation omitted)). Thompson's definition of the most useful form of the false positive error rate is identical to the "false discovery rate." Id.
32. Lempert, supra note 15, at 449 ("[T]he jury wants to know, given that a match has been reported, how likely is it that it is false.").
33. For simplicity, ignore the rate of inconclusive judgments, which we will assume to be similar for the two examiners.
34. The task is tougher for the second examiner because the match baserate is lower (100 out of 1,000 = 10% for the second examiner versus 900 out of 1,000 = 90% for the first examiner).
35. The same could be said about the false negative error rate statistic.
36. Lempert, supra note 15, at 449 ("[T]he implications of . . . [the false positive] error rate for the likelihood that a particular reported match is erroneous depend[] on the proportion of true matches in the cases the laboratory analyzes.").
37. Some courts understand this point. State v. Rose, No. K06-0545, at 24, 28 (Md. Cir. Ct. Oct. 19, 2007) (order granting motion to exclude testimony) ("The State is correct that fingerprint evidence has been used in criminal cases for almost a century. While that fact is worthy of consideration, it does not prove reliability. . . . An error rate, or lack thereof, must be demonstrated by reliable scientific studies, not by assumption."). Other courts do not. See e.g., United States v. Havvard, 117 F. Supp. 2d 848, 854 (S.D. Ind. 2000) (noting that "the error rate with latent print identification is vanishingly small when it is subject to fair adversarial testing and challenge").
38. DAVID L. FAIGMAN ET AL., SCIENCE IN THE LAW: FORENSIC SCIENCE ISSUES 72 n.68 (2002) ("[R]arely is it possible to test whether an examiner in a real case is correct or incorrect.");
39. Simon A. Cole, Grandfathering Evidence: Fingerprint Admissibility Rulings from Jennings to Llera Plaza and Back Again, 41 AMER. CRIM. L. REV. 1189, 1211 (2004) ("[I]n real life cases, no one knows the truth. Other actors in casework - detectives, attorneys, judges, juries - are not so much testing the fingerprint examiner's conclusions as following them.");
40. Lyn Haber & Ralph Norman Haber, Scientific Validation of Fingerprint Evidence Under Daubert, 7 LAW, PROBABILITY & RISK 87, 96 (2008) ("Ground truth is unknown during adversarial proceedings, so the outcome cannot be used to assess validity of the method being used.").
41. Cole, supra note 27, at 1211 ("[I]t is difficult to imagine what evidence would possibly convince any criminal justice system actor, including the defendant's own counsel, that the fingerprint evidence was erroneous."); Interview by Adrian Cho with Simon Cole, NEW SCIENTIST, June 16, 2001, at 42 ("Remember a trial is not an experiment because you have no way of validating your results. If you falsely match a fingerprint, then the guy is convicted and goes to jail. Who's going to believe he's really innocent unless there's some extraordinary circumstance, like DNA comes along or someone else confesses? There's no way you would ever know.").
42. Michael J. Saks et al., Merlin and Solomon: Lessons from the Law's Formative Encounters with Forensic Identification Science, 49 HASTINGS L.J. 1069, 1102 n.169 (1998) ("The allusion to 'thousands of cases without error' begs the question of validity. In actual disputed cases it rarely, if ever, is possible to tell whether the identification was correct or not; that is why the issue was before the fact finder.").
43. Ellen J. Langer & Lois G. Imber, When Practice Makes Imperfect: Debilitating Effects of Overlearning, 37 J. PERSONALITY & SOC. PSYCH. 2014, 2014 (1979);
44. see also Earl H. McKinney & Kevin J. Davis, Effects of Deliberate Practice on Crisis Performance, 45 HUM. FACTORS 436, 444 (2003).
45. Moshe Kam et al., Signature Authentication by Forensic Document Examiners, 46 J. FORENSIC SCIS. 884, 887 (2001) (noting that expert document examiners and laymen each detected amateur forgeries more than 90% of the time);
46. Jodi Sita et al., Forensic Handwriting Examiners' Expertise for Signature Comparison, 47 J. FORENSIC SCIS. 1117, 1121 (2002) (finding that expert document examiners correctly detected skilled forgeries 55% of the time; laymen detected the forgeries 57% of the time).
47. But see Moshe Kam & Erwei Lin, Writer Identification Using Hand-Printed and Non-Hand-Printed Questioned Documents, 48 J. FORENSIC SCIS. 1391, 1391 (2003) (finding that expert document examiners made fewer errors than laymen on hand-printed and non-hand-printed documents).
48. Margaret A. Berger, Laboratory Error Seen Through the Lens of Science and Policy, 30 U.C. DAVIS L. REV. 1081, 1088 (1997) ("We regularly admit other kinds of evidence, such as business records, without any thought of quantifying the error that undoubtedly attends their creation.").
49. FAIGMAN, supra note 27, at 74 ("It should matter to a factfinder whether the error rate of an identification technique is a fraction of a percent or 20 percent, even if the court regards neither level of error sufficient to exclude the expert testimony.").
50. Those who contend that a 20% industry-wide error rate should be ignored when some examiner-specific information is not accounted for in the industry-wide rate are committing a logical error called the base rate fallacy. The base rate ordinarily remains relevant even in the face of additional individuating information. A more sophisticated challenge to the base rate argument in this context occurs when there is credible information about the specific cause of testing errors (or the absence of testing errors) that either no longer exists or has changed significantly. If, for example, a disproportionate number of proficiency-test errors were committed by those who had less than one year of experience, but new rules forbid testimony from such relatively inexperienced analysts, then the error rate should be recomputed using the narrower examiner reference class (i.e., examiners who have at least one year of experience). That is, rather than rely on the error rate for all examiners, a more appropriate error rate would be the subset of examiners who have at least one year of experience (subject to sample size concerns). Paul E. Meehl & A. Rosen, Antecedent Probability and the Efficiency of Psychometric Signs, Patterns, or Cutting Scores, 52 PSYCH. BULL. 194, 194 (1955).
51. This point is less an indictment of the use of error rates than it is a reminder that error rates based on narrower reference classes are preferable to those based on broad reference classes when both are available. Jonathan J. Koehler, The Base Rate Fallacy Reconsidered: Normative, Descriptive and Methodological Challenges, 19 BEHAV. & BRAIN SCIS. 1, 10 (1996).
52. Koehler, supra note 4, at 874 ("It cannot be that every laboratory in the industry has an error rate that is less than the industry-wide average. Yet this is exactly the claim that is made every time laboratory personnel reject the industry-wide figures, and argue that their own error rate is substantially less or zero.").
53. See, e.g., United States v. Havvard, 117 F. Supp. 2d, 848, 854 (S.D. Ind. 2000) ("The government claims the error rate for the method is zero. The claim is breathtaking, but it is qualified by the reasonable concession that an individual examiner can of course make an error in a particular case.").
54. Id.; Joint Appendix at 168, United States v. Rogers, No. CR-90-1-BR (E.D.N.C. Dec. 7, 2000) ("[I]t's the government's position, with respect to fingerprints, there's no rate of error. One, [sic] the fingerprints match. There's no rate of error. It's a matter of separating the methodology from the practitioner. The method is foolproof. If there's a mistake it's with the practitioner and that goes to the qualifications of the witness."); see also Simon A. Cole, More Than Zero: Accounting for Error in Latent Fingerprint Identification, 95 J. CRIM. L. & CRIMINOLOGY 985, 1034-37 (2005) (criticizing the concept of methodological error rate).
55. Cole, supra note 37, at 1039.
56. David A. Stoney, Fingerprint Identification: Scientific Status, in 4 MODERN SCIENTIFIC EVIDENCE: THE LAW AND SCIENCE OF EXPERT TESTIMONY 82, 103 (Faigman et al. eds., 2002) ("[C]urrent fingerprint examination processes are inextricably linked with the human examiner and one cannot separate the human error of the examiner (whatever the cause) from the reliability of fingerprint evidence.").
57. See, e.g., IAN W. EVETT & BRUCE S. WEIR, INTERPRETING DNA EVIDENCE: STATISTICAL GENETICS FOR FORENSIC SCIENTISTS 240 (1998) (noting that "the undisputed subjectivity of a fingerprint comparison");
58. Robert Epstein, Fingerprints Meet Daubert: The Myth of Fingerprint "Science" is Revealed, 75 S. CAL. L. REV. 605, 638 n.185 (2002) ("Latent fingerprint examiners make subjective judgment calls throughout the entire comparison process . . . ."); Stoney, supra note 39, at 99. ("The criteria for absolute identification in fingerprint work are subjective and ill-defined.").
59. Ronald G. Nichols, Defending the Scientific Foundations of the Firearms and Tool Mark Identification Discipline: Responding to Recent Challenges, 52 J. FORENSIC SCIS. 586, 592 (2007) ("[P]roficiency tests can offer to the court a reliable practical indicator of how often the profession, using accepted procedures, practices, and controls, makes a false identification.").
60. See Forensics Program - Collaborative Testing Services, Inc., http://www.collaborativetesting.com/forensics/index.html (last visited Apr. 20, 2008). CTS is the primary provider of external forensic science proficiency tests across a variety of subfields, including fingerprint examination.
61. Id.
62. COLLABORATIVE TESTING SERVS., INC., FORENSIC TESTING PROGRAM: LATENT PRINTS SUMMARY REPORT TEST NO. 07-516 (2007) [hereinafter CTS TEST NO. 07-516], available at http://www.collaborativetesting.com/reports/2716_web.pdf.
63. Id. at 2.
64. Id. at 3.
65. See id. at 19-20.
66. Cole, supra note 37, at 1033 ("The existing data are inadequate to calculate a meaningful error rate for forensic fingerprint identification."); Lyn Haber & Ralph Norman Haber, Error Rates for Human Latent Fingerprint Examiners, in AUTOMATIC FINGERPRINT RECOGNITION SYSTEMS 339, 339 (Nalini K. Ratha & Ruud Bolle, eds., 2004) ("It is impossible to determine from existing data whether true error rates are miniscule or substantial.");
67. William A. Tobin & William C. Thompson, Evaluating and Challenging Forensic Identification Evidence, CHAMPION, July 2006, at 12, 19-20 ("[P]roficiency testing in forensic science is frequently worthless as a true indicator of examiner proficiency.").
68. CTS TEST NO. 07-516, supra note 44, at 2.
69. Id.
70. This is a frequent criticism of DNA proficiency tests. See United States v. Liera Plaza, 188 F. Supp. 2d 549, 558 (E.D. Pa. 2002) ("Llera Plaza II") (containing testimony from fingerprint expert Allen Bayle who claimed that the FBI's fingerprint proficiency tests are easier than casework because they contain too many matching prints and because the test prints had greater than usual clarity); Jonathan J. Koehler, Why DNA Likelihood Ratios Should Account for Error (Even When a National Research Council Report Says They Should Not), 37 JURIMETRICS 425, 429-30 (1997) ("[B]ecause most proficiency tests are relatively easy in the sense that they are nonblind, internal tests that use large, clean stains, performance on these tests would appear to provide a generously low estimate of (criminal) casework error rates.").
71. Lempert, supra note 15, at 447 (noting that existing proficiency tests "can be expected to understate error rates, for laboratory technicians can take special care when they know they are dealing with test samples"); Adina Schwartz, A Systematic Challenge to the Reliability and Admissibility of Firearms and Toolmark Identification, 6 COLUM. SCI. & TECH. REV. 2, 27 (2005) (noting that CTS proficiency tests "are likely to have understated day-to-day laboratory rates because the testing was declared, rather than blind").
72. But see Steve Gutowski, A Response to: A Systematic Challenge to the Reliability and Admissibility of Firearms and Toolmark Identification, a Recently Published Article by Adina Schwartz, FORENSIC BULL., Winter 2005, at 22, 24 ("[P]roficiency tests probably overestimate the error rate due to the irritation many examiners felt, at least in the past, of unnecessary testing which stopped people from getting on with their 'real work,' in addition to a tendency or misunderstanding that external tests were not supposed to undergo the usual technical and administrative reviews.") (emphasis added).
73. When fingerprint proficiency tests are more difficult - as they were in a test provided by CTS in 1995 which include a latent print from the identical twin of one of the knowns - the number of false identifications can increase dramatically. In this test, 33 out of 150 reporting laboratories (22%) made at least one misidentification on the seven samples. Twenty-nine labs made exactly one misidentification, two labs made two misidentifications, and two labs made six misidentifications. COLLABORATIVE TESTING SERVS., INC., FORENSIC TESTING PROGRAM, LATENT PRINTS EXAMINATION, REPORT NO. 9508 (1995).
74. As a former editor of the Journal of Forensic Identification noted, "reaction to the results of the CTS 1995 Latent Print Proficiency Test within the forensic science community has ranged from shock to disbelief." David L. Grieve, Possession of Truth, 46 J. FORENSIC IDENTIFICATION 521, 524 (1996). Subsequent tests have been less difficult in that they rarely include latents and knowns from such highly related individuals. Importantly, the absence of a clear metric by which to gauge print difficulty is a problem in its own right. Haber & Haber, supra note 27, at 95 ("[T]he profession lacks a quantitative measure of print quality (difficulty). One expert observed that the prints used in the FBI proficiency test are so easy they are a joke." (citation omitted)).
75. Ideally, the participants should be a random sample of this population.
76. Because the forensic science community is unlikely to embrace and voluntarily participate in tough, realistic proficiency tests, the courts might require such participation from laboratories that wish to offer results in a trial. This point arose in the context of DNA evidence as well. See Erin Murphy, The New Forensics: Criminal Justice, False Certainty, and the Second Generation of Scientific Evidence, 95 CAL. L. REV. 721, 796 (2007) (arguing that laboratory error rate data would have to be provided "if reliability were treated as a threshold question of admissibility");
77. Barry C. Scheck, DNA and Daubert, 15 CARDOZO L. REV. 1959, 1997 (1994) ("Unfortunately, forensic laboratories have historically resisted external blind proficiency testing and other efforts to assess laboratory error rates. . . . Unless there are strong decisions on the error rate issue, forensic DNA laboratories and legislators will not voluntarily do what Daubert requires." (footnote omitted)).
78. For a different view, see Berger, supra note 32, at 1088 (arguing that it is "pointless" to argue that reliable error rates should be admitted because error rates for individual laboratories are "unattainable"), and Edward J. Imwinkelried & D. H. Kaye, DNA Typing: Emerging or Neglected Issues, 76 WASH. L. REV. 413, 460 (arguing that the admissibility "standards relate to the capacity of an analytical procedure to generate accurate results when properly applied," not to whether the individual using those procedures is prone to err).
79. The highest quality proficiency testing is "double-blind" administered by an outside agency. In ideal double-blind testing, the test taker and test taker's supervisor(s) should not be aware that the sample(s) under analysis is a proficiency test. Anything less than double-blind testing administered by an outside agency significantly erodes the effectiveness and value of a proficiency test. Tobin & Thompson, supra note 48, at 21 n.25.
80. See S.D. Ferrara et al., Proficiency Testing for Psychoactive Substances in Italy, 113 INT'L J. LEGAL MED. 50, 52 (1999) (reporting false positive and false negative error rates from a large proficiency testing program for the analysis of substances in urine).
81. See Lyn Haber & Ralph Norman Haber, Letter to the Editor, 56 J. FORENSIC IDENTIFICATION 493, 496 (2006) (arguing that fingerprint error rate studies should adopt "conditions representative of case work," including varying the difficulty of latents, using latents in which the majority do not match known prints, and using representative examiners).
82. Proficiency test administrators commonly reuse materials in different tests. This is a poor idea because examiners sometimes recognize the materials and because it limits testing materials.
83. Development of a print-difficulty rating scheme will require some empirical testing. See Haber & Haber, supra note 27 (calling for an experimentally validated quantitative metric of latent print difficulty).
84. In open proficiency testing, examiners know they are being tested. Joseph L. Peterson et al., The Feasibility of External Blind DNA Proficiency Testing. I. Background and Findings, 48 J. FORENSIC SCIS. 21, 26 (2003).
85. Id.
86. Id. (stating open proficiency tests sometimes receive "special treatment").
87. Itiel E. Dror et al., Contextual Information Renders Experts Vulnerable to Making Erroneous Identifications, 156 FORENSIC SCI. INT'L 74, 76 (2006);
88. Itiel E. Dror & David Charlton, Why Experts Make Errors, 56 J. FORENSIC IDENTIFICATION. 600, 610 (2006);
89. see also D. Michael Risinger et al., The Daubert/Kumho Implications of Observer Effects in Forensic Science: Hidden Problems of Expectation and Suggestion, 90 CAL. L. REV. 1, 35-38 (2002).
90. Peterson et al., supra note 61 ("Currently, some proficiency testing programs, such as the Department of Defense's proficiency testing program for forensic urine drug testing [reference omitted] and HIV testing, are blind."); Steven Rand et al., The GEDNAP (German DNA Profiling Group) Blind Trial Concept, 116 INT'L J. LEGAL MED. 199, 201 (2002) (reporting results from DNA blind trials across 129 laboratories in twenty-eight European countries).
91. Peterson et al., supra note 61, ("[B]lind proficiency testing . . . provides a more realistic test of lab performance" than open proficiency testing.); Lempert, supra note 15, at 447 ("The obvious way to estimate laboratory error rates is through blind proficiency testing."); Risinger et al., supra note 64, at 45 ("The simplest, most powerful, and most useful procedure to protect against the distorting effects of unstated assumptions, collateral information, and improper expectations and motivations is blind testing."); cf. Alfred Biasotti & John Murdock, The Scientific Basis of Firearms and Toolmark Identification, in 3 MODERN SCIENTIFIC EVIDENCE THE LAW AND SCIENCE OF EXPERT TESTIMONY 495, 510 (Faigman et al., eds., 2002) ("It is clear that in such a high stakes game, laboratory administration will do everything possible to ensure that the proficiency tests results are correct before reporting them.").
92. Joseph L. Peterson et al., The Feasibility of External Blind DNA Proficiency Testing. II. Experience with Actual Blind Tests, 48 J. FORENSIC SCIS. 32, 39 (2003). One examiner was tipped off to the proficiency test by a friendly administrator. Id. at 38. This problem is less likely to arise when the administrators are disinterested parties.
93. Ideally, a large set of proficiency data would be collected from every examiner who wishes to testify in court. However, as others have noted in the context of DNA evidence, this would require an impractical amount of testing. JOSEPH L. PETERSON & R.E. GAENSSLEN, DEVELOPING CRITERIA FOR MODEL EXTERNAL DNA PROFICIENCY TESTING 109 (2001), available at http://www.ncjrs.gov/pdffiles1/nij/grants/192282.pdf ("At this stage, though, it is highly unlikely that declared or blind proficiency tests can be administered with sufficient frequency, relative to the quantity of most laboratories' caseloads, to allow a meaningful calculation of the lab's or any examiner's 'error rate.'").
94. Peterson et al., supra note 61, at 23 ("In fact, it has been demonstrated that mandated proficiency testing enhances overall quality of clinical laboratory testing, including turnaround time, accuracy of results, and training of laboratories, whereas self regulation has been found less effective in achieving those goals.").
95. Writing in the context of proficiency tests for firearms and toolmark examiners. Professor Adina Schwartz argues that "proficiency testing can (at most) establish an error rate for the particular people tested, not for firearms and toolmark examination as a whole." Schwartz, supra note 52, at 25. Professor Schwartz is right if her comment is interpreted in the context of current proficiency test practices: if those who take proficiency tests are not representative of the forensic science population, then we cannot be confident that the resultant error rates are valid estimates for the whole industry. However, if participants are randomly sampled, a statistically sound extrapolation can be made.
96. Although the initial studies with well-constructed proficiency tests are likely to produce error rate data that are broadly applicable, the research should not and will not end there. Ultimately, the goal is to understand the conditions under which the risk of error is more and less likely and by how much. Characteristics of the examiner, prints, crime, and technology, are some of the variables that may matter. See FAIGMAN, supra note 27, at 63 n.7 ("Since such errors can occur and have occurred, the questions move on to: what are the probabilities of such errors under varying conditions."); Lawson, supra note 3, at 44 ("Two critical questions must, however, be tested: 1) what is the probability of an erroneous match, and 2) under what circumstances are errors more likely or less likely to occur. These questions have not been asked of fingerprint identification evidence and the lack of knowledge of the real answers to these questions is what makes the conclusions drawn from fingerprint analysis suspect and arguably unreliable." (citations omitted)).
97. See supra Part III.A.
98. Some scholars recommend using the upper bound of the error rate's 95% confidence interval as an estimate of the error rate rather than the estimated error rate itself. Lempert, supra note 15, at 453.
99. See generally Koehler, supra, note 4.
100. The National Research Council made a similar call for behavioral research to understand how best to convey the significance of a DNA match. NAT'L RESEARCH COUNCIL, COMM. ON DNA FORENSIC SCIENCE: AN UPDATE, THE EVALUATION OF FORENSIC DNA EVIDENCE 203 (1996).
101. Peterson et al., supra note 61, at 28-29.
102. An estimate extrapolated from the authors' project was at the low end, an estimate from a government agency test provider was at the high end, and an estimate from a commercial test provider fell between the other two estimates but was closer to the lower end ($1,050,000). Id. at 29 tbl.3.
103. See generally Thompson, supra note 15.
104. OFFICE OF THE INSPECTOR GEN., U.S. DEP'T OF JUSTICE, supra note 2, 8.
105. Steering Comm. of the Physicians' Health Study Research Group, Final Report on the Aspirin Component of the Ongoing Physicians' Health Study, 321 NEW ENG. J. MED. 129 (1989).
106. Neither the participants nor the investigators knew whether a participant was in the experimental or placebo group. Group assignment was random.
107. See generally Videotape: Against All Odds: Inside Statistics, Program 12: Experimental Design (The Consortium for Mathematics and its Applications and Chedd-Angier 1989), available at http://www.learner.org/ resources/series65.html.
108. Steering Comm. of the Physicians' Health Study Research Group, supra note 80, at 129.
109. Id.
110. Jason Schklar & Shari Diamond, Juror Reactions to DNA Evidence: Errors and Expectancies, 23 LAW & HUM. BEHAV. 159, 159 (1999).
111. Another way to impress upon jurors that examiner testimony is opinion rather than fact is to require an instruction from the bench to this effect. Utah v. Quintana, 103 P.3d 168, 170, 171 n.3 (Utah Ct. App. 2004) (Thorne, J., concurring) ("[T]rial courts should be directed to instruct juries about the existing weaknesses of fingerprint examiner training and identification protocol. . . . [T]he jury should have been specifically instructed that the examiner's testimony was opinion and not fact and that the jurors should examine the fingerprint evidence independently. It is vital that we remove the near mystical awe that fingerprints evoke, and replace it with a more cautious regard for forensic evidence and its overall lack of certainty."); FAIGMAN, supra note 27, at 65 (suggesting that one solution to the problem of insufficient data to back fingerprint identification claims is a "judicial instruction of the jury about the limitations of the field"). The sufficiency of such judicial instruction is unknown.