Reservations about likelihood ratios (and some other aspects of forensic 'Bayesianism')

Law, Probability and Risk

Volumn 12, Issue 1, 2013, Pages 63-73

  Michael Risinger, D ^a  

^a SETON HALL UNIVERSITY   (United States)

Author keywords

Bayesianism; Expert evidence; Forensic science; Form of expert testimony

Indexed keywords

EID: 84875288413     PISSN: 14708396     EISSN: 1470840X     Source Type: Journal    
DOI: 10.1093/lpr/mgs011     Document Type: Article

References (93)

1. Wilde, Shaw and Bertrand Russell, among others, apparently made somewhat similar observations, but not as pithily as the saying finally became. Wilde was first, saying in 1887 (in The Cantorville Ghost) 'We really have everything in common with America nowadays, except, of course, language.'
2. See the discussion at http://forum.wordreference.com/showthread.php?t=146783.
3. The major exception to this generalization regarding forensic science seems to be Canada, where forensic science practices correspond more closely to those of its southern neighbor, the United States. Conversation with Dr. Madlen Margau, Forensic Scientist, Chemistry Section, Centre of Forensic Sciences, Toronto, Ontario, Canada.
4. Eighth International Conference on Forensic Inference and Statistics, Seattle, Washington, 18 July 2011
5. Which of course includes Australia and New Zealand, New Zealand being the source of one of the major texts, Bernard Robertson and G. A. Vignaux, Interpreting Evidence: Evaluating Forensic Science in the Courtroom (J.Wiley, 1995). But one must remember that Augustine wrote in Hippo, not in Rome.
6. There is some disagreement overwhether the proper possessive formis 'Bayes's Theorem' or 'Bayes' Theorem'.Compare, e.g. Stephen M. Stigler, Who Discovered Bayes's Theorem? 37 AMER. STATISTICIAN 290 (1983).
7. Reprinted in Stephen M. Stigler, Statistics on the Table 291 (1999), with the Wikipedia entry 'Bayes' Theorem', http://en.wikipedia.org/wiki/Bayes'_theorem.
8. Under the tutelage of my wife, I have finally settled on the general rule that I will always add the possessive 's' any time it is not horribly unpronounceable, as I have done here (although I will not 'sic' the other form in the notes that follow). Of course, as everyone knows, in its usual algebraic forms, it was not originated by Bayes at all, but by Laplace, but that's another story for another footNote
9. For a pre'cis, see D.Michael Risinger, Introduction to "BayesWars Redivivus," 8 INTERNATIONAL COMMENTARY ON EVIDENCE, article 1 (2010).
10. This is the footnote promised in footnote 5. The exposition in the single article on the subject attributed to Thomas Bayes, published posthumously in 1763 (after editing by Richard Price, and with an introduction and commentary by him) in the Philosophical Transactions of the Royal Society, set out certain important aspects of what is now known as Bayes's Theorem, without any specific algebraic formulae. See (as the title quaintly puts it) An Essay towards Solving a Problem in the Doctrine of Chances.
11. By the Late Rev. Mr. Bayes, F.R.S. Communicated by Mr. Price, in a Letter to John Canton, A.M.F.R.S., 53 PHILOSOPHICAL TRANSACTIONS 370 (1763), available at http://rstl.royalsocietypublishing.org/content/53/370.
12. But it was clearly Laplace, almost certainly independently, whose work gave the more generalized notion its impetus and content
13. See generally Stephen M. Stigler, The History of Statistics: Measurement of Uncertainty before 1900 (Belknap Press, 1986).
14. As Stephen Fienberg has noted (noting others who have noted the same thing), 'Bayes did not actually give us a statement of Bayes' Theorem, either in its discrete form (this came with Laplace), or in its continuous form with integration, although he solved a special case of the latter.'
15. Stephen E. Fienberg,When did Bayesian Inference Become "Bayesian"? BAYESIAN ANALYSIS 1, 3 (2006) (formula and note omitted). Available from http://ba.stat.cmu.edu/journal/2006/vol01/issue01/fienberg.pdf
16. I am aware of one sense in which all probabilities must be regarded as, in that sense, subjective, in the same way that language, being an abstraction, is in the same sense subjective. Both are products of mind, and are therefore mind-dependent. Something like this seems to have been the starting point for Bruno De Finetti.
17. See Bruno De Finetti, Theory of Probability, vol. 1, p. 1 (English translation, J. Wiley, 1974) "Probability does not exist".;
18. Quoted in Robert F. Nau, De Finetti Was Right: Probability Does Not Exist, 51 THEORY & DECISION 89 (2001).
19. Perhaps the mostmysterious thing, then, is why anyone claims any probabilities are not subjective. However, I think I get the main point of the frequentists: despite various levels of mystery (including the ontological status of mathematics in general, see Mary Leng, Mathematics and Reality (Oxford University Press, 2010).
20. Objective or 'frequentist' probability is the appropriatemodel when one has reason to believe that abstract notions of exact mathematical probability specification (represented in their simplest form by specified 'mental experiment' urn drawing problems or similarly specified mental experiments regarding gambling games) actually represent a reasonable model of the empirical world in which a problem is faced and must be answered. (real poker games, etc.).When such circumstances do not exist (and more often than not they don't) best guesses (subjective probabilities) must be resorted to, or else we must defer decision. But in both language and in notions of probability, there is a necessary contribution from an exterior physical reality, and more so in regard to warranted probability statements about factual conditions than in regard to some kinds of natural language expressions such as value statements. So my own emphasis is not on the subjective nature of probabilities vel non in this general sense, but on the empiricalwarrant that is, or ought to be, necessary to derive and use a probability statement in different contexts. It may be that numbers derived from subjective individual mental betting experiments are perfectly sufficient for some kinds of problems (contexts in which no iterative process is available to refine prior assumptions out of results, a decision must be made, and all the costs will fall on the betting party or parties, for instance), but not in others. However, appropriate criteria for acceptable best guesses for different classes of problems seemhardly ever to be discussed, at least so far as I have been able to discover. That is the most mysterious part about subjective probabilities, to me.
21. In a recent popular book on Bayesianism, Sharon Bertsch McGrain attributes to Jack Good the tongue-in-cheek claim to have identified 46 656 varieties of Bayesians. She then lists the following varieties: subjective, personalist, objective, empirical, semi-empirical, semi-Bayes, epistemic, intuitionist, logical, fuzzy, hierarchical, pseudo, quasi, compound, parametric, nonparametric, hyperparametric and non-hyperparametric Bayes.
22. See Sharon Bertsch McGrain, The Theory that Would Not Die: How Bayes' Rule Cracked the Enigma Code, Hunted Down Russian Submarines & Emerged Triumphant from Two Centuries of Controversy 129 (Yale University Press, 2011) (e. e. cummings-like lack of initial capitalization copied directly from spine and title page of book).
23. These categories obviously are not mutually exclusive. A further word about this book might be in order. It is an exercise in a kind of triumphalistWhig history, which fails to make very clear the actual contributions of 'Bayes' to many of the episodes it recounts as examples of Bayesian triumph. On the other hand, it is a terrific compendiumof anecdotes concerning the contempt in which 'frequentists' held 'Bayesians' (and vice versa) in the statistics community for over half a century. I have blessedly never seen such vitriol in the circles I inhabit.Whowould have guessed that statisticians could be somuchmore vicious than lawyers in any context.
24. See Robert E. Kass and Larry Wasserman, The Selection of Prior Distributions by Formal Rules, 91 J. AM. STAT. ASSOC. 1343 (1996).
25. The wellspring work is Howard Raiffa and Robert Schlaifer, Applied Statistical Decision Theory (Harvard University, 1961)
26. See generally Matt Jones and Bradley C. Love, Bayesian Fundamentalism or Enlightenment? On the Explanatory Status and Theoretical Contributions of the Bayesian Model of Cognition, 34 BEHAV. & BRAIN SCI. 169 (2011).
27. See D. Michael Risinger, Boxes in Boxes: Julian Barnes, Conan Doyle, Sherlock Holmes and the Edalji Case, 4 INT'L COMMENT. ON EVIDENCE, Iss. 2, Art. 3 at 7, n. 13, available at http://www.bepress.com/ice/vol4/-iss2/art3.
28. The foundational theorist of 19th-century positivism was Auguste Comte (1798-1857), who was, among many other things, dedicated to the proposition that the methods of science could be extended to give sure predictive knowledge to human social actions
29. See generally THE POSITIVE PHILOSOPHY OF AUGUSTE COMTE FREELY TRANSLATED AND CONDENSED BY HARRIET MARTINEAU (1855)
30. See also JOHN H. ZAMMITO, A NICE DERANGEMENT OF EPISTEMES: POST-POSITIVISM FROM QUINE TO LATOUR 6-8 (2004).
31. Comte, like Marx, was an enemy of religion who rather paradoxically founded a sort of social religion. While Comte's positions are looked upon as foundational to the social sciences, his views influenced popular views of science more than they influenced science itself. Ironically, he wrote at a time when the switch among natural scientists from certain knowledge goals to probabilistic goals was just beginning to take hold.
32. See LARRY LAUDAN, SCIENCE AND VALUES 83-85 (1984)
33. For decades, Professor Dennis V. Lindley has been a leader of the Bayesian movement in statistics.
34. See http://en.wikipedia.org/wiki/Dennis_Lindley.
35. See Ian Evett, Interpretation: A Personal Odyssey in Colin G.G. Aitken and David Stoney (eds), The Use of Statistics in Forensic Science (E. Hoorwood, 1991).
36. See Colin G.G. Aitken, Statistics and the Evaluation of Evidence for Forensic Scientists (J. Wiley, 1995).
37. The most influential series of articles on the topic of interpretation of the results of forensic testing to appear in any British forensic science journal is undoubtedly R. Cook, I.W. Evett, G. Jackson, P.J. Jones and J.A. Lambert, A model for case assessment and interpretation, 3 SCI. & JUSTICE 151 (1998).
38. R. Cook, I.W. Evett, G. Jackson, P.J. Jones and J.A. Lambert, A hierarchy of propositions: deciding which level to address in case work, 38 SCI. & JUSTICE 231 (1998).
39. I.W. Evett, G. Jackson and J.A. Lambert, More on the hierarchy of propositions: exploring the distinctions between explanations and propositions, 40 SCI. & JUSTICE 3 (2000).
40. See, e.g. IW. Evett and J. S. Buckleton, Some aspects of the Bayesian approach to evidence evaluation, 29 J. FORENS. SCI. SOCIETY 317 (1989). Graham Jackson has also been particularly active in publishing works on Bayesian interpretation in forensics.
41. See particularly Graham Jackson, Understanding Forensic Science Opinions, Ch. 16.
42. Jim Fraser and Robin Williams (eds), The Handbook of Forensic Science (Willan Publishing, 2009)
43. Professor Champod is on the faculty of law and forensic science, University of Lausanne, Lausanne, Switzerland. Lausanne is one of the leading forensic science institutions in the world, and, it is fair to say, a hotbed of Bayesianism. Professor Champod has published many works with Bayesian themes, including ones co-authored with Dr. Evett.
44. C. Champod C and I. W. Evett, A probabilistic approach to fingerprint evidence, 51 J. FORENS. IDENT. 101 (2001).
45. Others from the Lausanne faculty could be listed, but I have confined myself to the attendees at the 2011 ICFIS Conference, where the original paper on which the text is based was delivered
46. Colin Aitken and Franco Taroni, Statistics and the Evaluation of Evidence for Forensic Scientists (2nd ed., J. Wiley, 2004).
47. Recently the lead author on two further volumes, Franco Taroni, Colin Aitken, Paolo Garbolino and Alex Biedermann, Bayesian Networks and Probabalistic Inference in Forensic Science (J.Wiley, 2006).
48. Franco Taroni, Sylvia Bozza, Alex Biedermann, Paolo Garbolino and Colin Aitken, Data Analysis in Forensic Science (J. Wiley, 2010).
49. Dr. Cedric Neumann was trained at Lausanne, and is now an assistant professor of statistics and forensic science at Penn State University. I refer to him as a miracle worker because he is the lead author on Cedric Neumann, Ian Evett and J. Skerrett, Quantifying the Weight of Evidence from a Forensic Fingerprint Comparison: A New Paradigm, set for publication in the Journal of the Royal Statistical Society in 2012. Without going into too much detail, the article shows how to use common minutiae in fingermarks as anchors for triangles that can then be specified in such a way as to act as virtual alleles for that fingermark in a search of a large database. The search will give a population incidence rate for that triangle. Repeat it with other such triangle specifications and you can potentially get a truly useful random match probability that can easily be turned into a warranted likelihood ratio, should that be your preferred means of understanding such information. Brilliant, whether you are a Bayesian or not.
50. Dr. Charles Berger is principal scientist at the Netherlands Forensic Institute. At the conference, which generated this article, he joined Professor Aitken in giving a one-day workshop on Bayesian interpretation in forensic science. I am informed that Bayesian interpretation is not uniformly followed on the European continent, Lausanne and the Netherlands being its main centres. It is not generally used in Germany, for instance. Conversation with Dr. Stefan Becker, Scientific Director, Forensic Science Institute, Federal Criminal Police Office, Wiesbaden, Germany, August, 2011.
51. See Mike Redmayne, Paul Roberts, Colin Aitken and Graham Jackson, Forensic Evidence in Question, 2011 CRIM. L. REV. 347 (2011).
52. Professor Roberts hardly mentions Bayesianism in connection with expert evidence and forensic evidence interpretation in the chapter on expert evidence in his leading treatise (with Adrian Zuckerman), Paul Roberts and Adrian Zuckerman, Criminal Evidence 467-509 (2nd ed.,OxfordUniversity Press, 2010).
53. Bayesian Colin Aitken, Paul Roberts and Graham Jackson, Fundamentals of Probability and Statistical Evidence in Criminal Proceedings: Guidance for Judges, Lawyers, Forensic Scientists and Expert Witnesses (Royal Statistical Society, 2011)
54. The Forensic Institute, Commentary on the case of R. v. T. on footwear mark evidence and use of likelihood ratios, available at http://www.theforensicinstitute.com/PDF/Commentary%20on%20R%20v%20T%202011.pdf.
55. Geoffrey S. Morrison, The likelihood-ratio framework and forensic evidence in court: a response to R v T., 16 INT. J. EV. AND PROOF 1 (2012).
56. Bernard Robertson, C.A. Vignaux and Charles E. H. Berger, Extending the Confusion about Bayes, 74 MLR 444 (2011), among others.
57. that is, testimony in the formof 'this trace came fromthis source to the exclusion of all other potential sources inthe world', or some explicit or implied variant thereof, perhaps with a less absolute qualifier
58. This approach is described and critiqued in Dawn McQuiston-Surrett and Michael J. Saks, Communicating Opinion Evidence in the Forensic Identification Sciences: Accuracy and Impact, 59 HASTINGS L. J. 1159 (2008).
59. This approach is generally defended by the American footwear expert William Bodziak in William J. Bodziak, Traditional conclusions in footwear examinations versus the use of the Bayesian approach and likelihood ratio: A review of a recent UK appellate court decision. LAW PROBABILITY AND RISK, doi: 10.1093/lpr/mgs018 (2012).
60. See D. Michael Risinger and Lesley C. Risinger, Innocence is Different: Taking Innocence into Account in Reforming Criminal Procedure, 56 N.Y.L. L. REV. 869 (2012).
61. See the discussion in Michael J. Saks and D. Michael Risinger, Baserates, the Presumption of Guilt, Admissibility Rulings, and Erroneous Convictions, 2003 MICH. ST. L. REV. 1051 (2003).
62. Discussing the work of Daniel Kahneman and Amos Tversky on heurtistics and biases, reflected, e.g. in Amos Tversky and Daniel Kahneman, Heuristics and Biases, in Daniel Kahneman, Paul Slovic and Amos Tversky., (eds), Judgment Under Uncertainty: Heuristics and Biases (Cambridge University Press, 1982).
63. But as I and another co-author have previously observed: It should be noted that there is an ongoing debate over the extent to which ordinary humans are subject to processing errors from "probability blindness" in circumstances of decision presented by everyday life in the modern world. Some, most notably Gerd Gigerenzer, assert that the poor performance of people in laboratory experiments are more an artifact of the artificiality of the way information is presented in the experiment than a function of inaccurate judgment in normal circumstances.
64. See, e.g.,GerdGigerenzer&PeterM. Todd, Fast and FrugalHeuristics: The Adaptive Toolbox, in SIMPLE HEURISTICS THAT MAKE US SMART (Gerd Gigerenzer et al. eds., 1999).
65. Gerd Gigerenzer, How to Make Cognitive Illusions Disappear: Beyond "Heuristics and Biases," in 2 Eur. Rev. Soc. Psychol. (Wolfgang Stroebe & Miles Hewstone eds., 1991).
66. The debate appears to be a debate over whether our cognitive cup is half ormore empty, or half ormore full, since both sides concede that there are some problems we solve well, and some problems we deal with poorly
67. D. Michael Risinger and Jeffrey C. Loop, Three Card Monte, Monty Hall, Modus Operandi and "Offender Profiling": Some Lessons of Modern Cognitive Science for the Law of Evidence, 24 CARDOZO LAW REV 193, 196, fn 10 (2002).
68. For a recent review of the current state of the Kahneman/Tversky-Gigerenzer debate and the implications for the law, see generallyMark Kelman, The Heuristics Debate (Oxford University Press, 2011)
69. The term 'prosecutor's fallacy' for the transposition of the conditional was coined by William Thompson and Edward Schumann in a 1987 article, William C. Thompson and Edward L. Schumann, Interpretation of Statistical Evidence in Criminal Trials: The Prosecutor's Fallacy and theDefense Attorney's Fallacy, 11 L. & HUMAN BEHAV. 167 (1987).
70. Why this is an error (if it is not already well known to the reader, or obvious up a moment's reflection) is fully discussed in that article
71. This is because the hierarchy of propositions as set out and discussed concentrates on source attribution techniques. Obviously, much forensic science identification of chemical compounds is undertaken in the context of drug crimes or other possessory crimes, where the identity of the compound is in itself an element of the crime. In such situations, the distinction between the 'source level' and the 'offense level' disappears, and there is likewise little room left for propositions on the 'activity level' also.
72. See the extended discussion in D. Michael Risinger, Michael J. Saks, William C. Thompson and Robert Rosenthal, The Daubert/Kumho Implications of Observer Effects in Forensic Science: Hidden Problems of Expectation and Suggestion, 90 Cal. L. Rev. 1, 27-30 (2002).
73. The amount of case information and the level of detail thought appropriate in conducting 'activity level' analyses is well illustrated by the checklist given in Graham Jackson and Philip J. Jones, Case Assessment and Interpretation, (Wiley Online Library 2009), available at http://onlinelibrary.wiley.com/doi/10.1002/9780470061589.fsa124/abstract.
74. Ian Evett, Evaluation and Professionalism (editorial), 49 SCIENCE & JUSTICE 159 (2009)
75. D. Michael Risinger, The NAS/NRC Report on Forensic Science: A Glass Nine/Tenths Full (This Is About the Other Tenth), 50 JURIMETRICS 21, 22-34 (2009).
76. Itiel E. Dror and Greg Hampikan, Subjectivity and Bias in Forensic DNA Mixture Interpretation, 51 SCIENCE & JUSTICE 204 (2011).
77. Dan E. Krane, Simon Ford, Jason R. Gilder, Keith Inman, Allan Jamieson, Roger Koppl, Irving R. Kornfeld, D. Michael Risinger, Norah Rudin, Marc Scott Taylor, and William C. Thompson, Letter to the Editor: SequentialUnmasking: AMeans ofMinimizingObserver Effects in ForensicDNA Interpretation, 53 J. FORENSIC SCI. 1006 (2008).
78. William C. Thompson, Simon Ford, Jason R. Gilder, Keith Inman, Alan Jamieson, Roger Koppl, Irving Kornfield, Dan E. Krane, Jennifer L.Mnookin, D. Michael Risinger, Norah Rudin, Michael J. Saks, and Sandy Zabell, Commentary on Thornton re Sequential Unmasking (Letter to the Editor), 55 J. FORENSIC SCI. 1663 (2010).
79. These studies show the exquisite unreliability of the attribution of the source of bitemarks on human skin under most conditions that will be met with in practice
80. See Mary A. Bush, Raymond G. Miller, Peter J. Bush, and Robert B. J. Dorion, Biomechanical Factors in Human Dermal Bitemarks in a Cadaver Model, 54 J. Forensic Sci. 167 (2009).
81. Raymond G. Miller, Peter J. Bush, Robert B. J. Dorion, and Mary A. Bush, Uniqueness of the Dentition as Impressed in Human Skin: A Cadaver Model, 54 J. Forensic Sci. 909 (2009).
82. Mary A. Bush, Kyle Thorsrud, Raymond G. Miller, Robert B. J. Dorion, and Peter J. Bush, The Response of Skin to Applied Stress: Investigation of Bitemark Distortion in a Cadaver Model, 55 J. Forensic Sci. 71 (2010).
83. Mary A. Bush, Howard I. Cooper, and Robert B. J. Dorion, Inquiry into the Scientific Basis for Bitemark Profiling and Arbitrary Distortion Compensation 55 J Forensic Sci 976 (2010).
84. Mary A. Bush, Peter J. Bush, and H. David Sheets, Similarity andMatch Rates of the Human Dentition In Three Dimensions: Relevance to Bitemark Analysis. 125 International Journal of Legal Medicine 779 (2010).
85. H. David Sheets and Mary A. Bush, Mathematical Matching of a Dentition to Bitemarks: Use and Evaluation of AffineMethods. 207 Forensic Science International 111(2011).
86. Mary A. Bush, Peter J. Bush, and H. David Sheets, Statistical Evidence for the Similarity of the Human Dentition. 56 J Forensic Sci. 118 (2011).
87. H. David Sheets, Peter J. Bush, Cynthia Brzozowski, Lillian A. Nawrocki, Phyllis Ho, and Mary A. Bush, Dental Shape Match Rates in Selected and Orthodontically Treated Populations in New York State: A Two Dimensional Study, 56 J Forensic Sci, 621 (2011).
88. Mary A. Bush, Peter J. Bush, and H. David Sheets, A Study of Multiple Bitemarks Inflicted in Human Skin by a Single Dentition Using Geometric Morphometric Analysis. 211 Forensic Science International 1 (2011).
89. Steven Lubet, Lawyers' Poker: 52 Lessons That Lawyers Can Learn from Card Players (Oxford University Press, 2006)
90. That is, to the extent they are still usable after Regina v. T. The chart showing the likelihood ratios associated with each of the six affirmative categories of strength of support for the prosecution's proposition (weak support, moderate support, moderately strong support, strong support, very strong support and extremely strong support).
91. Association of Forensic Science Providers, Standards for the formulation of evaluative forensic science expert opinion, 49 SCI. AND JUSTICE 161, 163.
92. I know that one does not have to be a Bayesian to see the utility of likelihood ratios, and to generate them. But they are a device heavily associated with forensic science Bayesianism. Frankly, I think random match probabilities are sometimes more effective in communicating with ordinary folks than likelihood ratios, since I believe they invite simple set theory explanations that are easier to understand, and which can be graphically represented visuallymore easily in argument. I find it more congenial when talking to regular people about the implications of a well warranted random match probability to explain what it suggests about how many people would match the criterion in question in a population of a given size-a perpetrator candidate population, if you will. An RMP or similar measure enables that discussion directly in a way that a likelihood ratio does not. Plus, a likelihood ratio can generate its own version of the prosecutor's fallacy.
93. See Jonathan J. Koehler, On Conveying the Probative Value of DNA Evidence: Frequencies, Likelihood Ratios, and Error Rates, 67 U. COLO. L. REV. 859, 883 (1996).