Statistical Evidence, Sensitivity, and the Legal Value of Knowledge

Philosophy and Public Affairs

Volumn 40, Issue 3, 2012, Pages 197-224

  Enoch, David ^a   Spectre, Levi ^a   Fisher, Talia ^a  

^a NONE

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84870756550     PISSN: 00483915     EISSN: 10884963     Source Type: Journal    
DOI: 10.1111/papa.12000     Document Type: Article

References (45)

1. Previous versions of this article were presented at the Uncertainty in Morality and Law Conference, sponsored by the Law and Philosophy forum of the Hebrew University in Jerusalem, in June 2011; at the Theoretical and Practical Rationality group at the Institute for Advanced Studies in Jerusalem in the fall of 2011; at the Inter-Disciplinary Center, Herzeliya, Faculty of Law Colloquium in the fall of 2011; at the New England Consequentialism Workshop in February 2012; and at the Economic Theory Workshop at Tel Aviv University in March 2012. We thank the participants for the valuable discussion. We especially thank our commentator on that first occasion, Amit Pundik, for very detailed and helpful comments. Thanks to Hagit Benbaji, Selim Berker, Mitch Berman, Stewart Cohen, Cian Dorr, Kate Elgin, Nir Eyal, Yuval Eylon, Johann Frick, Caspar Hare, Amnon Knoll, Netanel Lipshitz, Ofra Magidor, Ofer Malcai, Jennifer Nagel, Christian Piller, Ariel Porat, Jim Pryor, Mike Redmayne, Melinda Roberts, Sergio Tenenbaum, Ruti Weintraub, Dan Wikler, and two referees and an editor for Philosophy & Public Affairs for helpful comments and discussions. David Enoch's research was supported by the Israel Science Foundation (grant no. 136/09).
2. See, e.g., Rebecca Haw, "Prediction Markets and Law: A Skeptical Account, " Harvard Law Review 122 (2009): 1217-38, at pp. 1217, 1229, and the references there.
3. For one discussion of this common hypothetical, see Mike Redmayne, "Exploring the Proof Paradoxes, " Legal Theory 14 (2008): 281-309.
4. See, e.g., Jonathan J. Koehler and Daniel Shaviro, "Veridical Verdicts: Increasing Verdict Accuracy through the Use of Overtly Probabilistic Evidence and Methods, " Cornell Law Review 75 (1990): 247-79, at p. 263.
5. Nothing in our relevant intuitions depends on the nonstatistical evidence being witness testimony. Think about other types of clearly individual evidence, like the videotape of a surveillance device, or some physical evidence (say, that the accused's gun was found on the scene).
6. Sometimes people write as if individual evidence is about the individual in a way that statistical evidence is not. (See, e.g., Richard Wright, "Causation, Responsibility, Risk, Probability, Naked Statistics, and Proof: Pruning the Bramble Bush by Clarifying the Concepts, " Iowa Law Review 73 [1988]: 1001-77, at p. 1050.) We have no objection to this way of putting the initial intuition here, but it is important not to mistake a new name for the problem for a solution. For it remains to be explained in what way exactly individual evidence is-and statistical evidence is not-about the individual. Surely, both are indicative of something about the individual. Is this not enough? (See Ferdinand Schoeman, "Statistical vs. Direct Evidence, " Noûs 21 [1987]: 179-98, at pp. 183-184, for a somewhat similar point.)
7. One major issue we will not try to tackle here is the reference class problem. We will assume, more or less throughout, that the statistical evidence latches on to the relevant frequencies. Such a simplifying assumption cannot be objected to in our context, as it arguably arises both for statistical and for individual, direct evidence (for instance, regarding the 70 percent reliability of the eyewitness).
8. For a list of examples, and some insistence on the significance of the differences between them, see Redmayne, "Exploring the Proof Paradoxes, " pp. 282-85.
9. See Alex Stein, Foundations of Evidence Law (Oxford: Oxford University Press, 2005).
10. Schoeman, "Statistical vs. Direct Evidence, " is very sympathetic to this possibility, offering a debunking explanation of our intuitions to the contrary in terms of some cognitive biases. But we think that his pessimism about the distinction is premature.
11. See ibid., p. 187, for a similar distinction. Redmayne, "Exploring the Proof Paradoxes, " p. 245, also introduces this distinction, but he adds a third strategy, in terms of attacking the inference from the statistical evidence to the relevant finding. We fail to see how this forms a third kind of strategy here: either the problem with the inference prevents it from establishing the relevant belief (on which the finding is based), or it does not; if it does, the problem is epistemic; if it does not, the problem seems practical; either way, the relevant cases fall into one or the other of the two strategies differentiated in the text. Notice that the practical strategy need not be limited to instrumental considerations.
12. See John Hawthorne, Knowledge and Lotteries (Oxford: Oxford University Press, 2004); and H. E. Kyburg, Probability and the Logic of Rational Belief (Middletown, Conn.: Wesleyan University Press, 1961).
13. This further condition may be needed in order to alleviate some internalist worries (in the sense "internalism" has in epistemological contexts); we do not think we need to worry about them here.Some epistemologists (following Timothy Williamson, Knowledge and Its Limits [Oxford: Oxford University Press, 2000]; and Williamson, "Reply to John Hawthorne and Maria Lasonen-Aarnio, " in Williamson on Knowledge, ed. P. Greenough and D. Pritchard [Oxford: Oxford University Press, 2009], pp. 313-30) may want to make a distinction here between epistemic probabilities and chances. Others (following Stewart Cohen, "How to Be a Fallibilist, " Philosophical Perspectives 2 [1988]: 91-123) may want to talk of a change in context due to salience of error possibilities for the knowledge ascriber (or in the epistemic subject's practical environment, as in Hawthorne, Knowledge and Lotteries). For our purposes here, though, we do not need to decide these issues.
14. Prominent proponents of this kind of view include Robert Nozick, Philosophical Explanations (Cambridge, Mass.: Harvard University Press, 1981.
15. DeRose K., "Solving the Sceptical Puzzle, " Philosophical Review 104 (1995): 1-52.
16. Alvin Goldman, "Discrimination and Perceptual Knowledge, " Journal of Philosophy 78 (1976): 771-791.
17. Fred Dretske, "Conclusive Reasons, " Australasian Journal of Philosophy 49 (1971): 1-22.
18. This qualification allows us to bypass some technical issues here, having to do with the common way of fleshing out counterfactual semantics in terms of possible worlds. (See, e.g., Robert Stalnaker, "A Theory of Conditionals, " in Studies in Logical Theory, American Philosophical Quarterly, Monograph Series 2 [Oxford: Blackwell, 1968], pp. 98-112; and David Lewis, Counterfactuals [Oxford: Basil Blackwell, 1973]; and for more recent discussions, see Theodore Sider, Logic for Philosophy [Oxford: Oxford University Press, 2010]; and Jonathan Bennett, A Philosophical Guide to Conditionals [New York: Oxford University Press, 2003].) On such proposals, roughly speaking, a counterfactual is true if and only if in the closest possible world in which its antecedent is true, so is its consequent. The problem is that worlds that are less likely to be the actual one (for instance, one where the newspaper made a mistake) are not guaranteed to be further from the actual one compared to more likely worlds. The most-probably qualification bypasses this problem.
19. Alvin Goldman, "Discrimination and Perceptual Knowledge, " Journal of Philosophy 78 (1976): 771-791.
20. For two recent contributions, and for references, see Keith DeRose, "Insensitivity Is Back, Baby!" Philosophical Perspectives 24 (2010): 161-187; Saul A. Kripke, "Nozick on Knowledge, " in Philosophical Troubles: Collected Papers, vol. 1, ed. Saul A. Kripke (Oxford: Oxford University Press, 2011), 162-224.
21. For one example of something in the vicinity of Sensitivity, see the discussion of normic support below.
22. The evidence-law literature on statistical evidence (and on the proof paradoxes) has recently come to appreciate this similarity with cases of the type of the lottery case, but has not appreciated in full the significance of this similarity. Thus, Stein, Foundations of Evidence Law, p. 67, mentions a lottery paradox in a related context, but deals with a version of the lottery paradox that is not relevant to our concerns. Redmayne, "Exploring the Proof Paradoxes, " p. 297, discusses our version of the paradox and explicitly draws the analogy between the evidence-law cases and the epistemological literature on the (relevant kind of) lottery paradox, but he fails to mention the relevance of something like Sensitivity (rather, he discusses the Safety condition, which is related but less appropriate here, and even that he discusses only in a very sketchy way). Hock Lai Ho, A Philosophy of Evidence Law (Oxford: Oxford University Press, 2008), pp. 168-169, briefly mentions the similarity but fails to put it to theoretical use-see here also note. Judy Thomson, "Liability and Individualized Evidence, " in Rights, Restitution, and Risk, ed. William Parent (Cambridge, Mass.: Harvard University Press, 1986), pp. 225-50, at pp. 236-239, does notice the parallel and makes more of it; indeed, there are some important similarities between her treatment of statistical evidence and ours. We hope to compare her view to ours in more detail elsewhere. Let us just note here that Thomson's discussion is located in the context of a causal theory of knowledge, which we (and pretty much everyone else these days) reject, and that she seems to assume that the law should significantly care about knowledge and epistemology, a point we emphatically deny in this article.
23. David Kaye, "The Paradox of the Gatecrasher and Other Stories, " Arizona State Law Journal (1978): 101-43, at p. 104.
24. Robert J. Rhee, "Probability, Policy, and the Problem of a Reference Class, " International Journal of Evidence and Proof 11 (2007): 286-91, at p. 289.
25. Amik Pundik, "Statistical Evidence and Individual Litigants: A Reconsideration of Wasserman's Argument from Autonomy, " International Journal of Evidence and Proof 12 (2008): 303-24, at p. 303, emphasizes that in some contexts-say, that of supplying a medical diagnosis-we are perfectly willing to rely on statistical evidence. He takes this as reason to believe that the problem is not as general as we say in this article that it is. But note that Pundik is interested in the extent to which we are willing to act on statistical evidence in different practical contexts. The point in this article is that even outside the context of any action at all, we are unhappy to attribute knowledge (and perhaps even justified belief) to a subject who bases her beliefs on statistical evidence alone. And here the reluctance seems to be very general indeed. We briefly revisit this point below.
26. A fuller discussion of mistakes than we need or can afford to conduct here would include-among other things-a discussion of the ways in which the mistakes of many procedures are not themselves entirely random (for example, newspapers printing erroneous lottery results are not equally likely to print all erroneous results). Notice that such details-it seems reasonable to expect-will be closely connected to the requirement to explain the relevant kind of mistake that is emphasized in this article.
27. Martin Smith, "What Else Justification Could Be, " Noûs 44 (2010): 10-31, goes much further than merely noting the explanatory point in the text. He thinks of normic support as grounding epistemic justification, indeed as doing so even against probabilities, so that one belief may be justified while the other is not, even if the latter is more probable (for the thinker), so long as the former is normically supported by the evidence. Relatedly, Mark Schroeder suggested to us in email correspondence that while statistical evidence can support credences, it cannot support all-out beliefs (which, it follows, do not supervene on credences). Furthermore, recent work by Timothy Williamson ("Reply" and "Improbable Knowing, " in Evidentialism and Its Discontents, ed. Trent Dougherty [Oxford: Oxford University Press, 2011], pp. 147-64) defending a sharp separation between chance and epistemic probability is relevant here. We do not need to discuss these interesting suggestions for our purposes here. At least one of us intends to elaborate-in future work-on some probabilistic differences that seem to mesh nicely with central cases of sensitivity and explanation.
28. Nevertheless, let us make the following points about Smith's interesting suggestion (in "What Else Justification Could Be"). First, he explicitly addresses Sensitivity (p. 23), rejecting it-if we understand him correctly-because misleading evidence that normically supports the relevant belief (and thus grounds epistemic justification) is not sensitive. But this, it seems to us, is beside the point: Of course bad or misleading evidence-individual evidence included-can fail Sensitivity. (If a bad or misleading eyewitness testifies that p, it may be the case that she would have so testified even had it been the case that not-p. Indeed, this is one of the standard ways of discrediting eyewitnesses.) The crucial point for us is that even good statistical evidence fails Sensitivity. Second, Smith's suggestion attempts to explain epistemic justification using thoughts about what does and what does not call for explanation. And we agree that he is on to something important here, at least regarding the correlation between good, justification- and knowledge-grounding evidence and what mistakes call for explanation. Still, given the opacity of calling for an explanation-the question "what calls for explanation?" seems to us profound, and we do not know of eye-opening answers to it: it is hard to see Smith's contribution as explanatory progress. Talking about which mistakes call for explanation (rather than about which evidence supports which beliefs) does not seem to reduce mysteriousness. Perhaps the truth-values of Sensitivity-like counterfactuals-or the law-like connections that support them-are better candidates for being the more basic explanatory story.
29. It is sometimes suggested that relying on statistical evidence amounts to preferring better consequences over doing justice in the specific case (e.g., Erik Lillquist, "Recasting Reasonable Doubt: Decision Theory and the Virtues of Variability, " U.C. Davis 36 [2002]: 85-198, at p. 140), or to punishing people for being members of a reference class (e.g., Mark Colyvan, Helen M. Regan, and Scott Ferson, "Is It a Crime to Belong to a Reference Class?" Journal of Political Philosophy 9 [2001]: 168-181). Both of these suggestions are wrong, though, for reasons related to the point in the text: Individuals seem entitled not to be subjected to procedures that are too inaccurate, and statistical evidence improves accuracy. And even a court that is only concerned with making justice in the specific case has to use some evidence-the question is whether, when the court relies on evidence, statistical evidence is not as good as other evidence. And of course, whatever the evidence is that the court relies on for conviction, it punishes the convicted for the crime, not for being subject to incriminating evidence about the crime.
30. We can put things here also in terms of luck, but it is not clear this will promote understanding rather than confusion. In a case of the kind described, in which system are right decisions right in a way that does not (or that does to a lesser extent) depend on luck? There is, we concede, a sense in which there is something about System A that makes it less vulnerable to luck (when it is mistaken, for instance, its mistakes call for an explanation and do not invite merely a "you win some, you lose some" kind of response). But when it comes to the law, it seems just as ill advised to prefer this over accuracy as it is to prefer knowledge or any other positive epistemic status.
31. See, e.g., Duncan H. Pritchard and John Turri, "The Value of Knowledge, " in Stanford Encyclopedia of Philosophy, , and the references there.
32. Notice that this remains so even if we engage in "knowledge-first" epistemology (see Williamson, Knowledge and Its Limits), perhaps partly because of a (purported) constitutive relation between assertability and knowledge. This is why Ho's way (in Philosophy of Evidence Law, pp. 140-43) of addressing the proof paradoxes in the law of evidence seems to us unsatisfying. Even given his Williamsonian assumptions, why should the law care about, say, assertability? Why should the law care about it enough to tolerate a higher rate and probability of mistakes? Indeed, for a Williamsonian there is a serious problem here having to do with reliance on evidence since the same body of evidence may diverge radically between probabilities as opposed to chances. For Williamson, that there is no close world where a belief is false does not entail that there is a low chance that the belief is false. Thus, one can know that p is true (i.e., the Williamsonian epistemic probability on a subject's evidence is 1) while the chance that p is true may be close to 0. Once this separation is imposed between chance and epistemic probability, it is not clear (in or outside the courtroom) what one should do if, say, the chance that p is low and the epistemic probability is high (supposing it is crucial for an agent's practical deliberation whether or not p). For more on related issues, see John Hawthorne and Maria Lasonen-Aarnio, "Knowledge and Objective Chance, " in Greenough and Pritchard, Williamson on Knowledge, pp. 92-108; Williamson, "Reply"; and Assaf Sharon and Levi Spectre, "Epistemic Closure under Deductive Inference: What Is It and Can We Afford It?" Synthese (forthcoming).
33. See here, e.g., Mitch Berman's interesting discussion of how the fact that the standard used to allow instant replay to reverse the initial call in many sports requires indisputable or conclusive evidence is best explained not by a concern for accuracy, but for other values. Mitchell N. Berman, "Replay, " California Law Review 99 (2011): 1683-1744.
34. Perhaps-we are not sure-such considerations can shed some light on the recent discussion of Gettier cases in the context of legal proof. See Michael S. Pardo, "The Gettier Problem and Legal Proof, " Legal Theory 16 (2010): 37-57; Pardo, "More on the Gettier Problem and Legal Proof, " Legal Theory 17 (2011): 75-80, and the references there.
35. This is a different kind of indirect relevance from the one highlighted in the previous paragraph. There, it was indirect relevance by transitivity. In our case, it will be indirect relevance by a kind of parallelism.
36. Chris Sanchirico, "Character Evidence and the Object of Trial, " Columbia Law Review 101 (2001): 1227-1311.
37. In particular, character evidence may be thought of as a kind of intrapersonal statistical evidence. And just as with statistical evidence there is an intuitive feeling that the evidence is not sufficiently about the relevant individual directly, with character evidence there is an intuitive feeling that the evidence is not sufficiently about the relevant specific action directly.
38. We thank Mitch Berman for making us see this point.
39. In the next subsection, we are going to argue for systematic parallels between the incentive story and the epistemological story. Still, that there may be some discrepancies between the incentive story and the counterfactual Sensitivity story is to be expected, if nothing else because incentives are evaluated ex ante, and the counterfactuals ex post. We discuss these complications elsewhere.
40. We thank Mike Redmayne and Christian Piller for discussions relevant to this point.
41. "Statistical Evidence: The Case for Sensitivity" (unpublished manuscript).
42. Our focus on Sanchirico's account neither entails nor presupposes that no other considerations can contribute here. But for any other account it will have to be checked whether it coincides with the epistemic story of Sensitivity as Sanchirico's does.
43. Wait: is DNA evidence not an obvious counterexample to our claims, a case where the law is happy relying on insensitive evidence? Things are complicated here, and we address these matters in detail in another article. Let us just note that actually the account developed here promises to explain the DNA exception to the usual suspicion with which statistical evidence is viewed. This is so, first, because DNA evidence does seem to be sensitive (had he not done it, in all likelihood we would have not found his DNA on the scene, and so in all likelihood we would have not convicted him), and second, because the incentive problem for statistical evidence does not seem to apply (because would-be perpetrators are hardly ever in a position to know whether DNA obtained on the scene will match theirs). But, to repeat, all this merits much more discussion, which we engage elsewhere. See "Statistical Evidence: The Case for Sensitivity."
44. And there may be other debunking explanations of these intuitions as well. For one attempt at a cognitive psychological one, see K. J. Heller, "The Cognitive Psychology of Circumstantial Evidence, " Michigan Law Review 105 (2006): 241-306.
45. We thank an anonymous referee for noting this.