A Bayesian truth Serum for subjective data

Science

Volumn 306, Issue 5695, 2004, Pages 462-466

  Prelec, Dražen ^a  

^a MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DATA REDUCTION; POPULATION STATISTICS; PUBLIC POLICY; SOCIAL ASPECTS;

INFORMATION SOURCES; SUBJECTIVE DATA;

INFORMATION ANALYSIS;

BEHAVIOR;

ARTICLE; BAYES THEOREM; CONSENSUS; DECISION MAKING; HUMAN; POLICY; PRIORITY JOURNAL; SCIENCE; SCORING SYSTEM; VALIDATION PROCESS;

BAYES THEOREM; BIAS (EPIDEMIOLOGY); GAMES, EXPERIMENTAL; HUMANS; JUDGMENT; MATHEMATICS; MOTIVATION; TRUTH DISCLOSURE;

EID: 5644221311     PISSN: 00368075     EISSN: None     Source Type: Journal    
DOI: 10.1126/science.1102081     Document Type: Article

References (39)

1. C. F. Turner, E. Martin, Eds., Surveying Subjective Phenomena (Russell Sage Foundation, New York, 1984), vols. I and II.
2. R. M. Cooke, Experts in Uncertainty (Oxford Univ. Press, New York, 1991).
3. A formalized scoring rule has diverse uses: training, as in psychophysical experiments (32); communicating desired performance (33); enhancing motivation and effort (34); encouraging advance preparation, as in educational testing; attracting a larger and more representative pool of respondents; diagnosing suboptimal judgments (4); and identifying superior respondents.
4. R. Winkler, J. Am. Stat. Assoc. 64, 1073 (1969).
5. W. H. Batchelder, A. K. Romney, Psychometrika 53, 71 (1988).
6. In particular, this precludes the application of a futures markets (27) or a proper scoring rule (29).
7. C. d'Aspremont, L.A. Gerard-Varet, J. Public Econ. 11, 25 (1979).
8. S. J. Johnson, J. Pratt, R. J. Zeckhauser, Econometrica 58, 873 (1990).
9. P. McAfee, P. Reny, Econometrica 60, 395 (1992).
10. H. A. Linstone, M. Turoff, The Delphi Method: Techniques and Applications (Addison-Wesley, Reading, MA, 1975).
11. R. M. Dawes, in Insights in Decision Making, R. Hogarth, Ed. (Univ. of Chicago Press, Chicago, IL, 1990), pp. 179-199.
12. S. J. Hoch, J. Pers. Soc. Psychol. 53, 221 (1987).
13. It is one of the most robust findings in experimental psychology that participants' self-reported characteristics - behavioral intentions, preferences, and beliefs - are positively correlated with their estimates of the relative frequency of these characteristics (35). The psychological literature initially regarded this as an egocentric error of judgment (a "false consensus") (36) and did not consider the Bayesian explanation, as was pointed out by Dawes (11, 14). There is still some dispute over whether the relationship is entirely consistent with Bayesian updating (37).
14. R. M. Dawes, J. Exp. Soc. Psychol. 25, 1 (1989).
15. D. Fudenberg, J. Tirole, Game Theory (MIT Press, Cambridge, MA, 2000).
16. With finite players, the truth-telling result holds provided that the number of players exceeds some finite n, which in turn depends on p(ω).
17. J. M. Bernardo, A. F. M. Smith, Bayesian Theory, Wiley Series in Probability and Statistics (Wiley, New York, 2000).
18. R. J. Aumann, Econometrica 55, 1 (1987).
19. s). The reverse implication (i.e., that different opinions imply different posteriors) is also called stochastic relevance (8).
20. q/(n - 2). The score for r is built up from pairwise comparisons of r against all other respondents s, excluding from the pairwise calculations the answers and predictions of respondents r and s. To prevent infinite scores associated with zero frequencies, I replace the empirical frequencies with Laplace estimates derived from these frequencies. This is equivalent to "seeding" the empirical sample with one extra answer for each possible choice. Any distortion in incentives can be made arbitrarily small by increasing the number of respondents, n. The scoring is zero-sum when α = 1.
21. T. M. Cover, J. A. Thomas, Elements of Information Theory (Wiley, New York, 1991).
22. S. Kullback, Information Theory and Statistics (Wiley, New York, 1954).
23. k|ω), which is inserted into the fraction, (d) follows from (c) by Bayes' rule.
24. The scoring system is not easy to circumvent by collective collusion, because if everyone agrees to give the same response then that response will no longer be surprisingly common, and will receive a zero information score. The prediction scores will also be zero in that case.
25. J. Cremer, R. P. McLean, Econometrica 56, 1247 (1988).
26. M. Rees, Our Final Century (Heinemann, London, 2003).
27. J. Berg, R. Forsythe, T. A. Rietz, in Understanding Strategic Interaction: Essays in the Honor of Reinhard Selten, W. Albers, W. Guth, B. Hammerstein, B. Moldovanu, E. van Damme, Eds. (Springer, New York, 1997), pp. 441-463.
28. Certainly, the assumption of independent credibility signals is unrealistic in that it implies that expert opinion can in aggregate predict the true outcome perfectly; a more realistic model would have to interpose some uncertainty between the outcome and the totality of expert opinion.
29. L. J. Savage, J. Am. Stat. Assoc. 66, 783 (1971).
30. B(ω). Unlike proper scoring, information scoring could be applied to verbal expressions of probability ("likely," "impossible," etc.).
31. Precisely, if the opinions of one type of respondent are a statistical "garbling" of the opinions of a second type, then the first type will receive a lower score in the truth-telling equilibrium. Garbling means that the more informed individual could replicate the statistical properties of the signal received by the less informed individual, simply by applying a randomization device to his own signal (38).
32. D. M. Green, J. A. Swets, Signal Detection Theory and Psychophyics (Peninsula Publishing, Los Altos, CA, 1989).
33. W. Edwards, Psychol. Rev. 68, 275 (1961).
34. C. F. Camerer, R. Hogarth, J. Risk Uncert. 18, 7 (1999).
35. G. Marks, N. Miller, Psychol. Bull. 102, 72 (1987).
36. L. Ross, D. Greene, P. House, J. Exp. Soc. Psychol. 13, 279 (1977).
37. J. Krueger, R. W. Clement, J. Pers. Soc. Psychol. 67, 596 (1994).
38. D. Blackwell, Ann. Math. Stat. 24, 265 (1953).
39. I thank D. Mijovic-Prelec, S. Frederick, D. Fudenberg, J. R. Hauser, M. Kearns, E. Kugelberg, R. D. Luce, D. McAdams, S. Seung, R. Weaver, and B. Wernerfelt for comments and criticism. I acknowledge early support for this research direction by Harvard Society of Fellows, MIT E-Business Center, and the MIT Center for Innovation in Product Development