Eliciting informative feedback: The peer-prediction method

Management Science

Volumn 51, Issue 9, 2005, Pages 1359-1373

  Miller, Nolan ^a   Resnick, Paul ^b   Zeckhauser, Richard ^a  

^a HARVARD UNIVERSITY   (United States)

^b UNIVERSITY OF MICHIGAN   (United States)

Author keywords

Electronic markets; Honest feedback; Proper scoring rules

Indexed keywords

ELECTRONIC MARKETS; HONEST FEEDBACK; PROPER SCORING RULES; RATERS;

ELECTRONIC EQUIPMENT; FEEDBACK; SIGNAL PROCESSING;

DECISION MAKING;

EID: 25844515285     PISSN: 00251909     EISSN: 15265501     Source Type: Journal    
DOI: 10.1287/mnsc.1050.0379     Document Type: Article

References (35)

1. Avery, C., P. Resnick, R. Zeckhauser. 1999. The market for evaluations. Amer. Econom. Rev. 89(3) 564-584.
2. Breese, J., D. Heckerman, C. Kadie. 1998. Empirical analysis of predictive algorithms for collaborative filtering. Proc. Fourteenth Conf. Uncertainty Artificial Intelligence. Morgan Kaufmann Publisher, Madison, WI.
3. Clemen, R. 2002. Incentive contracts and strictly proper scoring rules. Test 11(1) 195-217.
4. Congdon, P. 2001. Bayesian Statistical Modelling. Wiley, Chichester, UK.
5. Cooke, R. M. 1991. Experts in Uncertainty: Opinion and Subjective Probability in Science. Oxford University Press, New York.
6. Crémer, J., R. McLean. 1985. Optimal selling strategies under uncertainty for a discriminating monopolist when demands are interdependent. Econometrica 53(2) 345-361.
7. Crémer, J., R. McLean. 1988. Full extraction of surplus in Bayesian and dominant strategy auctions. Econometrica 56(6) 1247-1257.
8. d'Aspremont, C., L.-A. Gérard-Varet. 1979. Incentives and incomplete information. J. Public Econom. 11(1) 25-45.
9. d'Aspremont, C., L.-A. Gérard-Varet. 1982. Bayesian incentive compatible beliefs. J. Math. Econom. 10(1) 83-103.
10. Dellarocas, C. 2001. Analyzing the economic efficiency of eBay-like online reputation reporting mechanisms. Proc. 3rd ACM Conf. Electronic. Commerce, ACM Press, New York.
11. Friedman, D. 1983. Effective scoring rules for probabilistic forecasts. Management Sci. 29(4) 447-454.
12. Gollier, Cristian. 2001. The Economics of Risk and Time. MIT Press, Cambridge, MA.
13. Hanson, R. 2002. Logarithmic market scoring rules for modular combinatoral information aggregation. Working paper, Department of Economics, George Mason University, Fairfax, VA.
14. Jensen, F., C. Peterson. 1973. Psychological effects of proper scoring rules. Organ. Behavior Human Performance 9(2) 307-317.
15. Johnson, S., J. Pratt, R. Zeckhauser. 1990. Efficiency despite mutually payoff-relevant private information: The finite case. Econometrica 58(4) 873-900.
16. Johnson, S., N. Miller, J. Pratt, R. Zeckhauser. 2003. Efficient design with interdependent valuations and an informed center. Working paper RWP03-020, Kennedy School of Government, Harvard University, Cambridge, MA.
17. Kandori, M., H. Matsushima. 1998. Private observation, communication and collusion. Econometrica 66(3) 627-652.
18. Lampe, C., P. Resnick. 2004. Slash(dot) and burn: Distributed moderation in a large online conversation space. CHI Lett. 6(1) 543-550.
19. Lavalle, I. 1968. On cash equivalents and information evaluation in decisions under uncertainty: Part I: Basic theory. J. Amer. Statist. Assoc. 63(321) 252-276.
20. Miller, N., P. Resnick, R. Zeckhauser. 2005. A sufficient condition for correlated information in mechanism design. Mimeo, Harvard University, Cambridge, MA.
21. Nau, R. 1985. Should scoring rules be effective? Management Sci. 34(5) 527-535.
22. Nelson, R., D. Bessler. 1989. Subjective probabilities and scoring rules: Experimental evidence. Amer. J. Agricultural Econom. 71(2) 363-369.
23. Ottaviani, M., P. N. Sørensen. 2004. The strategy of professional forecasting. Finance Research Unit working paper 2004-2005, University of Copenhagen, Copenhagen, Denmark.
24. Pratt, J., H. Raiffa, R. Schlaifer. 1965. Introduction to Statistical Decision Theory. McGraw-Hill, New York.
25. Prelec, D. 2001. A two-person scoring rule for subjective reports. Working paper, Marketing Center, MIT Sloan School, Cambridge, MA.
26. Prendergast, C. 1993. A theory of yes-men. Amer. Econom. Rev. 83(4) 757-770.
27. Roth, A. 1995. Introduction to experimental economics. J. Kagel, A. Roth, eds. The Handbook of Experimental Economics, Princeton University Press, Princeton, NJ, 3-110.
28. Sarwar, B. M., G. Karypis, J. A. Konstan, J. Riedl. 2000. Analysis of recommender algorithms for e-commerce. Proc. 2nd ACM Conf. Electronic Commerce, ACM Press, New York, 158-167.
29. Savage, L. 1954. Foundations of Statistics. Dover Publications, New York.
30. Savage, L. 1971. Elicitation of personal probabilities and expectations. J. Amer. Statist. Assoc. 66(336) 783-801.
31. Selten, R. 1998. Axiomatic characterization of the quadratic scoring rule. Experiment. Econom. 1(1) 43-62.
32. Smith, C. 1961. Consistency in statistical inference and decision. J. Roy. Statist. Soc., Series B (Methodological) 23(1) 1-37.
33. Staël von Holstein, C.-A. 1970. Measurement of subjective probability. Acta Psych. 34(1) 146-159.
34. Winkler, R. 1969. Scoring rules and the evaluation of probability assessors. J. Amer. Statist. Assoc. 64(327) 1073-1078.
35. Winkler, R. 1996. Scoring rules and the evaluation of probabilities. Test 5(1) 1-60.