Identifying expertise to extract the wisdom of crowds

Management Science

Volumn 61, Issue 2, 2015, Pages 267-280

  Budescu, David V ^a   Chen, Eva ^b  

^a FORDHAM UNIVERSITY   (United States)

^b NONE   (United States)

Author keywords

Contribution; Dynamic modeling; Forecasting; Wisdom of the crowd

Indexed keywords

AGGREGATES; DYNAMIC MODELS;

A-WEIGHTING; ABSOLUTE PERFORMANCE; CONTRIBUTION; CURRENT EVENTS; ECONOMIC FORECAST; EUROPEAN CENTRAL BANK; WISDOM OF CROWDS; WISDOM OF THE CROWDS;

FORECASTING;

EID: 84922351651     PISSN: 00251909     EISSN: 15265501     Source Type: Journal    
DOI: 10.1287/mnsc.2014.1909     Document Type: Article

References (49)

1. Armstrong JS (2001) Principles of Forecasting: A Handbook for Researchers and Practitioners (Kluwer Academic, Boston).
2. Aspinall W (2010) A route to more tractable expert advice. Nature 463(7279):294-295.
3. Bedford T, Cooke R (2001) Probabilistic Risk Analysis: Foundations and Methods (Cambridge University Press, Cambridge, UK).
4. Bettman JR, Luce MF, Payne JW (1998) Constructive consumer choice processes. J. Consumer Res. 25(2):187-217.
5. Bickel E (2007) Some comparisons among quadratic, spherical, and logarithmic scoring rules. Decision Anal. 4(2):49-65.
6. Broomell S, Budescu DV (2009) Why are experts correlated? Decomposing correlations between judges. Psychometrika 74(3):531-553.
7. Budescu DV (2006) Confidence in aggregation of opinions from multiple sources. Fiedler K, Juslin P, eds. Information Sampling and Adaptive Cognition (Cambridge University Press, Cambridge, UK), 327-352.
8. Clemen RT (1989) Combining forecasts: A review and annotated bibliography. Internat. J. Forecasting 5(4):559-609.
9. Clemen RT (2008) Improving and measuring the effectiveness of decision analysis: Linking decision analysis and behavioral decision research. Kugler T, Smith JC, Connolly T, Son Y-J, eds. Decision Modeling and Behavior in Complex and Uncertain Environments (Springer, New York), 3-31.
10. Clemen RT, Winkler RL (1986) Combining economic forecasts. J. Bus. Econom. Statist. 4(1):39-46.
11. Clemen RT, Winkler RL (1999) Combining probability distributions from experts in risk analysis. Risk Anal. 19(2):187-203.
12. Cooke RM (1991) Experts in Uncertainty (Oxford University Press, Oxford, UK).
13. Cooke RM, Goossens LHJ (2008) TU Delft expert judgment data base. Reliability Engrg. System Safety 93(5):657-674.
14. Davis-Stober CP, Dana J, Budescu DV (2010) A constrained linear estimator for multiple regression. Psychometrika 75(3):521-541.
15. Davis-Stober CP, Budescu DV, Dana J, Broomell SB (2014) When is a crowd wise? Decision 1(2):79-101.
16. Dawes RM (1979) The robust beauty of improper linear models in decision making. Amer. Psychologist 34(7):571-582.
17. de Finetti B (1962) Does it make sense to speak of "Good probability appraisers"? Good IJ, et al., eds. The Scientist Speculates, An Anthology of Partly-Baked Ideas (Basic Books, New York), 357-364.
18. Denrell J, Fang C (2010) Predicting the next big thing: Success as a signal of poor judgment. Management Sci. 56(10):1653-1667.
19. Evgeniou T, Fang L, Hogarth RH, Karelaia N (2013) Competitive dynamics in forecasting: The interaction of skill and uncertainty. J. Behavioral Decision Making 26(4):375-384.
20. French S (1985) Group consensus probability distributions: A critical survey. Bernardo JM, DeGroot MH, Lindley DV, Smith AFM, eds. Bayesian Statistics, Vol. 2 (North-Holland, Amsterdam), 183-201.
21. French S (2011) Expert judgment, meta-analysis and participatory risk analysis. Decision Anal. 9(2):119-127.
22. Genest C, Zidek JV (1986) Combining probability distributions: A critique and annotated bibliography. Statist. Sci. 1(1):114-148.
23. Gilovich T, Griffin D, Kahneman D (2002) Heuristics and Biases: The Psychology of Intuitive Judgment (Cambridge University Press, Cambridge, UK).
24. Hastie R, Kameda T (2005) The robust beauty of majority rules in group decisions. Psych. Rev. 112(2):494-508.
25. Herzog SM, Hertwig R (2009) The wisdom of many in one mind: Improving individual judgments with dialectical bootstrapping. Psych. Sci. 20(2):231-237.
26. Herzog SM, Hertwig R (2011) The wisdom of ignorant crowds: Predicting sport outcomes by mere recognition. Judgment and Decision Making 6(1):58-72.
27. Hogarth RM (1978) A note on aggregating opinions. Organ. Behav. Human Performance 21(1):40-46.
28. Hora SC, Fransen BR, Hawkins N, Susel I (2013) Median aggregation of distribution functions. Decision Anal. 10(4):279-291.
29. Jose VRR, Grushka-Cockayne Y, Lichtendahl KC Jr (2014) Trimmed opinion pools and the crowd's calibration problem. Management Sci. 60(2):463-475.
30. th ed. (McGraw Hill, Irwin, CA).
31. Larrick RP, Mannes AE, Soll JB (2011) The social psychology of the wisdom of crowds. Krueger JI, ed. Frontiers of Social Psychology: Social Judgment and Decision Making (Psychology Press, New York), 227-242.
32. Lee MD, Zhang S, Shi J (2011) The wisdom of the crowd playing the price is right. Memory and Cognition 39(5):914-923.
33. Leonhardt D (2012) When the crowd isn't wise. New York Times (July 7), http://www.nytimes.com/2012/07/08/sunday-review/ when-the-crowd-isnt-wise.html.
34. Lichtendahl KC Jr, Grushka-Cockayne Y, Pfeifer PE (2013) The wisdom of the competitive crowds. Oper. Res. 61(6):1383-1398.
35. Lin S-W, Cheng C-H (2009) The reliability of aggregated probability judgments obtained through Cooke's classical model. J. Model. Management 4(2):149-161.
36. Lorenz J, Rauhutb H, Schweitzera F, Helbing D (2011) How social influence can undermine the wisdom of crowd effect. Proc. Natl. Acad. Sci. 108(2):9020-9025.
37. Makridakis S, Winkler RL (1983) Averages of forecasts: Some empirical results. Management Sci. 29(9):987-996.
38. Mannes AE, Soll JB, Larrick RP (2014) The wisdom of select crowds. J. Personality Soc. Psych. 107(2):276-299.
39. Simmons J, Nelson LD, Galak J, Frederick S (2011) Intuitive biases in choice vs. estimation: Implications for the wisdom of crowds. J. Consumer Res. 38(1):1-15.
40. Soll JB, Larrick RP (2009) Strategies for revising judgment: How (and how well) people use others' opinions. J. Experiment. Psych.: Learn., Memory and Cognition 35(3):780-805.
41. Sunstein CR (2006) Infotopia: How Many Minds Produce Knowledge (Oxford University Press, New York).
42. Surowiecki J (2004) The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies, and Nations (Little, Brown, London).
43. Tetlock PE (2005) Expert Political Opinion, How Good Is It? How Can We Know? (Princeton University Press, Princeton, NJ).
44. Turner BM, Steyvers M, Merkle EC, Budescu DV, Wallsten TS (2014) Forecast aggregation via recalibration. Machine Learn. 95(3):261-289.
45. Wallsten TS, Budescu DV (1983) Encoding subjective probabilities: A psychological and psychometric review. Management Sci. 29(2):151-173.
46. Wallsten TS, Diederich A (2001) Understanding pooled subjective probability estimates. Math. Soc. Sci. 41(1):1-18.
47. Wallsten TS, Budescu DV, Zwick R (1993) Comparing the calibration and coherence of numerical and verbal probability judgments. Management Sci. 39(2):176-190.
48. Wallsten TS, Budescu DV, Erev I, Diederich A (1997) Evaluating and combining subjective probability estimates. J. Behavioral Decision Making 10(3):243-268.
49. Wang G, Kulkarni SR, Poor HV, Osherson DN (2011) Aggregating large sets of probabilistic forecasts by weighted coherent adjustment. Decision Anal. 8(2):28-144.