Strategies for Revising Judgment: How (and How Well) People Use Others' Opinions

Journal of Experimental Psychology: Learning Memory and Cognition

Volumn 35, Issue 3, 2009, Pages 780-805

  Soll, Jack B ^a   Larrick, Richard P ^a  

^a DUKE UNIVERSITY   (United States)

Author keywords

advice taking; averaging forecasts and estimates; combining opinions; group decision making; social influence

Indexed keywords

ARTICLE; ASSOCIATION; CONCEPT FORMATION; CULTURAL ANTHROPOLOGY; DECISION MAKING; FEEDBACK SYSTEM; HUMAN; INTUITION; LEARNING; PEER GROUP; SOCIAL ENVIRONMENT;

CHOICE BEHAVIOR; CONCEPT FORMATION; CUES; CULTURE; DECISION MAKING; FEEDBACK; HUMANS; INTUITION; JUDGMENT; PEER GROUP; PROBABILITY LEARNING; SOCIAL ENVIRONMENT;

EID: 65749115325     PISSN: 02787393     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0015145     Document Type: Article

References (143)

1. Allen V.L. Situational factors in conformity. In: Berkowitz L. (Ed). Advances in experimental social psychology 2 (1965), Academic Press, New York 133-175
2. Allison P.D. Logistic regression using SAS: Theory and application (1999), SAS Institute, Cary, NC
3. Anderson L.R., and Holt C.A. Information cascades in the laboratory. American Economic Review 87 (1997) 847-862
4. Ariely D., Au W.T., Bender R.H., Budescu D.V., Dietz C.B., Gu H.B., et al. The effects of averaging subjective probability estimates between and within judges. Journal of Experimental Psychology: Applied 6 (2000) 130-147
5. Armstrong J.S. Combining forecasts. In: Armstrong J.S. (Ed). Principles of forecasting: A handbook for researchers and practitioners (2001), Kluwer, New York 417-439
6. Armstrong J.S., and Collopy F. Error measures for generalizing about forecasting methods: Empirical comparisons. International Journal of Forecasting 8 (1992) 69-80
7. Asch S.E. Social psychology (1952), Prentice Hall, Oxford, United Kingdom
8. Bettman J.R., Johnson E.J., Luce M.F., and Payne J.W. Correlation, conflict, and choice. Journal of Experimental Psychology: Learning, Memory, and Cognition 19 (1993) 931-951
9. Birnbaum M.H. Intuitive numerical prediction. American Journal of Psychology 89 (1976) 417-429
10. Birnbaum M.H., and Stegner S.E. Source credibility in social judgment: Bias, expertise, and the judge's point of view. Journal of Personality and Social Psychology 37 (1979) 48-74
11. Birnbaum M.H., Wong R., and Wong L.K. Combining information from sources that vary in credibility. Memory & Cognition 4 (1976) 330-336
12. Bonaccio S., and Dalal R.S. Advice taking and decision-making: An integrative literature review, and implications for the organizational sciences. Organizational Behavior and Human Decision Processes 101 (2006) 127-151
13. Bonner B.L. Expertise in group problem solving: Recognition, social combination, and performance. Group Dynamics: Theory, Research, and Practice 8 (2004) 277-280
14. Brehmer B. Social judgment theory and analysis of interpersonal conflict. Psychological Bulletin 83 (1976) 985-1003
15. Budescu D.V. Confidence in aggregation of opinions from multiple sources. In: Fiedler K., and Juslin P. (Eds). Information sampling and adaptive cognition (2006), Cambridge University Press, Cambridge, United Kingdom
16. Budescu D.V., Rantilla A.K., Yu H.-T., and Karelitz T.M. The effects of asymmetry among advisors on the aggregation of their opinions. Organizational Behavior and Human Decision Processes 90 (2003) 178-194
17. Burson K.A., Larrick R.P., and Klayman J. Skilled or unskilled, but still unaware of it: How perceptions of difficulty drive miscalibration in relative comparisons. Journal of Personality and Social Psychology 90 (2006) 60-77
18. Burt R.S. Structural holes (1992), Harvard University Press, Cambridge, MA
19. Burt R.S. Structural holes and good ideas. The American Journal of Sociology 110 (2004) 349-399
20. Camerer C.F., and Johnson E.J. The process-performance paradox in expert judgment: How can experts know so much and predict so badly?. In: Ericsson K.A., and Smith J. (Eds). Toward a general theory of expertise: Prospects and limits (1991), Cambridge University Press, Cambridge, United Kingdom
21. Campbell D.T. Conformity in psychology's theories of acquired behavioral dispositions. In: Berg I.A., and Bass B.M. (Eds). Conformity and deviation (1961), Harper, New York 100-142
22. Chambers J.R., and Windschitl P.D. Biases in social comparative judgments: The role of nonmotivated factors in above-average and comparative-optimism effects. Psychological Bulletin 130 (2004) 813-838
23. Chapman G.B., and Johnson E.J. Anchoring, activation and the construction of values. Organizational Behavior and Human Decision Processes 79 (1999) 115-153
24. Chater N., Oaksford M., Nakisa R., and Redington M. Fast, frugal, and rational: How rational norms explain behavior. Organizational Behavior and Human Decision Processes 90 (2003) 63-86
25. Chi M.T.H., Glaser R., and Farr M.J. The nature of expertise (1988), Erlbaum, Hillsdale, NJ
26. Clemen R.T. Combining forecasts: A review and annotated bibliography. International Journal of Forecasting 5 (1989) 559-583
27. Clemen R.T., and Winkler R.L. Combining economic forecasts. Journal of Business and Economic Statistics 4 (1986) 39-46
28. Dawes R.M. An inequality concerning correlation of composites vs. Composites of correlations (Methodological Note Vol. 1) (1970), Oregon Research Institute, Eugene, OR
29. Dawes R.M. The robust beauty of improper linear models in decision making. American Psychologist 34 (1979) 571-582
30. Dawes R.M., and Corrigan B. Linear models in decision making. Psychological Bulletin 81 (1974) 95-105
31. Deutsch M., and Gerard H.B. A study of normative and informational social influences upon individual judgment. Journal of Abnormal & Social Psychology 51 (1955) 629-636
32. Diggle P.J., Liang K.Y., and Zeger S.L. The analysis of longitudinal data (1994), Oxford University Press, New York
33. Einhorn H.J., and Hogarth R.M. Unit weighting schemes for decision making. Organizational Behavior and Human Performance 13 (1975) 171-192
34. Einhorn H.J., and Hogarth R.M. Confidence in judgment: Persistence of illusion of validity. Psychological Review 85 (1978) 395-416
35. Einhorn H.J., and Hogarth R.M. Behavioral decision theory: Processes of judgment and choice. Annual Review of Psychology 32 (1981) 53-88
36. Einhorn H.J., Hogarth R.M., and Klempner E. Quality of group judgment. Psychological Bulletin 84 (1977) 158-172
37. Einhorn H.J., Kleinmuntz D.N., and Kleinmuntz B. Linear regression and process-tracing models of judgment. Psychological Review 86 (1979) 465-485
38. Epley N., and Gilovich T. A tale of tuned decks? Anchoring as adjustment and anchoring as activation. In: Koehler D.J., and Harvey N. (Eds). The Blackwell Handbook of judgment and decision making (2004), Blackwell, Oxford, United Kingdom 240-256
39. Epley N., and Gilovich T. The anchoring-and-adjustment heuristic. Psychological Science 17 (2006) 311-318
40. Ericsson K.A., and Smith J. Toward a general theory of expertise: Prospects and limits (1991), Cambridge University Press, Cambridge, United Kingdom
41. Festinger L. A theory of social comparison processes. Human Relations 7 (1954) 117-140
42. Fiedler K. Beware of samples! A cognitive-ecological sampling approach to judgment biases. Psychological Review 107 (2000) 659-676
43. Frederick S. Cognitive reflection and decision making. Journal of Economic Perspectives 19 (2005) 25-42
44. Gigerenzer G. Adaptive thinking: Rationality in the real world (2000), Oxford University Press, New York
45. Gigerenzer G., and Goldstein D.G. Reasoning the fast and frugal way: Models of bounded rationality. Psychological Review 103 (1996) 650-669
46. Gigerenzer G., Switjtink Z., Porter T., Daston L., Beatty J., and Krüger L. The empire of chance: How probability changed science and everyday life (1989), Cambridge University Press, Cambridge, United Kingdom
47. Gigerenzer G., Todd P., and the ABC Research Group. Simple heuristics that make us smart (1999), Oxford University Press, New York
48. Gigone D., and Hastie R. Proper analysis of the accuracy of group judgments. Psychological Bulletin 121 (1997) 149-167
49. Gino F., and Moore D.A. Effects of task difficulty on use of advice. Journal of Behavioral Decision Making 20 (2007) 21-35
50. Goethals G.R., and Nelson R.E. Similarity in the influence process: The belief-value distinction. Journal of Personality and Social Psychology 25 (1973) 117-122
51. Goldberg L.R. Diagnosticians versus diagnostic signs: The diagnosis of psychosis vs. neurosis from MMPI. Psychological Monographs 79 9, Whole No. 602 (1965)
52. Gonzalez R. When words speak louder than actions: Another's evaluations can appear more diagnostic than their decisions. Organizational Behavior and Human Decision Processes 58 (1994) 214-245
53. Goodwin P., and Wright G. Decision analysis for management judgment (1998), Wiley, Chichester, United Kingdom
54. Graham J.R. Is a group of economists better than one? Than none?. Journal of Business 69 (1996) 193-232
55. Harvey N., and Fischer I. Taking advice: Accepting help, improving judgment, and sharing responsibility. Organizational Behavior and Human Decision Processes 70 (1997) 117-133
56. Harvey N., and Harries C. Effects of judges' forecasting on their later combination of forecasts for the same outcomes. International Journal of Forecasting 20 (2004) 391-409
57. Hastie R. Experimental evidence on group accuracy. In: Grofman B., and Owen G. (Eds). Information pooling and group decision making (1986), JAI Press, Greenwich, CT 129-157
58. Hastie R., and Kameda T. The robust beauty of majority rules in group decisions. Psychological Review 112 (2005) 494-508
59. Henry R.A. Improving group judgment accuracy-Information sharing and determining the best member. Organizational Behavior and Human Decision Processes 62 (1995) 190-197
60. Herzog S.M., and Hertwig R. The wisdom of many in one mind: Improving individual judgments with dialectical bootstrapping. Psychological Science 20 (2009) 231-237
61. Hibon M., and Evgeniou T. To combine or not to combine: Selecting among forecasts and their combinations. International Journal of Forecasting 21 (2005) 15-24
62. Hogarth R.M. Methods for aggregating opinions. In: Jungermann H., and de Zeeuw G. (Eds). Decision making and change in human affairs (1977), D. Reidel, Dordrecht-Holland
63. Hogarth R.M. Note on aggregating opinions. Organizational Behavior and Human Performance 21 (1978) 40-46
64. Hogarth R.M. Educating intuition (2002), The University of Chicago Press, Chicago
65. Hogarth R.M., and Karelaia N. "Take-the-best" and other simple strategies: Why and when they work "well" with binary cues. Theory and Decision 61 (2006) 205-249
66. Hogarth R.M., and Karelaia N. Heuristic and linear models of judgment: Matching rules and environments. Psychological Review 114 (2007) 733-758
67. Hutchinson J.W., Kamakura W.A., and Lynch Jr. J.G. Unobserved heterogeneity as an alternative explanation for "reversal" effects in behavioral research. Journal of Consumer Research 27 (2000) 324-344
68. Hyndman R.J., and Koehler A.B. Another look at measures of forecast accuracy. International Journal of Forecasting 22 (2006) 679-688
69. Jacobs R., and Campbell D.T. The perpetuation of an arbitrary tradition through several generations of a laboratory microculture. Journal of Abnormal & Social Psychology 62 (1961) 649-658
70. Johnson T.R., Budescu D.V., and Wallsten T.S. Averaging probability judgments: Monte Carlo analyses of asymptotic diagnostic value. Journal of Behavioral Decision Making 14 (2001) 123-140
71. Kameda T., and Tindale R.S. Groups as adaptive devices: Human docility and group aggregation mechanisms in evolutionary context. In: Schaller M., Simpson J., and Kenrick D. (Eds). Evolution and social psychology (2006), Psychology Press, New York 317-341
72. Keppel G., and Wickens T.D. Design and analysis: A researcher's handbook. 4th ed. (2004), Prentice Hall, Upper Saddle River, NJ
73. Klayman J., and Brown K. Debias the environment instead of the judge: An alternative approach to reducing error in diagnostic (and other) judgment. Cognition 49 (1993) 97-122
74. Klayman J., Soll J.B., González-Vallejo C., and Barlas S. Overconfidence: It depends on how, what, and whom you ask. Organizational Behavior and Human Decision Processes 79 (1999) 216-247
75. Koehler D.J., and Beauregard T.A. Illusion of confirmation from exposure to another's hypothesis. Journal of Behavioral Decision Making 19 (2006) 61-78
76. Krueger J.I. Return of the ego-Self-referent information as a filter for social prediction: Comment on Karniol (2003). Psychological Review 110 (2003) 585-590
77. Krueger J., and Mueller R.A. Unskilled, unaware, or both? The better-than-average heuristic and statistical regression predict errors in estimates of own performance. Journal of Personality and Social Psychology 82 (2002) 180-188
78. Kruger J. Lake Wobegon be gone! The "below-average effect" and the egocentric nature of comparative ability judgments. Journal of Personality and Social Psychology 77 (1999) 221-232
79. Kruger J., and Dunning D. Unskilled and unaware of it: How difficulties in recognizing one's own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology 77 (1999) 1121-1134
80. Larrick R.P., Burson K.A., and Soll J.B. Social comparison and confidence: When thinking you're better than average predicts overconfidence (and when it does not). Organizational Behavior and Human Decision Processes 102 (2007) 76-94
81. Larrick R.P., and Soll J.B. Intuitions about combining opinions: Misappreciation of the averaging principle. Management Science 52 (2006) 111-127
82. Lees C.D., and Triggs T.J. Intuitive prediction: Response strategies underlying cue weights in the relative-weight averaging model. American Journal of Psychology 110 (1997) 317-356
83. Lim J.S., and O'Connor M. Judgmental adjustment of initial forecasts: Its effectiveness and biases. Journal of Behavioral Decision Making 8 (1995) 149-168
84. Littlepage G.E., and Mueller A.L. Recognition and utilization of expertise in problem-solving groups: Expert characteristics and behavior. Group Dynamics: Theory, Research, and Practice 1 (1997) 324-328
85. Littlepage G.E., Robison W., and Reddington K. Effects of task experience and group experience on group performance, member ability, and recognition of expertise. Organizational Behavior and Human Decision Processes 69 (1997) 133-147
86. Littlepage G.E., Schmidt G.W., Whisler E.W., and Frost A.G. An input-process-output analysis of influence and performance in problem-solving groups. Journal of Personality and Social Psychology 69 (1995) 877-889
87. Lorge I., Fox D., Davitz J., and Brenner M. A survey of studies contrasting the quality of group performance and individual performance. Psychological Bulletin 55 (1958) 337-372
88. Makridakis S. Accuracy measures: Theoretical and practical concerns. International Journal of Forecasting 9 (1993) 527-529
89. Makridakis S., and Hibon M. The m3-competition: Results, conclusions and implications. International Journal of Forecasting 16 (2000) 451-476
90. Makridakis S., and Winkler R.L. Averages of forecasts: Some empirical results. Management Science 29 (1983) 987-996
91. Malkiel B.G. A random walk down Wall Street: The time-tested strategy for successful investing (2007), Norton, New York
92. McKenzie C.R.M. The accuracy of intuitive judgment strategies: Covariation assessment and Bayesian inference. Cognitive Psychology 26 (1994) 209-239
93. McKenzie C.R.M. Judgment and decision making. In: Lamberts K., and Goldstone R.L. (Eds). Handbook of cognition (2005), Sage, London 321-338
94. Miner F.C. Group versus individual decision making: An investigation of performance measures, decision strategies, and process losses/gains. Organizational Behavior and Human Performance 33 (1984)
95. Moore D.A. Not so above average after all: When people believe they are worse than average and its implications for theories of bias in social comparison. Organizational Behavior and Human Decision Processes 102 (2007) 42-58
96. Moore D.A., and Klein W.M.P. Use of absolute and comparative performance feedback in absolute and comparative judgments and decisions. Organizational Behavior and Human Decision Processes 107 (2008) 60-74
97. Mussweiler T., and Strack F. Hypothesis-consistent testing and semantic priming in the anchoring paradigm: A selective accessibility model. Journal of Experimental Social Psychology 35 (1999) 136-164
98. Newell B.R., and Shanks D.R. Take the best or look at the rest? Factors influencing "one-reason" decision making. Journal of Experimental Psychology: Learning, Memory, and Cognition 29 (2003) 53-65
99. Nisbett R.E. Rules for reasoning (1993), McGraw Hill, New York
100. Page S.E. The difference: How the power of diversity creates better groups, firms, schools, and societies (2007), Princeton University Press, Princeton, NJ
101. Payne J.W. Task complexity and contingent processing in decision making: An information search and protocol analysis. Organizational Behavior and Human Performance 16 (1976) 366-387
102. Payne J.W., Bettman J.R., and Johnson E.J. Adaptive strategy selection in decision making. Journal of Experimental Psychology: Learning Memory and Cognition 14 (1988) 534-552
103. Payne J.W., Bettman J.R., and Johnson E.J. The adaptive decision maker (1993), Cambridge University Press, New York
104. Peters E., Vastfjall D., Slovic P., Mertz C.K., Mazzocco K., and Dickert S. Numeracy and decision making. Psychological Science 17 (2006) 407-413
105. Raudenbush S.W., and Bryk A.S. Hierarchical linear models: Applications and data analysis methods. 2nd ed. (2002), Sage, Thousand Oaks, CA
106. Rieskamp J., and Otto P.E. SSL: A theory of how people learn to select strategies. Journal of Experimental Psychology: General 135 (2006) 207-236
107. Ross L., and Nisbett R.E. The person and the situation: Perspectives of social psychology (1991), McGraw Hill, New York
108. Russo J.E., and Dosher B.A. Strategies for multiattribute binary choice. Journal of Experimental Psychology: Learning, Memory, and Cognition 9 (1983) 676-696
109. Russo J.E., and Shoemaker P.J.H. Decision traps (1989), Simon & Schuster, New York
110. Shanteau J. Competence in experts: The role of task characteristics. Organizational Behavior and Human Decision Processes 53 (1992) 252-266
111. Sherif M. The psychology of social norms (1936), Harper, New York
112. Simon H.A. A behavioral model of rational choice. Quarterly Journal of Economics 69 (1955) 99-118
113. Simon H.A. Rational choice and the structure of the environment. Psychological Review 63 (1956) 129-138
114. Simon H.A., and Chase W.G. Skill in chess. American Scientist 61 (1973) 394-403
115. Sniezek J.A., and Buckley T. Cueing and cognitive conflict in judge-advisor decision making. Organizational Behavior and Human Decision Processes 62 (1995) 159-174
116. Soll J.B. Intuitive theories of information: Beliefs about the value of redundancy. Cognitive Psychology 38 (1999) 317-346
117. Soll, J. B., Larrick, R. P., & Zhu, X. (2006, June). The wisdom of the crowd in the mind. Paper presented at the Behavioral Decision Research in Management conference. Santa Monica, California.
118. Soll, J. B., Mannes, A., & Larrick, R. P. (2006, November). Averaging opinions: When does it work well and when do people do it? Paper presented at the annual meeting of the Institute for Operations Research and the Management Sciences, Pittsburgh, Pennsylvania.
119. Sorkin R.D., West R., and Robinson D.E. Group performance depends on the majority rule. Psychological Science 9 (1998) 456-463
120. Staël von Holstein C.-A.S. An experiment of probabilistic weather forecasting. Journal of Applied Meteorology 10 (1971) 635-645
121. Stasson M.F., and Hawkes W.G. Effect of group performance on subsequent individual performance: Does influence generalize beyond the issues discussed by the group. Psychological Science 6 (1995) 305-307
122. Stigler S.M. The history of statistics: The measurement of uncertainty before 1900 (1986), Belknap Press, Cambridge, MA
123. Stigler S.M. Statistics on the table (1999), Harvard University Press, Cambridge, MA
124. Surowiecki J. The wisdom of crowds: Why the many are smarter than the few and how collective wisdom shapes business, economies, societies, and nations (2004), Doubleday, New York
125. Tajfel H. Social and cultural factors in perception. In: Lindzey G., and Aronson E. (Eds). The handbook of social psychology. 2nd ed. 3 (1969), Addison-Wesley, Reading, MA 315-394
126. Tetlock P.E. Expert political judgment: How good is it? How can we know? (2005), Princeton University Press, Princeton, NJ
127. Thorngate W. Efficient decision heuristics. Behavioral Science 25 (1980) 219-225
128. Toda M. The design of a fungus-eater: A model of human behavior in an unsophisticated environment. Behavioral Science 7 (1962) 164-183
129. Todd P.M., and Gigerenzer G. Environments that make us smart: Ecological rationality. Current Directions in Psychological Science 16 (2007) 167-171
130. Trotman K.T., Yetton P.W., and Zimmer I.R. Individual and group judgments of internal control-systems. Journal of Accounting Research 21 (1983) 286-292
131. Tversky A., and Kahneman D. Judgment under uncertainty: Heuristics and biases. Science 185 (1974) 1124-1131
132. Van Swol L.M., and Ludutsky C.L. Tell me something I don't know: Decision makers' preference for advisors with unshared information. Communication Research 34 (2007) 297-312
133. von Winterfeldt D., and Edwards W. Decision analysis and behavioral research (1986), Cambridge University Press, Cambridge, United Kingdom
134. Vul E., and Pashler H. Measuring the crowd within: Probabilistic representations within individuals. Psychological Science 19 (2008) 645-647
135. Winkler R.L. Probabilistic prediction: Some experimental results. Journal of the American Statistical Association 66 (1971) 675-685
136. Winkler R.L. Combining probability distributions from dependent information sources. Management Science 27 (1981) 479-488
137. Winkler R.L. Combining forecasts. In: Makridakis S., Andersen A., Carbone R., Fildes R., Hibon M., Lewandowski R., Newton J., Parzen E., and Winkler R. (Eds). The forecasting accuracy of major time series methods (1984), Wiley, Chichester, United Kingdom 289-295
138. Winkler R.L. Bayesian inference and decision. 2nd ed. (2003), Probabilistic Publishing, Gainesville, FL
139. Winkler R.L., and Clemen R.T. Multiple experts vs. multiple methods: Combining correlation assessments. Decision Analysis 1 (2004) 167-176
140. Yaniv I. Receiving other people's advice: Influence and benefit. Organizational Behavior and Human Decision Processes 93 (2004) 1-13
141. Yaniv I., and Kleinberger E. Advice taking in decision making: Egocentric discounting and reputation formation. Organizational Behavior and Human Decision Processes 83 (2000) 260-281
142. Zajonc R.B. A note on group judgements and group size. Human Relations 15 (1962) 177-180
143. Zucker L.G. The role of institutionalization in cultural persistence. American Sociological Review 42 (1977) 726-743