Knowing the crowd within: Metacognitive limits on combining multiple judgments

Journal of Memory and Language

Volumn 71, Issue 1, 2014, Pages 17-38

  Fraundorf, Scott H ^a   Benjamin, Aaron S ^a  

^a UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

Author keywords

Judgment and decision making; Metacognition; Subjective fluency

Indexed keywords

EID: 84887902342     PISSN: 0749596X     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jml.2013.10.002     Document Type: Article

References (53)

1. Alter A.L., Oppenheimer D.M. Uniting the tribes of fluency to form a metacognitive nation. Personality and Social Psychology Review 2009, 13:219-235. 10.1177/108868309341564.
2. Benjamin A.S., Bjork R.A., Hirshman E. Predicting the future and reconstructing the past: A Bayesian characterization of the utility of subjective fluency. Acta Psychologica 1998, 98:267-290. 10.1016/S0001-6918(97)00046-2.
3. Benjamin A.S., Bjork R.A. Retrieval fluency as a metacognitive index. Implicit memory and metacognition 1996, 309-338. Erlbaum, Mahwah, NJ. L. Reder (Ed.).
4. Benjamin A.S., Bjork R.A., Schwartz B.L. The mismeasure of memory: When retrieval fluency is misleading as a metacognitive index. Journal of Experimental Psychology; General 1998, 127:55-68. 10.1037/0096-3445.127.1.55.
5. Bonaccio S., Dalal R.S. Advice taking and decision-making: An integrative literature review, and implications for the organizational sciences. Organizational Behavior and Human Performance 2006, 101:127-151. 10.1016/j.obhdp.2006.07.001.
6. Brown-Schmidt S., Hanna J.E. Talking in another person's shoes: Incremental perspective-taking in language processing. Dialogue & Discourse 2011, 2:11-33. 10.5087/dad.2011.102.
7. Central Intelligence Agency (2011). The world factbook. <>. https://www.cia.gov/library/publications/the-world-factbook/.
8. Craik F.I.M. On the transfer of information from temporary to permanent memory. Philosophical Transactions of the Royal Society of London, Series B 1983, 302:341-359. 10.1098/rstb.1983.0059.
9. Einhorn H.J., Hogarth R.M. Unit weighting schemes for decision making. Organizational Behavior and Human Performance 1975, 13:171-192. 10.1016/0030-5073(75)90044-6.
10. Finley, J. R., & Benjamin, A. S. (2012). Retrieval cue variability: When and why are two meanings better than one? (submitted for publication).
11. Friedman M.P., Burke C.J., Cole M., Keller L., Millward R.B., Estes W.K. Two-choice behavior under extended training with shifting probabilities of reinforcement. Studies in mathematical psychology 1964, Vol. 9:250-316. Stanford University Press, Stanford, CA.
12. Galton F. Vox populi. Nature 1907, 75:450-451. 10.1038/075450a0.
13. Gigerenzer G., Goldstein D.G. Reasoning the fast and frugal way: Models of bounded rationality. Psychological Review 1996, 103:650-669. 10.1037/0033-295X.103.4.650.
14. Harvey N., Fischer I. Taking advice: Accepting help, improving judgment, and sharing responsibility. Organizational Behavior and Human Decision Processes 1997, 70:117-133. 10.1006/obhd.1997.2697.
15. Harvey N., Harries C. Effects of judges' forecasting on their later combination of forecasts for the same outcomes. International Journal of Forecasting 2003, 20:391-409. 10.1016/j.ijforecast.2003.09.012.
16. Herzog S.M., Hertwig R. The crowd within and the benefits of dialectical bootstrapping: A reply to White and Antonakis. Psychological Science 2013, 24:117-119. 10.1177/0956797612457399.
17. Herzog S.M., Hertwig R. The wisdom of many in one mind: Improving individual judgments with dialectical bootstrapping. Psychological Science 2009, 20:231-237. 10.1111/j.1467-9280.2009.02271.x.
18. Hogarth R.M., Einhorn H.J. Order effects in belief updating: The belief-adjustment model. Cognitive Psychology 1992, 24:1-55. 10.1016/0010-0285(92)90002-J.
19. Hourihan K.L., Benjamin A.S. Smaller is better (when sampling from the crowd within): Low memory-span individuals benefit more from multiple opportunities from estimation. Journal of Experimental Psychology: Learning, Memory, and Cognition 2010, 36:1068-1074. 10.1037/a0019694.
20. Jacoby L.L., Kelley C.M. Unconscious influences of memory for a prior event. Personality and Social Psychology Bulletin 1987, 13:314-336. 10.1177/0146167287133003.
21. Jacoby L.L., Whitehouse K. An illusion of memory: False recognition influenced by unconscious perception. Journal of Experimental Psychology: General 1989, 118:126-135. 10.1037/0096-3445.118.2.126.
22. Kelley C.M., Jacoby L.L. Adult egocentrism: Subjective experience versus analytic bases for judgment. Journal of Memory and Language 1996, 35:157-175. 10.1006/jmla.1996.0009.
23. Keysar B., Barr D.J., Balin J.A., Brauner J.S. Taking perspective in conversation: The role of mutual knowledge in comprehension. Psychological Science 2000, 11:32-38. 10.1111/1467-9280.00211.
24. Koriat A. How do we know that we know? The accessibility model of the feeling of knowing. Psychological Review 1993, 100:609-639. 10.1037/0033-295X.100.4.609.
25. Koriat A. Monitoring one's own knowledge during study: A cue-utilization approach to judgments of learning. Journal of Experimental Psychology: General 1997, 126:349-370. 10.1037/0096-3445.126.4.349.
26. Koriat A. The self-consistency model of subjective confidence. Psychological Review 2012, 119:80-113. 10.1037/a0025648.
27. Koriat A., Bjork R.A., Sheffer L., Bar S.K. Predicting one's own forgetting: The role of experience-based and theory-based processing. Journal of Experimental Psychology; General 2004, 153:643-656. 10.1037/0096-3445.133.4.643.
28. Kornell N., Bjork R.A. A stability bias in human memory: Overestimating remembering and underestimating learning. Journal of Experimental Psychology: General 2009, 138:449-468. 10.1037/a0017350.
29. Kornell N., Rhodes M.G., Castel A.D., Tauber S.K. The ease-of-processing heuristic and the stability bias: Dissociating memory, memory beliefs, and memory judgments. Psychological Science 2011, 22:787-794. 10.1177/0956797611407929.
30. Krueger J.I. Return of the ego-Self-referent information as a filter for social prediction: Comment on Karniol (2003). Psychological Review 2003, 110:585-590. 10.1037/0033-295X.110.3.585.
31. Larrick R.P., Soll J.B. Intuitions about combining opinions: Misappreciation of the averaging principle. Management Science 2006, 52:111-127. 10.1287/mnsc.1050.0459.
32. Lim J.S., O'Connor M. Judgemental adjustment of initial forecasts: Its effectiveness and biases. Journal of Behavioral Decision Making 1995, 8:149-168. 10.1002/bdm.3960080302.
33. Lovie A.D., Lovie P. The flat maximum effect and linear scoring models for prediction. Journal of Forecasting 1986, 5:159-168. 10.1002/for.39850050303.
34. Mannes A.E. Are we wise about the wisdom of crowds? The use of group judgments in belief revision. Management Science 2009, 55:1267-1279. 10.1287/mnsc.1090.1031.
35. Mozer M.C., Pashler H., Homaei H. Optimal predictions in everyday cognition: The wisdom of indivdiuals or crowds?. Cognitive Science 2008, 32:1133-1147. 10.1080/03640210802353016.
36. Müller-Trede J. Repeated judgment sampling: Boundaries. Judgment and Decision Making 2011, 6:283-294.
37. Oppenheimer D.M. The secret life of fluency. Trends in Cognitive Sciences 2008, 12:237-241. 10.1016/j.tics.2008.02.014.
38. Payne J.W., Bettman J.R., Johnson E.J. Adaptive strategy selection in decision making. Journal of Experimental Psychology: Learning, Memory, and Cognition 1988, 14:534-552.
39. Peterson W.W., Birdsall T.G., Fox W.C. The theory of signal detectability. IRE Professional Group on Information Theory PGIT-4 1954, 171-212.
40. Postman L., Phillips L.W. Short-term temporal changes in free recall. Quarterly Journal of Experimental Psychology 1965, 17:132-138. 10.1080/17470216508416422.
41. Rauhut H., Lorenz J. The wisdom of crowds in one mind: How individuals can simulate the knowledge of diverse societies to reach better decisions. Journal of Mathematical Psychology 2010, 55:191-197. 10.1016/j.jmp.2010.10.002.
42. Rawson K.A., Dunlosky J., McDonald S.L. Influences of metamemory on performance predictions for text. The Quarterly Journal of Experimental Psychology 2002, 55A:505-524. 10.1080/02724980143000352.
43. Soll J.B. Intuitive theories of information: Beliefs about the value of redundancy. Cognitive Psychology 1999, 38:317-346. http://dx.doi.org/cogp.1998.0699.
44. Soll J.B., Larrick R.P. Strategies for revising judgment: How (and how well) people use others' opinions. Journal of Experimental Psychology: Learning, Memory, and Cognition 2009, 35:780-805. 10.1037/a0015145.
45. Surowiecki J. The wisdom of crowds: Why the many are smarter than the few and how collective wisdom shapes business, economies, societies, and nations 2004, Random House, New York.
46. Tullis J.G., Benjamin A.S. The effectiveness of updating metacognitive knowledge in the elderly: Evidence from metamnemonic judgments of word frequency. Psychology and Aging 2012, 27:683-690. 10.1037/a0025838.
47. Tulving E., Thomson D.M. Encoding specificity and retrieval processes in episodic memory. Psychological Review 1973, 80:353-373. 10.1037/h0020071.
48. Vul E., Pashler H. Measuring the crowd within: Probabilistic representations within individuals. Psychologial Science 2008, 19:645-647. 10.1111/j.1467-9280.2008.02136.x.
49. Welsh M.B., Lee M.D., Begg S.H. Repeated judgments in elicitation tasks: Efficacy of the MOLE method. Proceedings of the 30th annual conference of the Cognitive Science Society 2009, 64-70. Cognitive Science Society, Austin, TX. B.C. Love, K. McRae, V.M. Sloutsky (Eds.).
50. White C.M., Antonakis J. Quantifying accuracy improvements in sets of pooled judgments: Does dialectical bootstrapping work?. Psychological Science 2013, 24:115-116. 10.1177/0956797612449174.
51. Whittlesea B.W.A., Jacoby L.L., Girard K. Illusions of immediate memory: Evidence of an attributional basis for feelings of familiarity and perceptual quality. Journal of Memory and Language 1990, 29:716-732. 10.1016/0749-596X(90)90045-2.
52. Yaniv I. The benefit of additional opinions. Current Directions in Psychological Science 2004, 13:75-78. 10.1111/j.0963-7214.2004.00278.x.
53. Yaniv I., Choshen-Hillel S. Exploiting the wisdom of others to make better decisions: Suspending judgment reduces egocentrism and increases accuracy. Journal of Behavioral Decision Making 2012, 25:427-434. 10.1002/bdm.740.