The Cognitive Reflection Test as a predictor of performance on heuristics-and-biases tasks

Memory and Cognition

Volumn 39, Issue 7, 2011, Pages 1275-1289

  Toplak, Maggie E ^a   West, Richard F ^b   Stanovich, Keith E ^c  

^a YORK UNIVERSITY   (Canada)

^b JAMES MADISON UNIVERSITY   (United States)

^c UNIVERSITY OF TORONTO   (Canada)

Author keywords

Cognitive reflection test; Heuristics and biases; Intelligence; Rational thinking; Thinking dispositions

Indexed keywords

ADULT; ARTICLE; EXECUTIVE FUNCTION; FEMALE; HUMAN; INSTRUMENTATION; INTELLIGENCE; MALE; NEUROPSYCHOLOGICAL TEST; PHYSIOLOGY; PREDICTIVE VALUE; PSYCHOMETRY; REGRESSION ANALYSIS; THINKING;

ADULT; EXECUTIVE FUNCTION; FEMALE; HUMANS; INTELLIGENCE; MALE; NEUROPSYCHOLOGICAL TESTS; PREDICTIVE VALUE OF TESTS; PSYCHOMETRICS; REGRESSION ANALYSIS; THINKING; YOUNG ADULT;

EID: 84856413981     PISSN: 0090502X     EISSN: 15325946     Source Type: Journal    
DOI: 10.3758/s13421-011-0104-1     Document Type: Article

References (116)

1. Ackerman, P. L. (1994). Intelligence, attention, and learning: Maximal and typical performance. In D. K. Detterman (Ed.), Current topics in human intelligence (Vol. 4, pp. 1-27). Norwood: Ablex.
2. Ackerman, P. L. (1996). A theory of adult development: Process, personality, interests, and knowledge. Intelligence, 22, 227-257.
3. Ackerman, P. L., & Heggestad, E. D. (1997). Intelligence, personality, and interests: Evidence for overlapping traits. Psychological Bulletin, 121, 219-245.
4. Ackerman, P. L., & Kanfer, R. (2004). Cognitive, affective, and conative aspects of adult intellect within a typical and maximal performance framework. In D. Y. Dai & R. J. Sternberg (Eds.), Motivation, emotion, and cognition: Integrative perspectives on intellectual functioning and development (pp. 119-141). Mahwah: Erlbaum.
5. Aron, A. R. (2008). Progress in executive-function research: From tasks to functions to regions to networks. Current Directions in Psychological Science, 17, 124-129.
6. Baron, J. (2008). Thinking and deciding (4th ed.). New York: Cambridge University Press.
7. Baron, J., & Hershey, J. C. (1988). Outcome bias in decision evaluation. Journal of Personality and Social Psychology, 54, 569-579.
8. Best, J. R., Miller, P. H., & Jones, L. L. (2009). Executive functions after age 5: Changes and correlates. Developmental Review, 29, 180-200.
9. Beyth-Marom, R., & Fischhoff, B. (1983). Diagnosticity and pseudodiagnositicity. Journal of Personality and Social Psychology, 45, 1185-1195.
10. Bishop, M. A., & Trout, J. D. (2005). Epistemology and the psychology of human judgment. Oxford: Oxford University Press.
11. Bruine de Bruin, W., Parker, A. M., & Fischhoff, B. (2007). Individual differences in adult decision-making competence. Journal of Personality and Social Psychology, 92, 938-956.
12. Cacioppo, J. T., Petty, R. E., Feinstein, J., & Jarvis, W. (1996). Dispositional differences in cognitive motivation: The life and times of individuals varying in need for cognition. Psychological Bulletin, 119, 197-253.
13. Campitelli, G., & Labollita, M. (2010). Correlations of cognitive reflection with judgments and choices. Judgment and Decision Making, 5, 182-191.
14. Cokely, E. T., & Kelley, C. M. (2009). Cognitive abilities and superior decision making under risk: A protocol analysis and process model evaluation. Judgment and Decision Making, 4, 20-33.
15. Cronbach, L. J. (1949). Essentials of psychological testing. New York: Harper.
16. Dawes, R. M. (1976). Shallow psychology. In J. S. Carroll & J. W. Payne (Eds.), Cognition and social behavior (pp. 3-11). Hillsdale: Erlbaum.
17. Del Missier, F., Mantyla, T., & Bruine de Bruin, W. (2010). Executive functions in decision making: An individual differences approach. Thinking & Reasoning, 16, 69-97.
18. Denes-Raj, V., & Epstein, S. (1994). Conflict between intuitive and rational processing: When people behave against their better judgment. Journal of Personality and Social Psychology, 66, 819-829.
19. De Neys, W., & Glumicic, T. (2008). Conflict monitoring in dual process theories of thinking. Cognition, 106, 1248-1299.
20. Dohmen, T., Falk, A., Huffman, D., Marklein, F., & Sunde, U. (2009). Biased probability judgment: Evidence of incidence and relationship to economic outcomes from a representative sample. Journal of Economic Behavior and Organization, 72, 903-915.
21. Duckworth, A. L. (2009). Over and beyond high-stakes testing. American Psychologist, 64, 279-280.
22. Duncan, J., Parr, A., Woolgar, A., Thompson, R., Bright, P., Cox, S., et al. (2008). Goal neglect and Spearman's g: Competing parts of a complex task. Journal of Experimental Psychology: General, 137, 131-148.
23. Epstein, S., & Meier, P. (1989). Constructive thinking: A broad coping variable with specific components. Journal of Personality and Social Psychology, 57, 332-350.
24. Evans, J. St. B. T. (1984). Heuristic and analytic processes in reasoning. British Journal of Psychology, 75, 451-468.
25. Evans, J. St. B. T. (2003). In two minds: Dual-process accounts of reasoning. Trends in Cognitive Sciences, 7, 454-459.
26. Evans, J. St. B. T. (2008). Dual-processing accounts of reasoning, judgment and social cognition. Annual Review of Psychology, 59, 255-278.
27. Evans, J. St. B. T. (2010). Thinking twice: Two minds in one brain. Oxford: Oxford University Press.
28. Evans, J. St. B. T., Barston, J., & Pollard, P. (1983). On the conflict between logic and belief in syllogistic reasoning. Memory & Cognition, 11, 295-306.
29. Evans, J. St. B. T., & Frankish, K. (Eds.). (2009). In two minds: Dual processes and beyond. Oxford: Oxford University Press.
30. Evans, J. St. B. T., & Over, D. E. (1996). Rationality and reasoning. Hove: Psychology Press.
31. Finucane, M. L., & Gullion, C. M. (2010). Developing a tool for measuring the decision-making competence of older adults. Psychology and Aging, 25, 271-288.
32. Fong, G. T., Krantz, D. H., & Nisbett, R. E. (1986). The effects of statistical training on thinking about everyday problems. Cognitive Psychology, 18, 253-292.
33. Frederick, S. (2005). Cognitive reflection and decision making. Journal of Economic Perspectives, 19, 25-42.
34. Friedman, N. P., Haberstick, B. C., Willcutt, E. G., Miyake, A., Young, S. E., Corley, R. P., et al. (2007). Greater attention problems during childhood predict poorer executive functioning in late adolescence. Psychological Science, 18, 893-900.
35. Friedman, N. P., Miyake, A., Corley, R. P., Young, S. E., DeFries, J. C., & Hewitt, J. K. (2006). Not all executive functions are related to intelligence. Psychological Science, 17, 172-179.
36. Frisch, D. (1993). Reasons for framing effects. Organizational Behavior and Human Decision Processes, 54, 399-429.
37. Gal, I., & Baron, J. (1996). Understanding repeated simple choices. Thinking & Reasoning, 2, 81-98.
38. George, C. (1995). The endorsement of the premises: Assumption-based or belief-based reasoning. British Journal of Psychology, 86, 93-111.
39. Gilhooly, K. J., & Fioratou, E. (2009). Executive functions in insight versus non-insight problem solving: An individual differences approach. Thinking and Reasoning, 15, 355-376.
40. Gilhooly, K. J., & Murphy, P. (2005). Differentiating insight from non-insight problems. Thinking & Reasoning, 11, 279-302.
41. Gilovich, T., Griffin, D., & Kahneman, D. (Eds.). (2002). Heuristics and biases: The psychology of intuitive judgment. New York: Cambridge University Press.
42. Gonzalez, R., Grant, I., Miller, W., Taylor, M. J., Schweinsburg, B. C., Carey, C. L., et al. (2006). Demographically adjusted normative standards for new indices of performance on the Paced Auditory Serial Addition Task (PASAT). Clinical Neuropsychologist, 20, 396-413.
43. Gronwall, D. M. A. (1977). Paced auditory serial-addition task: A measure of recovery from concussion. Perceptual and Motor Skills, 44, 367-373.
44. Handley, S. J., Capon, A., Beveridge, M., Dennis, I., & Evans, J St B T. (2004). Working memory, inhibitory control and the development of children's reasoning. Thinking & Reasoning, 10, 175-195.
45. Hasher, L., Lustig, C., & Zacks, R. (2007). Inhibitory mechanisms and the control of attention. In A. Conway, C. Jarrold, M. Kane, A. Miyake, & J. Towse (Eds.), Variation in working memory (pp. 227-249). New York: Oxford University Press.
46. Jones, W., Russell, D., & Nickel, T. (1977). Belief in the paranormal scale: An objective instrument to measure belief in magical phenomena and causes. JSAS Catalog of Selected Documents in Psychology, 7(100), Ms. No. 1577.
47. Kahneman, D. (2000). A psychological point of view: Violations of rational rules as a diagnostic of mental processes. Behavioral and Brain Sciences, 23, 681-683.
48. Kahneman, D. (2003). A perspective on judgment and choice: Mapping bounded rationality. American Psychologist, 58, 697-720.
49. Kahneman, D., & Frederick, S. (2002). Representativeness revisited: Attribute substitution in intuitive judgment. In T. Gilovich, D. Griffin, & D. Kahneman (Eds.), Heuristics and biases: The psychology of intuitive judgment (pp. 49-81). New York: Cambridge University Press.
50. Kahneman, D., & Tversky, A. (1982). On the study of statistical intuitions. Cognition, 11, 123-141.
51. Kahneman, D., & Tversky, A. (1984). Choices, values and frames. American Psychologist, 39, 341-350. doi: 10. 1037/0003-066X. 39. 4. 341.
52. Kahneman, D., & Tversky, A. (1996). On the reality of cognitive illusions. Psychological Review, 103, 582-591.
53. Kahneman, D., & Tversky, A. (Eds.). (2000). Choices, values, and frames. Cambridge: Cambridge University Press.
54. Kirkpatrick, L., & Epstein, S. (1992). Cognitive-experiential self-theory and subjective probability: Evidence for two conceptual systems. Journal of Personality and Social Psychology, 63, 534-544.
55. Klaczynski, P. A. (2001). Analytic and heuristic processing influences on adolescent reasoning and decision making. Child Development, 72, 844-861.
56. Koehler, D. J., & James, G. (2010). Probability matching and strategy availability. Memory & Cognition, 38, 667-676.
57. Kokis, J., Macpherson, R., Toplak, M., West, R. F., & Stanovich, K. E. (2002). Heuristic and analytic processing: Age trends and associations with cognitive ability and cognitive styles. Journal of Experimental Child Psychology, 83, 26-52.
58. Lehman, D. R., Lempert, R. O., & Nisbett, R. E. (1988). The effect of graduate training on reasoning. American Psychologist, 43, 431-442.
59. Lieberman, M. D. (2007). Social cognitive neuroscience: A review of core processes. Annual Review of Psychology, 58, 259-289.
60. Lieberman, M. D. (2009). What zombies can't do: A social cognitive neuroscience approach to the irreducibility of reflective consciousness. In J St B T Evans & K. Frankish (Eds.), In two minds: Dual processes and beyond (pp. 293-316). Oxford: Oxford University Press.
61. Matthews, G., Zeidner, M., & Roberts, R. D. (2002). Emotional intelligence: Science & myth. Cambridge: MIT Press.
62. Miyake, A., Friedman, N., Emerson, M. J., & Witzki, A. H. (2000). The utility and diversity of executive functions and their contributions to complex "frontal lobe" tasks: A latent variable analysis. Cognitive Psychology, 41, 49-100.
63. Norris, S. P., & Ennis, R. H. (1989). Evaluating critical thinking. Pacific Grove: Midwest Publications.
64. Obrecht, N. A., Chapman, G. B., & Gelman, R. (2009). An encounter frequency account of how experience affects likelihood estimation. Memory & Cognition, 37, 632-643.
65. Oechssler, J., Roider, A., & Schmitz, P. W. (2009). Cognitive abilities and behavioral biases. Journal of Economic Behavior & Organization, 72, 147-152.
66. Parker, A. M., & Fischhoff, B. (2005). Decision-making competence: External validation through an individual differences approach. Journal of Behavioral Decision Making, 18, 1-27.
67. Pedhazur, E. J. (1997). Multiple regression in behavioral research: Explanation and prediction (3rd ed.). Fort Worth: Harcourt Brace.
68. Perkins, D. N. (1995). Outsmarting IQ: The emerging science of learnable intelligence. New York: Free Press.
69. Reitan, R. M. (1955). The relation of the Trail Making Test to organic brain damage. Journal of Consulting Psychology, 19, 393-394. doi: 10. 1037/h0044509.
70. Reitan, R. M. (1958). Validity of the trail making test as an indicator of organic brain damage. Perceptual and Motor Skills, 8, 271-276.
71. Reyna, V. F. (1991). Class inclusion, the conjunction fallacy, and other cognitive illusions. Developmental Review, 11, 317-336.
72. Reyna, V. F., & Brainerd, C. J. (1994). The origins of probability judgment: A review of data and theories. In G. Wright & P. Ayton (Eds.), Subjective probability (pp. 239-272). New York: Wiley.
73. Reyna, V. F., & Brainerd, C. J. (2008). Numeracy, ratio bias, and denominator neglect in judgments of risk and probability. Learning and Individual Differences, 18, 89-107.
74. Reyna, V. F., Lloyd, F. J., & Brainerd, C. J. (2003). Memory, development, and rationality: An integrative theory of judgment and decision making. In S. L. Schneider & J. Shanteau (Eds.), Emerging perspectives on judgment and decision research (pp. 201-245). New York: Cambridge University Press.
75. Sá, W., West, R. F., & Stanovich, K. E. (1999). The domain specificity and generality of belief bias: Searching for a generalizable critical thinking skill. Journal of Educational Psychology, 91, 497-510.
76. Sackett, P. R., Zedeck, S., & Fogli, L. (1988). Relations between measures of typical and maximum job performance. Journal of Applied Psychology, 73, 482-486.
77. Salthouse, T. A., Atkinson, T. M., & Berish, D. E. (2003). Executive functioning as a potential mediator of age-related cognitive decline in normal adults. Journal of Experimental Psychology: General, 132, 566-594.
78. Samuels, R., & Stich, S. P. (2004). Rationality and psychology. In A. R. Mele & P. Rawling (Eds.), The Oxford handbook of rationality (pp. 279-300). Oxford: Oxford University Press.
79. Shafir, E., & LeBoeuf, R. A. (2002). Rationality. Annual Review of Psychology, 53, 491-517.
80. Simon, H. A. (1955). A behavioral model of rational choice. Quarterly Journal of Economics, 69, 99-118.
81. Simon, H. A. (1956). Rational choice and the structure of the environment. Psychological Review, 63, 129-138.
82. Sloman, S. A. (1996). The empirical case for two systems of reasoning. Psychological Bulletin, 119, 3-22.
83. Sloman, S. A. (2002). Two systems of reasoning. In T. Gilovich, D. Griffin, & D. Kahneman (Eds.), Heuristics and biases: The psychology of intuitive judgment (pp. 379-396). New York: Cambridge University Press.
84. Slugoski, B. R., Shields, H. A., & Dawson, K. A. (1993). Relation of conditional reasoning to heuristic processing. Personality and Social Psychology Bulletin, 19, 158-166.
85. Stanovich, K. E. (1989). Implicit philosophies of mind-The dualism scale and its relation to religiosity and belief in extrasensory perception. Journal of Psychology, 123, 5-23.
86. Stanovich, K. E. (1999). Who is rational? Studies of individual differences in reasoning. Mahwah: Erlbaum.
87. Stanovich, K. E. (2004). The robot's rebellion: Finding meaning in the age of Darwin. Chicago: University of Chicago Press.
88. Stanovich, K. E. (2008). Higher-order preferences and the master rationality motive. Thinking & Reasoning, 14, 111-127.
89. Stanovich, K. E. (2009a). Distinguishing the reflective, algorithmic, and autonomous minds: Is it time for a tri-process theory? In J. Evans & K. Frankish (Eds.), In two minds: Dual processes and beyond (pp. 55-88). Oxford: Oxford University Press.
90. Stanovich, K. E. (2009b). What intelligence tests miss: The psychology of rational thought. New Haven: Yale University Press.
91. Stanovich, K. E. (2011). Rationality and the reflective mind. New York: Oxford University Press.
92. Stanovich, K. E., Toplak, M. E., & West, R. F. (2008). The development of rational thought: A taxonomy of heuristics and biases. Advances in child development and behavior, 36, 251-285.
93. Stanovich, K. E., & West, R. F. (1997). Reasoning independently of prior belief and individual differences in actively open-minded thinking. Journal of Educational Psychology, 89, 342-357.
94. Stanovich, K. E., & West, R. F. (1998a). Cognitive ability and variation in selection task performance. Thinking and Reasoning, 4, 193-230.
95. Stanovich, K. E., & West, R. F. (1998b). Individual differences in framing and conjunction effects. Thinking and Reasoning, 4, 289-317.
96. Stanovich, K. E., & West, R. F. (1998c). Individual differences in rational thought. Journal of Experimental Psychology: General, 127, 161-188.
97. Stanovich, K. E., & West, R. F. (1998d). Who uses base rates and P(D/~H)? An analysis of individual differences. Memory & Cognition, 26, 161-179.
98. Stanovich, K. E., & West, R. F. (1999). Discrepancies between normative and descriptive models of decision making and the understanding/acceptance principle. Cognitive Psychology, 38, 349-385.
99. Stanovich, K. E., & West, R. F. (2000). Individual differences in reasoning: Implications for the rationality debate? Behavioral and Brain Sciences, 23, 645-726.
100. Stanovich, K. E., & West, R. F. (2007). Natural myside bias is independent of cognitive ability. Thinking & Reasoning, 13, 225-247. doi: 10. 1080/13546780600780796.
101. Stanovich, K. E., & West, R. F. (2008a). On the failure of intelligence to predict myside bias and one-sided bias. Thinking & Reasoning, 14, 129-167.
102. Stanovich, K. E., & West, R. F. (2008b). On the relative independence of thinking biases and cognitive ability. Journal of Personality and Social Psychology, 94, 672-695.
103. Stanovich, K. E., West, R. F., & Toplak, M. E. (2011). Intelligence and rationality. In R. J. Sternberg & S. B. Kaufman (Eds.), Cambridge handbook of intelligence. New York: Cambridge University Press.
104. Sternberg, R. J. (2003). Wisdom, intelligence, and creativity synthesized. Cambridge: Cambridge University Press.
105. Strathman, A., Gleicher, F., Boninger, D. S., & Scott Edwards, C. (1994). The consideration of future consequences: Weighing immediate and distant outcomes of behavior. Journal of Personality and Social Psychology, 66, 742-752.
106. Strauss, E., Sherman, E. M. S., & Spreen, O. (2006). A compendium of neuropsychological tests (3rd ed.). New York: Oxford University Press.
107. Taylor, S. E. (1981). The interface of cognitive and social psychology. In J. H. Harvey (Ed.), Cognition, social behavior, and the environment (pp. 189-211). Hillsdale: Erlbaum.
108. Toplak, M., Liu, E., Macpherson, R., Toneatto, T., & Stanovich, K. E. (2007). The reasoning skills and thinking dispositions of problem gamblers: A dual-process taxonomy. Journal of Behavioral Decision Making, 20, 103-124.
109. Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185, 1124-1131.
110. Tversky, A., & Kahneman, D. (1981). The framing of decisions and the psychology of choice. Science, 211, 453-458.
111. Tversky, A., & Kahneman, D. (1983). Extensional versus intuitive reasoning: The conjunction fallacy in probability judgment. Psychological Review, 90, 293-315.
112. Wechsler, D. (1999). Wechsler abbreviated scale of intelligence (WASI). San Antonio: Harcourt Brace, Psychological Corp.
113. West, R. F., & Stanovich, K. E. (2003). Is probability matching smart? Associations between probabilistic choices and cognitive ability. Memory & Cognition, 31, 243-251.
114. West, R. F., Toplak, M. E., & Stanovich, K. E. (2008). Heuristics and biases as measures of critical thinking: Associations with cognitive ability and thinking dispositions. Journal of Educational Psychology, 100, 930-941.
115. Zeidner, M., & Matthews, G. (2000). Intelligence and personality. In R. J. Sternberg (Ed.), Handbook of intelligence (pp. 581-610). New York: Cambridge University Press.
116. Zelazo, P. D. (2004). The development of conscious control in childhood. Trends in Cognitive Sciences, 8, 12-17.