The cognitive reflection test revisited: exploring the ways individuals solve the test

Thinking and Reasoning

Volumn 23, Issue 3, 2017, Pages 207-234

  Szaszi, B ^a   Szollosi, A ^a,b   Palfi, B ^a   Aczel, B ^a  

^a EÖTVÖS LORÁND UNIVERSITY   (Hungary)

^b UNIVERSITY OF NEW SOUTH WALES   (Australia)

Author keywords

Cognitive reflection test; heuristics and biases; process tracing; reasoning; thinking errors

Indexed keywords

EID: 85014569262     PISSN: 13546783     EISSN: 14640708     Source Type: Journal    
DOI: 10.1080/13546783.2017.1292954     Document Type: Article

References (83)

1. Aczel, B., Bago, B., Szollosi, A., Foldes, A., & Lukacs, B., (2015). Measuring individual differences in decision biases:Methodological considerations. Frontiers in Psychology, 6, 1770.
2. Baron, J., (1993). Why teach thinking? An essay. Applied Psychology, 42(3), 191–214.
3. Baron, J., Badgio, P., & Gaskins, I. W., (1986). Cognitive style and its improvement:A normative approach. Advances in the Psychology of Human Intelligence, 3, 173–220.
4. Baron, J., Scott, S., Fincher, K., & Metz, S. E., (2014). Why does the cognitive reflection test (sometimes) predict utilitarian moral judgment (and other things)? Journal of Applied Research in Memory and Cognition, 4, 265–284
5. Barton, S. B., & Sanford, A. J., (1993). A case study of anomaly detection:Shallow semantic processing and cohesion establishment. Memory & Cognition, 21(4), 477–487.
6. Bates, D., Maechler, M., Bolker, B., & Walker, S., (2015). Lme4:Linear mixed-effects models using Eigen and S4. R Package, version 1.1-8. Retrieved from https://cran.r-project.org/web/packages/lme4/index.html
7. Böckenholt, U., (2012). The cognitive-miser response model:Testing for intuitive and deliberate reasoning. Psychometrika, 77(2), 388–399.
8. Brandstätter, E., & Gussmack, M., (2013). The cognitive processes underlying risky choice. Journal of Behavioral Decision Making, 26(2), 185–197.
9. Brosnan, M., Hollinworth, M., Antoniadou, K., & Lewton, M., (2014). Is empathizing intuitive and systemizing deliberative? Personality and Individual Differences, 66, 39–43.
10. Campitelli, G., & Gerrans, P., (2014). Does the cognitive reflection test measure cognitive reflection? A mathematical modeling approach. Memory & Cognition, 42(3), 434–447.
11. Campitelli, G., & Labollita, M., (2010). Correlations of cognitive reflection with judgments and choices. Judgment and Decision Making, 5(3), 182–191.
12. Cokely, E. T., Galesic, M., Schulz, E., Ghazal, S., & Garcia-Retamero, R., (2012). Measuring risk literacy:The Berlin numeracy test. Judgment and Decision Making, 7(1), 25–47.
13. Cokely, E. T., & Kelley, C. M., (2009). Cognitive abilities and superior decision making under risk:A protocol analysis and process model evaluation. Judgment and Decision Making, 4(1), 20–33.
14. Cokely, E. T., Parpart, P., & Schooler, L. J., (2009). On the link between cognitive control and heuristic processes. In N. A., Taatgnen & H. Van, Rijn (Eds.), Proceedings of the 31th annual conference of the cognitive science society (pp. 2926–2931). Austin, TX:Cognitive Science Society.
15. Del Missier, F., Mäntylä, T., & Bruin, W. B., (2012). Decision‐making competence, executive functioning, and general cognitive abilities. Journal of Behavioral Decision Making, 25(4), 331–351.
16. De Neys, W., & Bonnefon, J.-F., (2013). The “whys” and “whens” of individual differences in thinking biases. Trends in Cognitive Sciences, 17(4), 172–178.
17. De Neys, W., & Glumicic, T., (2008). Conflict monitoring in dual process theories of thinking. Cognition, 106(3), 1248–1299.
18. De Neys, W., Moyens, E., & Vansteenwegen, D., (2010). Feeling we're biased:Autonomic arousal and reasoning conflict. Cognitive, Affective, & Behavioral Neuroscience, 10(2), 208–216.
19. Dienes, Z., (2008). Understanding psychology as a science:An introduction to scientific and statistical inference. New York, NY:Palgrave Macmillan.
20. Dienes, Z., (2011). Bayesian versus orthodox statistics:Which side are you on? Perspectives on Psychological Science, 6(3), 274–290.
21. Dienes, Z., (2014). Using Bayes to get the most out of non-significant results. Frontiers in psychology, 5, 781.
22. Elqayam, S., & Evans, J. S. B., (2011). Subtracting “ought” from “is”:Descriptivism versus normativism in the study of human thinking. Behavioral and Brain Sciences, 34(5), 233–248.
23. Erickson, T. D., & Mattson, M. E., (1981). From words to meaning:A semantic illusion. Journal of Verbal Learning and Verbal Behavior, 20(5), 540–551.
24. Ericsson, K. A., & Simon, H. A., (1980). Verbal reports as data. Psychological Review, 87(3), 215–251.
25. Evans, J. S. B., (1996). Deciding before you think:Relevance and reasoning in the selection task. British Journal of Psychology, 87(2), 223–240.
26. Evans, J. S. B., (2003). In two minds:Dual-process accounts of reasoning. Trends in Cognitive Sciences, 7(10), 454–459.
27. Evans, J. S. B., (2009). How many dual-process theories do we need? One, two, or many? In In two minds:Dual processes and beyond (pp. 33–54). New York, NY:Oxford University Press.
28. Evans, J. S. B., & Ball, L. J., (2010). Do people reason on the Wason selection task:A new look at the data of Ball et al. (2003). Quarterly Journal of Experimental Psychology, 63(3), 434–441.
29. Evans, J. S. B., Barston, J. L., & Pollard, P., (1983). On the conflict between logic and belief in syllogistic reasoning. Memory & Cognition, 11(3), 295–306.
30. Evans, J. S. B., & Stanovich, K. E., (2013). Dual-process theories of higher cognition advancing the debate. Perspectives on Psychological Science, 8(3), 223–241.
31. Fific, M., (2014). Double jeopardy in inferring cognitive processes. Frontiers in Psychology, 5, 1130.
32. Finucane, M. L., & Gullion, C. M., (2010). Developing a tool for measuring the decision-making competence of older adults. Psychology and Aging, 25(2), 271–288.
33. Fox, M. C., Ericsson, K. A., & Best, R., (2011). Do procedures for verbal reporting of thinking have to be reactive? A meta-analysis and recommendations for best reporting methods. Psychological Bulletin, 137(2), 316–344.
34. Frederick, S., (2005). Cognitive reflection and decision making. The Journal of Economic Perspectives, 19(4), 25–42.
35. Gervais, W. M., & Norenzayan, A., (2012). Analytic thinking promotes religious disbelief. Science, 336(6080), 493–496.
36. Ghazal, S., Cokely, E. T., & Garcia-Retamero, R., (2014). Predicting biases in very highly educated samples:Numeracy and metacognition. Judgment and Decision Making, 9(1), 15–34.
37. Haran, U., Ritov, I., & Mellers, B. A., (2013). The role of actively open-minded thinking in information acquisition, accuracy, and calibration. Judgment and Decision Making, 8(3), 188–201.
38. Jacoby, L. L., Kelley, C. M., & McElree, B. D., (1999). The role of cognitive control:Early selection versus late correction. In S., Chaiken & Y., Trope (Eds.), Dual-process theories in social psychology (pp. 383–400). New York, NY:Guilford.
39. Jacoby, L. L., Shimizu, Y., Daniels, K. A., & Rhodes, M. G., (2005). Modes of cognitive control in recognition and source memory:Depth of retrieval. Psychonomic Bulletin & Review, 12(5), 852–857.
40. Jeffreys, H., (1961). The theory of probability. Oxford:Oxford University Press.
41. Johnson, E. D., Tubau, E., & De Neys, W., (2016). The doubting system 1:Evidence for automatic substitution sensitivity. Acta Psychologica, 164, 56–64.
42. Kagan, J., Rosman, B. L., Day, D., Albert, J., & Phillips, W., (1964). Information processing in the child:Significance of analytic and reflective attitudes. Psychological Monographs:General and Applied, 78(1), 1–37.
43. Kahneman, D., (2011). Thinking, fast and slow. New York, NY:Farrar, Straus, and Giroux.
44. Låg, T., Bauger, L., Lindberg, M., & Friborg, O., (2014). The role of numeracy and intelligence in health‐risk estimation and medical data interpretation. Journal of Behavioral Decision Making, 27(2), 95–108.
45. Liberali, J. M., Reyna, V. F., Furlan, S., Stein, L. M., & Pardo, S. T., (2012). Individual differences in numeracy and cognitive reflection, with implications for biases and fallacies in probability judgment. Journal of Behavioral Decision Making, 25(4), 361–381.
46. Mata, A., Ferreira, M. B., & Sherman, S. J., (2013). The metacognitive advantage of deliberative thinkers:A dual-process perspective on overconfidence. Journal of Personality and Social Psychology, 105(3), 353–355.
47. Mata, A., Schubert, A.-L., & Ferreira, M. B., (2014). The role of language comprehension in reasoning:How “good-enough” representations induce biases. Cognition, 133(2), 457–463.
48. Meyer, A., Spunt, R., & Frederick, S., (2015). The bat and ball problem. Unpublished manuscript.
49. Pachur, T., & Spaar, M., (2015). Domain-specific preferences for intuition and deliberation in decision making. Journal of Applied Research in Memory and Cognition, 4(3), 303–311.
50. Pacini, R., & Epstein, S., (1999). The relation of rational and experiential information processing styles to personality, basic beliefs, and the ratio-bias phenomenon. Journal of Personality and Social Psychology, 76(6), 972–987.
51. Paulhus, D. L., (1991). Measurement and control of response bias. In J. P., Robinson, P. R., Shaver, & L. S., Wrightsman (Eds.), Measures of personality and social psychological attitudes (pp. 17–59). San Diego, CA:Academic Press.
52. Paxton, J. M., Ungar, L., & Greene, J. D., (2012). Reflection and reasoning in moral judgment. Cognitive Science, 36(1), 163–177.
53. Pennycook, G., Cheyne, J. A., Koehler, D. J., & Fugelsang, J. A., (2015). Is the cognitive reflection test a measure of both reflection and intuition? Behavior Research Methods, 48(1), 341–348.
54. Pennycook, G., Cheyne, J. A., Seli, P., Koehler, D. J., & Fugelsang, J. A., (2012). Analytic cognitive style predicts religious and paranormal belief. Cognition, 123(3), 335–346.
55. Pennycook, G., Fugelsang, J. A., & Koehler, D. J., (2015). Everyday consequences of analytic thinking. Current Directions in Psychological Science, 24(6), 425–432.
56. Pennycook, G., & Ross, M. R., (2016). Commentary:Cognitive reflection vs. calculation in decision making. Frontiers in Psychology, 7, 9. Retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4722428/
57. Perkins, D., (1995). Outsmarting IQ:The emerging science of learnable intelligence. New York, NY:Free Press.
58. Peters, E., (2012). Beyond comprehension the role of numeracy in judgments and decisions. Current Directions in Psychological Science, 21(1), 31–35.
59. Peters, E., Dieckmann, N., Dixon, A., Hibbard, J. H., & Mertz, C. K., (2007). Less is more in presenting quality information to consumers. Medical Care Research and Review, 64(2), 169–190.
60. Piazza, J., & Sousa, P., (2013). Religiosity, political orientation, and consequentialist moral thinking. Social Psychological and Personality Science, 5(3), 334–342.
61. Primi, C., Morsanyi, K., Chiesi, F., Donati, M. A., & Hamilton, J., (2015). The development and testing of a new version of the cognitive reflection test applying item response theory (IRT). Journal of Behavioral Decision Making, 29. doi:10.1002/bdm.1883
62. Reisen, N., Hoffrage, U., & Mast, F. W., (2008). Identifying decision strategies in a consumer choice situation. Judgment and Decision Making, 3(8), 641–658.
63. Reyna, V. F., Nelson, W. L., Han, P. K., & Dieckmann, N. F., (2009). How numeracy influences risk comprehension and medical decision making. Psychological Bulletin, 135(6), 943–973.
64. Simon, H. A., (1956). Rational choice and the structure of the environment. Psychological Review, 63(2), 129–138.
65. Sinayev, A., & Peters, E., (2015). Cognitive reflection vs. calculation in decision making. Frontiers in Psychology, 6, 532. doi:10.3389/fpsyg.2015.00532
66. Stanovich, K. E., Toplak, M. E., & West, R. F., (2008). The development of rational thought:A taxonomy of heuristics and biases. Advances in Child Development and Behavior, 36, 251–285.
67. Stupple, E. J., Ball, L. J., & Ellis, D., (2013). Matching bias in syllogistic reasoning:Evidence for a dual-process account from response times and confidence ratings. Thinking & Reasoning, 19(1), 54–77.
68. Stupple, E. J., Gale, M., & Richmond, C. R., (2013). Working memory, cognitive miserliness and logic as predictors of performance on the cognitive reflection test. In M., Knauff, M., Pauen, N., Sebanz, & I., Wachsmuth (Eds.), Proceedings of the 35th annual conference of the cognitive science society (pp. 1396–1401). Austin, TX:Cognitive Science Society.
69. Svedholm-Häkkinen, A. M., (2015). Highly reflective reasoners show no signs of belief inhibition. Acta Psychologica, 154, 69–76.
70. Szaszi, B., (2016). The role of expertise and preference behind individuals’ tendency to use intuitive decision style. Journal of Applied Research in Memory and Cognition, 5(3), 329–330.
71. Szollosi, A., Bago, B., Szaszi, B., & Aczel, B., (in press). Exploring the determinants of confidence in the bat-and-ball problem.
72. Thomson, K. S., & Oppenheimer, D. M., (2016). Investigating an alternate form of the cognitive reflection test. Judgment and Decision Making, 11(1), 99–113.
73. Thompson, V. A., & Johnson, S. C., (2014). Conflict, metacognition, and analytic thinking. Thinking & Reasoning, 20(2), 215–244.
74. Thompson, V. A., Turner, J. P., & Pennycook, G., (2011). Intuition, reason and metacognition. Cognitive Psychology, 63(3), 107–140.
75. Thompson, V. A., Turner, J. P., Pennycook, G., Ball, L. J., Brack, H., Ophir, Y., & Ackerman, R., (2013). The role of answer fluency and perceptual fluency as metacognitive cues for initiating analytic thinking. Cognition, 128(2), 237–251.
76. Toplak, M. E., West, R. F., & Stanovich, K. E., (2011). The Cognitive Reflection Test as a predictor of performance on heuristics-and-biases tasks. Memory & Cognition, 39(7), 1275–1289.
77. Toplak, M. E., West, R. F., & Stanovich, K. E., (2014). Assessing miserly information processing:An expansion of the Cognitive Reflection Test. Thinking & Reasoning, 20(2), 147–168.
78. Tor, A., & Bazerman, M. H., (2003). Focusing failures in competitive environments:Explaining decision errors in the Monty Hall game, the acquiring a company problem, and multiparty ultimatums. Journal of Behavioral Decision Making, 16(5), 353–374.
79. Travers, E., Rolison, J. J., & Feeney, A., (2016). The time course of conflict on the cognitive reflection test. Cognition, 150, 109–118.
80. Tversky, A., & Kahneman, D., (1974). Judgment under uncertainty:Heuristics and biases. Science, 185(4157), 1124–1131.
81. Wason, P. C., & Evans, J. S. B., (1975). Dual processes in reasoning? Cognition, 3(2), 141–154.
82. Weller, J. A., Dieckmann, N. F., Tusler, M., Mertz, C. K., Burns, W. J., & Peters, E., (2013). Development and testing of an abbreviated numeracy scale:A Rasch analysis approach. Journal of Behavioral Decision Making, 26(2), 198–212.
83. Welsh, M., Burns, N., & Delfabbro, P., (2013). The Cognitive Reflection Test:How much more than numerical ability. In M., Knauff, M., Pauen, N., Sebanz, & I., Wachsmuth (Eds.), Proceedings of the 35th annual conference of the cognitive science society (pp. 1396–1401). Austin, TX:Cognitive Science Society.