Investigating an alternate form of the cognitive reflection test

Judgment and Decision Making

Volumn 11, Issue 1, 2016, Pages 99-113

  Thomson, Keela S ^a   Oppenheimer, Daniel M ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Cognitive reflection; CRT

Indexed keywords

EID: 84956941613     PISSN: None     EISSN: 19302975     Source Type: Journal    
DOI: None     Document Type: Article

References (61)

1. Ackerman, R. (2014). The diminishing criterion model for metacognitive regulation of time investment. Journal of Experimental Psychology: General, 143(3), 1349-1368.
2. Ackerman, R., & Zalmanov, H. (2012). The persistence of the fluency-confidence association in problem solving. Psychonomic Bulletin & Review, 19(6), 1187-1192.
3. Baron, J., Scott, S., Fincher, K. S., & Metz, S. E. (2015). Why does the Cognitive Reflection Test (sometimes) predict utilitarian moral judgment (and other things)? Jour-nal of Applied Research in Memory and Cognition, 4(3), 265-284.
4. Basso, M. R., Bornstein, R. A., & Lang, J. M. (1999). Practice effects on commonly used measures of executive function across twelve months. The Clinical Neuropsy-chologist, 13(3), 283-292.
5. Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012). Evaluating online labor markets for experimental research: Amazon.com's Mechanical Turk. Political Analysis, 20(3), 351-368.
6. Blackwell, L. S., Trzesniewski, K. H., & Dweck, C. S. (2007). Implicit theories of intelligence predict achievement across an adolescent transition: A longitudinal study and an intervention. Child Development, 78(1), 246-263.
7. Böckenholt, U. (2012). The cognitive-miser response model: Testing for intuitive and deliberative reasoning. Psychometrika, 77(2), 388-399.
8. Cacioppo, J. T. Petty, R.E. & Kao, C.F. (1996). The efficient assessment of need for cognition. Journal of Personality Assessment, 48(3), 306-307.
9. Campitelli, G., & Gerrans, P. (2014). Does the cognitive reflection test measure cognitive reflection? A mathematical modeling approach. Memory and Cognition, 42(3), 434-447.
10. Chandler, J., Mueller, P., & Paolacci, G. (2014). Nonnaivete among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Be-havioral Research Methods, 46(1), 112-130.
11. Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015). Using nonnaive participants can reduce effect sizes. Psychological Science, 26(7), 1131-1139.
12. Cokely, E. T., & Kelley, C. M. (2009). Cognitive abilities and superior decision making under risk: A protocol analysis and process model evaluation. Judgment and Deci-sion Making, 4(1), 20-33.
13. Del Missier, F., Mäntylä, T., & Bruine de Bruin, W. (2012). Decision-making competence, executive functioning, and general cognitive abilities. Journal of Be-havioral Decision-Making, 25(4), 331-351.
14. De Neys, W., & Franssens, S. (2009). Belief inhibition during thinking: Not always winning but at least taking part. Cognition, 113(1), 45-61.
15. Duckworth, A. L., & Seligman, M. E. P. (2005). Selfdiscipline outdoes IQ in predicting academic performance of adolescents. Psychological Science, 16, 939-944.
16. Finucane, M. L., & Gullion, C. M. (2010). Developing a tool for measuring the decision-making competence of older adults. Psychology and Aging, 25(2), 271.
17. Frederick, S. (2005). Cognitive reflection and decisionmaking. Journal of Economic Perspectives, 19(4), 25-42.
18. Graffeo, M., Polonio, L., & Bonini, N. (2015). Individual differences in competent consumer choice: The role of cognitive reflection and numeracy skills. Frontiers in Psy-chology, 6, 844.
19. Hoerger, M. (2013). ZH: An updated version of Steiger's Z and web-based calculator for testing the statistical significance of the difference between dependent correlations. Retrieved from http://www.psychmike.com/ dependent_correlations.php.
20. Hoppe, E. I., & Kusterer, D. J. (2011). Behavioral biases and cognitive reflection. Economics Letters, 110(12), 97-100.
21. Irwin, J. R. & McClelland, G. H. (2001). Misleading heuristics and moderated multiple regression models. Journal of Marketing Research, 38(1), 100-109.
22. Irwin, J. R. & McClelland, G. H. (2003). Negative effects of dichotomizing continuous predictor variables. Journal of Marketing Research, 40(3), 366-371.
23. Jaccard, J., Guilamo-Ramos, V., Johansson, M., & Bouris, A. (2006). Multiple regression analyses in clinical child and adolescent psychology. Journal of Clinical Child and Adolescent Psychology, 35(3), 456-479.
24. Johnson-Laird, P. N., & Bara, B. G. (1984). Syllogistic inference. Cognition, 16, 1-61.
25. Kahneman, D. (2011). Thinking, Fast and Slow. New York: Farrar, Strauss, Giroux.
26. Kirkpatrick, L., & Epstein, S. (1992). Cognitiveexperiential self-theory and subjective probability: Evidence for two conceptual systems. Journal of Personality and Social Psychology, 63(4), 534-544.
27. Koehler, D. J., & James, G. (2010). Probability matching and strategy availability. Memory & Cognition, 38(6), 667-676.
28. Levesque, H. J. (1986). Making believers out of computers. Artificial Intelligence, 30, 81-108.
29. Levesque, H. J. (1989). Logic and the complexity of reasoning. In R. H. Thomason (Ed.), Philosophical logic and artificial intelligence (pp. 73-107). Dordrecht, The Netherlands: Kluwer Academic Publishers.
30. Liberali, J. M., Reyna, V. F., Furlan, S., Stein, L. M., & Pardo, S. T. (2012). Individual differences in numeracy and cognitive reflection, with implications for biases and fallacies in probability judgment. Journal of Behavioral Decision Making, 25(4), 361-381.
31. Lipkus, I. M., Samsa, G., & Rimer, B. K. (2001). General performance on a numeracy scale among highly educated samples. Medical Decision Making, 21(1), 37-44.
32. Lubin, G. (2012). A simple logic question that most Harvard students get wrong. Business Insider. Retrieved from: http://www.businessinsider.com/questionthat-harvard-students-get-wrong-2012-12.
33. MacCallum, R. C., Zhang, S., Preacher, K. J., & Rucker, D. D. (2002). On the practice of dichotomization of quantitative variables. Psychological Methods, 7(1), 19-40.
34. Markovits, H. & Nantel, G. (1989). The belief-bias effect in the production and evaluation of logical conclusions. Memory & Cognition, 17(1), 11-17.
35. Maxwell, S. E. & Delaney, H. D. (1993). Bivariate median splits and spurious statistical significance. Psychological Bulletin, 113(1), 181-190.
36. Obrecht, N., Chapman, G., & Gelman, R. (2007) Intuitive t-tests: Lay use of statistical information. Psychonomic Bulletin and Review, 14(6), 1147-1152.
37. Obrecht, N. A., Chapman, G. B., & Gelman, R. (2009). An encounter frequency account of how experience affects likelihood estimation. Memory & Cognition, 37(5), 632-643.
38. Oechssler, J., Roider, A., & Schmitz, P. W. (2009). Cognitive abilities and behavioral biases. Journal of Economic Behavior & Organization, 72(1), 147-152.
39. Paxton, J. M., Ungar, L., & Greene, J. D. (2012). Reflection and reasoning in moral judgment. Cognitive Science, 36(1), 163-177.
40. Peters, E. (2012). Beyond comprehension: the role of numeracy in judgments and decisions. Current Directions in Psychological Science, 21(1), 31-35.
41. Peters, E., Dieckmann, N. F., Västfjäll, D., Mertz, C. K., Slovic, P., & Hibbard, J. H. (2009). Bringing meaning to numbers: the impact of evaluative categories on decisions. Journal of Experimental Psychology: Applied, 15(3), 213-227.
42. Peters, E., Västfjäll, D., Slovic, P., Mertz, C. K., Mazzocco, K., and Dickert, S. (2006). Numeracy and decision making. Psychological Science, 17(5), 407-413.
43. Postrel, A. (2006). Would you take a bird in the hand, or a 75% chance of two in the bush? The New York Times. Retrieved from: http://www.nytimes.com/2006/ 01/26/business/26scene.html?pagewanted=print&_r=0.
44. Primi, C., Morsanyi, K., Chiesi, F., Donati, M. A., & Hamilton, J. (2015). The development and testing of a new version of the cognitive reflection test applying item response theory (IRT). Journal of Behavioral Decision Making. http://dx.doi.org/10.1002/bdm.1883.
45. Rand, D. G., Peysakhovich, A., Kraft-Todd, G. T., Newman, G. E., Wurzbacher, O., Nowak, M.A., & Greene, J.D. (2014). Social heuristics shape intuitive cooperation. Nature Communications, 5, 1-30.
46. Reyna, V., Chapman, S., Dougherty, M., & Confrey, J. (Eds.). (2012). The adolescent brain: Learning, rea-soning, and decision making. Washington DC: American Psychological Association.
47. Reyna, V. F., Nelson, W. L., Han, P. K., & Dieckmann, N. F. (2009). How numeracy influences risk comprehension and medical decision making. Psychological Bulletin, 135(6), 943-973.
48. Riordan, H. (n.d.). See if you can figure out these trick questions … Allwomenstalk. Retrieved from http://funny. allwomenstalk.com/see-if-you-can-figure-out-thesetrick-questions.
49. Royzman, E. B., Landy, J. F., & Leeman, R. F. (2014). Are thoughtful people more utilitarian? CRT as a unique predictor of moral minimalism in the dilemmatic context. Cognitive Science, 39(2), 325-352.
50. Schwartz, L. M., Woloshin, S., Black, W. C., & Welch, G. H. (1997). The role of numeracy in understanding the benefit of screening mammography. Annals of Internal Medicine, 127 (11), 966-971.
51. Sinayev, A. & Peters, E. (2015). Cognitive reflection vs. calculation in decision making. Frontiers in Psychology, 6, 532.
52. Steiger, J. H. (1980). Tests for comparing elements of a correlation matrix. Psychological Bulletin, 87, 245-251.
53. Stewart, N., Ungemach, C., Harris, A. J. L., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015). The average laboratory samples a population of 7, 300 Amazon Mechanical Turk Workers. Judgment and Decision-Making, 10(5), 479-491.
54. Toplak, M. E., & Stanovich, K. E. (2002). The domain specificity and generality of disjunctive reasoning: Searching for a generalizable critical thinking skill. Jour-nal of Educational Psychology, 94(1), 197-209.
55. Toplak, M. E., West, R.F., & Stanovich, K. E. (2011). The Cognitive Reflection Test as a predictor of performance on heuristics-and-biases tasks. Memory & Cognition, 39(7), 1275-1289.
56. Toplak, M. E., West, R. F., & Stanovich, K. E. (2014a). Assessing miserly information processing: An expansion of the Cognitive Reflection Test. Thinking & Reasoning, 20(2), 147-168.
57. Toplak, M. E., West, R. F., & Stanovich, K. E. (2014b). Rational thinking and cognitive sophistication: Development, cognitive abilities, and thinking dispositions. De-velopmental Psychology, 50(4), 1037-1048.
58. Vargha, A., Rudas, T., Delaney, H. D., & Maxwell, S. E. (1996). Dichotomization, partial correlation, and conditional independence. Journal of Educational and Behav-ioral Statistics, 21(3), 264-282.
59. Weller, J. A., Dieckmann, N. F., Tusler, M., Mertz, C. K., Burns, W. J.,& Peters, E. (2013). Development and testing of an abbreviated numeracy scale: A Rasch analysis approach. Journal of Behavioral Decision Making, 26(2), 198-212.
60. Welsh, M., Burns, N., & Delfabbro, P. (2013). The cognitive reflection test: How much more than numerical ability? Paper presented at the Proceedings of the 35th Annual Meeting of the Cognitive Science Society, Berlin, Germany. pp.1587-1592.
61. West, R. F., Toplak, M. E., & Stanovich, K. E. (2008). Heuristics and biases as measures of critical thinking: Associations with cognitive ability and thinking dispositions. Journal of Educational Psychology, 100(4), 930-941.