Assessing the belief bias effect with ROCs: It's a response bias effect

Psychological Review

Volumn 117, Issue 3, 2010, Pages 831-863

  Dube, Chad ^a   Rotello, Caren M ^a   Heit, Evan ^b  

^a UNIVERSITY OF MASSACHUSETTS   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Belief bias; Multinomial modeling; Response bias; Signal detection; Syllogistic reasoning

Indexed keywords

ARTICLE; COGNITION; CULTURAL ANTHROPOLOGY; DECISION MAKING; HUMAN; PERCEPTIVE DISCRIMINATION; PROBLEM SOLVING; PSYCHOLOGICAL MODEL; RECEIVER OPERATING CHARACTERISTIC; SOCIAL PSYCHOLOGY; PREJUDICE;

COGNITION; CULTURE; HUMANS; JUDGMENT; MODELS, PSYCHOLOGICAL; PREJUDICE; PROBLEM SOLVING; ROC CURVE; SIGNAL DETECTION, PSYCHOLOGICAL;

EID: 77955320523     PISSN: 0033295X     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0019634     Document Type: Article

References (111)

1. Akaike H. Information theory as an extension of the maximum likelihood principle. Second International Symposium on Information Theory 1973, 267-281. Akademiai Kiado, Budapest, Hungary. B.N. Petrov, F. Csaki (Eds.).
2. Ashby F.G., Gott R.E. Decision rules in the perception and categorization of multidimensional stimuli. Journal of Experimental Psychology: Learning, Memory, and Cognition 1988, 14:33-53.
3. Pringles are potato chips, UK taxman decides. USA Today 2009, May 20, Associated Press, Retrieved from http://www.usatoday.com/money/industries/food/2009-05-20-pringles_N.htm.
4. Ball L.J., Phillips P., Wade C.N., Quayle J.D. Effects of belief and logic on syllogistic reasoning: Eye-movement evidence for selective processing models. Experimental Psychology 2006, 53:77-86.
5. Banks W.P. Signal detection theory and human memory. Psychological Bulletin 1970, 74:81-99.
6. Banks W.P. Recognition and source memory as multivariate decision processes. Psychological Science 2000, 11:267-273.
7. Bartlett F.C. Remembering: A study in experimental and social psychology 1932, Cambridge University Press, Cambridge, England.
8. Begg I., Denny J.P. Empirical reconciliation of atmosphere and conversion interpretations of syllogistic reasoning errors. Journal of Experimental Psychology 1969, 81:351-354.
9. Benjamin A.S., Diaz M., Wee S. Signal detection with criterion noise: Applications to recognition memory. Psychological Review 2009, 116:84-115.
10. Bröder A., Schütz J. Recognition ROCs are curvilinear-or are they? On premature arguments against the two-high-threshold model of recognition. Journal of Experimental Psychology: Learning, Memory, and Cognition 2009, 35:587-606.
11. Chater N., Oaksford M. The probability heuristics model of syllogistic reasoning. Cognitive Psychology 1999, 38:191-258.
12. Cherubini P., Garnham A., Oakhill J., Morley E. Can any ostrich fly? Some new data on belief bias in syllogistic reasoning. Cognition 1998, 69:179-218.
13. Cohen A.L., Rotello C.M., Macmillan N.A. Evaluating models of remember-know judgments: Complexity, mimicry, and discriminability. Psychonomic Bulletin & Review 2008, 15:906-926.
14. Creelman C.D., Donaldson W. ROC curves for discrimination of linear extent. Journal of Experimental Psychology 1968, 77:514-516.
15. Dickstein L.S. Effects of instructions and premise order on errors in syllogistic reasoning. Journal of Experimental Psychology: Human Learning and Memory 1975, 1:376-384.
16. Dickstein L.S. The effect of figure on syllogistic reasoning. Memory & Cognition 1978, 6:76-83.
17. Dickstein L.S. Conversion and possibility in syllogistic reasoning. Bulletin of the Psychonomic Society 1981, 18:229-232.
18. Dodson C.S., Prinzmetal W., Shimamura A.P. Using Excel to estimate parameters from observed data: An example from source memory data. Behavior Research Methods, Instruments, & Computers 1998, 30:517-526.
19. Dougal S., Rotello C.M. " Remembering" emotional words is based on response bias, not recollection. Psychonomic Bulletin & Review 2007, 14:423-429.
20. Egan J.P., Schulman A.I., Greenberg G.Z. Operating characteristics determined by binary decisions and by ratings. Journal of the Acoustical Society of America 1959, 31:768-773.
21. Emmerich D.S. ROCs obtained with two signal intensities presented in random order, and a comparison between yes-no and rating ROCs. Perception & Psychophysics 1968, 3:35-40.
22. Erdfelder E., Buchner A. Process-dissociation measurement models: Threshold theory or detection theory?. Journal of Experimental Psychology: General 1998, 127:83-96.
23. Evans J.St.B.T. The heuristic-analytic theory of reasoning: Extension and evaluation. Psychonomic Bulletin & Review 2006, 13:378-395.
24. Evans J.St.B.T. Hypothetical thinking: Dual processes in reasoning and judgement 2007, Psychology Press, New York, NY.
25. Evans J.St.B.T. Dual-processing accounts of reasoning, judgment, and social cognition. Annual Review of Psychology 2008, 59:255-278.
26. Evans J.St.B.T., Barston J.L., Pollard P. On the conflict between logic and belief in syllogistic reasoning. Memory & Cognition 1983, 11:295-306.
27. Evans J.St.B.T., Curtis-Holmes J. Rapid responding increases belief bias: Evidence for the dual-process theory of reasoning. Thinking & Reasoning 2005, 11:382-389.
28. Evans J.St.B.T., Handley S.J., Harper C.N.J. Necessity, possibility and belief: A study of syllogistic reasoning. Quarterly Journal of Experimental Psychology: Human Experimental Psychology 2001, 54(A):935-958.
29. Evans J.St.B.T., Newstead S.E., Allen J.L., Pollard P. Debiasing by instruction: The case of belief bias. European Journal of Cognitive Psychology 1994, 6:263-285.
30. Evans J.St.B.T., Newstead S.E., Byrne R.M.J. Human reasoning: The psychology of deduction 1993, Erlbaum, Hillsdale, NJ.
31. Galvin S.J., Podd J.V., Drga V., Whitmore J. Type 2 tasks in the theory of signal detectability: Discrimination between correct and incorrect decisions. Psychonomic Bulletin & Review 2003, 10:843-876.
32. Gilinsky A.S., Judd B.B. Working memory and bias in reasoning across the life span. Psychology and Aging 1994, 9:356-371.
33. Glanzer M., Kim K., Hilford A., Adams J.K. Slope of the receiver-operating characteristic in recognition memory. Journal of Experimental Psychology: Learning, Memory, and Cognition 1999, 25:500-513.
34. Goodman L.A., Kruskal W.H. Measures of association for cross classifications. Journal of the American Statistical Association 1954, 49:732-764.
35. Green D.M., Swets J.A. Signal detection theory and psychophysics 1966, Wiley, Oxford, England.
36. Hautus M.J., Macmillan N.A., Rotello C.M. Toward a complete decision model of item and source recognition. Psychonomic Bulletin & Review 2008, 15:889-905.
37. Hayes, B. K., Heit, E., & Swendsen, H. (in press). Inductive reasoning. Cognitive Science.
38. Healy A.F., Jones C. Can subjects maintain a constant criterion in a memory task?. Memory & Cognition 1975, 3:233-238.
39. Heathcote A. Item recognition memory and the receiver operating characteristic. Journal of Experimental Psychology: Learning, Memory, and Cognition 2003, 29:1210-1230.
40. Heit E. Modeling the effects of expectations on recognition memory. Psychological Science 1993, 4:244-252.
41. Heit E. Models of the effects of prior knowledge on category learning. Journal of Experimental Psychology: Learning, Memory, and Cognition 1994, 20:1264-1282.
42. Heit E. Knowledge and concept learning. Knowledge, concepts, and categories 1997, 7-41. Psychology Press, London, England. K. Lamberts, D. Shanks (Eds.).
43. Heit E. Background knowledge and models of categorization. Similarity and categorization 2001, 155-178. Oxford University Press, Oxford, England. U. Hahn, M. Ramscar (Eds.).
44. Heit E., Hahn U., Feeney A. Defending diversity. Categorization inside and outside the laboratory: Essays in honor of Douglas L. Medin 2005, 87-99. American Psychological Association, Washington, DC. W. Ahn, R. Goldstone, B.C. Love, A.B. Markman, P. Wolff (Eds.).
45. Heit E., Rotello C.M. Are there two kinds of reasoning?. Proceedings of the Twenty-Seventh Annual Meeting of the Cognitive Science Society 2005, 923-928. Erlbaum, Hillsdale, NJ. B.G. Bara, L. Barsalou, M. Bucciarelli (Eds.).
46. Heit E., Rotello C.M. Modeling two kinds of reasoning. Proceedings of the Thirtieth Annual Meeting of the Cognitive Science Society 2008, 1831-1836. Cognitive Science Society, Austin, TX. B.C. Love, K. McRae, V.M. Sloutsky (Eds.).
47. Heit, E., & Rotello, C. M. (in press). Relations between inductive reasoning and deductive reasoning. Journal of Experimental Psychology: Learning, Memory, and Cognition.
48. Higham P.A., Perfect T.J., Bruno D. Investigating strength and frequency effects in recognition memory using Type-2 signal detection theory. Journal of Experimental Psychology: Learning, Memory, and Cognition 2009, 35:57-80.
49. Johnson-Laird P.N. Mental models: Towards a cognitive science of language, inference, and consciousness 1983, Harvard University Press, Cambridge, MA.
50. Johnson-Laird P.N., Bara B.G. Syllogistic inference. Cognition 1984, 16:1-61.
51. Johnson-Laird P.N., Steedman M. The psychology of syllogisms. Cognitive Psychology 1978, 10:64-99.
52. Kapucu A., Rotello C.M., Ready R.E., Seidl K.N. Response bias in " remembering" emotional stimuli: A new perspective on age differences. Journal of Experimental Psychology: Learning, Memory, and Cognition 2008, 34:703-711.
53. Klauer, K. C., & Kellen, D. (in press). Toward a complete decision model of item and source recognition: A discrete-state approach. Psychonomic Bulletin & Review.
54. Klauer K.C., Musch J., Naumer B. On belief bias in syllogistic reasoning. Psychological Review 2000, 107:852-884.
55. Klayman J., Ha Y. Confirmation, disconfirmation, and information in hypothesis testing. Psychological Review 1987, 94:211-228.
56. Krantz D. Threshold theories of signal detection. Psychological Review 1969, 76:308-324.
57. Macmillan N.A., Creelman C.D. Detection theory: A user's guide 2005, Erlbaum, Mahwah, NJ. 2nd ed.
58. Macmillan N.A., Rotello C.M., Miller J.O. The sampling distributions of Gaussian ROC statistics. Perception & Psychophysics 2004, 66:406-421.
59. Maddox W.T., Dodd J.L. Separating perceptual and decisional attention processes in the identification and categorization of integral-dimension stimuli. Journal of Experimental Psychology: Learning, Memory, and Cognition 2003, 29:467-480.
60. Malmberg K.J. On the form of ROCs constructed from confidence ratings. Journal of Experimental Psychology: Learning, Memory, and Cognition 2002, 28:380-387.
61. Markovits H., Nantel G. The belief-bias effect in the production and evaluation of logical conclusions. Memory & Cognition 1989, 17:11-17.
62. Masson M.E.J., Rotello C.M. Sources of bias in the Goodman-Kruskal gamma coefficient measure of association: Implications for studies of metacognitive processes. Journal of Experimental Psychology: Learning, Memory, and Cognition 2009, 35:509-527.
63. Metz C.E. ROCKIT computer program 1998, University of Chicago, Department of Radiology, Chicago, IL, Available from http://xray.bsd.uchicago.edu/cgi-bin/roc_software.cgi.
64. Morley N.J., Evans J.St.B.T., Handley S.J. Belief bias and figural bias in syllogistic reasoning. Quarterly Journal of Experimental Psychology: Human Experimental Psychology 2004, 57(A):666-692.
65. Murphy G.L., Medin D.L. The role of theories in conceptual coherence. Psychological Review 1985, 92:289-316.
66. Myung I.J., Pitt M.A., Kim W. Model evaluation, testing, and selection. Handbook of cognition 2003, 422-436. Sage, London, England. K. Lamberts, R. Goldstone (Eds.).
67. Nelson T.O. A comparison of current measures of the accuracy of feeling-of-knowing predictions. Psychological Bulletin 1984, 95:109-133.
68. Nelson T.O. ROC curves and measures of discrimination accuracy: A reply to Swets. Psychological Bulletin 1986, 100:128-132.
69. Newstead S.E., Pollard P., Evans J.St.B.T., Allen J. The source of belief bias effects in syllogistic reasoning. Cognition 1992, 45:257-284.
70. Norenzayan A., Smith E.E., Jun Kim B., Nisbett R.E. Cultural preferences for formal versus intuitive reasoning. Cognitive Science 2002, 26:653-684.
71. Oakhill J., Johnson-Laird P.N., Garnham A. Believability and syllogistic reasoning. Cognition 1989, 31:117-140.
72. Oakhill J.V., Johnson-Laird P. The effects of belief on the spontaneous production of syllogistic conclusions. Quarterly Journal of Experimental Psychology: Human Experimental Psychology 1985, 37(A):553-569.
73. Oaksford M., Chater N. Bayesian rationality: The probabilistic approach to human reasoning 2007, Oxford University Press, Oxford, England.
74. Polk T.A., Newell A. Deduction as verbal reasoning. Psychological Review 1995, 102:533-566.
75. Pollack I., Norman D.A. A non-parametric analysis of recognition experiments. Psychonomic Science 1964, 1:125-126.
76. Prowse-Turner J.A., Thompson V.A. The role of training, alternative models, and logical necessity in determining confidence in syllogistic reasoning. Thinking & Reasoning 2009, 15:69-100.
77. Quayle J., Ball L. Working memory, metacognitive uncertainty, and belief bias in syllogistic reasoning. Quarterly Journal of Experimental Psychology: Human Experimental Psychology 2000, 53(A):1202-1223.
78. Ratcliff R., McKoon G., Tindall M. Empirical generality of data from recognition memory receiver-operating characteristic functions and implications for the global memory models. Journal of Experimental Psychology: Learning, Memory, and Cognition 1994, 20:763-785.
79. Ratcliff R., Sheu C., Gronlund S.D. Testing global memory models using ROC curves. Psychological Review 1992, 99:518-535.
80. Revlin R., Leirer V., Yopp H., Yopp R. The belief bias effect in formal reasoning: The influence of knowledge on logic. Memory & Cognition 1980, 8:584-592.
81. Revlis R. Two models of syllogistic reasoning: Feature selection and conversion. Journal of Verbal Learning and Verbal Behavior 1975, 14:180-195.
82. Rhodes M.G., Jacoby L.L. On the dynamic nature of response criterion in recognition memory: Effects of base rate, awareness, and feedback. Journal of Experimental Psychology: Learning, Memory, and Cognition 2007, 33:305-320.
83. Roberts M.J., Sykes E.D.A. Belief bias and relational reasoning. Quarterly Journal of Experimental Psychology: Human Experimental Psychology 2003, 56(A):131-154.
84. Rotello C.M., Heit E. Modeling the effects of argument length and validity on inductive and deductive reasoning. Journal of Experimental Psychology: Learning, Memory, and Cognition 2009, 35:1317-1330.
85. Rotello C.M., Macmillan N.A. Response bias in recognition memory. Skill and strategy in memory use 2008, 61-94. Elsevier, San Diego, CA. A.S. Benjamin, B.H. Ross (Eds.).
86. Rotello C.M., Macmillan N.A., Hicks J.L., Hautus M. Interpreting the effects of response bias on remember-know judgments using signal-detection and threshold models. Memory & Cognition 2006, 34:1598-1614.
87. Rotello C.M., Macmillan N.A., Reeder J.A. Sum-difference theory of remembering and knowing: A two-dimensional signal-detection model. Psychological Review 2004, 111:588-616.
88. Rotello C.M., Masson M.E.J., Verde M.F. Type I error rates and power analyses for single-point sensitivity measures. Perception & Psychophysics 2008, 70:389-401.
89. Rotello C.M., Zeng M. Analysis of RT distributions in the remember-know paradigm. Psychonomic Bulletin & Review 2008, 15:825-832.
90. Schwartz G. Estimating the dimension of a model. Annals of Statistics 1978, 6:461-464.
91. Shafto P., Coley J.D., Vitkin A. Availability in category-based induction. Inductive reasoning 2007, 114-136. Cambridge University Press, Cambridge, England. A. Feeney, E. Heit (Eds.).
92. Shynkaruk J.M., Thompson V.A. Confidence and accuracy in deductive reasoning. Memory & Cognition 2006, 34:619-632.
93. Skyrms B. Choice and chance: An introduction to inductive logic 2000, Wadsworth, Belmont, CA. 4th ed.
94. Snodgrass J.G., Corwin J. Pragmatics of measuring recognition memory: Applications to dementia and amnesia. Journal of Experimental Psychology: General 1988, 117:34-50.
95. Stanovich K.E., West R.F. Individual differences in rational thought. Journal of Experimental Psychology: General 1998, 127:161-188.
96. Stupple E.J.N., Ball L.J. Belief-logic conflict resolution in syllogistic reasoning: Inspection-time evidence for a parallel-process model. Thinking & Reasoning 2008, 14:168-181.
97. Swets J.A. Form of empirical ROCs in discrimination and diagnostic tasks: Implications for theory and measurement of performance. Psychological Bulletin 1986, 99:181-198.
98. Swets J.A. Indices of discrimination or diagnostic accuracy: Their ROCs and implied models. Psychological Bulletin 1986, 99:100-117.
99. Tanner T.A.J., Haller R.W., Atkinson R.C. Signal recognition as influenced by presentation schedules. Perception & Psychophysics 1967, 2:349-358.
100. Tanner T.A.J., Rauk J.A., Atkinson R.C. Signal recognition as influenced by information feedback. Journal of Mathematical Psychology 1970, 7:259-274.
101. Thompson V.A., Striemer C.L., Reikoff R., Gunter R.W., Campbell J.I.D. Syllogistic reasoning time: Disconfirmation disconfirmed. Psychonomic Bulletin & Review 2003, 10:184-189.
102. Unsworth S.J., Medin D.L. Cross cultural differences in belief bias with deductive reasoning?. Cognitive Science 2005, 29:525-529.
103. Van Zandt T. ROC curves and confidence judgments in recognition memory. Journal of Experimental Psychology: Learning, Memory, and Cognition 2000, 26:582-600.
104. z, and A'. Perception & Psychophysics 2006, 68:643-654.
105. Verde M.F., Rotello C.M. Does familiarity change in the revelation effect?. Journal of Experimental Psychology: Learning, Memory, and Cognition 2003, 29:739-746.
106. Verde M.F., Rotello C.M. ROC curves show that the revelation effect is not a single phenomenon. Psychonomic Bulletin & Review 2004, 11:560-566.
107. Watkins M.J., Peynircioglu Z.F. The revelation effect: When disguising test items induces recognition. Journal of Experimental Psychology: Learning, Memory, and Cognition 1990, 16:1012-1020.
108. Weaver C.A., Kelemen W.L. Processing similarity does not improve metamemory: Evidence against transfer-appropriate monitoring. Journal of Experimental Psychology: Learning, Memory, and Cognition 2003, 29:1058-1065.
109. Woodworth R.S., Sells S.B. An atmosphere effect in formal syllogistic reasoning. Journal of Experimental Psychology 1935, 18:451-460.
110. Yonelinas A.P. The nature of recollection and familiarity: A review of 30 years of research. Journal of Memory and Language 2002, 46:441-517.
111. Yonelinas A.P., Parks C.M. Receiver operating characteristics (ROCs) in recognition memory: A review. Psychological Bulletin 2007, 133:800-832.