On the measurement of criterion noise in signal detection theory: The case of recognition memory

Psychological Review

Volumn 119, Issue 3, 2012, Pages 457-479

  Kellen, David ^a   Klauer, Karl Christoph ^a   Singmann, Henrik ^a  

^a UNIVERSITY OF FREIBURG   (Germany)

Author keywords

Decision making; Recognition memory; Response criteria; Signal detection

Indexed keywords

DECISION MAKING; HUMAN; OBSERVER VARIATION; PERCEPTIVE DISCRIMINATION; PSYCHOLOGICAL THEORY; PSYCHOMOTOR PERFORMANCE; RECEIVER OPERATING CHARACTERISTIC; RECOGNITION; REVIEW; STATISTICAL ANALYSIS; STATISTICAL MODEL;

DATA INTERPRETATION, STATISTICAL; DECISION MAKING; HUMANS; MODELS, STATISTICAL; OBSERVER VARIATION; PSYCHOLOGICAL THEORY; PSYCHOMOTOR PERFORMANCE; RECOGNITION (PSYCHOLOGY); ROC CURVE; SIGNAL DETECTION, PSYCHOLOGICAL;

EID: 84867339566     PISSN: 0033295X     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0027727     Document Type: Article

References (86)

1. Atkinson, R. C. (1963). A variable sensitivity theory of signal detection. Psychological Review, 70, 91-106. doi:10.1037/h0041428
2. Balakrishnan, J. D. (1998). Some more sensitive measures of sensitivity and response bias. Psychological Methods, 3, 68-90. doi:10.1037/1082989X.3.1.68
3. Balakrishnan, J. D. (1999). Decision processes in discrimination: Fundamental misrepresentations of signal detection theory. Journal of Experimental Psychology: Human Perception and Performance, 25, 1189-1206. doi:10.1037/0096-1523.25.5.1189
4. Balakrishnan, J. D., & MacDonald, J. A. (2002). Decision criteria do not shift: Reply to Treisman (2002). Psychonomic Bulletin & Review, 9, 858-865. doi:10.3758/BF03196345
5. Balakrishnan, J. D., & MacDonald, J. A. (2011). Performance measures for dynamic signal detection. Journal of Mathematical Psychology, 55, 290-301. doi:10.1016/j.jmp.2011.05.001
6. Benjamin, A. S. (2008). Memory is more than just remembering: Strategic control of encoding, accessing memory, and making decisions. In A. S. Benjamin & B. H. Ross (Eds.), The psychology of learning and motivation: Skill and strategy in memory use (p. 175-223). London, England: Academic Press.
7. Benjamin, A. S., & Bawa, S. (2004). Distractor plausibility and criterion placement in recognition. Journal of Memory and Language, 51, 159-172. doi:10.1016/j.jml.2004.04.001
8. Benjamin, A. S., Diaz, M. L., & Wee, S. (2009). Signal detection with criterion noise: Applications to recognition memory. Psychological Review, 116, 84-115. doi:10.1037/a0014351
9. Bishop, Y. M. M., Fienberg, S. E., & Holland, P. W. (1975). Discrete multivariate analysis: Theory and practice. Cambridge, MA: MIT Press.
10. Bröder, A., & Schütz, J. (2009). Recognition ROCs are curvilinear-Or are they? On premature arguments against the two-high-threshold model of recognition. Journal of Experimental Psychology: Learning, Memory, and Cognition, 35, 587-606. doi:10.1037/a0015279
11. Brown, J. (1965). Multiple response evaluation of discrimination. British Journal of Mathematical and Statistical Psychology, 18, 125-137. doi: 10.1111/j.2044-8317.1965.tb00696.x
12. Brown, S. D., Marley, A. A. J., Donkin, C., & Heathcote, A. (2008). An integrated model of choices and response times in absolute identification. Psychological Review, 115, 396-425. doi:10.1037/0033295X.115.2.396
13. Brown, S. D., & Steyvers, M. (2005). The dynamics of experimentally induced criterion shifts. Journal of Experimental Psychology: Learning, Memory, and Cognition, 31, 587-599. doi:10.1037/0278-7393.31.4.587
14. Burnham, K. P., & Anderson, D. R. (2002). Model selection and multimodel inference: A practical information-theoretic approach. New York, NY: Springer.
15. Busemeyer, J. R., & Wang, Y.-M. (2000). Model comparisons and model selections based on generalization criterion methodology. Journal of Mathematical Psychology, 44, 171-189. doi:10.1006/jmps.1999.1282
16. Chechile, R. A., & Soraci, S. A. (1999). Evidence for a multiple-process account of the generation effect. Memory, 7, 483-508. doi:10.1080/ 741944921
17. Cohen, A. L., Sanborn, A. N., & Shiffrin, R. M. (2008). Model evaluation using grouped or individual data. Psychonomic Bulletin & Review, 15, 692-712. doi:10.3758/PBR.15.4.692
18. Criss, A. H., Malmberg, K. J., & Shiffrin, R. M. (2011). Output interference in recognition memory. Journal of Memory and Language, 64, 316-326. doi:10.1016/j.jml.2011.02.003
19. Criss, A. H., & McClelland, J. (2006). Differentiating the differentiation models: A comparison of the retrieving effectively from memory model (REM) and the subjective likelihood model (SLiM). Journal of Memory and Language, 55, 447-460.
20. Dalrymple-Alford, E. C. (1970). A model for assessing multiple-choice test performance. British Journal of Mathematical and Statistical Psychology, 23, 199-203. doi:10.1111/j.2044-8317.1970.tb00444.x
21. DeCarlo, L. T. (2010). On the statistical and theoretical basis of signal detection theory and extensions: Unequal variance, random coefficient, and mixture models. Journal of Mathematical Psychology, 54, 304-313. doi:10.1016/j.jmp.2010.01.001
22. Dube, C., Rotello, C. M., & Heit, E. (2010). Assessing the belief bias effect with ROCs: It's a response bias effect. Psychological Review, 117, 831-863. doi:10.1037/a0019634
23. Duyck, W., Desmet, T., Verbeke, L., & Brysbaert, M. (2004). WordGen: A tool for word selection and non-word generation in Dutch, German, English, and French. Behavior Research Methods, Instruments & Computers, 36, 488-499. doi:10.3758/BF03195595
24. Estes, W. K., & Maddox, W. T. (2005). Risks of drawing inferences about cognitive processes from model fits to individual versus average performance. Psychonomic Bulletin & Review, 12, 403-408. doi:10.3758/ BF03193784
25. Feller, W. (1966). An introduction to probability theory and its applications. New York, NY: Wiley.
26. Galinsky, T. L., Rosa, R. R., Warm, J. S., & Dember, W. N. (1993). Psychophysical determinants of stress in sustained attention. Human Factors, 35, 603-614.
27. Gigerenzer, G. (1991). From tools to theories: A heuristic of discovery in cognitive psychology. Psychological Review, 98, 254-267. doi:10.1037/ 0033-295X.98.2.254
28. Gilden, D. L., & Wilson, S. G. (1995). On the nature of streaks in signal detection. Cognitive Psychology, 28, 17-64. doi:10.1006/ cogp.1995.1002
29. Glanzer, M., Hilford, A., & Maloney, L. T. (2009). Likelihood ratio decisions in memory: Three implied regularities. Psychonomic Bulletin & Review, 16, 431-455. doi:10.3758/PBR.16.3.431
30. Glöckner, A. (2009). Investigating intuitive and deliberate processes statistically: The multiple-measure maximum likelihood strategy classification method. Judgment and Decision Making, 4, 186-199.
31. Green, D. M., & Moses, F. L. (1966). On the equivalence of two recognition measures of short-term memory. Psychological Bulletin, 66, 228-234. doi:10.1037/h0023645
32. Green, D. M., & Swets, J. A. (1966). Signal detection theory and psychophysics. New York, NY: Wiley.
33. Hautus, M. J., Macmillan, N. A., & Rotello, C. B. (2008). Toward a complete decision model of item and source recognition. Psychonomic Bulletin & Review, 15, 889-905. doi:10.3758/PBR.15.5.889
34. Iverson, G., & Bamber, D. (1997). The generalized area theorem in signal detection theory. In A. A. J. Marley (Ed.), Choice, decision, and measurement: Essays in honor of R. Duncan Luce (p. 301-318). Hillsdale, NJ: Erlbaum.
35. Jang, Y., Wixted, J. T., & Huber, D. E. (2009). Testing signal-detection models of yes/no and two-alternative forced-choice recognition memory. Journal of Experimental Psychology: General, 138, 291-306. doi: 10.1037/a0015525
36. Kellen, D., & Klauer, K. C. (2011). Evaluating models of recognition memory using firstand second-choice responses. Journal of Mathematical Psychology, 55, 251-266. doi:10.1016/j.jmp.2010.11.004
37. Klauer, K. C. (2010). Hierarchical multinomial processing tree models: A latent-trait approach. Psychometrika, 75, 70-98. doi:10.1007/s11336009-9141-0
38. Klauer, K. C., & Kellen, D. (2012). The law of categorical judgment (corrected) extended: A note on Rosner and Kochanski (2009). Psychological Review, 119, 216-220. doi:10.1037/a0025824
39. Klein, S. A. (2001). Measuring, estimating, and understanding the psychometric function: A commentary. Perception & Psychophysics, 63, 1421-1455. doi:10.3758/BF03194552
40. Kornbrot, D. E. (2006). Signal detection theory, the approach of choice: Model-based and distribution-free measures and evaluation. Perception & Psychophysics, 68, 393-414. doi:10.3758/BF03193685
41. Lahl, O., Göritz, A. S., Pietrowsky, R., & Rosenberg, J. (2009). Using the World-Wide Web to obtain large-scale word norms: 190,212 ratings on a set of 2,654 German nouns. Behavior Research Methods, 41, 13-19. doi:10.3758/BRM.41.1.13
42. Lu, Z.-L., & Dosher, B. A. (2008). Characterizing observer states using external noise and observer models: Assessing internal representations with external noise. Psychological Review, 115, 44-82. doi:10.1037/ 0033-295X.115.1.44
43. Macmillan, N. A., & Creelman, C. D. (2005). Detection theory: A user's guide (2nd ed.). Mahwah, NJ: Erlbaum.
44. Macmillan, N. A., Rotello, C. M., & Miller, J. O. (2004). The sampling distributions of Gaussian ROC statistics. Perception & Psychophysics, 66, 406-421. doi:10.3758/BF03194889
45. Maddox, W. T., & Bohil, C. J. (1998). Base-rate and payoff effects in multidimensional perceptual categorization. Journal of Experimental Psychology: Learning, Memory, and Cognition, 24, 1459-1482. doi: 10.1037/0278-7393.24.6.1459
46. Malmberg, K. J. (2008). Recognition memory: A review of the critical findings and an integrated theory for relating them. Cognitive Psychology, 57, 335-384. doi:10.1016/j.cogpsych.2008.02.004
47. Malmberg, K. J., & Annis, J. (2011). On the relationship between memory and perception: Sequential dependencies in recognition memory testing. Journal of Experimental Psychology: General. Advance online publication. doi:10.1037/a0025277
48. Malmberg, K. J., Criss, A. H., Gangwani, T., & Shiffrin, R. M. (2012). Overcoming the negative consequences of interference that results from recognition memory testing. Psychological Science, 23, 115-119. doi: 10.1177/0956797611430692
49. Malmberg, K. J., & Xu, J. (2006). The influence of averaging and noisy decision strategies on the recognition memory ROC. Psychonomic Bulletin & Review, 13, 99-105. doi:10.3758/BF03193819
50. Mickes, L., Hwe, V., Wais, P., & Wixted, J. T. (2011). Strong memories are hard to scale. Journal of Experimental Psychology: General, 140, 239-257. doi:10.1037/a0023007
51. Morrell, H. E. R., Gaitan, S., & Wixted, J. T. (2002). On the nature of the decision axis in signal detection-based models of recognition memory. Journal of Experimental Psychology: Learning, Memory, and Cognition, 28, 1095-1110. doi:10.1037/0278-7393.28.6.1095
52. Moshagen, M., & Hilbig, B. E. (2011). Methodological notes on model comparisons and strategy classification: A falsificationist proposition. Judgment and Decision Making, 6, 814-820.
53. Mueller, S. T., & Weidemann, C. T. (2008). Decision noise: An explanation for observed violations of signal detection theory. Psychonomic Bulletin & Review, 15, 465-494. doi:10.3758/PBR.15.3.465
54. Murdock, B. B. (1963). An analysis of the recognition process. In C. N. Cofer & B. S. Musgrave (Eds.), Verbal behavior and learning (pp. 10-32). New York, NY: McGraw-Hill. doi:10.1037/11178-002
55. Parks, C. M., & Yonelinas, A. P. (2009). Evidence for a memory threshold in second-choice recognition memory responses. Proceedings of the National Academy of Sciences, USA, 106, 11515-11519. doi:10.1073/ pnas.0905505106
56. Peirce, J. W. (2007). PsychoPy-Psychophysics software in Python. Journal of Neuroscience Methods, 162, 8-13. doi:10.1016/j.jneumeth .2006.11.017
57. Pleskac, T. J. (2007). A signal detection analysis of the recognition heuristic. Psychonomic Bulletin & Review, 14, 379-391. doi:10.3758/ BF03194081
58. Pleskac, T. J., & Busemeyer, J. (2010). Two-stage dynamic signal detection: A theory of confidence, choice, and response time. Psychological Review, 117, 864-901. doi:10.1037/a0019737
59. Pratte, M. S., Rouder, J. N., & Morey, R. D. (2010). Separating mnemonic process from participant and item effects in the assessment of ROC asymmetries. Journal of Experimental Psychology: Learning, Memory, and Cognition, 36, 224-232. doi:10.1037/a0017682
60. Ratcliff, R. (1978). A theory of memory retrieval. Psychological Review, 85, 59-108. doi:10.1037/0033-295X.85.2.59
61. Ratcliff, R., McKoon, G., & Tindall, M. (1994). Empirical generality of data from recognition memory receiver-operating characteristic functions and implications for the global memory models. Journal of Experimental Psychology: Learning, Memory, and Cognition, 20, 763-785. doi:10.1037/0278-7393.20.4.763
62. Ratcliff, R., & Starns, J. J. (2009). Modeling confidence and response time in recognition memory. Psychological Review, 116, 59-83. doi: 10.1037/a0014086
63. R Development Core Team. (2011). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing.
64. Rosner, B. S., & Kochanski, G. (2009). The law of categorical judgment (corrected) and the interpretation of changes in psychophysical performance. Psychological Review, 116, 116-128. doi:10.1037/a0014463
65. Rouder, J. N., Pratte, M. S., & Morey, R. D. (2010). Latent mnemonic strengths are latent: A comment on Mickes, Wixted, and Wais (2007). Psychonomic Bulletin & Review, 17, 427-435. doi:10.3758/ PBR.17.3.427
66. Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6, 461-464. doi:10.1214/aos/1176344136
67. Self, S. G., & Liang, K. (1987). Asymptotic properties of maximum likelihood estimators and likelihood ratio tests under nonstandard conditions. Journal of the American Statistical Association, 82, 605-610. doi:10.2307/2289471
68. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.
69. Shapiro, A. (1985). Asymptotic distribution of test statistics in the analysis of moment structures under inequality constraints. Biometrika, 72, 133-144. doi:10.1093/biomet/72.1.133
70. Shiffrin, R. M., & Steyvers, M. (1997). A model for recognition memory: REM-Retrieving effectively from memory. Psychonomic Bulletin & Review, 4, 145-166. doi:10.3758/BF03209391
71. Singer, M., & Wixted, J. T. (2006). Effect of delay on recognition decisions: Evidence for a criterion shift. Memory & Cognition, 34, 125-137. doi:10.3758/BF03193392
72. Solomon, J. A. (2007). Intrinsic uncertainty explains second responses. Spatial Vision, 20, 45-60. doi:10.1163/156856807779369715
73. Stretch, V., & Wixted, J. T. (1998). On the difference between strengthbased and frequency-based mirror effects in recognition memory. Journal of Experimental Psychology: Learning, Memory, and Cognition, 24, 1379-1396. doi:10.1037/0278-7393.24.6.1379
74. Swets, J. A., Tanner, W. P., Jr., & Birdsall, T. G. (1961). Decision processes in perception. Psychological Review, 68, 301-340. doi: 10.1037/h0040547
75. Thomas, E. A., & Legge, D. (1970). Probability matching as a basis for detection and recognition decisions. Psychological Review, 77, 65-72. doi:10.1037/h0028579
76. Treisman, M. (1987). Effects of the setting and adjustment of decision criteria on psychophysical performance. In E. E. Roskam & R. Suck (Eds.), Progress in mathematical psychology (pp. 253-297). New York, NY: Elsevier Science.
77. Treisman, M. (2002). Is signal detection theory fundamentally flawed? A response to Balakrishnan (1998a, 1998b, 1999). Psychonomic Bulletin & Review, 9, 845-857.
78. Treisman, M., & Faulkner, A. (1984). The effect of signal probability on the slope of the receiver operating characteristic given by the rating procedure. British Journal of Mathematical and Statistical Psychology, 37, 199-215. doi:10.1111/j.2044-8317.1984.tb00800.x
79. Treisman, M., & Williams, T. C. (1984). A theory of criterion setting with an application to sequential dependencies. Psychological Review, 91, 68-111. doi:10.1037/0033-295X.91.1.68
80. Turner, B., Van Zandt, T., & Brown, S. (2011). A dynamic stimulus-driven model of signal detection. Psychological Review, 118, 583-613.
81. Van Zandt, T. (2000). ROC curves and confidence judgments in recognition memory. Journal of Experimental Psychology: Learning, Memory, and Cognition, 26, 582-600. doi:10.1037/0278-7393.26.3.582
82. Verde, M. F., & Rotello, C. M. (2004). ROC curves show that the revelation effect is not a single phenomenon. Psychonomic Bulletin & Review, 11, 560-566. doi:10.3758/BF03196611
83. Wickens, T. D. (2002). Elementary signal detection theory. Oxford, England: Oxford University Press.
84. Wixted, J. T. (2007). Dual-process theory and signal-detection theory of recognition memory. Psychological Review, 114, 152-176. doi:10.1037/ 0033-295X.114.1.152
85. Wixted, J. T., & Gaitan, S. (2002). Cognitive theories as reinforcement history surrogates: The case of likelihood ratio models of human recognition memory. Animal Learning & Behavior, 30, 289-305. doi: 10.3758/BF03195955
86. Yonelinas, A. P., & Parks, C. M. (2007). Receiver operating characteristics (ROCs) in recognition memory: A review. Psychological Bulletin, 133, 800-832. doi:10.1037/0033-2909.133.5.800