When more data steer us wrong: replications with the wrong dependent measure perpetuate erroneous conclusions

Psychonomic Bulletin and Review

Volumn 22, Issue 4, 2015, Pages 944-954

  Rotello, Caren M ^a   Heit, Evan ^b   Dubé, Chad ^c  

^a UNIVERSITY OF MASSACHUSETTS   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c UNIVERSITY OF SOUTH FLORIDA   (United States)

Author keywords

Eyewitness memory; Signal detection theory; Social cognition; Statistical inference

Indexed keywords

CHILD; CHILD WELFARE; HUMAN; LOGIC; MEMORY; PERCEPTION; PERCEPTIVE DISCRIMINATION; PSYCHOLOGICAL THEORY; RECEIVER OPERATING CHARACTERISTIC; REPRODUCIBILITY; SOCIAL PSYCHOLOGY;

CHILD; CHILD WELFARE; HUMANS; LOGIC; MEMORY; PSYCHOLOGICAL THEORY; PSYCHOLOGY, SOCIAL; REPRODUCIBILITY OF RESULTS; ROC CURVE; SIGNAL DETECTION, PSYCHOLOGICAL; SOCIAL PERCEPTION;

EID: 84938967611     PISSN: 10699384     EISSN: 15315320     Source Type: Journal    
DOI: 10.3758/s13423-014-0759-2     Document Type: Article

References (70)

1. Alogna, V. K., Attaya, M. K., Aucoin, P., Bahnik, S., Birch, S., Birt, A. R., & Zwaan, R. A. (2014). Registered replication report: Schooler and Engstler-Schooler (1990). Perspectives on Psychological Science, 9, 556–578. doi:10.1177/1745691614545653
2. Buchner, A., Erdfelder, E., & Vaterrodt-Plünnecke, B. (1995). Toward unbiased measurement of conscious and unconscious memory processes with the process dissociation framework. Journal of Experimental Psychology: General, 124, 137–160. doi:10.1037/0096-3445.124.2.137
3. Carlson, C. A., & Carlson, M. A. (2014). An evaluation of lineup presentation, weapon presence, and a distinctive feature using ROC analysis. Journal of Applied Research in Memory and Cognition, 3, 45–53.
4. Correll, J. (2008). 1/f noise and effort on implicit measures of bias. Journal of Personality and Social Psychology, 94, 48–59. doi:10.1037/0022-3514.94.1.48
5. Correll, J., Park, B., Judd, C. M., Wittenbrink, B., Sadler, M. S., & Keesee, T. (2007a). Across the thin blue line: Police officers and racial bias in the decision to shoot. Journal of Personality and Social Psychology, 92, 1006–1023.
6. Correll, J., Park, B., Judd, C. M., & Wittenbrink, B. (2002). The police officer’s dilemma: Using ethnicity to disambiguate potentially threatening individuals. Journal of Personality and Social Psychology, 83, 1314–1329.
7. Correll, J., Park, B., Judd, C. M., & Wittenbrink, B. (2007b). The influence of stereotypes on decisions to shoot. European Journal of Social Psychology, 37, 1102–1117.
8. Dobolyi, D. G., & Dodson, C. S. (2013). Eyewitness confidence in simultaneous and sequential lineups: A criterion shift account for sequential mistaken identification overconfidence. Journal of Experimental Psychology: Applied, 19, 345–357.
9. Dougal, S., & Rotello, C. M. (2007). “Remembering” emotional words is based on response bias, not recollection. Psychonomic Bulletin & Review, 14, 423–429. doi:10.3758/BF03194083
10. Dube, C., & Rotello, C. M. (2012). Binary ROCs in perception and recognition memory are curved. Journal of Experimental Psychology: Learning, Memory, and Cognition, 38, 130–151. doi:10.1037/a0024957
11. Dube, C., Rotello, C. M., & Heit, E. (2010). Assessing the belief bias effect with ROCs: It’s a response bias effect. Psychological Review, 117, 831–863.
12. Dunn, J. C. (2004). Remember–know: A matter of confidence. Psychological Review, 111, 524–542. doi:10.1037/0033-295X.111.2.524
13. Dusoir, A. E. (1975). Treatments of bias in detection and recognition models: A review. Perception & Psychophysics, 17, 167–178.
14. Egu, C. L., & Weiss, D. J. (2003). The role of race and severity of abuse in teachers’ recognition or reporting of child abuse. Journal of Child and Family Studies, 12, 465–474.
15. Evans, J. S. B. T., Barston, J. L., & Pollard, P. (1983). On the conflict between logic and belief in syllogistic reasoning. Memory & Cognition, 11, 295–306.
16. Evans, J. S. B. T., & Curtis-Holmes, J. (2005). Rapid responding increases belief bias: Evidence for the dual-process theory of reasoning. Thinking & Reasoning, 11, 382–390.
17. Evans, K., Rotello, C. M., Li, X., & Rayner, K. (2009). Scene perception and memory revealed by eye movements and ROC analyses: Does a cultural difference truly exist? Quarterly Journal of Experimental Psychology, 62, 276–285.
18. Feynman, R. P., Leighton, R., & Hutchings, E. (1985). Surely you’re joking, Mr. Feynman! (Adventures of a curious character). New York, NY: W. W. Norton.
19. Fiedler, K., Kutzner, F., & Krueger, J. I. (2012). The long way from a-error control to validity proper: Problems with a short-sighted false-positive debate. Perspectives on Psychological Science, 7, 661–669. doi:10.1177/1745691612462587
20. Francis, G. (2012). Too good to be true: Publication bias in two prominent studies from experimental psychology. Psychonomic Bulletin & Review, 19, 151–156. doi:10.3758/s13423-012-0227-9
21. Goel, V., Buchel, C., Frith, C., & Dolan, R. (2000). Dissociation of mechanisms underlying syllogistic reasoning. NeuroImage, 12, 504–514.
22. Goel, V., & Dolan, R. J. (2003). Explaining modulation of reasoning by belief. Cognition, 87, 11–22.
23. Goodman, L. A., & Kruskal, W. H. (1954). Measures of association for cross classifications. Journal of the American Statistical Association, 49, 732–764.
24. Greenwald, A. G., Oakes, M. A., & Hoffman, H. G. (2003). Targets of discrimination: Effects of race on responses to weapons holders. Journal of Experimental Social Psychology, 39, 399–405.
25. Gronlund, S. D., Carlson, C. A., Neuschatz, J. S., Goodsell, C. A., Wetmore, S. A., Wooten, A., & Graham, M. (2012). Showups versus lineups: An evaluation using ROC analysis. Journal of Applied Research in Memory and Cognition, 1, 221–228.
26. Heit, E., & Rotello, C. M. (2010). Relations between inductive reasoning and deductive reasoning. Journal of Experimental Psychology: Learning, Memory, and Cognition, 36, 805–812.
27. Heit, E., & Rotello, C. M. (2012). The pervasive effects of argument length on inductive reasoning. Thinking & Reasoning (Special Issue on Reasoning and Argumentation), 18, 244–277.
28. Heit, E., & Rotello, C. M. (2014). Traditional difference-score analyses of reasoning are flawed. Cognition, 131, 75–91.
29. Ioannidis, J. P. A., Munafò, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014). Publication and other reporting biases in cognitive sciences: Detection, prevalence, and prevention. Trends in Cognitive Sciences, 18, 235–241. doi:10.1016/j.tics.2014.02.010
30. Jacoby, L. L. (1991). A process dissociation framework: Separating automatic from intentional uses of memory. Journal of Memory and Language, 30, 513–541. doi:10.1016/0749-596X(91)90025-F
31. Jaeggi, S. M., Buschkuehl, M., Jonides, J., & Shah, P. (2011). Short- and long-term benefits of cognitive training. Proceedings of the National Academy of Sciences, 108, 10081–10086. doi:10.1073/pnas.1103228108
32. Johnson-Laird, P. N. (1983). Mental models: Towards a cognitive science of language, inference, and consciousness. Cambridge, MA: Harvard University Press.
33. Klein, R. A., Ratliff, K., Vianello, M., Adams, R. B., Jr., Bahník, Š., Bernstein, M. J.,.. Nosek, B. A. (2014). Investigating variation in replicability: A “many labs” replication project. Retrieved from Open Science Framework, http://osf.io/wx7ck
34. Lambert, A. J., Payne, B. K., Jacoby, L. L., Shaffer, L. M., Chasteen, A. L., & Khan, S. R. (2003). Stereotypes as dominant responses: On the “social facilitation” of prejudice in anticipated public contexts. Journal of Personality and Social Psychology, 84, 277–295.
35. Macmillan, N. A., & Creelman, C. D. (2005). Detection theory: A user’s guide (2nd ed.). Mahwah, NJ: Erlbaum.
36. Mansell, J., Ota, R., Erasmus, R., & Marks, K. (2011). Reframing child protection: A response to a constant crisis of confidence in child protection. Children and Youth Services Review, 33, 2076–2086. doi:10.1016/j.childyouth.2011.04.019
37. Masson, M. E. J., & Rotello, C. M. (2009). Sources of bias in the Goodman–Kruskal gamma coefficient measure of association: Implications for studies of metacognitive processes. Journal of Experimental Psychology: Learning, Memory, and Cognition, 35, 509–527.
38. Miaskowski, C., Dodd, M., West, C., Paul, S. M., Schumacher, K., Tripathy, D., & Koo, P. (2007). The use of a responder analysis to identify differences in patient outcomes following a self-care intervention to improve cancer pain management. Pain, 129, 55–63. doi:10.1016/j.pain.2006.09.031
39. Mickes, L., Flowe, H. D., & Wixted, J. T. (2012). Receiver operating characteristic analysis of eyewitness memory: Comparing the diagnostic accuracy of simultaneous versus sequential lineups. Journal of Experimental Psychology: Applied, 18, 361–376.
40. Mumpower, J. L. (2010). Disproportionality at the “front end” of the child welfare services system: An analysis of rates of referrals, “hits”, “ misses”, and “false alarms”. Journal of Health and Human Services Administration, 33, 364–405.
41. Mumpower, J. L., & McClelland, G. H. (2014). A signal detection analysis of racial and ethnic disproportionality in the referral and substantiation processes of the U.S. child welfare services system. Judgment and Decision Making, 9, 114–128.
42. Open Science Collaboration (2012). An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science, 7, 657–660. doi:10.1177/1745691612462588
43. Payne, B. K. (2001). Prejudice and perception: The role of automatic and controlled processes in misperceiving a weapon. Journal of Personality and Social Psychology, 81, 181–192.
44. Payne, B. K., Lambert, A. J., & Jacoby, L. L. (2002). Best laid plans: Effects of goals on accessibility bias and cognitive control in race-based misperceptions of weapons. Journal of Experimental Social Psychology, 38, 384–396.
45. Pazzaglia, A. M., Dube, C., & Rotello, C. M. (2013). A critical comparison of discrete-state and continuous models of recognition memory: Implications for recognition and beyond. Psychological Bulletin, 139, 1173–1203. doi:10.1037/a0033044
46. Plant, E. A., & Peruche, B. M. (2005). The consequences of race for police officers’ responses to criminal suspects. Psychological Science, 16, 180–183.
47. Plant, E. A., Peruche, B. M., & Butz, D. A. (2005). Eliminating automatic racial bias: Making race non-diagnostic for responses to criminal suspects. Journal of Experimental Social Psychology, 41, 141–156.
48. Police Executive Research Forum. (2013). A national survey of eyewitness identification procedures in law enforcement agencies. Retrieved from www.policeforum.org/free-online-documents
49. Pollack, I., & Norman, D. A. (1964). A non-parametric analysis of recognition experiments. Psychonomic Science, 1, 125–126.
50. Risen, J. L., & Critcher, C. R. (2011). Visceral fit: While in a visceral state, associated states of the world seem more likely. Journal of Personality and Social Psychology, 100, 777–793.
51. Rotello, C. M., & Heit, E. (2009). Modeling the effects of argument length and validity on inductive and deductive reasoning. Journal of Experimental Psychology: Learning, Memory, and Cognition, 35, 1317–1330.
52. Rotello, C. M., & Heit, E. (2014). The neural correlates of belief bias: Activation in inferior frontal cortex reflects response rate differences. Frontiers in Human Neuroscience, 8, 862. doi:10.3389/fnhum.2014.00862
53. Rotello, C. M., Macmillan, N. A., Reeder, J. A., & Wong, M. (2005). The remember response: Subject to bias, graded, and not a process-pure indicator of recollection. Psychonomic Bulletin & Review, 12, 865–873. doi:10.3758/BF03196778
54. Rotello, C. M., Masson, M. E. J., & Verde, M. F. (2008). Type I error rates and power analyses for single-point sensitivity measures. Perception & Psychophysics, 70, 389–401. doi:10.3758/PP.70.2.389
55. Schooler, J. W., & Engstler-Schooler, T. Y. (1990). Verbal overshadowing of visual memories: Some things are better left unsaid. Cognitive Psychology, 22, 36–71. doi:10.1016/0010-0285(90)90003-M
56. Sedlak, A. J., Mettenburg, J., Basena, M., Petta, I., McPherson, K., Greene, A., & Li, S. (2010). Fourth National Incidence Study of Child Abuse and Neglect (NIS-4): Report to Congress. Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families.
57. Sherman, J. W., Gawronski, B., Gonsalkorale, K., Hugenberg, K., Allen, T. J., & Groom, C. J. (2008). The self-regulation of automatic associations and behavioral impulses. Psychological Review, 115, 314–335.
58. Steblay, N. K., Dysart, J. E., & Wells, G. L. (2011). Seventy-two tests of the sequential lineup superiority effect: A meta-analysis and policy discussion. Psychology, Public Policy, and Law, 17, 99–139. doi:10.1037/a0021650
59. Stollstorff, M., Vartanian, O., & Goel, V. (2012). Levels of conflict in reasoning modulate right lateral prefrontal cortex. Brain Research, 1428, 24–32.
60. Swets, J. A. (1986a). Form of empirical ROCs in discrimination and diagnostic tasks: Implications for theory and measurement of performance. Psychological Bulletin, 99, 181–198. doi:10.1037/0033-2909.99.2.181
61. Swets, J. A. (1986b). Indices of discrimination or diagnostic accuracy: Their ROCs and implied models. Psychological Bulletin, 99, 100–117. doi:10.1037/0033-2909.99.1.100
62. Tidwell, J. W., Dougherty, M. R., Chrabaszcz, J. R., Thomas, R. P., & Mendoza, J. L. (2014). What counts as evidence for working memory training? Problems with correlated gains and dichotomization. Psychonomic Bulletin & Review, 21, 620–628.
63. Trippas, D., Handley, S. J., & Verde, M. F. (2013). The SDT model of belief bias: Complexity, time, and cognitive ability mediate the effects of believability. Journal of Experimental Psychology: Learning, Memory, and Cognition, 39, 1393–1402.
64. z, and A'. Perception & Psychophysics, 68, 643–654. doi:10.3758/BF03208765
65. Verde, M. F., & Rotello, C. M. (2003). Does familiarity change in the revelation effect? Journal of Experimental Psychology: Learning, Memory, and Cognition, 29, 739–746. doi:10.1037/0278-7393.29.5.739
66. Verfaellie, M., & McGwin, J. (2011, December). The case of Diederik Stapel. Psychological Science Agenda. Retrieved March 7, 2014, from www.apa.org/science/about/psa/2011/12/diederik-stapel.aspx
67. Wells, G. L., & Lindsay, R. C. (1980). On estimating the diagnosticity of eyewitness nonidentifications. Psychological Bulletin, 88, 776–784. doi:10.1037/0033-2909.88.3.776
68. Wells, G. L., Malpass, R. S., Lindsay, R. C. L., Fisher, R. P., Turtle, J. W., & Fulero, S. M. (2000). From the lab to the police station: A successful application of eyewitness research. American Psychologist, 55, 581–598. doi:10.1037/0003-066X.55.6.581
69. White, C. N., Kapucu, A., Bruno, D., Rotello, C. M., & Ratcliff, R. (2014). Response bias for emotional words in immediate recognition memory is due to relatedness rather than emotional valence. Cognition and Emotion, 28, 867–880. doi:10.1080/02699931.2013.858028
70. Wixted, J. T., & Mickes, L. (2012). The field of eyewitness memory should abandon “probative value” and embrace receiver operating characteristic analysis. Perspectives on Psychological Science, 7, 275–278. doi:10.1177/1745691612442906