Participant Nonnaiveté and the reproducibility of cognitive psychology

Psychonomic Bulletin and Review

Volumn 25, Issue 5, 2018, Pages 1968-1972

  Zwaan, Rolf A ^a   Pecher, Diane ^a   Paolacci, Gabriele ^a   Bouwmeester, Samantha ^a   Verkoeijen, Peter ^a,b   Dijkstra, Katinka ^a   Zeelenberg, René ^a  

^a ERASMUS UNIVERSITY ROTTERDAM   (Netherlands)

^b AVANS UNIVERSITY OF APPLIED SCIENCES   (Netherlands)

Author keywords

Language; Memory; Perception; Replication; Reproducibility

Indexed keywords

ADULT; EXECUTIVE FUNCTION; FEMALE; HUMAN; INHIBITION (PSYCHOLOGY); LANGUAGE; MALE; MEMORY; PERCEPTION; PHYSIOLOGY; PSYCHOLOGY; PSYCHOMOTOR PERFORMANCE; REPRODUCIBILITY; STANDARDS; YOUNG ADULT;

ADULT; COGNITIVE SCIENCE; EXECUTIVE FUNCTION; FEMALE; HUMANS; INHIBITION (PSYCHOLOGY); LANGUAGE; MALE; MEMORY; PERCEPTION; PSYCHOLOGY; PSYCHOMOTOR PERFORMANCE; REPRODUCIBILITY OF RESULTS; YOUNG ADULT;

EID: 85025839187     PISSN: 10699384     EISSN: 15315320     Source Type: Journal    
DOI: 10.3758/s13423-017-1348-y     Document Type: Article

References (27)

1. Alogna, V. K., Attaya, M. K., Aucoin, P., Bahnik, S., Birch, S., Birt, A. R., & Zwaan, R. A. (2014). Registered replication report: Schooler & Engstler-Schooler (1990). Perspectives on Psychological Science, 9, 556–578
2. Bahník, S., & Vranka, M. A. (2017). If it’s difficult to pronounce, it might not be risky. The effect of fluency on judgment of risk does not generalize to new stimuli. Psychological Science, 28, 427–436
3. Bohannon, J. (2016). Mechanical Turk upends social sciences. Science, 352, 1263–1264
4. Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon's Mechanical Turk a new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6, 3–5
5. Chandler, J., Mueller, P., & Paolacci, G. (2014). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, 46, 112–130
6. Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015). Using nonnaive participants can reduce effect sizes. Psychological Science, 26, 1131–1139
7. Craft, J. L., & Simon, J. R. (1970). Processing symbolic information from a visual display: Interference from an irrelevant directional cue. Journal of Experimental Psychology, 83, 415–420
8. Crump, M. J. C., McDonnell, J. V., & Gureckis, V. M. (2013). Evaluating Amazon’s Mechanical Turk as a tool for experimental behavioral research. PLoS One, 8, e57410
9. DeVoe, S. E., & House, J. (2016). Replications with MTurkers who are naïve versus experienced with academic studies: A comment on Connors, Khamitov, Moroz, Campbell, and Henderson (2015). Journal of Experimental Social Psychology, 67, 65–67
10. Eerland, A., Sherrill, A. M., Magliano, J. P., Zwaan, R. A., Arnal, J. D., Aucoin, P., & Prenoveau, J. M. (2016). Registered replication report: Hart & Albarracín (2011). Perspectives on Psychological Science, 11, 158–171
11. Eriksen, B. A., & Eriksen, C. W. (1974). Effects of noise letters upon the identification of a target letter in a non search task. Perception and Psychophysics, 16, 143–149
12. Forster, K. I., & Davis, C. (1984). Repetition Priming and Frequency Attenuation. Journal of Experimental Psychology: Learning, Memory, and Cognition, 10, 4
13. Greene, R. L. (1989). Spacing effects in memory: Evidence for a two-process account. Journal of Experimental Psychology: Learning, Memory, and Cognition, 15, 371–377
14. Hagger, M. S., Chatzisarantis, N. L. D., Alberts, H., Anggono, C. O., Batailler, C., Birt, A. R.,.. Zwienenberg, M. (2016). A multilab preregistered replication of the ego-depletion effect. Perspectives on Psychological Science, 11, 546–573
15. Lykken, D. T. (1968). Statistical significance in psychological research. Psychological Bulletin, 70, 151–159
16. Meyer, D. E., & Schvaneveldt, R. W. (1971). Facilitation in recognizing pairs of words: Evidence of a dependence between retrieval operations. Journal of Experimental Psychology, 90, 227–234
17. Murdock, B. B., Jr. (1962). The serial position effect of free recall. Journal of Experimental Psychology, 64, 482–488
18. Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. doi:10.1126/science.aac4716
19. Paolacci, G., & Chandler, J. (2014). Inside the Turk: Understanding Mechanical Turk as a participant pool. Current Directions in Psychological Science, 23, 184–188
20. Popper, K. R. (1959). The Logic of Scientific Discovery, translation of Logik der Forschung. Oxford: Routledge
21. Rand, D. G., Peysakhovich, A., Kraft-Todd, G. T., Newman, G. E., Wurzbacher, O., Nowak, A. W., & Greene, J. D. (2014). Social heuristics shape intuitive cooperation. Nature Communications, 5, 4677
22. Roediger, H. L., & McDermott, K. B. (1995). Creating false memories: Remembering words not presented in lists. Journal of Experimental Psychology: Learning, Memory, and Cognition, 24, 803–814
23. Stewart, N., Ungemach, C., Harris, A. J. L., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015). The average laboratory samples a population of 7,30 Amazon Mechanical Turk workers. Judgment and Decision Making, 10, 479–491
24. JASP Team (2017). JASP (Version 0.8.1.2)[Computer software]
25. Wagenmakers, E.-J., Beek, T., Dijkhoff, L., Gronau, Q. F., Acosta, A., Adams, R. B., Jr., & Zwaan, R. A. (2016). Registered Replication Report: Strack, Martin, & Stepper (1988). Perspectives on Psychological Science, 11, 917–928
26. Zwaan, R. A., & Pecher, D. (2012). Revisiting mental simulation in language comprehension: Six replication attempts. PLoS One, 7, e51382
27. Zwaan, R. A., Yaxley, R., & Stanfield, R. (2002). Language comprehenders mentally represent the shape of objects. Psychological Science, 13, 168–171. Experiment 1