Promises and pitfalls of Web-based experimentation in the advance of replicable psychological science: A reply to Plant (2015)

Behavior Research Methods

Volumn 48, Issue 4, 2016, Pages 1713-1717

  van Steenbergen, Henk ^a,b   Bocanegra, Bruno R ^a  

^a LEIDEN UNIVERSITY   (Netherlands)

^b NONE   (Netherlands)

Author keywords

Critical timing; Online experimentation; Psychological science; Qualtrics Reaction Time Engine (QRTEngine); Reaction times; Replication; Web based psychology experiments

Indexed keywords

BEHAVIORAL RESEARCH; HUMAN; INTERNET; METHODOLOGY; PSYCHOLOGY;

BEHAVIORAL RESEARCH; HUMANS; INTERNET; PSYCHOLOGY; RESEARCH DESIGN;

EID: 84946430550     PISSN: 1554351X     EISSN: 15543528     Source Type: Journal    
DOI: 10.3758/s13428-015-0677-x     Document Type: Article

References (45)

1. Alogna, V. K., Attaya, M. K., Aucoin, P., Bahnik, S., Birch, S., Birt, A. R., & Zwaan, R. A. (2014). Registered replication report: Schooler & Engstler-Schooler (1990). Perspectives on Psychological Science, 9, 556–578.
2. Asendorpf, J. B., & Conner, M. (2012). Recommendations for increasing replicability in psychology. European Journal of Personality, 119, 108–119.
3. Barnhoorn, J. S., Haasnoot, E., Bocanegra, B. R., & van Steenbergen, H. (2015). QRTEngine: An easy solution for running online reaction time experiments using Qualtrics. Behavior Research Methods. doi:10.3758/s13428-014-0530-7
4. Brunswik, E. (1955). Representative design and probabilistic theory in a functional psychology. Psychological Review, 62, 193–217. doi:10.1037/h0047470
5. Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon’s Mechanical Turk a new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6, 3–5. doi:10.1177/1745691610393980
6. Chetverikov, A., & Upravitelev, P. (2015). Online versus offline: The Web as a medium for response time data collection. Behavior Research Methods. doi:10.3758/s13428-015-0632-x
7. Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013). Evaluating Amazon’s Mechanical Turk as a tool for experimental behavioral research. PLoS ONE, 8, e57410. doi:10.1371/journal.pone.0057410
8. de Leeuw, J. R. (2015). jsPsych: A JavaScript library for creating behavioral experiments in a Web browser. Behavior Research Methods, 47, 1–12. doi:10.3758/s13428-014-0458-y
9. de Leeuw, J. R., & Motz, B. A. (2015). Psychophysics in a Web browser? Comparing response times collected with JavaScript and Psychophysics Toolbox in a visual search task. Behavior Research Methods. doi:10.3758/s13428-015-0567-2
10. Doyen, S., Klein, O., Pichon, C. L., & Cleeremans, A. (2012). Behavioral priming: It’s all in the mind, but whose mind. PLoS ONE, 7, e29081. doi:10.1371/journal.pone.0029081
11. Eichstaedt, J. C., Schwartz, H. A., Kern, M. L., Park, G., Labarthe, D. R., Merchant, R. M., & Sap, M. (2015). Psychological language on twitter predicts county-level heart disease mortality. Psychological Science, 26, 159–169.
12. Eimer, M., & Schlaghecken, F. (2002). Links between conscious awareness and response inhibition: Evidence from masked priming. Psychonomic Bulletin & Review, 9, 514–520. doi:10.3758/BF03196307
13. Garaizar, P., Vadillo, M. A., & López-de-Ipiña, D. (2014). Presentation accuracy of the Web revisited: Animation methods in the HTML5 era. PLoS ONE, 9, e109812. doi:10.1371/journal.pone.0109812
14. Gratton, G., Coles, M. G. H., & Donchin, E. (1992). Optimizing the use of information: Strategic control of activation of responses. Journal of Experimental Psychology: General, 121, 480–506. doi:10.1037/0096-3445.121.4.480
15. Griffiths, T. L. (2015). Manifesto for a new (computational) cognitive revolution. Cognition, 135, 21–23. doi:10.1016/j.cognition.2014.11.026
16. Hommel, B., & Akyürek, E. G. (2005). Lag-1 sparing in the attentional blink: Benefits and costs of integrating two events into a single episode. Quarterly Journal of Experimental Psychology, 58A, 1415–1433. doi:10.1080/02724980443000647
17. Ioannidis, J. P. (2005). Why most published research findings are false. Chance, 18, 40–47.
18. Jolicœur, P., Snow, D., & Murray, J. (1987). The time to identify disoriented letters: Effects of practice and font. Canadian Journal of Psychology, 41, 303–316. doi:10.1037/h0084159
19. Keller, F., Gunasekharan, S., Mayo, N., & Corley, M. (2009). Timing accuracy of Web experiments: A case study using the WebExp software package. Behavior Research Methods, 41, 1–12. doi:10.3758/BRM.41.1.12
20. Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Jr., Bahník, Š., Bernstein, M. J., & Woodzicka, J. A. (2015). Investigating variation in replicability. Social Psychology, 45, 142–152.
21. Lehner, P. N. (1998). Handbook of ethological methods. Cambridge, UK: Cambridge University Press.
22. Martin, P., & Bateson, P. P. G. (1993). Measuring behaviour: An introductory guide. Cambridge, UK: Cambridge University Press.
23. Mason, W., & Suri, S. (2012). Conducting behavioral research on Amazon’s Mechanical Turk. Behavior Research Methods, 44, 1–23. doi:10.3758/s13428-011-0124-6
24. Metha, A. B., Vingrys, A. J., & Badcock, D. R. (1993). Calibration of a color monitor for visual psychophysics. Behavior Research Methods, Instruments, & Computers, 25, 371–383. doi:10.3758/BF03204528
25. Most, S. B., Chun, M. M., Widders, D. M., & Zald, D. H. (2005). Attentional rubbernecking: Cognitive control and personality in emotion-induced blindness. Psychonomic Bulletin & Review, 12, 654–661. doi:10.3758/BF03196754
26. Neath, I., Earle, A., Hallett, D., & Surprenant, A. M. (2011). Response time accuracy in Apple Macintosh computers. Behavior Research Methods, 43, 353–362. doi:10.3758/s13428-011-0069-9
27. Open Science Collaboration. (2012). An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science, 7, 657–660. doi:10.1177/1745691612462588
28. Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, aac4716. doi:10.1126/science.aac4716
29. Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411–419.
30. Park, G., Schwartz, H. A., Eichstaedt, J. C., Kern, M. L., Kosinski, M., Stillwell, D. J., & Seligman, M. E. P. (2015). Automatic personality assessment through social media language. Journal of Personality and Social Psychology, 108, 934–952.
31. Pashler, H., & Wagenmakers, E.-J. (2012). Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence? Perspectives on Psychological Science, 7, 528–530. doi:10.1177/1745691612465253
32. Plant, R. R. (2015). A reminder on millisecond timing accuracy and potential replication failure in computer-based psychology experiments: An open letter. Behavior Research Methods. doi:10.3758/s13428-015-0577-0
33. Plant, R. R., Hammond, N., & Turner, G. (2004). Self-validating presentation and response timing in cognitive paradigms: How and why? Behavior Research Methods, Instruments, & Computers, 36, 291–303. doi:10.3758/BF03195575
34. Plant, R. R., & Turner, G. (2009). Millisecond precision psychological research in a world of commodity computers: New hardware, new problems? Behavior Research Methods, 41, 598–614. doi:10.3758/BRM.41.3.598
35. Pohl, C., Kunde, W., Ganz, T., Conzelmann, A., Pauli, P., & Kiesel, A. (2014). Gaming to see: Action Video Gaming enhances processing of masked stimuli. Frontiers in Psychology, 5(70), 1–9.
36. Reimers, S., & Stewart, N. (2015). Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments. Behavior Research Methods, 47, 309–327. doi:10.3758/s13428-014-0471-1
37. Schlaghecken, F., & Maylor, E. A. (2005). Motor control in old age: Evidence of impaired low-level inhibition. Journals of Gerontology, 60B, 158–P161.
38. Schubert, T. W., Murteira, C., Collins, E. C., & Lopes, D. (2013). ScriptingRT: A software library for collecting response latencies in online studies of cognition. PLoS ONE, 8, e67769. doi:10.1371/journal.pone.0067769
39. Serrien, D. J., Sovijärvi-Spapé, M. M., & Rana, G. (2012). Subliminal priming and effects of hand dominance. Acta Psychologica, 141, 73–77.
40. Slote, J., & Strand, J. F. (2015). Conducting spoken word recognition research online: Validation and a new timing method. Behavior Research Methods. doi:10.3758/s13428-015-0599-7
41. Stroebe, W., & Strack, F. (2014). The alleged crisis and the illusion of exact replication. Perspectives on Psychological Science, 9, 59–71.
42. Sumner, P., Tsai, P. C., Yu, K., & Nachev, P. (2006). Attentional modulation of sensorimotor processes in the absence of perceptual awareness. Proceedings of the National Academy of Sciences, 103, 10520–10525.
43. Tinbergen, N. (1951). The study of instinct. Oxford, UK: Oxford University Press.
44. von Bastian, C. C., Locher, A., & Ruflin, M. (2013). Tatool: A Java-based open-source programming framework for psychological studies. Behavior Research Methods, 45, 108–115. doi:10.3758/s13428-012-0224-y
45. Vorberg, D., Mattler, U., Heinecke, A., Schmidt, T., & Schwarzbach, J. (2003). Different time courses for visual perception and action priming. Proceedings of the National Academy of Sciences, 100, 6275–6280. doi:10.1073/pnas.0931489100