Reaction time effects in lab- versus Web-based research: Experimental evidence

Behavior Research Methods

Volumn 48, Issue 4, 2016, Pages 1718-1724

  Hilbig, Benjamin E ^a,b  

^a Rheinland Pfälzische Technische Universität Kaiserslautern Landau   (Germany)

^b MAX PLANCK INSTITUTE FOR RESEARCH ON COLLECTIVE GOODS   (Germany)

Author keywords

Internet; Reaction time; Response latency; Web; Word frequency effect

Indexed keywords

BEHAVIORAL RESEARCH; HUMAN; INFORMATION PROCESSING; INTERNET; PHYSIOLOGY; PROCEDURES; REACTION TIME; RESEARCH; SOFTWARE; YOUNG ADULT;

BEHAVIORAL RESEARCH; DATA COLLECTION; HUMANS; INTERNET; REACTION TIME; RESEARCH; SOFTWARE; YOUNG ADULT;

EID: 84946434109     PISSN: 1554351X     EISSN: 15543528     Source Type: Journal    
DOI: 10.3758/s13428-015-0678-9     Document Type: Article

References (37)

1. Albrecht, T., & Vorberg, D. (2010). Long-lasting effects of briefly flashed words and pseudowords in ultrarapid serial visual presentation. Journal of Experimental Psychology: Learning, Memory, and Cognition, 36, 1339–1345. doi:10.1037/a0019999
2. Birnbaum, M. H. (2004). Human research and data collection via the Internet. Annual Review of Psychology, 55, 803–832. doi:10.1146/annurev.psych.55.090902.141601
3. Brand, A., & Bradley, M. T. (2012). Assessing the effects of technical variance on the statistical outcomes of web experiments measuring response times. Social Science Computer Review, 30, 350–357. doi:10.1177/0894439311415604
4. Chuah, S. C., Drasgow, F., & Roberts, B. W. (2006). Personality assessment: Does the medium matter? No. Journal of Research in Personality, 40, 359–376. doi:10.1016/j.jrp.2005.01.006
5. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
6. Corley, M., & Scheepers, C. (2002). Syntactic priming in English sentence production: Categorical and latency evidence from an Internet-based study. Psychonomic Bulletin & Review, 9, 126–131. doi:10.3758/bf03196267
7. Cronk, B. C., & West, J. L. (2002). Personality research on the Internet: A comparison of Web-based and traditional instruments in take-home and in-class settings. Behavior Research Methods, Instruments, & Computers, 34, 177–180.
8. Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013). Evaluating Amazon’s Mechanical Turk as a Tool for Experimental Behavioral Research. PLoS ONE, 8(e57410), 1–18. doi:10.1371/journal.pone.0057410
9. de Leeuw, J. R. (2015). jsPsych: A JavaScript library for creating behavioral experiments in a Web browser. Behavior Research Methods, 47, 1–12. doi:10.3758/s13428-014-0458-y
10. de Leeuw, J. R., & Motz, B. A. (2015). Psychophysics in a Web browser? Comparing response times collected with JavaScript and Psychophysics Toolbox in a visual search task. Behavior Research Methods. doi:10.3758/s13428-015-0567-2
11. Eichstaedt, J. (2001). An inaccurate-timing filter for reaction time measurement by JAVA applets implementing Internet-based experiments. Behavior Research Methods, Instruments, & Computers, 33, 179–186.
12. Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175–191. doi:10.3758/BF03193146
13. Germine, L., Nakayama, K., Duchaine, B. C., Chabris, C. F., Chatterjee, G., & Wilmer, J. B. (2012). Is the Web as good as the lab? Comparable performance from Web and lab in cognitive/perceptual experiments. Psychonomic Bulletin & Review, 19, 847–857. doi:10.3758/s13423-012-0296-9
14. Gordon, B. (1983). Lexical access and lexical decision: Mechanisms of frequency sensitivity. Journal of Verbal Learning and Verbal Behavior, 22, 24–44. doi:10.1016/S0022-5371(83)80004-8
15. Gosling, S. D., Vazire, S., Srivastava, S., & John, O. P. (2004). Should we trust web-based studies? A comparative analysis of six preconceptions about internet questionnaires. American Psychologist, 59, 93–104.
16. Hecht, H., Oesker, M., Kaiser, A., Civelek, H., & Stecker, T. (1999). A perception experiment with time-critical graphics animation on the World-Wide Web. Behavior Research Methods, Instruments, & Computers, 31, 439–445. doi:10.3758/bf03200724
17. Ihme, J. M., Lemke, F., Lieder, K., Martin, F., Muller, J. C., & Schmidt, S. (2009). Comparison of ability tests administered online and in the laboratory. Behavior Research Methods, 41, 1183–1189. doi:10.3758/BRM.41.4.1183
18. Keller, F., Gunasekharan, S., Mayo, N., & Corley, M. (2009). Timing accuracy of Web experiments: A case study using the WebExp software package. Behavior Research Methods, 41, 1–12. doi:10.3758/BRM.41.1.12
19. Kraut, R., Olson, J., Banaji, M., Bruckman, A., Cohen, J., & Couper, M. (2004). Psychological research online: Report of Board of Scientific Affairs’ Advisory Group on the Conduct of Research on the Internet. American Psychologist, 59, 105–117. doi:10.1037/0003-066x.59.2.105
20. Lang, F. R., John, D., Lütdke, O., Schupp, J., & Wagner, G. G. (2011). Short assessment of the Big Five: Robust across survey methods except telephone interviewing. Behavior Research Methods, 43, 548–567. doi:10.3758/s13428-011-0066-z
21. Linnman, C., Carlbring, P., Ahman, A., Andersson, H., & Andersson, G. (2006). The Stroop effect on the internet. Computers in Human Behavior, 22, 448–455. doi:10.1016/j.chb.2004.09.010
22. McGraw, K. O., Tew, M. D., & Williams, J. E. (2000). The integrity of Web-delivered experiments: Can you trust the data? Psychological Science, 11, 502–506. doi:10.1111/1467-9280.00296
23. Neath, I., Earle, A., Hallett, D., & Surprenant, A. M. (2011). Response time accuracy in Apple Macintosh computers. Behavior Research Methods, 43, 353–362. doi:10.3758/s13428-011-0069-9
24. Ollesch, H., Heineken, E., & Schulte, F. P. (2006). Physical or virtual presence of the experimenter: Psychological online-experiments in different settings. International Journal of Internet Science, 1, 71–81.
25. Plant, R. R., Hammond, N., & Whitehouse, T. (2003). How choice of mouse may affect response timing in psychological studies. Behavior Research Methods, Instruments, & Computers, 35, 276–284. doi:10.3758/bf03202553
26. Plant, R. R., & Turner, G. (2009). Millisecond precision psychological research in a world of commodity computers: New hardware, new problems? Behavior Research Methods, 41, 598–614. doi:10.3758/BRM.41.3.598
27. Ratcliff, R. (1979). Group reaction time distributions and an analysis of distribution statistics. Psychological Bulletin, 86, 446–461. doi:10.1037/0033-2909.86.3.446
28. Reimers, S., & Stewart, N. (2007). Adobe flash as a medium for online experimentation: A test of reaction time measurement capabilities. Behavior Research Methods, 39, 365–370. doi:10.3758/bf03193004
29. Reimers, S., & Stewart, N. (2015). Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments. Behavior Research Methods, 47, 309–327. doi:10.3758/s13428-014-0471-1
30. Reips, U.-D. (2002). Internet-based psychological experimenting: Five dos and five don’t@!s. Social Science Computer Review, 20, 241–249. doi:10.1177/08939302020003002
31. Reips, U.-D., & Birnbaum, M. H. (2011). Behavioral research and data collection via the internet. In R. W. Proctor & K.-P. L. Vu (Eds.), The handbook of human factors in Web design (2nd ed., pp. 563–585). Mahwah, NJ: Erlbaum.
32. Reips, U.-D., Buchanan, T., Krantz, J. H., & McGrawn, K. (in press). Methodological challenges in the use of the Internet for scientific research: Ten solutions and recommendations. Studia Psychologica. http://www.uni-konstanz.de/iscience/reips/pubs/papers/StudiaPsy_final.pdf
33. Reips, U.-D., & Krantz, J. H. (2010). Conducting true experiments on the Web. In S. D. Gosling & J. A. Johnson (Eds.), Advanced methods for conducting online behavioral research (pp. 193–216). Washington, DC: American Psychological Association.
34. Rubenstein, H., Garfield, L., & Millikan, J. A. (1970). Homographic entries in the internal lexicon. Journal of Verbal Learning and Verbal Behavior, 9, 487–494. doi:10.1016/S0022-5371(70)80091-3
35. Schneider, W., Eschman, A., & Zuccolotto, A. (2002). E-Prime: User’s guide. Pittsburgh, PA: Psychology Software Tools Inc.
36. Simcox, T., & Fiez, J. (2014). Collecting response times using Amazon Mechanical Turk and Adobe Flash. Behavior Research Methods, 46, 95–111. doi:10.3758/s13428-013-0345-y
37. Skitka, L. J., & Sargis, E. G. (2006). The Internet as psychological laboratory. Annual Review of Psychology, 57, 529–555. doi:10.1146/annurev.psych.57.102904.190048