Assessing the Effects of Technical Variance on the Statistical Outcomes of Web Experiments Measuring Response Times

Social Science Computer Review

Volumn 30, Issue 3, 2012, Pages 350-357

  Brand, Andrew ^a   Bradley, Michael T ^b  

^a KING'S COLLEGE LONDON   (United Kingdom)

^b UNIVERSITY OF NEW BRUNSWICK   (Canada)

Author keywords

effect sizes; response times; simulation; statistical power; technical variance; web experiments

Indexed keywords

EID: 84863422432     PISSN: 08944393     EISSN: 15528286     Source Type: Journal    
DOI: 10.1177/0894439311415604     Document Type: Article

References (31)

1. Birnbaum M.H.Introduction to behavioral research on the Internet. Upper Saddle River, NJ: Prentice Hall; 2001:.
2. Birnbaum M.H.Human research and data collection via the Internet.Annual Review of Psychology. 2004;55:803-832.
3. Box G.E. P.,Cox D.R.An analysis of transformations (with discussion).Journal of the Royal Statistical Society B. 1964;26:211-246.
4. Brand A.A web experiment based enquiry into the verbal overshadowing effect (PhD thesis). Cardiff: Cardiff University; 2004:.
5. Brand A.,Bradley M.T.,Best L.A.,Stocia G.Multiple trials may yield exaggerated effect size estimates.The Journal of General Psychology. 2011;138:1-11.
6. Cohen J.Statistical power analysis for the behavioral sciences. New York: Academic Press; 1977:.
7. Corley M.,Scheepers C.Syntactic priming in English sentence production: Categorical and latency evidence from an Internet-based study.Psychonomic Bulletin and Review. 2002;9:126-131.
8. Damian M.F.Does variability in human performance outweigh imprecision in response devices such as computer keyboards?.Behavior Research Methods. 2010;42:205-211.
9. Gosling S.D.,Vazire S.,Srivastava S.,John O.P.Should we trust web-based studies? A comparative analysis of six preconceptions about Internet questionnaires.American Psychologist. 2004;59:93-104.
10. Heathcote A.,Popiel S.J.,Mewhort D.J. K.Analysis of response time distributions: An example using the Stroop task.Psychological Bulletin. 1991;109:340-347.
11. Hecht H.,Oesker M.,Kaiser A.,Civelek H.,Stecker T.A perception experiment with time-critical graphics animation on the World-Wide Web.Behavior Research Methods, Instruments and Computers. 1999;31:439-445.
12. Hewson C.Conducting research on the Internet.The Psychologist. 2003;16:290-293.
13. Hohle R.H.Inferred components of reaction times as functions of foreperiod duration.Journal of Experimental Psychology. 1965;69:382-386.
14. Keller F.,Gunasekharan S.,Mayo N.,Corley M.Timing accuracy of web experiments: A case study using the WebExp software package.Behavior Research Methods. 2009;41:1-12.
15. Kelley K.Methods for the behavioral, educational, and social sciences: An R package.Behavior Research Methods. 2007;39:979-984.
16. Luce R.D.Response times: Their role in inferring elementary mental organization. New York, NY: Oxford University Press; 1986:.
17. McGill W.J.Handbook of mathematical psychology. Luce R.D.Bush R.R., ed. New York, NY: Wiley; 1963:309-360.
18. McGraw K.O.,Tew M.D.,Williams J.E.The integrity of web-delivered experiments: Can you trust the data?.Psychological Science. 2000;11:502-506.
19. Musch J.,Reips U.D.Psychological experiments on the Internet. Birnbaum M.H., ed. San Diego, CA: Academic Press; 2000:61-87.
20. Nosek B.A.,Banaji M.R.,Greenwald A.G.Harvesting implicit group attitudes and beliefs from a demonstration website.Group Dynamics. 2002;6:101-115.
21. Plant R.R.,Hammond N.,Whitehouse T.How choice of mouse may affect response timing in psychological studies.Behavior Research Methods. 2003;35:276-284.
22. Plant R.R.,Turner G.Millisecond precision psychological research in a word of commodity computers: New hardware, new problems?.Behavior Research Methods. 2009;41:598-614.
23. R Development Core Team. (2009). R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing. Retrieved fromhttp://www.r-project.org.
24. Reimers S.,Maylor E.A.Task switching across the life span: Effects of age on general and specific switch costs.Developmental Psychology. 2005;41:661-671.
25. Reimers S.,Stewart N.Adobe Flash as a medium for online experimentation: A test of RT measurement capabilities.Behavior Research Methods. 2007;39:365-370.
26. Reips U.D.Psychological experiments on the Internet. Birnbaum M.H., ed. SanDiego, CA: Academic Press; 2000:89-117.
27. Reips U.D.Standards for Internet-based experimenting.Experimental Psychology. 2002;49:243-256.
28. Stasinopoulos D.M.,Rigby R.A.Generalized additive models for location scale and shape (GAMLSS) in R.Journal of Statistical Software. 2007;23:1-46.
29. TewM. D.McGrawK. O. (2002). The accuracy of response timing by Authorware programs. Unpublished manuscript. Retrieved fromhttp://www.psych.uni.edu/psychexps/Scrapbook/Timing_2002.pdf.
30. Ulrich R.,Giray M.Time resolution of clocks: Effects on reaction time measurement: Good news for bad clocks.British Journal of Mathematical & Statistical Psychology. 1989;42:1-12.
31. Weinberger J.,Westen D.RATS, we should have used Clinton: Subliminal priming in political campaigns.Political Psychology. 2008;29:631-651.