Collecting response times using Amazon Mechanical Turk and Adobe Flash

Behavior Research Methods

Volumn 46, Issue 1, 2014, Pages 95-111

  Simcox, Travis ^a   Fiez, Julie A ^a  

^a UNIVERSITY OF PITTSBURGH SCHOOL OF MEDICINE   (United States)

Author keywords

ActionScript; Adobe flash; Amazon Mechanical Turk; Crowdsourcing; Response times; Rich media; Stimulus presentation; Timing; Web experiment

Indexed keywords

ADOLESCENT; ADULT; AGED; ANALYSIS OF VARIANCE; ARTICLE; BEHAVIORAL SCIENCE; CLINICAL TRIAL; COMPUTER INTERFACE; COMPUTER PROGRAM; CROWDSOURCING; DECISION MAKING; FEMALE; HUMAN; INFORMATION PROCESSING; INTERNET; MALE; METHODOLOGY; MIDDLE AGED; NORMAL HUMAN; PHYSIOLOGY; QUESTIONNAIRE; REACTION TIME; YOUNG ADULT;

ADOLESCENT; ADULT; AGED; ANALYSIS OF VARIANCE; BEHAVIORAL SCIENCES; CROWDSOURCING; DATA COLLECTION; DATA DISPLAY; DECISION MAKING; FEMALE; HEALTHY VOLUNTEERS; HUMANS; INTERNET; MALE; MIDDLE AGED; QUESTIONNAIRES; REACTION TIME; RESEARCH DESIGN; SOFTWARE; USER-COMPUTER INTERFACE; YOUNG ADULT;

EID: 84894644541     PISSN: 1554351X     EISSN: 15543528     Source Type: Journal    
DOI: 10.3758/s13428-013-0345-y     Document Type: Article

References (38)

1. Adobe Developer Connection. (2012). Flash Platform Developer Center. http://www. adobe. com/devnet/flashplatform. html Retrieved 2012-08-11.
2. AMT FAQ. (2012). https://requester. mturk. com/help/faq#examples_violations Retrieved 2012-08-11.
3. Behrend, T. S., Sharek, D. J., Meade, A. W., & Wiebe, E. N. (2011). The viability of crowdsourcing for survey research. Behavior Research Methods, 43, 800-813.
4. Berinsky, A. J., Huber, & Lenz, G. S. (2012). Evaluating online labor markets for experimental research: Amazon. com's Mechanical Turk. Political Analysis, 20, 351-368.
5. Brand, A., & Bradley, M. T. (2012). Assessing the effects of technical variance on the statistical outcomes of web experiments measuring response times. Social Science Computer Review, 30, 350-357.
6. Buhrmester, M. D., Kwang, T., & Gosling, S. D. (2011). Amazon's Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6, 3-5.
7. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, New Jersey: Lawrence Erlbaum.
8. Faul, F., Erdfelder, E., Lang, A. G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175-191.
9. Fowler, R. L. (1985). Point estimates and confidence intervals in measures of association. Quantitative Methods in Psychology, 98, 160-165.
10. Goodman, J. K., Cryder, C. E., & Cheema, A. (in press). Data collection in a flat world: Strengths and weaknesses of mechanical Turk samples. Journal of Behavioral Decision Making.
11. Grossman, G. & Huang, E. (2009). ActionScript 3. 0 Overview. Adobe systems incorporated. Retrieved 2012-08-11 from http://www. adobe. com/devnet/actionscript/articles/actionscript3_overview. html.
12. Halberda, J., Ly, R., Wilmer, J., Naiman, D., & Germine, L. (2012). Number sense across the lifespan as revealed by a massive internet-based sample. Proceedings of the National Academy of Sciences, 109, 11116-11120.
13. Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? Behavior and Brain Sciences. Retrieved from: http://www2. psych. ubc. ca/~henrich/pdfs/WeirdPeople. pdf.
14. Hewson, C. M., Laurent, D., & Vogel, C. M. (1996). Proper methodologies for psychological and sociological studies conducted via the Internet. Behavior Research Methods, Instruments, & Computers, 28, 186-191.
15. Houben, K., & Wiers, R. W. (2008). Measuring implicit alcohol associations via the Internet: Validation of Web-based implicit association tests. Behavior Research Methods, 40, 1134-1143.
16. Ipeirotis, P. (2010). Demographics of Mechanical Turk. CeDER Working Papers, CeDER-10-01, New York University, Stern School of Business. Retrieved Aug 2012 from: http://hdl. handle. net/2451/29585.
17. Lupker, S. J., Perea, M., & Davis, C. M. (2008). Transposed-letter effects: Consonants, vowels and letter frequency. Language & Cognitive Processes, 23, 93-116.
18. Mason, W., & Suri, S. (2012). Conducting behavioral research on Amazon's Mechanical Turk. Behavior Research Methods, 44, 1-23.
19. Mayo, C., Aubanel, V., & Cooke, M. (2012). Effect of prosodic changes on speech intelligibility. In Proc. Interspeech, Portland, OR, USA.
20. McDonnell, J., Domingo, D., & Gureckis, T. (2012). Is Mechanical Turk the future of cognitive science research? Retrieved Aug 2012 from http://gureckislab. org/blog/?p=1297.
21. Meyer, D. E., Osman, A. M., Irwin, D. E., & Yantis, S. (1988). Modern mental chronometry. Biological Psychology, 26, 3-67.
22. Meyer, D. E., & Roger, W. S. (1971). Facilitation in recognizing pairs of words: Evidence of a dependence between retrieval operations. Journal of Experimental Psychology, 90, 227-234.
23. Neath, I., Earle, A., Hallett, D., & Surprenant, A. M. (2011). Response time accuracy in Apple Macintosh computers. Behavior Research Methods, 43, 353-362.
24. Nieuwenhuis, S., Stins, J. F., Posthuma, D., Polderman, T. J., Boomsma, D. I., & de Geus, E. J. (2006). Accounting for sequential trial effects in the flanker task: Conflict adaptation or associative priming? Memory & Cognition, 34, 1260-1272.
25. Owen, A. M., Hampshire, A., Grahn, J. A., Stenton, R., Dajani, S., Burns, A. S., et al. (2010). Putting brain training to the test. Nature, 465, 775-778.
26. Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411-419.
27. Paré, D. E., & Cree, G. S. (2009). Web-based image norming: How do object familiarity and visual complexity ratings compare when collected in-lab versus online? Behavior Research Methods, 41, 699-704.
28. Plant, R., & Turner, G. (2009). Millisecond precision psychological research in a world of commodity computers: New hardware, new problems? Behavior Research Methods, 41, 598-614.
29. R Developmental Core Team. (2011). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0, URL http://www. R-project. org/.
30. Rand, D. G. (2012). The promise of Mechanical Turk: How online labor markets can help theorists run behavioral experiments. Journal of Theoretical Biology, 222, 172-179.
31. Reimers, S., & Stewart, N. (2007). Adobe Flash as a medium for online experimentation: A test of reaction time measurement capabilities. Behavior Research Methods, 39, 365-370.
32. Schmidt, W. C. (2001). Presentation accuracy of Web animation methods. Behavior Research Methods, Instruments, & Computers, 33, 187-200.
33. Simcox, T. (2012). [Compilation of user agent strings from Amazon Mechanical Turk workers]. Unpublished raw data.
34. Smithson, M. (2001). Correct confidence intervals for various regression effect sizes and parameters: The importance of noncentral distributions in computing intervals. Educational and Psychological Measurement, 61, 605-632.
35. The Mechanical Turk Blog. (2012, August 28). Improving quality with qualifications - tips for API requesters [Web log post]. Retrieved from: http://mechanicalturk. typepad. com/blog/2012/08/requesters-consistently-tell-us-that-using-qualifications-is-one-of-the-most-effective-strategies-for-optimizing-the-quality. html.
36. Ulrich, R., & Giray, M. (1989). The resolution of clocks: Effects on reaction time measurement-Good news for bad clocks. British Journal of Mathematical and Statistical Psychology, 42, 1-12.
37. Woltman, G. (2012). Prime95. Retrieved from: http://www. mersenne. org/freesoft/.
38. Yu C.H. (2003). Resampling methods: Concepts, applications, and justification. Practical Assessment, Research & Evaluation.