Collecting psycholinguistic response time data using Amazon Mechanical Turk

PLoS ONE

Volumn 10, Issue 3, 2015, Pages

  Enochson, Kelly ^a   Culbertson, Jennifer ^a,b  

^a GEORGE MASON UNIVERSITY   (United States)

^b UNIVERSITY OF EDINBURGH   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

ACCURACY; ARTICLE; CONTROLLED STUDY; CROWDSOURCING; DECISION MAKING; FORCED CHOICE METHOD; GRAMMAR; HUMAN; HUMAN EXPERIMENT; LINGUISTICS; MENTAL TASK; READING; RECALL; RESPONSE TIME; TASK PERFORMANCE; ADULT; INTERNET; PROCEDURES; REACTION TIME; YOUNG ADULT;

ADULT; CROWDSOURCING; HUMANS; INTERNET; PSYCHOLINGUISTICS; REACTION TIME; YOUNG ADULT;

EID: 84926305284     PISSN: None     EISSN: 19326203     Source Type: Journal    
DOI: 10.1371/journal.pone.0116946     Document Type: Article

References (41)

1. Gibson E, Piantadosi S, Fedorenko K (2011) Using Mechanical Turk to obtain and analyze English acceptability judgments. Lang Linguist Compass 5: 509-524.
2. Sprouse J (2011) A validation of Amazon Mechanical Turk for the collection of acceptability judgments in linguistic theory. Behav Res Methods 43: 155-167. doi: 10.3758/s13428-010-0039-7 PMID: 21287108
3. Schnoebelen T, Kuperman V (2010) Using Amazon mechanical turk for linguistic research. Psihologija 43: 441-464.
4. Fine AB, Jaeger TF, Farmer TA, Qian T (2013) Rapid Expectation Adaptation during Syntactic Comprehension. PLoS ONE 8: e77661. doi: 10.1371/journal.pone.0077661 PMID: 24204909
5. Munro R, Bethard S, Kuperman V, Lai VT, Melnick R, et al. (2010) Crowdsourcing and language studies: the new generation of linguistic data. Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk. Association for Computational Linguistics. pp. 122-130. Available: http://dl.acm.org/citation.cfm?id=1866715. Accessed 13 August 2014.
6. Culbertson J, Adger D (2014) Language learners privilege structured meaning over surface frequency. Proc Natl Acad Sci 111: 5842-5847. doi: 10.1073/pnas.1320525111 PMID: 24706789
7. Becker M, Nevins A, Levine J (2012) Asymmetries in generalizing alternations to and from initial syllables. Language 88: 231-268.
8. Henrich J, Heine SJ, Norenzayan A (2010) The weirdest people in the world? Behav Brain Sci 33: 61-83. doi: 10.1017/S0140525X0999152X PMID: 20550733
9. Crump MJC, McDonnell JV, Gureckis TM (2013) Evaluating Amazon's Mechanical Turk as a Tool for Experimental Behavioral Research. PLoS ONE 8: e57410. doi: 10.1371/journal.pone.0057410 PMID: 23516406
10. Keller F, Gunasekharan S, Mayo N, Corley M (2009) Timing accuracy of web experiments: A case study using the WebExp software package. Behav Res Methods 41: 1-12. doi: 10.3758/BRM.41.1.12 PMID: 19182118
11. Fine AB, Florian Jaeger T (2013) Evidence for Implicit Learning in Syntactic Comprehension. Cogn Sci 37: 578-591. doi: 10.1111/cogs.12022 PMID: 23363004
12. Demberg V (2013) Integration Costs on Auxiliaries? - a self-paced reading study using WebExp. Available: http://csjarchive.cogsci.rpi.edu/Proceedings/2013/papers/0396/paper0396.pdf. Accessed 14 October 2014.
13. Drummond A (2013) Ibex Farm. Available: http://spellout.net/ibexfarm/. Accessed 2014 October 14.
14. Schmidt S (2009) Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Rev Gen Psychol 13: 90.
15. Klein RA, Ratliff KA, Vianello M, Adams RB Jr, Bahník Š, et al. (2014) Investigating Variation in Replicability. Soc Psychol 45: 142-152.
16. Schubert TW, Murteira C, Collins EC, Lopes D (2013) ScriptingRT: A Software Library for Collecting Response Latencies in Online Studies of Cognition. PLoS ONE 8: e67769. doi: 10.1371/journal.pone.0067769 PMID: 23805326
17. Ariel M (1990) Accessing noun-phrase antecedents. Croom Helm Lond.
18. Gundel JK, Hedberg N, Zacharski R (1993) Cognitive status and the form of referring expressions in discourse. language: 274-307.
19. Gordon PC, Grosz BJ, Gilliom LA (1993) Pronouns, names, and the centering of attention in discourse. Cogn Sci 17: 311-347.
20. Warren T, Gibson E (2002) The influence of referential processing on sentence complexity. Cognition 85: 79-112. PMID: 12086714
21. Hofmeister P, Sag IA (2010) Cognitive Constraints and Island Effects. Language 86: 366-415. PMID: 22661792
22. Hawkins JA (1999) Processing complexity and filler-gap dependencies across grammars. Language: 244-285.
23. Wanner E, Marastos M (1978) An ATN approach to comprehension. Linguist Theory Psychol Real. Available: http://ci.nii.ac.jp/naid/10008829763/. Accessed 2014 May 8.
24. Crain S, Fodor JD (1985) How can grammars help parsers? Natural language parsing: Psychological, computational, and theoretical perspectives. pp. 94-128.
25. Stowe LA (1986) Parsing WH-constructions: Evidence for on-line gap location. Lang Cogn Process 1: 227-245.
26. Kimball J, Aissen J (1971) I think, you think, he think. Linguist Inq 2: 241-246.
27. Bock K, Miller CA (1991) Broken agreement. Cognit Psychol 23: 45-93. doi: 10.1016/0010-0285(91) 90003-7 PMID: 2001615
28. Pearlmutter NJ, Garnsey SM, Bock K (1999) Agreement processes in sentence comprehension. J Mem Lang 41: 427-456.
29. Wagers MW, Lau EF, Phillips C (2009) Agreement attraction in comprehension: Representations and processes. J Mem Lang 61: 206-237.
30. Vigliocco G, Nicol J (1998) Separating hierarchical relations and word order in language production: Is proximity concord syntactic or linear? Cognition 68: B13-B29. PMID: 9775519
31. BadeckerW(in prep) Agreement in Wh-Questions: Subject-oriented Retrieval Mechanisms in a Work-ing- Memory Retrieval Model of Sentence Production.
32. King J, Just MA (1991) Individual differences in syntactic processing: The role of working memory. J Mem Lang 30: 580-602. doi: 10.1016/0749-596X(91)90027-H
33. Rohde D (2003) Linger: a flexible platform for language processing experiments, version 2.94.
34. Ferreira F, Clifton C (1986) The independence of syntactic processing. J Mem Lang 25: 348-368.
35. Baayen RH, Milin P (2010) Analyzing reaction times. Int J Psychol Res 3: 12-28.
36. Barr DJ, Levy R, Scheepers C, Tily HJ (2013) Random effects structure for confirmatory hypothesis testing: Keep it maximal. J Mem Lang 68: 255-278. doi: 10.1016/j.jml.2012.11.001
37. R Core Team (2014) R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Available: http://www.R-project.org/. Accessed 1 March 2014.
38. Bates D, Maechler M, Bolker B (2012) lme4: Linear mixed-effects models using S4 classes. Available: http://www.citeulike.org/group/17044/article/12173300. Accessed 10 September 2014.
39. MacWhinney B, Bates E, Kliegl R (1984) Cue validity and sentence interpretation in English, German, and Italian. J Verbal Learn Verbal Behav 23: 127-150.
40. Solomon ES, Pearlmutter NJ (2004) Semantic integration and syntactic planning in language production. Cognit Psychol 49: 1-46. PMID: 15193971
41. Germine L, Nakayama K, Duchaine BC, Chabris CF, Chatterjee G, et al. (2012) Is the Web as good as the lab? Comparable performance from Web and lab in cognitive/perceptual experiments. Psychon Bull Rev 19: 847-857. PMID: 22829343