Using Amazon Mechanical Turk for linguistic research

Psihologija

Volumn 43, Issue 4, 2010, Pages 441-464

  Schnoebelen, Tyler ^a   Kuperman, Victor ^b  

^a STANFORD UNIVERSITY   (United States)

^b MCMASTER UNIVERSITY   (Canada)

Author keywords

Amazon mechanical turk; Crowdsourcing; Predictability; Semantic similarity; Web experiments

Indexed keywords

EID: 78851469710     PISSN: 00485705     EISSN: 14519283     Source Type: Journal    
DOI: 10.2298/PSI1004441S     Document Type: Article

References (39)

1. Baldwin, T., & Villavicencio, A. (2002). Extracting the unextractable: a case study on verb-particles. In Proceedings of the 6th Conference on Natural Language Learning (Vol. 20, pp. 1-7). Association for Computational Linguistics.
2. Bannard, C. (2002). Statistical techniques for automatically inferring the semantics of verb-particle constructions. LinGO Working Paper No. 2002-06.
3. Birnbaum, M. (2004). Human research and data collection via the Internet. Annual Review of Psychology, 55(1), 803-832. doi:10.1146/annurev.psych.55.090902.141601
4. Bolinger, D. (1971). The phrasal verb in English. Cambridge, MA: Harvard University Press.
5. Boston, M. (n.d.). DepParse 2.5. Retrieved February 21, 2010, from http://conf.ling.cornell.edu/Marisa/papers.html
6. Boston, M., Hale, J., Kliegl, R., Patil, U., & Vasishth, S. (2008). Parsing costs as predictors of reading difficulty: An evaluation using the Potsdam Sentence Corpus. Journal of Eye Movement Research, 2(1), 1-12.
7. Boston, M., Hale, J., Kliegl, R., & Vasishth, S. (2008). Surprising parser actions and reading difficulty. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Short Papers (pp. 5-8).
8. Boston, M., Hale, J., Vasishth, S., & Kliegl, R. (in press). Parallel processing and sentence comprehension difficulty. Language and Cognitive Processes.
9. Buchanan, T. (1998). Internet research: Self-monitoring and judgements of attractiveness. Presented at the The 28th Annual Convention of the Society for Computers in Psychology, Dallas, TX.
10. Dambacher, M., Kliegl, R., Hofmann, M., & Jacobs, A. (2006). Frequency and predictability effects on event-related potentials during reading. Brain Research, 1084(1), 89-103. doi:10.1016/j.brainres.2006.02.010
11. Dandurand, F., Shultz, T., & Onishi, K. (2008). Comparing online and lab methods in a problem-solving experiment. Behavior Research Methods, 40(2), 428-434. doi:10.3758/BRM.40.2.428
12. Engbert, R., Nuthmann, A., Richter, E., & Kliegl, R. (2005). SWIFT: A dynamical model of saccade generation during reading. Psychological Review, 112(4), 777-813. doi:10.1037/0033-295X.112.4.777
13. Fraser, B. (1974). The phrasal verb in English by Dwight Bolinger. Language, 50(3), 568. doi:10.2307/412224
14. Gibson, E. (1998). Linguistic complexity: locality of syntactic dependencies. Cognition, 68(1), 1-76. doi:10.1016/S0010-0277(98)00034-1
15. Gries, S. (2003). Multifactorial analysis in corpus linguistics: A study of particle placement. Continuum Intl Pub Group.
16. Hale, J. (2001). A probabilistic Earley parser as a psycholinguistic model. In Proceedings of NAACL, 2, 159-166.
17. Ipeirotis, P. (2008, March 19). Mechanical Turk: The demographics. A Computer Scientist in a Business School. Retrieved February 21, 2010, from http://behind-the-enemy-lines.blogspot.com/2008/03/mechanical-turk-demographics.html
18. Kennedy, A. (1920). The Modern English verb-adverb combination. Language and Literature, 1(1), 1-51.
19. Kliegl, R., Grabner, E., Rolfs, M., & Engbert, R. (2004). Length, frequency, and predictability effects of words on eye movements in reading. European Journal of Cognitive Psychology, 16 (1-2), 262-284. doi:10.1080/09541440340000213
20. Kliegl, R., Nuthmann, A., & Engbert, R. (2006). Tracking the mind during reading: The influence of past, present, and future words on fixation durations. Journal of Experimental Psychology: General, 135(1), 12-35. doi:10.1037/0096-3445.135.1.12
21. Krantz, J., & Dalal, R. (2000). Validity of Web-based psychological research. Psychological experiments on the Internet, 35-60.
22. Landis, J., & Koch, G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159. doi:10.2307/2529310
23. Levy, R. (2008). Expectation-based syntactic comprehension. Cognition, 106(3), 1126-1177. doi:10.1016/j.cognition.2007.05.006
24. Lohse, B., Hawkins, J., & Wasow, T. (2004). Domain minimization in English verb-particle constructions. Language, 80(2), 238-261. doi:10.1353/lan.2004.0089
25. McCarthy, D., Keller, B., & Carroll, J. (2003). Detecting a continuum of compositionality in phrasal verbs. In Proceedings of the ACL 2003 workshop on Multiword expressions: analysis, acquisition and treatment-Volume 18 (p. 80).
26. McGraw, K., Tew, M., & Williams, J. (2000). The integrity of Web-delivered experiments: Can you trust the data? Psychological Science, 11 (6), 502-506.
27. Rayner, K., & Well, A. (1996). Effects of contextual constraint on eye movements in reading: A further examination. Psychonomic Bulletin & Review, 3(4), 504-509.
28. Reichle, E., Pollatsek, A., Fisher, D., & Rayner, K. (1998). Toward a model of eye movement control in reading. Psychological Review, 105, 125-157.
29. Reichle, E., Warren, T., & McConnell, K. (2009). Using E-Z Reader to model the effects of higher level language processing on eye movements during reading. Psychonomic Bulletin & Review, 16(1), 1-21. doi:10.3758/PBR.16.1.1
30. Reips, U. (2002a). Theory and techniques of conducting Web experiments. In B. Batinic, U. Reips, & M. Bosnjak (Eds.), Online Social Sciences (pp. 229-250). Seattle: Hogrefe & Huber.
31. Reips, U. (2002b). Standards for Internet-based experimenting. Experimental Psychology (formerly "Zeitschrift für Experimentelle Psychologie"), 49(4), 243-256. doi:10.1026//1618-3169.49.4.243
32. Resnik, P. (1999). Semantic similarity in a taxonomy: An information-based measure and its application to problems of ambiguity in natural language. Journal of Artificial Intelligence Research, 11, 95-130.
33. Roland, D., Dick, F., & Elman, J. (2007). Frequency of basic English grammatical structures: A corpus analysis. Journal of Memory and Language, 57(3), 348-379. doi:10.1016/j.jml.2007.03.002
34. Schilling, H., Rayner, K., & Chumbley, I. (1998). Comparing naming, lexical decision, and eye fixation times: Word frequency effects and individual differences. Memory and Cognition, 26, 1270-1281.
35. Senior, C., Phillips, M., Barnes, J., & David, A. (1999). An investigation in the perception of dominance from schematic faces: A study using the World-Wide Web. Behavior Research Methods, Instruments, & Computers, 31, 341-346.
36. Snow, R., O'Connor, B., Jurafsky, D., & Ng, A. (2008). Cheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (pp. 254-263). Honolulu, Hawaii: Association for Computational Linguistics.
37. Taylor, W. (1953). Cloze procedure: A new tool for measuring readability. Journalism Quarterly, 30, 415-433.
38. Tsuruoka, Y., & Tsujii, J. (2005). Bidirectional inference with the easiest-first strategy for tagging sequence data. In Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing (pp. 467-474). Vancouver, British Columbia, Canada: Association for Computational Linguistics.
39. Van Petten, C., & Kutas, M. (1990). Interactions between sentence context and word frequency in event-related brain potentials. Memory & Cognition, 18(4), 380-393.