A validation of Amazon Mechanical Turk for the collection of acceptability judgments in linguistic theory

Behavior Research Methods

Volumn 43, Issue 1, 2011, Pages 155-167

  Sprouse, Jon ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Acceptability judgments; Amazon Mechanical Turk; Experimental syntax; Grammaticality judgments; Linguistic theory

Indexed keywords

ADULT; ARTICLE; COMPUTER PROGRAM; COST; DECISION MAKING; FEMALE; HUMAN; ILLUSION; INFORMATION PROCESSING; INTERNET; LANGUAGE; LINGUISTICS; MALE; METHODOLOGY; PHYSIOLOGY; PSYCHOLOGICAL ASPECT; PSYCHOMOTOR PERFORMANCE; REPRODUCIBILITY; STATISTICAL ANALYSIS; STATISTICS; UNITED KINGDOM;

COSTS AND COST ANALYSIS; DATA COLLECTION; DATA INTERPRETATION, STATISTICAL; ENGLAND; FEMALE; HUMANS; ILLUSIONS; INTERNET; JUDGMENT; LANGUAGE; LINGUISTICS; MALE; PSYCHOLINGUISTICS; PSYCHOMOTOR PERFORMANCE; REPRODUCIBILITY OF RESULTS; SOFTWARE; YOUNG ADULT;

EID: 79953709996     PISSN: 1554351X     EISSN: 15543528     Source Type: Journal    
DOI: 10.3758/s13428-010-0039-7     Document Type: Article

References (28)

1. Bard, E. G., Robertson, D., & Sorace, A. (1996). Magnitude estimation of linguistic acceptability. Language, 72, 32-68.
2. Chomsky, N. (1965). Aspects of the theory of syntax. Cambridge: MIT Press.
3. Chomsky, N. (1986). Barriers. Cambridge: MIT Press.
4. Cowart, W. (1997). Experimental syntax: Applying objective methods to sentence judgments. Thousand Oaks: Sage.
5. Featherston, S. (2005a). Magnitude estimation and what it can do for your syntax: Some wh-constraints in German. Lingua, 115, 1525-1550.
6. Featherston, S. (2005b). Universals and grammaticality: Wh-constraints in German and English. Linguistics, 43, 667-711.
7. Frazier, L. (1985). Syntactic complexity. In D. Dowty, L. Karttunen, & A. Zwicky (Eds.), Natural language processing: Psychological, computational and theoretical perspectives (pp. 129-189). Cambridge: Cambridge University Press.
8. Gibson, E., & Fedorenko, E. (in press). The need for quantitative methods in syntax. Language and Cognitive Processes.
9. Gibson, E., & Thomas, J. (1999). Memory limitations and structural forgetting: The perception of complex ungrammatical sentences as grammatical. Language and Cognitive Processes, 14, 225-248.
10. Grimshaw, J. (1986). Subjacency and the S/S' parameter. Linguistic Inquiry, 17, 364-369.
11. Hofmeister, P., & Sag, I. (2010). Cognitive constraints and island effects. Language, 86, 366-415.
12. Huang, C.-T. (1982). Move WH in a language without WH movement. Linguistic Review, 1, 369-416.
13. Ipeirotis, P. G. (2010). Demographics of Mechanical Turk. Center for Digital Economy Research Working Papers, 10. Available at http://hdl. handle. net/2451/29585.
14. Keller, F. (2000). Gradience in grammar: Experimental and computational aspects of degrees of grammaticality. University of Edinburgh: Unpublished doctoral dissertation.
15. Keller, F., Gunasekharan, S., Mayo, N., & Corley, M. (2009). Timing accuracy of Web experiments: A case study using the WebExp software package. Behavior Research Methods, 41, 1-12.
16. Kendall, M. (1938). A new measure of rank correlation. Biometrika, 30, 81-89.
17. Kuno, S. (1973). Constraints on internal clauses and sentential subjects. Linguistic Inquiry, 4, 363-385.
18. Marantz, A. (2005). Generative linguistics within the cognitive neuroscience of language. Linguistic Review, 22, 429-445.
19. Myers, J. (2009). The design and analysis of small-scale syntactic judgment experiments. Lingua, 119, 425-444.
20. Phillips, C., Wagers, M., & Lau, E. (in press). Grammatical illusions and selective fallibility in real-time language comprehension. Language and Linguistics Compass.
21. R Development Core Team. (2009). R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing. Available at www. R-project. org.
22. Ross, J. R. (1967). Constraints on variables in syntax. Unpublished doctoral dissertation, MIT, Cambridge, MA.
23. Schütze, C. (1996). The empirical base of linguistics: Grammaticality judgments and linguistic methodology. Chicago: University of Chicago Press.
24. Sorace, A., & Keller, F. (2004). Gradience in linguistic data. Lingua, 115, 1497-1524.
25. Sprouse, J. (2009). Revisiting satiation: Evidence for an equalization response strategy. Linguistic Inquiry, 40, 329-341.
26. Sprouse, J., & Almeida, D. (2010). A quantitative defense of linguistic methodology. Manuscript submitted for publication.
27. Sprouse, J., Wagers, M., & Phillips, C. (2010). A test of the relation between working memory capacity and island effects. Manuscript submitted for publication.
28. Wagers, M., Lau, E., & Phillips, C. (2009). Agreement attraction in comprehension: Representations and processes. Journal of Memory and Language, 61, 206-237.