Using mechanical turk to obtain and analyze English acceptability judgments

Linguistics and Language Compass

Volumn 5, Issue 8, 2011, Pages 509-524

  Gibson, Edward ^a   Piantadosi, Steve ^a   Fedorenko, Kristina ^b  

^a MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^b SMITH COLLEGE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 79960936614     PISSN: None     EISSN: 1749818X     Source Type: Journal    
DOI: 10.1111/j.1749-818X.2011.00295.x     Document Type: Article

References (41)

1. Adger, D. (2003). Core syntax: a minimalist approach. Oxford, UK: Oxford University Press.
2. Arnon, I., N. Snider, P. Hofmeister, T. F. Jaeger, and I. A. Sag. 2006. Cross-linguistic variation in a processing account: the case of multiple wh-questions. Proceedings of Berkeley Linguistics Society 32.
3. Baayen, R. H. 2008. Analyzing linguistic data: a practical introduction to statistics using R. Cambridge, UK: Cambridge University Press.
4. Bard, E. G., D. Robertson, and A. Sorace. 1996. Magnitude estimation of linguistic acceptability. Language 72. 107-50.
5. Bates, D., M. Maechler, and B. Dai. 2008. lme4: Linear mixed-effects models using S4 classes. R package version 0.999375-27. Retrieved on 4 April 2011 from: .
6. Bolinger, D. 1978. Asking more than one thing at a time. Questions, ed. by H. Hiz. Dordrecht: D. Reidel.
7. Clifton, C, Jr., G. Fanselow, and L. Frazier. 2006. Amnestying superiority violations: processing multiple questions. Linguistic Inquiry 37. 51-68.
8. Corley, M. M. B., and C. Scheepers. 2002. Syntactic priming in English sentence production: categorical and latency evidence from an internet-based study. Psychonomic Bulletin and Review 9. 126-31.
9. Cowart, W. 1997. Experimental syntax: applying objective methods to sentence judgments. Thousand Oaks, CA: Sage Publications.
10. Culicover, P. W., and R. Jackendoff. 2010. Quantitative methods alone are not enough: response to Gibson and Fedorenko. Trends in Cognitive Science 14. 234-5.
11. DeGroot, M. H., M. J. Schervish, X. Fang, L. Lu, and D. Li. 1986. Probability and statistics. Reading, MA: Addison-Wesley.
12. Edelman, S., and M. Christiansen. 2003. How seriously should we take Minimalist syntax? Trends in Cognitive Sciences 7. 60-1.
13. Featherston, S. 2005. Magnitude estimation and what it can do for your syntax: some wh-constraints in German. Lingua 115. 1525-50.
14. Featherston, S. 2007. Data in generative grammar: the stick and the carrot. Theoretical Linguistics 33. 269-318.
15. Fedorenko, E., and E. Gibson. 2010. Adding a third wh-element does not increase the acceptability of object-initial multiple-wh-questions. Syntax 13. 183-95.
16. Fedorenko, E., and E. Gibson. forthcoming. Syntactic parallelism as an account of superiority effects: empirical investigations in English and Russian.
17. Ferreira, F. 2005. Psycholinguistics, formal grammars, and cognitive science. The Linguistic Review 22. 365-80.
18. Gelman, A., and J. Hill. 2007. Data analysis using regression and multilevel/hierarchical models. Cambridge, UK: Cambridge University Press.
19. Gibson, E., and E. Fedorenko. 2010. Weak quantitative standards in linguistics research. Trends in Cognitive Science 14. 233-4.
20. Gibson, E., and E. Fedorenko. >forthcoming. The need for quantitative methods in syntax and semantics research. Language and Cognitive Processes.
21. Kayne, R. 1983. Connectedness. Linguistic Inquiry 14. 223-49.
22. Keller, F. 2000. Gradience in grammar: experimental and computational aspects of degrees of grammaticality. PhD Thesis, University of Edinburgh.
23. Keller, F., M. Corley, S. Corley, L. Konieczny, and A. Todirascu. 1998. WebExp: a Java toolbox for web-based psychological experiments (Technical Report No. HCRC/TR-99). Human Communication Research Centre, University of Edinburgh, Edinburgh, UK.
24. Keller, F., S. Gunasekharan, N. Mayo, and M. Corley. 2009. Timing accuracy of web experiments: a case study using the WebExp Software Package. Behavior Research Methods 41(1). 1-12.
25. Kuno, S., and J. Robinson. 1972. Multiple wh-questions. Linguistic Inquiry 3. 463-87.
26. Marantz, A. 2005. Generative linguistics within the cognitive neuroscience of language. The Linguistic Review 22. 429-45.
27. Myers, J. 2009a. Syntactic judgment experiments. Language and Linguistics Compass 3. 406-23.
28. Myers, J. 2009b. The design and analysis of small-scale syntactic judgment experiments. Lingua 119. 425-44.
29. Nickerson, R. S. 1998. Confirmation bias: a ubiquitous phenomenon in many guises. Review of General Psychology 2. 175-220.
30. Pesetsky, D. 2000. Phrasal movement and its kin. Cambridge, MA: MIT Press.
31. Piantadosi, S. T., J. B. Tenenbaum, and N. Goodman. forthcoming. Representation languages for complex compositional concepts. Department of Brain and Cognitive Sciences, MIT.
32. R Core Development Team. 2008. R: a language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. [Online]. Retrieved on 4 April 2011 from: .
33. Richards, N. 2001. Movement in language. Oxford: Oxford University Press.
34. Schütze, C. 1996. The empirical base of linguistics: grammaticality judgments and linguistic methodology. Chicago: University of Chicago Press.
35. Sprouse, J. 2011. A validation of Amazon Mechanical Turk for the collection of acceptability judgments in linguistic theory. Behavior Research Methods 43.
36. Sprouse, J., and D. Almeida. 2010. The 469 data points that form the empirical foundation of generative syntactic theory are at least 98% replicable using formal experiments. Irvine: University of California, ms.
37. Venables, W. N., and B. D. Ripley. 2000. S programming. New York: Springer Verlag.
38. Wason, P. C. 1960. On the failure to eliminate hypotheses in a conceptual task. Quarterly Journal of Experimental Psychology 12. 129-40.
39. Wasow, T., and J. Arnold. 2005. Intuitions in linguistic argumentation. Lingua 115. 1481-96.
40. Weskott, T., and G. Fanselow. 2009. Scaling issues in the measurement of linguistic acceptability. The fruits of empirical linguistics. Vol. 1: process, ed. by S. Featherston and S. Winkler, 229-45. Berlin: Mouton de Gruyter.
41. Weskott, T., and G. Fanselow. 2011. On the informativity of different measures of linguistic acceptability. Language 87. 249-73.