Online crowdsourcing for efficient rating of speech: A validation study

Journal of Communication Disorders

Volumn 53, Issue , 2015, Pages 70-83

  McAllister Byun, Tara ^a   Halpin, Peter F ^a   Szeredi, Daniel ^a  

^a NEW YORK UNIVERSITY   (United States)

Author keywords

Crowdsourcing; Research methods; Speech perception; Speech rating; Speech sound disorders

Indexed keywords

ADOLESCENT; ADULT; AMAZON MECHANICAL TURK; ARTICLE; BOOTSTRAPPING; CHILD; CONTROLLED STUDY; CORRELATION ANALYSIS; CROWDSOURCING; FEMALE; HUMAN; MALE; SPEECH ANALYSIS; SPEECH ARTICULATION; SPEECH RATE; TASK PERFORMANCE; VALIDATION STUDY; PROCEDURES; SPEECH; SPEECH DISORDERS; SPEECH PERCEPTION;

ADOLESCENT; ADULT; CHILD; CROWDSOURCING; FEMALE; HUMANS; MALE; SPEECH; SPEECH DISORDERS; SPEECH PERCEPTION;

EID: 84923095708     PISSN: 00219924     EISSN: 18737994     Source Type: Journal    
DOI: 10.1016/j.jcomdis.2014.11.003     Document Type: Article

References (37)

1. American Speech-Language-Hearing Association 2012 schools survey report: SLP workforce/work conditions 2012, http://www.asha.org/research/memberdata/schoolssurvey/.
2. Baylor C.R., Hula W., Donovan N.J., Doyle P., Kendall D., Yorkston K.M. An introduction to item response theory and Rasch models for speech-language pathologists. American Journal of Speech-Language Pathology 2011, 20:243-259.
3. Becker M., Levine J. Experigen - An online experiment platform 2010, Available from. https://github.com/tlozoot/experigen.
4. Becker M., Nevins A., Levine J. Asymmetries in generalizing alternations to and from initial syllables. Language 2012, 88(2):231-268.
5. Berinsky A.J., Huber G.A., Lenz G.S. Evaluating online labor markets for experimental research: Amazon.com's Mechanical Turk. Political Analysis 2012, 20(3):351-368.
6. Buhrmester M., Kwang T., Gosling S.D. Amazon's Mechanical Turk: A new source of inexpensive, yet high-quality, data?. Perspectives on Psychological Science 2011, 6(1):3-5.
7. Crump M.J., McDonnell J.V., Gureckis T.M. Evaluating Amazon's Mechanical Turk as a tool for experimental behavioral research. PLOS ONE 2013, 8(3):e57410.
8. Culbertson J., Adger D. Language learners privilege structured meaning over surface frequency. Proceedings of the National Academy of Sciences of the United States of America 2014, 10.1073/pnas.1320525111.
9. Delattre P., Freeman D.C. A dialect study of American r's by X-ray motion picture. Linguistics 1968, 6(44):29-68.
10. Efron B. Bootstrap methods: Another look at the jackknife. Annals of Statistics 1979, 7:1-26.
11. Flipsen P., Shriberg L.D., Weismer G., Karlsson H.B., McSweeny J.L. Acoustic phenotypes for speech-genetics studies: Reference data for residual /ε{right hook reversed}/ distortions. Clinical Linguistics & Phonetics 2001, 15(8):603-630.
12. Gibson E., Piantadosi S., Fedorenko K. Using Mechanical Turk to obtain and analyze English acceptability judgments. Language and Linguistics Compass 2011, 5(8):509-524.
13. Goodman J.K., Cryder C.E., Cheema A. Data collection in a flat world: The strengths and weaknesses of Mechanical Turk samples. Journal of Behavioral Decision Making 2013, 26(3):213-224.
14. Horton J.J., Rand D.G., Zeckhauser R.J. The online laboratory: Conducting experiments in a real labor market. Experimental Economics 2011, 14(3):399-425.
15. Ipeirotis P.G. Analyzing the Amazon Mechanical Turk marketplace. XRDS: Crossroads, The ACM Magazine for Students 2010, 17(2):16-21.
16. Ipeirotis P.G. Demographics of Mechanical Turk. Center for Digital Economy Research Working Papers 2010, 10. Available from. http://hdl.handle.net/2451/29585.
17. Ipeirotis P.G., Provost F., Sheng V.S., Wang J. Repeated labeling using multiple noisy labelers. Data Mining and Knowledge Discovery 2014, 28(2):402-441.
18. Khatib F., Cooper S., Tyka M.D., Xu K., Makedon I., Popović Z., et al. Algorithm discovery by protein folding game players. Proceedings of the National Academy of Sciences of the United States of America 2011, 108(47):18949-18953.
19. Klein H.B., Grigos M.I., McAllister Byun T., Davidson L. The relationship between inexperienced listeners' perceptions and acoustic correlates of children's /r/ productions. Clinical Linguistics and Phonetics 2012, 26(7):628-645.
20. Kleinschmidt D.F., Jaeger T.F. A continuum of phonetic adaptation: Evaluating an incremental belief-updating model of recalibration and selective adaptation 2012, Paper presented at the 34th annual conference of the cognitive science society, Sapporo, Japan.
21. Lee S., Potamianos A., Narayanan S. Acoustics of children's speech: Developmental changes of temporal and spectral parameters. Journal of the Acoustical Society of America 1999, 105(3):1455-1468.
22. Maas E., Farinella K.A. Random versus blocked practice in treatment for childhood apraxia of speech. Journal of Speech, Language and Hearing Research 2012, 55(2):561-578.
23. Mason W., Suri S. Conducting behavioral research on Amazon's Mechanical Turk. Behavior Research Methods 2012, 44(1):1-23.
24. McAllister Byun T., Hitchcock E.R. Investigating the use of traditional and spectral biofeedback approaches to intervention for /r/ misarticulation. American Journal of Speech-Language Pathology 2012, 21(3):207-221.
25. Munson B., Schellinger S.K., Urberg-Carlson K. Measuring speech-sound learning using visual analog scaling. Perspectives in Language Learning and Education 2012, 19:19-30.
26. Paolacci G., Chandler J., Ipeirotis P.G. Running experiments on Amazon Mechanical Turk. Judgment and Decision Making 2010, 5:411-419.
27. Payesteh B., Munson B. Clinical feasibility of visual analog scaling 2013, Poster presented at the 2013 Convention of the American Speech-Language Hearing Association, Chicago, IL.
28. Schnoebelen T., Kuperman V. Using Amazon Mechanical Turk for linguistic research. Psihologija 2010, 43(4):441-464.
29. Schubert T.W., Murteira C., Collins E.C., Lopes D. ScriptingRT: A software library for collecting response latencies in online studies of cognition. PLOS ONE 2013, 8(6):e67769.
30. Shapiro D.N., Chandler J., Mueller P.A. Using Mechanical Turk to study clinical populations. Clinical Psychological Science 2013, 1:213-220.
31. Shriberg L.D. Childhood speech sound disorders: From post-behaviorism to the postgenomic era. Speech sound disorders in children: In honor of Lawrence D. Shriberg 2009, 1-34. Plural Publishing, San Diego, CA. R. Paul, P.J. Flipsen (Eds.).
32. Shriberg L.D., Lof G.L. Reliability studies in broad and narrow phonetic transcription. Clinical Linguistics & Phonetics 1991, 5:225-279.
33. Shriberg L., Fourakis M., Hall S.D., Karlsson H.B., Lohmeier H.L., McSweeny J.L., et al. Perceptual and acoustic reliability estimates for the speech disorders classification system (SDCS). Clinical Linguistics and Phonetics 2010, 24(10):825-846.
34. Sprouse J. A validation of Amazon Mechanical Turk for the collection of acceptability judgments in linguistic theory. Behavior Research Methods 2011, 43(1):155-167.
35. Summerville A., Chartier C.R. Pseudo-dyadic interaction on Amazon's Mechanical Turk. Behavior Research Methods 2013, 45(1):116-124.
36. Suri S., Watts D.J. Cooperation and contagion in web-based, networked public goods experiments. PLoS ONE 2011, 6(3):e16836.
37. Zhu D., Carterette B. An analysis of assessor behavior in crowdsourced preference judgments 2010, Paper presented at the ACM SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation, Geneva, Switzerland.