Evaluating Speech Synthesis Intelligibility using Amazon Mechanical Turk

7th ISCA Workshop on Speech Synthesis, SSW 2010

Volumn , Issue , 2010, Pages 136-141

  Wolters, Maria K ^a   Isaac, Karl B ^a   Renals, Steve ^a  

^a UNIVERSITY OF EDINBURGH   (United Kingdom)

Author keywords

crowdsourcing; diphone; evaluation; HMM based synthesis; intelligibility; Mechanical Turk; semantically unpredictable sentences; unit selection

Indexed keywords

CROWDSOURCING; SPEECH COMMUNICATION; SPEECH RECOGNITION; SPEECH SYNTHESIS;

AMAZON'S MECHANICAL TURKS; DIPHONES; EVALUATION; HMM-BASED; HMM-BASED SYNTHESIS; INTELLIGIBILITY; MECHANICAL TURKS; MICROTASKS; SEMANTICALLY UNPREDICTABLE SENTENCE; UNIT SELECTION;

SPEECH INTELLIGIBILITY;

EID: 84885876466     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (24)

1. C. Callison-Burch and M. Dredze, “Creating speech and language data with amazon's mechanical turk,” in Proc. NAACL, 2010.
2. R. Snow, B. O'Connor, D. Jurafsky, and A. Ng, “Cheap and fast-but is it good?: evaluating non-expert annotations for natural language tasks,” in Proc. EMNLP, 2008.
3. S. Novotney and C. Callison-Burch, “Automatic speech recognition with non-expert transcription,” in Proc. NAACL, 2010.
4. M. Marge, S. Banerjee, and A. Rudnicky, “Using the Amazon Mechanical Turk for transcription of spoken language,” in Proc. ICASSP, 2010.
5. A. Kittur, E. Chi, and B. Suh, “Crowdsourcing user studies with Mechanical Turk,” in Proc. CHI, 2008.
6. M. Wolters, P. Campbell, C. DePlacido, A. Liddell, and D. Owens, “Making synthetic speech accessible to older people,” in Proc. SSW-6, Aug. 2007, pp. 288-293.
7. H. S. Venkatagiri, “Segmental intelligibility of four currently used text-to-speech synthesis methods.” J Acoust Soc Am, vol. 113, 2003.
8. K. Isaac, M. Wolters, and S. Renals, “How intelligible is synthetic speech in noise?” in Interspeech, subm.
9. H. Zen, T. Nose, J. Yamagishi, S. Sako, T. Masuko, A. Black, and K. Tokuda, “The HMM-based speech synthesis system (HTS) version 2.0,” in Proc. SSW-6, 2007.
10. K. S. Rhebergen and N. J. Versfeld, “A speech intelligibility index-based approach to predict the speech reception threshold for sentences in fluctuating noise for normal-hearing listeners,” J Acoust Soc Am, vol. 117, no. 4, 2005.
11. K. S. Rhebergen, N. J. Versfeld, and W. A. Dreschler, “Prediction of the intelligibility for speech in real-life background noises for subjects with normal hearing.”Ear Hear, vol. 29, no. 2, 2008.
12. U. D. Reips, “Standards for Internet-based experimenting,” Exp Psychol, vol. 49, 2002.
13. S. King and V. Karaiskos, “The Blizzard Challenge 2010,” in Proc. Blizzard Challenge Workshop, 2010.
14. P. G. Ipeirotis, “Demographics of Mechanical Turk,” New York University, Tech. Rep., 2010.
15. S. King and V. Karaiskos, “The Blizzard Challenge 2009,” in Proc. Blizzard Challenge Workshop, 2009.
16. C. W. Newman, B. E. Weinstein, G. P. Jacobson, and G. A. Hug, “The Hearing Handicap Inventory for Adults: psychometric adequacy and audiometric correlates.” Ear Hear, vol. 11, no. 6, 1990.
17. C. Benoît, M. Grice, and V. Hazan, “The SUS test: A method for the assessment of text-to-speech synthesis intelligibility using semantically unpredictable sentences,” Speech Communication, vol. 18, 1996.
18. H. T. Bunnell and J. Lilley, “Analysis methods for assessing TTS intelligibility,” in Proc. SSW-6, 2007.
19. J. Yamagishi, T. Nose, H. Zen, Z. Ling, T. Toda, K. Tokuda, S. King, and S. Renals, “Robust speaker-adaptive HMM-based text-to-speech synthesis,” IEEE Trans ASLP, vol. 17, no. 6, 2009.
20. A. Gelman and J. Hill, Data Analysis Using Regression and Multilevel/Hierarchical Models. Cambridge, UK: Cambridge University Press, 2007.
21. H. Baayen, Analyzing Linguistic Data. Cambridge University Press, 2008.
22. R Development Core Team, R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing, Vienna, Austria, 2009, ISBN 3-900051-07-0.
23. D. Bates and M. Maechler, lme4: Linear mixed-effects models using S4 classes, 2009, R package version 0.999375-31.
24. T. Hothorn, K. Hornik, M. van de Wiel, and A. Zeileis, “Implementing a class of permutation tests: The coin package,” Journal of Statistical Software, vol. 28, no. 8, 2008.