Collection of user judgments on spoken dialog system with crowdsourcing

2010 IEEE Workshop on Spoken Language Technology, SLT 2010 - Proceedings

Volumn , Issue , 2010, Pages 277-282

  Yang, Zhaojun ^a   Li, Baichuan ^a   Zhu, Yi ^a   King, Irwin ^a   Levow, Gina ^b   Meng, Helen ^a  

^a CHINESE UNIVERSITY OF HONG KONG   (Hong Kong)

^b UNIVERSITY OF WASHINGTON   (United States)

Author keywords

Amazon mechanical Turk; Crowdsourcing; Let's go; Spoken dialog system; User judgment

Indexed keywords

CROWDSOURCING; LET'S GO; MECHANICAL TURKS; SPOKEN DIALOG SYSTEMS; USER JUDGMENT;

RATING;

QUALITY CONTROL;

EID: 79951807212     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/SLT.2010.5700864     Document Type: Conference Paper

References (11)

1. M.F. McTear, "Spoken dialogue technology: enabling the conversational user interface," ACM Computing Surveys, vol. 34, no. 1, pp. 169, 2002.
2. S.Möller, "Parameters for quantifying the interaction with spoken dialogue telephone services," in Proc. of the SIGDIAL, 2005.
3. J. Glass, J. Polifroni, S. Seneff, and V. Zue, "Data collection and performance evaluation of spoken dialogue systems: the MIT experience," in Proc. of ICSLP, 2000.
4. H. Bonneau-Maynard, L. Devillers, and S. Rosset, "Predictive performance of dialog systems," in Proc. of LREC, 2000.
5. M.A.Walker, D.J. Litman, C.A. Kamm, and A. Abella, "PARADISE: a framework for evaluating spoken dialogue agents," in Proc. of European chapter of the Association for Computational Linguistics, 1997.
6. I. McGraw, C. ying Lee, L. Hetherington, and J. Glass, "Collecting voices from the crowd," in Proc. of LREC, 2010.
7. A. Raux, B. Langner, D. Bohus, A. Black, and M. Eskenazi, "Let's go public! taking a spoken dialog system to the real world," in Proc. of Interspeech, 2005.
8. ITU P series Rec, "Parameters Describing the Interaction with Spoken Dialogue Systems," ITU, Geneva, 2005.
9. M.M. Shoukri, Measures of interobserver agreement, 2004.
10. J.R. Landis and G.G. Koch, "The measurement of observer agreement for categorical data," Biometrics, vol. 33, no. 1, pp. 159-174, 1977.
11. K.P. Engelbrech, F. Gödde, F. Hartard, H. Ketabdar, and S. Möller, "Modeling user satisfaction with Hidden Markov Model," in Proc. of the SIGDIAL, 2009.