Predicting user satisfaction in spoken dialog system evaluation with collaborative filtering

IEEE Journal on Selected Topics in Signal Processing

Volumn 6, Issue 8, 2012, Pages 971-981

  Yang, Zhaojun ^a   Levow, Gina Anne ^b   Meng, Helen ^c  

^a University of Southern California   (United States)

^b UNIVERSITY OF WASHINGTON   (United States)

^c CHINESE UNIVERSITY OF HONG KONG   (Hong Kong)

Author keywords

Collaborative filtering; Crowdsourcing user satisfaction; Spoken dialog system evaluation

Indexed keywords

COLLABORATIVE FILTERING; CROWDSOURCING; LINEAR REGRESSION MODELS; MECHANICAL TURKS; SPOKEN DIALOG SYSTEMS; USER EVALUATIONS; USER SATISFACTION;

LINEAR REGRESSION; RATING;

FORECASTING;

EID: 84872169460     PISSN: 19324553     EISSN: None     Source Type: Journal    
DOI: 10.1109/JSTSP.2012.2229965     Document Type: Article

References (38)

1. A. Raux, B. Langner, D. Bohus, A. Black, and M. Eskenazi, "Let's go public! taking a spoken dialog system to the real world," in Proc. Interspeech, 2005.
2. L. Hirschman, M. Bates, D. Dahl,W. Fisher, J.Garofolo,D. Pallett,K. Hunicke-Smith, P. Price, A. Rudnicky, and E. Tzoukermann, "Multisite data collection and evaluation in spoken language understanding," in Proc. Workshop Human Lang. Technol., 1993, pp. 19-24.
3. H. Meng, P. Ching, S. Chan, Y. Wong, and C. Chan, "ISIS: An adaptive, trilingual conversational system with interleaving interaction and delegation dialogs," ACM Trans. Computer-Human Interaction (TOCHI), vol. 11, no. 3, pp. 268-299, 2004.
4. Z. Wu, H. Meng, H. Ning, and S. Tse, "A corpus-based approach for cooperative response generation in a dialog system," Chinese Spoken Lang. Process., pp. 614-626, 2006.
5. D. Litman, C. Rosé, K. Forbes-Riley, K. VanLehn, D. Bhembe, and S. Silliman, "Spoken versus typed human and computer dialogue tutoring," Int. J. Artif. Intell. in Educat., vol. 16, no. 2, pp. 145-170, 2006.
6. S. Möller, "Parameters for quantifying the interaction with spoken dialogue telephone services," in Proc. SIGdial Workshop Discourse and Dialogue, 2005, pp. 166-177.
7. M.McTear, "Spoken dialogue technology: Enabling the conversational user interface," ACM Comput. Surveys, vol. 34, no. 1, p. 169, 2002.
8. H. Bonneau-Maynard, L. Devillers, and S. Rosset, "Predictive performance of dialog systems," in Proc. Lang. Resources Eval. Conf. (LREC), 2000.
9. S.Moller and J. Skowronek, "Quantifying the impact of system characteristics on perceived quality dimensions of a spoken dialogue service," in Proc. 8th Eur. Conf. Speech Commun. Technol., 2003.
10. M. Walker, D. Litman, C. Kamm, and A. Abella, "PARADISE: A framework for evaluating spoken dialogue agents," in Proc. Assoc. Comput. Linguist. (ACL), 1997.
11. M. A. Walker, D. J. Litman, C. A. Kamm, and A. Abella, "Evaluating spoken dialogue agents with PARADISE: Two case studies," Comput. Speech Lang., vol. 12, no. 4, pp. 317-347, 1998.
12. K. Engelbrech, F. Gödde, F. Hartard, H. Ketabdar, and S. Möller, "Modeling user satisfaction with hidden Markov model," in Proc. SIGDIAL Workshop Discourse and Dialogue, 2009, pp. 170-177.
13. I. McGraw, C.Y. Lee, L.Hetherington, and J.Glass, "Collecting voices from the crowd," in Proc. Lang. Resources Eval. Conf. (LREC), 2010.
14. R. Snow, B. O'Connor, D. Jurafsky, and A. Ng, "Cheap and fast - But is it good?: Evaluating non-expert annotations for natural language tasks," in Proc. Conf. Empir. Meth. in Nat. Lang. Process., 2008, pp. 254-263.
15. S. Novotney and C. Callison-Burch, "Cheap, fast and good enough: Automatic speech recognition with non-expert transcription," in Proc. Assoc. Comput. Linguist. (ACL), 2010, pp. 207-215.
16. Z. Yang, B. Li, Y. Zhu, I. King, G. Levow, and H. Meng, "Collection of user judgments on spoken dialog system with crowdsourcing," in Proc. IEEE Workshop Spoken Lang. Technol. (SLT), 2010, pp. 277-282.
17. X. Su and T. M. Khoshgoftaar, "A survey of collaborative filtering techniques," Adv. Artif. Intell., 2009.
18. J.Wang, A. De Vries, andM. Reinders, "Unifying user-based and itembased collaborative filtering approaches by similarity fusion," in Proc. SIGIR, 2006, pp. 501-508.
19. R. Jin, J. Y. Chai, and L. Si, "An automatic weighting an automatic weighting an automatic weighting scheme for collaborative fltering," in Proc. SIGIR, 2004.
20. B. Sarwar, G. Karypis, J. Konstan, and J. Reidl, "Item-based collaborative filtering recommendation algorithms," in Proc. Int. Conf. World Wide Web (WWW), 2001, p. 295.
21. G. Linden, B. Smith, and J. York, "Amazon.com recommendations: Item-to-item collaborative filtering," IEEE Internet Comput., vol. 7, no. 1, pp. 76-80, Jan./Feb. 2003.
22. G. Adomavicius and A. Tuzhilin, "Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions," IEEE Trans. Knowl. Data Eng., pp. 734-749, 2005. (Pubitemid 40860454)
23. Z.Yang,Y. Zhu, B. Li, I. King, G. Levow, andH.Meng, "Collaborative filtering model for user satisfaction prediction in spoken dialog evaluation," in Proc. IEEEWorkshop Spoken Language Technol. (SLT), 2010, pp. 472-477.
24. K. Hone and R. Graham, "Subjective assessment of speech-system interface usability," in Proc. 7th Eur. Conf. Speech Commun. Technol., 2001.
25. I. Rec. "P. 851. Subjective quality evaluation of telephone services based on spoken dialogue systems," Int. Telecomm. Union, Geneva, 2003.
26. J.Glass, J. Polifroni, S. Seneff, andV. Zue, "Data collection and performance evaluation of spoken dialogue systems: The MIT experience," in Proc. Int. Conf. Spoken Lang. Process. (ICSLP), 2000.
27. M. McTear, "Modelling spoken dialogues with state transition diagrams: experiences with the CSLU toolkit," in Proc. ICSLP, 1998, pp. 1223-1226.
28. "Parameters describing the interaction with spoken dialogue systems," ITU. Geneva, 2005, I. P. series Rec.
29. P. Cohen, Empirical Methods for Artificial Intelligence. Cambridge, U.K.: MIT Press, 1995, vol. 55.
30. K. Forbes-Riley and D. Litman, "Modelling user satisfaction and student learning in a spoken dialogue tutoring system with generic, tutoring, and user affect parameters," in Proc. Assoc. Comput. Linguist. (ACL), 2006, pp. 264-271.
31. M. Walker, R. Passonneau, and J. Boland, "Quantitative and qualitative evaluation of DARPA communicator spoken dialogue systems," in Proc. Assoc. Comput. Linguist. (ACL), 2001, p. 522.
32. C. Kamm, M. Walker, and D. Litman, "Evaluating spoken language systems," in Proc. AVIOS, 1999.
33. M. Walker, C. Kamm, and D. Litman, "Towards developing general models of usabilitywith PARADISE," Nat. Lang. Eng., vol. 6, no. 3&4, pp. 363-377, 2000.
34. N. Beringer, U. Kartal, K. Louka, F. Schiel, and U. Türk, "Promise: A procedure for multimodal interactive system evaluation," in Proc. LREC Workshop Multimodal Resources and Multimodal Syst. Eval., 2002, pp. 90-95.
35. L. Dybkjaer, N. Bernsen, and W. Minker, "Evaluation and usability of multimodal spoken language dialogue systems," Speech Commun., vol. 43, no. 1-2, pp. 33-54, 2004.
36. A. Black, S. Burger, B. Langner, G. Parent, and M. Eskenazi, "Spoken dialog challenge 2010," in Proc. IEEE Workshop Spoken Lang. Technol., 2010, pp. 448-453.
37. M. Shoukri, Measures of Interobserver Agreement. Boca Raton, FL: CRC, 2004.
38. K. Engelbrecht and S. Möller, "Pragmatic usage of linear regression models for the prediction of user judgments," in Proc. SIGdial Workshop Discourse and Dialogue, 2007.