Data-driven user simulation for automated evaluation of spoken dialog systems

Computer Speech and Language

Volumn 23, Issue 4, 2009, Pages 479-509

  Jung, Sangkeun ^a   Lee, Cheongjae ^a   Kim, Kyungduk ^a   Jeong, Minwoo ^a   Lee, Gary Geunbae ^a  

^a Pohang University of Science and Technology (POSTECH)   (South Korea)

Author keywords

Data driven; Dialog simulation; Dialog system; Evaluation; Spoken dialog system; User simulation

Indexed keywords

DATA-DRIVEN; DIALOG SIMULATION; DIALOG SYSTEM; EVALUATION; SPOKEN DIALOG SYSTEM; USER SIMULATION;

KNOWLEDGE BASED SYSTEMS;

SYSTEMS ENGINEERING;

EID: 67349121246     PISSN: 08852308     EISSN: 10958363     Source Type: Journal    
DOI: 10.1016/j.csl.2009.03.002     Document Type: Article

References (42)

1. Ai H., and Diane J.L. Assessing Dialog System User Simulation Evaluation Measures Using Human Judges (2008), Association for Computational Linguistics
2. Boros, M., Eckert, W., Gallwitz, F., Gorz, G., Hanrieder, G., Niemann, H., 1996. Towards understanding spontaneous speech: Word accuracy vs. conceptaccuracy. In: 4th International Conference on Spoken Language Processing, vol. 2.
3. Bos, J., Klein, E., Lemon, O., Oka, T., 2003. DIPPER: Description and formalisation of an information-state update dialogue system architecture. In: 4th SIGdial Workshop on Discourse and Dialogue, Sapporo, Japan.
4. Chung G. Developing a Flexible Spoken Dialog System Using Simulation (2004), Association for Computational Linguistics pp. 63-70
5. Cuayahuitl H., Renals S., Lemon O., and Shimodaira H. Human-computer dialogue simulation using hidden Markov models. Automatic Speech Recognition and Understanding (2005) 100-105
6. Deng, Y., Mahajan, M., Acero. A., 2003. Estimating speech recognition error rate without acoustic test data. In: 8th European Conference on Speech Communication and Technology.
7. Eckert W., Levin E., and Pieraccini R. User modeling for spoken dialogue system evaluation. Automatic Speech Recognition and Understanding (1997) 80-87
8. Fosler-Lussier, E., Amdal, I., Kuo, H.K.J., 2002. On the road to improved lexical confusability metrics. In: ISCA Tutorial and Research Workshop (ITRW) on Pronunciation Modeling and Lexicon Adaptation for Spoken Language Technology.
9. Georgila, K., Henderson, J., Lemon, O., 2005. Learning user simulations for information state update dialogue systems. In: 9th European Conference on Speech Communication and Technology: ISCA.
10. Georgila, K., Henderson, J., Lemon, O., 2006. User simulation for spoken dialogue systems: Learning and evaluation. In: 9th International Conference on Spoken Language Processing.
11. Greenberg, S., Chang, S., Hollenback, J., 2000. An introduction to the diagnostic evaluation of switchboard-corpus automatic speech recognition systems. NIST Speech Transcription Workshop.
12. Hunt M.J. Figures of merit for assessing connected-word recognisers. Speech Input/Output Assessment and Speech Databases (1989)
13. Jeong, M., Lee, G.G., 2006. Jointly predicting dialog act and named entity for spoken language understanding. Spoken Language Technology Workshop. IEEE 66-69.
14. Kim, K., Lee, C., Jung, S., Lee, G.G., 2008. A frame-based probabilistic framework for spoken dialog management using dialog examples. SigDial.
15. Kneser, R., Ney, H., 1995. Improved backing-off for M-gram language modeling. In: International Conference on Acoustics, Speech, and Signal Processing, vol. 1.
16. Lafferty, J.D., McCallum, A., Pereira, F.C.N., 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In: Proceedings of the Eighteenth International Conference on Machine Learning table of contents, pp. 282-289.
17. Lee C., Jung S., Eun J., Jeong M., and Lee G.G. Example and situation based dialog management for spoken dialog system. Automatic Speech Recognition and Understanding (2005)
18. Lee, J., Kim, S., Lee, G.G., 2006. Grapheme-to-phoneme conversion using automatically extracted associative rules for Korean TTS system. In: 9th International Conference on Spoken Language Processing.
19. Lee, C., Jung, S., Lee, G.G., 2008. Robust dialog management with n-best hypotheses using dialog example and agenda. In: Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics, Ohio.
20. Lemon, O., Georgila, K., Henderson, J., 2006. Evaluating effectiveness and portability of reinforcement learned dialogue strategies with real users: The talk towninfo evaluation. In: Eurospeech.
21. Levin E., Pieraccini R., and Eckert W. A stochastic model of human-machine interaction for learning dialogstrategies. IEEE Transactions on Speech and Audio Processing 8 1 (2000) 11-23
22. Liu D.C., and Nocedal J. On the limited memory BFGS method for large scale optimization. Mathematical Programming 45 1 (1989) 503-528
23. López-Cózar R., De la Torre A., Segura J.C., and Rubio A.J. Assessment of dialogue systems by means of a new simulation technique. Speech Communication 40 3 (2003) 387-407
24. López-Cózar R., Callejas Z., and McTear M. Testing the performance of spoken dialogue systems by means of an artificially simulated user. Artificial Intelligence Review 26 4 (2006) 291-323
25. Needleman S.B., and Wunsch C.D. A general method applicable to the search for similarities in the amino acid sequence of two proteins. Journal of Molecular Biology 48 3 (1970) 443-453
26. Papineni, K., Roukos, S., Ward, T., Zhu, W.J., 2001. BLEU: A method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pp. 311-318.
27. Pietquin, O., 2004. A Framework for Unsupervised Learning of Dialogue Strategies. Ph.D. thesis, Faculty of Engineering, Mons, Belgium.
28. Pietquin O., and Dutoit T. A probabilistic framework for dialog simulation and optimal strategy learning. Audio, Speech and Language Processing, IEEE Transactions on [see also Speech and Audio Processing, IEEE Transactions on] 14 2 (2006) 589-599
29. Pietquin, O., Renals, S., 2002. ASR system modeling for automatic evaluation and optimization of dialogue systems. In: ICASSP.
30. Prommer, T, Holzapfel, H., Waibel, A., 2006. Rapid simulation-driven reinforcement learning of multimodal dialog strategies in human-robot interaction. In: ICSLP: ISCA.
31. Schatzmann, J., Georgila, K., Young, S., 2005. Quantitative evaluation of user simulation techniques for spoken dialogue systems. In: 6th SIGdial Workshop on Discourse and Dialogue.
32. Schatzmann, J, Thomson, B ., Weilhammer, K., Ye, H., Young, S., 2007a. Agenda-based user simulation for bootstrapping a POMDP dialogue system. In: Proc. of HLT/NAACL.
33. Schatzmann J., Thomson B., and Young S. Error simulation for training statistical dialogue systems. Automatic Speech Recognition and Understanding (2007) 526-531
34. Schatzmann, J., Thomson, B., Young, S., 2007c. Statistical user simulation with a hidden agenda. SigDial.
35. Scheffler, K., Young, S., 2000. Probabilistic simulation of human-machine dialogues. In: IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 2.
36. Scheffler, K., Young, S., 2001. Corpus-based dialogue simulation for automatic strategy learning and evaluation. NAACL Workshop on Adaptation in Dialogue Systems, pp. 64-70.
37. Seneff S. Response planning and generation in the Mercury flight reservation system. Computer Speech and Language 16 3 (2002) 283-312
38. Stuttle, M., Williams, J., Young, S., 2004. A framework for dialog systems data collection using a simulated ASR channel. In: International Conference on Spoken Language Processing.
39. Torres F., Sanchis E., and Segarra E. User simulation in a stochastic dialog system. Computer Speech and Language 22 3 (2008) 230-255
40. Williams J.D., and Young S. Partially observable Markov decision processes for spoken dialog systems. Computer Speech & Language 21 2 (2007) 393-422
41. Young S., Odell J.., Ollason D.., Valtchev V., and Woodland P. The HTK Book (1997), Entropic Cambridge Research Laboratory
42. Zukerman I., and Albrecht D.W. Predictive statistical models for user modeling. User Modeling and User-Adapted Interaction 11 1 (2001) 5-18