Testing the performance of spoken dialogue systems by means of an artificially simulated user

Artificial Intelligence Review

Volumn 26, Issue 4, 2006, Pages 291-323

  López Cózar, Ramón ^a   Callejas, Zoraida ^a   McTear, Michael ^b  

^a UNIVERSITY OF GRANADA   (Spain)

^b UNIVERSITY OF ULSTER   (United Kingdom)

Author keywords

Artificial intelligence; Natural language processing; Robust human computer interaction; Speech recognition; Speech understanding; Spoken dialogue systems; User simulation

Indexed keywords

ARTIFICIAL INTELLIGENCE; COMPUTER SIMULATION; HUMAN COMPUTER INTERACTION; MATHEMATICAL MODELS; NATURAL LANGUAGE PROCESSING SYSTEMS;

SPEECH UNDERSTANDING; SPOKEN DIALOGUE SYSTEMS; USER SIMULATION;

SPEECH RECOGNITION;

EID: 38349010687     PISSN: 02692821     EISSN: None     Source Type: Journal    
DOI: 10.1007/s10462-007-9059-9     Document Type: Article

References (39)

1. Allen J (1995) Natural language understanding. The Benjamin/Cummings Publishing Company Inc
2. Allen J, Byron D, Dzikovska M, Ferguson G, Galescu L and Stent A (2001). Towards conversational human-computer interaction. AI Mag 22(4): 27-38
3. Aust H, Oerder M (1995) Dialogue control in automatic inquiry systems. In: ESCA workshop on spoken dialogue systems, Vigso, Denmark, June pp 121-124
4. Carbini S, Delphin-Poulat L, Perron L and Viallet JE (2006). From a Wizard of Oz experiments to a real time speech and gesture multimodal interface. Signal Processing 86(12): 3559-3577
5. Chung G (2004) Developing a flexible spoken dialog system using simulation. In Proceedings of ACL, Barcelona, Spain, pp 63-70
6. Cuayáhuitl H, Renals S, Lemon O, Shimodaira H (2005) Human-computer dialogue simulation using Hidden Markov models. In: Proc. of 2005 IEEE automatic speech recognition and understanding workshop (ASRU), San Juan, Puerto Rico, 28 November-1 December 2005, pp 290-295
7. Dow S, MacIntyre B, Lee J, Oezbek C, Bolter JD and Gandy M (2005). Wizard of Oz support throughout an interative design process. IEEE Pervasive Comput 4(8): 18-26
8. Eckert W, Levin E, Pieraccini R (1997) User modelling for spoken dialogue system evaluation. In: Proceedings of 1997 IEEE automatic speech recognition and understanding workshop (ASRU), Santa Barbara, California, 14-17 December 1997, pp 80-87
9. Georgila K, Henderson J, Lemon O (2005) Learning user simulations for information state update dialogue systems. In: Proceedings of Interspeech-eurospeech, Lisbon, Portugal, 4-8 September 2005, pp 893-896
10. Hain T, Woodland PC, Niesler TR, Whittaker EWD (1999) The 1998 HTK system for transcription of conversational telephone speech. In: Proceedings of the 1999 international conference on acoustics, speech and signal processing, pp 57-60
11. Huang X, Acero A, Hon H (2001) Spoken language processing: a guide to theory, Algorithm and System Development, Prentice-Hall
12. Levin E, Pieraccini R and Eckert W (2000). A stochastic model of human-machine interaction for learning dialog strategies. IEEE Trans Speech Audio Process 8(1): 11-23
13. Litman D and Pan S (2002). Designing and evaluating an adaptive spoken dialogue system. User Model User-adapt Interact 12(2/3): 111-137
14. Lin BS and Lee LS (2001). Computer aided analysis and design for spoken dialogue systems based on quantitative simulations. IEEE Trans Speech Audio Process 9(5): 534-548
15. López-Cózar R, Araki M (2005) Spoken, multilingual and multimodal dialogue systems. Development and Assessment, John Wiley & Sons Publishers
16. López-Cózar R and Callejas Z (2005). Combining language models in the input interface of a spoken dialogue system. Comput Speech Lang 20: 420-440
17. López-Cózar R, Milone DH (2001) A new technique based on augmented language models to improve the performance of spoken dialogue systems. Proceedings of Eurospeech, pp 741-744
18. López-Cózar R, García P, Díaz J, Rubio AJ (1997) A voice activated dialogue system for fast-food restaurant applications. Proceedings of Eurospeech, pp 1783-1786
19. López-Cózar R, Rubio AJ, García P, Segura JC (1998) A spoken dialogue system based on a dialogue corpus analysis. Proceedings of LREC, pp 55-58
20. López-Cózar R, Rubio AJ, Díaz Verdejo JE, De la Torre A (2000) Evaluation of a dialogue system based on a generic model that combines robust speech understanding and mixed-initiative control. Proceedings of LREC, pp 743-748
21. López-Cózar R, De la Torre A, Segura JC, Rubio AJ, López-Soler JM (2002) A new method for testing dialogue systems based on simulations of real-world conditions. Proceedings of ICSLP, pp 305-308
22. López-Cózar R, De la Torre A, Segura JC, Rubio AJ and Sánchez V (2003). Assessment of dialogue systems by means of a new simulation technique. Speech Communi 40(3): 387-407
23. McTear M (2004). Spoken dialogue technology: toward the conversational user interface. Springer, London
24. Möller S (2004) Quality of telephone-based spoken dialogue systems, Springer
25. Möller S, Englert R, Engelbrecht K, Hafner V, Jameson A, Oulasvirta A, Raake A, Reithinger N (2006) MeMo: towards automatic usability evaluation of spoken dialogue services by user error simulations. Proceedings of Interspeech, pp 1786-1789
26. Okamoto M, Cho K, Okamoto Y, Yamasaki T, Hattori M (2005) User-model-based adaptability evaluation for context-aware systems. In: Proceedings of the international conference of pervasive services, Santorini, Greece, 11-14 July 2005, pp 470-473
27. Pieraccini R, Huerta J (2005) Where do we go from here? Research and commercial spoken dialog systems. In: Proceedings of 6th SIGdial workshop on dialogue and discourse, Lisbon, Portugal, 2-3 September 2005, pp 1-10
28. Pietquin O and Dutoit T (2006a). A probabilistic framework for dialog simulation and optimal strategy learning. IEEE Trans Audio Speech Lang Process 14(2): 589-599
29. Pietquin O, Dutoit T (2006b) Dynamic Bayesian networks for NLU simulation with applications to dialog optimal strategy learning. In: Proceedings of the 2006 IEEE international conference on acoustics, speech and signal processing (ICASSP), Toulouse, France, 15-19 May 2005, pp 49-52
30. Rabiner L, Juang BH (1993) Fundamentals of speech recognition, Prentice-Hall
31. Schatzmann J, Georgila K, Young S (2005a) Quantitative evaluation of user simulation techniques for spoken dialogue systems. In: Proceedings of the 6th SIGdial workshop on discourse and dialogue, Lisbon
32. Schatzmann J, Stuttle MN, Weilhammer K, Young S (2005b) Effects of the user model on simulation-based learning of dialogue strategies. In Proceedings of IEEE automatic speech recognition and understanding workshop (ASRU), San Juan, Puerto Rico 2005, pp 220-225
33. Scheffler K, Young S (2000) Probabilistic simulation of human-machine dialogues. In: Proceedings of the IEEE international conference on acoustics, speech and signal processing (ICASSP), Istanbul, Turkey, pp 1217-1220
34. Scheffler, K, Young S (2001) Corpus-based dialogue simulation for automatic strategy learning and evaluation. In: Proceedings of the NAACL workshop on adaptation in dialogue systems, Pittsburgh, pp 64-70
35. Singh S, Litman D, Kearns M and Walker M (2002). Optimizing dialogue managment with reinforcement learning: experiments with the NJFun system. J Artif Intell Res 16: 105-133
36. Walker M (2000). An application of reinforcement learning to dialogue strategy selection in a spoken dialogue system for email. J Artif Intell Res JAIR 12: 387-416
37. Walker M, Litman D, Kamm C, Abella A (1997) PARADISE: a framework for evaluating spoken dialogue agents. In: Proceedings of the 35th annual meeting of the association for computational linguistics (ACL-97), pp 271-280
38. Young S (2002) Talking to machines (Statistically Speaking) In: Proceedings of ICSLP'2002, 7th international conference on spoken language processing, Denver, Colorado
39. Young S, Kershaw D, Odell J, Ollason D, Valtchev V, Woodland P (2000) The HTK book (for HTK Version 3.0). Microsoft Corporation