Evaluating user simulations with the Cramér-von Mises divergence

Speech Communication

Volumn 50, Issue 10, 2008, Pages 829-846

  Williams, Jason D ^a  

^a AT AND T LABS RESEARCH   (United States)

Author keywords

Dialog management; Dialog simulation; User modeling; User simulation

Indexed keywords

HUMAN COMPUTER INTERACTION; INDUSTRIAL MANAGEMENT;

DIALOG MANAGEMENT; DIALOG SIMULATION; DIALOG SYSTEMS; DOMAIN-SPECIFIC; INFORMATION NEEDS; NOVEL METHODS; SCORING FUNCTIONS; STATISTICAL SIGNIFICANCE; USER MODELING; USER SIMULATION; VON MISES;

SOFTWARE AGENTS;

EID: 52949090026     PISSN: 01676393     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.specom.2008.05.007     Document Type: Article

References (43)

1. Ai, H., Litman, D.J., 2007. Knowledge consistent user simulations for dialog systems. In: Proc. Eurospeech, Antwerp, Belgium.
2. Anderson T. On the distribution of the two-sample Cramér-von Mises criterion. Ann. Math. Statist. 33 3 (1962) 1148-1159
3. Bui, T., Poel, M., Nijholt, A., Zwiers, J., 2007. A tractable DDN-POMDP approach to affective dialogue modeling for general probabilistic frame-based dialogue systems. In: Proc. Workshop on Knowledge and Reasoning in Practical Dialogue Systems, Internat. Joint Conf. on Artificial Intelligence (IJCAI), Hyderabad, India, pp. 34-37.
4. Cramér H. On the composition of elementary errors. second paper: Statistical applications. Skandinavisk Aktuarietidskrift 11 (1928) 171-180
5. Cuayáhuitl, H., Renals, S., Lemon, O., Shimodaira, H., 2005. Human-computer dialogue simulation using hidden markov models. In: Proc. IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), San Juan, Puerto Rico, USA, pp. 290-295.
6. Denecke, M., Dohsaka, K., Nakano, M., 2004. Learning dialogue policies using state aggregation in reinfocement learning. In: Proc. Internat. Conf. on Spoken Language Processing (ICSLP), Jeju, Korea, pp. 325-328.
7. Eadie, W., Drijard, D., James, F., Roos, M., Sadoulet, B., 1971. Statistical Methods in Experimental Physics. North Holland.
8. Filisko, E., Seneff, S., 2005. Developing city name acquisition strategies in spoken dialogue systems via user simulation. In: Proc. SIGdial Workshop on Discourse and Dialogue, Lisbon, Portugal.
9. Frampton, M., Lemon, O., 2006. Learning more effective dialogue strategies using limited dialogue move features. In: Proc. Association for Computational Linguistics (ACL), Sydney, pp. 185-192.
10. Georgila, K., Henderson, J., Lemon, O., 2005. Learning user simulations for information state update dilaogue systems. In: Proc. Eurospeech, Libson, Portugal, pp. 893-896.
11. Georgila, K., Henderson, J., Lemon, O., 2006. User simulation for spoken dialogue systems: Learning and evaluation. In: Proc. Internat. Conf. on Spoken Language Processing (ICSLP), Pittsburgh, Pennsylvania, USA.
12. Goddeau, D., Pineau, J., 2000. Fast reinforcement learning of dialog strategies. In: Proc. Internat. Conf. on Acoustics, Speech, and Signal Processing (ICASSP), Istanbul, pp. 1233-1236.
13. Heeman, P., 2007. Combining reinforcement learning with information-state update rules. In: Proc. Human Language Technologies: The Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), Rochester, New York, USA, pp. 268-275.
14. Henderson, J., Lemon, O., Georgila, K., 2005. Hybrid reinforcement/supervised learning for dialogue policies from Communicator data. In: Proc. Workshop on Knowledge and Reasoning in Practical Dialogue Systems, Internat. Joint Conf. on Artificial Intelligence (IJCAI), Edinburgh, pp. 68-75.
15. Kolmogorov A. Sulla determinazione empirica di una legge di distribuzione. Giorn. Ist. Ital. Attuari 4 (1933) 1-11
16. Kullback S., and Leibler R. On information and sufficiency. Ann. Math. Statist. 22 (1951) 79-86
17. Lemon, O., Georgila, K., Henderson, J., 2006. Evaluating effectiveness and portability of reinforcement learned dialogue strategies with real users: the TALK TownInfo evaluation. In: Proc. Workshop on Spoken Language Technologies (SLT), Aruba, pp. 178-181.
18. Levin, E., Pieraccini, R., 1997. A stochastic model of computer-human interaction for leaning dialog strategies. In: Proc. Eurospeech, Rhodes, Greece, pp. 1883-1886.
19. Levin, E., Pieraccini, R., 2006. Value-based optimal decision for dialog systems. In: Proc. Workshop on Spoken Language Technologies (SLT), Aruba, pp. 198-201.
20. Levin E., Pieraccini R., and Eckert W. A stochastic model of human-machine interaction for learning dialogue strategies. IEEE Trans. Speech and Audio Processing 8 1 (2000) 11-23
21. Pietquin, O., 2004. A framework for unsupervised learning of dialogue strategies. Ph.D. thesis, Faculty of Engineering, Mons (TCTS Lab), Belgium.
22. Pietquin, O., Renals, S., 2002. ASR modelling for automatic evaluation and optimisation of dialogue systems. In: Proc. Internat. Conf. on Acoustics, Speech, and Signal Processing (ICASSP), Florida, USA, pp. 46-49.
23. Roy, N., Pineau, J., Thrun, S., 2000. Spoken dialogue management using probabilistic reasoning. In: Proc. Association for Computational Linguistics (ACL), Hong Kong, pp. 93-100.
24. Schatzmann, J., Georgila, K., Young, S., 2005. Quantitative evaluation of user simulation techniques for spoken dialogue systems. In: Proc. SIGdial Workshop on Discourse and Dialogue, Lisbon, Portugal, pp. 178-181.
25. Schatzmann, J., Thomson, B., Young, S., 2007a. Error simulation for training statistical dialogue systems. In: Proc. IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), Kyoto, Japan, pp. 526-531.
26. Schatzmann, J., Thomson, B., Young, S., 2007b. Statistical user simulation with a hidden agenda. In: Proc. SIGdial Workshop on Discourse and Dialogue, Antwerp, Belgium, pp. 273-282.
27. Scheffler, K., 2002. Automatic design of spoken dialogue systems. Ph.D. thesis, Cambridge University.
28. Scheffler, K., Young, S., 2001. Corpus-based dialogue simulation for automatic strategy learning and evaluation. In: Proc. North American Association for Computational Linguistics (NAACL) Workshop on Adaptation in Dialogue Systems, Pittsburgh, USA, pp. 12-19.
29. Scheffler, K., Young, S., 2002. Automatic learning of dialogue strategy using dialogue simulation and reinforcement learning. In: Proc. Human Language Technologies (HLT), San Diego, USA, pp. 12-18.
30. Singh S., Litman D., Kearns M., and Walker M. Optimizing dialogue management with reinforcement leaning: experiments with the NJFun system. J. Artif. Intell. 16 (2002) 105-133
31. Stephens, M., 1992. Introduction to: Kolmogorov (1933) on the empirical determination of a distribution. In: Kotz, S., Johnson, N. (Eds.), Breakthrougth in statistics, Vol. II. Springer Verlag, pp. 93-105.
32. Stuttle, M., Williams, J., Young, S., 2004. A framework for Wizard-of-Oz experiments with a simulated ASR channel. In: Proc. Internat. Conf. on Spoken Language Processing (ICSLP), Jeju, Korea, pp. 241-244.
33. Thomson, B., Schatzmann, J., Welhammer, K., Ye, H., Young, S., 2007. Training a real-world POMDP-based dialog system. In: Proc. Human Language Technologies: The Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT) Workshop on Bridging the Gap: Academic and Industrial Research in Dialog Technologies, Rochester, New York, USA, pp. 9-17.
34. von Mises, R., 1931. Wahrscheinlichkeitsrechnung und ihre Anwendung in der Statistik und Theoretischen Physik. F Deuticke.
35. Walker, M., Fromer, J., Narayanan, S., 1998. Learning optimal dialogue strategies: a case study of a spoken dialogue agent for email. In: Proc Association for Computational Linguistics and Internat. Conf. on Computational Linguistics (ACL/COLING), Montreal, pp. 1345-1351.
36. Walker, M., Kamm, C., Litman, D., 2000. Towards developing general models of usability with PARADISE. Natural Language Engineering: Special Issue on Best Practice in Spoken Dialogue Systems. URL: citeseer.ist.psu.edu/article/walker00towards.html.
37. Walker, M., Rudnicky, A., Prasad, R., Aberdeen, J., Bratt, E., Garofolo, J., Hastie, H., Le, A., Pellom, B., Potamianos, A., Passonneau, R., Roukos, S., Sanders, G., Seneff, S., Stallard, D., 2002. DARPA Communicator: Cross-system results for the 2001 evaluation. In: Proc. Internat. Conf. on Spoken Language Processing (ICSLP), Colorado, USA, pp. 269-272.
38. Williams, J., 2006. Partially observable Markov decison processes for spoken dialogue management. Ph.D. thesis, Cambridge University.
39. Williams, J., 2007. Applying POMDPs to dialog systems in the troubleshooting domain. In: NAACL-HLT Workshop on Bridging the Gap: Academic and Industrial Research in Dialog Technologies, Rochester, New York, USA, pp. 1-8.
40. Williams J., and Young S. Scaling POMDPs for spoken dialog management. IEEE Trans. Audio, Speech, and Language Processing 15 7 (2007) 2116-2129
41. Williams J.D., and Young S. Partially observable Markov decision processes for spoken dialog systems. Computer Speech and Language 21 2 (2007) 393-422
42. Young, S., Schatzmann, J., Weilhammer, K., Ye, H., 2007. The hidden information state approach to dialog management. In: Proc. International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Honolulu, Hawaii, USA, pp. IV149-IV152.
43. Zhang, B., Cai, Q., Mao, J., Chang, E., Guo, B., 2001. Spoken dialogue management as planning and acting under uncertainty. In: Proc. Eurospeech, Aalborg, Denmark, pp. 2169-2172.