A method for evaluating and comparing user simulations: The Cramér-von Mises divergence

2007 IEEE Workshop on Automatic Speech Recognition and Understanding, ASRU 2007, Proceedings

Volumn , Issue , 2007, Pages 508-513

  Williams, Jason D ^a  

^a AT AND T LABS RESEARCH   (United States)

Author keywords

Dialog management; Dialog simulation; User modelling; User simulation

Indexed keywords

COMPUTER NETWORKS; MULTITASKING; SPEECH; SPEECH ANALYSIS;

AUTOMATIC SPEECH RECOGNITION (ASR); CRITICAL VALUES; DIALOG SYSTEMS; DOMAIN SPECIFIC; SCORING FUNCTIONS; STATISTICAL SIGNIFICANCE; VON MISES;

SPEECH RECOGNITION;

EID: 44849122956     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (19)

1. S Singh, DJ Litman, M Kearns, and MA Walker, "Optimizing dialogue management with reinforcement leaning: experiments with the NJFun system," Journal of Artificial Intelligence, vol. 16, pp. 105-133, 2002.
2. E Levin, R Pieraccini, and W Eckert, "A stochastic model of human-machine interaction for learning dialogue strategies," IEEE Trans on Speech and Audio Processing, vol. 8, no. 1, pp. 11-23, 2000.
3. K Scheffler and SJ Young, "Automatic learning of dialogue strategy using dialogue simulation and reinforcement learning," in Proc Human Language Technologies (HLT), San Diego, USA, 2002, pp. 12-18.
4. O Pietquin, A framework for unsupervised learning of dialogue strategies, Ph.D. thesis, Faculty of Engineering, Mons (TCTS Lab), Belgium, 2004.
5. J Henderson, O Lemon, and K Georgila, "Hybrid reinforcement/ supervised learning for dialogue policies from Communicator data," in Proc Workshop on Knowledge and Reasoning in Practical Dialogue Systems, IJCAI, Edinburgh, 2005, pp. 68-75.
6. N Roy, J Pineau, and S Thrun, "Spoken dialogue management using probabilistic reasoning," in Proc ACL, Hong Kong, 2000, pp. 93-100.
7. B Zhang, Q Cai, J Mao, E Chang, and B Guo, "Spoken dialogue management as planning and acting under uncertainty," in Proc Eurospeech, Aalborg, Denmark, 2001, pp. 2169-2172.
8. JD Williams, Partially Observable Markov Decison Processes for Spoken Dialogue Management, Ph.D. thesis, Cambridge University, 2006.
9. SJ Young, J Schatzmann, K Weilhammer, and H Ye, "The hidden information state approach to dialog management," in Proc ICASSP, Hawaii, 2007, pp. IV149-IV152.
10. TH Bui, M Poel, A Nijholt, and J Zwiers, "A tractable DDN-POMDP approach to affective dialogue modeling for general probabilistic frame-based dialgue systems," in Proc Workshop on Knowledge and Reasoning in Practical Dialogue Systems, IJCAI, Hyderabad, 2007, pp. 34-37.
11. Heriberto Cuayáhuitl, Steve Renals, Oliver Lemon, and Hiroshi Shimodaira, "Human-computer dialogue simulation using hidden markov models," in Proc ASRU, Puerto Rico, 2005, pp. 290-295.
12. J Schatzmann, K Georgila, and SJ Young, "Quantitative evaluation of user simulation techniques for spoken dialogue systems," in Proc SIGdial Workshop on Discourse and Dialogue, Lisbon, 2005, pp. 178-181.
13. E Levin and R Pieraccini, "Value-based optimal decision for dialog systems," in Proc SLT, Aruba, 2006, pp. 198-201.
14. MA Walker, CA Kamm, and DJ Litman, "Towards developing general models of usability with PARADISE," Natural Language Engineering: Special Issue on Best Practice in Spoken Dialogue Systems, 2000.
15. H Cramer, "On the composition of elementary errors. second paper: Statistical applications," Skandinavisk Aktuarietidskrift, vol. 11, pp. 171-180, 1928.
16. R von Mises, Wahrscheinlichkeitsrechnung und ihre Anwendung in der Statistik und Theoretischen Physik, F Deuticke, 1931.
17. TW Anderson, "On the distribution of the two-sample Cramér-von Mises criterion," Annals of Mathematical Statistics, vol. 33, no. 3, pp. 1148-1159, 1962.
18. MA Stephens, "Introduction to: Kolmogorov(1933) on the empirical determination of a distribution," in Breakthrougth in statistics, S Kotz and NL Johnson, Eds., vol. II, pp. 93-105. Springer Verlag, 1992.
19. WT Eadie, D Drijard, FE James, MGW Roos, and B Sadoulet, Statistical Methods in Experimental Physics, North Holland, 1971.