Assessing dialog system user simulation evaluation measures using human judges

ACL-08: HLT - 46th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference

Volumn , Issue , 2008, Pages 622-629

  Ai, Hua ^a   Litman, Diane J ^a  

^a UNIVERSITY OF PITTSBURGH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DIALOG SYSTEMS; EVALUATION MEASURES; GOLD STANDARDS; HUMAN JUDGMENTS; PREDICTION MODEL; RANKING MODEL; REGRESSION MODEL; SIMULATION MODEL; USER SIMULATION;

COMPUTER SIMULATION; MATHEMATICAL MODELS; REGRESSION ANALYSIS;

COMPUTATIONAL LINGUISTICS;

EID: 79959839791     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (13)

1. H. Ai and D. Litman. 2007. Knowledge Consistent User Simulations for Dialog Systems. In Proc. of Interspeech 2007.
2. G. Chung. 2004. Developing a Flexible Spoken Dialog System Using Simulation. In Proc. of ACL 04.
3. Y. Freund, R. Iyer, R.E. Schapire, and Y. Singer. 2003. An Efficient Boosting Algorithm for Combining Preferences. Journal of Machine Learning Research.
4. H. P. Grice 1975. Logic and Conversation. Syntax and Semantics III: Speech Acts, 41-58.
5. E. Levin, R. Pieraccini, and W. Eckert. 2000. A Stochastic Model of Human-Machine Interaction For learning Dialog Strategies. IEEE Trans. On Speech and Audio Processing, 8(1):11-23.
6. Linguistic Data Consortium. 2005. Linguistic Data Annotation Specification: Assessment of Fluency and Adequacy in Translations.
7. F. Mairesse, M. Walker, M. Mehl and R. Moore. 2007. Using Linguistic Cues for the Automatic Recognition of Personality in Conversation and Text. Journal of Artificial Intelligence Research, Vol 30, pp 457-501.
8. K.A. Papineni, S. Roukos, R.T. Ward, and W-J. Zhu. 2002. Bleu: A Method for Automatic Evaluation of Machine Translation. In Proc. of 40th ACL.
9. J. Schatzmann, K. Georgila, and S. Young. 2005. Quantitative Evaluation of User Simulation Techniques for Spoken Dialog Systems. In Proc. of 6th SIGdial.
10. K. Scheffler. 2002. Automatic Design of Spoken Dialog Systems. Ph.D. diss., Cambridge University.
11. J. D.Williams. 2007. A Method for Evaluating and Comparing User Simulations: The Cramer-von Mises Divergence. Proc IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU).
12. M. Walker, D. Litman, C. Kamm, and A. Abella. 1997. PARADISE: A Framework for Evaluating Spoken Dialog Agents. In Proc. of ACL 97.
13. M. Walker, O. Rambow, and M. Rogati. 2001. SPoT: A Trainable Sentence Planner. In Proc. of NAACL 01.