A Reinforcement Learning approach to evaluating state representations in spoken dialogue systems

Speech Communication

Volumn 50, Issue 8-9, 2008, Pages 683-696

  Tetreault, Joel R ^a   Litman, Diane J ^b  

^a EDUCATIONAL TESTING SERVICE   (United States)

^b University of Pittsburgh   (United States)

Author keywords

Adaptive systems; Affect; Evaluation; Feature selection; Machine learning; Markov decision processes; Reinforcement Learning; Spoken dialogue systems; Tutoring systems

Indexed keywords

LEARNING SYSTEMS; MANAGERS; REINFORCEMENT; REINFORCEMENT LEARNING; SPEECH; SPEECH PROCESSING; STATE SPACE METHODS;

BEST STATE; DIALOGUE CONTROL; DIALOGUE MANAGER; DIALOGUE SYSTEMS; ELSEVIER (CO); END TO END (ETE); HUMAN COMPUTER; METRICS (CO); REINFORCEMENT LEARNING (RL) MODEL; REINFORCEMENT LEARNING APPROACH; RELATIVE UTILITY; SPOKEN DIALOGUE SYSTEMS (SDS); SPOKEN DIALOGUES; STATE MODELING; TUTORING SYSTEMS; USER SATISFACTION;

HUMAN COMPUTER INTERACTION;

EID: 47349097051     PISSN: 01676393     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.specom.2008.05.002     Document Type: Article

References (46)

1. Ai, H., Litman, D., 2007. Knowledge Consistent user simulations for dialog systems (Interspeech).
2. Ai, H., Litman, D., Forbes-Riley, K., Rotaru, M., Tetreault, J., Purandare, A., 2006. Using system and user performance features to improve emotion detection in spoken tutoring dialogs (Interspeech).
3. Ammicht, E., Potamianos, A., Fosler-Lussier, E., 2001. Ambiguity representation and resolution in spoken dialogue systems (Eurospeech).
4. Ang, J., Dhillon, R., Krupski, A., Shriberg, E., and Stolcke, A., 2002. Prosody-based automatic detection of annoyance and frustration in human-computer dialog (Interspeech).
5. Bhatt K., Evens M., and Argamon S. Hedged responses and expressions of affect in human/human and human computer tutorial interactions. Cognit. Sci. (2004)
6. Byron, D., 2002. Resolving pronomial reference to abstract entities. In: 40th Annual Meeting of the Association for Computational Linguistics, pp. 80-87.
7. Core, M., Schubert, L., 1996. Dialog Parsing in the TRAINS System, University of Rochester Technical Report 612.
8. Danieli, M., Gerbino, E., 1995. Metrics for evaluating dialogue strategies in a spoken language system. In: AAAI Spring Symposium on Empirical Methods in Discourse Interpretation and Generation, pp. 34-39.
9. Dumais, S., Platt, J., Heckerman, D., Sahami. M., 1998. Inductive learning algorithms and representations for text categorization. In: Conf. on Information and Knowledge Management, pp. 148-155.
10. Forbes-Riley, K., Litman, D., 2005. Using bigrams to identify relationships between student certainness states and tutor responses in a spoken dialogue corpus. SIGDial.
11. Forbes-Riley K., Litman D., Huettner A., and Ward A. Dialogue-learning correlations in spoken dialogue tutoring. Artif. Intell. Educat. (2005)
12. Frampton, M., 2006. Learning more effective dialogue strategies using limited dialogue move features. COLING/ACL, pp. 185-192.
13. Frampton, M., Lemon, O., 2005. Reinforcement learning of dialogue strategies using the user's last dialogue act. In: IJCAI Workshop on K&R in Practical Dialogue Systems, pp. 62-67.
14. Georgila, K., Henderson, J., Lemon, O., 2005. Learning user simulations for information state update dialogue systems, pp. 893-896 (Interspeech).
15. Henderson, J., Lemon, O., Georgila, K., 2005. Hybrid reinforcement/supervised learning for dialogue policies from COMMUNICATOR data. In: IJCAI Workshop on K&R in Practical Dialogue Systems, pp. 68-75.
16. Hirschman, L., Pao, C., 1993. The cost of errors in a spoken language system. In: 3rd European Conf. on Speech Communication and Technology, pp. 1419-1422.
17. Hirschman, L., Dahl, D., McKay, D., Norton, L., Linebarger, M., 1990. Beyond class A: a proposal for automatic evaluation of discourse. In: Speech and Natural Language Workshop, pp. 109-113.
18. Jaulmes, R., Pineau, J., Precup, D., 2005. Active learning in partially observable markov decision processes. In: European Conf. on Machine Learning.
19. Jelinek F. Statistical Methods for Speech Recognition (1998), MIT Press
20. Levin, E., Pieraccini, R., 1997. A stochastic model of computer-human interaction for learning dialogues (EUROSPEECH).
21. Levin E., Pieraccini R., and Eckert W. A stochastic model of human-machine interaction for learning dialog strategies. IEEE Trans. Speech Audio Process. 8 (2000) 11-23
22. Liscombe, J., Hirschberg, J., Venditti, J., 2005. Detecting certainness in spoken tutorial dialogues (Interspeech).
23. Litman, D., Silliman, S., 2004. ITSPOKE: an intelligent tutoring spoken dialogue system, HLT/NAACL.
24. Litman, D., Kearns, M., Singh, S., Walker, M., 2000. Automatic optimization of dialogue management. In: Proc. 18th Internat. Conf. on Computational Linguistics (COLING-2000).
25. Nicholas, G., Rotaru, M., Litman, D., 2006. Exploiting word-level features for emotion prediction. In: IEEE/ACL Workshop on Spoken Language Technology.
26. Paek, T., Chickering, D., 2005. The Markov assumption in spoken dialogue management. In: 6th SIGDial Workshop on Discourse and Dialogue.
27. Polifroni, J., Hirschman, L., Seneff, S., Zue, V., 1992. Experiments in evaluating interactive spoken language systems. In: DARPA Speech and NL Workshop, pp. 28-31.
28. Rieser, V., Lemon, O., 2006. Using logistic regression to initialise reinforcement-learning dialogue systems. In: IEEE/ACL Workshop on Spoken Language Technology (SLT).
29. Rivals I., and Personnaz L. Construction of confidence intervals for neural networks based on least squares estimation. Neural Networks 15 1 (2002) 143-145
30. Roy, N., Pineau, J., Thrum, S., 2002. Spoken dialogue management using probabilistic reasoning. ACL.
31. Schapire, R., 2002. The boosting approach to machine learning: an overview. In: MSRI Workshop on Nonlinear Estimation and Classification.
32. Scheffler, K., Young, S., 2002. Automatic learning of dialogue strategy using dialogue simulation and reinforcement learning. HLT.
33. Singh, S., Kearns, M., Litman, D., Walker, M., 1999. Reinforcement learning for spoken dialogue systems. NIPS.
34. Singh S., Litman D., Kearns M., and Walker M. Optimizing dialogue management with reinforcement learning: experiments with the NJFun system. JAIR 16 (2002) 105-133
35. Sutton R., and Barto A. Reinforcement Learning (1998), The MIT Press
36. Tetreault, J., Litman, D., 2006. Using reinforcement learning to build a better model of dialogue state. EACL.
37. Tetreault, J., Litman, D., 2006. Comparing the utility of state features in spoken dialogue using reinforcement learning. NAACL.
38. Tetreault, J., Bohus, D., Litman, D., 2007. Estimating the reliability of MDP policies: a confidence interval approach. NAACL.
39. VanLehn K., Jordan P., Rosé C., Bhembe D., Bottner M., Gaydos A., Makatchev M., Pappuswamy U., Ringenberg M., Roque A., Siler S., Srivastava R., and Wilson R. The architecture of Why2-Atlas a coach for qualitative physics essay writing. Intell. Tutoring Systems (2002)
40. Walker M. An application of reinforcement learning to dialogue strategy selection in a spoken dialogue system for Email. JAIR (2000)
41. Walker M., Fromer, J., Narayanan, S., 1998. Learning optimal dialogue strategies: a case study of a spoken dialogue agent for Email. ACL/COLING.
42. Walker M., Litman D., Kamm C., and Abella A. Towards developing general models of usability with PARADISE. Nat. Lang. Eng. (2000)
43. Warnestal, P., 2005. User evaluation of a conversational recommender system. In: IJCAI Workshop on K&R in Practical Dialogue Systems, pp. 62-67.
44. Williams J., and Young S. Partially observable Markov decision processes for spoken dialog systems. Comput. Speech Lang. 24 (2006)
45. Williams, J., Poupart, P., Young, S., 2005a. Partially observable markov decision processes with continuous observations for dialogue management. SIGDial.
46. Williams, J., Poupart, P., Young, S., 2005b. Factored partially observable Markov decision processes for dialogue management. In: IJCAI Workshop on K&R in Practical Dialogue Systems, pp. 76-82.