Hybrid reinforcement/supervised learning of dialogue policies from fixed data sets

Computational Linguistics

Volumn 34, Issue 4, 2008, Pages 487-511

  Henderson, James ^a   Lemon, Oliver ^b   Georgila, Kallirroi ^b  

^a UNIVERSITY OF GENEVA   (Switzerland)

^b UNIVERSITY OF EDINBURGH   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

REINFORCEMENT LEARNING; SPEECH PROCESSING; SUPERVISED LEARNING;

AUTOMATIC EVALUATION; AUTOMATIC OPTIMIZATION; COMMUNICATOR SYSTEMS; DIALOGUE MANAGEMENT; HYBRID REINFORCEMENTS; INFORMATION STATE; LEARNING DIALOGUES; REINFORCEMENT LEARNING MODELS;

LEARNING SYSTEMS;

EID: 51449120317     PISSN: 08912017     EISSN: 15309312     Source Type: Journal    
DOI: 10.1162/coli.2008.07-028-R2-05-82     Document Type: Article

References (50)

1. Bohlin, Peter, Robin Cooper, Elisabet Engdahl, and Staffan Larsson. 1999. Information states and dialog move engines. Electronic Transactions in AI, 3(9). Available at www. ep. liu. se/ej /etai/1999/D/.
2. Bos, Johan, Ewan Klein, Oliver Lemon, and Tetsushi Oka. 2003. DIPPER: Description and formalisation of an information-state update dialogue system architecture. In Proceedings of the 4th SIGdial Workshop on Discourse and Dialogue, pages 115-124, Sapporo.
3. Cheyer, Adam and David Martin. 2001. The open agent architecture. Journal of Autonomous Agents and Multi-Agent Systems, 4(1/2):143-148.
4. Denecke, Matthias, Kohji Dohsaka, and Mikio Nakano, 2005. Fast reinforcement learning of dialogue policies using stable function approximation. In K. Y. Su, J. Tsujii, J.-H. Lee, and O. Y. Kwong, Natural Language Processing, IJCNLP 2004. Springer, Berlin, pages 1-11.
5. Frampton, Matthew and Oliver Lemon. 2005. Reinforcement learning of dialogue strategies using the user's last dialogue act. In Proceedings of the 4th Workshop on Knowledge and Reasoning in Practical Dialog Systems, International Joint Conference on Artificial Intelligence (IJCAI), pages 83-90, Edinburgh.
6. Frampton, Matthew and Oliver Lemon. 2006. Learning more effective dialogue strategies using limited dialogue move features. In Proceedings of the 44th Meeting of the Association for Computational Linguistics, pages 185-192, Sydney.
7. Gabsdil Malte and Oliver Lemon. 2004. Combining acoustic and pragmatic features to predict recognition performance in spoken dialogue systems. In Proceedings of the 42nd Meeting of the Association for Computational Linguistics, pages 344-351, Barcelona.
8. Georgila, Kallirroi, James Henderson, and Oliver Lemon. 2005. Learning user simulations for Information State Update dialogue systems. In Proceedings of the 9th European Conference on Speech Communication and Technology (Interspeech - Eurospeech), pages 893-896, Lisbon.
9. Georgila, Kallirroi, James Henderson, and Oliver Lemon. 2006. User simulation for spoken dialogue systems: Learning and evaluation. In Proceedings of the 9th International Conference on Spoken Language Processing (Interspeech - ICSLP), pages 1065-1068, Pittsburgh, PA.
10. Georgila, Kallirroi, Oliver Lemon, and James Henderson. 2005. Automatic annotation of COMMUNICATOR dialogue data for learning dialogue strategies and user simulations. In Proceedings of the Ninth Workshop on the Semantics and Pragmatics of Dialogue (SEMDIAL), pages 61-68, Nancy.
11. Georgila, Kallirroi, Oliver Lemon, James Henderson, and Johanna Moore. (submitted). Automatic annotation of context and speech acts for dialogue corpora.
12. Goddeau, D. and J. Pineau. 2000. Fast reinforcement learning of dialog strategies. In Proceedings of the IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP), pages 11-1233-1236, Istanbul.
13. Henderson, James, Oliver Lemon, and Kallirroi Georgila. 2005. Hybrid reinforcement/supervised learning for dialogue policies from COMMUNICATOR data. In Proceedings of the 4th Workshop on Knowledge and Reasoning in Practical Dialog Systems, International Joint Conference on Artificial Intelligence (IJCAI), pages 68-75, Edinburgh.
14. Larsson, Staffan and David Traum. 2000. Information state and dialogue management in the TRINDI Dialogue Move Engine Toolkit. Natural Language Engineering, 6(3-4):323-340.
15. Lemon, Oliver, Kallirroi Georgila, and James Henderson. 2006. Evaluating effectiveness and portability of reinforcement learned dialogue strategies with real users: the TALK TownInfo evaluation. In Proceedings of the IEEE/ACL 2006 Workshop on Spoken Language Technology, pages 178-181, Aruba.
16. Lemon, Oliver, Kallirroi Georgila, James Henderson, Malte Gabsdil, Ivan Meza-Ruiz, and Steve Young. 2005. Integration of learning and adaptivity with the ISU approach. Technical Report D4.1, TALK Project.
17. Lemon, Oliver, Kallirroi Georgila, James Henderson, and Matthew Stuttle. 2006. An ISU dialogue system exhibiting reinforcement learning of dialogue policies: generic slot-filling in the TALK in-car system. In Proceedings of the Demonstrations of EACL, pages 119-122, Trento.
18. Lemon, Oliver, Kallirroi Georgila, and Matthew Stuttle. 2005. Showcase exhibiting reinforcement learning for dialogue strategies in the in-car domain. Technical Report D4.2, TALK Project.
19. Levin, Esther and Roberto Pieraccini. 1997. A stochastic model of computer-human interaction for learning dialogue strategies. In Proceedings of the 5th European Conference on Speech Communication and Technology (Interspeech - Eurospeech), pages 1883-1886, Rhodes.
20. Levin, Esther, Roberto Pieraccini, and Wieland Eckert. 2000. A stochastic model of human-machine interaction for learning dialog strategies. IEEE Transactions on Speech and Audio Processing, 8(1):11-23.
21. Pietquin, Olivier. 2004. A Framework for Unsupervised Learning of Dialogue Strategies. Presses Universitaires de Louvain, SIMILAR Collection.
22. Pietquin, Olivier and Thierry Dutoit. 2006a. Dynamic Bayesian networks for NLU simulation with application to dialog optimal strategy learning. In Proceedings of the IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP), pages 49-52, Toulouse.
23. Pietquin, Olivier and Thierry Dutoit. 2006b. A probabilistic framework for dialog simulation and optimal strategy learning. IEEE Transactions on Speech and Audio Processing, 14(2):589-599.
24. Rieser, Verena and Oliver Lemon. 2006a. Cluster-based user simulations for learning dialogue strategies and the SUPER evaluation metric. In Proceedings of the 9th International Conference on Spoken Language Processing (Interspeech - ICSLP), pages 1766-1769, Pittsburgh, PA.
25. Rieser, Verena and Oliver Lemon. 2006b. Using logistic regression to initialise reinforcement-learning-based dialogue systems. In Proceedings of the IEEE/ACL 2006 Workshop on Spoken Language Technology, pages 190-193, Aruba.
26. Rieser, Verena and Oliver Lemon. 2006c. Using machine learning to explore human multimodal clarification strategies. In Proceedings of the Poster Session of the 44th Meeting of the Association for Computational Linguistics, pages 659-666, Sydney.
27. Roy, Nicholas, Joelle Pineau, and Sebastian Thrun. 2000. Spoken dialog management for robots. In Proceedings of the 38th Meeting of the Association for Computational Linguistics, pages 93-100, Hong Kong.
28. Schatzmann, Jost, Kallirroi Georgila, and Steve Young. 2005. Quantitative evaluation of user simulation techniques for spoken dialogue systems. In Proceedings of the 6th SIGdial Workshop on Discourse and Dialogue, pages 45-54, Lisbon.
29. Schatzmann, Jost, Matthew N. Stuttle, Karl Weilhammer, and Steve Young. 2005. Effects of the user model on simulation-based learning of dialogue strategies. In Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop, pages 220-225, San Juan, Puerto Rico.
30. Schatzmann, Jost, Karl Weilhammer, Matthew N. Stuttle, and Steve Young. 2006. A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies. The Knowledge Engineering Review, 21:97-126.
31. Scheffler, Konrad and Steve Young. 2001. Corpus-based dialogue simulation for automatic strategy learning and evaluation. In Proceedings of the NAACL Workshop on Adaptation in Dialogue Systems, pages 64-70, Pittsburgh, PA.
32. Scheffler, Konrad and Steve Young. 2002. Automatic learning of dialogue strategy using dialogue simulation and reinforcement learning. In Proceedings of the Human Language Technology Conference, pages 12-19, San Diego, CA.
33. Shawe-Taylor, John and Nello Cristianini. 2004. Kernel Methods for Pattern Analysis. Cambridge University Press.
34. Singh, Satinder, Michael Kearns, Diane Litman, and Marilyn Walker. 2000a. Empirical evaluation of a reinforcement learning dialogue system. In Proceedings of the AAAI, pages 645-651, Whistler.
35. Singh, Satinder, Michael Kearns, Diane Litman, and Marilyn Walker. 2000b. Reinforcement learning for spoken dialogue systems. In Advances in Neural Information Processing Systems, 12:956-962.
36. Singh, Satinder, Diane Litman, Michael Kearns, and Marilyn Walker. 2002. Optimizing dialogue management with reinforcement learning: Experiments with the NJFun system. Journal of Artificial Intelligence Research (JAIR), 16:105-133.
37. Sutton, Richard and Andrew Barto. 1998. Reinforcement Learning. MIT Press, Cambridge, MA.
38. Walker, M., J. Aberdeen, J. Boland, E. Bratt, J. Garofolo, L. Hirschman, A. Le, S. Lee, S. Narayanan, K. Papineni, B. Pellom, B. Polifroni, A. Potamianos, P. Prabhu, A. Rudnicky, G. Sanders, S. Seneff, D. Stallard, and S. Whittaker. 2001. DARPA communicator dialog travel planning systems: The June 2000 data collection. In Proceedings of the 7th European Conference on Speech Communication and Technology (Interspeech - Eurospeech), pages 1371-1374, Aalborg.
39. Walker, M. and R. Passonneau. 2001. DATE: A dialogue act tagging scheme for evaluation of spoken dialogue systems. In Proceedings of the Human Language Technology Conference, pages 1-8, San Diego, CA.
40. Walker, M., A. Rudnicky, J. Aberdeen, E. Bratt, J. Garofolo, H. Hastie, A. Le, B. Pellom, A. Potamianos, R. Passonneau, R. Prasad, S. Roukos, G. Sanders, S. Seneff, D. Stallard, and S. Whittaker. 2002a. DARPA Communicator Evaluation: Progress from 2000 to 2001. In Proceedings of the 7th International Conference on Spoken Language Processing (Interspeech - ICSLP), pages 273-276, Denver, CO.
41. Walker, M., A. Rudnicky, R. Prasad, J. Aberdeen, E. Bratt, J. Garofolo, H. Hastie, A. Le, B. Pellom, A. Potamianos, R. Passonneau, S. Roukos, G. Sanders, S. Seneff, and D. Stallard. 2002b. DARPA Communicator: Cross-system results for the 2001 evaluation. In Proceedings of the 7th International Conference on Spoken Language Processing (Interspeech - ICSLP), pages 269-272, Denver, CO.
42. Walker, Marilyn A., Jeanne C. Fromer, and Shrikanth Narayanan. 1998. Learning optimal dialogue strategies: A case study of a spoken dialogue agent for email. In Proceedings of the 17th International Conference on Computational Linguistics, pages 1345-1351, Montreal.
43. Walker, Marilyn A., Candace A. Kamm, and Diane J. Litman. 2000. Towards developing general models of usability with PARADISE. Natural Language Engineering, 6(3):363-377.
44. Walker, Marilyn A., Rebecca J. Passonneau, and Julie E. Boland. 2001. Quantitative and qualitative evaluation of DARPA Communicator spoken dialogue systems. In Proceedings of the 39th Meeting of the Association for Computational Linguistics, pages 515-522, Toulouse.
45. Williams, Jason. 2007. A method for evaluating and comparing user simulations: The Cramer-von Mises divergence. In Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop, pages 508-513, Kyoto.
46. Williams, Jason, Pascal Poupart, and Steve Young. 2005a. Factored partially observable Markov decision processes for dialogue management. In Proceedings of the 4th Workshop on Knowledge and Reasoning in Practical Dialog Systems, International Joint Conference on Artificial Intelligence (IJCAI), pages 76-82, Edinburgh.
47. Williams, Jason, Pascal Poupart, and Steve Young. 2005b. Partially observable Markov decision processes with continuous observations for dialogue management. In Proceedings of the 6th SIGdial Workshop on Discourse and Dialogue, pages 25-34, Lisbon.
48. Williams, Jason and Steve Young. 2003. Using Wizard-of-Oz simulations to bootstrap reinforcement-learning-based dialog management systems. In Proceedings of the 4th SIGdial Workshop on Discourse and Dialogue, pages 135-139, Sapporo.
49. Williams, Jason and Steve Young. 2005. Scaling up POMDPs for dialog management: The "Summary POMDP" method. In Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop, pages 177-182, San Juan, Puerto Rico.
50. Young, Steve. 2000. Probabilistic methods in spoken dialogue systems. Philosophical Transactions of the Royal Society (Series A), 358(1769):1389-1402.