A comprehensive reinforcement learning framework for dialogue management optimization

IEEE Journal on Selected Topics in Signal Processing

Volumn 6, Issue 8, 2012, Pages 891-902

  Daubigney, Lucie ^a,b   Geist, Matthieu ^a   Chandramohan, Senthilkumar ^a,c   Pietquin, Olivier ^a,d  

^a UMI Georgia Tech CNRS 2958   (France)

^b INRIA   (France)

^c UNIVERSITY OF AVIGNON   (France)

^d CNRS   (France)

Author keywords

Dialogue management; Reinforcement learning; Spoken dialogue system

Indexed keywords

DIALOGUE MANAGEMENT; DIALOGUE STRATEGY; DIALOGUE SYSTEMS; GOAL-ORIENTED; INTERACTION STRATEGY; NON-STATIONARITIES; SPOKEN DIALOGUE SYSTEM; TEMPORAL DIFFERENCES;

ALGORITHMS; OPTIMIZATION; SPEECH PROCESSING;

REINFORCEMENT LEARNING;

EID: 84872138024     PISSN: 19324553     EISSN: None     Source Type: Journal    
DOI: 10.1109/JSTSP.2012.2229257     Document Type: Article

References (48)

1. E. Levin, R. Pieraccini, and W. Eckert, "Learning dialogue strategies within the Markov decision process framework," in Proc. Autom. Speech Recognit. Understand. Workshop (ASRU'97), 1997.
2. R. Bellman, "A Markovian Decision Process," J. Math. Mech., vol. 6, pp. 679-684, 1957.
3. R. S. Sutton and A. G. Barto, Reinforcement Learning: An introduction. Cambridge, MA: MIT Press, 1998.
4. E. Levin and R. Pieraccini, "Using Markov decision process for learning dialogue strategies," in Proc. Int. Conf. Accoust., Speech, Signal Process. (ICASSP'98), 1998, pp. 201-204.
5. E. Levin, R. Pieraccini, andW. Eckert, "A stochastic model of humanmachine interaction for learning dialog strategies," IEEE Trans. Speech Audio Process., vol. 8, no. 1, pp. 11-23, Jan. 2000. (Pubitemid 30540744)
6. W. Eckert, E. Levin, and R. Pieraccini, "User modeling for spoken dialogue system evaluation," in Proc. Autom. Speech Recognit. Understand. Workshop (ASRU'97), 1997.
7. O. Pietquin and T. Dutoit, "A probabilistic framework for dialog simulation and optimal strategy learning," IEEE Trans. Speech Audio Process., vol. 14, no. 2, pp. 589-599, Mar. 2006. (Pubitemid 46405357)
8. J. Schatzmann, K.Weilhammer, M. Stuttle, and S. Young, "A survey of statistical user simulation techniques for RL of dialogue management strategies," Knowl. Eng. Rev., vol. 21, no. 2, pp. 97-126, 2006. (Pubitemid 44266297)
9. J. Schatzmann, M. N. Stuttle, K. Weilhammer, and S. Young, "Effects of the user model on simulation-based learning of dialogue strategies," in Proc. Autom. Speech Recognit. Understand. Workshop (ASRU'05), 2005.
10. O. Pietquin and H. Hastie, "A survey on metrics for the evaluation of user simulations," Knowledge Eng. Rev., 2011.
11. V. Rieser and O. Lemon, Reinforcement learning for adaptive dialogue systems: A data-driven methodology for dialogue management and natural language generation. New York: Springer, 2011, Theory and Applications of Natural Language Processing.
12. S. Singh, M. Kearns, D. Litman, and M. Walker, "Reinforcement learning for spoken dialogue systems," in Proc. Adv. Neural Inf. Process. Syst. (NIPS'99), 1999.
13. R. Bellman, Dynamic Programming, 6th ed. New York: Dover, 1957.
14. L. Li, S. Balakrishnan, and J. Williams, "Reinforcement learning for dialog management using least-squares policy iteration and fast feature selection," in Proc. Interspeech'09, 2009.
15. O. Pietquin, M. Geist, S. Chandramohan, and H. Frezza-Buet, "Sample-efficient batch reinforcement learning for dialogue management optimization," ACM Trans. Speech Audio Process., vol. 7, no. 3, pp. 1-21, 2011.
16. J. Henderson, O. Lemon, and K. Georgila, "Hybrid reinforcement/supervised learning of dialogue policies from fixed data sets," Comput. Linguist., vol. 34, no. 4, pp. 487-511, 2008.
17. M. Geist and O. Pietquin, "Managing uncertainty within the KTD framework," in Proc. AL&E Workshop, 2011, Journal of Machine Learning Research C& WP
18. M. Gašić, F. Jurčíček, S. Keizer, F.Mairesse, B. Thomson, K. Yu, and S. Young, "Gaussian processes for fast policy optimisation of POMDPbased dialogue managers," in Proc. SIGdial'10, 2010.
19. L.Daubigney,M. Geist, andO. Pietquin, "Off-policy learning in largescale POMDP-based dialogue systems," in Proc. Int. Conf. Acoust., Speech Signal Process. (ICASSP'12), 2012, pp. 4989-4992.
20. F. Jurčíček, B. Thomson, S. Keizer, M. Gašić, F.Mairesse, K.Yu, and S. Young, "Natural belief-critic: A reinforcement algorithm for parameter estimation in statistical spoken dialogue systems," in Proc. Interspeech' 10, 2010.
21. M. Geist and O. Pietquin, "Kalman temporal differences," J. Artif. Intell. Res. (JAIR), vol. 39, pp. 483-532, 2010.
22. J. Bos, E. Klein, O. Lemon, and T. Oka, "DIPPER: Description and formalisation of an information-state update dialogue system architecture," in Proc. SIGdial'03, 2003.
23. S. Young, J. Schatzmann, B. Thomson, H. Ye, and K. Weilhammer, "The HIS dialogue manager," in Proc. IEEE/ACL Workshop Spoken Lang. Technol. (SLT'06), 2006.
24. J. Tsitsiklis and B. Van Roy, "An analysis of temporal-difference learning with function approximation," IEEE Trans. Autom. Control, vol. 42, no. 5, pp. 674-690, May 1997. (Pubitemid 127760263)
25. S. Png and J. Pineau, "Bayesian reinforcement learning for POMDPbased dialogue systems," in Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP'11), 2011, pp. 2156-2159.
26. S. Larsson and D. R. Traum, "Information state and dialogue management in the trindi dialogue move engine toolkit," Natural Lang. Eng., vol. 6, pp. 323-340, 2000.
27. J. Williams and S. Young, "Scaling up POMDPs for dialogue management: The summary POMDP method," in Proc. Autom. Speech Recognit. Understanding Workshop (ASRU'05), 2005.
28. M. Walker, D. Litman, C. Kamm, and A. Abella, "PARADISE: A framework for evaluating spoken dialogue agents," in Proc. Meeting Assoc. Comput. Linguist. (ACL'97), 1997.
29. A. Boularias, H. Chinaei, and B. Chaib-draa, "Learning the reward model of dialogue POMDPs from data," in Proc. NIPS Workshop of Mach. Learn. for Assistive Tech., 2010.
30. L. E. Asri, R. Laroche, and O. Pietquin, "Reward function learning for dialogue management," in Proc. Starting Artif. Intell. Res. Symp. (STAIRS'12), 2012, pp. 95-106.
31. R. Kalman, "A new approach to linear filtering and prediction problems," Trans. ASME-J. Basic Eng., vol. 82, no. Series D, pp. 35-45, 1960.
32. E. Wan and R. Van Der Merwe, "The unscented Kalman filter for nonlinear estimation," in Adaptive Syst. for Signal Process., Commun., Control Symp. (AS-SPCC'00), 2000, pp. 153-158.
33. L. Daubigney, M. Gašić, S. Chandramohan, M. Geist, O. Pietquin, and S. Young, "Uncertainty management for on-line optimisation of a POMDP-based large-scale spoken dialogue system," in Proc. Interspeech' 11, 2011.
34. K. Georgila, J. Henderson, and O. Lemon, "Learning user simulations for information state update dialogue systems," in Proc. Eur. Conf. Speech Commun. Technol. (Interspeech - Eurospeech'05), 2005.
35. V. Rieser, S. Keizer, X. Liu, and O. Lemon, "Adaptive information presentation for spoken dialogue systems: Evaluation with human subjects," in Proc. Eur. Workshop Natural Lang. Generat. (ENLG'11), 2011.
36. J. Schatzmann, B. Thomson, K. Weilhammer, H. Ye, and S. Young, "Agenda-based user simulation for bootstrapping a POMDP dialogue system," in Proc. NAACL Conf. Human Lang. Technol. (HLT/NAACL'07), 2007.
37. F. Jurčíček, S. Keizer, M. Gašić, F. Mairesse, B. Thomson, K. Yu, and S. Young, "Real user evaluation of spoken dialogue systems using Amazon Mechanical Turk," in Proc. Interspeech'11, 2011.
38. M. Gašić, F. Jurčíček, B. Thomson, K. Yu, and S. Young, "On-line policy optimisation of spoken dialogue systems via live interaction with human subjects," in Proc. Autom. Speech Recognit. Understand. Workshop (ASRU'11), 2011, pp. 312-317.
39. S. Young, M. Gašić, S. Keizer, F. Mairesse, J. Schatzmann, B. Thomson, and K. Yu, "The hidden information state model: A practical framework for POMDP-based spoken dialogue management," Comput. Speech Lang., vol. 24, no. 2, pp. 150-174, 2010.
40. G.Gordon, "Stable function approximation in dynamic programming," in Proc. Int. Conf. Mach. Learn. (ICML'95), 1995.
41. M. G. Lagoudakis and R. Parr, "Least-squares policy iteration," J. Mach. Learn. Res. (JMLR), vol. 4, pp. 1107-1149, 2003.
42. L. Li, M. L. Littman, and C. R. Mansley, "Online exploration in leastsquares policy iteration," in Proc. Int. Conf. Autonom. Agents Multiagent Syst. (AAMAS'09), 2009, vol. 2, pp. 733-739.
43. Y. Engel, S. Mannor, and R. Meir, "Reinforcement learning with Gaussian processes," in Proc. Int. Conf. Mach. Learn. (ICML'05), 2005.
44. Y. Engel, S. Mannor, and R. Meir, "Sparse online greedy support vector regression," in Proc. Eur. Conf. Mach. Learn. (ECML'02, 2002, vol. 2430, pp. 84-96.
45. O. Lemon,K. Georgila, J.Henderson, and M. Stuttle, "An ISU dialogue system exhibiting reinforcement learning of dialogue policies: Generic slot-filling in the TALK in-car system," in Proc. Conf. Eur. Chapter Assoc. for Comput. Linguist. (EACL'06), 2006, pp. 119-122.
46. O. Pietquin,M. Geist, and S. Chandramohan, "Sample efficient on-line learning of optimal dialogue policies with Kalman temporal differences," in Proc. Int. Joint Conf. Artif. Intell. (IJCAI'11), 2011.
47. G. Tesauro, "Temporal difference learning and TD-Gammon," Commun. Assoc. for Comput. Mach. (ACM), vol. 38, no. 3, pp. 58-68, 1995.
48. M. Geist and O. Pietquin, "Statistically linearized least-squares temporal differences," in Proc. IEEE Int. Conf. UltraModern Control Syst (ICUMT'10), 2010, pp. 450-457.