Reinforcement learning for parameter estimation in statistical spoken dialogue systems

Computer Speech and Language

Volumn 26, Issue 3, 2012, Pages 168-192

  Jurčíček, Filip ^a   Thomson, Blaise ^a   Young, Steve ^a  

^a UNIVERSITY OF CAMBRIDGE   (United Kingdom)

Author keywords

Dialogue management; POMDP; Reinforcement learning; Spoken dialogue systems

Indexed keywords

DESIGN CONSIDERATIONS; DIALOGUE MANAGEMENT; DIALOGUE MODELS; DIALOGUE SYSTEMS; INFORMATION DOMAINS; MODEL PARAMETERS; PARTIALLY OBSERVABLE MARKOV DECISION PROCESS; POMDP; REINFORCEMENT ALGORITHMS; REINFORCEMENT TECHNIQUE; REWARD FUNCTION; SPOKEN DIALOGUE SYSTEM;

BEHAVIORAL RESEARCH; LEARNING ALGORITHMS; MARKOV PROCESSES; MATHEMATICAL MODELS; REINFORCEMENT; REINFORCEMENT LEARNING; SPEECH PROCESSING;

PARAMETER ESTIMATION;

EID: 84855300452     PISSN: 08852308     EISSN: 10958363     Source Type: Journal    
DOI: 10.1016/j.csl.2011.09.004     Document Type: Article

References (40)

1. Aberdeen, D.A., 2003. Policy-gradient algorithms for partially observable Markov decision processes. Ph.D. Thesis, Australian National University.
2. Amari S. Natural gradient works efficiently in learning Neural Computation 10 2 1998 251 276 (Pubitemid 128463152)
3. Bishop C. Pattern Recognition and Machine Learning 2006 Springer
4. Black A., and Lenzo K. Flite: A Small, Fast Speech Synthesis Engine 2008 http://www.speech.cs.cmu.edu/flite/index.html
5. Doshi F., and Roy N. Efficient model learning for dialog management HRI'07: Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction 2007 ACM New York, NY, USA 65 72 http://doi.acm.org/10.1145/1228716. 1228726 (Pubitemid 47327128)
6. Doshi F., and Roy N. Spoken language interaction with model uncertainty: an adaptive human robot interaction system Connection Science 20 4 2008 299 318 10.1080/09540090802413145
7. Doshi-Velez F. The infinite partially observable Markov decision process Bengio Y. Schuurmans D. Lafferty J. Williams C.K.I. Culotta A. Advances in Neural Information Processing Systems, vol. 22 2009 477 485
8. Dybkjaer L., Bernsen N.O., and Minker W. Evaluation and usability of multimodal spoken language dialogue systems Speech Communication 43 1-2 2004 33 54 10.1016/j.specom.2004.02.001
9. Georgila K., Lemon O., and Henderson J. Automatic annotation of COMMUNICATOR dialogue data for learning dialogue strategies and user simulations Ninth Workshop on the Semantics and Pragmatics of Dialogue 2005
10. Hoerl A.E., and Kennard R.W. Ridge regression: biased estimation for nonorthogonal problems Technometrics 12 1970 55 67
11. Jurčíček F., Keizer S., Gašić M., Mairesse F., Thomson B., Yu K., and Young S. Real user evaluation of spoken dialogue systems using Amazon Mechanical Turk Proc. Interspeech 2011
12. Jurčíček F., Thomson B., Keizer S., Mairesse F., Gašić M., Yu K., and Young S. Natural Belief-Critic: a reinforcement algorithm for parameter estimation in statistical spoken dialogue systems Kobayashi T. Hirose K. Nakamura S. Proc. Interspeech. ISCA 2010 90 93
13. Jurčíček F., Thomson B., and Young S. Natural actor and belief critic: reinforcement algorithm for learning parameters of dialogue systems modelled as pomdps ACM Transactions on Speech and Language Processing 7 June 2011 6:1-6:26. http://doi.acm.org/10.1145/1966407.1966411
14. Kim D., Sim H.S., Kim K., Kim J.H., Kim H., and Sung J.W. Effects of user modeling on POMDP-based dialogue systems Proceedings of Interspeech 2008
15. Konda V., and Tsitsiklis J. Actor-critic algorithms Advances in Neural Information Processing Systems 12 2000 1008 1014
16. Peters J., Vijayakumar S., and Schaal S. Natural Actor-Critic European Conference on Machine Learning (ECML) 2005 Springer 280 291
17. Png S., and Pineau J. Bayesian reinforcement learning for pomdp-based dialogue systems ICASSP'11: International Conference on Acoustics, Speech and Signal Processing 2011
18. Roy N., Pineau J., and Thrun S. Spoken dialogue management using probabilistic reasoning ACL'00: Proceedings of the 38th Annual Meeting on Association for Computational Linguistics 2000 Association for Computational Linguistics Morristown, NJ, USA 93 100 http://dx.doi.org/10.3115/1075218.1075231
19. Schatzmann, J., 2008. Statistical user modeling for dialogue systems. Ph.D. Thesis, University of Cambridge.
20. Schatzmann J., Stuttle M.N., Weilhammer K., and Young S. Effects of the user model on simulation-based learning of dialogue strategies IEEE ASRU'05: Proc. IEEE Workshop Automatic Speech Recognition and Understanding 2005
21. Sutton R., and Barto A. Reinforcement Learning: An Introduction. Adaptive Computation and Machine Learning 1998 MIT Press Cambridge, MA
22. Sutton R., McAllester D., Singh S., and Mansour Y. Policy gradient methods for reinforcement learning with function approximation Advances in Neural Information Processing Systems, vol. 12 2000 MIT Press 1057 1063
23. Syed U., and Williams J.D. Using automatically transcribed dialogs to learn user models in a spoken dialog system HLT Morristown, USA 2008 121 124
24. Thomson, B., 2010. Statistical methods for spoken dialogue management. Ph.D. Thesis, University of Cambridge.
25. Thomson B., Gašić M., Keizer S., Mairesse F., Schatzmann J., Yu K., and Young S. User study of the Bayesian Update of Dialogue State approach to dialogue management. Interspeech 2008 Brisbane, Australia 2008
26. Thomson B., Jurčíček F., Gašić M., Keizer S., Mairesse F., Yu K., and Young S. Parameter learning for POMDP spoken dialogue models IEEE SLT '10: Spoken Language Technology Workshop 2010 271 276 10.1109/SLT.2010.5700863
27. Thomson B., Schatzmann J., and Young S. Bayesian update of dialogue state for robust dialogue systems Int Conf Acoustics Speech and Signal Processing ICASSP Las Vegas 2008
28. Thomson B., and Young S. Bayesian update of dialogue state: a POMDP framework for spoken dialogue systems Computer Speech and Language 24 4 2010 562 588
29. Walker M.A., Litman D.J., Kamm C.A., and Abella A. Evaluating interactive dialogue systems: extending component evaluation to integrated system evaluation Interactive Spoken Dialog Systems on Bringing Speech and NLP Together in Real Applications. ISDS'97. Association for Computational Linguistics Stroudsburg, PA, USA 1997 1 8
30. Ward W. Understanding spontaneous speech Proc. Int. Conf. Acoustics, Speech and Signal Processing Toronto, Canada 1991 365 368
31. Wierstra D., Förster A., Peters J., and Schmidhuber J. Recurrent policy gradients Logic Journal of the IGPL 18 5 2010 620 634
32. Williams J.D. Integrating expert knowledge into POMDP optimization for spoken dialog systems Proceedings of the AAAI-08 Workshop on Advancements in POMDP Solvers 2008
33. Williams J.D., and Young S. Scaling up POMDPs for dialog management: the summary POMDP method. IEEE ASRU'05: Proc. IEEE Workshop Automatic Speech Recognition and Understanding Cancun, Mexico November 2005
34. Williams J.D., and Young S. Partially observable Markov decision processes for spoken dialog systems Computer Speech and Language 21 2 2007 393 422 (Pubitemid 44709839)
35. Williams J.D., and Young S. Scaling POMDPs for spoken dialog management IEEE Audio, Speech and Language Processing 15 7 2007 2116 2129
36. Williams R.J. Simple statistical gradient-following algorithms for connectionist reinforcement learning Machine Learning 8 1992 229 256
37. Young S. Talking to Machines (Statistically speaking) Int. Conf. Spoken Language Processing Denver, CO 2002
38. Young S. ATK: An Application Toolkit for HTK 2005 http://mi.eng.cam.ac. uk/research/dialogue/atk-home
39. Young, S., 2007. CUED Standard Dialogue Acts. Tech. rep., Cambridge University Engineering Dept. http://mi.eng.cam.ac.uk/research/dialogue/ LocalDocs/dastd.pdf.
40. Young S., Gašić M., Keizer S., Mairesse F., Schatzmann J., Thomson B., and Yu K. The Hidden Information State Model: a practical framework for POMDP-based spoken dialogue management Computer Speech and Language 24 2 2010 150 174