Natural actor and belief critic: Reinforcement algorithm for learning parameters of dialogue systems modelled as POMDPs

ACM Transactions on Speech and Language Processing

Volumn 7, Issue 3, 2011, Pages

  Jurčíček, Filip ^a   Thomson, Blaise ^a   Young, Steve ^a  

^a UNIVERSITY OF CAMBRIDGE   (United Kingdom)

Author keywords

POMDP; Reinforcement learning; Spoken dialogue systems

Indexed keywords

DIALOGUE MANAGER; DIALOGUE MODELS; DIALOGUE SYSTEMS; INFORMATION DOMAINS; LEARNING PARAMETERS; MAIN COMPONENT; MODEL PARAMETERS; NATURAL GRADIENT; NOVEL ALGORITHM; OPTIMAL MODEL; OPTIMAL POLICIES; OPTIMIZATION TECHNIQUES; PARTIALLY OBSERVABLE MARKOV DECISION PROCESS; POLICY GRADIENT METHODS; POMDP; PRIOR DISTRIBUTION; RANDOM SEARCH ALGORITHM; REINFORCEMENT ALGORITHMS; REWARD FUNCTION; SPOKEN DIALOGUE SYSTEM; STATE INFORMATION;

GRADIENT METHODS; LEARNING ALGORITHMS; MARKOV PROCESSES; OPTIMIZATION; SPEECH PROCESSING;

PARAMETER ESTIMATION;

EID: 80052051092     PISSN: 15504875     EISSN: 15504883     Source Type: Journal    
DOI: 10.1145/1966407.1966411     Document Type: Article

References (39)

1. ABERDEEN, D. A. 2003. Policy-gradient algorithms for partially observable Markov decision processes. Ph.D. thesis, Australian National University.
2. AMARI, S. 1998. Natural gradient works efficiently in learning. Neural Comput. 10, 2, 251-276. (Pubitemid 128463152)
3. BISHOP, C. 2006. Pattern Recognition and Machine Learning. Springer.
4. BUI, T. H., POEL, M., NIJHOLT, A., AND ZWIERS, J. 2009. A tractable hybrid DDN-POMDP approach to affective dialogue modeling for probabilistic frame-based dialogue systems. Natural Lang. Engin. 15, 2, 273-307.
5. DOSHI, F. AND ROY, N. 2007. Efficient model learning for dialog management. In Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (HRI'07). ACM, New York, NY, 65-72. (Pubitemid 47327128)
6. ENGEL, Y.,MANNOR, S., AND MEIR, R. 2005. Reinforcement learning with Gaussian Processes. In Proceedings of the 22nd International Conference on Machine Learning.
7. GEIST, M., PIETQUIN, O., AND FRICOUT, G. 2009. Kalman Temporal Differences: The deterministic case. In Proceedings of the IEEE International Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL'09). 185-192.
8. GEORGILA, K., LEMON, O., AND HENDERSON, J. 2005. Automatic annotation of COMMUNICATOR dialogue data for learnin dialogue strategies and user simulations. In Proceedings of the 9thWorkshop on the Semantics and Pragmatics of Dialogue.
9. HANSEN,N. ANDOSTERMEIER, A. 2001. Completely derandomized self-adaptation in evolution strategies. Evolut. Computat. 9, 2, 159-195.
10. HOERL, A. E. AND KENNARD, R. W. 1970. Ridge Regression: Biased Estimation for Nonorthogonal Problems. Technometrics 12, 55-67.
11. JURČÍČEK, F., THOMSON, B., KEIZER, S., GAŠIĆ, M., MAIRESSE, F., YU, K., AND YOUNG, S. 2010. Natural Belief-Critic: a reinforcement algorithm for parameter estimation in statistical spoken dialogue systems. In Proceedings of Interspeech.
12. KAELBLING, L. P., LITTMAN, M. L., AND CASSANDRA, A. R. 1998. Planning and Acting in Partially Observable Stochastic Domains. Art. Intell. 101, 99-134. (Pubitemid 128387390)
13. KAPADIA, S. 1998. Discriminative training of hidden Markov models. Ph.D. dissertation. Cambridge University.
14. KIM, D.,SIM, H. S.,KIM, K.,KIM, J. H.,KIM, H., AND SUNG, J.W. 2008. Effects of User Modeling on POMDP-based Dialogue Systems. In Proceedings of Interspeech.
15. KONDA, V. AND TSITSIKLIS, J. 2000. Actor-critic algorithms. In Advances in Neural Information Processing Systems 12. MIT Press.
16. OLIPHANT, T. E. 2007. Python for scientific computing. Comput. Sci. Engin. 9, 3, 10-20. (Pubitemid 46646860)
17. PETERS, J. AND SCHAAL, S. 2008a. Natural actor-critic. Neurocomput. 71, 7-9, 1180-1190.
18. PETERS, J. AND SCHAAL, S. 2008b. Reinforcement learning of motor skills with policy gradients. Neural Netw. 21, 4, 682-697.
19. PETERS, J., VIJAYAKUMAR, S., AND SCHAAL, S. 2003. Reinforcement Learning for Humanoid Robotics. In Proceedings of the 3rd IEEE-RAS International Conference on Humanoid Robots.
20. PETERS, J., VIJAYAKUMAR, S., AND SCHAAL, S. 2005. Natural Actor-Critic. In Proceedings of the European Conference on Machine Learning (ECML). Springer, 280-291.
21. RABINER, L. R. 1989. A tutorial on hidden Markov models and selected applications in speech recognition. In Proc. IEEE. 257-286.
22. RAUX, A., LANGNER, B., BOHUS, D., BLACK, A. W., AND ESKENAZI, M. 2005. Lets go public! taking a spoken dialog system to the real world. In Proceedings of Interspeech.
23. ROY, N., PINEAU, J., AND THRUN, S. 2000. Spoken dialogue management using probabilistic reasoning. In Proceedings of the 38th Annual Meeting on Association for Computational Linguistics (ACL'00). Association for Computational Linguistics, Morristown, NJ, 93-100.
24. SCHATZMANN, J., STUTTLE, M. N.,WEILHAMMER, K., AND YOUNG, S. 2005. Effects of the user model on simulationbased learning of dialogue strategies. In Proceedings of the IEEE Workshop on Automatic Speech Recognition and Understanding.
25. SPALL, J. C. 2003. Introduction to Stochastic Search and Optimization. John Wiley & Sons, Inc., New York, NY.
26. SUTTON, R. AND BARTO, A. 1998. Reinforcement Learning: An Introduction. Adaptive Computation and Machine Learning. MIT Press, Cambridge, MA.
27. SUTTON, R., MCALLESTER, D., SINGH, S., AND MANSOUR, Y. 2000. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems 12, MIT Press, 1057-1063.
28. SYED, U. AND WILLIAMS, J. D. 2008. Using automatically transcribed dialogs to learn user models in a spoken dialog system. In Proceedings of the Annual Meeting of the Association for Computational Linguistics on Human Language Technologies. 121-124.
29. THOMSON, B. 2010. Statistical methods for spoken dialogue management. Ph.D. thesis, University of Cambridge.
30. THOMSON, B. AND YOUNG, S. 2010. Bayesian update of dialogue state: A POMDP framework for spoken dialogue systems. Comput. Speech Lang. 24, 4, 562-588.
31. WIERSTRA, D., FOERSTER, A., PETERS, J., AND SCHMIDHUBER, J. 2007. Solving deep memory POMDPs with recurrent policy gradients. In Proceedings of the International Conference on Artificial Neural Networks.
32. WIERSTRA, D.,FÖRSTER, A.,PETERS, J., AND SCHMIDHUBER, J. 2010. Recurrent policy gradients. Logic J. IGPL 18, 5, 620-634.
33. WILLIAMS, J. 2008a. Demonstration of a POMDP voice dialer. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies (HLT'08). Association for Computational Linguistics, Morristown, NJ, 1-4.
34. WILLIAMS, J. D. 2008b. Integrating expert knowledge into POMDP optimization for spoken dialog systems. In Proceedings of the AAAI-08 Workshop on Advancements in POMDP Solvers.
35. WILLIAMS, J. D. AND YOUNG, S. 2005. Scaling Up POMDPs for dialog management: The Summary POMDP Method. In Proceedings of the IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU).
36. WILLIAMS, R. J. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn. 8.
37. YOUNG, S. 2007. CUED standard dialogue acts: http://mi.eng.cam.ac.uk/ research/dialogue/LocalDocs/dastd.pdf. Tech. rep., Engineering Department, Cambridge University.
38. YOUNG, S.,GAŠIĆ, M.,KEIZER, S.,MAIRESSE, F., SCHATZMANN, J., THOMSON, B., AND YU, K. 2010. The Hidden Information State Model: a practical framework for POMDP-based spoken dialogue management. Comput. Speech Lang. 24, 2, 150-174.
39. ZHANG, B.,CAI, Q.,MAO, J., AND GUO, B. 2001. Planning and acting under uncertainty: A new model for spoken dialogue system. In Proceedings of the 17th Conference on Uncertainty in Artificial Intelligence (UAI).