On-line active reward learning for policy optimisation in spoken dialogue systems

54th Annual Meeting of the Association for Computational Linguistics, ACL 2016 - Long Papers

Volumn 4, Issue , 2016, Pages 2431-2441

  Su, Pei Hao ^a   Gašić, Milica ^a   Mrkšić, Nikola ^a   Rojas Barahona, Lina ^a   Ultes, Stefan ^a   Vandyke, David ^a   Wen, Tsung Hsien ^a   Young, Steve ^a  

^a UNIVERSITY OF CAMBRIDGE   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL LINGUISTICS; E-LEARNING; GAUSSIAN DISTRIBUTION; GAUSSIAN NOISE (ELECTRONIC); NETWORK CODING; RECURRENT NEURAL NETWORKS; SPEECH PROCESSING;

CONTINUOUS SPACES; DIALOGUE REPRESENTATION; ENCODER-DECODER; GAUSSIAN PROCESS MODELS; GAUSSIAN PROCESSES; ONLINE LEARNING; REWARD FUNCTION; SPOKEN DIALOGUE SYSTEM;

REINFORCEMENT LEARNING;

EID: 85012034455     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.18653/v1/p16-1230     Document Type: Conference Paper

References (51)

1. Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, James Bergstra, Ian J. Goodfellow, Arnaud Bergeron, Nicolas Bouchard, and Yoshua Bengio. 2012. Theano: new features and speed improvements. Deep Learning and Unsupervised Feature Learning NIPS Workshop.
2. James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio. 2010. Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference.
3. Abdeslam Boularias, Hamid R Chinaei, and Brahim Chaib-draa. 2010. Learning the reward model of dialogue pomdps from data. In NIPS Workshop on Machine Learning for Assistive Techniques.
4. Eric Brochu, Vlad M Cora, and Nando De Freitas. 2010. A tutorial on bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv preprint arXiv: 1012.2599.
5. Inigo Casanueva, Thomas Hain, Heidi Christensen, Ricard Marxer, and Phil Green. 2015. Knowledge transfer between speakers for personalised dialogue management. In Proc of SigDial.
6. Lu Chen, Pei-Hao Su, and Milica Gasic. 2015. Hyper-parameter optimisation of gaussian process reinforcement learning for statistical dialogue management. In Proc of SigDial.
7. Weiwei Cheng, Johannes Fürnkranz, Eyke Hüllermeier, and Sang-Hyeun Park. 2011. Preference-based policy iteration: Leveraging preference learning for reinforcement learning. In Machine learning and knowledge discovery in databases. Springer.
8. Kyunghyun Cho, Bart van Merriënboer Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv: 1406.1078.
9. Peng Dai, Christopher H Lin, Daniel S Weld, et al. 2013. Pomdp-based control of workflows for crowdsourcing. Artificial Intelligence, 202.
10. Christian Daniel, Malte Viering, Jan Metz, Oliver Kroemer, and Jan Peters. 2014. Active reward learning. In Proc of RSS.
11. Lucie Daubigney, Matthieu Geist, Senthilkumar Chandramohan, and Olivier Pietquin. 2014. A comprehensive reinforcement learning framework for dialogue management optimisation. Journal of Selected Topics in Signal Processing, 6(8).
12. Layla El Asri, Romain Laroche, and Olivier Pietquin. 2014. Task completion transfer learning for reward inference. In Proc of MLIS.
13. Milica Gasic and Steve Young. 2014. Gaussian processes for pomdp-based dialogue manager optimization. TASLP, 22(1): 28-40.
14. Milica Gasic, Filip Jurcicek, Blaise. Thomson, Kai Yu, and Steve Young. 2011. Online policy optimisation of spoken dialogue systems via live interaction with human subjects. In IEEE ASRU.
15. Milica Gasic, Catherine Breslin, Matthew Henderson, Dongho Kim, Martin Szummer, Blaise Thomson, Pirros Tsiakoulis, and Steve J. Young. 2013. On-line policy optimisation of bayesian spoken dialogue systems via human interaction. In Proc of ICASSP.
16. Alax Graves, Navdeep Jaitly, and Abdel-rahman Mohamed. 2013. Hybrid speech recognition with deep bidirectional lstm. In IEEE ASRU.
17. Matthew Henderson, Milica Gasic, Blaise Thomson, Pirros Tsiakoulis, Kai Yu, and Steve Young. 2012. Discriminative spoken language understanding using word confusion networks. In IEEE SLT.
18. James Hensman, Nicolo Fusi, Ricardo Andrade, Nicolas Durrande, Alan Saul, Max Zwiessele, and Neil D. Lawrence. 2012. GPy: A gaussian process framework in python. http://github.com/SheffieldML/GPy.
19. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8).
20. Ashish Kapoor, Kristen Grauman, Raquel Urtasun, and Trevor Darrell. 2007. Active learning with gaussian processes for object categorization. In Proc of ICCV.
21. Dongho Kim, Catherine Breslin, Pirros Tsiakoulis, Matthew Henderson, and Steve J Young. 2014. Inverse reinforcement learning for micro-turn management. In IEEE SIT.
22. L.B. Larsen. 2003. Issues in the evaluation of spoken dialogue systems using objective and subjective measures. In IEEE ASRU.
23. Esther Levin and Roberto Pieraccini. 1997. A stochastic model of computer-human interaction for learning dialogue strategies. Eurospeech.
24. Omer Levy and Yoav Goldberg. 2014. Neural word embedding as implicit matrix factorization. In NIPS.
25. Christopher H Lin, Daniel S Weld, et al. 2014. To re (label), or not to re (label). In Second AAAI Conference on Human Computation and Crowdsourcing.
26. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In NIPS.
27. Hannes Nickisch and Carl Edward Rasmussen. 2008. Approximations for binary gaussian process classification. JMLR, 9(10).
28. Tim Paek and Roberto Pieraccini. 2008. Automating spoken dialogue management design using machine learning: An industry perspective. Speech communication, 50.
29. Carl Edward Rasmussen and Chris Williams. 2006. Gaussian processes for machine learning.
30. Verena Rieser and Oliver Lemon. 2011. Learning and evaluation of dialogue strategies for new applications: Empirical methods for optimization from small data sets. Computational Linguistics, 37(1).
31. Lina Maria Rojas Barahona and Christophe Cerisara. 2014. Bayesian Inverse Reinforcement Learning for Modeling Conversational Agents in a Virtual Environment. In Conference on Intelligent Text Processing and Computational Linguistics.
32. Nicholas Roy, Joelle Pineau, and Sebastian Thrun. 2000. Spoken dialogue management using probabilistic reasoning. In Proc of SigDial.
33. Stuart Russell. 1998. Learning agents for uncertain environments. In Proc of COLT.
34. Jost Schatzmann, Karl Weilhammer, Matt Stuttle, and Steve Young. 2006. A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies. The knowledge engineering review, 21 (02): 97-126.
35. Mian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2015. Hierarchical neural network generative models for movie dialogues. arXiv preprint arXiv: 1507.04808.
36. Burr Settles. 2010. Active learning literature survey. Computer Sciences Technical Report 1648.
37. Pei-Hao Su, David Vandyke, Milica Gasic, Dongho Kim, Nikola Mrksic, Tsung-Hsien Wen, and Steve Young. 2015a. Learning from real users: Rating dialogue success with neural networks for reinforcement learning in spoken dialogue systems. In Proc of Interspeech.
38. Pei-Hao Su, David Vandyke, Milica Gasic, Nikola Mrksic, Tsung-Hsien Wen, and Steve Young. 2015b. Reward shaping with recurrent neural networks for speeding up on-line policy learning in spoken dialogue systems. In Proc of SigDial.
39. Hiroaki Sugiyama, Toyomi Meguro, and Yasuhiro Minami. 2012. Preferencelearning based inverse reinforcement learning for dialog control. In Proc of Interspeech.
40. Blaise Thomson and Steve Young. 2010. Bayesian update of dialogue state: A pomdp framework for spoken dialogue systems. Computer Speech and Language, 24: 562-588.
41. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In Proc of ACL.
42. Stefan Ultes and Wolfgang Minker. 2015. Quality-adaptive spoken dialogue initiative selection and implications on reward modelling. In Proc of SigDial.
43. Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. JMLR, 9: 85.
44. David Vandyke, Pei-Hao Su, Milica Gasic, Nikola Mrksic, Tsung-Hsien Wen, and Steve Young. 2015. Multi-domain dialogue success classifiers for policy training. In IEEE ASRU.
45. Oriol Vinyals and Quoc Le. 2015. A neural conversational model. arXiv preprint arXiv: 1506.05869.
46. Marilyn A. Walker, Diane J. Litman, Candace A. Kamm, and Alicia Abella. 1997. PARADISE: A framework for evaluating spoken dialogue agents. In Proc of EACL.
47. Jason D. Williams and Steve Young. 2007. Partially observable Markov decision processes for spoken dialog systems. Computer Speech and Language, 21(2): 393-422.
48. Zhaojun Yang, G Levow, and Helen Meng. 2012. Predicting user satisfaction in spoken dialog system evaluation with collaborative filtering. IEEE Journal of Selected Topics in Signal Processing, 6(99): 971-981.
49. Steve Young, Milica Gasic, Blaise Thomson, and Jason Williams. 2013. Pomdp-based statistical spoken dialogue systems: a review. In Proc of IEEE, Volume 99, pages 1-20.
50. Chicheng Zhang and Kamalika Chaudhuri. 2015. Active learning from weak and strong labelers. CoRR, abs/1510.02847.
51. Liyue Zhao, Gita Sukthankar, and Rahul Sukthankar. 2011. Incremental relabeling for active learning with noisy crowdsourced annotations. In Proc of PASSAT and Proc of SocialCom.