Learning from real users: Rating dialogue success with neural networks for reinforcement learning in spoken dialogue systems

Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH

Volumn 2015-January, Issue , 2015, Pages 2007-2011

  Su, Pei Hao ^a   Vandyke, David ^a   Gašíc, Milica ^a   Kim, Dongho ^a   Mrkšíc, Nikola ^a   Wen, Tsung Hsien ^a   Young, Steve ^a  

^a UNIVERSITY OF CAMBRIDGE   (United Kingdom)

Author keywords

Dialogue success classification; Neural network; Real users; Reward prediction; Spoken dialogue systems

Indexed keywords

NEURAL NETWORKS; REINFORCEMENT LEARNING; SPEECH COMMUNICATION;

BASELINE SYSTEMS; BEST MODEL; NEURAL NETWORK MODEL; NON-TRIVIAL; PRIOR KNOWLEDGE; REAL USERS; SPOKEN DIALOGUE SYSTEM; USER FEEDBACK;

SPEECH PROCESSING;

EID: 84959164720     PISSN: 2308457X     EISSN: 19909772     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (31)

1. J. D. Williams and S. Young, "Partially observable Markov decisionprocesses for spoken dialog systems, " Computer Speech and Language, vol. 21, no. 2, pp. 393-422, 2007.
2. S. Young, M. Gasic, B. Thomson, and J. Williams, "Pomdp-basedstatistical spoken dialogue systems: A review, " in Proc of theIEEE, vol. 99, 2013, pp. 1-20.
3. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. MIT Press, 1999.
4. M. Gasic and S. Young, "Gaussian processes for pomdp-baseddialogue manager optimisation, " TASLP, vol. 22, 2014.
5. M. Gasic, D. Kim, P. Tsiakoulis, C. Breslin, M. Henderson, M. Szummer, B. Thomson, and S. J. Young, "Incremental on-lineadaptation of pomdp-based dialogue managers to extended domains, "in Interspeech 2014, 15th Annual Conference of the InternationalSpeech Communication Association, Singapore, September14-18, 2014, 2014, pp. 140-144.
6. O. Lemon and O. Pietquin, "Machine learning for spoken dialoguesystems, " in In Proceedings of the European Conferenceon Speech Communication and Technologies (Interspeech07), Anvers, 2007.
7. L. Daubigney, M. Geist, S. Chand ramohan, and O. Pietquin, "Acomprehensive reinforcement learning framework for dialoguemanagement optimization, " Selected Topics in Signal Processing, IEEE Journal of, vol. 6, no. 8, pp. 891-902, 2012.
8. E. Levin, R. Pieraccini, and W. Eckert, "A stochastic model ofhuman-machine interaction for learning dialog strategies, " Speechand Audio Processing, IEEE Transactions on, vol. 8, no. 1, pp. 11-23, Jan 2000.
9. J. Schatzmann and S. Young, "The hidden agenda user simulationmodel, " Audio, Speech, and Language Processing, IEEE Transactionson, vol. 17, no. 4, pp. 733-747, May 2009.
10. S. Keizer, M. Gasic, F. Jurcicek, F. Mairesse, B. Thomson, K. Yu, and S. Young, Proceedings of the SIGDIAL 2010 Conference. Association for Computational Linguistics, 2010, ch. Parameterestimation for agenda-based user simulation, pp. 116-123.
11. Z. Yang, G. Levow, and H. Meng, "Predicting user satisfaction inspoken dialog system evaluation with collaborative filtering, " SelectedTopics in Signal Processing, IEEE Journal of, vol. 6, no. 99, pp. 971-981, 2012.
12. C. Daniel, M. Viering, J. Metz, O. Kroemer, and J. Peters, "Activereward learning, " in Proceedings of Robotics Science & Systems, 2014.
13. M. Gasic, F. Jurcicek, B. Thomson, K. Yu, and S. Young, "On-linepolicy optimisation of spoken dialogue systems via live interactionwith human subjects, " in Automatic Speech Recognition and Understand ing (ASRU), 2011 IEEE Workshop on, Dec 2011, pp. 312-317.
14. M. A. Walker, D. J. Litman, C. A. Kamm, and A. Abella, "Paradise: A framework for evaluating spoken dialogue agents, " Proceedings of the eighth conference on European chapter ofthe Association for Computational Linguistics. Association forComputational Linguistics, 1997, pp. 271-280.
15. L. Larsen, "Issues in the evaluation of spoken dialogue systemsusing objective and subjective measures, " in Automatic SpeechRecognition and Understand ing, 2003. ASRU '03. 2003 IEEEWorkshop on, Nov 2003, pp. 209-214.
16. L. E. Asri, R. Laroche, and O. Pietquin, "Reward Function Learningfor Dialogue Management, " in Proceedings of the sixth StartingArtificial Intelligence Research Symposium (STAIRS 2012), Montpellier (France), August 2012, pp. 95-106.
17. -, "Task completion transfer learning for reward inference, " inProc of MLIS, 2014.
18. M. Gasic, C. Breslin, M. Henderson, D. Kim, M. Szummer, B. Thomson, P. Tsiakoulis, and S. J. Young, "On-line policy optimisationof Bayesian spoken dialogue systems via human interaction, "in IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2013, Vancouver, BC, Canada, May26-31, 2013, 2013, pp. 8367-8371.
19. B. Thomson and S. Young, "Bayesian update of dialogue state: A pomdp framework for spoken dialogue systems. " ComputerSpeech and Language, vol. 24, pp. 562-588, 2010.
20. M. Lukoeviius and H. Jaeger, "Reservoir computing approachesto recurrent neural network training, " Computer Science Review, vol. 3, no. 3, pp. 127-149, 2009.
21. T. Mikolov, M. Karafiát, L. Burget, J. Cernocky, and S. Khudanpur, "Recurrent neural network based language model. " in Interspeech2010, 11th Annual Conference of the International SpeechCommunication Association, Makuhari, Chiba, Japan, September26-30, 2010, 2010, pp. 1045-1048.
22. T. Mikolov, S. Kombrink, L. Burget, J. H. Cernocky, and S. Khudanpur, "Extensions of recurrent neural network language model, "in Acoustics, Speech and Signal Processing (ICASSP), 2011 IEEEInternational Conference on. IEEE, 2011, pp. 5528-5531.
23. A. Karpathy and L. Fei-Fei, "Deep visual-semantic alignments forgenerating image descriptions, " CoRR, vol. abs/1412. 2306, 2014.
24. G. Mesnil, Y. Dauphin, K. Yao, Y. Bengio, L. Deng, D. Hakkani-Tur, X. He, L. Heck, G. Tur, D. Yu, and G. Zweig, "Using recurrentneural networks for slot filling in spoken language understand ing, "IEEE/ACM Transactions on Audio, Speech, and LanguageProcessing, March 2015.
25. Y. LeCun and Y. Bengio, "The hand book of brain theory and neuralnetworks, " M. A. Arbib, Ed. Cambridge, MA, USA: MITPress, 1998, ch. Convolutional Networks for Images, Speech, and Time Series, pp. 255-258.
26. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-basedlearning applied to document recognition, " in Proceedings of theIEEE, 1998, pp. 2278-2324.
27. N. Kalchbrenner, E. Grefenstette, and P. Blunsom, "A convolutionalneural network for modelling sentences, " Proceedings ofthe 52nd Annual Meeting of the Association for ComputationalLinguistics, June 2014.
28. M. Henderson, M. Gasic, B. Thomson, P. Tsiakoulis, K. Yu, and S. Young, "Discriminative Spoken Language Understand ing UsingWord Confusion Networks, " in Spoken Language TechnologyWorkshop, 2012. IEEE, 2012.
29. J. Bergstra, O. Breuleux, F. Bastien, P. Lamblin, R. Pascanu, G. Desjardins, J. Turian, D. Warde-Farley, and Y. Bengio, "Theano: A CPU and GPU math expression compiler, " in Proceedingsof the Python for Scientific Computing Conference(SciPy), Jun. 2010, oral Presentation.
30. F. Bastien, P. Lamblin, R. Pascanu, J. Bergstra, I. J. Goodfellow, A. Bergeron, N. Bouchard, and Y. Bengio, "Theano: new featuresand speed improvements, " Deep Learning and Unsupervised FeatureLearning NIPS 2012 Workshop, 2012.
31. E. Ferreira and F. Lefèvre, "Social signal and user adaptation inreinforcement learning-based dialogue management, " in Proceedingsof the 2Nd Workshop on Machine Learning for InteractiveSystems: Bridging the Gap Between Perception, Action and Communication, ser. MLIS '13. New York, NY, USA: ACM, 2013, pp. 61-69.