POMDP-based statistical spoken dialog systems: A review

Proceedings of the IEEE

Volumn 101, Issue 5, 2013, Pages 1160-1179

  Young, Steve ^a   Gašić, Milica ^a   Thomson, Blaise ^a   Williams, Jason D ^b  

^a UNIVERSITY OF CAMBRIDGE   (United Kingdom)

^b MICROSOFT RESEARCH   (United States)

Author keywords

Belief monitoring; partially observable Markov decision process (POMDP); policy optimization; reinforcement learning; spoken dialog systems (SDSs)

Indexed keywords

APPROXIMATION ALGORITHMS; BAYESIAN NETWORKS; BEHAVIORAL RESEARCH; LEARNING ALGORITHMS; MARKOV PROCESSES;

BAYESIAN MODEL; MODEL REPRESENTATION; NOISY ENVIRONMENT; PARTIALLY OBSERVABLE MARKOV DECISION PROCESS; POLICY OPTIMIZATION; SPEECH RECOGNIZER; SPOKEN DIALOG SYSTEMS; STATE OF THE ART;

REINFORCEMENT LEARNING;

EID: 84876682878     PISSN: 00189219     EISSN: None     Source Type: Journal    
DOI: 10.1109/JPROC.2012.2225812     Document Type: Review

References (155)

1. R. De Mori, Spoken Dialogues With Computers, New York: Academic, 1998.
2. R. Smith and D. Hipp, Spoken Natural Language Dialog Systems: A Practical Approach. Oxford, U.K.: Oxford Univ. Press, 1994.
3. V. Zue and J. Glass, "Conversational interfaces: Advances and challenges," Proc. IEEE, vol. 88, no. 8, pp. 1166-1180, Aug. 2000.
4. S. Young, "Cognitive user interfaces," IEEE Signal Process. Mag., vol. 27, no. 3, pp. 128-140, May 2010.
5. M. Oshry, P. Baggia, K. Rehor, M. Young, R. Akolkar, X. Yang, J. Barnett, R. Hosn, R. Auburn, J. Carter, S. McGlashan, M. Bodell, and D. C. Burnett, Voice extensible markup language (VoiceXML) 3.0, W3C, W3C Working Draft, Dec. 2009. [Online]. Available: http://www.w3.org/TR/2009/WD-voicexml30-20091203/.
6. T. Paek and R. Pieraccini, "Automating spoken dialogue management design using machine learning: An industry perspective," Speech Commun., vol. 50, no. 8-9, pp. 716-729, 2008.
7. R. P. Lippmann, "Speech recognition by machines and humans," Speech Commun., vol. 22, no. 1, pp. 1-15, 1997.
8. A. Black, S. Burger, A. Conkie, H. Hastie, S. Keizer, O. Lemon, N. Merigaud, G. Parent, G. Schubiner, B. Thomson, J. Williams, K. Yu, S. Young, and M. Eskenazi, "Spoken dialog challenge 2010: Comparison of live and control test results," in Proc. Annu. Meeting Special Interest Group Discourse Dialogue, Portland, OR, 2011, pp. 2-7.
9. N. Roy, J. Pineau, and S. Thrun, "Spoken dialogue management using probabilistic reasoning," in Proc. Annu. Meeting Assoc. Comput. Linguist., Hong Kong, 2000, pp. 93-100.
10. S. Young, "Talking to machines (statistically speaking)," in International Conf Spoken Language Processing, Denver, CO, 2002, pp. 9-15.
11. J. Williams and S. Young, "Partially observable Markov decision processes for spoken dialog systems," Comput. Speech Lang., vol. 21, no. 2, pp. 393-422, 2007.
12. L. Kaelbling, M. Littman, and A. Cassandra, "Planning and acting in partially observable stochastic domains," Artif. Intell., vol. 101, pp. 99-134, 1998.
13. E. Hansen, "Solving POMDPs by searching in policy space," in Proc. Conf. Uncertainty Artif. Intell., Madison, WI, 1998, pp. 211-219.
14. M. L. Littman, R. S. Sutton, and S. Singh, "Predictive representations of state Advances in Neural Information Processing Systems 14. Cambridge, MA: MIT Press, 2002, pp. 1555-1561.
15. R. E. Bellman, Dynamic Programming. Princeton, NJ: Princeton Univ. Press, 1957.
16. J. Pineau, G. Gordon, and S. Thrun, "Point-based value iteration: An anytime algorithm for POMDPs," in Proc. Int. Joint Conf. Artif. Intell., Acapulco, Mexico, 2003, pp. 1025-1032.
17. T. Smith and R. G. Simmons, "Heuristic search value iteration for POMDPs," in Proc. Conf. Uncertainty Artif. Intell., Banff, AB, Canada, 2004, pp. 520-527.
18. M. Spaan and N. Vlassis, "Perseus: Randomized point-based value iteration for POMDPs," J. Artif. Intell. Res., vol. 24, pp. 195-220, 2005.
19. T. Smith and R. Simmons, "Point-based POMDP algorithms: Improved analysis and implementation," in Proc. 21st Annu. Conf. Uncertainty Artif. Intell., Arlington, VA, 2005, pp. 542-549.
20. S. Ji, R. Parr, H. Li, X. Liao, and L. Carin, "Point-based policy iteration," in Proc. Nat. Conf. Artif. Intell., Vancouver, BC, Canada, 2007, pp. 1243-1249.
21. J. Williams, P. Poupart, and S. Young, "Factored partially observable Markov decision processes for dialogue management," in Proc. IJCAI Workshop Knowl. Reason. Practical Dialog Syst., Edinburgh, U.K., 2005, pp. 76-82.
22. T. Bui, M. Poel, A. Nijholt, and J. Zwiers, "A tractable hybrid DDN-POMDP approach to affective dialogue modeling for probabilistic frame-based dialogue systems," Natural Lang. Eng., vol. 15, no. 2, pp. 273-307, 2009.
23. S. Young, J. Schatzmann, K. Weilhammer, and H. Ye, "The hidden information state approach to dialog management," in Proc. Int. Conf. Acoust. Speech Signal Process., Honolulu, HI, 2007, pp. 149-152.
24. K. Kim, C. Lee, S. Jung, and G. Lee, "A frame-based probabilistic framework for spoken dialog management using dialog examples," in Proc. Special Interest Group Discourse Dialogue (SIGDIAL) Workshop Discourse Dialogue, Columbus, OH, 2008, pp. 120-127.
25. J. Williams, "Incremental partition recombination for efficient tracking of multiple dialogue states," in Proc. Int. Conf. Acoust. Speech Signal Process., Dallas, TX, 2010, pp. 5382-5385.
26. M. Gašíc and S. Young, "Effective handling of dialogue state in the hidden information state POMDP dialogue manager," ACM Trans. Speech Lang. Process., vol. 7, no. 3, 2011, Article 4.
27. B. Thomson, J. Schatzmann, and S. Young, "Bayesian update of dialogue state for robust dialogue systems," in Proc. Int. Conf. Acoust. Speech Signal Process., Las Vegas, NV, 2008, pp. 4937-4940.
28. B. Thomson and S. Young, "Bayesian update of dialogue state: A POMDP framework for spoken dialogue systems," Comput. Speech Lang., vol. 24, no. 4, pp. 562-588, 2010.
29. J. Henderson and O. Lemon, "Mixture model POMDPs for efficient handling of uncertainty in dialogue management," in Proc. Annu. Meeting Assoc. Comput. Linguist., Columbus, OH, 2008, pp. 73-76.
30. D. Bohus and A. Rudnicky, "A 'K hypotheses + other' belief updating model," in Proc. AAAI Workshop Stat. Empirical Approaches Spoken Dialogue Syst., Boston, MA, 2006, pp. 13-18.
31. R. Higashinaka, M. Nakano, and K. Aikawa, "Corpus-based discourse understanding in spoken dialogue systems," in Proc. Annu. Meeting Assoc. Comput. Linguist., Sapporo, Japan, 2003, pp. 240-247.
32. C. Bishop, Pattern Recognition and Machine Learning. New York: Springer-Verlag, 2006.
33. J. D. Williams, "Using particle filters to track dialogue state," in Proc. IEEE Workshop Autom. Speech Recognit. Understand., Kyoto, Japan, 2007, pp. 502-507.
34. B. Thomson, K. Yu, S. Keizer, M. Gasic, F. Jurcicek, F. Mairesse, and S. Young, "Bayesian dialogue system for the let's go spoken dialogue challenge," in Proc. IEEE Spoken Lang. Technol. Workshop, Berkeley, CA, 2010, pp. 460-465.
35. P. A. Crook and O. Lemon, "Lossless value directed compression of complex user goal states for statistical spoken dialogue systems," in Proc. Annu. Conf. Int. Speech Commun. Assoc., Florence, Italy, 2011, pp. 1029-1032.
36. F. Doshi and N. Roy, "The permutable POMDP: Fast solutions to POMDPs for preference elicitation," in Proc. Int. Joint Conf. Autonom. Agents Multiagent Syst., 2008, pp. 493-500.
37. J. Williams and S. Young, "Scaling up POMDPs for dialog management: The 'summary POMDP' method," in Proc. IEEE Workshop Autom. Speech Recognit. Understand., Cancun, Mexico, 2005, pp. 177-182.
38. J. Williams and S. Young, "Scaling POMDPs for spoken dialog management," IEEE Trans. Audio Speech Lang. Process., vol. 15, no. 7, pp. 2116-2129, Jul. 2007.
39. D. Andre and S. Russell, "State abstraction for programmable reinforcement learning agents," in Proc. 18th Nat. Conf. Artif. Intell., Edmonton, AB, Canada, 2002, pp. 119-125.
40. J. D. Williams, "The best of both worlds: Unifying conventional dialog systems and POMDPs," in Proc. Annu. Conf. Int. Speech Commun. Assoc., Brisbane, Australia, 2008, pp. 1173-1176.
41. P. Lison, "Towards relational POMDPs for adaptive dialogue management," in Proc. Annu. Meeting Assoc. Comput. Linguist., Uppsala, Sweden, 2010, pp. 7-12.
42. S. Young, M. Gašíc, S. Keizer, F. Mairesse, J. Schatzmann, B. Thomson, and K. Yu, "The hidden information state model: A practical framework for POMDP-based spoken dialogue management," Comput. Speech Lang., vol. 24, no. 2, pp. 150-174, 2010.
43. L. Li, J. D. Williams, and S. Balakrishnan, "Reinforcement learning for dialog management using least-squares policy iteration and fast feature selection," in Proc. Annu. Conf. Int. Speech Commun. Assoc., Brighton, U.K., 2009, pp. 2475-2478.
44. O. Pietquin, M. Geist, S. Chandramohan, and H. Frezza-Buet, "Sample-efficient batch reinforcement learning for dialogue management optimization," ACM Trans. Speech Lang. Process., vol. 7, no. 3, pp. 1-21, 2011.
45. K. Scheffler and S. Young, "Automatic learning of dialogue strategy using dialogue simulation and reinforcement learning," in Proc. 2nd Int. Conf. Human Lang. Technol. Res., San Diego, CA, 2002, pp. 12-19.
46. H. Cuaýahuitl, S. Renals, O. Lemon, and H. Shimodaira, "Reinforcement learning of dialogue strategies with hierarchical abstract machines," in Proc. IEEE Spoken Lang. Technol. Workshop, Aruba, 2006, pp. 182-185.
47. H. Cuaýahuitl, S. Renals, and O. Lemon, "Learning multi-goal dialogue strategies using reinforcement learning with reduced state-action spaces," in Proc. Annu. Conf. Int. Speech Commun. Assoc., Pittsburgh, PA, 2006, pp. 547-565.
48. O. Pietquin and T. Dutoit, "A probabilistic framework for dialog simulation and optimal strategy learning," IEEE Trans. Audio Speech Lang. Process., vol. 14, no. 2, pp. 589-599, Mar. 2006.
49. J. Henderson, O. Lemon, and K. Georgila, "Hybrid reinforcement/supervised learning for dialogue policies from communicator data," in Proc. IJCAI Workshop Knowl. Reason. Practical Dialog Syst., Edinburgh, U.K., 2005, pp. 68-75.
50. M. Frampton and O. Lemon, "Learning more effective dialogue strategies using limited dialogue move features," in Proc. Annu. Meeting Assoc. Comput. Linguist., Sydney, Australia, 2006, pp. 185-192.
51. J. Henderson, O. Lemon, and K. Georgila, "Hybrid reinforcement/supervised learning of dialogue policies from fixed datasets," Comput. Linguist., vol. 34, no. 4, pp. 487-511, 2008.
52. H. Kurniawati, D. Hsu, and W. Lee, "SARSOP: Efficient point-based POMDP planning by approximating optimally reachable belief spaces," in Proc. Robot., Sci. Syst., 2008. [Online]. Available: http://www. roboticsproceedings.org/rss04/.
53. T. H. Bui, M. Poel, A. Nijholt, and J. Zwiers, "A tractable DDN-POMDP approach to affective dialogue modeling for general probabilistic frame-based dialogue systems," in Proc. IJCAI Workshop Knowl. Reason. Practical Dialog Syst., Hyderabad, India, 2007, pp. 34-37.
54. T. H. Bui, B. W. van Schooten, and D. H. W. Hofs, "Practical dialogue manager development using POMDPs," in Proc. Annu. Meeting Special Interest Group Discourse Dialogue, Antwerp, Belgium, 2007, pp. 215-218.
55. F. Pinault, F. Lefevre, and R. D. Mori, "Feature-based summary space for stochastic dialogue modeling with hierarchical semantic frames," in Proc. Annu. Conf. Int. Speech Commun. Assoc., Brighton, U.K., 2009, pp. 284-287.
56. A. Boularias, H. R. Chinaei, and B. Chaib-draa, "Learning the reward model of dialogue POMDPs from data," in Proc. NIPS Workshop Mach. Learn. Assistive Tech., 2010.
57. F. Pinault and F. Lefevre, "Semantic graph clustering for POMDP-based spoken dialog systems," in Proc. Annu. Conf. Int. Speech Commun. Assoc., Florence, Italy, 2011, pp. 1321-1324.
58. F. Pinault and F. Lefevre, "Unsupervised clustering of probability distributions of semantic frame graphs for pomdp-based spoken dialogue systems with summary space," in Proc. IJCAI Workshop Knowl. Reason. Practical Dialog Syst., Barcelona, Spain, 2011.
59. H. R. Chinaei and B. Chaib-draa, "Learning dialogue POMDP models from data," in Proc. Can. Conf. Artif. Intell., 2011, pp. 86-91.
60. H. R. Chinaei, B. Chaib-draa, and L. Lamontagne, "Learning observation models for dialogue POMDPs," in Proc. Can. Conf. Artif. Intell., 2012.
61. S. Varges, G. Riccardi, S. Quarteroni, and A. Ivanov, "POMDP concept policies and task structures for hybrid dialog management," in Proc. Int. Conf. Acoust. Speech Signal Process., Prague, Czech Republic, 2011, pp. 5592-5595.
62. J. D. Williams, "Integrating expert knowledge into POMDP optimization for spoken dialog systems," in Proc. AAAI Workshop Adv. POMDP Solvers, Chicago, Illinois, 2008.
63. R. Sutton and A. Barto, Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press, 1998, ser. Adaptive Computation and Machine Learning.
64. F. Lefevre, M. Gasic, F. Jurcicek, S. Keizer, F. Mairesse, B. Thomson, K. Yu, and S. J. Young, "k-nearest neighbor Monte-Carlo control algorithm for POMDP-based dialogue systems," in Proc. Annu. Meeting Special Interest Group Discourse Dialogue, London, U.K., 2009.
65. M. G. Lagoudakis and R. Parr, "Least-squares policy iteration," J. Mach. Learn. Res., vol. 4, pp. 1107-1149, 2003.
66. R. J. Williams, "Simple statistical gradient-following algorithms for connectionist reinforcement learning," Mach. Learn., vol. 8, pp. 229-256, 1992.
67. S. Amari, "Natural gradient works efficiently in learning," Neural Comput., vol. 10, no. 2, pp. 251-276, 1998.
68. J. Peters and S. Schaal, "Natural actor-critic," Neurocomputing, vol. 71, no. 7-9, pp. 1180-1190, 2008.
69. T. Misu, "Dialogue strategy optimization to assist user's decision for spoken consulting dialogue systems," in Proc. IEEE Spoken Lang. Technol. Workshop, Berkeley, CA, 2010, pp. 354-359.
70. F. Jurčíček, B. Thomson, and S. Young, "Natural actor and belief critic: Reinforcement algorithm for learning parameters of dialogue systems modelled as POMDPs," ACM Trans. Speech Lang. Process., vol. 7, no. 3, 2011, Article 6.
71. T. Misu, K. Sugiura, K. Ohtake, and C. Hori, "Modeling spoken decision making dialogue and optimization of its dialogue strategy," in Proc. Annu. Meeting Special Interest Group Discourse Dialogue, Portland, OR, 2011, pp. 221-224.
72. T. Misu, K. Georgila, A. Leuski, and D. Traum, "Reinforcement learning of question-answering dialogue policies for virtual museum guides," in Proc. Annu. Meeting Special Interest Group Discourse Dialogue, Seoul, Korea, 2012, pp. 84-93.
73. J. Schatzmann, K. Weilhammer, M. Stuttle, and S. Young, "A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies," Knowl. Eng. Rev., vol. 21, no. 2, pp. 97-126, Jun. 2006.
74. H. Hastie, "Metrics and evaluation of spoken dialogue systems," in Data-Driven Methods for Adaptive Spoken Dialogue Systems: Computational Learning for Conversational Interfaces. New York: Springer-Verlag, 2012.
75. O. Pietquin and H. Hastie, "A survey on metrics for the evaluation of user simulations," Knowl. Eng. Rev., 2012, DOI:http://dx.doi.org/10. 1017/S0269888912000343.
76. J. Schatzmann, K. Georgila, and S. Young, "quantitative evaluation of user simulation techniques for spoken dialogue systems," in Proc. Annu. Meeting Special Interest Group Discourse Dialogue, Lisbon, Portugal, 2005, pp. 45-54.
77. J. Williams, "Evaluating user simulations with the Cramervon Mises divergence," Speech Commun., vol. 50, pp. 829-846, 2008.
78. W. Eckert, E. Levin, and R. Pieraccini, "User modelling for spoken dialogue system evaluation," in Proc. IEEE Workshop Autom. Speech Recognit. Understand., Santa Barbara, CA, 1997, pp. 80-87.
79. K. Georgila, J. Henderson, and O. Lemon, "Learning user simulations for information state update dialogue systems," in Proc. Annu. Conf. Int. Speech Commun. Assoc., Lisbon, Portugal, 2005, pp. 893-896.
80. K. Georgila, J. Henderson, and O. Lemon, "User simulation for spoken dialogue systems: Learning and evaluation," in Proc. Int. Conf. Spoken Lang. Process., Pittsburgh, PA, 2006, pp. 45-54.
81. K. Scheffler and S. Young, "Probabilistic simulation of human-machine dialogues," in Proc. Int. Conf. Acoust. Speech Signal Process., Istanbul, Turkey, 2000, pp. 1217-1220.
82. V. Rieser and O. Lemon, "Cluster-based user simulations for learning dialogue strategies," in Proc. Annu. Conf. Int. Speech Commun. Assoc., Pittsburgh, PA, 2006.
83. O. Pietquin, "Consistent goal-directed user model for realistic man-machine task-oriented spoken dialogue simulation," in Proc. IEEE Int. Conf. Multimedia Expo, Toronto, ON, Canada, 2006, pp. 425-428.
84. J. Schatzmann, B. Thomson, K. Weilhammer, H. Ye, and S. Young, "Agenda-based user simulation for bootstrapping a POMDP dialogue system," in Proc. Conf. North Amer. Chapter Assoc. Comput. Linguist. Human Lang. Technol., Rochester, NY, 2007, pp. 149-152.
85. H. Ai and D. J. Litman, "Knowledge consistent user simulations for dialog systems," in Proc. Annu. Conf. Int. Speech Commun. Assoc., Antwerp, Belgium, 2007, pp. 2697-2700.
86. J. Schatzmann and S. Young, "The hidden agenda user simulation model," IEEE Trans. Audio Speech Lang. Process., vol. 17, no. 4, pp. 733-747, May 2009.
87. S. Keizer, M. Gasic, F. Jurcicek, F. Mairesse, B. Thomson, K. Yu, and S. Young, "Parameter estimation for agenda-based user simulation," in Proc. Annu. Meeting Special Interest Group Discourse Dialogue, Tokyo, Japan, 2010, pp. 116-123.
88. O. Pietquin, A Framework for Unsupervised Learning of Dialogue Strategies, Louvain, Belgium: Presses Universitaires de Louvain, 2004.
89. H. Cuayahuitl, S. Renals, O. Lemon, and H. Shimodaira, "Human-computer dialogue simulation using hidden Markov models," in Proc. IEEE Workshop Autom. Speech Recognit. Understand., Puerto Rico, 2005, pp. 290-295.
90. O. Pietquin and T. Dutoit, "A probabilistic framework for dialog simulation and optimal strategy learning," IEEE Trans. Audio Speech Lang. Process., vol. 14, no. 2, pp. 589-599, Mar. 2006.
91. S. Rossignol, M. Ianotto, and O. Pietquin, "Training a BN-based user model for dialogue simulation with missing data," in Proc. Int. Joint Conf. Natural Lang. Process., Chiang Mai, Thailand, Nov. 2011, pp. 598-604.
92. S. Jung, C. Lee, K. Kim, M. Jeong, and G. G. Lee, "Data-driven user simulation for automated evaluation of spoken dialog systems," Comput. Speech Lang., vol. 23, no. 4, pp. 479-509, 2009.
93. A. Ng and S. Russell, "Algorithms for inverse reinforcement learning," in Proc. Int. Conf. Mach. Learn., Stanford, CA, 2000, pp. 663-670.
94. S. Chandramohan, M. Geist, F. Lefevre, and O. Pietquin, "User simulation in dialogue systems using inverse reinforcement learning," in Proc. Annu. Conf. Int. Speech Commun. Assoc., Florence, Italy, 2011, pp. 1025-1028.
95. O. Pietquin and S. Renals, "ASR system modelling for automatic evaluation and optimisation of dialogue systems," in Proc. Int. Conf. Acoust. Speech Signal Process., Orlando, FL, 2002, pp. 46-49.
96. O. Pietquin and R. Beaufort, "Comparing ASR modeling methods for spoken dialogue simulation and optimal strategy learning," in Proc. Annu. Conf. Int. Speech Commun. Assoc., Lisbon, Portugal, 2005, pp. 861-864.
97. O. Pietquin and T. Dutoit, "Dynamic Bayesian networks for NLU simulation with application to dialog optimal strategy learning," in Proc. Int. Conf. Acoust. Speech Signal Process., Toulouse, France, 2006, pp. 49-52.
98. J. Schatzmann, B. Thomson, and S. Young, "Error simulation for training statistical dialogue systems," in Proc. IEEE Workshop Autom. Speech Recognit. Understand., Kyoto, Japan, 2007, pp. 526-531.
99. F. García, L. F. Hurtado, D. Griol, M. J. Castro, E. Segarra, and E. Sanchis, "Recognition and understanding simulation for a spoken dialog corpus acquisition," Proc. Text Speech Dialogue, pp. 574-581, 2007.
100. B. Thomson, K. Yu, M. Gasic, S. Keizer, F. Mairesse, J. Schatzmann, and S. Young, "Evaluating semantic-level confidence scores with multiple hypotheses," in Proc. Annu. Conf. Int. Speech Commun. Assoc., Brisbane, Australia, 2008, pp. 1153-1156.
101. J. Schatzmann, K. Weilhammer, M. Stuttle, and S. Young, "Effects of the user model on simulation-based learning of dialogue strategies," in Proc. IEEE Workshop Autom. Speech Recognit. Understand., Cancun, Mexico, 2005, pp. 220-225.
102. D. L. Hua Ai and J. Tetreault, "Comparing user simulation models for dialog strategy learning," in Proc. Conf. North Amer. Chapter Assoc. Comput. Linguist., Rochester, NY, 2007, pp. 1-4.
103. D. Kim, J. Kim, and K.-E. Kim, "Robust performance evaluation of POMDP-based dialogue systems," IEEE Trans. Audio Speech Lang. Process., vol. 19, no. 4, pp. 1029-1040, May 2011.
104. J. D. Williams, "Exploiting the ASR N-best by tracking multiple dialog state hypotheses," in Proc. Annu. Conf. Int. Speech Commun. Assoc., Brisbane, Australia, 2008, pp. 191-194.
105. S. Keizer, M. Gašíc, F. Mairesse, B. Thomson, K. Yu, and S. Young, "Modelling user behaviour in the HIS-POMDP dialogue manager," in Proc. IEEE Spoken Lang. Technol. Workshop, Goa, India, 2008, pp. 121-124.
106. F. Doshi and N. Roy, "Spoken language interaction with model uncertainty: An adaptive human-robot interaction system," Connection Sci., vol. 20, no. 4, pp. 299-318, 2008.
107. U. Syed and J. D. Williams, "Using automatically transcribed dialogs to learn user models in a spoken dialog system," Proc. Annu. Meeting Assoc. Comput. Linguist., pp. 121-124, 2008.
108. T. Minka, "Expectation propagation for approximate Bayesian inference," in Proc. Conf. Uncertainty Artif. Intell., Seattle, WA, 2001, pp. 362-369.
109. B. Thomson, F. Jurčíček, M. Gašíc, S. Keizer, F. Mairesse, K. Yu, and S. Young, "Parameter learning for POMDP spoken dialogue models," in Proc. IEEE Spoken Lang. Technol. Workshop, Berkeley, CA, 2010, pp. 271-276.
110. B. Thomson, "Statistical methods for spoken dialogue management," Ph.D. dissertation, Engineering Department, Cambridge Univ., Cambridge, U.K., 2009.
111. F. Jurčíček, B. Thomson, S. Keizer, F. Mairesse, M. Gašíc, K. Yu, and S. Young, "Natural belief-critic: A reinforcement algorithm for parameter estimation in statistical spoken dialogue systems," in Proc. Annu. Conf. Int. Speech Commun. Assoc., Tokyo, Japan, 2010, pp. 90-93.
112. C. Rasmussen and C. Williams, Gaussian Processes for Machine Learning. Cambridge, MA: MIT Press, 2006.
113. E. Engel, S. Mannor, and R. Meir, "Reinforcement learning with Gaussian processes," in Proc. Int. Conf. Mach. Learn., Bonn, Germany, 2005, pp. 201-208.
114. M. Gašíc, "Statistical dialogue modelling," Ph.D. dissertation, Engineering Department, Univ. Cambridge, Cambridge, U.K., 2011.
115. Y. Engel, "Algorithms and representations for reinforcement learning," Ph.D. dissertation, Engineering and Computer Science, Hebrew Univ., Jerusalem, Israel, 2005.
116. M. Gašíc, F. Jurčíček, S. Keizer, F. Mairesse, B. Thomson, K. Yu, and S. Young, "Gaussian processes for fast policy optimisation of a POMDP dialogue manager for a real-world task," in Proc. Annu. Meeting Special Interest Group Discourse Dialogue, Tokyo, Japan, 2010, pp. 201-204.
117. M. Gašíc, F. Jurčíček, B. Thomson, K. Yu, and S. Young, "On-line policy optimisation of spoken dialogue systems via live interaction with human subjects," in Proc. IEEE Workshop Autom. Speech Recognit. Understand., 2011, pp. 312-317.
118. M. Geist, O. Pietquin, and G. Fricout, "Kalman temporal differences: The deterministic case," in Proc. IEEE Symp. Adaptive Dyn. Programm. Reinforcement Learn., 2009, pp. 185-192.
119. M. Geist and O. Pietquin, "Managing uncertainty within the KTD framework," in Proc. Workshop Active Learn. Exp. Design, 2011, pp. 157-168.
120. O. Pietquin, M. Geist, and S. Chandramohan, "Sample efficient on-line learning of optimal dialogue policies with Kalman temporal differences," in Proc. Int. Joint Conf. Artif. Intell., Barcelona, Spain, 2011, pp. 1878-1883.
121. J. Williams, "Demonstration of a POMDP voice dialer," presented at the Annu. Meeting Assoc. Comput. Linguist., Demonstration Session, Columbus, Ohio, 2008.
122. S. Janarthanam, H. Hastie, O. Lemon, and X. Liu, "'The day after the day after tomorrow?' A machine learning approach to adaptive temporal expression generation: Training and evaluation with real users," in Proc. Annu. Meeting Special Interest Group Discourse Dialogue, Portland, OR, 2011, pp. 142-151.
123. K. Kim and G. G. Lee, "Multimodal dialog system using hidden information state dialog manager," in Proc. Int. Conf. Multimodal Interfaces Demonstration Session, Nagoya, Japan, 2007.
124. J. Williams, "Applying POMDPs to Dialog Systems in the Troubleshooting Domain," in Proc. HLT/NAACL Workshop Bridging the Gap: Acad. Ind. Res. Dialog Technol., Rochester, NY, 2007, pp. 1-8.
125. J. Williams, "An empirical evaluation of a statistical dialog system in public use," in Proc. Annu. Meeting Special Interest Group Discourse Dialogue, Portland, OR, 2011, pp. 130-141.
126. L. Hirschman and H. S. Thompson, Overview of Evaluation in Speech and Natural Language Processing. New York: Cambridge Univ. Press, 1997, pp. 409-414.
127. S. Gandhe and D. Traum, "An evaluation understudy for dialogue coherence models," in Proc. Annu. Meeting Special Interest Group Discourse Dialogue, Columbus, OH, 2008, pp. 172-181.
128. M. Walker, D. Litman, C. Kamm, and A. Abella, "Paradise: A framework for evaluating spoken dialogue agents," in Proc. Annu. Meeting Assoc. Comput. Linguist., Madrid, Spain, 1997, pp. 271-280.
129. E. Levin, R. Pieraccini, and W. Eckert, "Learning dialogue strategies within the Markov decision process framework," in Proc. IEEE Workshop Autom. Speech Recognit. Understand., Santa Barbara, CA, 1997, pp. 72-79.
130. S. Singh, M. Kearns, D. Litman, and M. Walker, "Reinforcement learning for spoken dialogue systems," in Proc. Neural Inf. Process. Syst., 1999.
131. T. Watambe, M. Araki, and S. Doshita, "Evaluating dialogue strategies under communication errors using computer-to-computer simulation," IEICE Trans. Inf. Syst., vol. E81-D, no. 9, pp. 1025-1033, 1998.
132. H. Ai and F. Weng, "User simulation as testing for spoken dialog systems," in Proc. Annu. Meeting Special Interest Group Discourse Dialogue, Columbus, OH, 2008, pp. 164-171.
133. J. Schatzmann, "Statistical user and error modelling for spoken dialogue systems," Ph.D. dissertation, Engineering Department, Univ. Cambridge, Cambridge, U.K., 2008.
134. M. Gašíc, F. Lefevre, F. Jurčíček, S. Keizer, F. Mairesse, B. Thomson, K. Yu, and S. Young, "Back-off action selection in summary space-based POMDP dialogue systems," in Proc. IEEE Workshop Autom. Speech Recognit. Understand., Merano, Italy, 2009, pp. 456-461.
135. F. Jurčíček, S. Keizer, M. Gašíc, F. Mairesse, B. Thomson, K. Yu, and S. Young, "Real user evaluation of spoken dialogue systems using amazon mechanical turk," in Proc. Annu. Conf. Int. Speech Commun. Assoc., Florence, Italy, 2011, pp. 3061-3064.
136. I. McGraw, C. Lee, L. Hetherington, S. Seneff, and J. Glass, "Collecting voices from the cloud," in Proc. Int. Conf. Lang. Resources Eval., Malta, 2010, pp. 1576-1583.
137. B. Zhang, Q. Cai, J. Mao, E. Chang, and B. Guo, "Spoken dialogue management as planning and acting under uncertainty," in Proc. EUROSPEECH, Aalborg, Denmark, 2001, pp. 2169-2172.
138. E. Levin and R. Pieraccini, "A stochastic model of computer-human interaction for learning dialogue strategies," in Proc. EUROSPEECH, Rhodes, Greece, 1997, pp. 1883-1886.
139. E. Levin, R. Pieraccini, and W. Eckert, "Using Markov decision processes for learning dialogue strategies," in Proc. Int. Conf. Acoust. Speech Signal Process., Seattle, WA, 1998, pp. 201-204.
140. M. Walker, J. Fromer, and S. Narayanan, "Learning optimal dialogue strategies: A case study of a spoken dialogue agent for email," in Proc. Annu. Meeting Assoc. Comput. Linguist./COLING, Montreal, QC, Canada, 1998, pp. 1345-1351.
141. M. Kearns, D. Litman, and M. Walker, "Automatic detection of poor speech recognition at the dialogue level," in Proc. Annu. Meeting Assoc. Comput. Linguist., College Park, MD, 1999, pp. 309-316.
142. D. Goddeau and J. Pineau, "Fast reinforcement learning of dialog strategies," in Proc. Int. Conf. Acoust. Speech Signal Process., Istanbul, Turkey, 2000, pp. 1233-1236.
143. S. Young, "Probabilistic methods in spoken dialogue systems," Philosoph. Trans. Roy. Soc. A, vol. 358, no. 1769, pp. 1389-1402, 2000.
144. S. Singh, D. Litman, M. Kearns, and M. Walker, "Optimizing dialogue management with reinforcement learning: Experiments with the NJFun system," J. Artif. Intell. Res., vol. 16, pp. 105-133, 2002.
145. M. Denecke, K. Dohsaka, and M. Nakano, "Learning dialogue policies using state aggregation in reinforcement learning," in Proc. Annu. Conf. Int. Speech Commun. Assoc., Jeju Island, Korea, 2004, pp. 325-328.
146. M. Denecke, K. Dohsaka, and M. Nakano, "Fast reinforcement learning of dialogue policies using stable function approximation," in Proc. Int. Joint Conf. Natural Lang. Process., 2005, DOI: 10.1007/978-3-540-30211-7-1.
147. O. Pietquin, "A probabilistic description of man-machine spoken communication," in Proc. IEEE Int. Conf. Multimedia Expo, Amsterdam, The Netherlands, 2005, pp. 410-413.
148. J. Tetreault and D. Litman, "Comparing the utility of state features in spoken dialogue using reinforcement learning," in Proc. Conf. Human Lang. Technol. North Amer. Chapter Assoc. Comput. Linguist., New York, 2006, pp. 272-279.
149. J. Tetreault and D. Litman, "Using reinforcement learning to build a better model of dialogue state," in Proc. European Association Computational Linguistics, Trento, Italy, 2006.
150. P. Heeman, "Combining reinforcement learning with information-state update rules," in Proc. Conf. Human Lang. Technol. North Amer. Chapter Assoc. Comput. Linguist., Rochester, NY, 2007, pp. 268-275.
151. O. Pietquin, "Learning to ground in spoken dialogue systems," in Proc. Int. Conf. Acoust. Speech Signal Process., Honolulu, HI, 2007, pp. 165-168.
152. S. Pulman, "Conversational games, belief revision and Bayesian networks," in Proc. CLIN VII: 7th Comput. Linguist. Meeting, The Netherlands, 1996.
153. D. Heckerman and E. Horwitz, "Inferring informational goals from free-text queries: A Bayesian approach," in Proc. Conf. Uncertainty Artif. Intell., Madison, WI, 1998, pp. 230-238.
154. E. Horvitz and T. Paek, "A Computational architecture for conversation," in Proc. 7th Int. Conf. User Model., Banff, AB, Canada, 1999, pp. 201-210.
155. H. Meng, C. Wai, and R. Pieraccini, "The use of belief networks for Mixed-Initiative dialog modeling," IEEE Trans. Speech Audio Process., vol. 11, no. 6, pp. 757-773, Nov. 2003.