Evaluating spoken dialogue agents with PARADISE: Two case studies

Computer Speech and Language

Volumn 12, Issue 4, 1998, Pages 317-347

  Walker, M A ^a   Litman, D J ^a   Kamm, C A ^a   Abella, A ^a  

^a AT AND T LABS RESEARCH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER SYSTEMS PROGRAMMING; OPTIMIZATION; SPEECH ANALYSIS; SPEECH PROCESSING;

PARADIGM FOR DIALOGUE SYSTEM EVALUATION (PARADISE);

NATURAL LANGUAGE PROCESSING SYSTEMS;

EID: 0032180111     PISSN: 08852308     EISSN: None     Source Type: Journal    
DOI: 10.1006/csla.1998.0110     Document Type: Article

References (67)

1. Abella, A., Brown, M. K. & Buntschuh, B. (1996). Development principles for dialog-based interfaces. ECAI-96 Spoken Dialog Processing Workshop, Hungary, 1996.
2. Allen, J. F. & Perrault, C. R. (1980). Analyzing intention in utterances. Artificial Intelligence, 15, 143-178.
3. Barto, A. G., Bradtke, S. J. & Singh, S. P. (1995). Learning to act using real-time dynamic programming. Artificial Intelligence, 72, 81-138.
4. Bates, M. & Ayuso, D. (1991). A proposal for incremental dialogue evaluation. Proceedings of the DARPA Speech and Natural Language Workshop, Pacific Grove, CA, pp 319-322.
5. Bellman, R. E. (1957). Dynamic Programming. Princeton University Press, Princeton, N.J.
6. Bernsen, N. O., Dybkjaer, H. & Dybkjaer, L. (1996). Principles for the design of cooperative spoken human-machine dialogue. International Conference on Spoken Language Processing, ICSLP 96, Philadelphia, PA, pp. 729-732.
7. Biermann, A. W. & Long, P. M. (1996). The composition of messages in speech-graphics interactive systems. Proceedings of the 1996 International Symposium on Spoken Dialogue, Philadelphia, PA, pp. 97-100.
8. Boyce, S. & Gorin, A. L. (1996). User interface issues for natural spoken dialogue systems. Proceedings of the 1996 International Symposium on Spoken Dialogue, Philadelphia, PA, pp. 65-68.
9. Carberry, S. (1989). Plan recognition and its use in understanding dialogue. In User Models in Dialogue Systems, (A. Kobsen & W. Wahlster, eds) pp. 133-162. Springer Verlag, Berlin.
10. Carletta, J. C. (1996). Assessing the reliability of subjective codings. Computational Linguistics, 22, 249-254.
11. Chu-Carroll, J. & Carberry, S. (1995). Response generation in collaborative negotiation. 33rd Annual Meeting of the Association for Computational Linguistics, Cambridge, MA, pp. 136-143.
12. Ciaremella, A. (1993). A prototype performance evaluation report. Technical Report Project Esprit 2218 SUNDIAL, WP8000-D3.
13. Cohen, P. R. (1995). Empirical Methods for Artificial Intelligence. MIT Press, Boston.
14. Danieli, M., Eckert, W., Fraser, N., Gilbert, N., Guyomard, M., Heisterkamp, P., Kharoune, M., Magadur, J., McGlashan, S., Sadek, D., Siroux, J. & Youd, N. (1992) Dialogue manager design evaluation. Technical Report Project Esprit 2218 SUNDIAL, WP6000-D3.
15. Danieli, M. & Gerbino, E. (1995). Metrics for evaluating dialogue strategies in a spoken language system. Proceedings of the 1995 AAAI Spring Symposium on Empirical Methods in Discourse Interpretation and Generation, Stanford, CA, pp. 34-39.
16. Doyle, J. (1992). Rationality and its roles in reasoning. Computational Intelligence, 8, 376-409.
17. Fraser, N. M. (1995). Quality standards for spoken dialogue systems: a report on progress in EAGLES. ESCA Workshop on Spoken Dialogue Systems Visgo, Denmark, pp. 157-160.
18. Gale, W., Church, K. W. & Yarowsky, D. (1992). Estimating upper and lower bounds on the performance of word-sense disambiguation programs. 30th Annual Meeting of the Association for Computational Linguistics, Newark, Delaware, pp. 249-256.
19. Geelhoed, E., Toft, P., Roberts, S. & Hyland, P. (1995). To influence time perception. Proceedings of the Conference on Computer Human Interaction, Denver, CO.
20. Goddeau, D., Meng, H., Polifroni, J., Seneff, S. & Busayapongchai, S. (1996). A form-based dialogue manager for spoken language applications. Fourth International Conference on Spoken Language Processing, Philadelphia, PA.
21. Goldberg, D. (1989). Genetic Algorithms in Search, Optimization and Machine Learning. Addison-Wesley.
22. Grice, H. P. (1967). Logic and conversation. William James Lecture.
23. Grosz, B. J. & Sidner, C. L. (1986). Attentions, intentions and the structure of discourse. Computational Linguistics, 12, 175-204.
24. Hirschberg, J. & Nakatani, C. (1996). A prosodic analysis of discourse segments in direction-giving monologues. 34th Annual Meeting of the Association for Computational Linguistics, Santa Cruz, CA, pp. 286-293.
25. Hirschman, L., Dahl, D. A., McKay, D. P., Norton, L. M. & Linebarger, M. C. (1990). Beyond class A: A proposal for automatic evaluation of discourse. Proceedings of the Speech and Natural Language Workshop, Hidden Valley, PA, pp. 109-113.
26. Hirschman, L., Bates, M., Dahl, D., Fisher, W., Garofolo, J., Pallett, D., Hunicke-Smith, K., Price, P., Rudnicky, A. & Tzoukermann, E. (1993). Multi-site data collection and evaluation in spoken language understanding. Proceedings of the Human Language Technology Workshop, Princeton, NJ, pp. 19-24.
27. Hirschman, L. & Pao, C. (1993). The cost of errors in a spoken language system. Proceedings of the Third European Conference on Speech Communication and Technology, Berlin, Germany, pp. 1419-1422.
28. Joshi, A. K., Webber, B. L. & Weischedel, R. M. (1984). Preventing false inferences. COLING84: Proc. 10th International Conference on Computational Linguistics, Stanford, CA, pp. 134-138.
29. Kamm, C. (1995). User interfaces for voice applications. In Voice Communication between Humans and Machines (D. Roe and J. Wilpon, eds), Washington, DC, pp. 422-442.
30. Kamm, C., Narayanan, S., Dutton, D. & Ritenour, R. (1997). Evaluating spoken dialog systems for telecommunication services. 5th European Conference on Speech Technology and Communication, EUROSPEECH 97. Rhodes, Greece.
31. Keeny, R. & Raiffa, H. (1976). Decisions with Multiple Objectives: Preferences and Value Tradeoffs. John Wiley and Sons.
32. Korth, H. (1991). Database System Concepts. McGraw-Hill, Berkeley, CA.
33. Krippendorf, K. (1980). Content Analysis: An Introduction to its Methodology. Sage Publications, CA.
34. Lee, K. F. (1988). Large Vocabulary Speaker-Independent Continuous Speech Recognizer: the Sphinx System. PhD thesis, Carnegie Mellon University.
35. Levin, E. & Pieracinni, R. (1997). A stochastic model of computer-human interaction for learning dialogue strategies. EUROSPEECH 97. Rhodes, Greece.
36. Litman, D. & Allen, J. (1990). Recognizing and relating discourse intentions and task-oriented plans. In Intentions in Communication (P. Cohen, J. Morgan & M. Pollack, eds). MIT Press. Cambridge, MA.
37. Litman, D., Pan, S. & Walker, M. (1998). Evaluating response strategies in a web-based spoken dialogue agent. Proceedings of ACL/COLING 98: 36th Annual Meeting of the Association of Computational Linguistics. Montreal, Canada.
38. Meng, H., Busayapongchai, S., Glass, J., Goddeau, D., Hetherington, L., Hurley, E., Pao, C., Polifroni, J., Seneff, S. & Zue, V. (1996). Wheels: A conversational system in the automobile classifieds domain. Proceedings of the Fourth International Conference on Spoken Language Processing. Philadelphia, PA.
39. Pallet, D. S. (1985). Performance assessment of automatic speech recognizers. Journal of Research of National Bureau of Standards, 90, 371-387.
40. Passonneau, R. J. & Litman, D. (1997). Discourse segmentation by human and automated means. Computational Linguistics, 23.
41. Pieraccini, R., Levin, E. & Eckert, W. (1997). The AT&T mixed initiative conversational architecture. Eurospeech.
42. Polifroni, J., Hirschman, L., Seneff, S. & Zue, V. (1992). Experiments in evaluating interactive spoken language systems. Proceedings of the DARPA Speech and NL Workshop, Harriman, NY, pp. 28-33.
43. Pollack, M., Hirschberg, J. & Webber, B. (1982). User participation in the reasoning process of expert systems. Proceedings National Conference on Artificial Intelligence, Pittsburgh, PA, pp. 358-361.
44. Price, P., Hirschman, L., Shriberg, E. & Wade, E. (1992). Subject-based evaluation measures for interactive spoken language systems. Proceedings of the DARPA Speech and NL Workshop, Harriman, NY, pp. 34-39.
45. Rabiner, L. R., Juang, B. H. & Lee, C. H. (1996). An overview of automatic speech recognition. In Automatic Speech and Speaker Recognition, Advanced Topics (C. H. Lee, F. K. Soong & K. K. Paliwal, eds), pp. 1-30. Kluwer Academic Publishers.
46. Ralston, J. V., Pisoni, D. B. & Mullennix, J. W. (1994). Perception and comprehension of speech. In Applied Speech Technology (A. K. Syrdal, R. W. Bennet & S. L. Greenspan, eds), pp. 233-287. CRC Press, Boca Raton.
47. Russell, S. & Norvig, P. (1995). Artificial Intelligence: A Modern Approach. Prentiss Hall, Englewood Cliffs, NJ.
48. Seneff, S., Zue, V., Polifroni, J., Pao, C., Hetherington, L., Godeau, D. & Glass, J. (1995). The preliminary development of a displayless PEGASUS system. ARPA Spoken Language Technology Workshop. Austin, TX.
49. Shriberg, E., Wade, E. & Price, P. (1992). Human-machine problem solving using spoken language systems (SLS): Factors affecting performance and user satisfaction. Proceedings of the DARPA Speech and NL Workshop, Harriman, NY, pp. 49-54.
50. Siegel, S. & Castellan, N. J. (1988). Nonparametric Statistics for the Behavioral Sciences. McGraw Hill.
51. Simpson, A. & Fraser, N. A. (1993). Black box and glass box evaluation of the SUNDIAL system. Proceedings of the Third European Conference on Speech Communication and Technology, pp. 1423-1426.
52. Smith, R. W. & Hipp, D. R. (1994). Spoken Natural Language Dialog Systems: A Practical Approach. Oxford University Press.
53. Smith, R. W. & Gordon, S. A. (1997). Effects of variable initiative on linguistic behavior in human-computer spoken natural language dialog. Computational Linguistics, 23.
54. Sparck-Jones, K. & Galliers, J. R. (1996). Evaluating Natural Language Processing Systems. Springer, Berlin.
55. Sproat, R. & Olive, J. (1995). An approach to text-to-speech synthesis. In Speech Coding and Synthesis (W. B. Kleijn & K. K. Paliwal, eds), pp. 611-633. Elsevier.
56. Sutton, R. S. (1991). Planning by incremental dynamic programming. Proceedings Ninth Conference on Machine Learning, pp. 353-357. Morgan-Kaufmann.
57. Tesauro, G. (1992). Practical Issues in Temporal Difference Learning. Machine Learning, 8, 257-277.
58. Walker, M. A. (1989). Evaluating discourse processing algorithms. Proceedings 27th Annual Meeting of the Association of Computational Linguistics, Vancouver, Canada, pp. 251-261.
59. Walker, M. A. (1993). Informational Redundancy and Resource Bounds in Dialogue. PhD thesis, University of Pennsylvania.
60. Walker, M. A. (1996). The effect of resource limits and task complexity on collaborative planning in dialogue. Artificial Intelligence, 85, 181-243.
61. Walker, M. A. & Whittaker, S. (1990). Mixed initiative in dialogue: An investigation into discourse segmentation. Proceedings of the 28th Annual Meeting of the ACL, Pittsburgh, PA, pp. 70-79.
62. Walker, M., Fromer, J., Di Fabbrizio, G., Mestel, C. & Hindle, D. (1998). What can I say: Evaluating a spoken language interface to email. Proceedings of the Conference on Computer Human Interaction (CHI 98). Los Angeles, CA
63. Walker, M., Fromer, J. & Narayanan, S. (1998). Learning optimal dialogue strategies: A case study of a spoken dialogue agent for email. ACL/COLING 98: Proceedings of the 36th Annual Meeting of the Association of Computational Linguistics. Montreal, Canada
64. Walker, M. A., Litman, D., Kamm, C. & Abella, A. (1997). PARADISE: A general framework for evaluating spoken dialogue agents. Proceedings of the 35th Annual Meeting of the Association of Computational Linguistics, ACL/EACL 97, Madrid, Spain, pp. 271-280.
65. Walker, M., Hindle, D., Fromer, J., Di Fabbrizio, G. & Mestel, C. (1997). Evaluating competing agent strategies for a voice email agent. Proceedings of the European Conference on Speech Communication and Technology, EUROSPEECH97. Rhodes, Greece.
66. Watkins, C. J. (1989). Models of Delayed Reinforcement Learning. PhD thesis, Cambridge University.
67. Webber, B. & Joshi, A. (1982). Taking the initiative in natural language database interaction: Justifying why. Coling 82, pp.413-419.