The PARADISE evaluation framework: Issues and findings

Computational Linguistics

Volumn 32, Issue 2, 2006, Pages 263-272

  Hajdinjak, Melita ^a,b   Mihelič, France ^a,b  

^a UNIVERSITY OF LJUBLJANA   (Slovenia)

^b Faculty for Electrical Engineering   (Slovenia)

Author keywords

[No Author keywords available]

Indexed keywords

SPEECH PROCESSING;

DEPENDENT VARIABLES; EVALUATION FRAMEWORK; GENERAL METHODOLOGIES; NORMALIZATION EFFECT; PERFORMANCE FUNCTIONS; REGRESSION PARAMETERS; SPOKEN LANGUAGE DIALOGUE SYSTEM; USER SATISFACTION;

SPEECH RECOGNITION;

EID: 33745816702     PISSN: 08912017     EISSN: 15309312     Source Type: Journal    
DOI: 10.1162/coli.2006.32.2.263     Document Type: Review

References (17)

1. Danieli, Morena and Elisabetta Gerbino. 1995. Metrics for evaluating dialogue strategies in a spoken language system. In Proceedings of the 1995 AAAI Spring Symposium on Empirical Methods in Discourse Interpretation and Generation, pages 34-39, Stanford.
2. Di Eugenio, Barbara and Michael Glass. 2004. The Kappa statistic: A second look. Computational Linguistics, 30(1):95-101.
3. Hajdinjak, Melita and France Mihelič. 2004a. Conducting the Wizard-of-Oz experiment. Informatica, 28(4): 425-430.
4. Hajdinjak, Melita and France Mihelič. 2004b. Information-providing dialogue management. In Proceedings of TSD, pages 595-602, Brno.
5. Hone, Kate S. and Robert Graham. 2000. Towards a tool for the Subjective Assessment of Speech System Interfaces (SASSI). Natural Language Engineering: Special Issue on Best Practice in Spoken Dialogue Systems, 6(3-4):287-303.
6. Kamm, Candace A., Marilyn A. Walker, and Diane J. Litman. 1999. Evaluating spoken language systems. In Proceedings of American Voice Input/Output Society, San Jose.
7. Litman, Diane J. and Pan Shimei. 2002. Designing and evaluating an adaptive spoken dialogue system. User Modeling and User-Adapted Interaction, 12:111-137.
8. Patel, Nitin R. and Peter C. Bruce. 1995. Resampling Stats - Data Mining in Excel: Lecture Notes and Cases. Trafford Holdings Ltd., Victoria, Canada.
9. Seber, George A. F. 1977. Linear Regression Analysis. John Wiley & Sons, New York.
10. Shriberg, Elizabeth, Elizabeth Wade, and Patti Price. 1992. Human-machine problem solving using Spoken Language Systems (SLS): Factors affecting performance and user satisfaction. In Proceedings of the DARPA Speech and NL Workshop, pages 49-54, New York.
11. Smith, Ronnie W. and Steven A. Gordon. 1997. Effects of variable initiative on linguistic behavior in human-computer spoken natural language dialog. Computational Linguistics, 23(1):141-168.
12. Tabachnick, Barbara G. and Linda S. Fidell. 1996. Using Multivariate Statistics, 3rd ed. Harper Collins, New York.
13. Walker, Marilyn A., Julie Borland, and Candace A. Kamm. 1999. The utility of elapsed time as a usability metric for spoken dialogue systems. In Proceedings of ASRU, pages 317-320, Keystone.
14. Walker, Marilyn A., Candace A. Kamm, and Diane J. Litman. 2000. Towards developing general models of usability with PARADISE. Natural Language Engineering: Special Issue on Best Practice in Spoken Dialogue Systems, 6(3-4):363-377.
15. Walker, Marilyn A., Diane J. Litman, Candace A. Kamm, and Alicia Abella. 1997. PARADISE: A general framework for evaluating spoken dialogue agents. In Proceedings of the 35th Annual Meeting of ACL, pages 271-280, Madrid.
16. Walker, Marilyn A., Diane J. Litman, Candace A. Kamm, and Alicia Abella. 1998. Evaluating spoken dialogue agents with PARADISE: Two case studies. Computer Speech and Language, 12(3):317-347.
17. Žibert, Janez, Sanda Martinčić-Ipšić, Melita Hajdinjak, Ivo Ipšić, and France Mihelič. 2003. Development of a bilingual spoken dialog system for weather information retrieval. In Proceedings of EUROSPEECH, pages 1917-1920, Geneva.