Comparing Objective and Subjective Measures of Usability in a Human-Robot Dialogue System

ACL-IJCNLP 2009 - Joint Conf. of the 47th Annual Meeting of the Association for Computational Linguistics and 4th Int. Joint Conf. on Natural Language Processing of the AFNLP, Proceedings of the Conf.

Volumn , Issue , 2009, Pages 879-887

  Foster, Mary Ellen ^a   Giuliani, Manuel ^a   Knoll, Alois ^a  

^a TECHNICAL UNIVERSITY OF MUNICH   (Germany)

Author keywords

[No Author keywords available]

Indexed keywords

ROBOTS; TOYS;

CONDITION; DIALOGUE SYSTEMS; EVALUATION FRAMEWORK; HUMAN USERS; HUMAN-ROBOT DIALOGUE; OBJECTIVE AND SUBJECTIVE MEASURES; PERFORMANCE FUNCTIONS; SUBJECTIVE AND OBJECTIVE MEASURES; USERS' SATISFACTIONS;

SPEECH PROCESSING;

EID: 85130574008     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (19)

1. M. Argyle and J. A. Graham. 1976. The Central Europe experiment: Looking at persons and looking at objects. Environmental Psychology and Nonverbal Behavior, 1(1):6–16. doi:10. 1007/BF01115461.
2. A. J. N. van Breemen. 2005. iCat: Experimenting with animabotics. In Proceedings of the AISB 2005 Creative Robotics Symposium.
3. C. Callison-Burch, M. Osborne, and P. Koehn. 2006. Re-evaluating the role of BLEU in machine translation research. In Proceedings of EACL 2006. ACL Anthology E06-1032.
4. M. E. Foster. 2008. Automated metrics that agree with human judgements on generated output for an embodied conversational agent. In Proceedings of INLG 2008. ACL Anthology W08-1113.
5. M. E. Foster, M. Giuliani, A. Isard, C. Matheson, J. Oberlander, and A. Knoll. 2009. Evaluating description and reference strategies in a cooperative human-robot dialogue system. In Proceedings of IJCAI 2009.
6. M. Huber, M. Rickert, A. Knoll, T. Brandt, and S. Glasauer. 2008. Human-robot interaction in handing-over tasks. In Proceedings of IEEE RO-MAN 2008. doi:10.1109/ROMAN.2008. 4600651.
7. C. Y. Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Proceedings of the ACL 2004 Workshop on Text Summarization. ACL Anthology W04-1013.
8. D. J. Litman and S. Pan. 2002. Designing and evaluating an adaptive spoken dialogue system. User Modeling and User-Adapted Interaction, 12(2–3):111–137. doi:10.1023/A: 1015036910358.
9. S. Möller, K.-P. Engelbrecht, and R. Schleicher. 2008. Predicting the quality and usability of spoken dialogue systems. Speech Communication, 50:730–744. doi:10.1016/j.specom. 2008.03.001.
10. D. G. Novick. 1997. What is effectiveness? In Working notes, CHI’97 Workshop on HCI Research and Practice Agenda Based on Human Needs and Social Responsibility. http://www.cs.utep.edu/novick/papers/eff.chi.html.
11. A. Ortony, G. L. Clore, and A. Collins. 1988. The Cognitive Structure of Emotions. Cambridge University Press.
12. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of ACL 2002. ACL Anthology P02-1040.
13. M. Rickert, M. E. Foster, M. Giuliani, T. By, G. Panin, and A. Knoll. 2007. Integrating language, vision and action for human robot dialog systems. In Proceedings of HCI International 2007. doi:10.1007/978-3-540-73281-5_108.
14. C. L. Sidner, C. Lee, C. D. Kidd, N. Lesh, and C. Rich. 2005. Explorations in engagement for humans and robots. Artificial Intelligence, 166(1–2):140–164. doi:10.1016/j.artint. 2005.03.005.
15. M. Walker, C. Kamm, and D. Litman. 2000. Towards developing general models of usability with PARADISE. Natural Language Engineering, 6(3–4):363–377.
16. M. A. Walker. 2000. An application of reinforcement learning to dialogue strategy selection in a spoken dialogue system for email. Journal of Artificial Intelligence Research, 12:387–416.
17. M. A. Walker, J. Fromer, G. D. Fabbrizio, C. Mes-tel, and D. Hindle. 1998. What can I say?: Evaluating a spoken language interface to email. In Proceedings of CHI 1998. doi:10.1145/274644.274722.
18. M. A. Walker, D. J. Litman, C. A. Kamm, and A. Abella. 1997. PARADISE: A framework for evaluating spoken dialogue agents. In Proceedings of ACL/EACL 1997. ACL Anthology P97-1035.
19. M. White, M. E. Foster, J. Oberlander, and A. Brown. 2005. Using facial feedback to enhance turn-taking in a multimodal dialogue system. In Proceedings of HCI International 2005.