Enabling multimodal human-robot interaction for the Karlsruhe humanoid robot

IEEE Transactions on Robotics

Volumn 23, Issue 5, 2007, Pages 840-851

  Stiefelhagen, Rainer ^a   Ekenel, Hazim Kemal ^a   Fügen, Christian ^a   Gieselmann, Petra ^b   Holzapfel, Hartwig ^a   Kraft, Florian ^a   Nickel, Kai ^a   Voit, Micheal ^a   Waibel, Alex ^a,c  

^a UNIVERSITY OF KARLSRUHE   (Germany)

^b Lucy Software and Services GmbH   (Germany)

^c CARNEGIE MELLON UNIVERSITY   (United States)

Author keywords

Audiovisual perception; Human centered robotics; Human robot interaction; Multimodal interaction

Indexed keywords

COMPUTER VISION; GESTURE RECOGNITION; MAN MACHINE SYSTEMS; SPEECH RECOGNITION; TRACKING (POSITION);

AUDIOVISUAL PERCEPTION; HUMAN-CENTERED ROBOTICS; HUMANOID ROBOT; LOCALIZATION; MULTIMODAL INTERACTION;

ROBOTICS;

EID: 35348860213     PISSN: 15523098     EISSN: None     Source Type: Journal    
DOI: 10.1109/TRO.2007.907484     Document Type: Article

References (57)

1. The SFB 588 Website. (2004). [Online]. Available: http:// www.sfb588.uni-karlsruhe.de/
2. "Special Issue on Human-Friendly Robots," J. Robot. Soc. Jpn., vol. 16, no. 3, 1998.
3. K. Akachi, K. Kaneko, N. Kanehira, S. Ota, G. Miyamori, M. Hirata, S. Kajita, and F. Kanehiro, "Development of humanoid robot HRP-3," in Proc. IEEE-RAS Int. Conf. Humanoid Robots, Dec. 2005, pp. 50-55.
4. R. Ambrose, H. Aldridge, R. Askew, R. Burridge, W. Bluethmann, M. Diftler, C. Lovchik, D. Magruder, and F. Rehnmark, "Robonaut: NASA's space humanoid," IEEE Intell. Syst. Appl., vol. 15, no. 4, pp. 57-63, Jul. 2000.
5. M. Argyle, Social Interaction. London, U.K.: Methuen, 1969.
6. T. Asfour, K. Regenstein, P. Azad, J. Schröder, A. Bierbaum, N. Vahrenkamp, and R. Dillmann, "ARMAR-III: An integrated humanoid platform for sensory-motor control," in Proc. IEEE-RAS Int. Conf. Humanoid Robots (Humanoids 2006), Genoa, Italy, Dec. 4-6, pp. 169-175.
7. I. Bakx, K. van Turnhout, and J. Terken, "Facial orientation during multi-party interaction with information kiosks," presented at the Interact 2003, Zurich, Switzerland.
8. P. N. Belhumeur, J. P. Hespanha, and D. J. Kriegman, "Eigenfaces vs. Fisherfaces: Recognition using class specific linear projection," IEEE Trans. Pattern Anal. Mach. Intell., vol. 19, no. 7, pp. 711-720, Jul. 1997.
9. C. Breazeal, "Social interactions in HRI: The robot view," IEEE Trans. Syst., Man, Cybern. C, Appl. Rev., vol. 34, no. 2, pp. 181-186, May 2004.
10. B. Brumitt and J. J. Cadiz, "Let there be light: Comparing interfaces for homes of the future," Microsoft Res., Microsoft Corp., Redmond, WA, Tech. Rep. MSR-TR-2000-92, Sep. 21, 2000.
11. J. Chen, J. Benesty, and Y. Huang, "Robust time delay estimation exploiting redundancy among multiple microphones," IEEE Trans. Speech Audio Process., vol. 11, no. 6, pp. 549-557, Nov. 2003.
12. K. Dautenhahn, M. Walters, S. Woods, K. L. Koay, C. L. Nehaniv, A. Sisbot, R. Alami, and T. Simeon, "How may i serve you?: A robot companion approaching a seated person in a helping context," in Proc. 1st ACM SIGCHI/SIGART Conf. Human Robot Interact. New York: ACM Press, 2006, pp. 172-179.
13. M. Denecke, "Rapid prototyping for spoken dialogue systems," in Proc. 19th Int. Conf. Comput. Linguist., Taipei, Taiwan, 2002, vol. 1, pp. 1-7.
14. H. Ekenel and R. Stiefelhagen, "A generic face representation approach for local appearance based face verification," presented at the Workshop Face Recognit. Grand Challenge Exp., 2005.
15. H. K. Ekenel and A. Pnevmatikakis, "Video-based face recognition evaluation in the CHIL project - run 1," in Proc. Int. Conf. Autom. Face Gesture Recognit., 2006, pp. 85-90.
16. H. K. Ekenel and R. Stiefelhagen, "Local appearance-based face recognition using discrete cosine transform," presented at the 13th Eur. Signal Process. Conf., Antalya, Turkey, 2005.
17. H. Ekenel and R. Stiefelhagen, "Analysis of local appearance-based face recognition on FRGC 2.0 database," presented at the Face Recogit. Grand Challenge Workshop (FRGC), Arlington, VA, Mar. 2006.
18. T. Fong, I. Nourbakhsh, and K. Dautenhahn, "A survey of socially interactive robots," Robot. Auton. Syst., vol. 42, pp. 143-166, 2003.
19. C. Fügen, H. Holzapfel, and A. Waibel, "Tight coupling of speech recognition and dialog management - dialog-context dependent grammar weighting for speech recognition," in Proc. Int. Conf. Spoken Lang. Process., Jeju-Islands, Korea, 2004.
20. Y. Gao, H. Erdogan, Y. Li, V. Goel, and M. Picheny, "Recent advances in speech recognition system for IBM DARPA communicator," presented at the Eurospeech, Aalborg, Denmark, Sep. 2001.
21. N. Gourier, D. Hall, and J. L. Crowley, "Estimating face orientation from robust detection of salient facial features," presented at the Pointing 2004, ICPR, Int. Workshop Vis. Obs. Deictic Gestures, Cambridge, U.K., 2004.
22. H. Holzapfel, "Building multilingual spoken dialogue systems," in Archives of Control Sci., Special Issue on Human Language Technologies as a Challenge for Computer Science and Linguistics (Part II), vol. 15, no. 4, pp. 555-566, 2005.
23. H. Holzapfel, K. Nickel, and R. Stiefelhagen, "Implementation and evaluation of a constraint based multimodal fusion system for speech and 3D pointing gestures," presented at the Int. Conf. Multimodal Interfaces, State College, PA, 2004.
24. H. Holzapfel, T. Schaaf, H. Ekenel, C. Schaa, and A. Waibel, "A robot learns to know people - first contacts of a robot," in Proc. 29th German Conf. Artif. Intell. (KI 2006), Lect. Notes Artif. Intell., Bremen, Germany.
25. H. Holzapfel and A. Waibel, "A multilingual expectations model for contextual utterances in mixed-initiative spoken dialogue," in Proc. Interspeech, Pittsburgh, PA, 2006.
26. M. Isard and A. Blake, "Condensation - conditional density propagation for visual tracking," Int. J. Comput. Vis., vol. 29, no. 1, pp. 5-28, 1998.
27. J. W. Tankard, "Effects of eye position on person perception," Percept. Mot. Skills, vol. 31, no. 3, pp. 883-893, 1970.
28. M. Katzenmaier, R. Stiefelhagen, T. Schultz, I. Rogina, and A. Waibel, "Identifying the addressee in human-human-robot interactions based on head pose and speech," presented at the Int. Conf. Multimodal Interfaces, State College, PA, Oct. 2004.
29. D. Kee, G. Wyeth, A. Hood, and A. Drury, "GuRoo: Autonomous humanoid platform for walking gait research," presented at the Conf. Auton. Minirobots Res. Edutainment, Brisbane, Australia, Feb. 2003.
30. J. Kittler, M. Hatef, R. P. W. Dum, and J. Maras, "On combining classifiers," IEEE Trans. Pattern Anal. Mach. Intell., vol. 20, no. 3, pp. 226-239, Mar. 1998.
31. C. L. Kleinke, A. A. Bustos, F. B. Meeker, and R. A. Staneski, "Effects of self-attributed and other-attributed gaze in interpersonal evaluations between males and females," J. Exp. Soc. Psychol., no. 9, pp. 154-163, 1973.
32. F. Kraft, R. Malkin, T. Schaaf, and A. Waibel, "Temporal ICA for classification of acoustic events in a kitchen environment," presented at the Interspeech, Lisbon, Portugal, 2005.
33. J. Lee, I. Park, J. Kim, and J. Oh, "Mechanical design of humanoid robot platform KHR-3 (Kaist humanoid robot-3: Hubo)," in Proc. IEEE-RAS Int. Conf. Humanoid Robots, Dec. 2005, pp. 321-326.
34. O. Lemon, "Context-sensitive speech recognition in ISU-dialogue systems: Results for the grammar-switching approach," presented at the Catalog'04, 8th Workshop Semantics Pragmatics Dialogue, Barcelona, Spain.
35. R. Lienhart and J. Maydt, "An extended set of Haar-like features for rapid object detection," in Proc. Int. Conf. Image Process., Sep. 2002, vol. 1, pp. 900-903.
36. P. P. Maglio, T. Matlock, C. S. Campbell, S. Zhai, and B. A. Smith, "Gaze and speech in attentive user interfaces," in Proc. Int. Conf. Multimodal Interfaces, Lect. Notes Comput. Sci. London, U.K.: Springer-Verlag, 2000, vol. 1948, pp. 1-7.
37. F. Metze, Q. Jin, C. Fugen, Y. Pan, and T. Schultz, "Issues in meeting transcription - The Meeting Transcription System," presented at the Interspeech 2004 - 4CSLP, Jeju Island, Korea, Oct.
38. B. Moghaddam, T. Jebara, and A. Pentland, "Bayesian face recognition," Pattern Recogit., vol. 33, no. 11, pp. 1771-1782, 2000.
39. K. Nickel, T. Gehrig, R. Stiefelhagen, and J. McDonough, "A joint particle filter for audio-visual speaker tracking," in Proc. 7th Int. Conf. Multimodal Interfaces, Trento, Italy, Oct. 4-6, 2005, pp. 61-68.
40. K. Nickel and R. Stiefelhagen, "Visual recognition of pointing gestures for human-robot interaction," Image Vis. Comput., to be published.
41. K. Nishiwaki, T. Sugihara, S. Kagami, F. Kanehiro, M. Inaba, and H. Inoue, "Design and development of research platform for perception action integration in humanoid robots: H6," in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst., 2000, pp. 1559-1564.
42. M. Omologo and P. Svaizer, "Acoustic event localization using a crosspower-spectrum phase based technique," in Proc. Int. Conf. Acoust. Speech Signal Process., Adelaide, Australia, Apr. 1994, pp. II-273-II-276.
43. B. Pellom, W. Ward, and S. Pradhan, "The cu communicator: An architecture for dialogue systems," presented at the Int. Conf. Spoken Lang. Process., Beijing, China, Oct. 2000.
44. T. Prommer, H. Holzapfel, and A. Waibel, "Rapid simulation-driven reinforcement learning of multimodal dialog strategies in human-robot interaction," presented at the Interspeech (Int. Conf. Spoken Lang. Process.), 2006.
45. J. Ruusuvuori, "Looking means listening: Coordinating displays of engagement in doctor-patient interaction," Soc. Sci. Med., vol. 52, pp. 1093-1108, 2001.
46. S. Sakagami, T. Watanabe, C. Aoyama, S. Matsunage, N. Higaki, and K. Fujimura, "The intelligent ASIMO: System overview and integration," in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst., 2002, pp. 2478-2483.
47. K. Scheffler and S. Young, "Automatic learning of dialogue strategy using dialogue simulation and reinforcement learning," presented at the Hum. Lang. Technol., San Diego, CA, 2002.
48. S. Seneff, R. Lau, and J. Polifroni, "Organization, communication, and control in the galaxy-II conversational system," presented at the Eurospeech, Budapest, Hungary, Sep. 1999.
49. R. Snelick, U. Uludag, A. Mink, M. Indovina, and A. K. Jain, "Large-scale evaluation of multimodal biometric authentication using state-of-the-art systems," IEEE Trans. Pattern Anal. Mach. Intell. vol. 27, no. 3, pp. 450-455, Mar. 2005.
50. H. Soltau, F. Metze, C. Migen, and A. Waibel, "A one pass-decoder based on polymorphic linguistic context assignment," presented at the Autom. Speech Recognit. Understanding, Madonna di Campiglio, Trento, Italy, Dec. 2001.
51. R. Stiefelhagen, K. Bernardin, R. Bowers, J. Garofolo, D. Mostefa, and P. Soundararajan, "The CLEAR 2006 evaluation," in Proc. 1st Int. CLEAR Eval. Workshop Multimodal Technol. Percept. Hum., 2006, Lect. Notes Comput. Sci., vol. 4122. London, U.K.: Springer-Verlag, 2007, pp. 1-45.
52. R. Stiefelhagen, "Tracking focus of attention in meetings," in Proc. Int. Conf. Multimodal Interfaces. Pittsburgh, PA: IEEE, Oct. 2002, pp. 273-280.
53. M. Turk and A. Pentland, "Eigenfaces for recognition," J. Cogn. Neursci., vol. 3, no. 1, pp. 71-86, 1991.
54. P. Viola and M. Jones, "Robust real-time object detection," presented at the ICCV Workshop Star. Comput. Theories Vis., San Diego, CA, Jul. 2001.
55. M. Voit, K. Nickel, and R. Stiefelhagen, "Neural network-based head pose estimation and multi-view fusion," presented at the 1st Int. CLEAR Eval. Workshop Multimodal Technol. Percept. Hum., 2006, Southampton, U.K.
56. Y.-Y. Wang, A. Acero, and C. Chelba, "Is word error rate a good indicator for spoken language understanding accuracy," presented at the Autom. Speech Recognit. Understanding, St. Thomas, U.S. Virgin Islands, 2003.
57. W. Zhao, R. Chellappa, P. J. Phillips, and A. Rosenfeld, "Face recognition: A literature survey," ACM Comput. Surv., vol. 35, no. 4, pp. 399-458, 2003.