From a Wizard of Oz experiment to a real time speech and gesture multimodal interface

Signal Processing

Volumn 86, Issue 12, 2006, Pages 3559-3577

  Carbini, S ^a   Delphin Poulat, L ^a   Perron, L ^a   Viallet, J E ^a  

^a ORANGE LABS   (France)

Author keywords

Bimanual gesture, speech recognition; Context dependent interpretation; Head, hands detection, tracking; Interpersonal communication; Mediated collaboration; Non intrusive multimodal interface; Non verbal behaviour

Indexed keywords

BIMANUAL GESTURE; CONTEXT DEPENDENT INTERPRETATION; INTERPERSONAL COMMUNICATION; MEDIATED COLLABORATION; NONINTRUSIVE MULTIMODAL INTERFACE; NONVERBAL BEHAVIOUR;

GESTURE RECOGNITION; HUMAN COMPUTER INTERACTION; INTERFACES (COMPUTER); MODAL ANALYSIS; REAL TIME SYSTEMS; TRACKING (POSITION);

SPEECH RECOGNITION;

EID: 33749439566     PISSN: 01651684     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.sigpro.2006.04.001     Document Type: Article

References (32)

1. S. Oviatt, P. Cohen, L. Wu, J. Vergo, L. Duncan, B. Suhm, T. Holzman, T. Winograd, J. Landay, J. Larson, D. Ferro, Designing the user interface for multimodal speech and gesture applications: state-of-the-art systems and research directions, in: R. (Ed.), Human Computer Interaction, Addison-Wesley Press, MA, 2001, pp. 263-322 (chapter 19).
2. L. Perron, An avatar with your own gestures, Appropriate Methodology for Empirical Studies of Privacy, INTERACT'05 Workshop, pp. 12-16, Rome, Italy, 2005.
3. Buisine S., Martin J.C., and Bernsen N.O. Children's gesture and speech in conversation with 3D characters. International Conference on Human-Computer Interaction. Las Vegas, NV, USA (2005)
4. Silva A., Raimundo G., and Paiva A. Tell me that bit again...Bringing interactivity to a virtual storyteller. International Conference on Virtual Storytelling. Toulouse, France (2003) 146-154
5. Corradini A., and Cohen P.R. On the relationships among speech, gestures, and object manipulation in virtual environments: initial evidence. International CLASS Workshop on Natural, Intelligent and Effective Interaction in Multimodal Dialogue Systems. Copenhagen, Denmark (2002) 52-61
6. Bolt R.A. "Put-that-there". voice and gesture at the graphics interface. Proceedings of the Seventh Annual Conference on Computer Graphics and Interactive Techniques. Washington, Seattle, WA, USA (1980) 262-270
7. Kaiser E., Olwal A., McGee D., Benko H., Corradini A., Li X., Cohen P., and Feiner S. Mutual disambiguation of 3D multimodal interaction in augmented and virtual reality. Proceedings of the Fifth International Conference on Multimodal. Vancouver, Canada (2003) 12-19
8. Stiefelhagen R., Fuegen C., Gieselmann P., Holzapfel H., Nickel K., and Waibel A. Natural human-robot interaction using speech, gaze and gestures. IEEE/RSJ International Conference on Intelligent Robots and Systems. Sendai, Japan (2004) 2422-2427
9. Sharma R., Yeasin M., Krahnstoever N., Rauschert I., Cai G., Brewer I., MacEachren A., and Sengupta K. Speech-gesture driven multimodal interfaces for Crisis Management. Proc. IEEE 91 9 (2003) 1327-1354
10. Chen F., Choi E., Epps J., Lichman S., Ruiz N., Shi Y., Taib R., and Wu M. A study of manual gesture-based selection for the PEMMI multimodal transport management interface. International Conference on Multimodal Interfaces (ICMI). Trento, Italy (2005) 274-281
11. Wang S.B., and Demirdjian D. Inferring body pose using speech content. International Conference on Multimodal Interfaces (ICMI). Trento, Italy (2005) 53-60
12. Epps J., Oviatt S., and Chen F. Integration of speech and gesture inputs during multimodal interaction. Australian International Conference on Computer-Human Interaction. Wollongong, Australia (2004)
13. Rimé B., and Schiaratura L. Gesture and speech. In: Feldman, and Rimé (Eds). Fundamentals of Non Verbal Behavior (1991), Press Syndicate of University of Cambridge, Cambridge 239-281
14. Atienza R., and Zelinsky A. Interactive skills using active gaze tracking. International Conference on Multimodal Interfaces (ICMI). Vancouver, Canada (2003) 188-195
15. Nickel K., Seemann E., and Stiefelhagen R. 3D-tracking of head and hands for pointing gesture recognition in a human-robot interaction scenario. IEEE International Conference on Automatic Face and Gesture Recognition. Seoul, Korea (2004) 565-570
16. Kato T., Kurata T., and Sakaue K. Vizwear-active: distributed Monte Carlo face tracking for wearable active camera. International Conference on Pattern Recognition, Quebec, Canada vol. 1 (2002) 395-400
17. Jojic N., Brumitt B., Meyers B., and Harris S. Detecting and estimating pointing gestures in dense disparity maps. IEEE International Conference on Face and Gesture Recognition. Grenoble, France (2000) 468-475
18. Yamamoto Y., Yoda I., and Sakaue K. Arm-pointing gesture interface using surrounded stereo cameras system. International Conference on Pattern Recognition (ICPR). Cambridge, UK (2004) 965-970
19. Demirdjian D., and Darrell T. 3-D articulated pose tracking for untethered diectic reference. International Conference on Multimodal Interfaces (ICMI). Pittsburgh, PA, USA (2002) 267
20. PointGrey. 〈http://www.ptgrey.com/products/bumblebee/index.html〉.
21. Feraud R., Bernier O., Viallet J.E., and Collobert M. A fast and accurate face detector based on neural networks. Pattern Anal. Mach. Intell. 23 1 (2001) 42-53
22. Viola P.A., and Jones M.J. Robust real-time face detection. Internat. J. Comput. Vision 57 2 (2004) 137-154
23. Garcia C., and Delakis M. Convolutional face finder: a neural architecture for fast and robust face detection. Pattern Anal. Mach. Intell. 26 11 (November 2004) 1408-1423
24. Carbini S., Viallet J.E., and Bernier O. Pointing gesture visual recognition for large display. Pointing'04 International Conference on Pattern Recognition Workshop. Cambridge, UK (2004)
25. Dempster A.P., Laird N.M., and Rubin D.B. Maximum likelihood from incomplete data via the EM algorithm. J. Roy. Statist. Soc. Ser. B 39 1 (1977) 1-38
26. Bartkova K., and Jouvet D. Modelization of allophones in a speech recognition system. International Congress of Phonetic Science (ICPhS). Aixen-Provence, France (1991) 474-477
27. Schwartz R., and Chow Y.-L. The N-best algorithm. an efficient and exact procedure for finding the N most likely sentence hypothesis. International Conference on Acoustic Speech and Signal Processing (ICASSP). Albuquerque, NM, USA (1990) 81-84
28. Mauuary L., and Monné J. Speech/non-speech detection for voice response systems. European Conference on Speech Communications and Technology (EUROSPEECH). Berlin, Germany (1993) 1097-1100
29. Eisenstein J., and Christoudias C.M. A salience-based approach to gesture-speech alignment. HLT-NAACL Conference. Boston, MA, USA (2004) 25-32
30. Bellik Y. Technical requirements for a successful multimodal interaction. International Workshop on Information Presentation and Natural Multimodal Dialogue. Verona, Italy (2001)
31. Xboard. 〈http://www.tim-mann.org/xboard.html〉.
32. Carbini S., Viallet J.E., Bernier O., and Bascle B. Tracking body parts of multiple people for multi-person multimodal interface, Computer Vision in Human-Computer Interaction. ICCV 2005 Workshop. Beijing, China (2005) 16-25