A survey of speech-hand gesture recognition for the development of multimodal interfaces in computer games

2010 IEEE International Conference on Multimedia and Expo, ICME 2010

Volumn , Issue , 2010, Pages 1564-1569

  Liu, Jing ^a   Kavakli, Manolya ^a  

^a MACQUARIE UNIVERSITY   (Australia)

Author keywords

Computer games; Hand gestures; Multimodal interfaces; Speech

Indexed keywords

APPLICATION AREA; COMPUTER GAME; HAND GESTURE; HAND-GESTURE RECOGNITION; HUMAN COMMUNICATIONS; KEY ISSUES; MULTI-MODAL INTERFACES;

COMPUTER SOFTWARE; GESTURE RECOGNITION; HUMAN COMPUTER INTERACTION; INTERACTIVE COMPUTER SYSTEMS; SPEECH COMMUNICATION; SPEECH RECOGNITION; SURVEYS;

INTERFACES (COMPUTER);

EID: 78349281859     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICME.2010.5583252     Document Type: Conference Paper

References (43)

1. AG Hauptmann, "Speech and gestures for graphic image manipulation," ACM SIGCHI Bulletin, vol. 20, pp. 241- 245, 1989.
2. RA Bolt, "put-that-there: Voice and gesture at the graphics interface," 1980, pp. 262-270, ACM New York, NY, USA.
3. S Oviatt, P Cohen, L Wu, L Duncan, B Suhm, J Bers, T Holzman, T Winograd, J Landay, and J Larson, "Designing the user interface for multimodal speech and penbased gesture applications: State-of-the-art systems and future research directions," Human-Computer Interaction, vol. 15, no. 4, pp. 263-322, 2000.
4. K Moustakas, G Nikolakis, D Tzovaras, B Deville, I Marras, and J Pavlek, "Multimodal tools and interfaces for the intercommunication between visually impaired and deaf and mute people," Proceedings of eNTERFACE, 2006.
5. JD Bolter and R Grusin, Remediation, MIT Press, 1999.
6. F Maddix, Human-computer interaction: theory and practice, Ellis Horwood Ltd, 1990.
7. Zhang Jie, Zhao Ji, Bai Shuanhu, and Huang Zhiyong, "Applying speech interface to mahjong game," in Multimedia Modelling Conference, 2004. Proceedings. 10th International, Brisbane, Australia, 2004, pp. 86-92.
8. MP Johnson, A Wilson, B Blumberg, C Kline, and A Bobick, "Sympathetic interfaces: using a plush toy to direct synthetic characters," 1999, pp. 152-158, ACM New York, NY, USA.
9. Lee Kue-Bum, Kim Jung-Hyun, and Hong Kwang-Seok, "An implementation of multi-modal game interface based on pdas," in Software Engineering Research, Management & Applications 2007. SERA 2007. 5th ACIS International Conference on, Busan, Korea, 2007, pp. 759-768.
10. J Payne, P Keir, J Elgoyhen, M McLundie, M Naef, M Horner, and P Anderson, "Gameplay issues in the design of spatial 3d gestures for video games," in CHI'06 extended abstracts on Human factors in computing systems. 2006, p. 1222, ACM.
11. H Kang, CWoo Lee, and K Jung, "Recognition-based gesture spotting in video games," Pattern Recognition Letters, vol. 25, no. 15, pp. 1701-1714, 2004.
12. S Carbini, L Delphin-Poulat, L Perron, and JE Viallet, "From a wizard of oz experiment to a real time speech and gesture multimodal interface," Signal Processing, vol. 86, no. 12, pp. 3559-3577, 2006.
13. RA Cole, J Mariani, H Uszkoreit, A Zaenen, and V Zue, Survey of the state of the art in human language technology, Cambridge University Press Cambridge, UK, 1997.
14. Lee Yong-Hee, Lee Dong-Woo, Choi Eun-Jung, and Park Jun-Seok, "Design of multimodal interface framework," in Advanced Communication Technology, The 9th International Conference on, Phoenix Park, Korea, 2007, vol. 1, pp. 345-348.
15. S Oviatt, "Multimodal system processing in mobile environments," 2000, pp. 21-30, ACM New York, NY, USA.
16. S Carbini, L Delphin-Poulat, L Perron, and JE Viallet, "From a wizard of oz experiment to a real time speech and gesture multimodal interface," Signal Processing, vol. 86, no. 12, pp. 3559-3577, 2006.
17. M Paleari and CL Lisetti, "Toward multimodal fusion of affective cues," 2006, pp. 99-108, ACM New York, NY, USA.
18. I Wachsmuth, "Communicative rhythm in gesture and speech," ADVANCES IN CONSCIOUSNESS RESEARCH, vol. 35, pp. 117-132, 2002.
19. S. S. Fels and G. E. Hinton, "Glove-talkii-a neuralnetwork interface which maps gestures to parallel formant speech synthesizer controls," Neural Networks, IEEE Transactions on, vol. 9, no. 1, pp. 205-212, 1998.
20. Y Kuno, T Ishiyama, K Jo, N Shimada, and Y Shirai, "Vision-based human interface system: Selectively recognizing intentional hand gestures," 1998, pp. 219-223.
21. DM Krum, O Omoteso, W Ribarsky, T Starner, and LF Hodges, "Speech and gesture multimodal control of a whole earth 3d visualization environment," 2002, pp. 195- 200, Eurographics Association Aire-la-Ville, Switzerland, Switzerland.
22. Data Gloves, http://www.vrealities.com/glove.html.
23. Xu Zhang, Xiang Chen, Wen-hui Wang, Ji-hai Yang, Vuokko Lantz, and Kong-qiao Wang, "Hand gesture recognition and virtual game control based on 3d accelerometer and emg sensors," 2009.
24. M. Kavakli, D. Richards, M. Dras, and J. Porte, "An immersive virtual reality training simulation for risk management," SimTecT 2007: Simulation Conference: Simulation - Improving Capability and Competitiveness, pp. 1-6, 2007.
25. J Yoon, S Kim, J Ryu, and W Woo, "Multimodal gumdo game: The whole body interaction with an intelligent cyber fencer," Lecture Notes in Computer Science, pp. 1088- 1095, 2002.
26. HS Park, DJ Jung, and HJ Kim, "Vision-based game interface using human gesture," Lecture Notes in Computer Science, vol. 4319, pp. 662, 2006.
27. H Sakoe, R Isotani, K Yoshida, KI Iso, and T Watanabe, "Speaker-independent word recognition using dynamic programmingneural networks," in Acoustics, Speech, and Signal Processing, 1989. ICASSP-89., 1989 International Conference on, 1989, pp. 29-32.
28. E Trentin andMGori, "A survey of hybrid ann/hmm models for automatic speech recognition," Neurocomputing, vol. 37, no. 1, pp. 91-126, 2001.
29. A Erol, G Bebis,MNicolescu, RD Boyle, and X Twombly, "Vision-based hand pose estimation: A review," Computer Vision and Image Understanding, vol. 108, no. 1-2, pp. 52-73, 2007.
30. R. Sharma, M. Yeasin, N. Krahnstoever, I. Rauschert, G. Cai, I. Brewer, A. M. MacEachren, and K. Sengupta, "Speech-gesture driven multimodal interfaces for crisis management," Proceedings of the IEEE, vol. 91, no. 9, pp. 1327-1354, 2003.
31. Sebastien Marcel Agnes Just, "A comparative study of two state-of-the-art sequence processing techniques for hand gesture recognition," Comput. Vis. Image Underst., vol. 113, no. 4, pp. 532-543, 2009.
32. P Hong, M Turk, and T Huang, "Constructing finite state machines for fast gesture recognition," in INTERNATIONAL CONFERENCE ON PATTERN RECOGNITION, Barcelona, Spain, 2000, vol. 15, pp. 691-694.
33. S. Kettebekov, M. Yeasin, and R. Sharma, "Prosody based co-analysis for continuous recognition of coverbal gestures," in Multimodal Interfaces, 2002. Proceedings. Fourth IEEE International Conference on, Pittsburgh, USA, 2002, pp. 161-166.
34. L Kratz, M Smith, and FJ Lee, "Wiizards: 3d gesture recognition for game play input," in Proceedings of the 2007 conference on Future Play. 2007, pp. 209-212, ACM.
35. DL Bruno Dumas and S Oviatt, "Multimodal interfaces: A survey of principles, models and frameworks," Human Machine Interaction: Research Results of the MMI Program.
36. LWu, SL Oviatt, and PR Cohen, "Multimodal integrationa statistical view," IEEE Transactions on Multimedia, vol. 1, no. 4, pp. 334-341, 1999.
37. David McNeill, Hand and Mind: what Gestures Reveal about Thought, University of Chicago Press, 1992.
38. Jacob Eisenstein and C. Mario Christoudias, "A saliencebased approach to gesture-speech alignment," in IN PROCEEDINGS OF HLT/NAACL04. 2004, pp. 25-32, ACL Press.
39. M Renwick, S Shattuck-Hufnagel, and Y Yasinnik, "The timing of speech-accompanying gestures with respect to prosody," The Journal of the Acoustical Society of America, vol. 115, pp. 2397, 2004.
40. M Johnston, PR Cohen, D McGee, SL Oviatt, JA Pittman, and I Smith, "Unification-based multimodal integration," 1997, pp. 281-288, Association for Computational Linguistics Morristown, NJ, USA.
41. M Johnston and S Bangalore, "Finite-state multimodal parsing and understanding," in Proceedings of the 18th conference on Computational linguistics-Volume 1. 2000, pp. 369-375, Association for Computational Linguistics Morristown, NJ, USA.
42. JY Chai, P Hong, MX Zhou, and Z Prasov, "Optimization in multimodal interpretation," 2004, Association for Computational Linguistics Morristown, NJ, USA.
43. Microsoft Natal Project, http://www.groundreport.com/Lifestyle/Microsoft- Natal-Project/2900419.