Multimodal interaction: A review

Pattern Recognition Letters

Volumn 36, Issue 1, 2014, Pages 189-195

  Turk, Matthew ^a  

^a University of California   (United States)

Author keywords

Multimodal integration; Multimodal interaction; Perceptual interface; Review

Indexed keywords

INTEGRATION; INTERACTIVE COMPUTER SYSTEMS; REVIEWS;

DESKTOP COMPUTING; MULTI-MODAL; MULTI-MODAL INTERACTIONS; MULTIMODAL HUMAN COMPUTER INTERACTION; MULTIMODAL INTEGRATION; PERCEPTUAL INTERFACES; SENSORY INTEGRATION;

HUMAN COMPUTER INTERACTION;

EID: 84893689533     PISSN: 01678655     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.patrec.2013.07.003     Document Type: Review

References (66)

1. Andersen, RA., 1997. Multimodal integration for the representation of space in the posterior parietal cortex. Philos. Trans. R. Soc. Lond. B Biol. Sci. 352 (1360), 1421-1428. (Pubitemid 27482180)
2. Arabzadeh, E., Clifford, C.W., Harris, J.A., 2008. Vision merges with touch in a purely tactile discrimination. Psychol. Sci. 19 (7), 635-641.
3. Arnal, L.H., Morillon, B., Kell, CA., Giraud, A.L., 2009. Dual neural routing of visual facilitation in speech processing. J. Neurosci. 29 (43), 13445-13453.
4. Blattner, M.M., Glinert, E.P., 1996. Multimodal integration. IEEE Multimedia 3 (4), 14-24. (Pubitemid 126574420)
5. Bohus, D., Horvitz, E., 2010. Facilitating multiparty dialog with gaze, gesture, and speech. In: ACM International Conference on Multimodal Interfaces (ICMI-MLMI), Beijing, China.
6. Bolt, RA., 1980. "Put-that-there,": voice and gesture at the graphics interface. ACM Comput. Graphic. 14 (3), 262-270.
7. Bunt, H., Beun, R.-J., Borghuis, T., 1998. Multimodal human-computer communication systems, techniques, and experiments. Lect. Notes Comput. Sci. 1374.
8. Calvert, G., 2001. Crossmodal processing in the human brain: insights from functional neuroimaging studies. Cereb. Cortex 11 (12), 1110-1123. (Pubitemid 33108238)
9. Calvert, GA, Brammer, M.J., Iversen, S.D., 1998. Crossmodal identification. Trends Cogn. Sci. 2 (7), 247-253. (Pubitemid 28394242)
10. Campi, K.L., Bales, K.L., Grunewald, R., Krubitzer, L., 2010. Connections of auditory and visual cortex in the prairie vole (Microtus ochrogaster): evidence for multisensory processing in primary sensory areas. Cereb. Cortex 20 (1), 89-108.
11. Cappe, C, Thelen, A., Romei, V., Thut, G., Murray, M.M., 2012. Looming signals reveal synergistic principles of multisensory integration. J. Neurosci. 32 (4), 1171-1182.
12. Chai, J.Y., Hong, P., Zhou, MX, 2004. A probabilistic approach to reference resolution in multimodal user interfaces. In: ACM International Conference on Intelligent User, Interfaces, pp. 70-77. (Pubitemid 40673704)
13. Chang, J., Bourguet, M.-L, 2008. Usability framework for the design an devaluation of multimodal interaction. In: The 22nd British HCl Group Annual Conference on People and Computers: Culture, Creativity, Interaction, Vol. 2, pp. 123-126.
14. Chen, F., Ruiz, N., Choi, E., Epps, J., Khawaja, A., Taib, R., Yin, B., Wang, Y., 2012. Multimodal behaviour and interaction as indicators of cognitive load. ACM Trans. Interact. Intell. Syst. 2 (4).
15. Chen, T., 2001. Audiovisual speech processing. IEEE Signal Process. Mag., 9-21.
16. Coen, M.H., 2001. Multimodal integration - a biological view. Int. Joint Conf. Artif Intell. 17 (1), 1417-1424.
17. Cohen, P.R., Johnston, M., McGee, D., Oviatt, S., Pittman, J., Smith, I., Chen, L., Clow, J., 1997. QuickSet: multimodal interaction for distributed applications. In: ACM International Conference on Multimedia, Seattle, WA, pp. 31-40.
18. Coutaz, J., Salber, D., Balbo, S., 1993. Towards automatic evaluation of multimodal user interfaces. Knowl.-Based Syst. 6 (4), 267-274.
19. Cutugno, F., and Leano., V. A., Rinaldi, R., Mignini, G., 2012. Multimodal framework for mobile interaction. In: International Working Conference on Advanced Visual Interfaces, Naples, Italy, pp. 197-203.
20. Cytowic, R.E., 2002. Synesthesia: A Union of the Senses, 2nd ed. MIT Press, Cambridge, Massachusetts.
21. Dumas, B., Lalanne, D., Oviatt, S., 2009. Multimodal interfaces: a survey of principles, models and frameworks. Human Machine Interaction. Lect. Notes Comput. Sci. 5440, 3-26, Springer.
22. Falchier, A., Clavagnier, S., Barone, P., Kennedy, H., 2002. Anatomical evidence of multimodal integration in primate striate cortex. J. Neurosci. 22 (13), 5749-5759. (Pubitemid 35386498)
23. Gauglitz, S., Lee, C, Turk, M., Höllerer, T., 2012. Integrating the physical environment into mobile remote collaboration. In: ACM International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI), San Francisco, CA.
24. Jaimes, A., Sebe, N., 2007. Multimodal human-computer interaction: a survey. Comput. Vis. Image Underst. 108 (1-2), 116-134, Elsevier, Amsterdam. (Pubitemid 47333395)
25. Johnston, M., 1998. Unification-based multimodal parsing. In: The 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Vol. 1, pp. 624-630.
26. Johnston, M., Bangalore, S., 2005. Finite-state multimodal integration and understanding. Nat. Lang. Eng. 11 (2), 159-188.
27. Johnston, M., Cohen, P.R., McGee, D., Oviatt, S.L., Pittman, JA, Smith, I., 1997. Unification-based multimodal integration. In: 35th Annual Meeting of the Association for Computational Linguistics, Montreal, Quebec, Canada, pp. 281-288.
28. Koons, D., Sparrell, C, Thorisson, K., 1993. Integrating simultaneous input from speech, gaze, and hand gestures. In: Maybury, M. (Ed.), Intelligent Multimedia Interfaces. MIT Press, Cambridge, pp. 257-276.
29. Lalanne, D., Nigay, L, Robinson, P., Vanderdonckt, J., Ladry, J.F., 2009. Fusion engines for multimodal input: a survey. In: ACM International Conference on Multimodal Interfaces (ICMI-MLMI), Cambridge, MA, pp. 153-160.
30. Leeb, R., Lancelle, M., Kaiser, V., Fellner, D.W., Pfurtscheller, G., 2013. Thinking penguin: multi-modal brain-computer interface control of a VR game. IEEE Trans. Comput. Intell. AI Games.
31. Leitão, J., Thielscher, A, Werner, S., Pohmann, R., Noppeney, U., 2012. Effects of parietal TMS on visual and auditory processing at the primary cortical level - a concurrent TMS-fMRI study. Cereb. Cortex. Oxford University Press. Advance Access available April 5, 2012.
32. Lewkowicz, D.J., Kraebel, K., 2004. The value of multimodal redundancy in the development of intersensory perception. In: Calvert, G., Spence, C, Stein, B. (Eds.), Handbook of Multisensory Processing. MIT Press.
33. McGurk, H., MacDonald, J., 1976. Hearing lips and seeing voices. Nature, 746-748.
34. Mendonça, H., Lawson, J.Y.L, Vybornova, O., Macq, B., Vanderdonckt, J., 2009. A fusion framework for multimodal interactive applications. In: ACM International Conference on Multimodal Interfaces (ICMI-MLMI), Cambridge, MA, pp. 161-168.
35. Nakamura, S., 2002. Statistical multimodal integration for audio-visual speech processing. IEEE Trans. Neural Networks 13 (4), 854-866.
36. Neal, J.G., Thielman, C.Y., Dobes, Z., Haller, S.M., Shapiro, S.C., 1989. Natural language with integrated deictic and graphic gestures. In: ACL Workshop on Speech and Natural Language. Association for Computational Linguistics, Stroudsburg, PA.
37. Nielsen, J., 1986. A virtual protocol model for computer-human interaction. Int. J. Man Mach. Stud. 24 (3), 301-312. (Pubitemid 16587961)
38. Nigay, L., Coutaz, J., 1993. A design space for multimodal systems: concurrent processing and data fusion. In: INTERCHI: Conference on Human Factors in Computing Systems, Amsterdam, The Netherlands, pp. 172-178.
39. Oviatt, S., 1999. Ten myths of multimodal interaction. Commun. ACM 42 (11), 74-81. (Pubitemid 129576755)
40. Oviatt, S., 2003. Advances in robust multimodal interface design. IEEE Comput. Graphics Appl. 23 (5), 62-68.
41. Oviatt, S., Cohen, P., 2000. Perceptual user interfaces: multimodal interfaces that process what comes naturally. Commun. ACM 43 (3), 45-53.
42. Oviatt, S.L., Cohen, P.R., Wu, L., Vergo, J., Duncan, L., Suhm, B., Bers, J., Holzman, T., Winograd, T., Landay, J., Larson, J., Ferro, D., 2000. Designing the user interface for multimodal speech and gesture applications: state-of-the-art systems and research directions. Human Comput. Interact. 15 (4), 263-322. (Pubitemid 32371462)
43. Oviatt, S., Coulston, R., and Lundsford, R., 2004. When do we interact multimodally? Cognitive load and multimodal communication patterns. In: ACM International Conference on Multimodal Interfaces, State College, PA, pp. 129-136. (Pubitemid 40371901)
44. Oviatt, S., Lunsford, R., Coulston, R, 2005. Individual differences in multimodal integration patterns: what are they and why do they exist? In: ACM SIGCHI Conference on Human Factors in Computing Systems, Vol. 2, No. 7, pp. 241-249. (Pubitemid 41809061)
45. Portillo, P.M., Garcia, G.P., Carredano, G.A, 2006. Multimodal fusion: a new hybrid strategy for dialogue systems. In: ACM International Conference on Multimodal Interfaces, Banff, Canada, pp. 357-363. (Pubitemid 47113469)
46. Powers III, AR., Hevey, M.A., Wallace, M.T., 2012. Neural correlates of multisensory perceptual learning. J. Neurosci. 32 (18), 6263-6274.
47. Cmek, F., McNeill, D., Bryll, R., Duncan, S., Ma, X.F., Kirbas, C, McCullough, K.E., Ansari, R., 2002. Multimodal human discourse: gesture and speech. ACM Trans. Comput-Human Interact. 9 (3), 171-193.
48. Reeves, L.M., Lai, J., Larson, J.A., Oviatt, S., Balaji, T.S., Buisine, S., Collings, P., Cohen, P., Kraal, B., Martin, J.-C, McTear, M., Raman, T.V., Stanney, K.M., Su, H., Wang, Q.Y., 2004. Guidelines for multimodal user interface design. Commun. ACM 47 (1), 57-59.
49. Ruiz, N., Chen, F., Oviatt, S., 2010. Multimodal input. In: Thiran, J.P., Marques, F., Bourlard, H. (Eds.), Multimodal Signal Processing: Theory and Applications for Human-Computer Interaction. Academic Press, pp. 231-255, Chapter 12.
50. Song, Y., Morency, LP., Davis, R., 2012. Multimodal human behavior analysis: learning correlation and interaction across modalities. In: ACM International Conference on Multimodal Interfaces, Santa Monica, CA.
51. Stork, D.G., Hennecke, M.E., 1996. Speechreading by Humans and Machines. Springer.
52. Swallow, K.M., Makovski, T., Jiang, Y.V., 2012. Selection of events in time enhances activity throughout early visual cortex. J. Neurophysiol. 108, 3239-3252.
53. Turk, M., 1998. Moving from GUIs to PUIs. In: Symposium on Intelligent Information Media, Tokyo, Japan.
54. Turk, M., Robertson, G., 2000. Perceptual user interfaces. Commun. ACM 43 (3), 32-34.
55. Turk, M., Kölsch, M., 2004. Perceptual interfaces. In: Medioni, G., Kang, S.B. (Eds.), Emerging Topics in Computer Vision. Prentice Hall
56. van Dam, A., 1997. Post-WIMP user interfaces. Commun. ACM 40 (2), 63-67. (Pubitemid 127441124)
57. van Wassenhove, V., Grant, K.W., Poeppel, D., 2005. Visual speech speeds up the neural processing of auditory speech. Proc. Nat. Acad. Sci. 102, 1181-1186. (Pubitemid 40170746)
58. Vasconcelos, N., Pantoja, J., Belchior, H., Caixeta, F.V., Faber, J., Freire, M.A.M., Cota, V.R, de Macedo, E.A., Laplagne, D.A., Gomes, H.M., Ribeiro, S., 2011. Cross-modal responses in the primary visual cortex encode complex objects and correlate with tactile discrimination. Proc. Nat. Acad. Sci. 108 (37), 15408-15413.
59. Wahlster, W., 2003. Towards symmetric multimodality: fusion and fission of speech, gesture, and facial expression. In: The 26th German Conference on Artificial Intelligence, Hamburg, Germany, pp. 1-18, September. (Pubitemid 37171106)
60. Waibel, A., Vo, M.T., Duchnowski, P., Manke, S., 1996. Multimodal interfaces. Artif. Intell. Rev. 10 (3), 299-319. (Pubitemid 126706900)
61. Wasinger, R., 2006. Multimodal Interaction with Mobile Devices: Fusing a Broad Spectrum of Modality Combinations. IOS Press.
62. Werner, S., Noppeney, U., 2011. The contributions of transient and sustained response codes to audiovisual integration. Cereb. Cortex 21 (4), 920-931.
63. Wu, L, Oviatt, S.L., Cohen, P.R., 1999. Multimodal integration - a statistical view. IEEE Trans. Multimedia 1 (4), 334-341.
64. Xiao, B., Girand, C, Oviatt, S.L., 2002. Multimodal integration patterns in children. In: International Conference on Spoken Language Processing, pp. 629-632.
65. Xiao, B., Lunsford, R., Coulston, R, Wesson, M., Oviatt, S., 2003. Modeling multimodal integration patterns and performance in seniors: toward adaptive processing of individual differences. In: ACM International Conference on Multimodal Interfaces, Vancouver, Canada, pp. 265-272. (Pubitemid 40681881)
66. Zhou, W., Zhang, X., Chen, J., Wang, L., Chen, D., 2012. Nostril-specific olfactory modulation of visual perception in binocular rivalry. J. Neurosci. 32 (48), 17225-17229.