Multimodal information presentation: Design guidance and research challenges

International Journal of Industrial Ergonomics

Volumn 36, Issue 5, 2006, Pages 439-445

  Sarter, Nadine B ^a  

^a UNIVERSITY OF MICHIGAN   (United States)

Author keywords

Adaptation and adaptability; Crossmodal links in attention; Design guidance; Multimodal information presentation; Touch

Indexed keywords

DISPLAY DEVICES; INFORMATION ANALYSIS; MODAL ANALYSIS; PATTERN RECOGNITION; VIRTUAL REALITY;

ADAPTATION AND ADAPTABILITY; CROSSMODAL LINKS IN ATTENTION; DESIGN GUIDANCE; MULTIMODAL INFORMATION PRESENTATION; MULTIMODAL INTERFACES; TOUCH;

INTERFACES (COMPUTER);

ARTICLE; COMPUTER INTERFACE; ENVIRONMENT; IMMERSION; INFORMATION PROCESSING; METHODOLOGY; PRIORITY JOURNAL; RESEARCH; SENSORY SYSTEM; SMELLING; TIME; TOUCH; VIRTUAL REALITY;

EID: 33646108361     PISSN: 01698141     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.ergon.2006.01.007     Document Type: Article

References (67)

1. Andre A.D., and Wickens C.D. When users want what's not best for them. Ergonomics in Design (1995) 10-14
2. Barfield W., and Danas E. Comments on the Use of Olfactory Displays for Virtual Environments. Presence 5 1 (1996) 109-121
3. Benoit C., and Le Goff B. Audio-visual speech synthesis from French text: eight years of models, designs and evaluation at the ICP. Speech Communication 26 (1998) 117-129
4. Boulter L.R. Attention and reaction times to signals of uncertain modality. Journal of Experimental Psychology: Human Perception and Performance 3 (1977) 379-388
5. Brickman B.J., Hettinger L.J., and Haas M.W. Multisensory interface design for complex task domains: replacing information overload with meaning in tactical crew stations. International Journal of Aviation Psychology 10 3 (2000) 273-290
6. Buisine, S., Martin, J.-C., 2003. Design principles for cooperation between modalities in bi-directional multimodal interfaces. In: Proceedings of the CHI 2003 workshop on "Principles for multimodal user interface design", Ft. Lauderdale, FL.
7. Calvert, G.A., Brammer, M.J., Bullmore, E.T., Campbell, R., Iversen, S.D., David, A.S., 1999. Response amplification in sensory-specific cortices during crossmodal binding. NeuroReport 10, 2619-2623.
8. Cholewiak R., and Collins A. Sensory and physiological basis of touch. In: Heller M., and Schiff W. (Eds). The psychology of touch (1991), LEA, Mahwah, NJ 23-60
9. Cohen P.R., and Oviatt S.L. The role of voice in human-machine communication. In: Roe D., and Wilpon J. (Eds). Voice Communication Between Humans and Machines (1994), National Academy of Sciences Press, Washington, DC 34-75
10. Cohen, P.R., Johnston, M., McGee, S., Oviatt, S., Pittman, J., 1997. Quickset: Multimodal interaction for simulation set-up and control. In: Proceedings of the Fifth Applied Natural Language Processing Meeting. Washington, DC.
11. Dingus T.A., and Hulse M.C. Some human factors design issues and recommendations for automobile navigation information systems. Transportation Research 1C2 (1993) 119-131
12. Driver, J., Grossenbacher, P.G., 1996. Multimodal spatial constraints on tactile selective attention. In: Innui, T., McClelland, J.L., (Eds.), Attention and Performance: Information integration in perception and Communication, vol. 16. MIT Press, Cambridge, MA, pp. 209-235.
13. Driver J., and Spence C. Multisensory perception: Beyond modularity and convergence. Current Biology 10 (2000) R731-R735
14. Driver J., and Spence C. Crossmodal spatial attention: evidence from human performance. In: Spence C., and Driver J. (Eds). Crossmodal Space and Crossmodal Attention (2004), Oxford University Press, Oxford, UK 179-220
15. Eimer M. Electrophysiological studies of multisensory attention. In: Calvert G.A., Spence C., and Stein B.E. (Eds). The Handbook of Multisensory Processes (2004), M.I.T. Press, Cambridge, MA 549-562
16. Eimer M., and Driver J. Crossmodal links in endogenous and exogenous spatial attention: evidence from event-related brain potential studies. Neuroscience and Biobehavioral Reviews 25 6 (2001) 497-511
17. ETSI (European Telecommunications Standards Institute), 2003. Multimodal interaction, communication and navigation guidelines (Report ETSI EG 202 191 V1.1.1 (2003-08). ETSI, Sophia Antipolis, France.
18. Ferstl R., Hanewinkel R., and Krag P. Is the modality-shift effect specific for schizophrenic patients?. Schizophrenia Bulletin 2 (1994) 367-373
19. Fitch, T., Kramer, G., 1994. Sonifying the body electric: Superiority of an auditory over a visual display in a complex, multi-variate system. In: Kramer, G., (Ed.), Auditory display: Sonification, audification and auditory interfaces. Proceedings of the First International Conference on Auditory Display (ICAD) 1992. Addison-Wesley, Reading, MA, pp. 307-326.
20. Gibson J.J. Observations on active touch. Psychological Review 69 (1962) 477-490
21. Goldsmith M., and Yeari M. Modulation of object-based attention by spatial focus under endogenous and exogenous orienting. Journal of Experimental Psychology-Human Perception and Performance 29 5 (2003) 897-918
22. Hale K.S., and Stanney K.M. Deriving haptic design guidelines from human physiological, psychophysical, and neurological foundation. IEEE Computer Graphics and Applications 24 2 (2004) 33-39
23. Ho, C.-Y., Sarter, N.B., 2004. Supporting synchronous distributed communication and coordination through multimodal information exchange. In: Proceedings of the 48th Annual Meeting of the Human Factors and Ergonomics Society, New Orleans, LS.
24. Ho, C.-Y., Nikolic, M.I., Sarter, N.B., 2001. Supporting timesharing and interruption management through multimodal information presentation. In: Proceedings of the 45th Annual Meeting of the Human Factors and Ergonomics Society, Minneapolis, MN.
25. ISO (International Organization for Standardization), 1998. ISO 14915 Multimedia user interface design software ergonomic requirements, Part 3: Media combination and selection, author.
26. Kaye, J.N., 2001. Symbolic olfactory displays. Unpublished Master of Science thesis, Massachusetts Institute of Technology, Cambridge, Boston, MA. Available: http://web.media.mit.edu/~jofish/thesis/.
27. Kennett S., Eimer M., Spence C., and Driver J. Tactile-visual links in exogenous spatial attention under different postures: Convergent evidence from psychophysics and ERPs. Journal of Cognitive Neuroscience 13 4 (2001) 462-478
28. Knasko S.C., and Gilbert A.N. Emotional state, physical well-being, and performance in the presence of feigned ambient odor. Journal of Applied Social Psychology 20 16 (1990) 1345-1357
29. Kramer G. Auditory display: Sonification, audification, and auditory interfaces (1993), Perseus Publishing
30. Kramer, G., Walker, B., Bonebright, T., Cook, P., Flowers, J., Miner, N., Neuhoff, J., 1999. Sonification report: Status of the field and research agenda. Report prepared for the National Science Foundation by members of the International Community for Auditory Display.
31. Krueger M.W. Olfactory stimuli in virtual reality medical training (1995), Artificial Reality Corporation, Vernon, CT
32. LaBerge D. Attentional Processing (1995), Harvard University Press, Cambridge, MA
33. Latorella K.A. Investigating Interruptions: Implications for Flightdeck Performance (NASA/TM-1999-209707) (1999), NASA Langley Research Center, Hampton, VA
34. Lee J.D., Caven B., Haake S., and Brown T.L. Speech-based interaction with in-vehicle computers: the effects of speech-based e-mail on drivers' attention to the road. Human Factors 43 4 (2001) 631-640
35. Macaluso E., Frith C.D., and Driver J. Directing attention to locations and to sensory modalities: multiple levels of selective processing revealed with PET. Cerebral Cortex 12 (2002) 357-368
36. Means L.G., Fleischman R.N., Carpenter J.T., Szczublewski F.E., Dingus T.A., and Krage M.K. Design of TravTek auditory interface. In NRC- TRB, Driver performance: measurement and modeling (1993), National Academy Press, Washington, DC
37. Michaelis P.R., and Wiggins R.H. A human factors engineer's introduction to speech synthesizers. In: Badre A., and Shneiderman B. (Eds). Directions in Human-Computer Interaction (1982), Ablex, Norwood, NJ 149-178
38. Mollenhauer M.A., Hulse M.C., Dingus T.A., Jahns S.K., and Carney C. Design decision aids and human factors guidelines for ATIS displays. In: Noy Y.I. (Ed). Ergonomics and Safety of Intelligent Driver Interfaces (1997), LEA, Mahwah, NJ 23-61
39. Nass C., and Lee K.N. Does computer-synthesized speech manifest personality? Experimental tests of recognition, similarity-attraction, and consistency-attraction. Journal of Experimental Psychology: Applied 7 3 (2001) 171-181
40. Nikolic M.I., and Sarter N.B. Peripheral visual feedback: A powerful means of supporting attention allocation and human-automation coordination in highly dynamic data-rich environments. Human Factors 43 1 (2001) 30-38
41. Oviatt, S.L., 1997. Multimodal interactive maps: Designing for human performance. Human-Computer Interaction (special issue on Multimodal Interfaces), 93-129.
42. Oviatt S. Multimodal interfaces. In: Jacko J., and Sears A. (Eds). Handbook of Human-Computer Interaction (2002), LEA, Hillsdale, NJ
43. Oviatt S.L., and Cohen P.R. Multimodal systems that process what comes naturally. Communications of the ACM 43 3 (2000) 45-53
44. Pentland, A., 1996. Smart room. Scientific American 54-62.
45. Posner M.I., and Petersen S.E. The attention system of the human brain. Annual Review of Neuroscience 13 (1990) 25-42
46. Post L.J., and Chapman C.E. The effects of cross-modal manipulations of attention on the detection of vibrotactile stimuli in humans. Somatosensory and Motor Research 8 2 (1991) 149-157
47. Raman T.V. XForms-XML powered web forms (2003), Addison Wesley, Reading, MA
48. Reeves L.M., Lai J., Larson J.A., Oviatt S., Balaji T.S., Buisine S., Collings P., Cohen P., Kraal B., Martin J.-C., McTear M., Raman T.V., Stanney K.M., Su H., and Wang Q.-Y. Guidelines for multimodal user interface design. Communications of the ACM 47 1 (2004) 57-59
49. Sarter N.B. The need for multisensory feedback in support of effective attention allocation in highly dynamic event-driven environments: The case of cockpit automation. International Journal of Aviation Psychology 10 3 (2000) 231-245
50. Sarter N.B. Multimodal information presentation in support of human-automation communication and coordination. In: Salas E. (Ed). Advances in human performance and cognitive engineering research (2002), JAI Press, New York 13-36
51. Sklar A.E., and Sarter N.B. Good vibrations: the use of tactile feedback in support of mode awareness on advanced technology aircraft. Human Factors 41 4 (1999) 543-552
52. Spence C., and Driver J. Crossmodal links in attention between audition, vision, and touch: Implications for interface design. International Journal of Cognitive Ergonomics 1 4 (1997) 351-373
53. Spence C., and McDonald J. The cross-modal consequences of the exogenous spatial orienting of attention. In: Calvert G.A., Spence C., and Stein B.E. (Eds). The Handbook of Multisensory Processes (2004), M.I.T. Press, Cambridge, MA 3-26
54. Spence C., Pavani F., and Driver J. Crossmodal links between vision and touch in covert endogenous spatial attention. Journal of Experimental Psychology: Human Perception and Performance 26 (2000) 1298-1319
55. In: Stork D.G., and Hennecke M.E. (Eds). Speechreading by Humans and Machines (1995), Springer, New York
56. Strayer D.L., Drews F.A., and Johnston W.A. Cell phone-induced failures of visual attention during simulated driving. Journal of Experimental Psychology: Applied 9 1 (2003) 23-32
57. Sutcliffe A. Multimedia and virtual reality: designing multisensory user interfaces (2003), LEA, Mahwah, NJ
58. Theeuwes J. Endogenous and exogenous control of visual selection. Perception 23 4 (1994) 429-440
59. Treviranus, J., 1999. Adding feeling, sound, and equal access to distance education. In: Proceedings of CSUN conference on "Technology and Persons with Disabilities". Los Angeles.
60. In: Turk M., and Robertson G. (Eds). Perceptual user interfaces. Communications of the ACM 43 3(special issue) (2000) 32-70
61. Vo, M.T., Wood, C., 1996. Building an application framework for speech and pen input integration in multimodal learning interfaces. In: Proceedings of the International Conference on Acoustics, Speech, and Signal Processing, pp. 3545-3548.
62. Wang, J., 1995. Integration of eye-gaze, voice and manual response in multimodal user interface. In: Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics, pp. 3938-3942.
63. Wickens C.D. Engineering Psychology and Human Performance. second ed (1992), Harper Collins Publishers, New York
64. Woods, D.D., 1993. The price of flexibility. In: Proceedings of the First International Conference on Intelligent User Interfaces. Orlando, FL, pp. 19-25.
65. Woods, D.D., Johannesen, L., Potter, S., 1994. The sophistry of guidelines: Revisiting recipes for color use in human-computer interface design. In: Proceedings of the 36th Annual Meeting of the Human Factors and Ergonomics Society.
66. Zhai, S., Morimoto, C., Ihde, S., 1999. Manual and gaze input cascaded (MAGIC) pointing. In: Proceedings of the Conference on Human Factors in Computing Systems (CHI'99), ACM Press, New York, pp. 246-253.
67. Zubin J. Problems of attention in schizophrenia. In: Kietzman M.L., Sutton S., and Zubin J. (Eds). Experimental Approaches to Psychopathology (1975), Academic Press, New York 139-166