When do we interact multimodally? Cognitive load and multimodal communication patterns

ICMI'04 - Sixth International Conference on Multimodal Interfaces

Volumn , Issue , 2004, Pages 129-136

  Oviatt, Sharon ^a   Coulston, Rachel ^a   Lunsford, Rebecca ^a  

^a OREGON HEALTH AND SCIENCE UNIVERSITY   (United States)

Author keywords

Cognitive load; Dialogue context; Human performance; Individual differences; Multimodal integration; Multimodal interaction; Speech and pen input; System adaptation; Task difficulty; Unimodal interaction

Indexed keywords

COGNITIVE SYSTEMS; COMPUTER AIDED INSTRUCTION; LINGUISTICS; MULTIMEDIA SYSTEMS; RELIABILITY; VISUAL COMMUNICATION;

INDIVIDUAL DIFFERENCES; MULTIMODAL INTEGRATION; MULTIMODAL INTERACTION; TASK DIFFICULTY;

SPEECH COMMUNICATION;

EID: 14944366296     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (29)

1. Almor, A., Noun-phrase anaphora and focus: The informational load hypothesis. Psychological Review, 1999. 106: 748-765.
2. Baddeley, A., Working Memory. Science, 1992. 255: 556-559.
3. Benoit, C., J.-C. Martin, C. Pelachaud, L. Schomaker, & B. Suhm, Audio-visual and multimodal speech-based systems, Handbook of Multimodal and Spoken Dialogue Systems: Resources, Terminology and Product Evaluation, R. Moore, ed. 2000, Kluwer Academic Publishers: Boston, MA. 102-203.
4. Calvert, G., C. Spence, & B.E. Stein, eds. The handbook of multisensory processing. 2004, MIT Press: Cambridge, MA.
5. Chandler, P. & J. Sweller, Cognitive load theory and the format of instruction. Cognition and Instruction, 1991. 8: 293-332.
6. Grant, K.W. & S. Greenberg. Speech intelligibility derived from asynchronous processing of auditory-visual information. Workshop on Audio-Visual Speech Processing (AVSP-2001). 2001. Scheelsminde, Denmark
7. Grice, H.P., Logic and conversation, Syntax and Semantics: Speech Acts, J.L. Morgan, ed. 1975, Acad Press: NY. 41-58.
8. Hinckley, K., Pierce J., E. Horvitz, & M. Sinclair, Foreground and background interaction with sensor-enhanced mobile devices. ACM Transactions on Computer Human Interaction, in press (Special Issue on Sensor-Based Interaction).
9. Jacko, J., L. Barnard, T. Kongnakorn, K. Moloney, P. Edwards, V. Emery, & F. Sainfort. Isolating the effects of visual impairment: Exploring the effect of AMD on the utility of multimodal feedback. Conf. on Human Factors in Comp. Systems: CHI '04. 2004. NY, NY: ACM Press
10. Jameson, A., Adaptive interfaces and agents, The human-computer interaction handbook, A. Sears, ed. 2003, Lawrence Erlbaum Associates: Mahwah NJ. 305-330.
11. Mayer, R.E. & R. Moreno, A split-attention effect in multimedia learning: evidence for dual processing systems in working memory. Journal of Educational Psychology, 1998. 90(2): 312-320.
12. Mousavi, S.Y., R. Low, & J. Sweller, Reducing cognitive load by mixing auditory and visual presentation modes. Journal of Educational Psychology, 1995. 87(2): 319-334.
13. Müller, C., B. Großmann-Hutter, A. Jameson, R. Rummer, & F. Wittig. Recognizing time pressure and cognitive load on the basis of speech: an experimental study. User Modeling. 2001: Springer
14. Oviatt, S.L., Predicting spoken disfluencies during human-computer interaction. Computer Speech and Language, 1995. 9: 19-35.
15. Oviatt, S.L., Multimodal interactive maps: Designing for human performance. Human Computer Interaction, 1997, 12(1-2): 93-129.
16. Oviatt, S.L., Ten myths of multimodal interaction. Communications of the ACM, 1999. 42(11): 74-81.
17. Oviatt, S.L. Mutual disambiguation of recognition errors in a multimodal architecture. ACMSIGCHI Conf. on Human Factors in Comp. Sys. (CHI'99). 1999. Pittsburgh, PA: ACM Press: 576-583.
18. Oviatt, S.L. Multimodal system processing in mobile environments. 13th ACM Symp. on User Interface Software Tech. (UIST'2000). 2000. New York: ACM Press: 21-30.
19. Oviatt, S.L., P.R. Cohen, L. Wu, J. Vergo, L. Duncan, B. Suhm, J. Bers, T.G. Holzman, T. Winograd, J. Landay, J. Larson, & D. Ferro, Designing the user interface for multimodal speech and gesture applications: State-of-the-art systems and research directions. Human Computer Interaction, 2000. 15(4): 263-322.
20. Oviatt, S.L., R. Coulston, S. Tomko, B. Xiao, R. Lunsford, M. Wesson, & L. Carmichael. Toward a theory of organized multimodal integration patterns during human-computer interaction. Internat. Conf. on Multimodal Interfaces. 2003. Vancouver, B.C.: ACM Press: 44-51.
21. Oviatt, S.L., T. Darrell, & M. Flickner, Multimodal Interfaces that flex, adapt, and persist, Comm. of the ACM. 2004. 30-33
22. Penney, C.G., Modality effects and the structure of shortterm verbal memory. Memory and Cognition, 1989. 17: 398-422.
23. Prince, E., Toward a taxonomy of given-new information, Radical Pragmatics, P. Cole, ed. 1986, Academic: NY. 223-255.
24. Sweller, J., Cognitive load during problem solving: Effects on learning. Cognitive Science, 1988. 12: 257-285.
25. Technology for adaptive aging: Reports and papers. 2003, Nat. Acad. of Sci. Workshop: Nat. Acad. Press. http://www.nap.edu/books/0309091160/html/
26. Teder-Sälejärvi, W.A., J.J. McDonald, F. Di Russo, & S.A. Hillyard, An analysis of audio-visual crossmodal integration by means of event-related potential (ERP) recordings. Cognitive Brain Research, 2002. 14: 106-114.
27. Tindall-Ford, S., P. Chandler, & J. Sweller, When two sensory modes are better than one. Journal of Experimental Psychology: Applied, 1997. 3(3): 257-287.
28. Wickens, C., Sandry, D., and Vidulich, M., Compatibility and resource competition between modalities of input, central processing, and output. Human Factors, 1983. 25(2): 227-248.
29. Xiao, B., R. Lunsford, R. Coulston, M. Wesson, & S.L. Oviatt. Modeling multimodal integration patterns and performance in seniors: Toward adaptive processing of individual differences. Internat. Conf. on Multimodal Interfaces. 2003. Vancouver, BC: ACM Press: 265-272.