Visual display, pointing, and natural language: The power of multimodal interaction

Proceedings of the Workshop on Advanced Visual Interfaces AVI

Volumn , Issue , 1998, Pages 164-173

  De Angeli, Antonella ^a   Gerbino, Walter ^a   Cassano, Giulia ^a   Petrelli, Daniela ^b  

^a UNIVERSITY OF TRIESTE   (Italy)

^b ITC IRST   (Italy)

Author keywords

Cross modal integration; Referent identification strategies

Indexed keywords

CROSS-MODAL; MULTI-MODAL INTERACTIONS; MULTI-MODAL INTERFACES; MULTIMODAL COMMUNICATIONS; MULTIMODAL HUMAN COMPUTER INTERACTION; MULTIMODAL SYSTEM; NATURAL LANGUAGES; REFERENT IDENTIFICATIONS;

AUTOMATIC VEHICLE IDENTIFICATION; BEHAVIORAL RESEARCH; COMPUTER SIMULATION; INTERACTIVE COMPUTER SYSTEMS;

HUMAN COMPUTER INTERACTION;

EID: 56049101459     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/948496.948519     Document Type: Conference Paper

References (40)

1. Brennan, S.E. Conversation With and Through Computers. User Modeling and User-Adapted Interaction, 1, 1991, 67-86.
2. Buxton, B. The Natural Language of Interaction: A Perspective on Nonverbal Dialogues. In B. Laurel (Ed.) The Art of Human-Computer Interface Design, Addison-Wesley, 1991, 405-416.
3. Caramazza, A. and Hillis, A.E. Lexical Organization of Nouns and Verbs in Brain. Nature, 349, 1991, 788-790.
4. Cassell, J. and Prévost, S. Distribution of Semantic Features Across Speech and Gesture by Humans and Machine. In L. Messing (Ed.) Proc. of the Workshop of the Integration of Gesture in Language and Speech, University of Delaware, 1996.
5. Cohen, P.R., Dalrymple M., Moran, D.B. Perdra, F.C.N., Sullivan, J.W, Gargan, R.A., Schlossberg, J.L., and Tyler., S.W. Synergistic Use of Direct Manipulation and Natural Language. In CHI'89 Conf, Proc, ACM, Addison-Wesley, New York, 227-234.
6. Clark, H. H., Schreuder, R., and Buttrick, S. Common Ground and the Understanding of Demostrative Reference. Journal of Verbal Learning and Verbal Behavior, 22, 1983, 245-258.
7. th European Conf. of ACL, 1989, 291-298.
8. De Angeli, A., Petrelli, D. and Gerbino, W. Interface Features Affecting Deixis Production: A Simulation Study. In L. Messing (Ed.) Proc. of the Workshop of the Integration of Gesture in Language and Speech, University of Delaware, 1996, 195-204.
9. De Angeli, A., Petrelli, D., Gerbino, W. & Cassano, G. How Computer Literacy Affects Natural Language Interaction. Technical Report of the Cognitive Technology Laboratory, University of Trieste, 1997.
10. Fraser, N. M. and Gilbert, G. N. Simulating Speech System. Computer, Speech and Languages, 5, 1991, 81-99.
11. Gentner, D. and Nielsen, J. The Anti-Mac Interface. Communication of the ACM, 39(8), 1996, 70-82.
12. Gerbino W. Solving by Redundancy and Misunderstanding by Simplification. In V. Cantoni, V. Di Gesu, A. Setti and D. Tegolo (Eds.) Human and Machine Perception: Information Fusion. New York, Plenum, 1997, 147-154.
13. Glenberg, A., and McDaniel, M. Mental Models, Pictures, and Text: Integration of Spatial and Verbal Information. Memory and Cognition, 20 (5), 1992, 458-460.
14. Gullberg, M. Deictic Gesture and Strategy in Second Language Narrative. In L. Messing (Ed.) Proc. of the Workshop of the Integration of Gesture in Language and Speech, University of Delaware, 1996, 155-174.
15. Johnsson, A. and Dahlback, N. Talking To a Computer is Not Like Talking To Your Best Friend. Proc. of Scandinavian Conf. of Artificial Intelligence, 1988, 53-68.
16. Kendon, A. Gesticulation and Speech: Two Aspects of the Process of Utterances. In M. R. Key (Ed.) Nonverbal Communication and Language, The Hague: Mounton, 1980.
17. Levelt, W. J., Richardson, G., and La Heij, W. Pointing and Voicing in Deictic Expressions. Journal of Memory and Language, 24, (1985), 133-164.
18. Levialdi, S., Mussio, P., and Mastronardi, G. Characters, Pixels, and Phonemes. In V. Cantoni, V. Di Gesù, A. Setti and D. Tegolo (Eds.) Human and Machine Perception: Information Fusion. New York, Plenum, 1997.
19. Levinson, S. C. Pragmatics. Cambridge: Cambridge University Press, 1983.
20. Lewkowicz, D. J. and Lickliter, R. The Development of Intersensory Perception: Comparative Perspectives. Hillsdale, NJ: LEA, 1994.
21. Mac Aogain, E. and Reilly, R. Discourse Theory and Interface Design: The Case of Pointing With the Mouse. Intl. Journal of Man-Machine Studies, 32, 1990, 591-602.
22. Maybury, M.T. (Ed.), Intelligent Multimedia Interfaces. Cambridge, Mass: MIT Press, 1993.
23. McNeill, D. Hand and Mind: What Gestures Reveal about Thought. Chicago: University of Chicago Press, 1992.
24. McNeill, D. Language as Gesture (Gesture as Language). In L. Messing (Ed.) Proc. of the Workshop of the Integration of Gesture in Language and Speech, University Of Delaware, 1996, 1-20.
25. Naughton, K. Spontaneous Gesture and Sign: A Study of ASL Signs Co-occurring with Speech. In L. Messing (Ed.) Proc. of the Workshop of the Integration of Gesture in Language and Speech, University Of Delaware, 1996, 125-34.
26. Norman, D. A. and Draper, S. W. User Centered System Design: New Perspectives on Human-Computer Interaction. Hillsdale, New Jersey: Lawrence Erlbaum Associates, 1986.
27. Oviatt, S. Multimodal Interfaces for Dynamic Interactive Maps. CHI'96 Conf. Proc, New York: ACM Press, 1996, 95-102.
28. Oviatt, S. and Olsen, E. Integration Themes in Multimodal Human-Computer Interaction. In Proc. of the Intl. Conf. on Spoken Language Processing, Yokohama, Japan, 1994, 551-554.
29. Oviatt, S. and vanGent, R. Error Resolution During Multimodal Human-Computer Interaction. In Proc. of the Intl. Conf. on Spoken Language Processing, Philadelfia, USA, 1996.
30. Oviatt, S., Cohen, P. R. and Wang, M. Toward Interface Design for Human Language Technology: Modality and Structure as Determinants of Linguistic Complexity. Speech Communication, 15, 1994, 283-300.
31. Oviatt, S., Cohen, P. R., Fong, M. and Frank, M. A Rapid Semi-automatic Simulation Technique for Investigating Interactive Speech and Handwriting. Proc. of the Intl. Conf. on Spoken Language Processing, vol. 2, Alberta, 1992, 1351-1354.
32. Oviatt, S., De Angeli, A. and Kuhn, K. Integration and Synchronization of Input Modes During Multimodal Human-Computer Interaction. In CHI'96 Conf. Proc, New York: ACM Press, 1997, 415-422.
33. Petrelli, D., De Angeli, A., Gerbino, W. and Cassano, G. Referring in Multimodal Systems: The Importance of User Expertise and System Features. In Proc. of the Workshop on Referring Phenomena in a Multimedia Context and Their Computational Treatment, ACL-EACL, 1997, 14-19.
34. Schmauks, D. Natural and Simulated Pointing. Proc. of the 3rd European ACL Conf, Kopenhagen, Danmark, 1987, 179-185.
35. Schmauks, D. and Reithinger, N. Generating Multimodal Output-Conditions, Advantages and Problems. In Proc. of the Intl. Conf. on Computational Linguistics, Budapest, Hungary, 1988, 584-588.
36. Schmauks, D. and Wille M. Integration of Communicative Hand Movements into Human-Computer-Interaction. Computers and the Humanities, 25, 1991, 129-140.
37. Shneiderman, B. Designing the User Interface: Strategies for Effective Human-Computer Interaction. Reading, MA: Addison-Wesley, second edition, 1992.
38. Siroux, J., Guyomard, M., Multon, F. and Remondeau, C. Oral and Gestural Activities of the Users in the Georal System. In Proc. of the Intl. Conf. on Cooperative Multimodal Communication, vol. 2, 1995, 287-298.
39. Stock, O. A Third Modality? Artificial Intelligence Review, 9, 1995, 129-146.
40. van Vliet, P. J., Kletke, M. G. and Chakraborty, G. The measurement of computer literacy: a comparison of self-appraisal and objective tests. Intl. Journal of Human-Computer Studies, 40, 1994, 835-875.