Multimodal feedback: An assessment of performance and mental workload

Ergonomics

Volumn 46, Issue 1-3, 2003, Pages 68-87

  Vitense, H S ^a   Jacko, J A ^b   Emery, V K ^b  

^a MEDTRONIC   (United States)

^b Georgia Institute of Technology   (United States)

Author keywords

Auditory; Feedback; Haptic; Human computer interaction; Multimodald; Visual

Indexed keywords

FEEDBACK; HAPTIC INTERFACES; HUMAN COMPUTER INTERACTION;

MULTIMODAL INTERFACES;

ERGONOMICS;

ARTICLE; AUDITORY RESPONSE; FEEDBACK SYSTEM; HUMAN; HUMAN COMPUTER INTERACTION; HUMAN EXPERIMENT; MENTAL LOAD; MENTAL PERFORMANCE; MENTAL TASK; NORMAL HUMAN; SELF REPORT; TACTILE STIMULATION; VISUAL STIMULATION; WORKLOAD; ADULT; BIOENGINEERING; COMPUTER INTERFACE; COMPUTER PROGRAM; DYSTHYMIA; METHODOLOGY; PATHOPHYSIOLOGY; PHYSIOLOGY; PSYCHOLOGICAL ASPECT; PSYCHOLOGICAL MODEL; QUESTIONNAIRE; TASK PERFORMANCE; TOUCH; VISION;

ADULT; FEEDBACK; HUMAN ENGINEERING; HUMANS; MENTAL FATIGUE; MODELS, PSYCHOLOGICAL; QUESTIONNAIRES; SOFTWARE; TASK PERFORMANCE AND ANALYSIS; TOUCH; USER-COMPUTER INTERFACE; VISION; WORKLOAD;

EID: 0037440234     PISSN: 00140139     EISSN: None     Source Type: Journal    
DOI: 10.1080/00140130303534     Document Type: Article

References (48)

1. Akamatsu, M. 1992, The influence of combined visual and tactile information on figure and eye movements during shape tracing, Ergonomics, 35, 647-660.
2. Akamatsu, M. and Mackenzie, S. I., 1996, Movement characteristics using a mouse with tactile and force feedback, International Journal of Human - Computer Studies, 45, 483-493.
3. Akamatsu, M. and Sato, S. 1994, A multi-modal mouse with tactile and force feedback, International Journal of Human - Computer Studies, 40, 443-453.
4. Aretz, A. J., Johanssen, C. and Ober, K. 1996, An empirical validation of subjective workload ratings, in Proceedings of the Human Factors Society 40th Annual Meeting, Philadelphia, PA, 91-95.
5. Arsenault, R. and Ware, C. 2000, Eye-hand co-ordination with force feedback, in Proceedings of the 2000 ACM Conference Companion on Human Factors in Computing Systems: Common Ground (CHI'00), The Hague, the Netherlands, 408-414.
6. Blattner, M., Sumikawa, D. and Greenberg, R. 1989, Earcons and icons: Their structure and common design principles, Human - Computer Interaction, 4, 11-44.
7. Bolt, R. A. 1980, 'Put-that-there': Voice and gesture at the graphics interface, Computer Graphics, 14, 262-270.
8. Brewster, S. A. 1998a. Sonically-enhanced drag and drop, in Proceedings of International Conference on Auditory Display (ICAD '98), Glasgow, UK, 1-7.
9. Brewster, S. A. 1998b, Using non-speech sounds to provide navigation cues, ACM Transactions on Computer - Human Interaction, 5, 224-259.
10. Brewster, S. A. and Crease, M. G. 1999, Correcting menu usability problems with sound, Behavior and Information Technology, 18, 165-177.
11. Brewster, S. A., Raty, V.-P. and Kortekangas, A. 1996, Enhancing scanning input with non-speech sounds, in Proceedings for the Second Annual International ACM/SIGCAPH Conference on Assistive Technologies (ASSETS '96), Vancouver Canada, 10-14.
12. Brewster, S. A., Wright, P. C. and Edwards, A. D. N. 1994, The design and evaluation of an auditory-enhanced scrollbar, in Proceedings of the 1994 ACM Conference on Human Factors in Computing Systems (CHI'94), Boston, MA, 173-179.
13. Byers, J. C., Bittner, A. C., Hill, S. G., Zaklad, A. L. and Christ, R. E. 1988, Workload assessment of a remotely piloted vehicle (RPV) system, in Proceedings of the Human Factors Society 32nd Annual Meeting, Anaheim, CA, 1145-1149.
14. Campbell, C. S., Zhai, S., May, K. W. and Maglio, P. P. 1999, What you feel must be what you see: Adding tactile feedback to the trackpoint, in M. A. Sasse and C. Johnson (eds), Human - Computer Interaction - Proceedings of INTERACT '99 (Amsterdam: IOS Press), 383-390.
15. Cohen, P. R., Dalrymple, M., Moran, D. B., Pereira, F. C. and Sullivan, J. W. 1989, Synergistic use of direct manipulation and natural language, in Proceedings of the 1989 ACM Conference on Human Factors in Computing Systems (CHI'89), Austin, TX, 227-233.
16. Cohen, P. R., Johnston, M., McGee, D., Oviatt, S., Pittman, J., Smith, I., Chen, L. and Clow, J. 1997, QuickSet: Multimodal interaction for simulation set-up and control, in Proceedings of the 5th Applied Natural Language Conference, San Francisco, CA, 20-24.
17. Engel, F. L., Goossens, P. and Haakma, R. 1994, Improved Efficiency through I- and E-feedback: A trackball with contextual force feedback, International Journal of Human - Computer Studies, 41, 949-974.
18. Gopher, D. and Donchin, E. 1986, Workload - An examination of the concept, in K. R. Boff, L. Kaufman and J. P. Thomas (eds), Handbook of Perception and Human Performance (New York: Wiley), 41-49.
19. Grasso, M. A., Ebert, D. S. and Finin, T. W. 1998, The integrally of speech in multimodal interfaces, ACM Transactions on Computer - Human Interaction, 5, 303-325.
20. Hart, S. G. and Staveland, L. E. 1988, Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research, in P. A. Handcock and N.Meshkati (eds), Human Mental Workload (New York: Elsevier), 5-39.
21. Hart, S. G. and Wickens, C. D. 1990, Workload assessment and prediction, in H. R. Booher (ed.), MANPRINT: An Approach to Systems Integration (New York: Van Norstrand Reinhold), pp. 257-296.
22. Hauptmann, A. G. 1989, Speech and gestures for graphic image manipulation, in Proceedings of the 1989 ACM Conference on Human Factors in Computing Systems (CHI'89), Austin, TX, 241-245.
23. Hill, S. G., Iavecchia, H. P., Byers, J. C., Bittner, A. C., Zaklad, A. L. and Christ, R. E. 1992, Comparison of four subjective workload rating scales, Human Factors, 34, 429-439.
24. Keates, S., Clarkson, P. J. and Robinson, P. 2001, Cognitive considerations in the design of multimodal input systems, in C. Stephanidis (ed.), Universal Access in HCI: Towards and Information Society for All (New York: Lawrence Erlbaum), 353-357.
25. Keppel, G. 1991, Design and Analysis: A Research's Handbook (Upper Saddle River: Prentice-Hall).
26. Liu, J., Pastoor, S., Seifert, K. and Hurtienne, J. 2000, Three-dimensional PC: Toward novel forms of human-computer interaction, in Proceedings of the Three-Dimensional Video and Display: Devices and Systems (SPIE CR76), Boston, MA, 250-281.
27. Ma, J., Gao, W. and Wang, R. 2000, A parallel multistream modal for integration of sign language recognition and lip motion, in T. Tan, Y. Shi and Gao (eds), Advances in Multimodal Interfaces - ICMI 2000 (New York: Springer), 353-357.
28. McGee, M. R. 1999, A haptically enhanced scrollbar: Force-feedback as a means of reducing the problems associated with scrolling, in Proceedings for the First PHANToM Users Research Symposium (PURS'99), Heidelberg, Germany, 17-20.
29. Miller, C. A. and Hannen, M. D. 1999, User acceptance of an intelligent user interface: A Rotorcraft Pilot's Associate example, in Proceedings of the 1999 International Conference on Intelligent User Interfaces, Redondo Beach, CA, 109-116.
30. Nelson, R. J., Mccandlish, C. A. and Douglas, V. D. 1990, Reaction times for hand movements made in response to visual versus vibratory cues, Somatosensory and Motor Research, 7, 337-352.
31. Nickerson, R. S. and Landauer, T. K. 1997, Human-computer interaction: Background and issue, in M. Helander, T. K. Landauer and P. Prabhu (eds), Handbook of Human-Computer Interaction (New York: Elsevier), 3-31.
32. Nigay, L. and Coutaz, J. 1995, A generic platform for addressing the multimodal challenge, in Proceedings of the 1995 ACM Conference on Human Factors in Computing Systems (CHI'95), Denver, CO, 98-105.
33. Oakley, I., Brewster, S. A. and Gray, P. D. 2001, Solving multi-target haptic problems in menu interaction, in Proceedings of the 2001 ACM Conference on Human Factors in Computing Systems (CHI'01), Seattle, WA, 357-358.
34. Oviatt, S. 2000, Multimodal system processing in mobile environments, in Proceedings of the 2000 ACM Symposium on User Interface Software and Technology (UIST'00), San Diego, CA, 21-29.
35. Oviatt, S., Cohen, P., Suhm, B., Bers, J., Wu, L., Holzman, T., Winograd, T., Vergo, J., Duncan, L., Landay, J., Larson, J. and Ferro, D. 2002, Designing the user interface for multimodal speech and gesture applications: State-of-the-art systems and research directions from 2000 and beyond, Human-Computer Interaction in the New Millennium (New York: ACM Press) (in press).
36. Oviatt, S., Deangeli, A. and Kuhn, K. 1997, Integration and synchronization of input modes during multimodal human-computer interaction, in Proceedings of the 1997 ACM Conference on Human Factors in Computing Systems (CHI'97), Atlanta, GA, 415-422.
37. Pitt, I. J. and Edwards, A. D. N. 1997, An improved auditory interface for the exploration of lists, in Proceedings of the Fifth ACM International Conference on Multimedia (ACM Multimedia'97), Seattle, WA, 51-61.
38. Pittman, J. A., Smith, I., Cohen, P., Oviatt, S. and Yang, T.-C. 1996, QuickSet: A multimodal interface for military simulation, in Proceedings of the 6th Conference Computer-Generated Forces and Behavioral Representation, Orlando, FL, 217-224.
39. Ramstein, C., Martial, O., Duffresne, A., Carigna, M., Chassé, P. and Mabilleau, P. 1996, Touching and hearing GUI's: Design issues for the PC-access system, in Proceedings for the Second Annual International ACM/SIGCAPH Conference on Assistive Technologies (ASSETS '96), Vancouver, Canada, 2-9.
40. Ren, X., Zhang, G. and Dai, G. 2000, An experimental study of input modes for multimodal human-computer interaction, in T. Tan, Y. Shi and W. Gao (eds), Advances in Multimodal Interfaces - ICMI 2000 (New York: Springer), 49-56.
41. Rosenberg, L. and Brave, S. 1996, Using force feedback to enhance human performance in graphical user interfaces, in Proceedings of the 1996 ACM Conference on Human Factors in Computing Systems (CHI'96): Short Papers, Vancouver, Canada, 291-292.
42. Salvendy, G. and Carayon, P. 1997, Data collection and evaluation of outcome measures, in G. Salvendy (ed.), Handbook of Human Factors and Ergonomics (New York: Wiley), 1451-1470.
43. Sanders, M. and Mccormick, E. (1993), Human Factors in Engineering and Design, 7th edn (New York: McGraw Hill), 176-178.
44. Smith, A., Dunaway, J., Demasco, P. and Peischl, D. 1996, Multimodal input for computer access and augmentative communication, in Proceedings for the Second Annual International ACM/SIGCAPH Conference on Assistive Technologies (ASSETS '96), Vancouver, Canada, 80-85.
45. Vidulich, M. A. and Tsang, P. S. 1986, Techniques of subjective workload assessment: A comparison of SWAT and the NASA-Bipolar methods. Ergonomics, 29, 1385-1398.
46. Vo, M. T. and Wood, C. 1996, Building and application framework for speech and pen input integration in multimodal learning interfaces, in Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Atlanta, GA, 3545-3548.
47. Wickens, C. D. 1992, Engineering Psychology and Human Performance (New York: HarperCollins).
48. Xie, B. and Salvendy, G. 2000, Prediction of mental workload in single and multiple tasks environments, International Journal of Cognitive Ergonomics, 4, 213-242.