Symmetric multimodality revisited: Unveiling users' physiological activity

IEEE Transactions on Industrial Electronics

Volumn 54, Issue 2, 2007, Pages 692-698

  Prendinger, Helmut ^a   Ishizuka, Mitsuru ^b  

^a NATIONAL INSTITUTE OF INFORMATICS   (Japan)

^b UNIVERSITY OF TOKYO   (Japan)

Author keywords

User interface human factors

Indexed keywords

BEHAVIORAL RESEARCH; HUMAN COMPUTER INTERACTION; PHYSIOLOGICAL MODELS; USER INTERFACES;

COGNITIVE-MOTIVATIONAL STATE; SYMMETRIC MULTIMODALITY;

MAN MACHINE SYSTEMS;

EID: 34047114777     PISSN: 02780046     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIE.2007.891646     Document Type: Article

References (40)

1. W. Wahlster, "Towards symmetric multimodality: Fusion and fission of speech, gesture and facial expression," in Proc. 26th German Conf. Artif. Intell., 2003, pp. 1-18.
2. J. Kim, E. André, M. Rehm, T. Vogt, and J. Wagner, "Integrating information from speech and physiological signals to achieve emotional sensitivity," in Proc. 9th Eur. Conf. Speech Commun. and Technol., 2005, pp. 809-812.
3. H. Prendinger and M. Ishizuka, Eds., Life-Like Characters. Tools, Affective Functions, and Applications, ser. Cognitive Technologies, Berlin, Germany: Springer-Verlag, 2004.
4. C. Pelachaud and M. Bilvi, "Computational model of believable conversational agents," in Communication in Multiagent Systems: Background, Current Trends, and Future, vol. LNCS 2650. New York: Springer-Verlag, 2003, pp. 300-317.
5. H. Prendinger, S. Descamps, and M. Ishizuka, "MPML: A markup language for controlling the behavior of life-like characters," J. Vis. Lang. Comput., vol. 15, no. 2, pp. 183-203, 2004.
6. M. Ishizuka and H. Prendinger, "Describing and generating multimodal contents featuring affective lifelike agents with MPML," New Gener. Comput., vol. 24, no. 2, pp. 97-128, Jan. 2006.
7. R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias, W. Fellenz, and J. Taylor, "Emotion recognition in human-computer interaction," IEEE Signal Process. Mag., vol. 18, no. 1, pp. 32-80, Jan. 2001.
8. J. Allanson, "Electrophysiologically interactive computer systems," Computer, vol. 35, no. 3, pp. 60-65, Mar. 2002.
9. D. Bersak, G. McDarby, N. Augenblick, P. McDarby, D. McDonnell, B. McDonald, and R. Karkun, "Intelligent biofeedback using an immersive competitive environment," in Proc. Online, Ubicomp Workshop Designing Ubiquitous Comput. Games, 2001.
10. A. T. Duchowski, Eye Tracking Methodology: Theory and Practice. London, U.K.: Springer-Verlag, 2003.
11. S. Shimojo, C. Simion, E. Shimojo, and C. Scheier, "Gaze bias both reflects and influences preference," Nat. Neurosci., vol. 6, no. 12, pp. 1317-1322, Dec. 2003.
12. H. Kautz, "A formal theory of plan recognition," Ph.D. dissertation, Univ. Rochester, Rochester, NY, 1987.
13. R. W. Picard, Affective Computing. Cambridge, MA: MIT Press, 1997.
14. R.W. Levenson, "Emotion and the autonomic nervous system: A prospectus for research on autonomic specificity," in Social Psychophysiology and Emotion: Theory and Clinical Applications, H. L. Wagner, Ed. Hoboken, NJ: Wiley, 1988, pp. 17-42.
15. R.W. Levenson, "Autonomic specificity and emotion," in Handbook of Affective Sciences, R. J. Davidson, K. R. Scherer, and H. H. Goldsmith, Eds. Oxford, U.K.: Oxford Univ. Press, 2003, pp. 212-224.
16. P. Ekman, R. W. Levenson, and W. V. Friesen, "Autonomic nervous system activity distinguishes among emotions," Science, vol. 221, no. 4616, pp. 1208-1210, Sep. 1983.
17. P. J. Lang, "The emotion probe: Studies of motivation and attention," Amer. Psychol., vol. 50, no. 5, pp. 372-385, May 1995.
18. L. Feldman-Barrett and J. A. Russell, "The structure of current affect: Controversies and emerging consensus," Curr. Dir. Psychol. Sci., vol. 8, no. 1, pp. 10-14, 1999.
19. J. L. Andreassi, Psychophysiology. Human Behavior & Physiological Response, 4th ed. Mahwah, NJ: Lawrence Erlbaum Associates, 2000.
20. A. Bechara, H. Damasio, D. Tranel, and A. R. Damasio, "Deciding advantageously before knowing the advantageous strategy," Science, vol. 275, no. 5304, pp. 1293-1295, Feb. 1997.
21. E. H. Hess, "Pupillometrics: A method of studying mental, emotional and sensory processes," in Handbook of Psychophysiology, N. Greenfield and R. Sternbach, Eds. New York: Holt, Rinehart and Winston, 1972, pp. 491-531.
22. T. Partala and V. Surakka, "Pupil size variation as an indication of affective processing," Int. J. Human-Comput. Stud., vol. 59, no. 1/2, pp. 185-198, Jul. 2003.
23. H. Umemuro and J. Yamashita, "Detection of user's confusion and surprise based on pupil dilation," Jpn. J. Ergonomics, vol. 39, no. 4, pp. 153-161, 2003.
24. M. Satyanarayanan, "Pervasive computing: Vision and challenges," IEEE Pers. Commun., vol. 8, no. 4, pp. 10-17, Aug. 2001.
25. J. A. Healey, "Wearable and automotive systems for affect recognition from physiology," Ph.D. dissertation, MIT, Cambridge, MA, 2000.
26. J. H. Goldberg and X. P. Kotval, "Computer interface evaluation using eye movements: Methods and constructs," Int. J. Ind. Ergon., vol. 24, no. 6, pp. 631-645, Oct. 1999.
27. R. J. K. Jacob, "The use of eye movements in human-computer interaction techniques: What you look at is what you get," ACM Trans. Inf. Syst., vol. 9, no. 3, pp. 152-169, 1991.
28. D. D. Salvucci and J. H. Goldberg, "Identifying fixations and saccades in eye-tracking protocols," in Proc. Eye Tracking Res. and Appl. Symp., 2000, pp. 71-78.
29. G. Loftus and N.Mackworth, "Cognitive determinants of fixation location during picture viewing," J. Exp. Psychol., Hum. Percept. Perform., vol. 4, no. 4, pp. 565-572, Nov. 1978.
30. T. S. Huang, L. S. Chen, H. Tao, T. Miyasato, and R. Nakatsu, "Bimodal emotion recognition by man and machine," in Proc. ATR Workshop Virtual Commun. Environments, 1998.
31. H. Prendinger and M. Ishizuka, "The empathic companion: A character-based interface that addresses users' affective states," Int. J. Appl. Artif. Intell., vol. 19, no. 3, pp. 267-285, Mar./Apr. 2005.
32. A. Kapoor, R. W. Picard, and Y. Ivanov, "Probabilistic combination of multiple modalities to detect interest," in Proc. Int. Conf. Pattern Recog., 2005, pp. 969-972.
33. Z. Zeng, J. Tu, M. Liu, T. Zhang, N. Rizzolo, Z. Zhang, T. S. Huang, D. Roth, and S. Levinson, "Bimodal HCI-related affect recognition," in Proc. 6th ICMI, 2004, pp. 137-143.
34. T. Koshizen, Y. Hasegawa, H. Tusjino, M. Kon, K. Aihara, and H. Prendinger, "A learning system for user modeling by combined cognitive and affective modeling for user interest estimation," in Proc. 7th ICCM, 2006, pp. 184-189.
35. S. Kopp, B. Jung, N. Lessmann, and I.Wachsmuth, "Max - A multimodal assistant in virtual reality construction," KI Zeitschift (German Magazine of Artificial Intelligence)-Special Issue on Embodied Conversational Agents, vol. 4, no. 3, pp. 11-17, 2003.
36. C. Becker, H. Prendinger, M. Ishizuka, and I. Wachsmuth, "Evaluating affective feedback of the 3D agent Max in a competitive cards game," in Proc. 1st Int. Conf. ACII. Berlin, Germany: Springer-Verlag, 2005, vol. LNCS 3784, pp. 466-473.
37. H. Prendinger, C. Becker, and M. Ishizuka, "A study in users' physiological response to an empathic interface agent," Int. J. Humanoid Robotics, vol. 3, no. 3, pp. 371-391, Sep. 2006.
38. H. Prendinger, C. Ma, J. Yingzi, A. Nakasone, and M. Ishizuka, "Understanding the effect of life-like interface agents through eye users' eye movements," in Proc. 7th ICMI, 2005, pp. 108-115.
39. R. M. Cooper, "The control of eye fixation by the meaning of spoken language: A new methodology for the real-time investigation of speech perception, memory, and language processing," Cogn. Psychol., vol. 6, no. 1, pp. 84-107, Jan. 1974.
40. C. Peters, C. Pelachaud, E. Bevacqua, M. Mancini, and I. Poggi, "A model of attention and interest using gaze behavior," in Proc. 5th Int. Working Conf. IVA. New York: Springer-Verlag, 2005, vol. LNAI 3661, pp. 229-240.