Affective medicine: A review of affective computing efforts in medical informatics

Methods of Information in Medicine

Volumn 49, Issue 3, 2010, Pages 207-218

  Luneski, A ^a,b   Konstantinidis, E ^a   Bamidis, P D ^a  

^a ARISTOTLE UNIVERSITY OF THESSALONIKI   (Greece)

^b UNIVERSITY OF SHEFFIELD   (Greece)

Author keywords

Affective computing; Autism; Biosensor networks; Emotions; Health; Human computer interaction; Learning; Medicine

Indexed keywords

COMPUTER INTERFACE; EMOTION; HUMAN; MEDICAL INFORMATICS; REVIEW;

EMOTIONS; HUMANS; MEDICAL INFORMATICS; USER-COMPUTER INTERFACE;

EID: 77952409131     PISSN: 00261270     EISSN: None     Source Type: Journal    
DOI: 10.3414/ME0617     Document Type: Review

References (110)

1. Picard RW. Affective Computing. 1st ed. The MIT Press; 1997.
2. Cacioppo J. Introduction: Emotion and Health. In: Handbook of Affective Stress. Oxford University Press; 2003.
3. Picard RW. Affective medicine: Technology with emotional intelligence. Future of Health Technology 2001. pp 69-85.
4. Webster C. Medical affective computing: medical informatics meets affective computing. Stud Health Technol Inform 1998. p 52.
5. Smith RN, Frawley WJ. Affective Computing: Medical Applications. In: Proceedings of HCI International (the 8th International Conference on Human-Computer Interaction): Ergonomics and User Interfaces. L. Erlbaum Associates Inc.; 1999. pp 843-847.
6. Prendinger H, Ishizuka M. What Affective Computing and Life-Like Character Technology Can Do for Tele-Home Health Care. In: Workshop on HCI and Homecare: Connecting Families and Clinicians (Online Proceedings) in conj. with CHI; 2004.
7. Bickmore TW, Picard RW. Establishing and maintaining long-term human-computer relationships. ACM Trans Comput-Hum Interact 2005; 12 (2): 293-327.
8. Glaser R, Pearl DK, Kiecolt-Glaser JK, Malarkey WB. Plasma cortisol levels and reactivation of latent Epstein-Barr virus in response to examination stress. Psychoneuroendocrinology 1994; 19 (8): 765-772.
9. Cohen S, Doyle WJ, Skoner DP, Rabin BS, Gwaltney JM. Social ties and susceptibility to the common cold. JAMA: The Journal of the American Medical Association 1997; 277 (24): 1940-1944.
10. Burg M. Stress, Behavior, and Heart Disease. In: Heart Book. Yale University School of Medicine; 2002.
11. Paulhaus D, Fridhandler B, Hayes S. Psychological Defense: Contemporary Theory and Research. In: Handbook of Personality Psychology. San Diego: Academic Press; 1997.
12. Vaillant GE. Ego mechanisms of defense and personality psychopathology. J Abnorm Psychol 1994; 103 (1): 44-50.
13. O'Leary A. Self-efficacy and health: Behavioural and stress-physiological mediation. Cognitive Therapy and Research 1992; 16 (2): 229-245.
14. Scheier MF, Magovern G, Abbot RA, Matthews KA, Owens JF, Lefembre R, Carver CS. Dispositional optimism and recovery from coronary artery bypass surgery: the beneficial effects on physical and psychological well-being. Journal of personality and social psychology 1989; 57 (6): 1024-1040.
15. Wiebbe DJ, Williams PG. Hardiness and health: a social psychophysiological perspective on stress and adaptation. Journal of social and clinical psychology 1992; 11 (3): 238-262.
16. Stone AA, Cox DS, Valdimarsdottir H, Jandorf L, Neale JM. Evidence that secretory IgA antibody is associated with daily mood. J Pers Soc Psychol 1987; 52 (5): 988-993.
17. Middleton R, Byrd E. Psychosocial Factors and Hospital Readmission Status of Older Persons with Cardiovascular Disease. Journal of Applied Re - habilitation Counseling 1996. pp 273-1210
18. Martin LR, Friedman HS, Tucker JS, Tomlinson-Keasey C, Criqui MH, Schwartz JE. A life course perspective on childhood cheerfulness and its relation to mortality risk. Personality and Social Psychology Bulletin 2002; 28 (9): 1155.
19. Kimonides VG, Spillantini MG, Sofroniew MV, Fawcett JW, Herbert J. Dehydroepiandrosterone antagonizes the neurotoxic effects of corticosterone and translocation of stress-activated protein kinase 3 in hippocampal primary cultures. Neuroscience 1999; 89 (2): 429-436.
20. Eysenck MW. Anxiety and cognition: A unified theory. Hove, UK: Psychology Press; 1997.
21. Fox E, Russo R, Bowles RJ, Dutton K. Do threatening stimuli draw or hold visual attention in subclinical anxiety? Journal of Experimental Psychology: General, 2001; 130 (4): 681-700.
22. Pennebaker JW, Mehl MR, Niederhoffer KG. Psychological aspects of natural language use: Our words, our selves. Annual review of psychology 2003; 54 (1): 547-577.
23. Liu H, Lieberman H, Selker T. A model of textual affect sensing using real-world knowledge. In: Proceedings of the 8th international conference on intelligent user interfaces. Miami, Florida, USA: ACM; 2003. pp 125-132.
24. Fellbaum C. WordNet An electronic lexical database. Cambridge, MA: MIT Press; 1998.
25. Valitutti A, Strapparava C, Stock O. Developing affective lexical resources. Psychology Journal 2004; 2 (1): 61-83.
26. Kim S, Hovy E. Identifying and analyzing judgment opinions. In: Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics; New York, 2006. New York: Association for Computational Linguistics; 2006. pp 200-207.
27. Whissell C. The dictionary of affect in language. The Measurement of Emotions: Theory, Research and Experience. 1989. p 113.
28. Subasic P, Huettner A. Affect analysis of text using fuzzy semantic typing. IEEE Transactions on Fuzzy Systems 2001; 9 (4): 483-496.
29. Nasukawa T, Yi J. Sentiment analysis: capturing favorability using natural language processing. In: Proceedings of the 2nd International Conference on Knowledge Capture; 2003. Sanibel Island, FL, USA; 2003. pp 70-77.
30. Dellaert F. Recognizing emotion in speech. In: Proceedings of the ICSLP '96. 1996.
31. Petrushin VA. Emotion in speech: Recognition and application to call centers. In: Engr 1999. pp 7-10.
32. Yu F, Chang E, Xu Y, Shum H. Emotion Detection from Speech to Enrich Multimedia Content. In: Advances in Multimedia Information Processing - PCM'01; 2001. pp 550-557.
33. Yacoub S, Simske S, Lin X, Burns J. Recognition of emotions in interactive voice response systems. In: Proc Eurospeech. Geneva; 2003. pp 1-4.
34. Neiberg D, Elenius K, Laskowski K. Emotion recognition in spontaneous speech using GMMs. In: Proc of Interspeech 2006. Pittsburgh; 2006.
35. Roy D, Pentland A. Automatic spoken affect classification and analysis. In: Proceedings of the 2nd International Conference on Automatic Face and Gesture Recognition (FG '96). IEEE Computer Society; 1996. p 363.
36. Montero JM, Gutierrez-Arriola JM, Palazuelos S, Enriquez E, Aguilera S, Pardo JM. Emotional speech synthesis: from speech database to TTS. In: Fifth International Conference on Spoken Language Processing. ISCA; 1998.
37. Zovato E, Pacchiotti A, Quazza S, Sandri S. Towards emotional speech synthesis: a rule based approach. In: Fifth ISCA Workshop on Speech Synthesis. ISCA; 2004.
38. Burkhardt F, Sendlmeier WF. Verification of acoustical correlates of emotional speech using formantsynthesis. In: ISCA Tutorial and Research Workshop (ITRW) on Speech and Emotion. ISCA; 2000.
39. Edgington M. Investigating the limitations of concatenative synthesis. In: Fifth European Conference on Speech Communication and Technology; 1997.
40. Heuft B, Portele T, Rauth M. Emotions in time domain synthesis. In: Proceedings of the Fourth International Conference on Spoken Language - ICSLP'96; 1996; 3: 1974-1977.
41. Reilly J, Klima ES, Bellugi U. Once more with feeling: affect and language in atypical populations. Development and Psychopathology 2008; 2 (04): 367-391.
42. Gillberg C. Asperger syndrome and high-functioning autism (review). British Journal of Psychiatry 1998; 172200-172209.
43. Dara C, Monetta L, Pell MD. Vocal emotion processing in Parkinson's disease: reduced sensitivity to negative emotions. Brain Research 2008; 1188: 100-111.
44. Ekman P. Emotion in the human face. Cambridge University Press; 1982.
45. Ekman P, Friesen WV. Facial Action Coding System. Consulting Psychologist Press; 1977.
46. Lisetti CL, Schiano DJ. Automatic facial expression interpretation: where human-computer interaction, artificial intelligence and cognitive science intersect. Perspective 2000; 8 (1): 185-235.
47. Cohn JF, Kanade T. Use of automated facial image analysis for measurement of emotion expression. Handbook of Emotion Elicitation and Assessment; 2007.
48. Kanade T, Cohn JF, Tian YL. Comprehensive database for facial expression analysis. In: Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition (FG'00), Grenoble, France; 2000. pp 46-53.
49. Schmidt K, Cohn JF, Tian YL. Signal characteristics of spontaneous facial expressions: Automatic movement in solitary and social smiles. Biological Psychology 2003; 65: 49-66.
50. Cohn JF, Schmidt KL. The timing of facial motion in posed and spontaneous smiles. International Journal of Wavelets, Multiresolution and Infor - mation Processing 2004; 2: 1-12.
51. Cootes TF, Edwards GJ, Taylor CJ. Active appearance models. IEEE Transactions on Pattern Analysis and Machine Intelligence; 2001. pp 681-685.
52. Wen Z, Huang TS. Capturing subtle facial motions in 3D face tracking. International Conference on Computer Vision, Nice, France; 2003. pp 1343-1350.
53. Valstar MF, Pantic M, Patras I. Motion history for facial action detection in video. In: Proceedings of the IEEE Conference on Systems, Man, and Cybernetics, the Hague, the Netherlands; 2004. pp 635-640. (Pubitemid 40415930)
54. Cassell J, Sullivan J, Prevost S, Churchill EF. Embodied conversational agents. Cambridge, MA: MIT Press; 2000.
55. Bailenson J, Beall A, Blascovich J, Raimundo M, Weishbush M. Intelligent agents who wear your face: user's reactions to the virtual self. Transfer Report. Center for the Virtual Environment and Behaviors Department of Psychology, University of California, Santa Barbara; 2000.
56. George P, McIllhagga M. The communication of meaningful emotional information for children interacting with virtual actors. In: Affective interactions: towards a new generation of computer interfaces. Springer-Verlag; 2000. pp 35-48.
57. Hone K. Empathic agents to reduce user frustration: The effects of varying agent characteristics. Interact Comput 2006; 18 (2): 227-245.
58. Moore D, Cheng Y, McGrath P, Powell NJ. Collaborative virtual environment technology for people with autism. Focus on Autism and Other Developmental Disabilities 2005; 20 (4): 231.
59. Luneski A, Konstantinidis E, Hitoglou-Antoniadou M, Bamidis PD. Affective computer-aided learning platform for autistic children. In: Proceedings of the 1st Workshop on Child, Computer and Interaction, Crete, Greece: 2008.
60. Woodworth M, Waschbusch D. Emotional processing in children with conduct problems and callous/ unemotional traits. Child Care Health Dev 2008; 34 (2): 234-244.
61. Shiffrar M, Pinto J. Visual analysis of bodily motion. In: Common mechanisms in perception and action: Attention and Performance. Oxford: Oxford University Press; 2002.
62. Burgoon JK, Jensen ML, Meservy TO, Kruse J, Nunamaker JF. Augmenting human identification of emotional states in video. In: Proceedings of the International Conference on Intelligent Data Analysis. 2005.
63. Bianchi-Berthouze N, Kleinsmith A. A categorical approach to affective gesture recognition. Connection Science 2003; 15 (4): 259-269.
64. Kapur A, Kapur A, Virji-Babul N, Tzanekatis G, Driessen PF. Gesture-based affective computing on motion capture data. In: Affective computing and intelligent interaction - ACII 2005. Springer Verlag; 2005.
65. Bernhardt D, Robinson P. Detecting Affect from Non-stylised Body Motions. In: Proceedings of the 2nd international conference on Affective Computing and Intelligent Interaction. Lisbon, Portugal: Springer-Verlag; 2007. pp 59-70.
66. Castellano G, Kessous L, Caridakis G. Emotion Recognition through Multiple Modalities: Face, Body Gesture, Speech. In: Affect and Emotion in Human-Computer Interaction. 2008. pp 92-103.
67. Balomenos T, Raouzaiou A, Ioannou S, Drosopoulos A, Karpouzis K, Kollias S. Emotion analysis in man-machine interaction systems. Machine Learning for Multimodal Interaction 2004. pp 318-328.
68. Norman DA. The Invisible Computer. Cambridge, MA: MIT Press; 1999.
69. Healey JA, Picard RW. Detecting stress during realworld driving tasks using physiological sensors. IEEE Transactions on Intelligent Transportation Systems 2005; 6 (2):156-166.
70. Papadelis C, Chen Z, Kourtidou-Papadeli C, Bamidis PD, Chouvarda I, Bekiaris E, Maglaveras N. Monitoring sleepiness with on-board electro-physiological recordings for preventing sleep-deprived traffic accidents. Clin Neurophysiol 2007; 118 (9): 1906-1922.
71. Skinner MJ, Simpson PA. Workload issues in military tactical airlift. The International Journal of Aviation Psychology 2002 ; 12 (1): 79-93.
72. Andreassi JL. Psychophysiology: Human Behavior and Physiological Response. 5th ed. Lawrence Erlbaum Associates; 2000.
73. Haag A, Goronzy S, Schaich P, Williams J. Emotion recognition using bio-sensors: First steps towards an automatic system. Lecture Notes in Computer Science 2004; 306: 836-848.
74. Nasoz F, Alvarez K, Lisetti L, Finkelstein N. Emotion recognition from physiological signals using wireless sensors for presence technologies. Cogn Technol Work 2004; 6 (1): 4-14.
75. Picard RW, Vyzas E, Healey J. Toward machine emotional intelligence: analysis of affective physiological state. IEEE transactions on pattern analysis and machine intelligence 2001; 23 (10): 1175-1191.
76. Bamidis PD, Frantzidis C, Konstantinidis E, Luneski A. An Integrated Approach to Emotion Recognition for Advanced Emotional Intelligence. San Diego: 2009.
77. Wagner J, Kim J, Andre E. From physiological signals to emotions: Implementing and comparing selected methods for feature extraction and classification. In: IEEE International Conference on Multimedia and Expo, 2005. ICME 2005. 2005. pp 940-943.
78. Lisetti C, Nasoz F, LeRouge C, Ozyer O, Alvarez K. Developing multimodal intelligent affective interfaces for tele-home health care. Int J Hum-Comput Stud 2003; 59 (1-2): 245-255.
79. Barreto A, Zhai J, Adjouadi M. Non-intrusive physiological monitoring for automated stress detection in human-computer interaction. In: Human-Computer Interaction. 2007. pp 29-38.
80. Paiva A, Picard R, Aylett R, Kolias S, Ortony A, Tao J, Prada R. Preface, In: Paiva A, Prada R, Picard R, editors. Proceedings of the 2nd International Conference on Affective Computing and Intelligent Interaction (ACII 2007); 2007. Berlin: Springer-Verlag; 2007.
81. Van Bemmel J, Musen MA, editors. Handbook of medical informatics. Houten: Springer-Verlag; 1997.
82. e-Health trends. http://ec.europa.eu/information-society/ehealth. Last access May 10, 2009.
83. Maojo V, Kulikowski CA. Bioinformatics and medical informatics: collaborations on the road to genomic medicine? JAMIA 2003; 10 (6): 515-522.
84. Riva G. AmI in health care. CyberPsychology & Behavior 2003; 6 (3): 295-300.
85. Alcaniz, M, Rey B. New technologies for AmI. In: AmI: The Evolution of Technology, Communication and Cognition towards the Future of HCI. 2005. pp 3-16.
86. Cearreta I, Lopez J, Hernandez C, Garay N, Grana M, Alvarez A. Affective computing as a component of AmI. In: Proceedings of the 10th Joint Conference on Information Sciences 2007. Salt Lake City, Utah, USA; 2007. pp 1580-1586.
87. Zhou J, Yu C, Riekki J, Karkkainen E. AmE framework: a model for emotion-aware AmI. In: Proceedings of the 2nd International Conference on Affective Computing and Intelligent Interaction (ACII 2007); 2007. Berlin: Springer-Verlag; 2007.
88. Ducatel K, Bogdanowicz M, Scapolo F, Leijten J, Burgelman J. ISTAG Scenarios for AmI in 2010; 2001. http://cordis.europa.eu/ist/istag-reports.htm
89. Riva G, Vatalaro F, Davide F, Alaniz M. AmI: The Evolution of Technology, Communication and Cognition towards the Future of HCI. IOS Press, Fairfax; 2005.
90. Crist TM, Kaufman SB, Crampton KR. Home telemedicine: a home health care agency strategy for maximizing resources. Home Health Care Management Practice 1996; 8 (4): 1-9.
91. Kenner AN. The effect of task differences, attention and personality on the frequency of body-focused hand movements. Journal of Nonverbal Behavior 1984; 8 (3): 159-171.
92. Alonso MB, Varkevisser M, Hekkert P, Keyson D. Exploring manipulative hand movements during a stressful condition. In: Proceedings of the 2nd Affective Computing and Intelligent Interaction conference (ACII 2007); 2007. pp 757-758.
93. Liu KKL. A personal, mobile system for understanding stress and interruptions. Cambridge: MIT; 2004.
94. Healey J, Picard R. Smartcar: detecting driver stress. In: Proceedings of the 15th International Conference on Pattern Recognition; 2000.
95. Metaxas D, Venkataraman S, Vogler C. Image-based stress recognition using a model-based dynamic face tracking system. In: Computational Science - ICCS 2004; 2004. pp 813-821.
96. Worgan S, Moore RK. Enabling reinforcement learning for open dialogue systems through speech stress detection. In: The Fourth International Workshop on Human-Computer Conversation. Bellagio, Italy; 2008.
97. Hook C. A holistic approach to ubiquitous stress management. In: Proceedings of UbiComp 2009 (in press).
98. Fabri M, Moore D. The use of emotionally expressive avatars in collaborative virtual environments. In: Proceeding of Symposium on Empathic Interaction with Synthetic Characters, at Artificial Intelligence and Social Behaviour Convention 2005. 2005.
99. Murray D. Autism and information technology: therapy with computers. Autism and learning: a guide to good practice. 1997. pp 100-117.
100. Enyon A. Computer interaction: an update on the avatar program. Communication 1997; 18.
101. Eddon G. Danny's rooms. In: Proceedings of the Johns Hopkins National Search for Computing Applications to Assist Persons with Disabilities. IEEE Computing Society Press; 1992.
102. Green SJ. Computer-Based Simulations in the Education and Assessment of Autistic Children. In: Rethinking the Roles of Technology in Education. Cambridge, MA: Massachusetts Institute of Technology; 1993. pp 334-336.
103. Luneski A, Konstantinidis E, Hitoglou-Antoniadou M, Bamidis PD. Affective Computer-Aided Learning Platform for Autistic Children. Chania, Crete, Greece: 2008.
104. Kaliouby R, Picard R, Barron-Cohen S. Affective computing and autism. Annals of the New York Academy of Sciences. 2006; 1093 (1 Progress in Convergence: Technologies for Human Wellbeing): 228-248.
105. Blocher K, Picard RW. Affective social quest: emotion recognition therapy for autistic children. Socially Intelligent Agents: Creating Relationships with Computers and Robots. 133-140.
106. Frantzidis C, Bamidis PD. Description and future trends of ICT solutions offered towards independent living: the case of the LLM project. In: Proceedings of the 2nd ACM International Conference on Pervasive Technologies Related to Assistive Environments (PETRA 2009); Greece 2009. In press.
107. Brain Fitness. http://www.brainfitness.com. Last access May 12, 2009.
108. Knaup P, Bott O, Kohl C, Lovis C, Garde S. Electronic patient records: moving from islands and bridges towards electronic health records for continuity of care. Meth Inf Med 2007; 46: 31-46.
109. Meystre S. Electronic patient records: some answers to the data representation and reuse challenges. Findings from the section on patient records. In: Geissbuhler A, Haux R, Kulikowski C, editors. IMIA Yearbook of Medical Informatics 2007. Methods Inf Med 2007; 46 (1): 47-49.
110. Bamidis PD, Papadelis C, Kourtidou-Papadeli C, Pappas C, Vivas A. Affective computing in the era of contemporary neurophysiology and health informatics. Interacting with Computers 2004; 16 (4): 715-721.