The role of nonlinear dynamics in affective valence and arousal recognition

IEEE Transactions on Affective Computing

Volumn 3, Issue 2, 2012, Pages 237-249

  Valenza, Gaetano ^a   Lanata, Antonio ^a   Scilingo, Enzo Pasquale ^a  

^a UNIVERSITY OF PISA   (Italy)

Author keywords

affective computing; Emotion recognition; feature extraction; nonlinear analysis

Indexed keywords

AFFECTIVE COMPUTING; AFFECTIVE STATE; AUTOMATIC ASSESSMENT; CROSS VALIDATION; DYNAMIC TECHNIQUES; ELECTRODERMAL RESPONSE; EMOTION RECOGNITION; EXPERIMENTAL PROTOCOLS; MULTI-CLASS; NON-LINEAR METHODS; PHYSIOLOGICAL SIGNALS; PICTURE SYSTEM; QUADRATIC DISCRIMINANT CLASSIFIER; RECOGNITION ACCURACY; REFERENCE LEVELS; RESPIRATION ACTIVITY; SPATIAL LOCALIZATION; STANDARD METHOD;

FEATURE EXTRACTION; NONLINEAR ANALYSIS;

ELECTROCARDIOGRAPHY;

EID: 84863908750     PISSN: 19493045     EISSN: None     Source Type: Journal    
DOI: 10.1109/T-AFFC.2011.30     Document Type: Article

References (100)

1. J. Russell and J.M. Carroll, "On the Bipolarity of Positive and Negative Affect," Psychological Bull., vol. 125, no. 1, pp. 3-30, 1999.
2. D. Watson, D. Wiese, J. Vaidya, and A. Tellegen, "The Two General Activation Systems of Affect: Structural Findings, Evolutionary Considerations, and Psychobiological Evidence," J. Personality and Social Psychology, vol. 76, no. 5, pp. 820-838, 1999.
3. D. Watson and L. Clark, "On Traits and Temperament: General and Specific Factors of Emotional Experience and Their Relation to the Five-Factor Model," J. Personality, vol. 60, no. 2, pp. 441-476, 1992.
4. C. Darwin, The Expression of the Emotions in Man and Animals; with an Introduction, Afterword, and Commentaries by Paul Ekman. Oxford Univ. Press, 1872.
5. P. Ekman, "Universal Facial Expressions of Emotion," Culture and Personality: Contemporary Readings, p. 8, Aldine Pub. Co., 1974.
6. P. Ekman, "Basic Emotions," Handbook of Cognition and Emotion, pp. 45-60, John Wiley & Sons, 1999.
7. S. Tompkins, Affect Imagery Consciousness: The Positive Affects, vol. 1. Springer Publishing Company, 1962.
8. C. Izard, The Face of Emotion, vol. 23. Appleton-Century-Crofts, 1971.
9. R. Plutchik, "Emotions: A General Psychoevolutionary Theory," Approaches to Emotion, pp. 197-219, L. Erlbaum Assoc., 1984.
10. P. Ekman, "Cross-Cultural Studies of Facial Expression," Darwin and Facial Expression: A Century of Research in Review, pp. 169-222, Academic Press, 1973.
11. J. Watson, Behaviorism. Transaction Publishers, 1997.
12. A. Ortony and T. Turner, "What Is Basic about Basic Emotions," Psychological Rev., vol. 97, no. 3, pp. 315-331, 1990.
13. W. Wundt, Grundriss der Psychologie [Fundamentals of Psychology], seventh rev. ed. Engelman, 1905.
14. H. Schlosberg, "Three Dimensions of Emotion," Psychological Rev., vol. 61, no. 2, pp. 81-88, 1954.
15. C. Osgood, The Measurement of Meaning. Univ. of Illinois Press, 1975.
16. J. Davitz, The Language of Emotion. Academic Press, 1969.
17. P. Lang, M. Bradley, and B. Cuthbert, "Emotion, Motivation, and Anxiety: Brain Mechanisms and Psychophysiology," Biological Psychiatry, vol. 44, no. 12, pp. 1248-1263, 1998. (Pubitemid 29015863)
18. J. Panskepp, Affective Neuroscience: The Foundations of Human and Animal Emotions. Oxford Univ. Press, 1998.
19. C. Breazeal, "Emotion and Sociable Humanoid Robots," Int'l J. Human-Computer Studies, vol. 59, nos. 1/2, pp. 119-155, 2003.
20. J. Russell, "A Circumplex Model of Affect," J. Personality and Social Psychology, vol. 39, no. 6, pp. 1161-1178, 1980.
21. J. Russell and A. Mehrabian, "Evidence for a Three-Factor Theory of Emotions* 1," J. Research in Personality, vol. 11, no. 3, pp. 273-294, 1977.
22. M. Arnold, An Excitatory Theory of Emotion. McGraw-Hill, 1950.
23. N. Frijda, The Emotions. Cambridge Univ. Press, 1986.
24. A. Ortony, G. Clore, and A. Collins, The Cognitive Structure of Emotions, Cambridge Univ. Press, 1990.
25. K. Scherer and P. Ekman, Approaches to Emotions. Lawrence Erlbaum Assoc., Publisher, 1984.
26. C. Lisetti and P. Gmytrasiewicz, "Can a Rational Agent Afford to Be Affectless? A Formal Approach," Applied Artificial Intelligence, vol. 16, pp. 1-33, 2002.
27. K. Scherer, A. Schorr, and T. Johnstone, Appraisal Processes in Emotion: Theory, Methods, Research, Oxford Univ. Press, 2001.
28. A. Egges, S. Kshirsagar, and N. Magnenat-Thalmann, "A Model for Personality and Emotion Simulation," Proc. Knowledge-Based Intelligent Information and Eng. Systems, pp. 453-461, 2003. (Pubitemid 37230708)
29. J. Posner, J. Russell, and B. Peterson, "The Circumplex Model of Affect: An Integrative Approach to Affective Neuroscience, Cognitive Development, and Psychopathology," Development and Psychopathology, vol. 17, no. 3, pp. 715-734, 2005.
30. L. Bialoskorski, J. Westerink, and E. Broek, "Mood Swings: An Affective Interactive Art System," Proc. Third Int'l Conf. Intelligent Technologies for Interactive Entertainment, pp. 181-186, 2009.
31. C. Lisetti and F. Nasoz, "Using Noninvasive Wearable Computers to Recognize Human Emotions from Physiological Signals," J. Applied Signal Processing, vol. 2004, pp. 1672-1687, 2004.
32. J. Kim and E. André, "Emotion Recognition Based on Physiological Changes in Music Listening," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 30, no. 12, pp. 2067-2083, Dec. 2008.
33. J. Janssen, E. Van den Broek, and J. Westerink, "Personalized Affective Music Player," Proc. Third Int'l Conf. Affective Computing and Intelligent Interaction and Workshops, pp. 1-6, 2009.
34. Y. Lin, C. Wang, T. Jung, T. Wu, S. Jeng, J. Duann, and J. Chen, "EEG-Based Emotion Recognition in Music Listening," IEEE Trans. Biomedical Eng., vol. 57, no. 7, pp. 1798-1806, July 2010.
35. E. Van den Broek, M. Schut, J. Westerink, and K. Tuinenbreijer, "Unobtrusive Sensing of Emotions (USE)," J. Ambient Intelligence and Smart Environments, vol. 1, no. 3, pp. 287-299, 2009.
36. E. Van den Broek and J. Westerink, "Considerations for Emotion-Aware Consumer Products," Applied Ergonomics, vol. 40, no. 6, pp. 1055-1064, 2009.
37. Z. Zeng, M. Pantic, G. Roisman, and T. Huang, "A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 31, no. 1, pp. 39-58, Jan. 2009.
38. K. Poels and S. Dewitte, "How to Capture the Heart? Reviewing 20 Years of Emotion Measurement in Advertising," J. Advertising Research, vol. 46, no. 1, p. 18, 2006.
39. E. Leon, G. Clarke, V. Callaghan, and F. Sepulveda, "A User-Independent Real-Time Emotion Recognition System for Software Agents in Domestic Environments," Eng. Applications of Artificial Intelligence, vol. 20, no. 3, pp. 337-345, 2007. (Pubitemid 46366646)
40. G. Chanel, J. Kierkels, M. Soleymani, and T. Pun, "Short-Term Emotion Assessment in a Recall Paradigm," Int'l J. Human-Computer Studies, vol. 67, no. 8, pp. 607-627, 2009.
41. J. Healey, "Affect Detection in the Real World: Recording and Processing Physiological Signals," Proc. Third Int'l Conf. Affective Computing and Intelligent Interaction and Workshops, pp. 1-6, 2009.
42. J. Healey and R. Picard, "Detecting Stress during Real-World Driving Tasks Using Physiological Sensors," IEEE Trans. Intelligent Transportation Systems, vol. 6, no. 2, pp. 156-166, June 2005. (Pubitemid 40925766)
43. P. Lang, M. Bradley, and B. Cuthbert, "International Affective Picture System IAPS): Digitized Photographs, Instruction Manual and Affective Ratings," Technical Report A-6, Univ.of Florida, 2005.
44. P. Lang, M. Bradley, and B. Cuthbert, "International Affective Picture System (IAPS): Technical Manual and Affective Ratings," NIMH Center for the Study of Emotion and Attention, 1997.
45. P. Lang, M. Greenwald, M. Bradley, and A. Hamm, "Looking at Pictures: Affective, Facial, Visceral, and Behavioral Reactions," Psychophysiology, vol. 30, no. 3, pp. 261-273, 1993. (Pubitemid 23108336)
46. P. Lang et al., "Behavioral Treatment and Bio-Behavioral Assessment: Computer Applications," Technology in Mental Health Care Delivery Systems, pp. 119-137, Ablex Pub. Corp., 1980.
47. S. Grimm, C. Schmidt, F. Bermpohl, A. Heinzel, Y. Dahlem, M. Wyss, D. Hell, P. Boesiger, H. Boeker, and G. Northoff, "Segregated Neural Representation of Distinct Emotion Dimensions in the Prefrontal Cortex-An fMRI Study," Neuroimage, vol. 30, no. 1, pp. 325-340, 2006. (Pubitemid 43243222)
48. A. Hariri, V. Mattay, A. Tessitore, F. Fera, and D. Weinberger, "Neocortical Modulation of the Amygdala Response to Fearful Stimuli," Biological Psychiatry, vol. 53, no. 6, pp. 494-501, 2003. (Pubitemid 36331888)
49. R. Picard, E. Vyzas, and J. Healey, "Toward Machine Emotional Intelligence: Analysis of Affective Physiological State," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 10, pp. 1175-1191, Oct. 2001. (Pubitemid 33047955)
50. C. Lisetti and F. Nasoz, "Using Noninvasive Wearable Computers to Recognize Human Emotions from Physiological Signals," EURASIP J. Applied Signal Processing, vol. 2004, pp. 1672-1687, 2004.
51. A. Haag, S. Goronzy, P. Schaich, and J. Williams, "Emotion Recognition Using Bio-Sensors: First Steps towards an Automatic System," Affective Dialogue Systems, vol. 1, pp. 36-48, 2004. (Pubitemid 38851666)
52. K. Kim, S. Bang, and S. Kim, "Emotion Recognition System Using Short-Term Monitoring of Physiological Signals," Medical and Biological Eng. and Computing, vol. 42, no. 3, pp. 419-427, 2004. (Pubitemid 38786706)
53. S. Yoo, C. Lee, Y. Park, N. Kim, B. Lee, and K. Jeong, "Neural Network Based Emotion Estimation Using Heart Rate Variability and Skin Resistance," Proc. First Int'l Conf. Advances in Natural Computation, pp. 818-824, 2005. (Pubitemid 41449993)
54. A. Choi and W. Woo, "Physiological Sensing and Feature Extraction for Emotion Recognition by Exploiting Acupuncture Spots," Proc. First Int'l Conf. Affective Computing and Intelligent Interaction, pp. 590-597, 2005. (Pubitemid 43763379)
55. L. Li and J. Chen, "Emotion Recognition Using Physiological Signals," Pro. 16th Int'l Conf. Advances in Artificial Reality and Tele-Existence, pp. 437-446, 2006.
56. P. Rani, C. Liu, N. Sarkar, and E. Vanman, "An Empirical Study of Machine Learning Techniques for Affect Recognition in Human-Robot Interaction," Pattern Analysis & Applications, vol. 9, no. 1, pp. 58-69, 2006. (Pubitemid 43823166)
57. P. Rainville, A. Bechara, N. Naqvi, and A. Damasio, "Basic Emotions Are Associated with Distinct Patterns of Cardiorespiratory Activity," Int'l J. Psychophysiology, vol. 61, no. 1, pp. 5-18, 2006. (Pubitemid 44209107)
58. J. Zhai and A. Barreto, "Stress Detection in Computer Users Based on Digital Signal Processing of Noninvasive Physiological Variables," Proc. IEEE 28th Ann. Int'l Conf. Eng. in Medicine and Biology Soc., pp. 1355-1358, 2006. (Pubitemid 46520124)
59. C. Liu, K. Conn, N. Sarkar, and W. Stone, "Physiology-Based Affect Recognition for Computer-Assisted Intervention of Children with Autism Spectrum Disorder," Int'l J. Human-Computer Studies, vol. 66, no. 9, pp. 662-677, 2008.
60. C. Katsis, N. Katertsidis, G. Ganiatsas, and D. Fotiadis, "Toward Emotion Recognition in Car-Racing Drivers: A Biosignal Processing Approach," IEEE Trans. Systems, Man, and Cybernetics, Part A: Systems and Humans, vol. 38, no. 3, pp. 502-512, May 2008. (Pubitemid 351649883)
61. G. Yannakakis and J. Hallam, "Entertainment Modeling through Physiology in Physical Play," Int'l J. Human-Computer Studies, vol. 66, no. 10, pp. 741-755, 2008.
62. C. Katsis, N. Katertsidis, and D. Fotiadis, "An Integrated System Based on Physiological Signals for the Assessment of Affective States in Patients with Anxiety Disorders," Biomedical Signal Processing and Control, vol. 6, pp. 261-268, 2011.
63. R. Kohavi and F. Provost, "Glossary of Terms," Machine Learning, vol. 30, pp. 271-274, 1998.
64. F. van der Heiden, R. Duin, D. de Ridder, and D. Tax, Classification, Parameter Estimation, State Estimation: An Engineering Approach Using MatLab. Wiley, 2004.
65. A. Schlogl, "Time Series Analysis-A Toolbox for the Use with Matlab," 2002.
66. K. Kroenke, R. Spitzer, and J. Williams, "The phq-9," J. General Internal Medicine, vol. 16, no. 9, pp. 606-613, 2001.
67. U. Rajendra Acharya, K. Paul Joseph, N. Kannathal, C. Lim, and J. Suri, "Heart Rate Variability: A Review," Medical and Biological Eng. and Computing, vol. 44, no. 12, pp. 1031-1051, 2006. (Pubitemid 47090818)
68. A. Lanata, E. Scilingo, E. Nardini, G. Loriga, R. Paradiso, and D. De-Rossi, "Comparative Evaluation of Susceptibility to Motion Artifact in Different Wearable Systems for Monitoring Respiratory Rate," IEEE Trans. Information Technology in Biomedicine, vol. 14, no. 2, pp. 378-386, Mar. 2010.
69. W. Winton, L. Putnam, and R. Krauss, "Facial and Autonomic Manifestations of the Dimensional Structure of Emotion* 1," J. Experimental Social Psychology, vol. 20, no. 3, pp. 195-216, 1984.
70. P. Lang, "The Emotion Probe," Am. Psychologist, vol. 50, no. 5, pp. 372-385, 1995.
71. H. McCURDY, "Consciousness and the Galvanometer," Psychological Rev., vol. 57, no. 6, pp. 322-327, 1950.
72. P. Venables and M. Christie, "Electrodermal Activity," Techniques in Psychophysiology, pp. 3-67, Wiley, 1980.
73. A. Jain and R. Mao, "Statistical Pattern Recognition: A Review," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 1, pp. 4-37, Jan. 2000. (Pubitemid 32132953)
74. B. Kohler, C. Hennig, and R. Orglmeister, "The Principal of Software QRS Detection," IEEE Eng. in Medicine and Biology, vol. 6, no. 1, pp. 42-57, Jan./Feb. 2002.
75. J. Pan and W. Tompkins, "A Real-Time QRS Detection Algorithm," IEEE Trans. Biomedical Eng., vol. 32, no. 3, pp. 230-236, Mar. 1985. (Pubitemid 15148852)
76. R. Berger, S. Akselrod, D. Gordon, and R. Cohen, "An Efficient Algorithm for Spectral Analysis of Heart Rate Variability," IEEE Trans. Biomedical Eng., vol. 33, no. 9, pp. 900-904, Sept. 1986.
77. S. Tiinanen, M. Tulppo, and T. Seppänen, "RSA Component Extraction from Heart Rate Signal by Independent Component Analysis," Proc. Computers in Cardiology, pp. 161-164, 2010.
78. A. Ishchenko and P. Shev'ev, "Automated Complex for Multiparameter Analysis of the Galvanic Skin Response Signal," Biomedical Eng., vol. 23, no. 3, pp. 113-117, 1989. (Pubitemid 20254642)
79. H. Akaike, "Fitting Autoregressive Models for Prediction," Annals of the Inst. of Statistical Math., vol. 21, no. 1, pp. 243-247, 1969.
80. A. Boardman, F. Schlindwein, A. Rocha, and A. Leite, "A Study on the Optimum Order of Autoregressive Models for Heart Rate Variability," Physiological Measurement, vol. 23, pp. 325-336, 2002. (Pubitemid 34538279)
81. A. Camm et al., "Heart Rate Variability: Standards of Measurement, Physiological Interpretation, and Clinical Use," Circulation, vol. 93, no. 5, pp. 1043-1065, 1996.
82. S. Koelstra, A. Yazdani, M. Soleymani, C. Mühl, J. Lee, A. Nijholt, T. Pun, T. Ebrahimi, and I. Patras, "Single Trial Classification of EEG and Peripheral Physiological Signals for Recognition of Emotions Induced by Music Videos," Proc. Int'l Conf. Brain Informatics, pp. 89-100, 2010.
83. J. Mendel, "Tutorial on Higher-Order Statistics (Spectra) in Signal Processing and System Theory: Theoretical Results and Some Applications," Proc. IEEE, vol. 79, no. 3, pp. 278-305, Mar. 1991.
84. C. Nikias, Higher-Order Spectral Analysis: A Nonlinear Signal Processing Framework. PTR Prentice-Hall, Inc., 1993.
85. D.R. Brillinger, M. Rosenblatt, and P. Petropulu, "Computation and Interpretation of kth Order Spectra," Spectral Analysis of Time Series, pp. 189-232, Wiley, 1967.
86. T.N. Chang and S. Sun, "Blind Detection of Photomontage Using Higher Order Statistics," Proc. IEEE Int'l Symp. Circuits and Systems, 2004.
87. K. Chua, V. Chandran, U. Acharya, and C. Lim, "Cardiac State Diagnosis Using Higher Order Spectra of Heart Rate Variability," J. Medical Eng. & Technology, vol. 32, no. 2, pp. 145-155, 2008. (Pubitemid 351302367)
88. N. Marwan, M. Carmen Romano, M. Thiel, and J. Kurths, "Recurrence Plots for the Analysis of Complex Systems," Physics Reports, vol. 438, nos. 5/6, pp. 237-329, 2007.
89. A. Monk and A. Compton, "Review of Modern Physics," The Am. Physical Soc., pp. 173-179, 1939.
90. J. Eckmann, S. Kamphorst, and D. Ruelle, "Recurrence Plots of Dynamical Systems," Europhysics Letters, vol. 4, p. 973, 1987.
91. S. Schinkel, O. Dimigen, and N. Marwan, "Selection of Recurrence Threshold for Signal Detection," The European Physical J.-Special Topics, vol. 164, no. 1, pp. 45-53, 2008.
92. J. Zbilut and C. Webber Jr., Recurrence Quantification Analysis. Wiley Online Library, 2006.
93. C. Peng, S. Havlin, H. Stanley, and A. Goldberger, "Quantification of Scaling Exponents and Crossover Phenomena in Nonstationary Heartbeat Time Series," Chaos: An Interdisciplinary J. Nonlinear Science, vol. 5, no. 1, p. 82, 1995.
94. I. Jolliffe, Principal Component Analysis. Wiley Online Library, 2002.
95. R. Duda, P. Hart, and D. Stork, Pattern Classification. Citeseer, 2001.
96. K. Scherer, "Emotions are Emergent Processes: They Require a Dynamic Computational Architecture," Philosophical Trans. Royal Soc. B: Biological Sciences, vol. 364, no. 1535, pp. 3459-3474, 2009.
97. V. Marmarelis, Nonlinear Dynamic Modeling of Physiological Systems. Wiley-IEEE Press, 2004.
98. N. Fragopanagos and J. Taylor, "Emotion Recognition in Human-Computer Interaction," Neural Networks, vol. 18, no. 4, pp. 389-405, 2005. (Pubitemid 40922647)
99. R. Cowie, E. Douglas-Cowie, K. Karpouzis, G. Caridakis, M. Wallace, and S. Kollias, "Recognition of Emotional States in Natural Human-Computer Interaction," Multimodal User Interfaces, pp. 119-153, Springer, 2008.
100. D. Grandjean, D. Sander, and K. Scherer, "Conscious Emotional Experience Emerges as a Function of Multilevel, Appraisal-Driven Response Synchronization," Consciousness and Cognition, vol. 17, no. 2, pp. 484-495, 2008.