Learning deep physiological models of affect

IEEE Computational Intelligence Magazine

Volumn 8, Issue 2, 2013, Pages 20-33

  Martinez, Hector P ^a   Bengio, Yoshua ^b   Yannakakis, Georgios ^c  

^a IT UNIVERSITY OF COPENHAGEN   (Denmark)

^b UNIVERSITÉ DE MONTRÉAL   (Canada)

^c UNIVERSITY OF MALTA   (Malta)

Author keywords

[No Author keywords available]

Indexed keywords

AFFECTIVE ANNOTATIONS; AFFECTIVE COMPUTING; AFFECTIVE MODELING; COMPUTATIONAL MODELS OF EMOTIONS; HUMAN COMPUTER INTERACTION (HCI); MODELING METHODOLOGY; PHYSIOLOGICAL SIGNALS; RESEARCH QUESTIONS;

ARTIFICIAL INTELLIGENCE; HUMAN COMPUTER INTERACTION; PHYSIOLOGY; SIGNAL DETECTION;

PHYSIOLOGICAL MODELS;

EID: 84876251010     PISSN: 1556603X     EISSN: None     Source Type: Journal    
DOI: 10.1109/MCI.2013.2247823     Document Type: Article

References (69)

1. R. Picard, Affective Computing. Cambridge, MA: MIT Press, 2000.
2. R. Calvo and S. D'Mello, "Affect detection: An interdisciplinary review of models, methods, and their applications," IEEE Trans. Affective Comput., vol. 1, no. 1, pp. 18-37, 2010.
3. G. E. Hinton, S. Osindero, and Y. Teh, "A fast learning algorithm for deep belief nets," Neural Comput., vol. 18, no. 7, pp. 1527-1554, 2006. (Pubitemid 44024729)
4. Y. Bengio, "Learning deep architectures for AI," Found. Trends® Mach. Learn., vol. 2, no. 1, pp. 1-127, 2009.
5. I. Arel, D. Rose, and T. Karnowski, "Deep machine learning-A new frontier in artificial intelligence research [Research Frontier]," IEEE Comput. Intell. Mag., vol. 5, no. 4, pp. 13-18, Nov. 2010.
6. M. Dash and H. Liu, "Feature selection for classification," Intell. data anal., vol. 1, nos. 1-4, pp. 131-156, 1997.
7. M. Ranzato, F. Huang, Y. Boureau, and Y. LeCun, "Unsupervised learning of invariant feature hierarchies with applications to object recognition," in Proc. IEEE Conf. Computer Vision Pattern Recognition, 2007, pp. 1-8.
8. G. Hinton and R. Salakhutdinov, "Reducing the dimensionality of data with neural networks," Science, vol. 313, no. 5786, pp. 504-507, 2006. (Pubitemid 44148451)
9. Y. Bengio and O. Delalleau, "On the expressive power of deep architectures," in Algorithmic Learning Theory. Berlin, Germany: Springer-Verlag, 2011, pp. 18-36.
10. E. Vyzas and R. Picard, "Affective pattern classification," in Proc. AAAI 1998 Fall Symp. Emotional Intelligent: The Tangled Knot Cognition, pp. 176-182, 1998.
11. H. P. Martínez and G. N. Yannakakis, "Mining multimodal sequential patterns: A case study on affect detection," in Proc. 13th. Int. Conf. Multimodal Interfaces, 2011, pp. 3-10.
12. G. N. Yannakakis and J. Hallam, "Ranking vs. preference: A comparative study of selfreporting," in Proc. 4th Int. Conf. Affective Computing Intelligent Interaction, 2011, pp. 437-446.
13. J. Morris, "Observations: SAM: The self-assessment manikinan efficient cross-cultural measurement of emotional response," J. Advertising Res., vol. 35, no. 6, pp. 63-68, 1995.
14. J. Russell, "A circumplex model of affect," J. Personality Social Psychol., vol. 39, no. 6, p. 1161, 1980.
15. R. Picard, E. Vyzas, and J. Healey, "Toward machine emotional intelligence: Analysis of affective physiological state," IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 10, pp. 1175-1191, 2001. (Pubitemid 33047955)
16. D. Ververidis and C. Kotropoulos, "Automatic speech classification to five emotional states based on gender information," in Proc. Eusipco, Vienna, pp. 341-344, 2004.
17. D. Giakoumis, D. Tzovaras, K. Moustakas, and G. Hassapis, "Automatic recognition of boredom in video games using novel biosignal moment-based features," IEEE Trans. Affective Comput., vol. 2, no. 3, pp. 119-133, July-Sept. 2011.
18. G. N. Yannakakis and J. Hallam, "Entertainment modeling through physiology in physical play," Int. J. Human-Comput. Stud., vol. 66, no. 10, pp. 741-755, Oct. 2008.
19. S. Pincus, "Approximate entropy as a measure of system complexity," in Proc. National Academy Sciences, 1991, vol. 88, no. 6, pp. 2297-2301. (Pubitemid 21916403)
20. N. Lesh, M. Zaki, and M. Ogihara, "Mining features for sequence classification," in Proc. 5th ACM Int. Conf. Knowledge Discovery Data Mining, 1999, pp. 342-346.
21. Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle, "Greedy layerwise training of deep networks," in Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press 2007, vol. 19, p. 153.
22. P. Ekman and W. Friesen, "Facial action coding system: A technique for the measurement of facial movement," in From Appraisal to Emotion: Differences Among Unpleasant Feelings, Motivation and Emotion, P. C. Ellsworth, and C. A. Smith, Eds. Palo Alto, CA: Consulting Psychologists Press, 1988, vol. 12, pp. 271-302.
23. G. Caridakis, S. Asteriadis, K. Karpouzis, and S. Kollias, "Detecting human behavior emotional cues in natural interaction," in Proc. 17th Int. Conf. Digital Signal Processing, July 2011, pp. 1-6.
24. A. Kleinsmith and N. Bianchi-Berthouze, "Affective body expression perception and recognition: A survey," IEEE Trans. Affective Comput., vol. PP, no. 99, p. 1, 2012.
25. Y. LeCun and Y. Bengio, "Convolutional networks for images, speech, and time series," in The Handbook of Brain Theory and Neural Networks. Cambridge, MA: MIT Press 1995, vol. 3361, pp. 255-258.
26. M. Matsugu, K. Mori, Y. Mitari, and Y. Kaneda, "Subject independent facial expression recognition with robust face detection using a convolutional neural network," Neural Netw., vol. 16, no. 5, pp. 555-559, 2003. (Pubitemid 36794327)
27. S. Rifai, Y. Bengio, A. Courville, P. Vincent, and M. Mirza, "Disentangling factors of variation for facial expression recognition," in Proc. European Conf. Computer Vision, 2012, pp. 802-822.
28. J. Susskind, A. Anderson, and G. E. Hinton, "The Toronto face dataset," U. Toronto, Toronto, ON, Canada, Tech. Rep. UTML TR 2010-001, 2010.
29. A. Kapoor, W. Burleson, and R. Picard, "Automatic prediction of frustration," Int. J. Human-Comput. Stud., vol. 65, no. 8, pp. 724-736, 2007. (Pubitemid 46810294)
30. S. Tognetti, M. Garbarino, A. Bonarini, and M. Matteucci, "Modeling enjoyment preference from physiological responses in a car racing game," in Proc. IEEE Conf. Computational Intelligence Games, 2010, pp. 321-328.
31. C. Lee and S. Narayanan, "Toward detecting emotions in spoken dialogs," IEEE Trans. Speech Audio Processing, vol. 13, no. 2, pp. 293-303, 2005. (Pubitemid 40320247)
32. J. Wagner, J. Kim, and E. André, "From physiological signals to emotions: Implementing and comparing selected methods for feature extraction and classification," in Proc. IEEE Int. Conf. Multimedia and Expo, 2005, pp. 940-943. (Pubitemid 44669023)
33. G. N. Yannakakis, H. P. Martínez, and A. Jhala, "Towards affective camera control in games," User Model. User-Adapted Interact., vol. 20, no. 4, pp. 313-340, 2010.
34. H. P. Martínez and G. N. Yannakakis, "Genetic search feature selection for affective modeling: A case study on reported preferences," in Proc. 3rd Int. Workshop Affective Interaction Natural Environments, 2010, pp. 15-20.
35. D. Giakoumis, A. Drosou, P. Cipresso, D. Tzovaras, G. Hassapis, T. Zalla, A. Gaggioli, and G. Riva, "Using activity-related behavioural features towards more effective automatic stress detection," PLoS ONE, vol. 7, no. 9, p. e43571, 2012.
36. O. AlZoubi, R. Calvo, and R. Stevens, "Classification of EEG for affect recognition: An adaptive approach," in AI 2009 Proc. 22nd Australasian Joint Conf. Advances in Artificial Intelligence, pp. 52-61. 2009.
37. M. Soleymani, M. Pantic, and T. Pun, "Multimodal emotion recognition in response to videos," IEEE Trans. Affective Comput., vol. 3, no. 2, pp. 211-223, 2012.
38. S. Mcquiggan, B. Mott, and J. Lester, "Modeling self-efficacy in intelligent tutoring systems: An inductive approach," User Model. User-Adapted Interact., vol. 18, no. 1, pp. 81-123, 2008.
39. H. Gunes and M. Piccardi, "Bi-modal emotion recognition from expressive face and body gestures," J. Netw. Comput. Appl., vol. 30, no. 4, pp. 1334-1345, 2007. (Pubitemid 47263518)
40. R. Mandryk and M. Atkins, "A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies," Int. J. Human-Comput. Stud., vol. 65, no. 4, pp. 329-347, 2007. (Pubitemid 46216840)
41. J. F. Grafsgaard, K. E. Boyer, and J. C. Lester, "Predicting facial indicators of confusion with hidden Markov models," in Affective Computing and Intelligent Interaction, (Series Lecture Notes in Computer Science), S. D'Mello, A. Graesser, B. Schuller, and J.-C. Martin, Eds. Berlin, Germany: Springer-Verlag, 2011, vol. 6974, pp. 97-106.
42. R. Kaliouby and P. Robinson, "Real-time inference of complex mental states from facial expressions and head gestures," in Real-Time Vision Human-Computer Interaction. New York: Springer-Verlag, 2005, pp. 181-200.
43. H. Kobayashi and F. Hara, "Dynamic recognition of basic facial expressions by discrete-time recurrent neural network," in Proc. Int. Joint Conf. Neural Networks, Oct. 1993, vol. 1, pp. 155-158.
44. K. Kim, S. Bang, and S. Kim, "Emotion recognition system using short-term monitoring of physiological signals," Med. Biol. Eng. Comput., vol. 42, no. 3, pp. 419-427, 2004. (Pubitemid 38786706)
45. J. Hernandez, R. R. Morris, and R. W. Picard, "Call center stress recognition with person-specific models," in Affective Computing and Intelligent Interaction, (Series Lecture Notes in Computer Science), S. D'Mello, A. Graesser, B. Schuller, and J.-C. Martin, Eds. Berlin, Germany: Springer-Verlag, 2011, vol. 6974, pp. 125-134.
46. J. Bailenson, E. Pontikakis, I. Mauss, J. Gross, M. Jabon, C. Hutcherson, C. Nass, and O. John, "Real-time classification of evoked emotions using facial feature tracking and physiological responses," Int. J. Human-Computer Stud., vol. 66, no. 5, pp. 303-317, 2008.
47. J. Fürnkranz and E. Hüllermeier, "Preference learning," Künstliche Intell., vol. 19, no. 1, pp. 60-61, 2005.
48. G. N. Yannakakis, "Preference learning for affective modeling," in Proc. Int. Conf. Affective Computing Intelligent Interaction, Amsterdam, The Netherlands, Sept. 2009, pp. 126-131.
49. S. Tognetti, M. Garbarino, A. Bonanno, M. Matteucci, and A. Bonarini,"Enjoyment recognition from physiological data in a car racing game," in Proc. 3rd Int. Workshop Affective Interaction Natural Environments, 2010, pp. 3-8.
50. G. N. Yannakakis, J. Hallam, and H. H. Lund, "Entertainment capture through heart rate activity in physical interactive playgrounds," User Model. User-Adapted Interact., vol. 18, no. 1, pp. 207-243, 2008. (Pubitemid 351185637)
51. G. N. Yannakakis and J. Hallam, "Entertainment modeling through physiology in physical play," Int. J. Human-Comput. Stud., vol. 66, no. 10, pp. 741-755, 2008.
52. A. Krizhevsky, I. Sutskever, and G. Hinton, "ImageNet classification with deep convolutional neural networks," in Advances in Neural Information Processing Systems 25. Cambridge, MA: MIT Press, 2012.
53. C. Farabet, C. Couprie, L. Najman, Y. LeCun, "Learning hierarchical features for scene labeling," IEEE Trans. Pattern Anal. Mach. Intell., p. 1-15, 2013.
54. P. Hamel, S. Lemieux, Y. Bengio, and D. Eck, "Temporal pooling and multiscale learning for automatic annotation and ranking of music audio," in Proc. 12th Int. Conf. Music Information Retrieval, 2011, pp. 729-734.
55. P. Mirowski, Y. LeCun, D. Madhavan, and R. Kuzniecky, "Comparing SVM and convolutional networks for epileptic seizure prediction from intracranial EEG," in Proc. IEEE Workshop Machine Learning Signal Processing, 2008, pp. 244-249.
56. P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol, "Extracting and composing robust features with denoising autoencoders," in Proc. Int. Conf. Machine Learning, 2008, pp. 1096-1103.
57. D. Rumelhart, Backpropagation: Theory, Architectures, and Applications. Hillsdale, NJ: Lawrence Erlbaum, 1995.
58. B. Bai, J. Weston, D. Grangier, R. Collobert, K. Sadamasa, Y. Qi, O. Chapelle, and K. Weinberger, "Learning to rank with (a lot of ) word features," Inform. Retrieval, vol. 13, no. 3, pp. 291-314, 2010.
59. D. Grangier and S. Bengio, "Inferring document similarity from hyperlinks," in Proc. ACM Int. Conf. Information Knowledge Management, 2005, pp. 359-360. (Pubitemid 44022203)
60. G. E. Hinton and R. S. Zemel, "Autoencoders, minimum description length, and Helmholtz free energy," in Proc. Neural Information Processing System NIPS'1993, 1994, pp. 3-10.
61. G. Alain, Y. Bengio, and S. Rifai, "Regularized auto-encoders estimate local statistics," Dept. IRO, Université de Montréal, Montreal, QC, Canada, Tech. Rep. Arxiv Report 1211.4246, 2012.
62. Y. Bengio, A. Courville, and P. Vincent, "Unsupervised feature learning and deep learning: A review and new perspectives," Université de Montréal, Tech. Rep. Arxiv Report 1206.5538, 2012.
63. J. Masci, U. Meier, D. Cireşan, and J. Schmidhuber, "Stacked convolutional autoencoders for hierarchical feature extraction," in Proc. Int. Conf. Artificial Neural Networks and Machine Learning, pp. 52-59, 2011.
64. R. Collobert and J. Weston, "A unified architecture for natural language processing: Deep neural networks with multitask learning," in Proc. Int. Conf. Machine Learning, 2008, pp. 160-167.
65. J. Goldberger, S. Challapalli, R. Tung, M. Parker, and A. Kadish, "Relationship of heart rate variability to parasympathetic effect," Circulation, vol. 103, no. 15, p. 1977, 2001.
66. N. Ravaja, T. Saari, M. Salminen, J. Laarni, and K. Kallinen, "Phasic emotional reactions to video game events: A psychophysiological investigation," Media Psychol., vol. 8, no. 4, pp. 343-367, 2006.
67. H. P. Martínez, M. Garbarino, and G. N. Yannakakis, "Generic physiological features as predictors of player experience," Affective Comput. Intell. Interact., pp. 267-276, 2011.
68. W. Zhao, R. Chellappa, and A. Krishnaswamy, "Discriminant analysis of principal components for face recognition," in Proc 3rd IEEE Int. Conf. IEEE Automatic Face Gesture Recognition, 1998, pp. 336-341.
69. H. P. Martínez, K. Hullett, and G. N. Yannakakis, "Extending neuro-evolution preference learning through player modeling," in Proc. IEEE Conf. Computational Intelligence and Games, Copenhagen, Denmark, Aug. 2010, pp. 313-320.