Video affective content analysis: A survey of state-of-the-art methods

IEEE Transactions on Affective Computing

Volumn 6, Issue 4, 2015, Pages 410-430

  Wang, Shangfei ^a   Ji, Qiang ^b  

^a UNIVERSITY OF SCIENCE AND TECHNOLOGY OF CHINA   (China)

^b RENSSELAER POLYTECHNIC INSTITUTE   (United States)

Author keywords

And content based video retrieval; Emotion recognition; Video affective content analysis

Indexed keywords

IMAGE RETRIEVAL;

AUDIO-VISUAL FEATURES; AUTOMATIC ANALYSIS; CONTENT ANALYSIS; CONTENT-BASED VIDEO RETRIEVAL; DIRECT APPROACH; EMOTION RECOGNITION; STATE-OF-THE-ART METHODS; VIDEO CONTENTS;

SURVEYS;

EID: 84944354809     PISSN: 19493045     EISSN: None     Source Type: Journal    
DOI: 10.1109/TAFFC.2015.2432791     Document Type: Article

References (143)

1. F. Nack, C. Dorai, and S. Venkatesh, "Computational media aesthetics: Finding meaning beautiful, " IEEE MultiMedia, vol. 8, no. 4, pp. 10-12, Oct.-Dec. 2001.
2. A. Hanjalic and L.-Q. Xu, "Affective video content representation and modeling, " IEEE Trans. Multimedia, vol. 7, no. 1, pp. 143-154, Feb. 2005.
3. S. Wang, Y. Zhu, G. Wu, and Q. Ji, "Hybrid video emotional tagging using users EEG and video content, " Multimedia Tools Appl., vol. 72, no. 2, pp. 1257-1283, Sept. 2014.
4. S. Wang and X. Wang, "Emotional semantic detection from multimedia: A brief overview, " in Kansei Engineering and Soft Computing: Theory and Practice. Hershey, PA, USA: IGI Global, 2010.
5. M. Soleymani and M. Pantic, "Human-centered implicit tagging: Overview and perspectives, " in Proc. IEEE Int. Conf. Syst., Man Cybern., 2012, pp. 3304-3309.
6. P. Ekman, "Basic emotions, " in Handbook of Cognition and Emotion. Sussex, U.K.: Wiley, 1999, pp. 45-60.
7. W. M. Wundt, Grundzuge de physiologischen Psychologie. Leipzig, Germany: Engelman, 1905.
8. H.-B. Kang, "Affective content detection using HMMs, " in Proc. 11th ACM Int. Conf. Multimedia, New York, NY, USA, 2003, pp. 259-262.
9. H.-B. Kang, "Emotional event detection using relevance feedback, " in Proc. Int. Conf. Image Process., 2003, vol. 1, pp. I- 721-I-724.
10. K. Sun and J. Yu, "Video affective content representation and recognition using video affective tree and hidden Markov models, " in Proc. 2nd Int. Conf. Affective Comput. Intell. Interaction, 2007, pp. 594-605.
11. R. M. A. Teixeira, T. Yamasaki, and K. Aizawa, "Determination of emotional content of video clips by lowlevel audiovisual features, " Multimedia Tools Appl., vol. 61, no. 1, pp. 21-49, 2012.
12. M. Xu, X. He, J. S. Jin, Y. Peng, C. Xu, and W. Guo, "Using scripts for affective content retrieval, " in Proc. Adv. Multimedia Inf. Process. 11th Pacific Rim Conf. Multimedia, Shanghai, China, 2010, pp. 43-51.
13. X. Y. Chen and Z. Segall, "XV-Pod: An emotion aware, affective mobile video player, " in Proc. WRI World Congr. Comput. Sci. Inf. Eng., 2009, pp. 277-281.
14. G. Irie, T. Satou, A. Kojima, T. Yamasaki, and K. Aizawaki, "Affective audio-visual words and latent topic driving model for realizing movie affective scene classification, " IEEE Trans. Multimedia, vol. 12, no. 6, pp. 523-535, Oct. 2010.
15. H. L. Wang and L.-F. Cheong, "Affective understanding in film, " IEEE Trans. Circuits Syst. Video Technol., vol. 16, no. 6, pp. 689- 704, Jun. 2006.
16. K.-M. Ong and W. Kameyama, "Classification of video shots based on human affect, " Inf. Media Technol., vol. 4, no. 4, pp. 903- 912, 2009.
17. A. Yazdani, J.-S. Lee, and T. Ebrahimi, "Implicit emotional tagging of multimedia using EEG signals and brain computer interface, " in Proc. 1st SIGMM Workshop Social Media, 2009, pp. 81-88.
18. S. Arifin and P. Y. K. Cheung, "A computation method for video segmentation utilizing the pleasure-arousal-dominance emotional information, " in Proc. 15th Int. Conf. Multimedia, 2007, pp. 68-77.
19. S. Arifin and P. Cheung, "Affective level video segmentation by utilizing the pleasure-arousal-dominance information, " IEEE Trans. Multimedia, vol. 10, no. 7, pp. 1325-1341, Nov. 2008.
20. S. Zhao, H. Yao, X. Sun, P. Xu, X. Liu, and R. Ji, "Video indexing and recommendation based on affective analysis of viewers, " in Proc. 19th ACM Int. Conf. Multimedia, 2011, pp. 1473-1476.
21. H.-W. Yoo and S.-B. Cho, "Video scene retrieval with interactive genetic algorithm, " Multimedia Tools Appl., vol. 34, no. 3, pp. 317- 336, 2007.
22. S. C. Watanapa, B. Thipakorn, and N. Charoenkitkarn, "A sieving ANN for emotion-based movie clip classification, " IEICE Trans. Inf. Syst., vol. 91, no. 5, pp. 1562-1572, May 2008.
23. M. Xu, J. Wang, X. He, J. S. Jin, S. Luo, and H. Lu, "A three-level framework for affective content analysis and its case studies, " Multimedia Tools Appl., vol. 70, no. 2, pp. 757-779, May 2014.
24. X. Ding, B. Li, W. Hu, W. Xiong, and Z. Wang, "Horror video scene recognition based on multi-view multi-instance learning, " in Proc. 11th Asian Conf. Comput. Vis., 2013, pp. 599-610.
25. A. G. Money and H. Agius, "Feasibility of personalized affective video summaries, " in Affect and Emotion Human-Computer Interaction. Berlin, Germany: Springer-Verlag, 2008, pp. 194-208.
26. A. G. Money and H. Agius, "Analysing user physiological responses for affective video summarisation, " Displays, vol. 30, no. 2, pp. 59-70, 2009.
27. L. Canini, S. Benini, and R. Leonardi, "Affective recommendation of movies based on selected connotative features, " IEEE Trans. Circuits and Systems for Video Technol., vol. 23, no. 4, pp. 636-647, Apr. 2013.
28. S. Koelstra and I. Patras, "Fusion of facial expressions and EEG for implicit affective tagging, " Image Vis. Comput., vol. 31, no. 2, pp. 164-174, 2013.
29. M. Soleymani, M. Pantic, and T. Pun, "Multimodal emotion recognition in response to videos, " IEEE Trans. Affective Comput., vol. 3, no. 2, pp. 211-223, Apr.-Jun. 2012.
30. Y. Cui, S. Luo, Q. Tian, S. Zhang, Y. Peng, L. Jiang, and J. Jin, "Mutual information-based emotion recognition, " in The Era of Interactive Media. New York, NY, USA: Springer, 2013, pp. 471-479.
31. S. Zhang, Q. Huang, S. Jiang, W. Gao, and Q. Tian, "Affective visualization and retrieval for music video, " IEEE Trans. Multimedia, vol. 12, no. 6, pp. 510-522, Oct. 2010.
32. L. Canini, S. Gilroy, M. Cavazza, R. Leonardi, and S. Benini, "Users' response to affective film content: A narrative perspective, " in Proc. Int. Workshop Content-Based Multimedia Indexing, Grenoble, France, Jun. 2010, pp. 1-6.
33. H. Katti, K. Yadati, M. Kankanhalli, and C. Tat-Seng, "Affective video summarization and story board generation using pupillary dilation and eye gaze, " in Proc. IEEE Int. Symp. Multimedia, 2011, pp. 319-326.
34. J. J. Kierkels, M. Soleymani, and T. Pun, "Queries and tags in affect-based multimedia retrieval, " in Proc. IEEE Int. Conf. Multimedia Expo, Jul. 2009, pp. 1436-1439.
35. S. Koelstra, C. Muhl, M. Soleymani, J.-S. Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt, and I. Patras, "DEAP: A database for emotion analysis; using physiological signals, " IEEE Trans. Affective Comput., vol. 3, no. 1, pp. 18-31, Jan.-Mar. 2012.
36. J. A. Russell, "A circumplex model of affect, " J. Personality Social Psychol., vol. 39, no. 6, pp. 1161-1178, 1980.
37. H. Gunes, B. Schuller, M. Pantic, and R. Cowie, "Emotion representation, analysis and synthesis in continuous space: A survey, " in Proc. IEEE Int. Conf. Automat. Face Gesture Recog., 1st Int. Workshop Emotion Synthesis, rePresentation, Anal. Continuous spacE, Santa Barbara, CA, USA, Mar. 2011, pp. 827-834.
38. Y. Cui, J. S. Jin, S. Zhang, S. Luo, and Q. Tian, "Music video affective understanding using feature importance analysis, " in Proc. ACM Int. Conf. Image Video Retrieval, New York, NY, USA, 2010, pp. 213-219.
39. S. Zhang, Q. Tian, Q. Huang, W. Gao, and S. Li, "Utilizing affective analysis for efficient movie browsing, " in Proc. 16th IEEE Int. Conf. Image Process., 2009, pp. 1853-1856.
40. A. Hanjalic and L.-Q. Xu, "User-oriented affective video content analysis, " in Proc. IEEE Workshop Content-Based Access Image Video Libraries, 2001, pp. 50-57.
41. A. Hanjalic, "Multimodal approach to measuring excitement in video, " in Proc. Int. Conf. Multimedia Expo, 2003, pp. 289-292.
42. S. Zhang, Q. Tian, S. Jiang, Q. Huang, and W. Gao, "Affective MTV analysis based on arousal and valence features, " in Proc. IEEE Int. Conf. Multimedia Expo, 2008, pp. 1369-1372.
43. L. Canini, S. Benini, P. Migliorati, and R. Leonardi, "Emotional identity of movies, " in Proc. 16th IEEE Int. Conf. Image Process., 2009, pp. 1821-1824.
44. C. H. Chan and G. J. Jones, An Affect-Based Video Retrieval System with Open Vocabulary Querying. New York, NY, USA: Springer, 2011.
45. C. H. Chan and G. J. Jones, "Affect-based indexing and retrieval of films, " in Proc. 13th Annu. ACM Int. Conf. Multimedia, New York, NY, USA, 2005, pp. 427-430.
46. J. H. French, "Automatic affective video indexing: Sound energy and object motion correlation discovery: Studies in identifying slapstick comedy using low-level video features, " in Proc. IEEE Southeastcon, 2013, pp. 1-6.
47. S. Moncrieff, C. Dorai, and S. Venkatesh, "Affect computing in film through sound energy dynamics, " in Proc. 9th ACM Int. Conf. Multimedia, New York, NY, USA, 2001, pp. 525-527.
48. M. Xu, C. Xu, X. He, J. S. Jin, S. Luo, and Y. Rui, "Hierarchical affective content analysis in arousal and valence dimensions, " Signal Process., vol. 93, no. 8, pp. 2140-2150, 2013.
49. A. Yazdani, E. Skodras, N. Fakotakis, and T. Ebrahimi. (2013). Multimedia content analysis for emotional characterization of music video clips, EURASIP J. Image Video Process. 1(26), pp. 1-10 [Online]. Available: Http://jivp.eurasipjournals.com/content/2013/1/26.
50. A. Yazdani, K. Kappeler, and T. Ebrahimi, "Affective content analysis of music video clips, " in Proc. 1st Int. ACM Workshop Music Inf. Retrieval User-Centered Multimodal Strategies, New York, NY, USA, 2011, pp. 7-12.
51. L. Lu, H. Jiang, and H. Zhang, "A robust audio classification and segmentation method, " in Proc. 9th ACM Int. Conf. Multimedia, 2001, pp. 203-211.
52. R. Bachu, S. Kopparthi, B. Adapa, and B. Barkana, "Separation of voiced and unvoiced using zero crossing rate and energy of the speech signal, " in Proc. Amer. Soc. Eng. Edu. Zone Conf., 2008, pp. 1-7.
53. M. Radmard and M. Hadavi, M. M. Nayebi, "A new method of voiced/unvoiced classification based on clustering, " J. Signal Inf. Process., vol. 2, no. 04, pp. 336-247, May 2011.
54. T. Zhang and C.-C. Kuo, "Audio content analysis for online audiovisual data segmentation and classification, " IEEE Trans. Speech Audio Process., vol. 9, no. 4, pp. 441-457, May 2001.
55. C. E. Williams and K. N. Stevens, "Vocal correlates of emotional states, " in Speech Evaluation in Psychiatry. New York, NY, USA: Grune and Stratton, 1981, pp. 221-240.
56. K. R. Scherer and M. R. Zentner, "Emotional effects of music: Production rules, " in Music and Emotion: Theory and Research. London, U.K.: Oxford Univ. Press, 2001, pp. 361-392.
57. R. W. Picard, Affective Computing. Cambridge, MA, USA: MIT Press, 2000.
58. R. Banse and K. R. Scherer, "Acoustic profiles in vocal emotion expression." J. Personality Social Psychol., vol. 70, no. 3, p. 614, 1996.
59. D. Wu, T. D. Parsons, and S. S. Narayanan, "Acoustic feature analysis in speech emotion primitives estimation, " in Proc. Interspeech, 2010, pp. 785-788.
60. M. El-Ayadi, M. S. Kamel, and F. Karray, "Survey on speech emotion recognition: Features, classification schemes, and databases, " Pattern Recognition, vol. 44, no. 3, pp. 572-587, 2011.
61. M. Xu, J. S. Jin, S. Luo, and L. Duan, "Hierarchical movie affective content analysis based on arousal and valence features, " in Proc. 16th ACM Int. Conf. Multimedia, 2008, pp. 677-680.
62. P. N. Juslin and J. A. Sloboda, Music and Emotion: Theory and Research. London, U.K.: Oxford Univ. Press, 2001.
63. D. Liu, L. Lu, and H. Zhang, "Automatic mood detection from acoustic music data, " in Proc. 4th Int. Conf. Music Inf. Retrieval, 2003, pp. 81-87.
64. Y. E. Kim, E. M. Schmidt, R. Migneco, B. G. Morton, P. Richardson, J. J. Scott, J. A. Speck, and D. Turnbull, "State of the art report: Music emotion recognition: A state of the art review, " in Proc. 11th Int. Soc. Music Inf. Retrieval Conf., 2010, pp. 255-266.
65. Y. H. Yang and H. H. Chen, "Machine recognition of music emotion: A review, " ACM Trans. Intell. Syst. Technol., vol. 3, no. 3, pp. 40:1-40:30, 2012.
66. T. Eerola and J. K. Vuoskoski, "A review of music and emotion studies: Approaches, emotion models, and stimuli, " Music Perception, vol. 30, no. 3, pp. 307-340, 2013.
67. D. Bordwell, K. Thompson, and J. Ashton, Film Art: An Introduction, vol. 7. New York, NY, USA: McGraw-Hill, 1997.
68. G. M. Smith, Film Structure and the Emotion System. Cambridge, U.K.: Cambridge Univ. Press, 2007.
69. C. R. Plantinga and G. M. Smith, Passionate Views: Film, Cognition, and Emotion. Baltimore, MD, USA: Johns Hopkins Univ. Press, 1999.
70. B. H. Detenber, R. F. Simons, and G. G. Bennett Jr, "Roll em!: The effects of picture motion on emotional responses, " J. Broadcasting Electron. Media, vol. 42, no. 1, pp. 113-127, 1998.
71. R. F. Simons, B. H. Detenber, T. M. Roedema, and J. E. Reiss, "Emotion processing in three systems: The medium and the message, " Psychophysiol., vol. 36, no. 5, pp. 619-627, 1999.
72. B. Adams, C. Dorai, and S. Venkatesh, "Novel approach to determining tempo and dramatic story sections in motion pictures, " in Proc. Int. Conf. Image Process., 2000, vol. 2, pp. 283-286.
73. K. Choros, "Video shot selection and content-based scene detection for automatic classification of tv sports news, " in Internet- Technical Development and Applications. New York, NY, USA: Springer, 2009, pp. 73-80.
74. The why, the where and the when to move the camera [Online]. Available: Http://filmmakeriq.com/2013/04/the-why-the-whereand-the-when-to-move-the-camera/, 2013.
75. Movie making manual, wikibook [Online]. Available: Http://en.wikibooks.org/wiki/Movie-Making-Manual/Cinematography/Moving-the-camera, 2015.
76. Z. Herbert, Sight, Sound, Motion: Applied Media Aesthetics. Belmont, CA, USA: Wadsworth 1999.
77. K. McLaren, "XIII the development of the cie 1976 (l∗ a∗ b∗) uniform colour space and colour-difference formula, " J. Soc. Dyers Colourists, vol. 92, no. 9, pp. 338-341, 1976.
78. G. R. Bradski and J. W. Davis, "Motion segmentation and pose recognition with motion history gradients, " Mach. Vis. Appl., vol. 13, no. 3, pp. 174-184, 2002.
79. P. Valdez and A. Mehrabian, "Effects of color on emotions, " J. Experimental Psychol.: General, vol. 123, no. 4, pp. 394-409, 1994.
80. C. Y. Wei, N. Dimitrova, and S.-F. Chang, "Color-mood analysis of films based on syntactic and psychological models, " in Proc. IEEE Int. Conf. Multimedia Expo, 2004, pp. 831-834.
81. S. Arifin and P. Y. K. Cheung, "A novel probabilistic approach to modeling the pleasure-arousal-dominance content of the video based on 'working memory', " in Proc. Int. Conf. Semantic Comput., 2007, pp. 147-154.
82. T. G. Dietterich, R. H. Lathrop, and T. Lozano-Perez, "Solving the multiple instance problem with axis-parallel rectangles, " Artif. Intell., vol. 89, no. 1, pp. 31-71, 1997.
83. J. Wang, B. Li, W. Hu, and O. Wu, "Horror video scene recognition via multiple-instance learning, " in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2011, pp. 1325-1328.
84. M. Xu, L.-T. Chia, and J. Jin, "Affective content analysis in comedy and horror videos by audio emotional event detection, " in Proc. IEEE Int. Conf. Multimedia Expo, 2005, p. 4.
85. Z. Wang, Y. Li, S. Wang, and Q. Ji, "Capturing global semantic relationships for facial action unit recognition, " in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 3304-3311.
86. S. Nie and Q. Ji, "Capturing global and local dynamics for human action recognition, " in Proc. 22nd Int. Conf. Pattern Recog., 2014, pp. 1946-1951.
87. E. Acar, F. Hopfgartner, and S. Albayrak, "Understanding affective content of music videos through learned representations, " in Proc. 20th Anniversary Int. Conf. Multi Media Model., 2014, pp. 303-314.
88. I. J. Roseman, "Cognitive determinants of emotion: A structural theory, " in Review of Personality & Social Psychology. Newbury Park, CA, USA: Sage, 1984.
89. K. R. Scherer, "What are emotions? and how can they be measured?" Social Sci. Inf., vol. 44, no. 4, pp. 695-729, 2005.
90. S. Wang, Z. Liu, Y. Zhu, M. He, X. Chen, and Q. Ji, "Implicit video emotion tagging from audiences facial expression, " Multimedia Tools Appl., vol. 32, no. 10, pp. 682-691, Jan. 2014.
91. M. Soleymani, G. Chanel, J. Kierkels, and T. Pun, "Affective characterization of movie scenes based on multimedia content analysis and user's physiological emotional responses, " in Proc. 10th IEEE Int. Symp. Multimedia, Dec. 2008, no. 10455061, pp. 228-235.
92. M. Pantic and A. Vinciarelli, "Implicit human-centered tagging [social sciences], " IEEE Signal Process. Mag., vol. 26, no. 6, pp. 173-180, 2009.
93. S. D. Kreibig, "Autonomic nervous system activity in emotion: A review, " Biological Psychol., vol. 84, no. 3, pp. 394-421, 2010.
94. P. Rainville, A. Bechara, N. Naqvi, and A. R. Damasio, "Basic emotions are associated with distinct patterns of cardiorespiratory activity, " Int. J. Psychophysiol., vol. 61, no. 1, pp. 5-18, 2006.
95. M. K. Abadi, M. Kia, R. Subramanian, P. Avesani, and N. Sebe, "Decoding affect in videos employing the MEG brain signal, " in Proc. 10th IEEE Int. Conf. Workshops Autom. Face Gesture Recog., 2013, pp. 1-6.
96. C. Chenes, G. Chanel, M. Soleymani, and T. Pun, "Highlight detection in movie scenes through inter-users, physiological linkage, " in Social Media Retrieval. New York, NY, USA: Springer, 2013, pp. 217-237.
97. J. Fleureau, P. Guillotel, and Q. Huynh-Thu, "Physiologicalbased affect event detector for entertainment video applications, " IEEE Trans. Affective Comput., vol. 3, no. 3, pp. 379-385, Jul.-Sep. 2012.
98. S. Koelstra, C. Muhl, and I. Patras, "EEG analysis for implicit tagging of video data, " in Proc. 3rd Int. Conf. Affective Comput. Intell. Interaction Workshops, 2009, pp. 1-6.
99. S. Koelstra, A. Yazdani, M. Soleymani, C. Muhl, J.-S. Lee, A. Nijholt, T. Pun, T. Ebrahimi, and I. Patras, "Single trial classification of EEG and peripheral physiological signals for recognition of emotions induced by music videos, " in Proc. Int. Conf. Brain Informatics, 2010, pp. 89-100.
100. A. T. Krzywicki, G. He, and B. L. O'Kane, "Analysis of facial thermal variations in response to emotion: Eliciting film clips, " in Proc. SPIE Defense, Security, Sensing, 2009, pp. 734312-734312.
101. A. G. Money and H. Agius, "ELVIS: Entertainment-led video summaries, " ACM Trans. Multimedia Comput. Comm. Appl., vol. 6, no. 3, pp. 17:1-17:30, 2010.
102. M. Soleymani, G. Chanel, J. J. Kierkels, and T. Pun, "Affective ranking of movie scenes using physiological signals and content analysis, " in Proc. 2nd ACM Workshop Multimedia Semantics, 2008, pp. 32-39.
103. M. Soleymani and M. Pantic, "Multimedia implicit tagging using EEG signals, " in Proc. IEEE Int. Conf. Multimedia Expo, 2013, pp. 1-6.
104. A. F. Smeaton and S. Rothwell, "Biometric responses to musicrich segments in films: The CDVPlex, " in Proc. 7th Int. Workshop Content-Based Multimedia Indexing, 2009, pp. 162-168.
105. S. Toyosawa and T. Kawai, "An experience oriented video digesting method using heart activity and its applicable video types, " in Proc. 11th Pacific Rim Conf. Adv. Multimedia Inf. Process., 2010, pp. 260-271.
106. M. K. Abadi, S. M. Kia, R. Subramanian, P. Avesani, and N. Sebe, "User-centric affective video tagging from MEG and peripheral physiological responses, " in Proc. Humaine Assoc. Conf. Affective Comput. Intell. Interaction, Geneva, Switzerland, Sep. 2013, pp. 582-587.
107. H. Joho, J. M. Jose, R. Valenti, and N. Sebe, "Exploiting facial expressions for affective video summarisation, " in Proc. ACM Int. Conf. Image Video Retrieval, 2009, pp. 31:1-31:8.
108. D. McDuff, R. El Kaliouby, D. Demirdjian, and R. Picard, "Predicting online media effectiveness based on smile responses gathered over the internet, " in Proc. 10th IEEE Int. Conf. Workshops Autom. Face Gesture Recog., 2013, pp. 1-7.
109. W.-T. Peng, C.-H. Chang, W.-T. Chu, W.-J. Huang, C.-N. Chou, W.-Y. Chang, and Y.-P. Hung, "A real-time user interest meter and its applications in home video summarizing, " in Proc. IEEE Int. Conf. Multimedia Expo, 2010, pp. 849-854.
110. W. T. Peng, W. J. Huang, W. T. Chu, C. N. Chou, W. Y. Chang, C. H. Chang, and Y. P. Hung, "A user experience model for home video summarization, " in Proc. 15th Int. Multimedia Model. Conf. Advances Multimedia Model., 2009, pp. 484-495.
111. M. Yeasin, B. Bullot, and R. Sharma, "Recognition of facial expressions and measurement of levels of interest from video, " IEEE Trans. Multimedia, vol. 8, no. 3, pp. 500-508, Jun. 2006.
112. J. A. Coan and J. J. Allen, Handbook of Emotion Elicitation and Assessment. London, U.K.: Oxford Univ. Press, 2007.
113. W. T. Peng, W. T. Chu, C. H. Chang, C. N. Chou, W. J. Huang, W. Y. Chang, and Y. P. Hung, "Editing by viewing: Automatic home video summarization by viewing behavior analysis, " IEEE Trans. Multimedia, vol. 13, no. 3, pp. 539-550, Jun. 2011.
114. I. Arapakis, Y. Moshfeghi, H. Joho, R. Ren, D. Hannah, and J. M. Jose, "Enriching user profiling with affective features for the improvement of a multimodal recommender system, " in Proc. ACM Int. Conf. Image Video Retrieval, 2009, pp. 29:1-29:8.
115. Z. Liu, S. Wang, Z. Wang, and Q. Ji, "Implicit video multi-emotion tagging by exploiting multi-expression relations, " in Proc. 10th IEEE Int. Conf. Workshops Autom. Face Gesture Recog., 2013, pp. 1-6.
116. M. Soleymani, J. Lichtenauer, T. Pun, and M. Pantic, "A multimodal database for affect recognition and implicit tagging, " IEEE Trans. Affective Comput., vol. 3, no. 1, pp. 42-55, Jan.-Mar. 2012.
117. M. Soleymani, "Implicit and automated emotional tagging of videos, " Ph.D. dissertation, Univ. Geneva, London, U.K. 2011.
118. I. Arapakis, I. Konstas, and J. M. Jose, "Using facial expressions and peripheral physiological signals as implicit indicators of topical relevance, " in Proc. 17th ACM Int. Conf. Multimedia, 2009, pp. 461-470.
119. Y. Baveye, J.-N. Bettinelli, E. Dellandrea, L. Chen, and C. Chamaret, "A large video database for computational models of induced emotion, " in Proc. Humaine Assoc. Conf. Affective Comput. Intell. Interaction, Geneva, Switzerland, Sep. 2013, pp. 13-18.
120. A. Schaefer, F. Nils, X. Sanchez, and P. Philippot, "Assessing the effectiveness of a large database of emotion-eliciting films: A new tool for emotion researchers, " Cognition Emotion, vol. 24, no. 7, pp. 1153-1172, 2010.
121. M. Soleymani and M. Larson, "Crowdsourcing for affective annotation of video: Development of a viewer-reported boredom corpus, " in Proc. SIGIR Workshop Crowdsourcing Search Eval., 2010.
122. J. Tarvainen, M. Sjoberg, S. Westman, J. Laaksonen, and P. Oittinen, "Content-based prediction of movie style, aesthetics and affect: Data set and baseline experiments, " IEEE Trans. Multimedia, vol. 16, no. 8, pp. 2085-2098, Dec. 2014.
123. S. Wang, Z. Liu, S. Lv, Y. Lv, G. Wu, P. Peng, F. Chen, and X. Wang, "A natural visible and infrared facial expression database for expression recognition and emotion inference, " IEEE Trans. Multimedia, vol. 12, no. 7, pp. 682-691, Nov. 2010.
124. M. Abadi, R. Subramanian, S. Kia, P. Avesani, I. Patras, and N. Sebe, "DECAF: MEG-based multimodal database for decoding affective physiological responses, " IEEE Trans. Affective Comput., no. 99, 2015, Doi:10.1109/TAFFC.2015.2392932.
125. J. D. Laird and C. Bressler, "The process of emotion experience: A self-perception theory, " Margaret S. Clark (Ed), Emotion Review of personality and social psychology, Thousand Oaks, CA, US: Sage Publications, Inc, no. 13, pp. 213-234, 1992.
126. J. Healey, "Recording affect in the field: Towards methods and metrics for improving ground truth labels, " in Proc. 4th Int. Conf. Affective Comput. Intell. Interaction, 2011, pp. 107-116.
127. J. J. Gross and R. W. Levenson, "Emotion elicitation using films, " Cognition Emotion, vol. 9, no. 1, pp. 87-108, 1995.
128. P. Philippot, "Inducing and assessing differentiated emotionfeeling states in the laboratory, " Cognition Emotion, vol. 7, no. 2, pp. 171-193, 1993.
129. Z. Wang, S. Wang, M. He, Z. Liu, and Q. Ji, "Emotional tagging of videos by exploring multiple emotions' coexistence, " in Proc. 10th IEEE Int. Conf. Workshops Autom. Face Gesture Recog., 2013, pp. 1-6.
130. S. Benini, L. Canini, and R. Leonardi, "A connotative space for supporting movie affective recommendation, " IEEE Trans. Multimedia, vol. 13, no. 6, pp. 1356-1370, Dec. 2011.
131. Y. Bengio, "Learning deep architectures for AI, " Found. Trends Mach. Learning, vol. 2, no. 1, pp. 1-127, Jan. 2009.
132. M. Ranzato, J. Susskind, V. Mnih, and G. Hinton, "On deep generative models with applications to recognition, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2011, pp. 2857-2864.
133. D. Yu, G. Hinton, N. Morgan, J. T. Chien, and S. Sagayama, "Introduction to the special section on deep learning for speech and language processing, " IEEE Trans. Audio, Speech, Language Process., vol. 20, no. 1, pp. 4-6, Jan. 2012.
134. Z. Wang, S. Lyu, G. Schalk, and Q. Ji, "Learning with target prior, " in Proc. Annu. Conf. Neural Inf. Process. Syst., 2012, pp. 2231- 2239.
135. Z. Wang, S. Lyu, G. Schalk, and Q. Ji, "Deep feature learning using target priors with applications in ECoG signal decoding for BCI, " in Proc. 23rd Int. Joint Conf. Artif. Intell., 2013, pp. 1785- 1791.
136. X. Hu, K. Li, J. Han, X. Hua, L. Guo, and T. Liu, "Bridging the semantic gap via functional brain imaging, " in IEEE Trans. Multimedia, vol. 14, no. 2, pp. 314-325, Apr. 2012.
137. V. Vapnik and A. Vashist, "A new learning paradigm: Learning using privileged information, " Neural Netw., vol. 22, no. 5, pp. 544-557, 2009.
138. P. Winoto and T. Y. Tang, "The role of user mood in movie recommendations, " Expert Syst. Appl., vol. 37, no. 8, pp. 6086-6092, 2010.
139. M. Soleymani, J. Kierkels, G. Chanel, and T. Pun, "A Bayesian framework for video affective representation, " in Proc. 3rd Int. Conf. Affective Comput. Intell. Interaction Workshops, 2009, pp. 1-7.
140. C. Lithari, C. Frantzidis, C. Papadelis, A. B. Vivas, M. Klados, C. Kourtidou-Papadeli, C. Pappas, A. Ioannides, and P. Bamidis, "Are females more responsive to emotional stimuli? a neurophysiological study across arousal and valence dimensions, " Brain Topography, vol. 23, no. 1, pp. 27-40, 2010.
141. R. Srivastava, S. Yan, T. Sim, and S. Roy, "Recognizing emotions of characters in movies, " in Proc. IEEE Int. Conf. Acoust., Speech Signal Proc., 2012, pp. 993-996.
142. M. Soleymani, M. Larson, T. Pun, and A. Hanjalic, "Corpus development for affective video indexing, " IEEE Trans. Multimedia, vol. 16, no. 4, pp. 1075-1089, Jun. 2014.
143. D. Hovy, T. Berg-Kirkpatrick, A. Vaswani, and E. H. Hovy, "Learning whom to trust with mace, " in Proc. HLT-NAACL, 2013, pp. 1120-1130.