Affective audio-visual words and latent topic driving model for realizing movie affective scene classification

IEEE Transactions on Multimedia

Volumn 12, Issue 6, 2010, Pages 523-535

  Irie, Go ^a,b   Satou, Takashi ^a   Kojima, Akira ^a   Yamasaki, Toshihiko ^b   Aizawa, Kiyoharu ^b  

^a NTT CORPORATION   (Japan)

^b UNIVERSITY OF TOKYO   (Japan)

Author keywords

Affective audio visual word; affective scene classification; latent topic driving model; Plutchik's basic emotions

Indexed keywords

AUDIO-VISUAL; AUDIO-VISUAL FEATURES; BASIC EMOTIONS; CLASSIFICATION MODELS; CONVENTIONAL APPROACH; LATENT TOPIC DRIVING MODEL; MARKOVIAN DYNAMIC SYSTEMS; MOVIE SCENES; NOVEL METHODS; SCENE CLASSIFICATION; USER-CENTRIC;

FEATURE EXTRACTION;

MOTION PICTURES;

EID: 77956779599     PISSN: 15209210     EISSN: None     Source Type: Journal    
DOI: 10.1109/TMM.2010.2051871     Document Type: Article

References (43)

1. M. S. Lew, N. Sebe, C. Djeraba, and R. Jain, "Content-based multimedia information retrieval: State of the art and challenges," ACM Trans. Multimedia Comput., Commun., Appl., vol.2, no.1, pp. 1-19, 2006.
2. A. Hanjalic and L.-Q. Xu, "Affective video content representation and modeling," IEEE Trans. Multimedia, vol.7, no.1, pp. 143-154, Feb. 2005.
3. B. T. Truong and S. Venkatesh, "Video abstraction: A systematic review and classification," ACM Trans. Multimedia Comput., Commun., Appl., vol.3, no.1, article no. 3, 2007.
4. Z. Xiong, R. Radhakrishnan, and A. Divakaran, "Generation of sports highlights using motion activity in combination with a common audio feature extraction framework," in Proc. Int. Conf. Image Processing (ICIP), 2003, pp. 5-8.
5. S. Nepal, U. Srinivasan, and G. Reynolds, "Automatic detection of 'Goal' segments in basketball videos," in Proc. ACM Int. Conf. Multimedia (ACM MM), 2001, pp. 261-269.
6. Y. Rui, A. Gupta, and A. Acero, "Automatically extracting highlights for TV baseball programs," in Proc. ACM Int. Conf. Multimedia (ACM MM), 2000, pp. 105-115.
7. Z. Zhao, S. Jiang, Q. Huang, and G. Zhu, "Highlight summarization in sports video based on replay detection," in Proc. IEEE Int. Conf. Multimedia & Expo. (ICME), 2006, pp. 1613-1616.
8. S. Lu, I. King, and M. Lyu, "Video summarization using greedy method in a constraint satisfaction framework," in Proc. Int. Conf. Distributed Multimedia Systems, 2003, pp. 456-461.
9. C. Xu, X. Shao, N. C. Maddage, and M. S. Kankanhalli, "Automatic music video summarization based on audio-visual-text analysis and alignment," in Proc. ACM SIGIR Conf. (SIGIR), 2005, pp. 361-368.
10. A. Lang, K. Dhillon, and Q. Dong, "The effects of emotional arousal and valence on television viewers' cognitive capacity and memory," J. Broadcast. Electron. Media, vol.39, no.3, pp. 313-327, 1995.
11. W. Schultz and A. Dickinson, "Neuronal coding of prediction errors," Annu. Rev. Neurosci., vol.23, pp. 473-500, 2000.
12. P. C. Fletcher, J.M. Anderson, and D. R. Shanks, "Responses of human frontal cortex to surprising events are predicted by formal associative learning theory," Nature Neurosci., vol.4, no.10, pp. 1043-1048, 2001.
13. G. Irie, K. Hidaka, T. Satou, A. Kojima, T. Yamasaki, and K. Aizawa, "Latent topic driving model for movie affective scene classification," in Proc. ACM Int. Conf. Multimedia (ACM MM), 2009, pp. 565-568.
14. L. F. Barrett, "Discrete emotions or dimensions? The role of valence focus and arousal focus," Cognit. Emotion, vol.12, no.4, pp. 579-599, 1998.
15. J. A. Coan and J. J. B. Allen, Handbook of Emotion Elicitation and Assessment. Oxford, U.K.: Oxford Univ. Press, 2007.
16. A. Mehrabian, "Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in temperament," Current Psychol., vol.14, no.4, pp. 261-292, 1996.
17. J. A. Russell, "A circumplex model of affect," J. Personal. Social Psychol., vol.39, no.6, pp. 1161-1178, 1980.
18. P. Ekman, "Universals and cultural differences in facial expressions of emotion," in Proc. Nebraska Symp. Motivation, J. Cole, Ed., Lincoln, NE: Univ. Nebraska Press, 1972, vol.19, pp. 207-282.
19. R. Plutchik, "The nature of emotions," Amer. Scientist, vol.89, no.4, pp. 344-350, 2001.
20. H. B. Kang, "Affective content detection using HMMs," in Proc. ACM Int. Conf. Multimedia (ACM MM), 2003, pp. 259-262.
21. M. Xu, L.-T. Chia, and J. Jin, "Affective content analysis in comedy and horror videos by audio emotional event detection," in Proc. IEEE Int. Conf. Multimedia & Expo. (ICME), 2005, pp. 622-625.
22. G. Irie, K. Hidaka, T. Satou, T. Yamasaki, and K. Aizawa, "Affective video segment retrieval for consumer generated videos based on correlation between emotions and emotional audio events," in Proc. IEEE Int. Conf. Multimedia & Expo. (ICME), 2009, pp. 525-527.
23. S. Arifin and P. Y. K. Cheung, "Affective level video segmentation by utilizing the pleasure-arousal-dominance information," IEEE Trans. Multimedia, vol.10, no.7, pp. 1325-1341, Nov. 2008.
24. S. Arifin and P. Y. K. Cheung, "A computation method for video segmentation utilizing the pleasure-arousal-dominance emotional information," in Proc. ACM Int. Conf. Multimedia (ACM MM), 2007, pp. 68-77.
25. M. Xu, J. S. Jin, S. Luo, and L. Duan, "Hierarchical movie affective content analysis based on arousal and valence features," in Proc. ACM Int. Conf. Multimedia (ACM MM), 2008, pp. 677-680.
26. K. Sun and J. Q. Yu, "Video affective content representation and recognition using video affective tree and hidden Markov models," in Proc. Int. Conf. Affective Computing and Intelligent Interaction, 2007, pp. 594-605.
27. D. M. Blei, A. Y. Ng, and M. I. Jordan, "Latent dirichlet allocation," J. Mach. Learn. Res., vol.3, pp. 993-1022, 2003.
28. S. Fine, Y. Singer, and N. Tishby, "The hierarchical hidden Markov model: Analysis and applications," Mach. Learn., vol.32, pp. 41-62, 1998.
29. N. Oliver, A. Garg, and E. Horvitz, "Layered representations for learning and inferring office activity from multiple sensory channels," Comput. Vis. Image Understand., vol.96, pp. 163-180, 2004.
30. T. Hofmann, "The cluster-abstraction model: Unsupervised learning of topic hierarchies from text data," in Proc. Int. Joint Conf. Artificial Intelligence (IJCAI), 1999, pp. 682-685.
31. D. M. Blei, T. L. Griffiths, M. L. Jordan, and J. B. Tenenbaum, "Hierarchical topic models and the nested chinese restaurant process," in Proc. Neural Information Processing Systems (NIPS), 2003, pp. 17-24.
32. T. Hofmann, "Probabilistic latent semantic analysis," in Proc. Conf. Uncertainty in Artificial Intelligence (UAI), 1999, pp. 289-296.
33. C. D. Manning, P. Raghavan, and H. Schütze, Introduction to Information Retrieval. Cambridge, U.K.: Cambridge Univ. Press, 2008.
34. Y. Tonomura, A. Akutsu, Y. Taniguchi, and G. Suzuki, "Structured video computing," IEEE Multimedia, vol.1, no.3, pp. 34-43, Autumn/ Fall, 1994.
35. G. Csurka, C. R. Dance, L. Fan, J. Willamowski, and C. Bray, "Visual categorization with bags of keypoints," in Proc. Eur. Conf. Computer Vision (ECCV), 2004, pp. 1-22.
36. D. Talkin, "A Robust Algorithm for Pitch Tracking," in Speech Coding and Synthesis, W. B. Kleijn and J. K. K. Paliwal, Eds. New York: Elsevier, 1995, pp. 495-518.
37. F. Souvannavong, B. Merialdo, and B. Huet, "Latent semantic analysis for an effective region-based video retrieval system," in Proc. Int.Workshop Multimedia Information Retrieval (MIR), 2004, pp. 243-250.
38. A. McCallum and F. Pereira, "Maximum entropy Markov models for information extraction and segmentation," in Proc. Int. Conf. Machine Learning (ICML), 2000, pp. 591-598.
39. D. Gildea and T. Hofmann, "Topic-based language models using EM," in Proc. EUROSPEECH, 1999, pp. 2167-2170.
40. R. Kneser, J. Peters, and D. Klakow, "Language model adaptation using dynamic marginals," in Proc. EUROSPEECH, 1997, pp. 1971-1974.
41. J. J. Gross and R. W. Levenson, "Emotion elicitation using films," Cognit. Emotion, vol.9, no.1, pp. 87-108, 1995.
42. G. Tsoumakas and I. Katakis, "Multi-label classification: An overview," Int. J. Data Warehousing and Mining, vol.3, no.3, pp. 1-13, 2007.
43. A. Aizawa, "An information-theoretic perspective of Tf-Idf measures," Inf. Process. Manage., vol.39, pp. 45-65, 2003.