Continuous Probability Distribution Prediction of Image Emotions via Multitask Shared Sparse Regression

IEEE Transactions on Multimedia

Volumn 19, Issue 3, 2017, Pages 632-645

  Zhao, Sicheng ^a   Yao, Hongxun ^b   Gao, Yue ^a   Ji, Rongrong ^c   Ding, Guiguang ^a  

^a TSINGHUA UNIVERSITY   (China)

^b HARBIN INSTITUTE OF TECHNOLOGY   (China)

^c XIAMEN UNIVERSITY   (China)

Author keywords

Gaussian mixture model; image emotion; multi task learning; probability distribution; shared sparse regression (SSR); valence arousal

Indexed keywords

FORECASTING; GAUSSIAN DISTRIBUTION; IMAGE ANALYSIS; IMAGE CLASSIFICATION; IMAGE SEGMENTATION; ITERATIVE METHODS; MAXIMUM PRINCIPLE; PROBABILITY; REGRESSION ANALYSIS;

GAUSSIAN MIXTURE MODEL; IMAGE EMOTION; MULTITASK LEARNING; SPARSE REGRESSION; VALENCE-AROUSAL;

PROBABILITY DISTRIBUTIONS;

EID: 85013465865     PISSN: 15209210     EISSN: None     Source Type: Journal    
DOI: 10.1109/TMM.2016.2617741     Document Type: Article

References (72)

1. D. Borth, R. Ji, T. Chen, T. Breuel, and S.-F. Chang, "Large-scale visual sentiment ontology and detectors using adjective noun pairs," in Proc. ACM Int. Conf. Multimedia, 2013, pp. 223-232.
2. S. Zhao et al., "Exploring principles-of-art features for image emotion recognition," in Proc. ACM Int. Conf. Multimedia, 2014, pp. 47-56.
3. T. Chen et al., "Object-based visual sentiment concept analysis and application," in Proc. ACM Int. Conf. Multimedia, 2014, pp. 367-376.
4. A. Hanjalic, "Extracting moods from pictures and sounds: Towards truly personalized tv," IEEE Signal Process. Mag., vol. 23, no. 2, pp. 90-100, Mar. 2006.
5. D. Joshi et al., "Aesthetics and emotions in images," IEEE Signal Process. Mag., vol. 28, no. 5, pp. 94-115, Sep. 2011.
6. K.-C. Peng, A. Sadovnik, A. Gallagher, and T. Chen, "A mixed bag of emotions: Model, predict, and transfer emotion distributions," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., Jun. 2015, pp. 860-868.
7. S. Zhao, H. Yao, X. Jiang, and X. Sun, "Predicting discrete probability distribution of image emotions," in Proc. IEEE Int. Conf. Image Process., Sep. 2015, pp. 2459-2463.
8. S. Zhao et al., "Predicting personalized emotion perceptions of social images," in Proc. ACM Int. Conf. Multimedia, 2016, pp. 1385-1394.
9. S. Zhao, H. Yao, and X. Jiang, "Predicting continuous probability distribution of image emotions in valence-arousal space," in Proc. ACM Int. Conf. Multimedia, 2015, pp. 879-882.
10. W.-N. Wang, Y.-L. Yu, and S.-M. Jiang, "Image retrieval by emotional semantics: A study of emotional space and feature extraction," in Proc. IEEE Int. Conf. Syst., Man Cybern., Oct. 2006, pp. 3534-3539.
11. J. Machajdik and A. Hanbury, "Affective image classification using features inspired by psychology and art theory," in Proc. ACM Int. Conf. Multimedia, 2010, pp. 83-92.
12. V. Yanulevskaya et al., "Emotional valence categorization using holistic image features," in Proc. IEEE Int. Conf. Image Process., Oct. 2008, pp. 101-104.
13. X. Lu et al., "On shape and the computability of emotions," in Proc. ACM Int. Conf. Multimedia, 2012, pp. 229-238.
14. B. Li, W. Xiong, W. Hu, and X. Ding, "Context-aware affective images classification based on bilayer sparse representation," in Proc. ACM Int. Conf. Multimedia, 2012, pp. 721-724.
15. J. Tang et al., "Quantitative study of individual emotional states in social networks," IEEE Trans. Affect. Comput., vol. 3, no. 2, pp. 132-144, Apr.- Jun. 2012.
16. J. Jia et al., "Canwe understand van Gogh'smood? learning to infer affects from images in social networks," in Proc. ACM Int. Conf. Multimedia, 2012, pp. 857-860.
17. Y. Yang et al., "How do your friends on social media disclose your emotions?," in Proc. AAAI Conf. Artif. Intell., 2014, pp. 306-312.
18. H. Schlosberg, "Three dimensions of emotion." Psychological Rev., vol. 61, no. 2, pp. 81-88, 1954.
19. S. Benini, L. Canini, and R. Leonardi, "A connotative space for supporting movie affective recommendation," IEEE Trans. Multimedia, vol. 13, no. 6, pp. 1356-1370, Dec. 2011.
20. A. Hanjalic and L.-Q. Xu, "Affective video content representation and modeling," IEEE Trans. Multimedia, vol. 7, no. 1, pp. 143-154, Feb. 2005.
21. A. B. Warriner, V. Kuperman, and M. Brysbaert, "Norms of valence, arousal, and dominance for 13,915 english lemmas," Behavior Res. Methods, vol. 45, no. 4, pp. 1191-1207, 2013.
22. S. Siersdorfer, E. Minack, F. Deng, and J. Hare, "Analyzing and predicting sentiment of images on the social web," in Proc. ACM Int. Conf. Multimedia, 2010, pp. 715-718.
23. J. Yuan, S. Mcdonough, Q. You, and J. Luo, "Sentribute: image sentiment analysis from a mid-level perspective," in Proc. ACMInt. Workshop Issues Sentiment Discovery Opinion Mining, 2013, Art. no. 10.
24. Y. Yang, P. Cui, W. Zhu, and S. Yang, "User interest and social influence based emotion prediction for individuals," in Proc. ACM Int. Conf. Multimedia, 2013, pp. 785-788.
25. Y. Yang et al., "Emotionally representative image discovery for social events," in Proc. ACM Int. Conf. Multimedia Retrieval, 2014, Art. no. 177.
26. S. Zhao, H. Yao, Y. Yang, and Y. Zhang, "Affective image retrieval via multi-graph learning," in Proc. ACM Int. Conf. Multimedia, 2014, pp. 1025-1028.
27. Y.-L. Boureau, F. Bach, Y. LeCun, and J. Ponce, "Learning mid-level features for recognition," in Proc. IEEEConf. Comput. Vis. Pattern Recog., Jun. 2010, pp. 2559-2566.
28. S. Zheng, A. Yuille, and Z. Tu, "Detecting object boundaries using low-, mid-, and high-level information," Comput. Vis. Image Understand., vol. 114, no. 10, pp. 1055-1067, 2010.
29. Y. Guo, G. Ding, X. Jin, and J. Wang, "Learning predictable and discriminative attributes for visual recognition." in Proc. AAAI Conf. Artif. Intell., 2015, pp. 3783-3789.
30. P. Yang, Q. Liu, and D. N. Metaxas, "Exploring facial expressions with compositional features," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., Jun. 2010, pp. 2638-2644.
31. S. Zhao et al., "Video indexing and recommendation based on affective analysis of viewers," in Proc. ACM Int. Conf. Multimedia, 2011, pp. 1473-1476.
32. G. Sandbach, S. Zafeiriou, M. Pantic, and L. Yin, "Static and dynamic 3D facial expression recognition: A comprehensive survey," Image Vis. Comput., vol. 30, no. 10, pp. 683-697, 2012.
33. C.-H. Wu, W.-L. Wei, J.-C. Lin, and W.-Y. Lee, "Speaking effect removal on emotion recognition from facial expressions based on eigenface conversion," IEEE Trans. Multimedia, vol. 15, no. 8, pp. 1732-1744, Dec. 2013.
34. S. Zhao, H. Yao, and X. Sun, "Video classification and recommendation based on affective analysis of viewers," Neurocomputing, vol. 119, pp. 101-110, 2013.
35. A. Kleinsmith and N. Bianchi-Berthouze, "Affective body expression perception and recognition: A survey," IEEE Trans. Affect. Comput., vol. 4, no. 1, pp. 15-33, Jan.-Mar. 2013.
36. S. Happy and A. Routray, "Automatic facial expression recognition using features of salient facial patches," IEEE Trans. Affect. Comput., vol. 6, no. 1, pp. 1-12, Jan-Mar. 2015.
37. M. Tkalcic, A. Odic, A. Kosir, and J. Tasic, "Affective labeling in a contentbased recommender system for images," IEEE Trans. Multimedia, vol. 15, no. 2, pp. 391-400, Feb. 2013.
38. B. Pang and L. Lee, "Opinion mining and sentiment analysis," Inf. Retrieval, vol. 2, no. 1/2, pp. 1-135, 2008.
39. S. G. Koolagudi and K. S. Rao, "Emotion recognition from speech: A review," Int. J. Speech Technol., vol. 15, no. 2, pp. 99-117, 2012.
40. Q. Mao, M. Dong, Z. Huang, and Y. Zhan, "Learning salient features for speech emotion recognition using convolutional neural networks," IEEE Trans. Multimedia, vol. 16, no. 8, pp. 2203-2213, Dec. 2014.
41. Y.-H. Yang and H. H. Chen, "Machine recognition of music emotion: A review," ACM Trans. Intell. Syst. Technol., vol. 3, no. 3, 2012, Art. no. 40.
42. Y.-H. Yang and J.-Y. Liu, "Quantitative study of music listening behavior in a social and affective context," IEEE Trans. Multimedia, vol. 15, no. 6, pp. 1304-1315, Oct. 2013.
43. P.-C. Chen, K.-S. Lin, and H. H. Chen, "Emotional accompaniment generation system based on harmonic progression," IEEE Trans. Multimedia, vol. 15, no. 7, pp. 1469-1479, Nov. 2013.
44. S. Zhao, H. Yao, F. Wang, X. Jiang, and W. Zhang, "Emotion based image musicalization," in Proc. IEEE Int. Conf. Multimedia Expo Workshops, Jul. 2014, pp. 1-6.
45. S. Zhao, H. Yao, X. Sun, X. Jiang, and P. Xu, "Flexible presentation of videos based on affective content analysis," in Proc. Int. Conf. Multimedia Model., 2013, pp. 368-379.
46. K. Yadati, H. Katti, and M. Kankanhalli, "Cavva: Computational affective video-in-video advertising," IEEE Trans. Multimedia, vol. 16, no. 1, pp. 15-23, Jan. 2014.
47. M. Soleymani, M. Larson, T. Pun, and A. Hanjalic, "Corpus development for affective video indexing," IEEE Trans. Multimedia, vol. 16, no. 4, pp. 1075-1089, Jun. 2014.
48. S. E. Shepstone, Z.-H. Tan, and S. H. Jensen, "Using audio-derived affective offset to enhance tv recommendation," IEEE Trans. Multimedia, vol. 16, no. 7, pp. 1999-2010, Nov. 2014.
49. S. Wang and Q. Ji, "Video affective content analysis: A survey of state of the art methods," IEEE Trans. Affect. Comput., vol. 6, no. 4, pp. 410-430, Oct.-Dec. 2015.
50. M. Carney, P. Cunningham, J. Dowling, and C. Lee, "Predicting probability distributions for surf height using an ensemble of mixture density networks," in Proc. ACM Int. Conf. Mach. Learn., 2005, pp. 113-120.
51. H. Liu, Z. Hu, D. Zhou, and H. Tian, "Cumulative probability distribution model for evaluating user behavior prediction algorithms," in Proc. IEEE Int. Conf. Social Comput., Sep. 2013, pp. 385-390.
52. G. Pipa, S. Grün, and C. van Vreeswijk, "Impact of spike train autostructure on probability distribution of joint spike events," Neural Comput., vol. 25, no. 5, pp. 1123-1163, 2013.
53. J. Wright, A. Y. Yang, A. Ganesh, S. S. Sastry, and Y. Ma, "Robust face recognition via sparse representation," IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 2, pp. 210-227, Feb. 2009.
54. X.-T. Yuan, X. Liu, and S. Yan, "Visual classification with multitask joint sparse representation," IEEE Trans. Image Process., vol. 21, no. 10, pp. 4349-4360, Oct. 2012.
55. J. Zhou, J. Chen, and J. Ye, "MALSAR: Multi-task learning via structural regularization," Tempe, AZ, USA: Arizona State University, 2012.
56. S. Bickel, J. Bogojeska, T. Lengauer, and T. Scheffer, "Multi-task learning for HIV therapy screening," in Proc. ACM Int. Conf. Mach. Learn., 2008, pp. 56-63.
57. A. Argyriou, T. Evgeniou, and M. Pontil, "Multi-task feature learning," Adv. Neural Inf. Process. Syst., vol. 19, 2007, Art. no. 41.
58. P. Gong, J. Ye, and C. Zhang, "Robust multi-task feature learning," in Proc. ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, 2012, pp. 895-903.
59. R. K. Ando and T. Zhang, "A framework for learning predictive structures from multiple tasks and unlabeled data," J. Mach. Learn. Res., vol. 6, pp. 1817-1853, 2005.
60. H. Wang et al., "Sparse multi-task regression and feature selection to identify brain imaging predictors for memory performance," in Proc. IEEE Int. Conf. Comput. Vis., Nov. 2011, pp. 557-562.
61. K. R. Scherer, "What are emotions? and how can they be measured?" Social Sci. Inf., vol. 44, no. 4, pp. 695-729, 2005.
62. J. A. Mikels et al., "Emotional category data on images from the international affective picture system," Behavior Res. Methods, vol. 37, no. 4, pp. 626-630, 2005.
63. J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba, "SUN database: Large-scale scene recognition from abbey to zoo," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., Jun. 2010, pp. 3485-3492.
64. G. Patterson and J. Hays, "SUN attribute database: Discovering, annotating, and recognizing scene attributes," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., Jun. 2012, pp. 2751-2758.
65. F. X. Yu, L. Cao, R. S. Feris, J. R. Smith, and S.-F. Chang, "Designing category-level attributes for discriminative visual recognition," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., Jun. 2013, pp. 771-778.
66. H. Zhang, Y. Yang, H. Luan, S. Yang, and T.-S. Chua, "Start from scratch: Towards automatically identifying, modeling, and naming visual attributes," in Proc. ACM Int. Conf. Multimedia, 2014, pp. 187-196.
67. P. Lucey et al., "The extended Cohn-Kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression," in Proc. IEEE Conf. Comput. Vis. Pattern Recog. Workshops, Jun. 2010, pp. 94-101.
68. P. Viola and M. J. Jones, "Robust real-time face detection," Int. J. Comput. Vis., vol. 57, no. 2, pp. 137-154, 2004.
69. R. Chartrand and W. Yin, "Iteratively reweighted algorithms for compressive sensing," in Proc. IEEE Int. Conf. Acoust, Speech Signal Process., Mar.-Apr. 2008, pp. 3869-3872.
70. C. Chen, J. Huang, L. He, and H. Li, "Preconditioning for accelerated iteratively reweighted least squares in structured sparsity reconstruction," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., Jun. 2014, pp. 2713-2720.
71. Y. Gao, S. Zhao, Y. Yang, and T.-S. Chua, "Multimedia social event detection in microblog," in Proc. Int. Conf. Multimedia Model., 2015, pp. 269-281.
72. G. Ding, Y. Guo, and J. Zhou, "Collective matrix factorization hashing for multimodal data," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., Jun. 2014, pp. 2075-2082.