Data-driven affective filtering for images and videos

IEEE Transactions on Cybernetics

Volumn 45, Issue 10, 2015, Pages 2336-2349

  Li, Teng ^a   Ni, Bingbing ^b   Xu, Mengdi ^c   Wang, Meng ^d   Gao, Qingwei ^a   Yan, Shuicheng ^e  

^a ANHUI UNIVERSITY   (China)

^b ADVANCED DIGITAL SCIENCES CENTER   (Singapore)

^c INSTITUTE FOR INFOCOMM RESEARCH   (Singapore)

^d HEFEI UNIVERSITY OF TECHNOLOGY   (China)

^e NATIONAL UNIVERSITY OF SINGAPORE   (Singapore)

Author keywords

Affective filtering; data driven; Gaussian mixture model (GMM); linear piece wise transformation

Indexed keywords

LINEAR TRANSFORMATIONS; MATHEMATICAL TRANSFORMATIONS; PIECEWISE LINEAR TECHNIQUES;

EMPIRICAL KNOWLEDGE; FEATURE DISTRIBUTION; FILTERING FRAMEWORK; GAUSSIAN MIXTURE MODEL; OBJECTIVE FUNCTIONS; PIECEWISE-LINEAR TRANSFORMATION; REGULARIZATION TERMS; STATISTICAL MODELING;

EDGE DETECTION;

EID: 84960128021     PISSN: 21682267     EISSN: None     Source Type: Journal    
DOI: 10.1109/TCYB.2015.2392156     Document Type: Article

References (72)

1. A. Rooij, J. Broekens, and M. H. Lamers, "Abstract expressions of affect. Int. J. Syn. Emot, vol. 4, no. 1, pp. 1-31, 2013.
2. Amazon. Mechanic Turk. [Online]. Available: https://www.mturk.com/mturk/welcome
3. S. Arifin, and P. Y. K. Cheung, "A computation method for video segmentation utilizing the pleasure-Arousal-dominance emotional information. in Proc. ACM Int. Conf. Multimedia (MM), Augsburg, Germany. 2007, pp. 68-77.
4. T. Balomenos, et al., "Emotion analysis in man-machine interaction systems. in Machine Learning for Multimodal Interaction. Berlin, Germany: Springer. 2004, pp. 318-328.
5. N. Bianchi-Berthouze, "K-DIME: An affective image filtering system. IEEE Trans. Multimedia, vol. 10, no. 3, pp. 103-106, Jul./Sep. 2003.
6. C. M. Bishop, et al., Pattern Recognition, and Machine Learning, vol. 1. New York, NY, USA: Springer, 2006.
7. S. Boyd, and L. Vandenberghe, Convex Optimization. Cambridge, U.K: Cambridge Univ. Press, 2004.
8. M. Bradley, Emotional Memory: A Dimensional Analysis. Hillsdale, NJ, USA: Lawrence Erlbaum, 1994.
9. Y. Chang, S. Saito, and M. Nakajima, "Example-based color transformation of image, and video using basic color categories. IEEE Trans. Image Process, vol. 16, no. 2, pp. 329-336, Feb. 2007.
10. T. Chen, and K.-H. Yap, "Discriminative BoW framework for mobile landmark recognition. IEEE Trans. Cybern, vol. 44, no. 5, pp. 695-706, May 2014.
11. B. Cheng, B. Ni, S. Yan, and Q. Tian, "Learning to photograph. in Proc. ACM Int. Conf. Multimedia (MM), Florence, Italy. 2010, pp. 291-300.
12. S. B. Cho, "Emotional image, and musical information retrieval with interactive genetic algorithm. Proc. IEEE, vol. 92, no. 4, pp. 702-711, Apr. 2004.
13. T. S. Chua, et al., "NUS-WIDE: A real-world web image database from National University of Singapore. in Proc. ACM Int. Conf. Image Video Retrieval (CIVR), Santorini, Greece. 2009, pp. 1-9.
14. C. Colombo, A. D. Bimbo, and P. Pala, "Semantics in visual information retrieval. IEEE Trans. Multimedia, vol. 6, no. 3, pp. 38-53, Jul./Sep. 1999.
15. N. Dalal, and B. Triggs, "Histograms of oriented gradients for human detection. in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), San Diego, CA, USA. 2005, pp. 886-893.
16. R. D'Andrade, and M. Egan, "The colors of emotion. Amer. Ethnol, vol. 1, no. 1, pp. 49-63, 1974.
17. L. Eisemann, Pantone Guide to Communicating With Color. Florida, FL, USA: Grafix Press, 2000.
18. P. Ekman, "Facial expression, and emotion. Amer. Psychol, vol. 48, no. 4, pp. 384-392, 1993.
19. P. Ekman, et al., "Universals, and cultural differences in the judgments of facial expressions of emotion. J. Pers. Soc. Psychol, vol. 53, no. 4, pp. 712-717, 1987.
20. P. Felzenszwalb, and D. Huttenlocher, "Efficient graph-based image segmentation. Int. J. Comput. Vis. (IJCV), vol. 59, no. 2, pp. 167-181, 2004.
21. Flickr. Online Sharing Photos. [Online]. Available: http://www.flickr.com/
22. B. Frey, Graphical Models for Machine Learning, and Digital Communication. Cambridge, MA, USA: MIT Press, 1998.
23. K. Fukunaga, Introduction to Statistical Pattern Recognition. San Diego, CA, USA: Academic Press, 1990.
24. M.-J. Han, C.-H. Lin, and K.-T. Song, "Robotic emotional expression generation based on mood transition, and personality model. IEEE Trans. Cybern, vol. 43, no. 4, pp. 1290-1303, Aug. 2013.
25. A. Hanjalic, "Extracting moods from pictures, and sounds: Towards truly personalized TV. IEEE Signal Process. Mag, vol. 23, no. 2, pp. 90-100, Mar. 2006.
26. A. Hanjalic, and L.-Q. Xu, "Affective video content representation, and modeling. IEEE Trans. Multimedia, vol. 7, no. 1, pp. 143-154, Feb. 2005.
27. J. A. Hartigan, and M. A. Wong, "Algorithm as 136: A k-means clustering algorithm. J. Roy. Stat. Soc. C, vol. 28, no. 1, pp. 100-108, 1979.
28. L. He, H. Qi, and R. Zaretzki, "Emotion transfer for images based on color combinations. Comput. Graph., arXiv preprint arXiv:1307.3581, 2013.
29. R. Hong, et al., "Movie2Comics: A feast of multimedia artwork. in Proc. ACM Int. Conf. Multimedia (MM), Florence, Italy. 2010, pp. 611-614.
30. J. Itten, The Art of Color: The Subjective Experience, and Objective Rationale of Color. New York, NY, USA: Wiley, 1997.
31. A. Johnson, O. Johnson, and M. Baksh, "The colors of emotions in Machiguenga. Amer. Anthropol, vol. 88, no. 3, pp. 674-681, 1986.
32. H. Kang, "Affective content detection using HMMs. in Proc. ACM Int. Conf. Multimedia (MM), Berkeley, CA, USA. 2003, pp. 259-262.
33. S. Kobayashi, Color Image Scale. Tokyo, Japan: Kodansha Int., 1992.
34. P. Lang, Perspectives on Anger, and Emotion. Hillsdale, NJ, USA: Lawrence Erlbaum. 1993, pp. 109-134.
35. T. Li, T. Mei, I.-S. Kweon, and X.-S. Hua, "Contextual bag-of-words for visual categorization. IEEE Trans. Circuits Syst. Video Technol, vol. 21, no. 4, pp. 381-392, Apr. 2011.
36. Y. Lin, et al., "Large-scale image classification: Fast feature extraction, and SVM training. in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Providence, RI, USA. 2011, pp. 1689-1696.
37. J. Machajdik, and A. Hanbury, "Affective image classification using features inspired by psychology, and art theory. in Proc. ACM Int. Conf. Multimedia (MM), Florence, Italy. 2010, pp. 83-92.
38. J. Mas, and G. Fernandez, "Video shot boundary detection based on color histogram. Notebook Papers TRECVID, NIST, Gaithersburg, MD, USA, 2003.
39. G. Mckeown, M. F. Valstar, R. Cowie, M. Pantic, and M. Schroeder, "The SEMAINE database: Annotated multimodal records of emotionally coloured conversations between a person, and a limited agent. IEEE Trans. Affect. Comput, vol. 3, no. 1, pp. 5-17, Jan./Mar. 2012.
40. G. McLachlan, Mixture Models. New York, NY, USA: Marcel Dekker, 1988.
41. J. Mehrabian, "Pleasure-Arousal-dominance: A general framework for describing, and measuring individual differences in temperament. Current Psychol, vol. 14, no. 4, pp. 261-292, 1996.
42. J. A. Mikels, et al., "Emotional category data on images from the international affective picture system. Behav. Res. Methods, vol. 37, no. 4, pp. 626-630, 2005.
43. B. Ni, Z. Song, and S. Yan, "Web image mining towards universal age estimator. in Proc. ACM Int. Conf. Multimedia (MM), Beijing, China. 2009, pp. 85-94.
44. A. Ortony, and T. J. Turner, "What?s basic about basic emotions?. Psychol. Rev, vol. 97, no. 3, pp. 315-331, 1990.
45. L.-C. Ou, M. R. Luo, A. Woodcock, and A. Wright, "A study of colour emotion, and colour preference. Color Res. Appl, vol. 29, no. 4, pp. 292-298, 2004.
46. M. Pantic, and L. J. M. Rothkrantz, "Toward an affect-sensitive multimodal human-computer interaction. Proc. IEEE, vol. 91, no. 9, pp. 1370-1390, Sep. 2003.
47. H. Permuter, J. Francos, and I. Jermyn, "A study of Gaussian mixture models of color, and texture features for image classification, and segmentation. Pattern Recognit, vol. 39, no. 4, pp. 695-706, 2006.
48. R. Picard, Affective Computing. Cambridge, MA, USA: MIT Press, 1997.
49. Z. Rasheed, Y. Sheikh, and M. Shah, "On the use of computable features for film classification. IEEE Trans. Circuits Syst. Video Technol, vol. 15, no. 1, pp. 52-64, Jan. 2005.
50. R. A. Redner, and H. F. Walker, "Mixture densities, maximum likelihood, and the EM algorithm. SIAM Rev, vol. 26, no. 2, pp. 195-239, 1984.
51. E. Reinhard, M. Ashikhmin, B. Gooch, and P. Shirley, "Color transfer between images. IEEE Comput. Graph. Appl, vol. 21, no. 5, pp. 34-41, Sep./Oct. 2001.
52. H. Schlosberg, "Three dimensions of emotion. Psychol. Rev, vol. 61, no. 2, pp. 81-88, 1954.
53. P. Shaver, J. Schwartz, D. Kirson, and C. O'Connor, Emotions in Social Psychology: Key Readings in Social Psychology. Florence, KY, USA: Psychology Press. 2001, pp. 26-56.
54. G. Smith, Film Structure, and the Emotion System. Cambridge, U.K: Cambridge Univ. Press, 2003.
55. M. Stricker, and M. Orengo, "Similarity of color images. Proc. SPIE, vol. 2420, pp. 381-392, Mar. 1995.
56. S. Tanaka, Y. Iwadate, and S. Inokuchi, "An attractiveness evaluation model based on the physical features of image regions. in Proc. Int. Conf. Pattern Recognit. (ICPR), vol. 2. Barcelona, Spain. 2000, pp. 793-796.
57. W. B. Thompson, P. Shirley, and J. A. Ferwerda, "A spatial postprocessing algorithm for images of night scenes. J. Graph. Tools, vol. 7, no. 1, pp. 1-12, 2002.
58. H. L. Wang, and L. F. Cheong, "Affective understanding in film. IEEE Trans. Circuits Syst. Video Technol, vol. 16, no. 6, pp. 689-704, Jun. 2006.
59. J. Wang, et al., "Locality-constrained linear coding for image classification. in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), San Francisco, CA, USA. 2010, pp. 3360-3367.
60. M. Wang, et al., "Facilitating image search with a scalable, and compact semantic mapping. IEEE Trans. Cybern., 2014, to be published
61. W. N. Wang, Y. L. Yu, and S. M. Jiang, "Image retrieval by emotional semantics: A study of emotional space, and feature extraction. in Proc. IEEE Int. Conf. Syst. Man Cybern., Taipei, Taiwan. 2006, pp. 3534-3539.
62. C. Wei, N. Dimitrova, and S. Chang, "Color-mood analysis of films based on syntactic, and psychological models. in Proc. Int. Conf. Multimedia Expo. (ICME), vol. 2. Taipei, Taiwan. 2004, pp. 831-834.
63. B. Whelan, Color Harmony 2. Gloucester, MA, USA: Rockport, 1994.
64. Wikipedia. Mixture Model. [Online]. Available: http://en.wikipedia.org/wiki/Mixture-model
65. Q. Wu, C. Zhou, and C. Wang, "Content-based affective image classification, and retrieval using support vector machines. in Affect Computing, and Intelligent Interaction. Berlin, Germany: Springer, 2005.
66. M. Xu, B. Ni, J. Tang, and S. Yan, "Image re-emotionalizing. in Proc. Pac.-Rim Conf. Multimedia (PCM), Sydney, NSW, Australia. 2011, pp. 3-14.
67. M. Xu, et al., "Hierarchical affective content analysis in arousal, and valence dimensions. Signal Process, vol. 93, no. 8, pp. 2140-2150, 2013.
68. C.-K. Yang, and L.-K. Peng, "Automatic mood-Transferring between color images. IEEE Comput. Graph. Appl, vol. 28, no. 2, pp. 52-61, Mar./Apr. 2008.
69. J. Yang, Y. G. Jiang, A. G. Hauptmann, and C.-W. Ngo, "Evaluating bag-of-visual-words representations in scene classification. in Proc. Int. Workshop Multimedia Inf. Retrieval, Augsburg, Germany. 2007, pp. 197-206.
70. V. Yanulevskaya, et al., "Emotional valence categorization using holistic image features. in Proc. IEEE Int. Conf. Image Process. (ICIP), San Diego, CA, USA. 2008, pp. 101-104.
71. Youtube. Online Sharing Video. [Online]. Available: http://www.youtube.com/
72. H. Zettle, Sight Sound Motion: Applied Media Aesthetics. Belmont, CA, USA: Wadsworth, 1999.