Modeling the affective content of music with a Gaussian mixture model

IEEE Transactions on Affective Computing

Volumn 6, Issue 1, 2015, Pages 56-68

  Wang, Ju Chiang ^a   Yang, Yi Hsuan ^b   Wang, Hsin Min ^a   Jeng, Shyh Kang ^c  

^a INSTITUTE OF INFORMATION SCIENCE   (Taiwan)

^b RESEARCH CENTER FOR INFORMATION TECHNOLOGY INNOVATION   (Taiwan)

^c NATIONAL TAIWAN UNIVERSITY   (Taiwan)

Author keywords

Arousal; Gaussian mixture model; Music emotion recognition; Music information retrieval; Subjectivity; Valence

Indexed keywords

ASSOCIATION REACTIONS; AUDIO ACOUSTICS; BEHAVIORAL RESEARCH; GAUSSIAN DISTRIBUTION; HUMAN COMPUTER INTERACTION; INFORMATION RETRIEVAL; LEARNING ALGORITHMS; PROBABILITY DISTRIBUTIONS;

AROUSAL; GAUSSIAN MIXTURE MODEL; MUSIC EMOTIONS; MUSIC INFORMATION RETRIEVAL; SUBJECTIVITY; VALENCE;

COMPUTER MUSIC;

EID: 84924020940     PISSN: 19493045     EISSN: None     Source Type: Journal    
DOI: 10.1109/TAFFC.2015.2397457     Document Type: Article

References (91)

1. A. J. Lonsdale and A. C. North, "Why do we listen to music? a uses and gratifications analysis," Brit. J. Psychol., vol. 102, pp. 108-134, 2011.
2. M. Zentner, D. Grandjean, and K. R. Scherer, "Emotions evoked by the sound of music: Characterization, classification, and measurement,"Emotion, vol. 8, no. 4, pp. 494-521, 2008.
3. P. Juslin and P. Laukka, "Expression, perception, and induction of musical emotions: A review and a questionnaire study of everyday listening," J. New Music Res., vol. 33, no. 3, pp. 217-238, 2004.
4. J. H. Lee and J. S. Downie, "Survey of music information needs, uses, and seeking behaviours: Preliminary findings," in Proc. Int. Soc. Music Inf. Retrieval Conf., 2004, pp. 441-446.
5. M. Kamalzadeh, D. Baur, and T. Möller, "A survey on music listening and management behaviours," in Proc. Int. Soc. Music Inf. Retrieval Conf., 2012, pp. 373-378.
6. Y.-H. Yang and H. H. Chen, Music Emotion Recognition. Boca Raton, FL, USA: CRC Press, 2011.
7. H. Katayose, M. Imai, and S. Inokuchi, "Sentiment extraction in music," in Proc. Int. Conf. Pattern Recog., 1998, pp. 1083-1087.
8. E. Schubert, "Modeling perceived emotion with continuous musical features," Music Perception, vol. 21, no. 4, pp. 561-585, 2004.
9. M. Leman, V. Vermeulen, L. D. Voogdt, D. Moelants, and M. Lesaffre, "Prediction of musical affect using a combination of acoustic structural cues," J. New Music Res., vol. 34, no. 1, pp. 39-67, 2005.
10. A. Hanjalic and L. Xu, "Affective video content representation and modeling," IEEE Trans. Multimedia, vol. 7, no. 1, pp. 143-154, 2005.
11. L. Lu, D. Liu, and H. Zhang, "Automatic mood detection and tracking of music audio signals," IEEE Trans. Audio, Speech Lang. Process., vol. 14, no. 1, pp. 5-18, Jan. 2006.
12. X. Hu, J. S. Downie, C. Laurier, M. Bay, and A. F. Ehmann, "The 2007 MIREX audio mood classification task: Lessons learned,"in Proc. Int. Soc. Music Inf. Retrieval Conf., 2008, pp. 462-467.
13. Y.-H. Yang, C. C. Liu, and H. H. Chen, "Music emotion classification: A fuzzy approach," in Proc. ACM Multimedia, 2006, pp. 81-84.
14. C.-C. Yeh, S.-S. Tseng, P.-C. Tsai, and J.-F. Weng, "Building a personalized music emotion prediction system," in Prof. Pacific Rim Conf. Multimedia, 2006, pp. 730-739.
15. K. F. MacDorman, S. Ough, and C.-C. Ho, "Automatic emotion prediction of song excerpts: Index construction, algorithm design, and empirical comparison," J. New Music Res., vol. 36, no. 4, pp. 281-299, 2007.
16. Y.-H. Yang, Y.-C. Lin, Y.-F. Su, and H. H. Chen, "A regression approach to music emotion recognition," IEEE Trans. Audio, Speech Lang. Process., vol. 16, no. 2, pp. 448-457, Feb. 2008.
17. X. Hu and J. S. Downie, "When lyrics outperform audio for music mood classification: A feature analysis," in Proc. Int. Soc. Music Inf. Retrieval Conf., 2010, pp. 619-624.
18. B. Schuller, C. Hage, D. Schuller, and G. Rigoll, "'Mister D.J., Cheer Me Up!': Musical and textual features for automatic mood classification," J. New Music Res., vol. 39, no. 1, pp. 13-34, 2010.
19. T. Eerola, O. Lartillot, and P. Toiviainen, "Prediction of multidimensional emotional ratings in music from audio using multivariate regression models," in Proc. Int. Soc. Music Inf. Retrieval Conf., 2009, pp. 621-626.
20. Y.-H. Yang and H. H. Chen, "Ranking-based emotion recognition for music organization and retrieval," IEEE Trans. Audio, Speech Lang. Process., vol. 19, no. 4, pp. 762-774, May 2011.
21. Y.-H. Yang and H. H. Chen, "Prediction of the distribution of perceived music emotions using discrete samples," IEEE Trans. Audio, Speech Lang. Process., vol. 19, no. 7, pp. 2184-2196, Sep. 2011.
22. Y.-H. Yang, Y.-F. Su, Y.-C. Lin, and H. H. Chen, "Music emotion recognition: The role of individuality," in Proc. ACM Int. Workshop Human-Centered Multimedia, 2007, pp. 13-21.
23. Y.-H. Yang, Y.-C. Lin, and H. H. Chen, "Personalized music emotion recognition," in Proc. ACM SIG Inf. Retrieval, 2009, pp. 748-749.
24. E. M. Schmidt and Y. E. Kim, "Modeling musical emotion dynamics with conditional random fields," in Proc. Int. Soc. Music Inf. Retrieval Conf., 2011, pp. 777-782.
25. E. M. Schmidt and Y. E. Kim, "Prediction of time-varying musical mood distributions from audio," in Proc. Int. Soc. Music Inf. Retrieval Conf., 2010, pp. 465-470.
26. Y. E. Kim, E. M. Schmidt, R. Migneco, B. G. Morton, P. Richardson, J. J. Scott, J. A. Speck, and D. Turnbull, "Music emotion recognition: A state of the art review," in Proc. Int. Soc. Music Inf. Retrieval Conf., 2010, pp. 255-266.
27. Y.-H. Yang and H.-H. Chen, "Machine recognition of music emotion: A review," ACM Trans. Intell. Syst. Technol., vol. 3, no. 4, p. 40, 2012.
28. M. Barthet, G. Fazekas, and M. Sandler, "Multidisciplinary perspectives on music emotion recognition: Implications for content and context-based models," in Proc. Int. Symp. Comput. Music Model. Retrieval, 2012, pp. 492-507.
29. J.-C. Wang, Y.-H. Yang, H.-M. Wang, and S.-K. Jeng, "The acoustic emotion Gaussians model for emotion-based music annotation and retrieval," in Proc. ACM Multimedia, 2012, pp. 89-98.
30. T.-L. Wu and S.-K. Jeng, "Probabilistic estimation of a novel music emotion model," in Proc. Int. Multimedia Model. Conf., 2008, pp. 487-497.
31. A. C. Mostafavi, Z. W. Raś, and A. Wieczorkowska, "Developing personalized classifiers for retrieving music by mood," in Proc. Int. Workshop New Frontiers Mining Complex Patterns, 2013.
32. P. Saari, T. Eerola, G. Fazekasy, M. Barthet, O. Lartillot, and M. Sandler, "The role of audio and tags in music mood prediction: A study using semantic layer projection," in Proc. 14th Int. Soc. Music Inf. Retrieval Conf., 2013, pp. 201-206.
33. R. Panda, R. Malheiro, B. Rocha, A. Oliveira, and R. P. Paiva, "Multi-modal music emotion recognition: A new dataset, methodology and comparative analysis," in Proc. Int. Symp. Comput. Music Model. Retrieval, 2013, pp. 1-13.
34. V. Imbrasaité, T. Baltrušaitis, and P. Robinson, "Emotion tracking in music using continuous conditional random fields and baseline feature representation," in Proc. Int. Workshop Affective Anal. Multimeda, 2013, pp. 1-6.
35. M. Soleymani, M. N. Caro, E. Schmidt, C.-Y. Sha, and Y.-H. Yang, "1000 songs for emotional analysis of music," in Proc. Int. Workshop Crowdsourcing Multimedia, 2013, pp. 1-6.
36. S. J. Breckler, R. B. Allen, and V. J. Konečni, "Mood-optimizing strategies in aesthetic-choice behavior," Music Perception, vol. 2, no. 4, pp. 459-470, 1985.
37. N. Kummer, D. Kadish, A. Dulic, and H. Najjaran, "The empathy machine," in Proc. IEEE Int. Conf. Syst., Man, Cybern., 2012, pp. 2265-2271.
38. Y.-H. Yang and J.-Y. Liu, "Quantitative study of music listening behavior in a social and affective context," IEEE Trans. Multimedia, vol. 15, no. 6, pp. 1304-1315, 2013.
39. R. W. Picard, Affective Computing. Cambridge, MA, USA: MIT Press, 1997.
40. R. A. Calvo and S. D'Mello, "Affect detection: An interdisciplinary review of models, methods, and their applications," IEEE Trans. Affective Comput., vol. 1, no. 1, pp. 18-37, Jan. 2010.
41. S. Koelstra, C. Mühl, M. Soleymani, J.-S. Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt, and I. Patras, "DEAP: A database for emotion analysis; using physiological signals," IEEE Trans. Affective Comput., vol. 3, no. 1, pp. 18-31, Jan.-Mar. 2012.
42. M. Soleymani, J. Lichtenauer, T. Pun, and M. Pantic, "A multimodal database for affect recognition and implicit tagging," IEEE Trans. Affective Comput., vol. 3, no. 1, pp. 42-55, Jan.-Mar. 2012.
43. I. Siegert, R. Böck, and A. Wendemuth, "Inter-rater reliability for emotion annotation in human-computer interaction: Comparison and methodological improvements," J. Multimodal User Interfaces, vol. 8, no. 1, pp. 17-28, 2013.
44. H. Gunesa and B. Schuller, "Categorical and dimensional affect analysis in continuous input: Current trends and future directions," Image Vis. Comput., vol. 31, no. 2, pp. 120-136, 2013.
45. C. L. Pelletier, "The effect of music on decreasing arousal due to stress:Ameta-analysis," J. Music Therapy, vol. 41, pp. 192-214, 2004.
46. P. Laukka, "Uses of music and psychological well-being among the elderly," J. Happiness Stud., vol. 8, no. 2, pp. 215-241, 2007.
47. I. Wallis, T. Ingalls, and E. Campana, "Computer-generating emotional music: The design of an affective music algorithm," in Proc. Int. Conf. Digital Audio Effects, 2008, pp. 1-6.
48. M. Zentner and K. R. Scherer, "Emotional effects of music: production rules," in Music and Emotion: Theory and Research, P. N. Juslin and J. A. Sloboda, Eds. New York, NY, USA: Oxford Univ. Press, 2001.
49. K. R. Scherer, "Which emotions can be induced by music? what are the underlying mechanisms? and how can we measure them,"J. New Music Res., vol. 33, no. 5, pp. 239-251, 2004.
50. J. Hanratty, "Individual and situational differences in emotional expression," Ph.D. dissertation, School of Psychol., Queen's Univ. Belfast, Belfast, U.K., 2010.
51. J.-C. Wang, Y.-H. Yang, H.-M. Wang, and S.-K. Jeng, "Personalized music emotion recognition via model adaptation,"in Proc. APSIPA Annu. Summit Conf., 2012, pp. 1-7.
52. J.-C. Wang, Y.-H. Yang, I. Jhuo, Y.-Y. Lin, and H.-M. Wang, "The acousticvisual emotion Gaussians model for automatic generation of music video," in Proc. ACM Multimedia, 2012, pp. 1379-1380.
53. D. Su and P. Fung, "Personalized music emotion classification via active learning," in Proc. ACM Int. Workshop Music Inf. Retrieval User-Centered Multimodal Strategies, 2012, pp. 51-62.
54. J. A. Russell, "A circumplex model of affect," J. Personality Soc. Sci., vol. 39, no. 6, pp. 1161-1178, 1980.
55. A. Gabrielsson, "Emotion perceived and emotion felt: Same or different?"Musicae Sci., vol. 5, pp. 123-147, 2002.
56. K. Krippendorff, Content Analysis: An Introduction to its Methodology. Thousand Oaks, CA, USA: Sage, 2013.
57. K. V. Mardia, "Measures of multivariate skewness and kurtosis with applications," Biometrika, vol. 57, no. 3, pp. 519-530, 1970.
58. Y.-H. Yang, Y.-C. Lin, H.-T. Cheng, and H. H. Chen, "Mr. Emo: Music retrieval in the emotion plane," in Proc. ACM Multimedia, 2008, pp. 1003-1004.
59. D. A. Reynolds, T. F. Quatieri, and R. B. Dunn, "Speaker verification using adapted Gaussian mixture models," Digital Signal Process., vol. 10, no. 1-3, pp. 19-41, 2000.
60. E. Ho€orster, R. Lienhart, and M. Slaney, "Continuous visual vocabulary models for pLSA-based scene recognition," in Proc. ACM Int. Conf. Image Video Retrieval, 2008, pp. 319-328.
61. K. Yoshii, M. Goto, K. Komatani, T. Ogata, and H. Okuno, "An efficient hybrid music recommender system using an incrementally trainable probabilistic generative model," IEEE Trans. Audio, Speech Lang. Process., vol. 16, no. 2, pp. 435-447, 2008.
62. N. Dehak, P. J. Kenny, R. Dehak, P. Dumouchel, and P. Ouellet, "Front-end factor analysis for speaker verification," IEEE Trans. Audio, Speech Lang. Process., vol. 19, no. 4, pp. 788-798, 2011.
63. J.-C. Wang, H.-S. Lee, H.-M. Wang, and S.-K. Jeng, "Learning the similarity of audio music in bag-of-frames representation from tagged music data," in Proc. Int. Soc. Music Inf. Retrieval Conf., 2011, pp. 85-90.
64. B. Schölkopf, A. J. Smola, R. C. Williamson, and P. L. Bartlett, "New support vector algorithms," Neural Comput., vol. 12, pp. 1207-1245, 2000.
65. B. L. Sturm, "Evaluating music emotion recognition: Lessons from music genre recognition," in Proc. Int. Workshop Affective Anal. Multimeda, 2013, pp. 1-6.
66. R. E. Thayer, The Biopsychology of Mood and Arousal. New York, NY, USA: Oxford Univ. Press, 1989.
67. J. A. Sloboda and P. N. Juslin, "At the interface between the inner and outer world: Psychological perspectives," in Handbook of Music and Emotion: Theory, Research, Applications, P. N. Juslin and J. A. Sloboda, Eds. New York, NY, USA: Oxford Univ. Press, 2010.
68. J.-C. Wang, H.-M. Wang, and G. Lanckriet, "A histogram density modeling approach to music emotion recognition," in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., accepted for publication, 2015.
69. D. M. Blei, A. Y. Ng, and M. I. Jordan, "Latent dirichlet allocation," J. Mach. Learn. Res., vol. 3, pp. 993-1022, 2003.
70. X. Huang, A. Acero, and H.-W. Hon, Spoken Language Processing. Englewood Cliffs, NJ, USA: Prentice Hall PTR, 2001.
71. M. Hoffman, D. Blei, and P. Cook, "Easy as CBA: A simple probabilistic model for tagging music," in Proc. Int. Soc. Music Inf. Retrieval Conf., 2009, pp. 369-374.
72. L. Su, C.-C. M. Yeh, J.-Y. Liu, J.-C. Wang, and Y.-H. Yang, "A systematic evaluation of the bag-of-frames representation for music information retrieval," IEEE Trans. Multimedia, vol. 16, no. 5, pp. 1188-1200, Aug. 2014.
73. Y. Vaizman, B. McFee, and G. Lanckriet, "Codebook-based audio feature representation for music information retrieval,"IEEE/ACM Trans. Audio, Speech, Lang. Process., vol. 22, no. 10, pp. 1483-1493, Oct. 2014.
74. J.-C. Wang, H.-S. Lee, S.-K. Jeng, and H.-M. Wang, "Posterior weighted Bernoulli mixture model for music tag annotation and retrieval," in Proc. APSIPA Annu. Summit Conf., 2010, pp. 247-252.
75. K. Seyerlehner, "Content-based music recommender systems: Beyond simple frame-level audio similarity," Ph.D. dissertation, Johannes Kepler Univ. Linz, Linz, Austria, 2010.
76. Y.-A. Chen, J.-C. Wang, Y.-H. Yang, and H. H. Chen, "Linear regression-based adaptation of music emotion recognition models for personalization," in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2014, pp. 2168-2172.
77. E. Bigand, S. Vieillard, F. Madurell, J. Marozeau, and A. Dacquet, "Multidimensional scaling of emotional responses to music: The effect of musical expertise and of the duration of the excerpts,"Cognition Emotion, vol. 19, no. 8, pp. 1113-1139, 2005.
78. G. Collier, "Beyond valence and activity in the emotional connotations of music," Psychol. Music, vol. 35, no. 1, pp. 110-131, 2007.
79. C. M. Bishop, Pattern Recognition and Machine Learning. New York, NY, USA: Springer-Verlag, 2006.
80. J.-C. Wang, M.-S. Wu, H.-M. Wang, and S.-K. Jeng, "Query by multi-tags with multi-level preferences for content-based music retrieval," in Proc. Int. Conf. Multimedia Expo, 2011, pp. 1-6.
81. O. Lartillot and P. Toiviainen, "A Matlab toolbox for musical feature extraction from audio," in Proc. Int. Conf. Digital Audio Effects, 2007, pp. 237-244.
82. S. Furui, "Speaker-independent isolated word recognition based on emphasized spectral dynamics," in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 1986, vol. 11, pp. 1991-1994.
83. P. N. Juslin, "Cue utilization in communication of emotion in music performance: Relating performance to perception," J. Exp. Psychol.: Human Perception Perform., vol. 16, no. 6, pp. 1797-1813, 2000.
84. A. Gabrielsson and E. Lindströom, "The influence of musical structure on emotional expression," in Music and Emotion: Theory and Research, P. N. Juslin and J. A. Sloboda, Eds. New York, NY, USA: Oxford Univ. Press, 2001.
85. J. Hershey and P. Olsen, "Approximating the Kullback Leibler divergence between Gaussian mixture models," in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2007, vol. 4, pp. 317-320.
86. Y.-H. Yang and X. Hu, "Cross-cultural music moood classification: A comparison on English and Chinese songs," in Proc. 8th Int. Soc. Music Inf. Retrieval Conf., 2012, pp. 19-24.
87. J.-C. Wang, Y.-H. Yang, and H.-M. Wang, "Affective music information retrieval," in Emotions and Personality in Personalized Services, M. Tkalcic et al., Eds., in press, 2015.
88. J.-C. Wang, H.-M. Wang, and S.-K. Jeng, "Playing with tagging: A real-time tagging music player," in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2012, pp. 77-80.
89. C.-C. Chang and C.-J. Lin, "LIBSVM: A library for support vector machines," ACM Trans. Intell. Syst. Technol., vol. 2, pp. 27:1-27:27, 2011.
90. J.-C. Wang, Y.-H. Yang, K. Chang, H.-M. Wang, and S.-K. Jeng, "Exploring the relationship between categorical and dimensional emotion semantics of music," in Proc. Int. ACM Workshop Music Inf. Retrieval User-Centered Multimodal Strategies, 2012, pp. 63-68.
91. M. Quirin, M. Kazén, and J. Kuhl, "When nonsense sounds happy or helpless: The implicit positive and negative affect test (ipanat),"J. Personality Soc. Psychol., vol. 97, no. 3, p. 500, 2009.