A neural network model for the prediction of musical emotions

Advances in Cognitive Systems

Volumn , Issue , 2010, Pages 333-370

  Coutinho, Eduardo ^a   Cangelosi, Angelo ^a  

^a INSTITUTO SUPERIOR TÉCNICO   (Portugal)

Author keywords

Acoustic signal processing; Bioacoustics; Computational representation; Connectionist network model; Dynamic analysis; Emotional response; Hearing; Heart rate; Human subjective feeling; Listener affective response; Mean pitch; Mean STFT flux; Music; Music piece; Music structural aspect; Musical acoustics; Musical emotion prediction; Neural nets; Neural network model; Perceptual process; Peripheral feedback; Physiological activation; Physiological arousal; Physiological cue; Physiology; Pitch variation; Power spectrum centroid; Psychoacoustic dimension; Psychoacoustic feature; Psychology; Self perception; Sound loudness; Sound sharpness; Sound tempo; Spatio temporal neural network; Timbre

Indexed keywords

ACOUSTIC SIGNAL PROCESSING; ACOUSTICS; ACTIVATION ANALYSIS; AUDITION; BEHAVIORAL RESEARCH; BIOACOUSTICS; CHEMICAL ACTIVATION; DYNAMIC ANALYSIS; FORECASTING; HEART; NEURAL NETWORKS; NEUROPHYSIOLOGY; PHYSIOLOGY; POWER SPECTRUM; SIGNAL PROCESSING;

AFFECTIVE RESPONSE; COMPUTATIONAL REPRESENTATION; CONNECTIONIST NETWORKS; EMOTIONAL RESPONSE; HEART RATES; HUMAN SUBJECTIVE FEELING; MEAN PITCH; MUSIC; MUSIC PIECE; MUSICAL ACOUSTICS; MUSICAL EMOTION; NEURAL NETWORK MODEL; PHYSIOLOGICAL AROUSAL; PHYSIOLOGICAL CUE; PITCH VARIATION; PSYCHOACOUSTIC DIMENSIONS; PSYCHOACOUSTIC FEATURE; PSYCHOLOGY; SPATIO TEMPORAL; STRUCTURAL ASPECTS; TIMBRE;

PHYSIOLOGICAL MODELS;

EID: 85014296970     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1049/PBCE071E_ch12     Document Type: Chapter

References (95)

1. Balkwill, L., & Thompson, W. (1999). ‘A cross-cultural investigation of the perception of emotion in music: psychophysical and cultural cues’. Music Perception, 17(1), 43-64.
2. Bechara, A., Damasio, H., & Damasio, A. (2003.) ‘Role of the amygdala in decision-making’. Annals of the New York Academy of Sciences, 985(1) 356-369.
3. Berlyne, D. (1974). Studies in the New Experimental Aesthetics: Steps Toward an Objective Psychology of Aesthetic Appreciation. London, UK: Halsted Press.
4. Berntson, G., Shafi, R., Knox, D., & Sarter, M. (2003). ‘Blockade of epinephrine priming of the cerebral auditory evoked response by cortical cholinergic deafferentation’. Neuroscience, 116(1), 179-186.
5. Blood, A., & Zatorre, R. (2001). ‘Intensely pleasurable responses to music correlate with activity in brain regions implicated in reward and emotion’. Proceedings of the National Academy of Sciences, 98(20), 11818-11823.
6. Blood, A., Zatorre, R., Bermudez, P., & Evans, A. (1999). ‘Emotional responses to pleasant and unpleasant music correlate with activity in paralimbic brain regions’. Nature Neuroscience, vol. 2, pp. 382-387.
7. Bradley, M., & Lang, P. (1994). ‘Measuring emotion: the Self-Assessment Manikin and the semantic differential’. Journal of Behavior Therapy and Experimental Psychiatry, 25(1), 49-59.
8. Brown, S. (2000). ‘Are Music and Language Homologues’? In N. Wallin, B. Merker, & S. Brown (Eds.), The Origins of Music (pp. 271-300). Cambridge USA: MIT Press.
9. Cabrera, D. (1999). ‘Psysound: a computer program for psychoacoustical analysis’. Proceedings of the Annual Conference of the Acoustical Society of Australia (pp. 47-54). Melbourne, Australia: Australian Acoustical Society.
10. Cacioppo, J., Klein, D., Berntson, G., & Hatfield, E. (1993). ‘The psychophysiology of emotion’. In M. Lewis & J. Haviland (Eds.), Handbook of Emotions (pp. 119-142). New York and London: The Guildford Press.
11. Coutinho, E. (2008). Computational and psycho-physiological investigations of musical emotions. Unpublished doctoral dissertation, University of Plymouth.
12. Coutinho, E., & Cangelosi, A. (2009). ‘The use of spatio-temporal connectionist models in psychological studies of musical emotions’. Music Perception. 27(1), 1-15.
13. Coutinho, E., & Dibben, N. (in press). ‘Music, Speech and Emotion: psychophysiological and computational investigations’. Submitted for publication.
14. Damasio, A. (1994). Descarte’s Error: Emotion, Reason and the Human Brain, New York: Grosset/Putnam Books.
15. Damasio, A. (2000). The Feeling of What Happens: Body, Emotion and the Making of Consciousness. London: Vintage.
16. Davidson, R., Scherer, K., & Goldsmith, H. (2003). Handbook of Affective Sciences. Oxford, New York: Oxford University Press.
17. Dell, G. (2002). ‘A spreading-activation theory of retrieval in sentence production’. Psycholinguistics: Critical Concepts in Psychology, 93(3), 283-321.
18. Deutsch, D., Henthorn, T., & Dolson, M. (2004). ‘Absolute pitch, speech, and tone language: some experiments and a proposed framework’. Music Perception, 21(3), 339-356.
19. Dibben, N. (2004, Fall). ‘The role of peripheral feedback in emotional experience with music’. Music Perception, 22(1), 79-115.
20. Dissanayake, E. (2000). ‘Antecedents of the temporal arts in early motherinfant interaction’. In N.L. Wallin, B. Merker, & S. Brown (Eds.), The Origins of Music (pp. 389-410). Cambridge, USA: MIT Press.
21. Dowling, W., & Harwood D. (1986). Music Cognition. San Diego, CA: Academic Press.
22. Ekman, P., & Davidson, R. (1994). The Nature of Emotion: Fundamental Questions. Cambridge, MA: Oxford University Press.
23. Elman, J. (1990). ‘Finding structure in time’. Cognitive Science, vol. 14, pp. 179-211.
24. Elman, J. (1991). ‘Distributed representations, simple recurrent networks, and grammatical structure’. Machine Learning, 7(2-3), 195-225.
25. Feldman, L. (1995). ‘Valence focus and arousal focus: individual differences in the structure of affective experience’. Journal of Personality and Social Psychology, 69(1), 153-166.
26. Frijda, N. (1986). The Emotions. London, UK: Cambridge University Press.
27. Fritz, T., Jentschke, S., Gosselin, N., Sammler, D., Peretz, I., & Turner, R. (2009). ‘Universal recognition of three basic emotions in music’. Current Biology, 19(7), 1-4.
28. Gabrielsson, A. (2002). ‘Emotion perceived and emotion felt: same or different’? Musicae Scientiae, Special Issue 2001-2002, pp. 123-147.
29. Gabrielsson, A., & Juslin, P. (1996). ‘Emotional expression in music performance: between the performer’s intention and the listener’s experience’. Psychology of Music, 24(1), 68.
30. Gabrielsson, A., & Lindström, E. (2001). ‘The influence of musical structure on emotional expression’. In P. Juslin & J. Sloboda (Eds.), Music and Emotion: Theory and Research (pp. 223-248). New York: Oxford University Press.
31. Giles, C., Lawrence, S., & Tsoi, A. (2001). ‘Noisy time series prediction using a recurrent neural network and grammatical inference’. Machine Learning, 44(1/2), 161-183.
32. Goldstein, A. (1980). ‘Thrills in response to music and other stimuli’. Physiological Psychology, 8(1), 126-129.
33. Greenwald, M., Cook, E., & Lang, P. (1989). ‘Affective judgment and psychophysiological response: dimensional covariation in the evaluation of pictorial stimuli’. Journal of Psychophysiology, 3(1), 51-64.
34. Grewe, O., Nagel, F., Kopiez, R., & Altenmuller, E. (2005). ‘How does music arouse ‘chills’? Investigating strong emotions, combining psychological, physiological, and psychoacoustical methods’. Annals of the New York Academy of Sciences, 1060(1), 446.
35. Harrer, G., & Harrer, H. (1977). ‘Music and the brain: studies in the neurology of music’. In M. Critchley & R. A. Henson (Eds.), Music and the Brain. Studies in the Neurology of Music (pp. 202-216). London, UK: William Heinemann Medical Books.
36. Hevner, K. (1936, April). ‘Experimental studies of the elements of expression in music’. The American Journal of Psychology, 48(2), 246-268.
37. Hochreiter, S., & Schmidhuber, J. (1997). ‘Long short-term memory’. Neural Computation, 9(8), 1735-1780.
38. Hotelling, H. (1936). ‘Relations between two sets of variables’. Biometrika, 28(3), 321-377.
39. Iwanaga, M., & Tsukamoto, M. (1997). ‘Effects of excitative and sedative music on subjective and physiological relaxation’. Perceptual and Motor Skills, 85(1), 287-296.
40. Jordan, M. (1990). ‘Attractor dynamics and parallelism in a connectionist sequential machine’. In J. Diederich (Ed.), Artificial Neural Networks: Concept Learning (pp. 112-127). Piscataway, NJ: IEEE Press.
41. Juslin, P., & Laukka, P. (2003). ‘Communication of emotions in vocal expression and music performance: different channels, same code’? Psychological Bulletin, 129(5), 770-814.
42. Juslin, P., & Laukka, P. (2004). ‘Expression, perception, and induction of musical emotions: a review and a questionnaire study of everyday listening’. Journal of New Music Research, 33(3), 217-238.
43. Juslin, P., & Sloboda, J. (2001). Music and Emotion: Theory and Research. New York: Oxford University Press.
44. Kellaris, J., & Kent, R. (1993). ‘An exploratory investigation of responses elicited by music varying in tempo, tonality, and texture’. Journal of Consumer Psychology, 2(4), 381-401.
45. Khalfa, S., Peretz, I., Blondin, J., & Manon, R. (2002). ‘Event-related skin conductance responses to musical emotions in humans’. Neuroscience Letters, 328(2), 145-149.
46. Kivy, P. (1990). Music Alone: Philosophical Reflections on the Purely Musical Experience. Ithaca, NY: Cornell University Press.
47. Koelsch, S., Fritz, T., Cramon, D., Müller, K., & Friederici, A. (2006). ‘Investigating emotion with music: an fMRI study’. Human Brain Mapping, vol. 27, pp. 239-250.
48. Korhonen, M. (2004a). Modeling continuous emotional appraisals of music using system identification. Unpublished master’s thesis, University of Waterloo.
49. Korhonen, M. (2004b, August). Modeling continuous emotional appraisals of music using system identification. Available from http://www.sauna.org/kiulu/emotion.html [Accessed 8 December 2008].
50. Korhonen, M., Clausi, D., & Jernigan, M. (2004). ‘Modeling continuous emotional appraisals using system identification’. In S. Libscomb, R. Ashley, R. Gjerdingen, & P. Webster, (Eds.), Proceedings of the 8th international conference on music perception and cognition. Evaston. Adelaide: Causal Productions.
51. Kremer, S. (2001). ‘Spatiotemporal connectionist networks: a taxonomy and review’. Neural Computation, 13(2), 249-306.
52. Krumhansl, C.L. (1996). ‘A perceptual analysis of Mozart’s piano sonata k. 282: segmentation, tension, and musical ideas’. Music Perception, 13(3), 401-432.
53. Krumhansl, C.L. (1997). ‘An exploratory study of musical emotions and psychophysiology’. Canadian Journal of Experimental Psychology, 51(4), 336-353.
54. Lang, P., Bradley, M., & Cuthbert, B. (1998, September). ‘Emotion and motivation: measuring affective perception’. Journal of Clinical Neurophysiology, 15(5), 397-408.
55. Langer, S. (1942). Philosophy in a New Key. Cambridge: Harvard University Press.
56. Lawrence, S., Giles, C., & Fong, S. (2000, Jan/Feb). ‘Natural language grammatical inference with recurrent neural networks’. IEEE Transactions on Knowledge and Data Engineering, 12(1), 126-140.
57. Lazarus, R. (1991). ‘Cognition and motivation in emotion’. American Psychologist, 46(4), 352-367.
58. Levitin, D. (2006). This is Your Brain on Music. New York: Dutton.
59. Ljung, L. (1986). System Identification: Theory for the User. New Jersey: Prentice-Hall.
60. Madsen, C., Brittin, R., & Capperella-Sheldon, D. (1993). ‘An empirical investigation of the aesthetic response to music’. Journal of Research in Music Education, vol. 43, pp. 57-69.
61. McClelland, J., & Elman, J. (1986). ‘The TRACE model of speech perception’. Cognitive Psychology, vol. 18, 1-86.
62. Meyer, L. (1956). Emotion and Meaning in Music. Chicago, IL: The University of Chicago Press.
63. Mozer, M. (1999). ‘Neural network music composition by prediction: exploring the benefits of psychoacoustic constraints and multiscale processing’. In N. Griffith & P. Todd (Eds.), Musical Networks: Parallel Distributed Perception and Performance (pp. 227-259). Cambridge, MA: MIT Press.
64. Nagel, F., Kopiez, R., Grewe, O., & Altenmüller, E. (2007). ‘Emujoy: software for continuous measurement of perceived emotions in music’. Behavior Research Methods, 39(2), 283-290.
65. Nielsen, F. (1987). ‘The semiotic web ’86: an international year-book’. In T.A. Seboek & J. Umiker-Seboek (Eds.), The Semiotic Web, (pp. 491-513). Berlin: Mouton de Gruyter.
66. Panksepp, J. (1998). Affective Neuroscience: The Foundations of Human and Animal Emotions. New York: Oxford University Press.
67. Panksepp, J., & Bernatzky, G. (2002). ‘Emotional sounds and the brain: the neuro-affective foundations of musical appreciation’. Behavioural Processes, 60(2), 133-155.
68. Parncutt, R. (1989). Harmony: A Psychoacoustical Approach. Berlin, Germany: Springer Verlag.
69. Patel, A., & Balaban, E. (2000). ‘Temporal patterns of human cortical activity reflect tone sequence structure’. Nature, vol. 404, pp. 80-84.
70. Peretz, I., Gagnon, L., & Bouchard, B. (1998). ‘Music and emotion: perceptual determinants, immediacy, and isolation after brain damage’. Cognition, 68(2), 111-141.
71. Philippot, P., Chapelle, G., & Blairy, S. (2002). ‘Respiratory feedback in the generation of emotion’. Cognition & Emotion, 16(5), 605-627.
72. Resnicow, J., Salovey, P., & Repp, B. (2004). ‘Is recognition of emotion in music performance an aspect of emotional intelligence’? Music Perception, 22(1), 145-158.
73. Rickard, N.S. (2004). ‘Intense emotional responses to music: a test of the physiological arousal hypothesis’. Psychology of Music, 32(4), 371-388.
74. Robazza, C., Macaluso, C., & D’Urso, V. (1994). ‘Emotional reactions to music by gender, age, and expertise’. Perceptual Motor Skills, 79(2), 939-944.
75. Rumelhart, D., Hintont, G., & Williams, R. (1986). ‘Learning representations by back-propagating errors’. Nature, 323(6088), 533-536.
76. Rumelhart, D., & McClelland, J. (1986). ‘PDP models and general issues in cognitive science’. In D. Rumelhart & J. McClelland (Eds.), Parallel Distributed Processing: Explorations in the Microstructure of Cognition, (vol. 1). Cambridge, MA: MIT Press.
77. Russell, J. (1980). ‘A circumplex model of affect’. Journal of Personality and Social Psychology, 39(6), 1161-1178.
78. Russell, J. (1989). ‘Measures of emotion’. In R. Plutchik & H. Kellerman (Eds.), Emotion: Theory, Research, and Experience, (vol. 4). Toronto, Canada: Academic.
79. Scherer, K. (2004). ‘Which emotions can be induced by music? what are the underlying mechanisms? and how can we measure them’? Journal of New Music Research, 33(3), 239-251.
80. Scherer, K., & Oshinsky, J. (1977). ‘Cue utilization in emotion attribution from auditory stimuli’. Motivation and Emotion, 1(4), 331-346.
81. Scherer, K.R., & Zentner M. (2001). ‘Emotional effects of music: Production rules’. In: P.N. Juslin, & J. A. Sloboda (Eds.), Music and Emotion: Theory and Research. Oxford, New York: Oxford University Press.
82. Schubert, E. (1999a). Measurement and time series analysis of emotion in music. Unpublished doctoral dissertation, University of New South Wales.
83. Schubert, E. (1999b). ‘Measuring emotion continuously: validity and reliability of the two dimensional emotion space’. Australian Journal of Psychology, 51(3), 154-165.
84. Schubert, E. (2001). ‘Continuous measurement of self-report emotional response to music’. In P. Juslin, & J. Sloboda (Eds.), Music and Emotion: Theory and Research, (vol. 393-414). Oxford, UK: Oxford University Press.
85. Schubert, E. (2004). ‘Modeling perceived emotion with continuous musical features’. Music Perception, 21(4), 561-585.
86. Sloboda, J. (1991). ‘Music structure and emotional response: some empirical findings’. Psychology of Music, vol. 19, pp. 110-120.
87. Sloboda, J., & Lehmann, A. (2001). ‘Tracking performance correlates of changes in perceived intensity of emotion during different interpretations of a Chopin piano prelude’. Music Perception, 19(1), 87-120.
88. Thayer, J. (1986). Multiple indicators of affective response to music. Unpublished doctoral dissertation, New York University, Graduate School of Arts and Science.
89. Thompson, W.F., & Balkwill, L.-L. (2006). ‘Decoding speech prosody in five languages’. Semiotica, 158(1/4), 407-424.
90. Tzanetakis, G., & Cook, P. (2000). ‘Marsyas: a framework for audio analysis’. Organised Sound, 4(03), 169-175.
91. Witvliet, C., & Vrana, S. (1996). ‘The emotional impact of instrumental music on affect ratings, facial EMG, autonomic measures, and the startle reflex: effects of valence and arousal’. Psychophysiology, 33 (Supplement 1), 91.
92. Witvliet, C., Vrana, S., & Webb-Talmadge, N. (1998). ‘In the mood: emotion and facial expressions during and after instrumental music and during an emotional inhibition task’. Psychophysiology, 35 (Supplement 1), 88.
93. Wundt, W. (1896). Grundriss der Psychologie (Outlines of Psychology). Leipzig: Engelmann.
94. Zatorre, R.J. (2005). ‘Music, the food of neuroscience?’ Nature, vol. 434, pp. 312-315.
95. Zwicker, E., & Fastl, H. (1990). Psychoacoustics. New York, NY: Springer.