Emotional speech processing: Disentangling the effects of prosody and semantic cues

Cognition and Emotion

Volumn 25, Issue 5, 2011, Pages 834-853

  Pell, Marc D ^a   Jaywant, Abhishek ^a   Monetta, Laura ^b   Kotz, Sonja A ^c  

^a MCGILL UNIVERSITY   (Canada)

^b UNIVERSITÉ LAVAL   (Canada)

^c MAX PLANCK INSTITUTE FOR HUMAN COGNITIVE AND BRAIN SCIENCES   (Germany)

Author keywords

Audio visual priming; Emotions; Facial expression; Speech perception; Vocal cues

Indexed keywords

ACCURACY; ADULT; ARTICLE; ASSOCIATION; ASSOCIATIVE MEMORY; AUDITORY STIMULATION; EMOTION; EXPERIMENTAL DESIGN; FACIAL EXPRESSION; FEMALE; HUMAN; HUMAN EXPERIMENT; LATENT PERIOD; LINGUISTICS; MALE; NORMAL HUMAN; PRIORITY JOURNAL; PROSODY; RECOGNITION; SEMANTICS; SPEECH PERCEPTION; TASK PERFORMANCE; VALIDATION PROCESS; VISUAL STIMULATION;

ACOUSTIC STIMULATION; ADULT; CUES; DECISION MAKING; EMOTIONS; FACIAL EXPRESSION; FEMALE; HUMANS; LINGUISTICS; MALE; PHOTIC STIMULATION; PSYCHOMOTOR PERFORMANCE; REACTION TIME; SEMANTICS; SPEECH PERCEPTION;

EID: 79961226203     PISSN: 02699931     EISSN: 14640600     Source Type: Journal    
DOI: 10.1080/02699931.2010.516915     Document Type: Article

References (83)

1. Astesano, C., Besson, M., & Alter, K. (2004). Brain potentials during semantic and prosodic processing in French. Cognitive Brain Research, 18, 172-184.
2. Attardo, S., Eisterhold, J., Hay, J., & Poggi, I. (2003). Multimodal markers of irony and sarcasm. Humor-International Journal of Humor Research, 16(2), 243-260.
3. Banse, R., & Scherer, K. R. (1996). Acoustic profiles in vocal emotion expression. Journal of Personality and Social Psychology, 70(3), 614-636.
4. Beaucousin, V., Lacheret, A., Turbelin, M.-R., Morel, M. L., Mazoyer, B., & Tzourio-Mazoyer, N. (2007). fMRI study of emotional speech comprehension. Cerebral Cortex, 17(2), 339-352.
5. Borod, J. C., Cicero, B. A., Obler, L. K., Welkowitz, J., Erhan, H. M., Santschi, C., et al. (1998). Right hemisphere emotional perception: Evidence across multiple channels. Neuropsychology, 12(3), 446-458.
6. Borod, J. C., Pick, L. H., Hall, S., Sliwinski, M., Madigan, N., Obler, L. K., et al. (2000). Relationships among facial, prosodic, and lexical channels of emotional perceptual processing. Cognition and Emotion, 14(2), 193-211.
7. Bower, G. H. (1981). Mood and memory. American Psychologist, 36(2), 129-148.
8. Bower, G. H. (1987). Commentary on mood and memory. Behaviour Research and Therapy, 25(6), 443-455.
9. Bowers, D., Bauer, R., & Heilman, K. (1993). The nonverbal affect lexicon: Theoretical perspectives from neuropsychological studies of affect perception. Neuropsychology, 7, 433-444.
10. Bowers, D., Coslett, H. B., Bauer, R. M., Speedie, L. J., & Heilman, K. M. (1987). Comprehension of emotional prosody following unilateral hemispheric lesions: Processing defect versus distraction defect. Neuropsychologia, 25(2), 317-328.
11. Breitenstein, C., Daum, I., & Ackermann, H. (1998). Emotional processing following cortical and subcortical brain damage: Contribution of the frontostriatal circuitry. Behavioural Neurology, 11, 29-42.
12. Buchanan, T. W., Lutz, K., Mirzazade, S., Specht, K., Jon Shah, N., Zilles, K., et al. (2000). Recognition of emotional prosody and verbal components of spoken language: An fMRI study. Cognitive Brain Research, 9, 227-238.
13. Caffi, C., & Janney, R. W. (1994). Toward a pragmatics of emotive communication. Journal of Pragmatics, 22, 325-373.
14. Carroll, N. C., & Young, A. W. (2005). Priming of emotion recognition. The Quarterly Journal of Experimental Psychology, 58A(7), 1173-1197.
15. Cheang, H. S., & Pell, M. D. (2008). The sound of sarcasm. Speech Communication, 50, 366-381.
16. Dara, C., Monetta, L., & Pell, M. D. (2008). Vocal emotion processing in Parkinson's disease: Reduced sensitivity to negative emotions. Brain Research, 1188, 100-111.
17. de Gelder, B., Bocker, K. B. E., Tuomainen, J., Hensen, M., & Vroomen, J. (1999). The combined perception of emotion from voice and face: Early interaction revealed by human brain responses. Neuroscience Letters, 260, 133-136.
18. de Gelder, B., & Vroomen, J. (2000). The perception of emotions by ear and by eye. Cognition and Emotion, 14(3), 289-311.
19. Dimberg, U., &.Öhman, A. (1996). Beyond the wrath: Psychophysiological responses to facial stimuli. Motivation and Emotion, 20(2), 149-182.
20. Eckstein, K., & Friederici, A. D. (2006). It's early: Event-related potential evidence for initial interaction of syntax and prosody in speech comprehension. Journal of Cognitive Neuroscience, 18(10), 1696-1711.
21. Ekman, P. (1992). An argument for basic emotions. Cognition and Emotion, 6, 169-200.
22. Fazio, R. H. (2001). On the automatic activation of associated evaluations: An overview. Cognition and Emotion, 15(2), 115-141.
23. Friederici, A. D. (2002). Towards a neural basis of auditory sentence processing. Trends in Cognitive Sciences, 6(2), 78-84.
24. Grandjean, D., Banziger, T., & Scherer, K. R. (2006). Intonation as an interface between language and affect. Progress in Brain Research, 156, 235-268.
25. Grimshaw, G. M. (1998). Integration and interference in the cerebral hemispheres: Relations with hemispheric specialization. Brain and Cognition, 36(2), 108-127.
26. Hagoort, P., Hald, L., Bastiaansen, M., & Petersson, K. M. (2004). Integration of word meaning and world knowledge in language comprehension. Science, 304, 438-441.
27. Hansen, C., & Shantz, C. (1995). Emotion-specific priming: Congruence effects on affect and recognition across negative emotions. Personality and Social Psychology Bulletin, 21, 548-557.
28. Hietanen, J. K., Leppanen, J. M., Illi, M., & Surakka, V. (2004). Evidence for the integration of audiovisual emotional information at the perceptual level of processing. European Journal of Cognitive Psychology, 16(6), 769-790.
29. Hsu, S.-M., Hetrick, W. P., & Pessoa, L. (2008). Depth of facial expression processing depends on stimulus visibility: Behavioral and electrophysiological evidence of priming effects. Cognitive, Affective Behavioral Neuroscience, 8(3), 282-292.
30. Innes-Ker, A., & Niedenthal, P. (2002). Emotion concepts and emotional states in social judgment and categorization. Journal of Personality and Social Psychology, 83, 804-816.
31. Ishii, K., Reyes, J. A., & Kitayama, S. (2003). Spontaneous attention to word content versus emotional tone: Differences among three cultures. Psychological Science, 14(1), 39-46.
32. Johnson, W. F., Emde, R. N., Scherer, K. R., & Klinnert, M. D. (1986). Recognition of emotion from vocal cues. Archives of General Psychiatry, 43, 280-283.
33. Johnstone, T., van Reekum, C., Oakes, T., & Davidson, R. J. (2006). The voice of emotion: An fMRI study of neural responses to angry and happy vocal expressions. Social, Cognitive, and Affective Neuroscience, 1, 242-249.
34. Kitayama, S., & Howard, S. (1994). Affective regulation of perception and comprehension: Amplification and semantic priming. In P. M. Niedenthal & S. Kitayama (Eds.), The heart's eye: Emotional influences in perception and attention (pp. 41-65). San Diego, CA: Academic Press.
35. Kitayama, S., & Ishii, K. (2002). Word and voice: Spontaneous attention to emotional utterances in two languages. Cognition and Emotion, 16(1), 29-59.
36. Kotz, S. A., & Paulmann, S. (2007). When emotional prosody and semantics dance cheek to cheek: ERP evidence. Brain Research, 115, 107-118.
37. Kreifelts, B., Ethofer, T., Grodd, W., Erb, M., & Wildgruber, D. (2007). Audiovisual integration of emotional signals in voice and face: An event-related fMRI study. NeuroImage, 37, 1445-1456.
38. Kreifelts, B., Ethofer, T., Shiozawa, T., Grodd, W., & Wildgruber, D. (2009). Cerebral representation of non-verbal emotional perception: fMRI reveals audiovisual integration area between voice-and face-sensitive regions in the superior temporal sulcus. Neuropsychologia, 47(14), 3059-3066.
39. Laukka, P. (2005). Categorical perception of vocal emotion expressions. Emotion, 5, 277-295.
40. Leppänen, J. M., & Hietanen, J. K. (2004). Positive facial expressions are recognized faster than negative facial expressions, but why? Psychological Research, 69, 22-29.
41. Mahon, B. Z., & Caramazza, A. (2008). A critical look at the embodied cognition hypothesis and a new proposal for grounding conceptual content. Journal of Physiology - Paris, 102, 59-70.
42. Mahon, B. Z., & Caramazza, A. (2009). Concepts and categories: A cognitive neuropsychological perspective. Annual Review of Psychology, 60, 27-51.
43. Massaro, D., & Egan, P. (1996). Perceiving affect from the voice and the face. Psychonomic Bulletin and Review, 3(2), 215-221.
44. Mitchell, R. L. C., Elliott, R., Barry, M., Cruttenden, A., & Woodruff, P. W. R. (2003). The neural response to emotional prosody, as revealed by functional magnetic resonance imaging. Neuropsychologia, 41(10), 1410-1421.
45. Mogg, K., & Bradley, B. P. (1999). Orienting of attention to threatening facial expressions presented under conditions of restricted awareness. Cognition and Emotion, 13(6), 713-740.
46. Morais, J., & Ladavas, E. (1987). Hemispheric interactions in the recognition of words and emotional intonations. Cognition and Emotion, 1(1), 89-100.
47. Morton, J., & Trehub, S. (2001). Children's understanding of emotion in speech. Child Development, 72, 834-843.
48. Neely, J. H., Keefe, D. E., & Ross, K. L. (1989). Semantic priming in the lexical decision task: Roles of prospective prime-generated expectancies and retrospective semantic matching. Journal of Experimental Psychology: Learning, Memory and Cognition, 15(6), 1003-1019.
49. Niedenthal, P. M. (1990). Implicit perception of affective information. Journal of Experimental Social Psychology, 26, 505-527.
50. Niedenthal, P. M., Halberstadt, J. B., & Innes-Ker, A. H. (1999). Emotional response categorization. Psychological Review, 106(2), 337-361.
51. Niedenthal, P. M., Halberstadt, J. B., & Setterlund, M. B. (1997). Being happy and seeing "happy": Emotional state mediates visual word recognition. Cognition and Emotion, 11(4), 403-432.
52. Nygaard, L., & Lunders, E. (2002). Resolution of lexical ambiguity by emotional tone of voice. Memory & Cognition, 30(4), 583-593.
53. Nygaard, L. C., & Queen, J. S. (2008). Communicating emotion: Linking affective prosody and word meaning. Journal of Experimental Psychology: Human Perception and Performance, 34(4), 1017-1030.
54. Paulmann, S., Jessen, S., & Kotz, S. A. (2009). Investigating the multimodal nature of human communication: Insights from ERPs. Journal of Psychophysiology, 23(2), 63-76.
55. Paulmann, S., & Kotz, S. A. (2008). An ERP investigation on the temporal dynamics of emotional prosody and emotional semantics in pseudo-and lexical-sentence context. Brain and Language, 105, 59-69.
56. Paulmann, S., & Pell, M. D. (2009). Decoding facial expressions as a function of emotional meaning status: ERP evidence. NeuroReport, 20, 1603-1608.
57. Paulmann, S., & Pell, M. D. (2010). Contextual influences of emotional speech prosody on face processing: how much is enough? Cognitive Affective and Behavioral Neuroscience, 10(2), 230-242.
58. Pell, M. D. (2002). Evaluation of nonverbal emotion in face and voice: Some preliminary findings on a new battery of tests. Brain and Cognition, 48, 499-504.
59. Pell, M. D. (2005a). Nonverbal emotion priming: evidence from the "facial affect decision task". Journal of Nonverbal Behavior, 29(1), 45-73.
60. Pell, M. D. (2005b). Prosody-face interactions in emotional processing as revealed by the facial affect decision task. Journal of Nonverbal Behavior, 29(4), 193-215.
61. Pell, M. D. (2006). Cerebral mechanisms for understanding emotional prosody in speech. Brain and Language, 96(2), 221-234.
62. Pell, M. D., & Baum, S. R. (1997). The ability to perceive and comprehend intonation in linguistic and affective contexts by brain-damaged adults. Brain and Language, 57(1), 80-99.
63. Pell, M. D., Monetta, L., Paulmann, S., & Kotz, S. A. (2009a). Recognizing emotions in a foreign language. Journal of Nonverbal Behaviour, 33(2), 107-120.
64. Pell, M. D., Paulmann, S., Dara, C., Alasseri, A., & Kotz, S. A. (2009b). Factors in the recognition of vocally expressed emotions: A comparison of four languages. Journal of Phonetics, 37, 417-435.
65. Pell, M. D., & Skorup, V. (2008). Implicit processing of emotional prosody in a foreign versus native language. Speech Communication, 50, 519-530.
66. Pourtois, G., de Gelder, B., Vroomen, J., Rossion, B., & Crommelinck, M. (2000). The time-course of intermodal binding between seeing and hearing affective information. NeuroReport, 11(6), 1329-1333.
67. Preston, S., & Stansfield, R. B. (2008). I know how you feel: Task-irrelevant facial expressions are spontaneously processed at a semantic level. Cognitive, Affective Behavioral Neuroscience, 8, 54-64.
68. Rossell, S. L., & Nobre, A. (2004). Semantic priming of different affective categories. Emotion, 4(4), 354-363.
69. Scherer, K. R. (1986). Vocal affect expression: A review and a model for future research. Psychological Bulletin, 99(2), 143-165.
70. Scherer, K. R., Banse, R., Wallbott, H. G., & Goldbeck, T. (1991). Vocal cues in emotion encoding and decoding. Motivation and Emotion, 15(2), 123-148.
71. Schirmer, A., & Kotz, S. A. (2003). ERP evidence for a sex-specific Stroop effect in emotional speech. Journal of Cognitive Neuroscience, 15(8), 1135-1148.
72. Schirmer, A., & Kotz, S. A. (2006). Beyond the right hemisphere: Brain mechanisms mediating vocal emotional processing. Trends in Cognitive Sciences, 10, 24-30.
73. Schirmer, A., Kotz, S. A., & Friederici, A. D. (2002). Sex differentiates the role of emotional prosody during word processing. Cognitive Brain Research, 14, 228-233.
74. Schirmer, A., Kotz, S. A., & Friederici, A. D. (2005). On the role of attention for the processing of emotions in speech: Sex differences revisited. Cognitive Brain Research, 24, 442-452.
75. Shuster, L. I. (2009). The effect of sublexical and lexical frequency on speech production: An fMRI investigation. Brain & Language, 111(1), 66-72.
76. Tanenhaus, M. K., & Brown-Schmidt, S. (2008). Language processing in the natural world. Philosophical Transactions of the Royal Society B: Biological Sciences, 363, 1105-1122.
77. Techentin, C., Voyer, D., & Klein, R. M. (2009). Between- and within-ear congruency and laterality effects in an auditory semantic/emotional prosody conflict task. Brain and Cognition, 70, 201-208.
78. Thompson, W., & Balkwill, L.-L. (2006). Decoding speech prosody in five languages. Semiotica, 158(1/4), 407-424.
79. Vingerhoets, G., Berckmoes, C., & Stroobant, N. (2003). Cerebral hemodynamics during discrimination of prosodic and semantic emotion in speech studied by transcranial Doppler ultrasonography. Neuropsychology, 17(1), 93-99.
80. Vroomen, J., Driver, J., & de Gelder, B. (2001). Is cross-modal integration of emotional expressions independent of attentional resources? Cognitive, Affective Behavioral Neuroscience, 1, 382-387.
81. Wambacq, I. J. A., & Jerger, J. F. (2004). Processing of affective prosody and lexical-semantics in spoken utterances as differentiated by event-related potentials. Cognitive Brain Research, 20(3), 427-437.
82. Wilson, D., & Wharton, T. (2006). Relevance and prosody. Journal of Pragmatics, 38, 1559-1579.
83. Wurm, L. H., Vakoch, D. A., Strasser, M. R., Calin-Jageman, R., & Ross, S. E. (2001). Speech perception and vocal expression of emotion. Cognition and Emotion, 15(6), 831-852.