Chapter 17 Seeing cries and hearing smiles: Crossmodal perception of emotional expressions

Advances in Psychology

Volumn 129, Issue C, 1999, Pages 425-438

  de Gelder, Beatrice ^a,b   Vroomen, Jean ^a   Pourtois, Gilles ^a,b  

^a TILBURG UNIVERSITY   (Netherlands)

^b UNIVERSITÉ CATHOLIQUE DE LOUVAIN   (Belgium)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 77956784975     PISSN: 01664115     EISSN: None     Source Type: Book Series    
DOI: 10.1016/S0166-4115(99)80040-5     Document Type: Article

References (87)

1. Ahern G.L., and Schwartz G.E. Differential lateralization for positive and negative emotion in the human brain: EEG spectral analysis. Neuropsychologia 23 6 (1985) 745-755
2. Austin J.L. Other minds. In: Urmson J.O., and Warnock G.J. (Eds). Philosophical papers. 2nd ed. (1970), Oxford University Press, London 47-63
3. Baylis G.C., Rolls E.T., and Leonard C.M. Selectivity between faces in the responses of a population of neurons in the cortex in the superior temporal sulcus of the monkey. Brain Research 342 (1985) 91-102
4. Bertelson P. Starting from the ventriloquist: The perception of multimodal events. In: Sabourin M., Craik F.I.M., and Robert M. (Eds). Advances in Psychological Science. Vol. 1: Biological and Cognitive Aspects (1998), Psychology Press, Hove, UK 419-439
5. (this volume). Bertelson P. Ventriloquism: Cross-modal patterning under spatial conflict. In: Aschersleben G., Bachmann T., and Müsseler J. (Eds). Cognitive contributions to the perception of spatial and temporal events (1999), Elsevier, Amsterdam
6. Bertelson P., Vroomen J., de Gelder B., and Driver J. Auditory-visual spatial interaction and the orientation, of visual attention. (submitted manuscript) (1999)
7. Bower T.G.R. Development in infancy (1974), W.H. Freeman, San-Fransisco
8. Bruyer R. Asymmetry of facial expression in brain damaged subjects. Neuropsychologia 19 4 (1981) 615-624
9. Calder A.J., Young A.W., Benson P.J., and Perrett D.L. Self priming from distinctive and caricatured faces. British Journal of Psychology 87 1 (1996) 141-162
10. Calvert G.A., Bullmore E.T., Brammer M.J., Campbell R., Williams S.C.R., McGuire P.K., Woodruff P.W.R., Iversen S.D., and David A.S. Activation of auditory cortex during silent lipreading. Science 276 (1997) 593-596
11. Campbell R. Seeing speech in space and time. Proceedings of the International Conference on Spoken Language Processing (1996) 1493-1498
12. Campbell R., Dodd B., and Burnham D. Hearing by Eye II: advances in the psychology of speechreading and auditory-visual speech (1998), Psychology Press, Hove
13. Carmon A., and Nachshon I. Ear asymmetry in perception of emotional non-verbal stimuli. Acta Psychologica 37 6 (1973) 351-357
14. Crowder R., and Surprenant A. On the linguistic module in auditory memory. In: de Gelder B., and Morais J. (Eds). Speech and reading: A comparative approach (1995), Erlbaum, Hove 94
15. Darwin C. The expressions of emotion in man and animals (1872), John Murray, London
16. Davidson R.J., and Tomarken A.J. Laterality and emotion: An electrophysiological approach. In: Boller F., and Grafman J. (Eds). Handbook of neuropsychology Vol. 3 (1989), Elsevier, Amsterdam 419-441
17. de Gelder B., Bachoud-Levi A., and Vroomen J. Emotion by ear and by eye: Implicit processing of emotion using a Cross-modal approach. Boston, March 23-25. Fourth Annual Meeting of the Cognitive Neuroscience Society 49 (1997) 73
18. de Gelder B., Teunisse J.-P., and Benson P. Categorical perception of facial expressions: Categories and their internal structure. Cognition and Emotion 11 (1997) 1-23
19. Pt. 2. de Gelder B., and Vroomen J. Categorical perception of emotional speech. The Journal of the Acoustical Society 100 4 (1996) 2818
20. de Gelder B., and Vroomen J. Memory for consonants versus vowels in heard and lipread speech. Journal of Memory and Language 31 (1994) 737-756
21. de Gelder B., and Vroomen J. Perceiving emotions by ear and by eye. (submitted manuscipt) (1999)
22. de Gelder B., and Vroomen J. Categorical perception of speech prosody. (manuscript in preparation) (1999)
23. de Gelder B., Vroomen J., and Bertelson P. Upright but not inverted faces modify the perception of emotion in the voice. Current Psychology of Cognition 17 (1998) 1021-1031
24. de Gelder B., Vroomen J., and Bachoud-Lévi A. Impaired speechreading and audiovisual speech integration in prosopagnosia. In: Campbell R., Dodd B., and Burnham D. (Eds). Hearing by Eye 11, Advances in the Psychology of Speechreading and Auditoryvisual Speech (1998), Psychology Press LTD, Hove 195-207
25. de Gelder B., Vroomen J., and Bertelson P. Aspects of Modality in audio-visual processes. In: Stork D.G., and Hennecke M.E. (Eds). Speechreading by Humans and Machines. NATO ASI Series F Vol. 150 (1996), Springer-Verlag, Berlin 197
26. de Gelder B., Böcker K.B.E., Tuomainen J., Hensens M., and Vroomen J. The combined perception of emotion from voice and face: early interaction revealed by electric brain responses. Letters in Neuroscience 260 (1999) 133
27. Delis D.C., Kiefner M.G., and Fridlund A.J. Visuospatial dysfunction following unilateral brain damage: Dissociations in hierarchical hemispatial analysis. Journal of Clinical and Experimental Neuropsychology 10 4 (1988) 421-431
28. Dodd B., and Campbell R. Hearing by Eye: the psychology of lipreading (1987), Lawrence Erlbaum, Hove, UK
29. Dopson W.G., Beckwith B.E., Tucker D.M., and Bullard-Bates P.C. Asymmetry of facial expression in spontaneous emotion. Cortex 20 2 (1984) 243-251
30. Ekman P., Friesen W.V., and O'Sullivan M. Smiles when lying. Journal of Personality and Social Psychology 54 3 (1988) 414-420
31. Ekman P. An argument for basic emotions. Cognition and Emotion 6 (1992) 169-200
32. Etcoff N., and Magee J. Categorical perception of facial expressions. Cognition 44 (1992) 227-240
33. Fodor J.A. The modularity of mind (1983), MIT Press, Cambridge
34. Frijda N. The Emotions (1989), CUP, Cambridge
35. Fried I., Mateer C., Ojemann G., Wohns R., and Fedio P. Organization of visuospatial functions in the human cortex. Brain 105 (1982) 349-371
36. George M.S., Parekh P.I., Rosinsky N., Ketter T.A., Kimbrell T.A., Heilman K.M., Herscovitch P., and Post R.M. Understanding emotional prosody activates right hemisphere regions. Archive of Neurology 53 (1996) 665-670
37. Giard, M. H., & Peronnet, F. (in press). Auditory-visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study. Journal of Cognitive Neuroscience.
38. Gainotti G. Emotional behavior and hemispheric side of lesion. Cortex 8 (1972) 41-55
39. Gainotti G. Disorders of emotions and affect in patients with unilateral brain damage. In: Boller F., and Grafman J. (Eds). Handbook of Neuropsychology Vol. 3 (1989), Elsevier, Amsterdam 345-361
40. Gianotti M., and Pereira N. Psychological approaches to the cerebral palsied child. Revista Brasiliera de deficiencia mental 7 1 (1972) 5-9
41. Gibson J.J. The senses considered as perceptual systems (1966), Houghton Mifflin, Boston
42. Humphreys G.W., and Riddoch J.M. To see or not to see (1987), Lawrence Erlbaum, Hove
43. James W. What is an emotion?. Mind 9 (1884) 188-205
44. Johnson W.F., Emde R., Scherer K., and Klinnert M.D. Recognition of emotion from vocal cues. Archives of General Psychiatry 43 (1986) 280-283
45. Landis T., Assal G., and Perret E. Opposite cerebral hemispheric superiorities for visual associative processing of emotional facial expressions and objects. Nature 278 (1979) 739-740
46. Lane R.D., Reiman E.M., Geoffrey L.A., Schwatrz G.E., and Davidson R.J. Neuroanatomical correlates of happiness, sadness and disgust. The American Journal of Psychiatry 154 (1997) 926-933
47. Ledoux J. The emotional brain (1996), Simon and Schuster, New York
48. Lewkowicz D.J. Developmental changes in infants' bisensory response to synchronous durations. Infant Behavior and Development 8 (1986) 335-353
49. Leonard C.M., Rolls E.T., Wilson F.A., and Baylis G.C. Neurons in the amygdala of the monkey with responses selective for faces. Behavioural Brain Research 15 2 (1985) 159-176
50. Liberman A.M. Speech: A special code (1996), MIT Press, Cambridge, MA
51. Liberman A.M., and Mattingley G. The motor theory of speech perception revised. Cognition 21 (1993) 1-36
52. Lieberman P., and Michaels S.B. Some aspects of fundamental frequency and envelope amplitude as related to emotional content of speech. Journal of the Acoustical Society of America 34 (1962) 922-927
53. Locke J. In: Yolton J.W. (Ed). An essay concerning human understanding 2 vol. (1689-1967-1968), Dent, London
54. MacLeod C.M. Half a century of research on the stroop effect: An integrative review. Psychological Bulletin 109 (1991) 163-203
55. Mahoney A.M., and Sainsbury R.S. Hemispheric asymmetry in the perception of emotional sounds. Brain and Cognition 6 2 (1987) 216-233
56. Massaro D.W. Speech perception by ear and eye (1987), Lawrence Erlbaum, Hillsdale, NJ
57. Massaro D.W., and Egan P.B. Perceiving affect from the voice and the face. Psychonomic Bulletin & Review 3 (1996) 215-221
58. McGurk H., and MacDonals J. Hearing lips and seeing voices. Nature 264 (1976) 746-748
59. Meltzoff A. Towards a developmental cognitive science: The implications of crossmodal matching and imitation for the development of representation and memory. Annals of the New-York Academy of Sciences 608 (1990) 1-37
60. Milner A.D., and Goodale M.A. The visual brain in action (1995), Oxford University Press, Oxford
61. Morris J.S., Frith C.D., Perrett D.I., Rowland D., Young A.W., Calder A.J., and Dolan R.J. A differential neural response in the human amygdala to fearful and happy facial expressions. Nature 383 (1996) 812-815
62. Murray I.R., and Arnott J.L. Toward the simulation of emotion in synthetic speech: A review of the literature on human vocal emotion. Journal of the Acoustical Society of America 93 (1993) 1097-1108
63. Näätänen R. Attention and brain function (1992), Lawrence Erlbaum, Hillsdale, NJ
64. Nahm F.K., Tranel D., Damasio H., and Damasio A.R. Cross-modal associations and the hyman amygdala. Neurospychologia 31 8 (1993) 727-744
65. Piaget J. The origins of intelligence in children (1952), International Universities Press, New York
66. Pollack I., Rubenstein H., and Horowitz A. Communication of verbal modes of expression. Language and Speech 3 (1960) 121-130
67. (submitted manuscript). Pourtois G., de Gelder B., Vroomen J., Rossion B., Guerit J.M., and Crommelinck M. Multimodal perception of emotion: an event-related potentials study (1999)
68. Pylyshyn, Z. (in press). Is vision continuous with cognition? The case for cognitive impenetrability of visual perception. Behavioral and Brain Sciences.
69. Sams M., Aulanko R., Hämäläinen M., Hari R., Lounasmaa O.V., Lu S.T., and Simola J. Seeing speech: visual information from lip movements modifies activity in the human auditory cortex. Neuroscience Letters 127 (1991) 141-145
70. Scherer K.R. Non-linguistic vocal indicators of emotions and psychopathology. In: Izard C.E. (Ed). Emotions in personality and psychopathology (1979), Plenum Press, New York
71. Scott S.K., Young A.W., Calder A.J., Hellawell D.J., et al. Impaired auditory recognition of fear and anger following bilateral amygda la lesions. Nature 385 6613 (1997) 254-275
72. Searcy J.H., and Bartlett J.C. Inversion and processing of component and spatial-relational information in faces. Journal of Experimental Psychology: Human Perception and Performance 22 4 (1996) 904-915
73. Stein B.E., and Meredith M.A. The merging of the senses (1993), MIT Press, Cambridge, Mass
74. Suberi M., and McKeever W.F. Differential right hemispheric memory storage of emotional and non-emotional faces. Neuropsychologia 15 6 (1977) 757-768
75. Summerfield, Q. (1992). Towards understanding audiovisual speech recognition. In G. Mattingly & M. Studdert-Kennedy (Eds.), Modularity and the motor theory of speech perception (pp. 117-137).
76. Tartter V., and Braun D. Hearing smiles and frowns in normal and whisper registers. Journal of the Acoustical Society of America 96 (1994) 2101-2107
77. Teunisse, J. P., & de Gelder, B. (in press). Impaired categorical expression of emotions in autistics. Child Neuropsychology.
78. van Lancker D. Rags to riches: Our increasing appreciation of cognitive and communicative abilities of the human right hemisphere. Brain and Language 57 (1997) 1-11
79. van Lancker D., and Sidtis J.J. The identification of affective-prososic stimuli by left-and right-hemisphere-damaged subjects: All errors are not created equal. Journal of Speech and Hearing Research 35 5 (1992) 963-970
80. Vroomen J., Collier R., and Mozziconacci S. Duration and intonation in emotional speech. Proceedings of the Third European Conference on Speech Communication and Technology. Berlin (1993) 577-580
81. Vroomen J., and de Gelder B. Phoneme detection in resyllabified word. The Journal of the Acoustical Society 100 4, Pt. 2 (1996) 2818-2819
82. Walker A., and Grolnick W. Discrimination of vocal expressions by young infants. Infant Behavior and Development 6 (1983) 491-498
83. Weiskrantz L. Consciousness Lost and Found (1997), Oxford University Press, Oxford
84. (this volume). Welch R.B. Meaning, attention and the "Unity Assumption" in the intersensory bias of spatial and temporal perceptions. In: Aschersleben G., Bachmann T., and Müsseler J. (Eds). Cognitive Contributions to the Perception of Spatial and Temporal Events (1999), Elsevier, Amsterdam
85. Werner H. Comparative psychology of mental development (1973), International Universities Press, New-York
86. Williams C.E., and Stevens K.N. Emotions and speech: some acoustic correlates. Journal of the Acoustical Society of America 52 (1972) 1238-1250
87. Yin R.K. Looking at upside-down faces. Journal of Experimental Psychology 81 (1969) 141-145