Facial expression and prosodic prominence: Effects of modality and facial area

Journal of Phonetics

Volumn 36, Issue 2, 2008, Pages 219-238

  Swerts, Marc ^a   Krahmer, Emiel ^a  

^a TILBURG UNIVERSITY   (Netherlands)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 42649101998     PISSN: 00954470     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.wocn.2007.05.001     Document Type: Article

References (58)

1. Baum S., Pell M., Leonard C., and Gordon J. The ability of right- and left-hemisphere-damaged individuals to produce and interpret prosodic cues marking phrasal boundaries. Language and Speech 40 (1997) 313-330
2. Bertelson P., Vroomen J., de Gelder B., and Driver J. The ventriloquist effect does not depend on the direction of deliberate visual attention. Perception & Psychophysics 62 (2000) 321-332
3. Beskow, J., Granström, B., & House, D. (2006). Visual correlates to prominence in several expressive modes. Proceedings of Interspeech (pp. 1272-1275), Pittsburgh, PA.
4. Borod J.C., Andelma F., Oble L.K., Tweed J.R., and Welkowit J. Right hemisphere specialization for the appreciation of emotional words and sentences: Evidence from stroke patients. Neuropsychologia 30 (1992) 827-844
5. Borod J.C., Koff E., Yecker S., Santschi C., and Schmidt J.M. Facial asymmetry during emotional expression: Gender, valence, and measurement technique. Neuropsychologica 11 (1998) 1209-1215
6. Butler S., Gilchrist I.D., Burt D.M., Perrett D.I., Jones E., and Harvey M. Are the perceptual biases found in chimeric face processing reflected in eye-movement patterns?. Neuropshychologia 44 (2004) 52-59
7. Cassell, J., Vihjálmsson, H., & Bickmore, T. (2001). BEAT: The behavior expression animation toolkit. Proceedings of SIGGRAPH01 (pp. 477-486).
8. Cavé, C., Guai{dotless}̈tella, I., Bertrand, R., Santi, S., Harlay, F., & Espesser, R. (1996). About the relationship between eyebrow movements and F0 variations. Proceedings of the ICSLP (pp. 2175-2179), Philadelphia.
9. Cho T., and McQueen J. Prosodic influences on consonant production in Dutch: Effects of prosodic boundaries phrasal accent and lexical stress. Journal of Phonetics 33 (2005) 121-157
10. Cruttenden A. Intonation (1986), Cambridge University Press, Cambridge
11. Dahan D., Tanenhaus M.K., and Chambers C.G. Accent and reference resolution in spoken-language comprehension. Journal of Memory and Language 47 (2002) 292-314
12. Davis, C. & Kim, J. (2007). Audio-visual speech perception off the topic of the head. Cognition, 100, B21-B31.
13. Dijkstra, C., Krahmer, E., & Swerts, M. (2006). Manipulating uncertainty-the contribution of different audiovisual prosodic cues to the perception of confidence. Proceedings of the Speech Prosody 2006, Dresden, Germany, May 2006.
14. Dik S.C. Functional grammar (1978), North-Holland, Amsterdam
15. Dohen, M. (2005). Deixis prosodique multisensorielle: Production et perception audiovisuelle de la focalisation contrastive on français. PhD thesis, Institut National Polytechnique de Grenoble (INPG), Grenoble.
16. Dohen M., Lœvenbruck H., Cathiard M.-A., and Schwartz J.-L. Visual perception of contrastive focus in reiterant French speech. Speech Communication 44 (2004) 155-172
17. Ekman P. About brows: Emotional and conversational signals. In: van Cranach M., et al. (Ed). Human ethology (1979), Cambridge University Press, Cambridge 169-202
18. Erber N.P. Effects of angle, distance, and illumination on the visual reception of speech by profoundly deaf children. Journal of Speech and Hearing Research 17 (1974) 99-112
19. Erickson D., Fujimura O., and Pardo B. Articulatory correlates of prosodic control: Emotion and emphasis. Language and Speech 3-4 (1998) 399-417
20. de Gelder B., Vroomen J.H.M., and Bertelson P. The role of face parts: The perception of emotions in the voice and face. In: Grim Cabral L., and Morais J. (Eds). Investigando a linguagem (1999), Mulheres, Florianópolis 262-266
21. Graf, H. P., Cosatto, E., Strom, V., & Huang, F. J. (2002). Visual prosody: Facial movements accompanying speech. Proceedings of the IEEE international conference on automatic face and gesture recognition (pp. 381-386), Washington, DC.
22. Granström, B., House, D., & Lundeberg, M. (1999). Prosodic cues to multimodal speech perception. Proceedings of the 14th ICPhS, San Francisco.
23. Gussenhoven C., Repp B.H., Rietveld A., Rump H.H., and Terken J. The perceptual prominence of fundamental frequency peaks. Journal of the Acoustical Society of America 102 (1997) 3009-3022
24. Hess U., Kappas A., and Scherer K. Multichannel communication of emotion: Synthetic signal production. In: Scherer K. (Ed). Facets of emotion: Recent research (1988), Erlbaum, Hillsdale, NJ 161-182
25. House, D., Beskow, J., & Granström, B. (2001). Timing and interaction of visual cues for prominence in audiovisual speech perception. Proceedings of Eurospeech 2001 (pp. 1957-1960), Denver, CO.
26. Janzen E.K. A balanced smile-a most important treatment objective. American Journal of Orthodontics 72 (1977) 359-372
27. Joanette Y., Goulet P., and Hannequin D. Right hemisphere and verbal communication (1990), Springer, New York
28. Jordan T., and Sergeant P. Effects of distance on visual and audio-visual speech recognition. Language and Speech 43 1 (2000) 107-124
29. Keating, P., Baroni, M., Mattys, S., Scarborough, R., Alwan, A., Auer, E., et al. (2003). Optical phonetics and visual perception of lexical and phrasal stress in English. In Proceedings of the international conference of phonetic sciences (ICPhS) (pp. 2071-2074), Barcelona, Spain.
30. Kohlrausch, A., & van de Par, S. (1999). Audio-visual interaction: From fundamental research in cognitive psychology to (possible) applications. In Proceedings of the SPIE (Vol. 3644, pp. 34-44).
31. Kowatari Y., Yamamoto M., Takahashi T., Kansaku K., Kitazawa S., Ueno S., et al. Dominance of the left oblique view in activating the cortical network for face recognition. Neuroscience Research 50 (2004) 475-480
32. Krahmer, E., Ruttkay, Zs., Swerts, M., & Wesselink, W. (2002). Pitch, eyebrows, and the perception of focus. Proceedings of speech prosody 2002 (pp. 443-446), Aix-en-Provence.
33. Krahmer, E. & Swerts, M. (2001). On the alleged existence of contrastive accent. Speech communication, 34, 391-405
34. Krahmer E., and Swerts M. More about brows. In: Ruttkay Zs., and Pelachaud C. (Eds). Evaluating ECAs (2004), Kluwer Academic Press, Dordrecht
35. Krahmer, E. & Swerts, M. (in press). The effects of visual beats on prosodic prominence: Acoustic analyses, auditory perception and visual perception. Journal of Memory and Language, accepted for publication.
36. Ladd D.R. Intonational phonology (1996), Cambridge University Press, Cambridge
37. Lansing C.R., and McConkie G.W. Attention to facial regions in segmental and prosodic visual speech perception tasks. Journal of Speech and Hearing Research 42 (1999) 526-539
38. Massaro D. Perceiving talking faces: From speech perception to a behavioural principle (1998), MIT, Cambridge
39. Massaro D. Multimodal speech perception: A paradigm for speech science. In: Granström B., House D., and Karlsson I. (Eds). Multimodality in language and speech systems (2002), Kluwer, Dordrecht 45-71
40. Massaro D., Cohen M., and Smeele P. Perception of asynchronous and conflicting visual and auditory speech. Journal of the Acoustical Society of America 100 (1996) 1777-1786
41. Massaro D., and Egan P. Perceiving affect from the voice and the face. Psychonomic Bulletin and Review 3 (1996) 215-221
42. McGurk H., and MacDonald J. Hearing lips and seeing voices. Nature 264 (1976) 746-748
43. Mehrabian A., and Ferris S. Inference of attitudes from nonverbal communication in two channels. Journal of Consulting Psychology 31 (1967) 248-252
44. Mertens I., Siegmund H., and Grüsser O.-J. Gaze motor asymmetries in the perception of faces during a memory task. Neuropsychologia 31 (1993) 989-998
45. Munhall K.G., Jones J.A., Callan D.E., Kuratate T., and Vatikiotis-Bateson E. Visual prosody and speech intelligibility. Head movement improves auditory speech perception. Psychological Science 15 (2004) 133-137
46. Munhall K.G., and Vatikiotis-Bateson E. The moving face during speech communication. In: Campbell R., Dodd B., and Burnham D. (Eds). Hearing by eye II. Advances in the psychology of speechreading and auditory-visual speech (1996), Psychology Press, London
47. Nicholls M.E.R., Searle D., and Bradshaw J.L. Read my lips: Asymmetries in the visual expression and perception of speech revealed through the McGurk effect. Psychological Science 15 (2004) 138-141
48. Pelachaud C., Badler N., and Steedman M. Generating facial expressions for speech. Cognitive Science 20 (1996) 1-46
49. Pourtois G., Debatisse D., Despland P.A., and de Gelder B. Facial expressions modulate the time course of long latency auditory brain potentials. Cognitive Brain Research 14 (2002) 99-105
50. Richardson C.K., Bowers D., Bauer R.M., Heilman K.M., and Leonard C.M. Digitizing the moving face during dynamic displays of emotion. Neuropsychologia 38 (2000) 1028-1039
51. Sakamoto, S., Tanaka, A., Tsumura, K., & Suzuki, Y. (2007). Effect of speed difference between time-expanded speech and talker's moving image on word or sentence intelligibility. Proceedings AVSP conference, Hilvarenbeek, The Netherlands, September 2007.
52. Srinivasan R., and Massaro D. Perceiving prosody from the face and voice: Distinguishing statements from echoic questions in English. Language and Speech 46 (2003) 1-22
53. Swerts, M., & Krahmer, E. (2004). Congruent and incongruent audiovisual cues to prominence. Proceedings of speech prosody 2004, Nara, Japan.
54. Terken J., and Nooteboom S. Opposite effects of accentuation and deaccentuation on verification latencies for given and new information. Language and Cognitive Processes 2 (1987) 145-163
55. Thompson L.A., Malmberg J., Goodell N.K., and Boring R.L. The distribution of attention across a talker's face. Discourse Processes 38 1 (2004) 145-168
56. Walker A., and Grolnick W. Discrimination of vocal expressions by young infants. Infant Behavior and Development 6 (1983) 491-498
57. Walker J.P., Daigle T., and Buzzard M. Hemispheric specialization in processing prosodic structures: Re-visited. Aphasiology 16 (2002) 1155-1172
58. Yehia H.C., Kuratate T., and Vatikiotis-Bateson E. Linking facial animation, head motion and speech acoustics. Journal of Phonetics 30 (2002) 555-568