Visual prosody of newsreaders: Effects of information structure, emotional content and intended audience on facial expressions

Journal of Phonetics

Volumn 38, Issue 2, 2010, Pages 197-206

  Swerts, Marc ^a   Krahmer, Emiel ^a  

^a TILBURG UNIVERSITY   (Netherlands)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 77952545677     PISSN: 00954470     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.wocn.2009.10.002     Document Type: Article

References (39)

1. Barkhuysen P.N., Krahmer E., Swerts M. Problem detection in human-machine interactions based on facial expressions of users. Speech Communication 2005, 45:343-359.
2. Beckman, M., & Ayers Elam, G. (1997). Guidelines for ToBI labelling. The Ohio State University Research Foundation, Ohio State University.
3. Beskow, J., Granström, B., & House, D. (2006). Visual correlates to prominence in several expressive modes. In Proceedings of interspeech, Pittsburgh, PA (pp. 1272-1275).
4. Burnham D., Kitamura C., Vollmer-Conna U. What's new pussycat? On talking to babies and animals. Science 2002, 296:1435.
5. Cassell, J., Vihjálmsson, H., & Bickmore, T. (2001). BEAT: The behavior expression animation toolkit. In Proceedings of SIGGRAPH'01 (pp. 477-486).
6. Graf, H. P., Cosatto, E., Strom, V., & Huang, F. J. (2002). Visual prosody: Facial movements accompanying speech. In Proceedings of the fifth IEEE international conference on automatic face and gesture recognition (FGR 02) (pp. 396-401).
7. Cavé, C., Guaïtella, I., Bertrand, R., Santi, S., Harlay, F., & Espesser, R. (1996). About the relationship between eyebrow movements and F0 variations. In Proceedings of ICSLP, Philadelphia (pp. 2175-2179).
8. Chafe W.L. Givenness, contrastiveness, definiteness, subjects, topics, and point of view. Subject and topic 1976, Academic Press, New York. C.N. Li (Ed.).
9. Clark H.H. Arenas of language use 1992, University of Chicago Press, Chicago.
10. Dohen, M. (2005). Deixis prosodique multisensorielle: Production et perception audiovisuelle de la focalisation contrastive on français. Ph.D. thesis, Institut National Polytechnique de Grenoble (INPG), Grenoble.
11. Dohen M., Loevenbruck H. Interaction of audition and vision for the perception of prosodic contrastive focus. Language and Speech 2009, 52:177-206.
12. Dohen M., Loevenbruck H., Cathiard M.-A., Schwartz J.-L. Visual perception of contrastive focus in reiterant French speech. Speech Communication 2004, 44:155-172.
13. Flecha-Garcia, M. L. (2006a). Eyebrow raising, discourse structure, and utterance function in face-to-face dialogue. In Proceedings of CogSci-2006, Vancouver, Canada (pp. 1311-1316).
14. Flecha-Garcia, M. L. (2006b). Eyebrow raising in dialogue: Discourse structure, utterance function, and pitch accents. Ph.D. thesis, University of Edinburgh, UK.
15. Flecha-Garcia, M. L. (2007). Non-verbal communication in dialogue: Alignment between eyebrow raises and pitch accents in English. In Proceedings of CogSci-2007, Austin, TX, USA (p. 1753).
16. Granström, B., House, D., & Lundeberg, M. (1999). Prosodic cues to multimodal speech perception. In Proceedings of the 14th ICPhS, San Francisco.
17. Gussenhoven C. Transcription of dutch intonation. Prosodic typology: The phonology of intonation and phrasing 2005, 118-145. Oxford University Press, Oxford. S.-A. Jun (Ed.).
18. Hirschberg J., Litman D., Swerts M. Prosodic and other cues to speech recognition failures. Speech Communication 2004, 43:155-175.
19. Jordan T., Sergeant P. Effects of distance on visual and audio-visual speech recognition. Language and Speech 2000, 43(1):107-124.
20. Krahmer, E., Ruttkay, Zs., Swerts, M., & Wesselink, W. (2002). Pitch, eyebrows, and the perception of focus. In Proceedings of the speech prosody 2002, Aix-en-Provence (pp. 443-446).
21. Krahmer E., Swerts M. On the alleged existence of contrastive accents. Speech Communication 2001, 34(4):391-405.
22. Krahmer E., Swerts M. More about brows: A cross-linguistic study via analysis-by-synthesis. Evaluating ECAs 2004, 191-216. Kluwer Academic Press, Dordrecht. Z. Ruttkay, C. Pelachaud (Eds.).
23. Krahmer E., Swerts M. The effects of visual beats on prosodic prominence: Acoustic analyses, auditory perception and visual perception. Journal of Memory and Language 2007, 57:396-414.
24. Ladd D.R. Intonational phonology 1996, Cambridge University Press, Cambridge.
25. Mo, Y., Cole, J., & Lee, E.-K. (2008). Naïve listeners prominence and boundary perception. In Proceedings of the speech prosody conference, Campinas, Brazil.
26. Mol L., Krahmer E.J., Maes A.A., Swerts M. The communicative import of gestures: Evidence from a comparative analysis of human-human and human-machine interactions. Gesture 2009, 9(1):97-126.
27. Munhall K.G., Jones J.A., Callan D.E., Kuratate T., Vatikiotis-Bateson E. Visual prosody and speech intelligibility. Head movement improves auditory speech perception. Psychological Science 2004, 15:133-137.
28. Ostendorf, M., Price, P. J., & Shattuck-Hufnagel, S. (1995). The Boston University radio news corpus. Boston University Technical Report No. ECS-95-001.
29. Oviatt S.L., Bernard J., Levow G. Linguistic adaptation during error resolution with spoken and multimodal systems. Language and Speech 1999, 41:415-438.
30. Pelachaud C., Badler N., Steedman M. Generating facial expressions for speech. Cognitive Science 1996, 20:1-46.
31. Petajan, E. (1985). Automatic Lipreading to Enhance Speech Recognition. In Proceeding of the IEEE conference on computer vision and pattern recognition (pp. 40-47).
32. Scarborough R., Keating P., Mattys S.L., Cho T., Alwan A., Auer E.T. Optical phonetics and visual perception of lexical and phrasal stress in English. Language and Speech 2009, 52:135-175.
33. Shahid S., Krahmer E.J., Swerts M. Alone or together: Exploring the effect of physical co-presence on the emotional expressions of game playing children across cultures. Fun and games. Lecture notes in computer science 2008, Vol. 5294:94-105. Springer, Berlin.
34. Sumby W.H., Pollack I. Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America 1954, 26:212-215.
35. Swerts M. Prosodic features at discourse boundaries of different strength. Journal of the Acoustical Society of America 1997, 101(1):514-521.
36. Swerts, M., & Krahmer, E. J. (2008). Facial expressions and prosodic prominence: Comparing modalities and facial areas. Journal of Phonetics, 36, 219-238.
37. Wang, S., Demirdjian, D., Kjellström, H., & Darrell, T. (2007). Multimodal communication error detection for driver-car interaction. In Proceedings of ICINCO 2007, Angers, France.
38. Wilting, J., Krahmer, E., & Swerts, M. (2006). Real vs. acted emotional speech. In Proceedings of the international conference on spoken language processing (interspeech 2006), Pittsburgh, PA, USA, September 2006.
39. Yasinnik Y., Renwick M., Shattuck-Hufnagel S. The timing of speech-accompanying gestures with respect to prosody. Proceedings from sound to sense: 50+ years of discoveries in speech communication 2004, C97-C102. MIT, Cambridge, MA.