The effects of visual beats on prosodic prominence: Acoustic analyses, auditory perception and visual perception

Journal of Memory and Language

Volumn 57, Issue 3, 2007, Pages 396-414

  Krahmer, Emiel ^a   Swerts, Marc ^a  

^a TILBURG UNIVERSITY   (Netherlands)

Author keywords

Acoustics; Audio visual speech; Beats; Facial expressions; Gestures; Speech perception; Speech production

Indexed keywords

EID: 34548256866     PISSN: 0749596X     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jml.2007.06.005     Document Type: Article

References (81)

1. Alibali M., Heath D., and Myers H. Effects of visibility between speaker and listener on gesture production: Some gestures are meant to be seen. Journal of Memory and Language 44 (2001) 169-188
2. Alibali M., Kita S., and Young A. Gesture and the process of speech production: We think, therefore we gesture. Language and Cognitive Processes 15 (2000) 593-613
3. Barkhuysen P., Krahmer E., and Swerts M. Problem detection in human-machine interactions based on facial expressions of users. Speech Communication 45 (2005) 343-359
4. Bernardis P., and Gentilucci M. Speech and gesture share the same communication system. Neuropsychologia 44 (2006) 178-190
5. Bernstein N. The coordination and regulation of movements (1967), Pergamon, London
6. Birdwhistell R. Kinesics and context (1970), University of Pennsylvania Press
7. Boersma, P. & Weenink, D. (2006). Praat: Doing phonetics by computer (Version 4.5.07), .
8. Bolinger D. Intonation and gesture. American Speech 58 (1983) 156-174
9. Bolinger D. Intonation and its parts (1985), Edward Arnold, London
10. Cassell J., McNeill D., and McCullough K. Speech-gesture mismatches: Evidence for one underlying representation of linguistic and non-linguistic information. Pragmatics and Cognition 7 (1999) 1-33
11. 0 variations. In Proceedings of the International Conference on Spoken Language Processing (ICSLP) (pp. 2175-2179). Philadelphia.
12. Condon W. An analysis of behavioral organization. Sign Language Studies 13 (1976) 285-318
13. Corballis M. On the evolution of language and generativity. Cognition 44 (1992) 197-226
14. Cruttenden A. Intonation. 2nd ed. (1997), Cambridge University Press, Cambridge
15. Cutler A. Stress and accent in language production and understanding. In: Gibbon D., and Richter H. (Eds). Intonation, accent and rhythm studies in discourse phonology (1984), de Gruyter Berlin 77-90
16. de Gelder B., and Vroomen J. The perception of emotions by ear and by eye. Cognition and Emotion 14 (2000) 289-311
17. de Ruiter J.P. The production of gesture and speech. In: McNeill D. (Ed). Language and gesture (2000), Cambridge University Press, Cambridge 284-311
18. Diehl R., Lotto A., and Holt L. Speech perception. Annual Review of Psychology 55 (2004) 149-179
19. Dodd B., and Campbell R. Hearing by eye: The psychology of lip-reading (1987), Lawrence Erlbaum Associates, New Jersey
20. Efron D. Gesture and environment (1941), King's Crown Press, New York
21. Eibl-Eibesfelt I. Similarities and differences between cultures in expressive movements. In: Hinde R. (Ed). Non-verbal communication (1972), Cambridge University Press, Cambridge
22. Ekman P. About brows: Emotional and conversational signals. In: von Cranach M., Foppa K., Lepenies W., and Ploog D. (Eds). Human ethology: Claims and limits of a new discipline (1979), Cambridge University Press, Cambridge 169-202
23. Ekman P., and Friesen W. The repertoire of nonverbal behavioral categories: Origins, usage, and coding. Semiotica 1 (1969) 49-98
24. Fitch W.T. The evolution of speech: A comparative review. Trends in Cognitive Science 4 (2000) 258-267
25. Flanagan R., Feldman A., and Ostry D. Control of human jaw and multi-joint arm movements. In: Hammond G. (Ed). Cerebral control of speech and limb movements (1990), North-Holland, Amsterdam 29-58
26. Flanders M., Helms Tillery S., and Soechting J. Early stages in sensorimotor transformation. Behavioral and Brain Sciences 15 (1992) 309-362
27. Fowler C. Auditory perception is not special: We see the world, we feel the world, we hear the world. Journal of the Acoustical Society of America 88 (1991) 1236-1249
28. Fowler C. Listeners do hear sounds, not tongues. Journal of the Acoustical Society of America 99 (1996) 1730-1741
29. Fowler C., and Dekle D. Listening with eye and hand: Crossmodal contributions to speech perception. Journal of Experimental Psychology: Human Perception and Performance 26 (1991) 877-888
30. Glave R., and Rietveld A. Bimodal cues for speech loudness. Journal of the Acoustical Society of America 66 (1979) 1018-1022
31. Goldin-Meadow S. Hearing gesture: How our hands help us think (2003), Harvard University Press, Cambridge, MA
32. Goldin-Meadow S., and Wagner S. How our hands help us learn. Trends in Cognitive Sciences 9 (2005) 234-240
33. Grant K., and Seitz P. The use of visible speech cues for improving auditory detection of spoken sentences. Journal of the Acoustical Society of America 108 (2000) 1197-1208
34. Gussenhoven C. The phonology of tone and intonation (2004), Cambridge University Press, Cambridge
35. Gussenhoven C., Repp B., Rietveld A., Rump H., and Terken J. The perceptual prominence of fundamental frequency peaks. Journal of the Acoustical Society of America 102 (1997) 3009-3022
36. Hadar U., Steiner T., Grant E., and Rose F. Head movement correlates to juncture and stress at sentence level. Language and Speech 26 (1983) 117-129
37. Hammond G. Cerebral control of speech and limb movements (1990), North-Holland, Amsterdam
38. Hirschberg J., Litman D., and Swerts M. Prosodic and other cues to speech recognition failures. Speech Communication 43 (2004) 155-175
39. Hiscock M., and Chipuer H. Concurrent performance of rhythmically compatible or incompatible vocal and manual tasks: Evidence for two sources of interference in verbal-manual timesharing. Neuropsychologia 24 (1986) 691-698
40. Holden C. The origin of speech. Science 303 (2004) 1316-1319
41. Keating, P., Baroni, M., Mattys, S., Scarborough, R., Alwan, A., & Auer, E. (2003). Optical phonetics and visual perception of lexical and phrasal stress in English. In Proceedings 16th International Conference of the Phonetic Sciences (ICPhS) (pp. 2071-2074). Barcelona, Spain.
42. Kelso J., and Holt K. Exploring a vibratory systems account of human movement production. Journal of Neurophysiology 43 (1980) 1183-1196
43. Kelso J., Tuller B., and Harris K. A "Dynamic Pattern" perspective on the control and coordination of movement. In: MacNeilage P. (Ed). The production of speech (1983), Springer Verlag, New York 137-173
44. Kendon A. Gesticulation and speech: Two aspects of the process of utterance. In: Key M. (Ed). The relationship of verbal and nonverbal communication (1980), Mouton, The Hague 207-227
45. Kendon A. Gesture. Annual Review of Anthropology 26 (1997) 109-128
46. Kita S., and Özyürek A. What does cross-linguistic variation in semantic coordination of speech and gesture reveal?: Evidence for an interface representation of spatial thinking and gesture. Journal of Memory and Language 48 (2003) 16-32
47. Krahmer E., and Swerts M. On the alleged existence of contrastive accents. Speech Communication 34 (2001) 391-405
48. Krahmer E., and Swerts M. More about brows. In: Ruttkay Zs., and Pelachaud C. (Eds). From brows to trust: Evaluating embodied conversational agents (2004), Kluwer Academic Press, Dordrecht 191-216
49. Krahmer E., and Swerts M. How children and adults signal and detect uncertainty in audiovisual speech. Language and Speech 48 (2005) 29-54
50. Krauss R., Chen Y., and Chawla P. Nonverbal behavior and nonverbal communication: What do conversational hand gestures tell us?. In: Zanna M. (Ed). Advances in experimental social psychology (1996), Academic Press, San Diego 389-450
51. Ladd D. Intonational phonology (1996), Cambridge University Press, Cambridge
52. Levelt W. Speaking: From intention to articulation (1989), MIT Press, Cambridge
53. Liberman A. Some results of research on speech perception. Journal of the Acoustical Society of America 29 (1957) 117-123
54. Liberman A., and Mattingly I. The motor theory of speech perception revised. Cognition 21 (1985) 1-36
55. Massaro D., Cohen M., and Smeele P. Perception of asynchronous and conflicting visual and auditory speech. Journal of the Acoustical Society of America 100 (1996) 1777-1786
56. Mayberry R., and Nicoladis E. Gesture reflects language development: Evidence from bilingual children. Current Directions in Psychological Science 9 (2000) 192-196
57. McClave E. Pitch and manual gestures. Journal of Psycholinguistic Research 27 (1998) 69-89
58. McGurk H., and MacDonald J. Hearing lips and seeing voices. Nature 264 (1976) 746-748
59. McNeill D. Hand and mind: What gestures reveal about thought (1992), University of Chicago Press, Chicago
60. McNeill D., and Duncan S.D. Growth points in thinking-for-speaking. In: McNeill D. (Ed). Language and gesture (2000), Cambridge University Press, Cambridge
61. Morgan B. Question melodies in American English. American Speech 2 (1953) 181-191
62. Munhall K., Jones J., Callan D., Kuratate T., and Vatikiotis-Bateson E. Visual prosody and speech intelligibility. Psychological Science 15 (2004) 133-137
63. Özyürek A. Do speakers design their cospeech gestures for their addressees? The effects of addressee location on representational gestures. Journal of Memory and Language 46 (2002) 688-704
64. Pelachaud C., Badler N., and Steedman M. Generating facial expressions for speech. Cognitive Science 20 (1996) 1-46
65. Pierrehumbert J., and Hirschberg J. The meaning of intonational contours in interpretation of discourse. In: Cohen P., Morgan J., and Pollack M. (Eds). Intentions in communication (1990), MIT Press, Cambridge, MA
66. Pikovsky A., Rosenblum M., and Kurths J. Synchronization. A universal concept in nonlinear sciences (2001), Cambridge University Press, Cambridge
67. Rauscher F., Krauss R., and Chen U. Gesture, speech and lexical access: The role of lexical movements in speech production. Psychological Science 7 (1996) 226-231
68. Rump H., and Collier R. Focus conditions and the prominence of pitch accented syllables. Language and Speech 39 (1996) 1-17
69. Saltzman E., and Byrd D. Task-dynamics of gestural timing: Phase windows and multifrequency rhythms. Human Movement Science 19 (2000) 499-526
70. Schwartz J.-L., Berthommier F., and Savariaux C. Seeing to hear better: Evidence for early audio-visual interactions in speech identification. Cognition 93 (2004) B69-B78
71. Srinivasan R., and Massaro D. Perceiving prosody from the face and voice: Distinguishing statements from echoic questions in English. Language and Speech 46 (2003) 1-22
72. Swerts, M. & Krahmer, E. (2007). Facial expressions and prosodic prominence: Comparing modalities and facial areas. Journal of Phonetics, in press.
73. Swerts M., and Krahmer E. Audiovisual prosody and feeling of knowing. Journal of Memory and Language 53 (2005) 81-94
74. Swerts M., Krahmer E., and Avesani C. Prosodic marking of information status in Dutch and Italian: A comparative analysis. Journal of Phonetics 30 (2002) 629-654
75. Terken J., and Nooteboom S. Opposite effects of accentuation and deaccentuation on verification latencies for given and new information. Language and Cognitive Processes 2 (1987) 145-163
76. Tuomainen J., Andersen T., Tiippana K., and Sams M. Audio-visual speech perception is special. Cognition 96 (2005) B13-B22
77. Turvey M. Coordination. American Psychologist 45 (1990) 938-953
78. van Bergem D. Acoustic vowel reduction. Speech Communication 12 (1993) 1-23
79. Vatikiotis-Bateson E., Eigsti I.-M., Yano S., and Munhall K.G. Eye movement of perceivers during audiovisual speech perception. Perception & Psychophysics 60 (1998) 926-940
80. Wagner S., Nusbaum H., and Goldin-Meadow S. Probing the mental representation of gesture: Is handwaving spatial?. Journal of Memory and Language 50 (2004) 395-407
81. Wilson G.B. Three Rs for vocal skill development in the choral rehearsal. Music Educators Journal 77 (1991) 42-46