Prosodic temporal alignment of co-speech gestures to speech facilitates referent resolution

Journal of Experimental Psychology: Human Perception and Performance

Volumn 38, Issue 6, 2012, Pages 1567-1581

  Jesse, Alexandra ^a   Johnson, Elizabeth K ^b  

^a UNIVERSITY OF MASSACHUSETTS   (United States)

^b UNIVERSITY OF TORONTO   (Canada)

Author keywords

Audiovisual perception; Prosody; Referent resolution; Speech; Synchrony

Indexed keywords

ADULT; ARTICLE; ASSOCIATION; ATTENTION; CONCEPT FORMATION; FEMALE; GESTURE; HUMAN; LEARNING; MALE; NETHERLANDS; SEMANTICS; SPEECH PERCEPTION; TIME; VISION;

ADULT; ATTENTION; CONCEPT FORMATION; CUES; FEMALE; GESTURES; HUMANS; MALE; NETHERLANDS; SEMANTICS; SPEECH PERCEPTION; TIME FACTORS; VERBAL LEARNING; VISUAL PERCEPTION; YOUNG ADULT;

EID: 84874734807     PISSN: 00961523     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0027921     Document Type: Article

References (70)

1. Bahrick, L. E., & Lickliter, R. (2003). Intersensory redundancy guides early perceptual and cognitive development. Advances in Child Development and Behavior, 30, 153-187. doi:10.1016/S0065-2407(02) 80041-6
2. Baldwin, D. A. (1991). Infants' contribution to the achievement of joint attention. Child Development, 62, 875-890. doi:10.2307/1131140
3. Baldwin, D. A., Markman, E. M., Bill, B., Desjardins, R. N., Irwin, J. M., & Tidball, G. (1996). Infants' reliance on a social criterion for establishing word-object relations. Child Development, 67, 3135-3153. doi: 10.2307/1131771
4. Berman, J. M. J., Chambers, C. G., & Graham, S. A. (2010). Preschoolers' appreciation of speaker vocal affect as a cue to referential intent. Journal of Experimental Child Psychology, 107, 87-99. doi:10.1016/ j.jecp.2010.04.012
5. Bernal, S., Lidz, J., Millotte, S., & Christophe, A. (2007). Syntax constrains the acquisition of verb meaning. Language Learning and Development, 3, 325-341. doi:10.1080/15475440701542609
6. Bernstein, L. E., Eberhardt, S. P., & Demorest, M. E. (1989). Singlechannel vibrotactile supplements to visual perception of intonation and stress. Journal of the Acoustical Society of America, 85, 397-405. doi:10.1121/1.397690
7. Bloomfield, L. (1935/1976). Language (13th ed.). London, UK: Allen & Unwin.
8. Boersma, P., & Weenink, D. (2005). PRAAT: Doing phonetics by computer [Computer program]. Version 5.1.19. Retrieved October 21, 2009, from http://www.praat.org
9. Brand, R. J., Baldwin, D. A., & Ashburn, L. A. (2002). Evidence for "motionese": Modifications in mothers' infant-directed action. Developmental Science, 5, 72-83. doi:10.1111/1467-7687.00211
10. Brand, R. J., Shallcross, W. L., Sabatos, M. G., & Massie, K. P. (2007). Fine-grained analysis of motionese: Eye gaze, object exchanges, and action units in infantversus adult-directed action. Infancy, 11, 203-214. doi:10.1111/j.1532-7078.2007.tb00223.x
11. Brooks, R., & Meltzoff, A. (2005). The development of gaze following and its relation to language. Developmental Science, 8, 535-543. doi: 10.1111/j.1467-7687.2005.00445.x
12. Bull, P., & Connelly, G. (1985). Body movement and emphasis in speech. Journal of Nonverbal Behavior, 9, 169-187. doi:10.1007/BF01000738
13. Cave, C., Guaitella, I., Bertrand, R., Santi, S., Harley, F., & Espesser, R. (1996). About the relationship between eyebrow movements and F0 variations. Paper presented at the ICSLP Fourth International Conference on Spoken Language. Philadelphia, PA, October 6, 1996.
14. Clark, E. V. (1990). On the pragmatics of contrast. Journal of Child Language, 17, 417-431. doi:10.1017/S0305000900013842
15. Collis, G. M., & Schafer, H. R. (1975). Synchronization of visual attention in mother-infant pairs. Journal of Child Psychology and Psychiatry, 16, 315-320. doi:10.1111/j.1469-7610.1975.tb00365.x
16. Condon, W. S. (1976). An analysis of behavioral organization. Sign Language Studies, 13, 285-318.
17. Condon, W. S., & Ogston, M. B. (1966). Sound film analysis of normal and pathological behavior patterns. Journal of Nervous and Mental Disease, 143, 338-347. doi:10.1097/00005053-196610000-00005
18. Cutler, A., & Otake, T. (1994). Mora or phoneme? Further evidence for language-specific listening. Journal of Memory and Language, 33, 824-844. doi:10.1006/jmla.1994.1039
19. De Saussure, F. (1915/1966). Course in general linguistics. New York, NY: McGraw-Hill.
20. Diesendruck, G., & Markson, L. (2001). Children's avoidance of lexical overlap: A pragmatic account. Developmental Psychology, 37, 630-641. doi:10.1037/0012-1649.37.5.630
21. Dittmann, A. T. (1972). The body movement-speech rhythm relationship as a cue to speech encoding. In A. W. Siegman & B. Pope (Eds.), Studies in dyadic communication (pp. 131-151). Elmsford, NY: Pergamon Press.
22. Dohen, M., Loevenbruck, H., Cathiard, M.-A., & Schwartz, J.-L. (2004). Visual perception of contrastive focus in reiterant French speech. Speech Communication, 44, 155-172. doi:10.1016/j.specom.2004.10.009
23. Efron, D. (1941). Gesture and environment. New York, NY: Kings Crown Press.
24. Ekman, P., & Friesen, W. V. (1969). The repertoire of nonverbal behavior: Categories, origins, usage and coding. Semiotica, 1, 49-98.
25. Fernald, A., & Simon, T. (1984). Expanded intonation contours in mothers' speech to newborns. Developmental Psychology, 20, 104-113. doi: 10.1037/0012-1649.20.1.104
26. Feyereisen, P., Van de Wiele, M., & Dubois, F. (1988). The meaning of gestures: What can be understood without speech? Cahiers de Psychologie Cognitive/Current Psychology of Cognition, 8, 3-25.
27. Freedman, N., & Hoffman, S. (1967). Kinetic behavior in altered clinical states: Approach to objective analysis of motor behavior during clinical interviews. Perceptual and Motor Skills, 24, 527-539. doi:10.2466/ pms.1967.24.2.527
28. Gogate, L. J., & Bahrick, L. E. (1998). Intersensory redundancy facilitates learning of arbitrary relations between vowel sounds and objects in seven-month-old infants. Journal of Experimental Child Psychology, 69, 133-149.
29. Gogate, L. J., & Bahrick, L. E. (2001). Intersensory redundancy and 7-month-old infants' memory for arbitrary syllable-object relations. Infancy, 2, 219-231. doi:10.1207/S15327078IN0202_7
30. Gogate, L. J., Bahrick, L. E., & Watson, J. D. (2000). A study of multimodal motherese: The role of temporal synchrony between verbal labels and gestures. Child Development, 71, 878-894. doi:10.1111/1467-8624.00197
31. Gogate, L. J., Bolzani, L. H., & Betancourt, E. A. (2006). Attention to maternal multimodal naming by 6to 8-month-old infants and learning of word-object relations. Infancy, 9, 259-288. doi:10.1207/ s15327078in0903_1
32. Gogate, L. J., Prince, C. G., & Matatyaho, D. J. (2009). Two-month-old infants' sensitivity to changes in arbitrary syllable-object pairings: The role of temporal synchrony. Journal of Experimental Psychology: Human Perception and Performance, 35, 508-519. doi:10.1037/a0013623
33. Golinkoff, R. M., Hirsh-Pasek, K., Bailey, L. M., & Wenger, N. R. (1992). Young children and adults use lexical principles to learn new nouns. Developmental Psychology, 28, 99-108. doi:10.1037/0012-1649.28.1.99
34. Granström, B., & House, D. (2005). Audiovisual representation of prosody in expressive speech communication. Speech Communication, 46, 473-484. doi:10.1016/j.specom.2005.02.017
35. Grant, K. W., & Walden, B. E. (1996). Spectral distribution of prosodic information. Journal of Speech & Hearing Research, 39, 228-238.
36. Hadar, U., Steiner, T. J., & Clifford Rose, F. (1984). The relationships between head movement and speech dysfluencies. Language and Speech, 27, 333-342.
37. Hadar, U., Steiner, T. J., Grant, E. C., & Rose, F. C. (1983). Head movement correlates of juncture and stress at sentence level. Language and Speech, 26, 117-129.
38. Hadar, U., Steiner, T. J., Grant, E. C., & Rose, F. C. (1984). The timing of shifts of head postures during conversation. Human Movement Science, 3, 237-245. doi:10.1016/0167-9457(84)90018-6
39. Harris, M., Jones, D., & Grant, J. (1983). The nonverbal context of mothers' speech to infants. First Language, 4, 21-30. doi:10.1177/ 014272378300401003
40. Herold, D. S., Nygaard, L. C., Chicos, K. A., & Namy, L. L. (2011). The developing role in prosody in novel word interpretation. Journal of Experimental Child Psychology, 108, 229-241. doi:10.1016/j.jecp .2010.09.005
41. Hirose, N. (2002). An ecological approach to embodiment and cognition. Cognitive Systems Research, 3, 289-299. doi:10.1016/S1389-0417(02)00044-X
42. Hirotani, M., Stets, M., Striano, T., & Friederici, A. D. (2009). Joint attention helps infants learn new words: Event-related potential evidence. NeuroReport, 20, 600-605. doi:10.1097/WNR.0b013e32832a0a7c
43. Hockett, C. F. (1960). The origin of speech. Scientific American, 203, 88-96. doi:10.1038/scientificamerican0960-88
44. Hollich, G. J., Hirsh-Pasek, K., & Golinkoff, R. (2000). Breaking the language barrier: An emergentist coalition model for the origins of word learning. Monographs of the Society for Research in Child Development, 65, i-vi, 1-123.
45. Jacobson, J. L., Boersma, D. C., Fields, R. B., & Olson, K. L. (1983). Paralinguistic features of adult speech to infants and small children. Child Development, 54, 436-442. doi:10.2307/1129704
46. Jolly, H. R., & Plunkett, K. (2008). Inflectional bootstrapping in 2-yearolds. Language and Speech, 51, 45-59. doi:10.1177/ 00238309080510010401
47. Kendon, A. (1972). Some relationships between body motion and speech. In A. Seigman & B. Pope (Eds.), Studies in dyadic communication (pp. 177-216). Elmsford, NY: Pergamon Press.
48. Kendon, A. (1980). Gesticulation and speech: Two aspects of the process of utterance. In M. R. Key (Ed.), The relation between verbal and nonverbal communication (pp. 207-227). The Hague, The Netherlands: Morton.
49. Kidd, C., White, K. S., & Aslin, R. N. (2011). Toddlers use speech disfluencies to predict speakers' referential intentions. Developmental Science, 14, 925-934. doi:10.1111/j.1467-7687.2011.01049.x
50. Krahmer, E., & Swerts, M. (2007). The effects of visual beats on prosodic prominence: Acoustic analyses, auditory perception and visual perception. Journal of Memory and Language, 57, 396-414. doi:10.1016/ j.jml.2007.06.005
51. Levelt, W. J. M., Richardson, G., & La Heij, W. (1985). Pointing and voicing in deictic expressions. Journal of Memory and Language, 24, 133-164. doi:10.1016/0749-596X(85)90021-X
52. Markman, E. M. (1990). Constraints children place on word meanings. Cognitive Science: A Multidisciplinary Journal, 14, 55-77. doi:10.1207/ s15516709cog1401_4
53. Masur, E. F. (1982). Mothers' responses to infants' object-related gestures: Influences on lexical development. Journal of Child Language, 9, 23-30. doi:10.1017/S0305000900003585
54. McClave, E. (1994). Gestural beats: The rhythm hypothesis. Journal of Psycholinguistic Research, 23, 45-66. doi:10.1007/BF02143175
55. McNeill, D. (2000). Language and gesture: Window into thought and action. Cambridge, UK: Cambridge University Press.
56. Messer, D. J. (1978). The integration of mothers' referential speech with joint play. Child Development, 49, 781-787. doi:10.2307/1128248
57. Munhall, K. G., Jones, J. A., Callan, D. E., Kuratate, T., & VatikiotisBateson, E. (2004). Visual prosody and speech intelligibility: Head movement improves auditory speech perception. Psychological Science, 15, 133-137. doi:10.1111/j.0963-7214.2004.01502010.x
58. Nygaard, L. C., Herold, D. S., & Namy, L. L. (2009). The semantics of prosody: Acoustic and perceptual evidence of prosodic correlates to word meaning. Cognitive Science: A Multidisciplinary Journal, 33, 127-146. doi:10.1111/j.1551-6709.2008.01007.x
59. Nygaard, L. C., & Lunders, E. R. (2002). Resolution of lexical ambiguity by emotional tone of voice. Memory & Cognition, 30, 583-593. doi: 10.3758/BF03194959
60. Nygaard, L. C., & Queen, J. S. (2008). Communicating emotion: Linking affective prosody and word meaning. Journal of Experimental Psychology: Human Perception and Performance, 34, 1017-1030. doi:10.1037/ 0096-1523.34.4.1017
61. Otake, T., Hatano, G., Cutler, A., & Mehler, J. (1993). Mora or syllable? Speech segmentation in Japanese. Journal of Memory and Language, 32, 258-278. doi:10.1006/jmla.1993.1014
62. Parault, S. J., & Schwanenflugel, P. J. (2006). Sound-symbolism: A piece in the puzzle of word learning. Journal of Psycholinguistic Research, 35, 329-351. doi:10.1007/s10936-006-9018-7
63. Pruden, S. M., Hirsh-Pasek, K., Golinkoff, R., & Hennon, E. A. (2006). The birth of words: Ten-month-olds learn words through perceptual salience. Child Development, 77, 266-280. doi:10.1111/j.1467-8624.2006.00869.x
64. Risberg, A., & Lubker, J. (1978). Prosody and speech-reading. Speech Transmission Laboratory Quarterly Progress Status Report, 4, 1-16.
65. Saberi, K., & Perrott, D. R. (1999). Cognitive restoration of reversed speech. Nature, 398, 760. doi:10.1038/19652
66. Scarborough, R., Keating, P. A., Mattys, S. L., Cho, T., & Alwan, A. (2009). Optical phonetics and visual perception of lexical and phrasal stress in English. Language and Speech, 52, 135-175. doi:10.1177/ 0023830909103165
67. Shintel, H., Nusbaum, H. C., & Okrent, A. (2006). Analog acoustic expression in speech communication. Journal of Memory and Language, 55, 167-177. doi:10.1016/j.jml.2006.03.002
68. Smith, L., & Yu, C. (2008). Infants rapidly learn word-referent mappings via cross-situational statistics. Cognition, 106, 1558-1568. doi:10.1016/j.cognition.2007.06.010
69. Thompson, D. M. (1934). On the detection of emphasis in spoken sentences by means of visual, tactual, and visual-tactual cues. Journal of General Psychology, 11, 160-172. doi:10.1080/00221309.1934.9917823
70. Yehia, H. C., Kuratate, T., & Vatikiotis-Bateson, E. (2002). Linking facial animation, head motion and speech acoustics. Journal of Phonetics, 30, 555-568. doi:10.1006/jpho.2002.0165