Effects of perturbation and prosody on the coordination of speech and gesture

Speech Communication

Volumn 57, Issue , 2014, Pages 283-300

  Rusiewicz, Heather Leavy ^a   Shaiman, Susan ^b   Iverson, Jana M ^c   Szuminsky, Neil ^b  

^a DUQUESNE UNIVERSITY   (United States)

^b UNIVERSITY OF PITTSBURGH   (United States)

^c UNIVERSITY OF PITTSBURGH SCHOOL OF MEDICINE   (United States)

Author keywords

Capacitance; Entrainment; Gesture; Perturbation; Prosody; Speech hand coordination

Indexed keywords

DELAYED AUDITORY FEEDBACKS; EXPERIMENTAL RESEARCH; GESTURE; PERTURBATION; PROSODY; SPEECH-HAND COORDINATION; TEMPORAL COORDINATION; TEMPORAL RELATIONSHIPS;

AIR ENTRAINMENT; CAPACITANCE; COMPUTER APPLICATIONS; COMPUTER SIMULATION;

SPEECH;

EID: 84888306609     PISSN: 01676393     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.specom.2013.06.004     Document Type: Article

References (93)

1. J.H. Abbs, J.W. Folkins, and M. Sivarajan Motor impairment following blockade of the infraorbital nerve: Implications for the use of anesthetization techniques in speech research J. Speech Hear. Res. 19 1976 19 35
2. J.H. Abbs, and V.L. Gracco Evidence for speech muscle functional compartmentalization: Theoretical and methodological implications J. Acoust. Soc. Am. 71 1982 S33 S34
3. J.H. Abbs, and V.L. Gracco Control of complex motor gestures: orofacial muscle responses to lo (Pubitemid 14170132)
4. P.A. Barbosa, Explaining cross-linguistic rhythmic variability via coupled-oscillator model of rhythm production, Paper presented at the Speech Prosody, Aix-en-Provence, France, 2002.
5. C. Bard, Y. Turrell, M. Fleury, N. Teasdale, Y. Lamarre, and O. Martin Deafferentation and pointing with visual double-step perturbations Exp. Brain Res. 125 1999 410 416 (Pubitemid 29178675)
6. R.L. Birdwhistell Introduction to Kinesics: An Annotation System for Analysis of Body Motion and Gesture 1952 University of Louisville Press Louisville
7. R.L. Birdwhistell Kinesics and Context: Essays on Body Motion Communication 1970 University of Pennsylvania Press Philadelphia
8. A.C. Bluedorn The Human Organization of Time: Temporal Realities and Experience 2002 Stanford University Press Stanford, CA
9. P. Bull, and G. Connelly Body movement and emphasis in speech J. Nonverbal Behav. 9 1985 169 187
10. B.D. Burke Susceptibility to delayed auditory feedback and dependence on auditory or oral sensory feedback J. Com. Dis. 8 1975 75 96
11. P. Chang, and G.R. Hammond Mutual interactions between speech and finger movements J. Motor Behav. 19 1987 265 275
12. M. Clayton, R. Sager, and U. Will In time with the music: the concept of entrainment and its significance for ethnomusicology ESEM Counterpoint 1 2004 1 82
13. W.S. Condon, and W.D. Ogston Sound film analysis of normal and pathological behavior patterns J. Nerv. Mental Dis. 143 1966 338 347
14. K.H. Corriveau, and U. Goswami Rhythmic motor entrainment in children with speech and language impairments: tapping to the beat Cortex 45 2009 119 130
15. Cummins, F., Port, R.F., 1996. Rhythmic commonalities between hand gestures and speech. In: Proceedings of the Eighteenth Meeting of the Cognitive Science Society, Lawrence Erlbaum, London, pp. 415-419.
16. F. Cummins, and R.F. Port Rhythmic constraints on stress timing in English J. Phonetic 26 1998 145 171 (Pubitemid 128179514)
17. A. Cutler, and D.M. Carter The predominance of strong initial syllables in the English vocabulary Comput. Speech Lang. 2 1987 133 142
18. J.P. de Ruiter Gesture and Speech Production, Unpublished Doctoral Dissertation 1998 Katholieke Universiteit Nijmegen, Germany
19. J.P. de Ruiter The production of gesture and speech D. McNeill, Language and Gesture 2000 Cambridge University Press Cambridge, UK 284 311
20. A.T. Dittman, and L.G. Llewellyn Body movement and speech rhythm in social conversation J. Pers. Soc. Psychol. 11 1969 98 106
21. J.L. Elman Oral vs. manual tapping with delayed auditory feedback as measures of cerebral dominance J. Speech Lang. Hear. Res. 26 1983 106 110 (Pubitemid 13103665)
22. Esteve-Gibert, N., Prieto, P., 2011. Temporal alignment of gesture and speech in Catalan-babbling children. In: Proceedings from the 2nd Gesture and Speech in Interaction (GESPIN), Bielefield, Germany. Retrieved from: < http://coral2.spectrum.uni-bielefeld.de/gespin2011/final/Esteve-Gibert-Prieto. pdf >.
23. I.M. Franks, P. Nagelkerke, M. Ketelaars, and P. van Donkelaar Response preparation and control of movement sequences Can. J. Exp. Psych 52 1998 93 101
24. J.W. Folkins, and J.H. Abbs Lip and jaw motor control during speech: Responses to resistive loading of the jaw J. Sp. Hear. Res. 18 1975 207 220
25. J.W. Folkins, and G.N. Zimmermann Jaw-muscle activity during speech with the mandible fixed J. Acoust. Soc. Amer. 69 1981 1441 1445 (Pubitemid 11090153)
26. E.A. Franz, H.N. Zelaznik, and A. Smith Evidence of common timing processes in the control of manual, orofacial, and speech movement J. Motor Behav. 24 1992 281 287
27. M. Gentilucci, F. Benuzzi, M. Gangitano, and S. Grimaldi Grasp with hand and mouth: Kinematic study on healthy subjects J. Neurophysiol. 86 2001 1685 1699 (Pubitemid 32953220)
28. M. Gentilucci, P. Santunione, A.C. Roy, and S. Stefanini Execution and observation of bringing a fruit to the mouth affect syllable pronunciation Eur. J. Neurosci. 19 2004 190 202 (Pubitemid 38122085)
29. L. Goffman, and C. Malin Metrical effects on speech movements in children and adults J. Sp. Lang. Hear. Res. 42 1999 1003 1015 (Pubitemid 29367789)
30. S. Goldin-Meadow The role of gesture in communication and thinking Trends Cogn. Sci. 3 1999 419 429 (Pubitemid 29499810)
31. V.L. Gracco, and J.H. Abbs Dynamic control of the perioral system during speech: kinematic analyses of autogenic and nonautogenic sensorimotor processes J. Neuropath. 54 1985 418 432 (Pubitemid 15010913)
32. V.L. Gracco, and J.H. Abbs Variant and invariant characteristics of speech movements Exp. Brain. Res. 65 1986 155 166
33. F.H. Guenther, M. Hampson, and D. Johnson A theoretical investigation of reference frames for the planning of speech movements Psych. Rev. 105 1998 611 633 (Pubitemid 128456526)
34. H. Haken, J.A.S. Kelso, and H. Bunz A theoretical model of phase transitions in human hand movements Bio. Cybernetics. 51 1985 347 356 (Pubitemid 15135510)
35. M. Hiscock, and H. Chipuer Concurrent performance of rhythmically compatible or incompatible vocal and manual tasks: evidence for two sources of interference in verbal-manual timesharing Neuropsychologia 24 1986 691 698 (Pubitemid 17181143)
36. P. Howell, and A. Archer Susceptibility to the effects of delayed auditory feedback Percepti. Psychophysics. 36 1984 296 302 (Pubitemid 15180043)
37. P. Howell, and S. Sackin Timing interference to speech in altered listening conditions J. Acoust. Soc. Am. 111 2002 2842 2852 (Pubitemid 34760270)
38. J.M. Iverson, and E. Thelen Hand, mouth and brain: the dynamic emergence of speech and gesture J. Consciousness Stud. 6 1999 19 40
39. M.R. Jones, and M. Boltz Dynamic attending and response to time Psychol. Rev. 96 1989 459 491
40. J.A.S. Kelso, B. Tuller, and K.S. Harris A "dynamic pattern" perspective on the control and coordination of movement P.F. MacNeilage, The Production of Speech 1981 Springer-Verlag New York 137 173
41. J.A. Kelso, and B. Tuller "Compensatory articulation" under conditions of reduced afferent information: A dynamic formulation J. Sp. Hear. Res. 26 1983 217 224 (Pubitemid 13077622)
42. J.A. Kelso, and B. Tuller Converging evidence in support of common dynamical principles for speech and movement coordination. Amer. J. Physio.: Regulat Integrat. Compar. Physio. 246 1984 R935 R938
43. J.A. Kelso, B. Tuller, E. Vatikiotis-Bateson, and C.A. Fowler Functionally specific articulatory cooperation following jaw perturbations during speech: Evidence for coordinative structures. J. Exper. Psych.: Hum Percept. Perf. 6 1984 812 832
44. A. Kendon Some relationships between body motion and speech A.W. Seigman, B. Pope, Studies in Dyadic Communication 1972 Pergamon Elmsford, NY 177 210
45. Kendon, A., 1980. Gesticulation and speech: Two aspects of the process of utterance. In: Key, M.R. (Ed.), The Relationship of Verbal and Nonverbal Communication Mouton, The Hague, the Netherlands, pp. 207-227.
46. E. Komilis, D. Pelisson, and C. Prablanc Error processing in pointing at randomly feedback induced double step stimuli J. Motor Behav. 25 1993 299 308
47. E. Krahmer, and M. Swerts The effects of visual beats on prosodic prominence: acoustic analyses, auditory perception and visual perception J. Mem. Lang. 57 2007 396 414
48. R.M. Krauss, Y. Chen, and R.F. Gottesman Lexical gestures and lexical access: a process model D. McNeill, Language and Gesture 2000 Cambridge University Press Cambridge, UK 261 283
49. E.W. Large, and M.R. Jones The dynamics of attending: how people track time varying events Psychol. Rev. 106 1999 119 159
50. T. Leonard, and F. Cummins The temporal relation between beat gestures and speech Lang. Cogn. Proc. 26 2011 1457 1471
51. W.J.M. Levelt, G. Richardson, and W. La Heij Pointing and voicing in deictic expressions J. Mem. Lang. 24 1985 133 164
52. A.M. Liberman, and D.H. Whalen Of the relation of speech to language Trends Cog. Sci. 4 2000 187 196
53. D.P. Loehr, Gesture and Intonation: Doctoral dissertation, Georgetown University, Dissertation Abstracts International, 65 (2004) 2180. (UMI No. 3137056).
54. D.P. Loehr Aspects of rhythm in gesture and speech Gesture 7 2007 179 214
55. S. Lorenzo, E. Diederich, R. Telgmann, and C. Schutte Discrimination of dynamical system models for biological and chemical processes J. Comput. Chem. 28 2007 1384 1399 (Pubitemid 46742228)
56. W.D. Marslen-Wilson, and L.K. Tyler Central processes in speech understanding Philos. Trans. R. Soc., Ser. B 295 1981 317 332
57. R.I. Mayberry, and J. Jaques Gesture production during stuttered speech: insights into the nature of gesture-speech integration D. McNeill, Language and Gesture 2000 Cambridge University Press Cambridge 199 214
58. R.I. Mayberry, J. Jaques, and G. DeDe What stuttering reveals about the development of gesture speech relationship New Direction Child Dev. 79 1998 77 87
59. E. McClave Gestural beats: the rhythm hypothesis J. Psycolinguist. Res. 23 1994 45 66
60. E. McClave Pitch and manual gestures J. Psycolinguist. Res. 27 1998 69 89 (Pubitemid 128445235)
61. D. McNeill Hand and Mind 1992 University of Chicago Press Chicago
62. B. Merker, G. Madison, and P. Eckerdal On the role and origin of isochrony in human rhythmic entrainment Cortex 45 2009 2009 1 17
63. S. Nobe, Representational gestures, cognitive rhythms, and acoustic aspects of speech: a network/threshold model of gesture production: Doctoral dissertation, University of Chicago, Dissertation Abstracts International, 57 (1996) 4736. (UMI No. 9636827).
64. M. O'Dell, and T. Nieminen Coupled oscillator model of speech rhythm J.J. Ohala, Y. Hasegawa, M. Ohala, D. Granville, A.C. Bailey, Proceedings of the XIVth International Congress of Phonetic Sciences 2 1999 American Institute of Physics New York 1075 1078
65. Y. Paulignan, M. Jeannerod, C. MacKenzie, and R. Marteniuk Selective perturbation of visual input during prehension movements Exp. Brain Res. 87 1991 407 420
66. B. Peter, and W.H. Raskind Evidence for a familial speech sound disorder subtype in a multigenerational study of oral and hand motor sequencing ability Top. Lang. Dis. 31 2011 145 167
67. R.F. Port Meter and speech J. Phonetic 31 2003 599 611 (Pubitemid 37495932)
68. R. Port, K. Tajima, and F. Cummins Speech and rhythmic behavior G.J.P. Savelsburgh, H. van der Maas, P.C.L. van Geert, The Nonlinear Analysis of Developmental Processes 1998 Royal Dutch Academy of Arts and Sciences Amsterdam 5 45
69. C. Prablanc, and O. Martin Automatic control during hand reaching at undetected two-dimensional target displacements J. Neurophysiol. 67 1992 455 469
70. R.L. Ringel, and M.D. Steer Some effects of tactile and auditory alterations on speech output J. Sp. Hear. Res. 13 1963 369 378
71. A. Rochet-Capellan, R. Laboissierre, A. Galvan, and J. Schwartz The speech focus position effect on jaw-finger coordination in a pointing task J. Speech Lang. Hear. Res. 51 2008 1507 1521
72. Rusiewicz, H.L., 2011. Synchronization of speech and gesture: a dynamic systems perspective. In: Proceedings from the 2nd Gesture and Speech in Interaction (GESPIN), Bielefield, Germany. Abstract retrieved from < http://coral2.spectrum.uni-bielefield.de/gespin2011/final/rusiewicz.pdf >.
73. H. Rusiewicz, S. Shaiman, J. Iverson, and N. Szuminsky Effects of prosody and position on the timing of deictic gestures J. Speech Lang. Hear. Res. 56 2013 458 470
74. E. Saltzman, and D. Byrd Task-dynamics of gestural timing: Phase window and multifrequency rhythms Hum. Movement Sci. 19 2000 499 526
75. E.L. Saltzman, and K.G. Munhall A dynamical approach to gestural patterning in speech production Ecolog. Psych. 1 1989 330 380
76. R.A. Schmidt, and T.D. Lee Motor Control and Learning: A Behavioral Emphasis third ed. 1999 Human Kinetics Champaign, IL
77. S. Shaiman Kinematic and electromyographic responses to perturbation of the jaw J. Acoust. Soc. Am. 86 1989 78 88 (Pubitemid 19175064)
78. S. Shaiman, and V.L. Gracco Task-specific sensorimotor interactions in speech production Exp. Brain Res. 146 2002 411 418
79. L.D. Shriberg, T.F. Campbell, H.B. Karlsson, R.L. Brown, J.L. McSweeney, and C.J. Nadler A diagnostic marker for childhood apraxia of speech: the lexical stress ratio Clin. Linguist. Phonetics 17 2003 549 574 (Pubitemid 37361309)
80. A. Smith The control of orofacial movements in speech Crit. Rev. Oral Bio. Med. 3 1992 233 267
81. A. Smith, D.H. McFarland, and C.M. Weber Interactions between speech and finger movements: an exploration of dynamic pattern perspective J. Speech Lang. Hear. Res. 29 1986 471 480 (Pubitemid 17213072)
82. A. Stuart, J. Kalinowski, M.P. Rastatter, and K. Lynch Effect of delayed auditory feedback on normal speakers at two speech rates J. Acoust. Soc. Am. 111 2002 2237 2241 (Pubitemid 34517492)
83. E. Thelen, and L.B. Smith A Dynamic System Approach to the Development of Cognition and Action 2002 MIT Press Cambridge, MA
84. K. Tuite The production of gesture Semiotica 93 1993 8 105
85. B. Tuller, K.S. Harris, and J.A. Kelso Stress and rate: Differential transformation of articulation J. Acoust. Soc. Amer. 71 1982 1534 1543 (Pubitemid 12019688)
86. B. Tuller, J.A. Kelso, and K.S. Harris On the kinematics of articulatory control as a function of stress and rate J. Acoust. Soci. Am. 72 1982 S103
87. B. Tuller, K.S. Harris, and J.A. Kelso Converging evidence for the role of relative timing in speech. J. Exp. Psych.: Human Percept Perf. 9 1983 829 833
88. M.T. Turvey Coordination Am. Psychol. 45 1990 938 953
89. D. van Kuijk, and L. Boves Acoustic characteristics of lexical stress in continuous telephone speech Speech Commun. 27 1999 95 111
90. P.H.H.M. van Lieshout Dynamic systems theory and its application in speech B. Maassen, R. Kent, H. Peters, P. van Lieshout, W. Hulstijn, Speech Motor Control: In Normal and Disordered Speech 2004 Oxford University Press New York 51 82
91. E. von Holst Relative coordination as a phenomenon and as a method of analysis of central nervous system function R. Martin, The Collected Papers of Erich von Holst: The Behavioral Physiology of Animal and Man 1973 University of Miami Press Coral Gables
92. Włodarczak, M., Šimko, J., Wagner, P., 2012. Syllable boundary effect: temporal entrainment in overlapped speech. Paper presented at speech prosody. In: 6th International Conference, Shanghai, China.
93. Yasinnik, Y., Renwick, M., Shattuck-Hufnagel, S., 2004. The timing of speech-accompanying gestures with respect to prosody. In: Paper presented at the Meeting the Sound to Sense: 50+ years of Discoveries in Speech Communication, Cambridge, MA.