The interplay between the auditory and visual modality for end-of-utterance detection

Journal of the Acoustical Society of America

Volumn 123, Issue 1, 2008, Pages 354-365

  Barkhuysen, Pashiera ^a   Krahmer, Emiel ^a   Swerts, Marc ^a  

^a TILBURG UNIVERSITY   (Netherlands)

Author keywords

[No Author keywords available]

Indexed keywords

DATA ACQUISITION; SENSORY PERCEPTION;

AUDITORY MODALITY; BIMODAL CONDITION;

SPEECH ANALYSIS;

ADULT; ARTICLE; AUDITORY STIMULATION; AUDITORY SYSTEM; FEMALE; HUMAN; HUMAN EXPERIMENT; MALE; PRIORITY JOURNAL; RESPONSE TIME; SPEECH; SPEECH ARTICULATION; STATISTICAL ANALYSIS; VISION; VISUAL STIMULATION;

ADULT; AUDITORY PERCEPTION; CUES; FEMALE; HUMANS; JUDGMENT; MALE; MIDDLE AGED; SIGNAL DETECTION (PSYCHOLOGY); SPEECH; SPEECH PERCEPTION; SPEECH PRODUCTION MEASUREMENT; VISUAL PERCEPTION;

EID: 37849018667     PISSN: 00014966     EISSN: None     Source Type: Journal    
DOI: 10.1121/1.2816561     Document Type: Article

References (45)

1. Argyle, M., and Cook, M. (1976). Gaze and Mutual Gaze (Cambridge University Press, Cambridge, UK).
2. Beattie, G. W., Cutler, A., and Pearson, M. (1982). " Why is Mrs. Thatcher interrupted so often? " Nature (London) 300, 744-747.
3. Bertelson, P., Vroomen, J., and de Gelder, B. (2003). " Visual recalibration of auditory speech identification: A McGurk aftereffect.. " Psychol. Sci. 14, 592-597.
4. Brebner, J., and Welford, A. (1980). " Introduction: An historical background sketch., " in Reaction Times edited by, A. Welford, (Academic, New York), pp. 1-23.
5. Carlson, R., Hirschberg, J., and Swerts, M. (2005). " Cues to upcoming Swedish prosodic boundaries: Subjective judgment studies and acoustic correlates.. " Speech Commun. 46, 326-333.
6. Caspers, J. (1998). " Who's next? The melodic marking of questions vs. continuation in Dutch.. " Lang Speech 41, 375-398.
7. Cassell, J., Nakano, Y. I., Bickmore, T. W., Sidner, C. L., and Rich, C. (2001). " Non-verbal cues for discourse structure.. " Proceedings of the 41st Annual Meeting of the Association of Computational Linguistics (ACL), Toulouse, France, July 9-11, pp. 114-123.
8. Couper-Kuhlen, E. (1993). English Speech Rhythm (Benjamins, Philadelphia).
9. de Pijper, J. R., and Sanderman, A. A. (1994). " On the perceptual strength of prosodic boundaries and its relation to suprasegmental cues.. " J. Acoust. Soc. Am. 96, 2037-2047.
10. de Ruiter, J. P., Mitterer, H., and Enfield, N. (2006). " Projecting the end of a speaker's turn: A cognitive cornerstone of conversation.. " Language 82, 515-535.
11. Doherty-Sneddon, G., Bonner, L., and Bruce, V. (2001). " Cognitive demands of face monitoring: Evidence for visuospatial overload.. " Mem. Cognit. 29, 909-917.
12. Doughty, M. J. (2001). " Consideration of three types of spontaneous eyeblink activity in normal humans: During reading and video display terminal use, in primary gaze, and while in conversation.. " Optom. Vision Sci. 78, 712-725.
13. Duncan, S. (1972). " Some signals and rules for taking speaking turns in conversations.. " J. Pers Soc. Psychol. 23, 283-292.
14. Ekman, P. (1979). " About brows: Emotional and conversational signals., " in Human Ethology: Claims and Limits of a New Discipline, edited by, M. von Cranach,,, K. Foppa,,, W. Lepenies,, and, D. Ploog, (Cambridge University Press, Cambridge, UK), pp. 169-202.
15. Froeberg, S. (1907). " The relation between the magnitude of stimulus and the time of reaction.. " Arch. Psychol. (Frankf) 8.
16. Goodwin, C. (1980). " Restarts, pauses, and the achievement of a state of mutual gaze at turn-beginning.. " Sociological Inquiry 50, 272-302.
17. Grosjean, F. (1983). " How long is the sentence? Prediction and prosody in the on-line processing of language.. " Linguistics 21, 501-529.
18. Grosjean, F. (1996). " Gating.. " Lang. Cognit. Processes 11, 597-604.
19. Hearst, E. (1991). " Psychology and nothing.. " Am. Psychol. 79, 432-443.
20. Kendon, A. (1967). " Some functions of gaze-direction in social interaction.. " Acta Psychol. 26, 22-63.
21. Kim, J., Davis, C., and Krins, P. (2004). " Amodal processing of visual speech as revealed by priming.. " Cognition 93, B39-B47.
22. Kobayashi, H., and Kohshima, S. (1997). " Unique morphology of the human eye.. " Nature (London) 387, 767-768.
23. Koiso, H., Horiuchi, Y., Tutiya, S., Ichikawa, A., and Den, Y. (1998). " An analysis of turn-taking and backchannels based on prosodic and syntactic features in Japanese map task dialogs.. " Lang Speech 41, 295-321.
24. Krahmer, E., and Swerts, M. (2004). " More about brows., " in From Brows to Trust: Evaluating Embodied Conversational Agents, edited by, C. Pelachaud, and, Zs. Ruttkay, (Kluwer, Dordrecht), pp. 191-216.
25. Leroy, L. (1984). " The psychological reality of fundamental frequency declination.. " Antwerp Papers in Linguistics (Antwerp University Press, Antwerp, Belgium), Vol. 40.
26. Levinson, S. (1983). Pragmatics (Cambridge University Press, Cambridge, UK).
27. Luce, R. (1986). Response Times: Their Role in Inferring Elementary Mental Organization (Oxford University Press, New York).
28. Maynard, S. K. (1987). " Interactional functions of a nonverbal sign: Head movement in Japanese dyadic casual conversation.. " J. Pragmat. 11, 589-606.
29. Nakano, Y. I., Reinstein, G., Stocky, T., and Cassell, J. (2003). " Towards a model of face-to-face grounding.. " Proceedings of Annual Meeting of the Association for Computational Linguistics (ACL), Sapporo, Japan, July 7-12, pp. 553-561.
30. Novick, D. G., Hansen, B., and Ward, K. (1996). " Coordinating turn-taking with gaze.. " Proceedings of the International Conference on Spoken Language Processing (ICSLP), Philadelphia, PA, October 3-6, pp. 1888-1891.
31. Price, P., Ostendorf, M., Shattuck-Hufnagel, S., and Fong, S. (1991). " The use of prosody in syntactic disambiguation.. " J. Acoust. Soc. Am. 90, 2956-2970.
32. Schwartz, J.-L., Berthommier, F., and Savariaux, C. (2004). " Seeing to hear better: Evidence for early audio-visual interactions in speech identification.. " Cognition 93, B69-B78.
33. Silverman, S., and Pierrehumbert, J. (1990). " The timing of prenuclear high accents in English., " Laboratory Phonology: Between the Grammar and Physics of Speech, edited by, J. Kingston, and, M. Beckman, (Cambridge University Press, Cambridge, UK), Vol. I, pp. 71-106.
34. Swerts, M. (1997). " Prosodic features at discourse boundaries of different strength.. " J. Acoust. Soc. Am. 101, 514-521.
35. Swerts, M. (1998). " Filled pauses as markers of discourse structure.. " J. Pragmat. 30, 485-496.
36. Swerts, M., Bouwhuis, D., and Collier, R. (1994a). " Melodic cues to the perceived finality of utterances.. " J. Acoust. Soc. Am. 96, 2064-2075.
37. Swerts, M., Collier, R., and Terken, J. (1994b). " Prosodic predictors of discourse finality in spontaneous monologues.. " Speech Commun. 15, 79-90.
38. Swerts, M., and Geluykens, R. (1994). " Prosody as a marker of information flow in spoken discourse.. " Lang Speech 37, 21-43.
39. Teichner, W., and Krebs, M. (1974). " Laws of visual choice reaction time.. " Psychol. Rev. 81, 75-98.
40. Ulrich, R., Rinkenauer, G., and Miller, J. (1998). " Effects of stimulus duration and intensity on simple reaction time and response force.. " J. Exp. Psychol. Hum. Percept. Perform. 24, 915-928.
41. Vertegaal, R., Slagter, R., van de Veer, G., and Nijholt, A. (2000). " Why conversational agents should catch the eye.. " Proceedings of the Internation Computer-Human Interaction Conference (CHI), The Hague, The Netherlands, April 1-6, pp. 257-258.
42. Walley, A. C., Michela, V., and Wood, D. (1995). " The gating paradigm: Effects of presentation format on spoken word recognition by children and adults.. " Percept. Psychophys. 57, 343-351.
43. Ward, N., and Tsukahara, W. (2000). " Prosodic features which cue back-channel responses in English and Japanese.. " J. Pragmat. 23, 1177-1207.
44. Wells, G. (1913). " The influence of stimulus duration on RT.. " Psychol. Monogr. 15.
45. Wightman, C., Shattuck-Hufnagel, S., Ostendorf, M., and Price, P. (1992). " Segmental durations in the vicinity of prosodic phrase boundaries.. " J. Acoust. Soc. Am. 91, 1707-1717.