Automatic detection of disfluency boundaries in spontaneous speech of children using audio-visual information

IEEE Transactions on Audio, Speech and Language Processing

Volumn 17, Issue 1, 2009, Pages 2-12

  Yildirim, Serdar ^a,b   Narayanan, Shrikanth ^a  

^a UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

^b MUSTAFA KEMAL UNIVERSITY   (Turkey)

Author keywords

Disfluency detection; Feature selection; Information fusion; Spoken language processing; Spontaneous children speech

Indexed keywords

AUDIO-VISUAL INFORMATION; AUTOMATIC DETECTION; AUTOMATIC RECOGNITION; COGNITIVE STATE; COMPUTER GAME; DECISION LEVELS; DETECTION ACCURACY; DETECTION ERROR RATE; DETECTION SYSTEM; DISFLUENCIES; DISFLUENCY DETECTION; FEATURE LEVEL; FEATURE SELECTION; INFORMATION SOURCES; LANGUAGE FEATURES; MULTI-MODAL; SPOKEN LANGUAGE PROCESSING; SPONTANEOUS CHILDREN SPEECH; SPONTANEOUS SPEECH; VISUAL INFORMATION;

ERROR DETECTION; HUMAN COMPUTER INTERACTION; INFORMATION FUSION; LINGUISTICS; SPEECH RECOGNITION; VISUAL COMMUNICATION;

INFORMATION USE;

EID: 70350442414     PISSN: 15587916     EISSN: None     Source Type: Journal    
DOI: 10.1109/TASL.2008.2006728     Document Type: Article

References (40)

1. S. Banerjee, J. E. Beck, and J. Mostow, "Evaluating the effect of predicting oral reading miscues," in Proc. Eurospeech, Geneva, Switzerland, 2003, pp. 3165-3168.
2. K. Lee, A. Hagen, N. Romanyshyn, S. Martin, and B. Pellom, "Analysis and detection of reading miscues for interactive literacy tutors," in Proc. 20th Int. Conf. Comput. Linguist. (Coling), Geneva, Switzerland, Aug. 2004, Article 1524.
3. M. Black, J. Tepperman, S. Lee, P. Price, and S. Narayanan, "Automatic detection and classification of disfluent reading miscues in young childrens speech for the purpose of assessment," in Proc. InterSpeech ICSLP, Antwerp, Belgium, Aug. 2007, pp. 206-209.
4. E. E. Shriberg, "Preliminaries to a Theory of Speech Disfluencies," Ph.D. dissertation, Univ. of California, Berkley, 1994.
5. A. Stolcke and E. Shriberg, "Statistical language modeling for speech disfluencies," in Proc. ICASSP, Atlanta, GA, 1996, vol.1, pp. 405-408.
6. E. Shriberg, R. Bates, and A. Stolcke, "A prosody-only decision-tree model for disfluency detection," in Proc. Eurospeech, 1997, pp. 2383-2386.
7. A. Stolcke, E. Shriberg, R. Bates, M. Ostendorf, D. Hakkani, M. Plauche, G. Tür, and Y. Lu, "Automatic detection of sentence boundaries and disfluencies based on recognized words," in Proc. ICSLP, 1998, no.5, pp. 2247-2250.
8. Y. Liu, E. Shriberg, and A. Stolcke, "Automatic disfluency identification in conversational speech using multiple knowledge source," in Proc. Eurospeech, Geneva, Switzerland, 2003, pp. 957-960.
9. S. Lee, A. Potamianos, and S. Narayanan, "Acoustics of children's speech: Developmental changes of temporal and spectral parameters," J. Acoust. Soc. Amer., vol.105, pp. 1455-1468, Mar. 1999.
10. S. Narayanan and A. Potamianos, "Creating conversational interfaces for children," IEEE Trans. Speech Audio Process., vol.10, no.2, pp. 65-78, Feb. 2002.
11. J.Wilpon and C. Jacobsen, "A study of speech recognition for children and elderly," in Proc. ICASSP, 1996, pp. 349-352.
12. S. Das, D. Nix, and M. Picheny, "Improvements in children's speech recognition performance," in Proc. ICASSP, 1998, pp. 433-436.
13. A. Hagen, B. Pellom, and R. Cole, "Childrens speech recognition with application to interactive books and tutors," in Proc. IEEE ASRUWorkshop, St. Thomas, Virgin Islands, Dec. 2003.
14. Q. Li and M. J. Russell, "An analysis of the causes of increased error rated in children's speech recognition," in Proc. ICSLP, Denver, CO, 2002, pp. 2337-2340.
15. S. L. Oviatt and B. Adams, , J. Cassell, J. Sullivan, S. Prevost, and E. Churchill, Eds., "Designing and evaluating conversational interfaces with animated characters," in Embodied Conversational Agents.. Cambridge, MA: MIT Press, 2000, pp. 319-343.
16. D. Wang and S. Narayanan, "A multi-pass linear fold algorithm for sentence boundary detection using prosodic cues," in Proc. ICASSP, May 2004, vol.1, pp. 525-528.
17. D. McNeill, F. Quek, K.-E. McCullough, S. D. N. Furuyama, R. Bryll, X.-F. Ma, and R. Ansari, "Catchments, prosody, and discourse," Gesture, vol.1, no.1, pp. 9-33, 2001.
18. F. Quek, D. McNeill, R. Bryll, S. Duncan, X.-F. Ma, C. Kirbas, K. E. McCullough, and R. Ansari, "Multimodal human discourse: Gesture and speech," in ACM Trans. Comput.-Human Interaction, 2002, vol.9, no.3, pp. 171-193.
19. C. Busso and S. Narayanan, "Interrelation between speech and facial gestures in emotional utterances: A single subject study," IEEE Trans. Speech, Audio, Lang. Process., vol.15, no.8, pp. 2331-2347, Nov. 2007.
20. C. Busso, Z. Deng, M. Grimm, U. Neumann, and S. Narayanan, "Rigid head motion in expressive speech animation: Analysis and synthesis," IEEE Trans. Speech, Audio, Lang. Process., vol.15, no.3, pp. 1075-1086, Mar. 2007.
21. L. Valbonesi, R. Ansari, D. McNeill, F. Quek, S. Duncan, K. McCullough, and R. Bryll, "Analysis of speech and gesture frequency during fluent and hesitant phases in speech," in Proc. 6th Multi-Conf. Syst., Cybern., Inf. (SCI 2002), Orlando, FL, Jul. 14-18, 2002.
22. J. Cassell, Y. I. Nakano, T. W. Bickmore, C. L. Sidner, and C. Rich, "Non-verbal cues for discourse structure," in Proc. 39th Meeting Assoc. Comput. Linguist., Toulouse, France, Jul. 2001, pp. 106-115.
23. L. Chen, M. Harper, and F. Quek, "Gesture patterns during speech repairs," in Proc. ICSLP, Denver, CO, 2002, pp. 629-632.
24. S. Kettebekov, M. Yeasin, and R. Sharma, "Prosody based audiovisual coanalysis for coverbal gesture recognition," IEEE Trans. Multimedia, vol.7, no.2, pp. 234-242, Apr. 2005.
25. L. Chen, Y. Liu, M. Harper, and E. Shriberg, "Multimodal model integration for sentence unit detection," in Proc. ICMI, State College, PA, 2004, pp. 121-128.
26. N. Landwehr, M. Hall, and E. Frank, "Logistic model trees," Mach. Learn. J., vol.59, no.1-2, pp. 161-205, 2005.
27. J. Ma, J.Yan, and R. Cole, "Cu animate tools for enabling conversations with animated characters," in Proc. ICSLP, 2002, vol.1, pp. 197-200.
28. P. Boersma and D. Weenink, "Praat: Doing Phonetics by Computer," Jul. 20, 2005 [Online]. Available: http://www.praat.org, (version 4.3.19) [computer program], 2005, retrieved from
29. M. Kipp, "Anvil-A generic annotation tool for multimodal dialogue," in Proc. Eurospeech, 2001, pp. 1367-1370.
30. S. Montanari, S. Yildirim, S. Khurana, M. Landes, L. Lawyer, E. Andersen, and S. Narayanan, "Analyzing the interplay between spoken language and gestural cues in conversational child-machine interactions in pre/early literate age group," in Proc. InStil, Jul. 2004, paper ID 047.
31. S. Montanari, S. Yildirim, E. Andersen, and S. Narayanan, "Reference marking in childrens computer-directed speech: An integrated analysis of discourse and gesture," in Proc. ICSLP, Oct. 2004, pp. 1841-1844.
32. S. Oviatt, "Predicting spoken disfluecies during human-computer interaction," Comput. Speech Lang., vol.9, pp. 19-35, 1995.
33. E. Shriberg, "Disfluencies in switchboard," in Proc. ICSLP, 1996, pp. 11-14.
34. A. Stolcke, "Srilm-An extensible language modeling toolkit," in Proc. ICSLP, 2002, vol.2, pp. 901-904.
35. J. L. Barren, D. J. Fleet, and S. Beauchemin, "Performance of optical flow techniques," Int. J. Comput. Vision, vol.12, pp. 43-77, 1994.
36. D. J. Fleet and K. Langley, "Recursive filters for optical flow," IEEE Trans. Pattern Anal. Mach. Intell., vol.17, no.1, pp. 61-67, Jan. 1995.
37. R. A. Peters, C. W. G. Clifford, and C. S. Evans, "Measuring the structure of dynamic visual signals," Animal Behaviour, vol.64, pp. 131-146, 2002.
38. R. Duda, P. Hart, and D. Stork, Pattern Classification, 2nd ed. New York: Wiley, 2001
39. D. Tax, M. van Breukelen, R. Duin, and J. Kittler, "Combining multiple classifiers by averaging or by multiplying," Pattern Recognition, vol.33, pp. 1475-1485, 2000.
40. L. Kuncheva and C. Whitaker, "Measure of diversity in classifier esembles," Mach. Learn., vol.51, pp. 181-207, 2003.