Recognizing articulatory gestures from speech for robust speech recognition

Journal of the Acoustical Society of America

Volumn 131, Issue 3, 2012, Pages 2270-2287

  Mitra, Vikramjit ^a   Nam, Hosung ^b   Espy Wilson, Carol ^c   Saltzman, Elliot ^d   Goldstein, Louis ^e  

^a SRI INTERNATIONAL   (United States)

^b HASKINS LABORATORIES   (United States)

^c UNIVERSITY OF MARYLAND   (United States)

^d BOSTON UNIVERSITY   (United States)

^e UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICULATORY GESTURES; AUTOMATIC SPEECH RECOGNITION SYSTEM; DATA SETS; DIGIT RECOGNITION; DYNAMIC BAYESIAN NETWORK; DYNAMIC PARAMETERS; NATURAL SPEECH; RECOGNITION PERFORMANCE; RECOGNITION RATES; ROBUST SPEECH RECOGNITION; SPEECH RECOGNITION SYSTEMS; SPEECH SIGNALS; SYNTHETIC SPEECH; THREE STAGES; WORD RECOGNITION;

GESTURE RECOGNITION; NETWORK ARCHITECTURE; SPEECH SYNTHESIS; VOCABULARY CONTROL;

SPEECH RECOGNITION;

ARTICLE; AUTOMATIC SPEECH RECOGNITION; BAYES THEOREM; GESTURE; HUMAN; LINGUISTICS; PHONETICS; PHYSIOLOGY; SPEECH; SPEECH PERCEPTION;

BAYES THEOREM; GESTURES; HUMANS; PHONETICS; SPEECH; SPEECH ACOUSTICS; SPEECH PERCEPTION; SPEECH RECOGNITION SOFTWARE; VOCABULARY;

EID: 84858976368     PISSN: 00014966     EISSN: None     Source Type: Journal    
DOI: 10.1121/1.3682038     Document Type: Article

References (63)

1. Atal, B. S. (1983), Efficient coding of LPC parameters by temporal decomposition., Proceedings of ICASSP, Boston, MA, pp. 81-84.
2. Bengio, Y., and Le Cun, Y. (2007), Scaling learning algorithms toward AI., in Large Scale Kernel Machines, edited by, L. Bottou, O. Chapelle, D. De-Coste, and, J. Weston, (MIT Press, Cambridge, MA), pp. 321-360.
3. Bilmes, J., and Zweig, G. (2002), The graphical models Toolkit: An open source software system for speech and time-series processing., Proceedings of ICASSP, Orlando, FL, Vol. 4, pp. 3916-3919.
4. Browman, C., and Goldstein, L. (1989), Articulatory gestures as phonological units., Phonology 6, 201-251. 10.1017/S0952675700001019
5. Browman, C., and Goldstein, L. (1992), Articulatory phonology: An overview., Phonetica 49, 155-180. 10.1159/000261913
6. Chen, C., and Bilmes, J. (2007), MVA processing of speech features., IEEE Trans. Audio Speech Lang. Processing 15 (1), 257-270. 10.1109/TASL.2006.876717
7. Chomsky, N., and Halle, M. (1968), The Sound Pattern of English (MIT Press, Cambridge, MA), 484 pp.
8. Cooke, M., Barker, J., Cunningham, S., and Shao, X. (2006), An audio-visual corpus for speech perception and automatic speech recognition., J. Acoust. Soc. Am. 120, 2421-2424. 10.1121/1.2229005 (Pubitemid 44631681)
9. Cui, X., and Alwan, A. (2005), Noise robust speech recognition using feature compensation based on polynomial regression of utterance SNR., IEEE Trans. Speech Audio Processing 13 (6), 1161-1172. 10.1109/TSA.2005.853002 (Pubitemid 41605019)
10. D'Ausilio, A., Pulvermüller, F., Salmas, P., Bufalari, I., Begliomini, C., and Fadiga, C. (2009), The motor somatotopy of speech perception., Curr. Biol. 19, 381-385. 10.1016/j.cub.2009.01.017
11. Demuth, H., Beale, M., and Hagan, M. (2008), Neural Network ToolboxTM6, User's Guide., The MathWorks Inc., Natick, MA. www.mathworks.com/access/ helpdesk/help/pdf-doc/nnet/nnet.pdf (Last viewed June 28, 2010).
12. Deng, L., and Sun, D. (1994), A statistical approach to automatic speech recognition using atomic units constructed from overlapping articulatory features., J. Acoust. Soc. Am. 95 (5), 2702-2719. 10.1121/1.409839 (Pubitemid 24152864)
13. Deshmukh, O., Espy-Wilson, C., Salomon, A., and Singh, J. (2005), Use of temporal information: Detection of the periodicity and aperiodicity profile of speech., IEEE Trans. Speech Audio Process. 13 (5), 776-786. 10.1109/TSA.2005. 851910 (Pubitemid 41558894)
14. European Telecommunications Standards Institute-Advanced Front End (2007), Speech processing, transmission and quality aspects (STQ); Distributed speech recognition; Adv. frontend feature extraction algorithm; Compression algorithms., ES 202 050 Ver. 1.1.5
15. Flynn, R., and Jones, E. (2008), Combined speech enhancement and auditory modelling for robust distributed speech recognition., Speech Comm. 50, 797-809. 10.1016/j.specom.2008.05.004
16. Frankel, J., and King, S. (2001), ASR-Articulatory speech recognition., Proceedings of Eurospeech, Aalborg, Denmark, pp. 599-602.
17. Frankel, J., and King, S. (2005), A hybrid ANN/DBN approach to articulatory feature recognition., Proc. of Eurospeech, Interspeech, Lisbon, Portugal, pp. 3045-3048.
18. Frankel, J., Wester, M., and King, S. (2004), Articulatory feature recognition using dynamic Bayesian networks., Proc. of ICSLP, Jeju, Korea, pp. 1202-1205.
19. Ghahramani, Z. (1998), Learning dynamic Bayesian networks., in Adaptive Processing of Temporal Information, edited by, C. L. Giles, and, M. Gori, (Springer-Verlag, Berlin), pp. 168-197. (Pubitemid 128056031)
20. Ghosh, P., Narayanan, S., Divenyi, P., Goldstein, L., and Saltzman, E. (2009), Estimation of articulatory gesture patterns from speech acoustics., Proceedings of Interspeech, Brighton, UK, pp. 2803-2806.
21. Gillick, L., and Cox, S. J. (1989), Some statistical issues in the comparison of speech recognition algorithms., Proceedings of ICASSP, pp. 532-535. (Pubitemid 20604171)
22. Hanson, H. M., and Stevens, K. N. (2002), A quasiarticulatory approach to controlling acoustic source parameters in a Klatt-type formant synthesizer using HLsyn., J. Acoust. Soc. Am. 112 (3), 1158-1182. 10.1121/1.1498851 (Pubitemid 35006671)
23. Hermansky, H., and Morgan, N. (1994), RASTA processing of speech., IEEE Trans. Speech Audio Process., 2, 578-589. 10.1109/89.326616
24. Hermansky, H., Ellis, D., and Sharma, S. (2000), Tandem connectionist feature stream extraction for conventional HMM systems., Proceedings of ICASSP, Istanbul, Turkey, pp. 1635-1638.
25. Hogden, J., Nix, D., and Valdez, P. (1998), An articulatorily constrained, maximum likelihood approach to speech recognition., Tech. Report, LA-UR-96-3945 (Los Alamos National Laboratory, Los Alamos, NM).
26. Hu, C., Zhuang, X., and Hasegawa-Johnson, M. (2010), FSM-based pronunciation modeling using articulatory phonological code., Proceedings of Interspeech, pp. 2274-2277.
27. Jakobson, R., Fant, C. G. M., and Halle, M. (1952), Preliminaries to speech analysis: The distinctive features and their correlates., MIT Acoustics Laboratory Technical Report 13 (MIT Press, Cambridge, MA).
28. Jordan, M. I., and Rumelhart, D. E. (1992), Forward models-supervised learning with a distal teacher., Cogn. Sci. 16, 307-354. 10.1207/ s15516709cog1603-1
29. Juneja, A., (2004), Speech recognition based on phonetic features and acoustic landmarks., Ph.D. thesis, University of Maryland, College Park, MD.
30. Jung, T. P., Krishnamurthy, A. K., Ahalt, S. C., Beckman, M. E., and Lee, S. H. (1996), Deriving gestural scores from articulator-movement records using weighted temporal decomposition., IEEE Trans. Speech Audio Process. 4 (1), 2-18. 10.1109/TSA.1996.481448
31. Jurafsky, D., Ward, W., Jianping, Z., Herold, K., Xiuyang, Y., and Sen, Z. (2001), What kind of pronunciation variation is hard for triphones to model?, Proceedings of ICASSP, Utah, Vol. 1, pp. 577-580. (Pubitemid 32839316)
32. King, S., Frankel, J., Livescu, K., McDermott, E., Richmond, K., and Wester, M. (2007), Speech production knowledge in automatic speech recognition., J. Acoust. Soc. Am. 121 (2), 723-742. 10.1121/1.2404622 (Pubitemid 46192674)
33. Kirchhoff, K. (1999), Robust speech recognition using articulatory information., Ph.D. Thesis, University of Bielefeld, Germany.
34. Lapedes, A., and Farber, R. (1988), How neural networks work., Technical Report LA-UR-88-418, Los Alamos National Library, Los Alamos, NM.
35. Lippmann, R. (1997), Speech recognition by machines and humans., Speech Comm. 22, 1-15. 10.1016/S0167-6393(97)00021-6 (Pubitemid 127403436)
36. Markov, K., Dang, J., and Nakamura, S. (2006), Integration of articulatory and spectrum features based on the hybrid HMM/BN modeling framework., Speech Comm. 48, 161-175. 10.1016/j.specom.2005.07.003 (Pubitemid 43012029)
37. McGowan, R. S. (1994), Recovering articulatory movement from formant frequency trajectories using task dynamics and a genetic algorithm: Preliminary model tests., Speech Comm. 14 (1), 19-48. 10.1016/0167-6393(94)90055-8
38. Meister, I. G., Wilson, S. M., Deblieck, C., Wu, A. D., and Iacoboni, M. (2007), The essential role of premotor cortex in speech perception., Curr. Biol. 17, 1692-1696. 10.1016/j.cub.2007.08.064 (Pubitemid 47503812)
39. Nam, H., Goldstein, L., Saltzman, E., and Byrd, D. (2004), TaDA: An enhanced, portable task dynamics model in MATLAB., J. Acoust. Soc. Am. 115 (5), pp. 2430.
40. Nam, H., Mitra, V., Tiede, M., Saltzman, E., Goldstein, L., Espy-Wilson, C., and Hasegawa- Johnson, M. (2010), A procedure for estimating gestural scores from natural speech., Proceedings of Interspeech, Makuhari, Japan, pp. 30-33.
41. Mitra, V., Nam, H., Espy-Wilson, C., Saltzman, E., and Goldstein, L. (2010), Retrieving tract variables from acoustics: A comparison of different machine learning strategies., IEEE J. Selected Topics Signal Process. 4, 1027-1045. 10.1109/JSTSP.2010.2076013
42. Mitra, V., Nam, H., Espy-Wilson, C., Saltzman, E., and Goldstein, L. (2011), Speech inversion: Benefits of tract variables over pellet trajectories., Proceedings of International Conference on Acoustics, Speech and Signal Processing, Prague, Czech Republic, pp. 5188-5191.
43. Mitra, V., Özbek, I., Nam, H., Zhou, X., and Espy-Wilson, C. (2009), From acoustics to vocal tract time functions., Proceedings of ICASSP, pp. 4497-4500.
44. Neiberg, D., Ananthakrishnan, G., and Engwall, O. (2008), The acoustic to articulation mapping: Non-linear or non-unique?, Proceedings of Interspeech, Brisbane, Australia, pp. 1485-1488.
45. Ostendorf, M. (1999), Moving beyond the beads-on-a-string' model of speech., in Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop, Colorado, Vol. 1, pp. 79-83.
46. Pearce, D., and Hirsch, H. G. (2000), The aurora experimental framework for the performance evaluation of speech recognition systems under noisy conditions., Proceedings of ICSLP, ASR, Beijing, China, pp. 181-188.
47. Pruthi, T. (2007), Analysis, vocal-tract modeling and automatic detection of vowel nasalization., Ph.D. Thesis, University of Maryland, College Park, MD.
48. Qin, C., and Carreira-Perpin, M. (2007), An empirical investigation of the nonuniqueness in the acoustic-to-articulatory mapping., Proceedings of Interspeech, Antwerp, Belgium, pp. 74-77.
49. Ramanarayanan, V., Bresch, E., Byrd, D., Goldstein, L., and Narayanan, S. (2009), Analysis of pausing behavior in spontaneous speech using real-time magnetic resonance imaging of articulation., J. Acoust. Soc. Am. 126 (5), EL160-EL165. 10.1121/1.3213452
50. Richardson, M., Bilmes, J., and Diorio, C. (2003), Hidden-articulator Markov models for speech recognition., Speech Comm. 41 (2-3), 511-529. 10.1016/S0167-6393(03)00031-1
51. Richmond, K. (2001), Estimating articulatory parameters from the acoustic speech signal., Ph.D. Thesis, University of Edinburgh, UK.
52. Richmond, K. (2007), Trajectory mixture density network with multiple mixtures for acoustic-articulatory inversion., ITRW on Non-Linear Speech Processing, NOLISP-07, Paris, France, pp. 67-70.
53. Rubin, P. E., Baer, T., and Mermelstein, P. (1981), An articulatory synthesizer for perceptual research., J. Acoust. Soc. Am. 70, 321-328. 10.1121/1.386780 (Pubitemid 11011801)
54. Saltzman, E., and Munhall, K. (1989), A dynamical approach to gestural patterning in speech production., Ecol. Psychol. 1 (4), 332-382. 10.1207/s15326969eco0104-2
55. Schmidbauer, O. (1989), Robust statistic modelling of systematic variabilities in continuous speech incorporating acoustic-articulatory relations., Proceedings of ICASSP, pp. 616-619. (Pubitemid 20604192)
56. Stevens, K. (1960), Toward a model for speech recognition., J. Acoust. Soc. Am. 32, 47-55. 10.1121/1.1907874
57. Sun, J. P., and Deng, L. (2002), An overlapping-feature-based phonological model incorporating linguistic constraints: Applications to speech recognition., J. Acoust. Soc. Am. 111 (2), 1086-1101. 10.1121/1.1420380 (Pubitemid 34127489)
58. Toutios, A., and Margaritis, K. (2005), A support vector approach to the acoustic-to- articulatory mapping., Proceedings of Interspeech, Lisbon, Portugal, pp. 3221-3224.
59. Virag, N. (1999), Single channel speech enhancement based on masking properties of the human auditory system., IEEE Trans. Speech Audio Process. 7 (2), 126-137. 10.1109/89.748118
60. Westbury, J. (1994), X-ray microbeam speech production database user's handbook., University of Wisconsin, Madison.
61. Wrench, A. A., and Hardcastle, W. J. (2000), A multichannel articulatory database and its application for automatic speech recognition., in 5th Seminar on Speech Production: Models and Data, Bavaria, Germany, pp. 305-308.
62. Xiao, X., Li, J., Chng, E. S., Li, H., and Lee, C. (2010), A study on the generalization capability of acoustic models for robust speech recognition., IEEE Trans. Audio Speech Lang. Process. 18 (6), 1158-1169. 10.1109/TASL.2009. 2031236
63. Zhuang, X., Nam, H., Hasegawa-Johnson, M., Goldstein, L., and Saltzman, E. (2009), Articulatory phonological code for word classification., Proceedings of Interspeech, Brighton, UK, pp. 2763-2766.