Face active appearance modeling and speech acoustic information to recover articulation

IEEE Transactions on Audio, Speech and Language Processing

Volumn 17, Issue 3, 2009, Pages 411-422

  Katsamanis, Athanassios ^a   Papandreou, George ^a   Maragos, Petros ^a  

^a NATIONAL TECHNICAL UNIVERSITY OF ATHENS   (Greece)

Author keywords

Active appearance models (AAMs); Audiovisual to articulatory speech inversion; Canonical correlation analysis (CCA); Multimodal fusion

Indexed keywords

ACTIVE APPEARANCE MODELS; ACTIVE APPEARANCE MODELS (AAMS); APPEARANCE MODELING; AUDIO FEATURES; AUDIOVISUAL-TO-ARTICULATORY SPEECH INVERSION; CANONICAL CORRELATION ANALYSIS; CANONICAL CORRELATION ANALYSIS (CCA); DYNAMIC INFORMATION; ELECTROMAGNETIC ARTICULOGRAPHY; FACE TRACKING; FACIAL ANALYSIS; ILL POSED; ILL-POSEDNESS; INVERSION PROCESS; INVERSION SCHEME; LINE SPECTRAL FREQUENCIES; LINEAR MAPPING; MARKOVIAN; MEL-FREQUENCY CEPSTRAL COEFFICIENTS; MODEL SWITCHING; MULTI-MODAL; MULTI-STREAM HIDDEN MARKOV MODEL; MULTIMODAL FUSION; PIECEWISE LINEAR MODELS; POINTS OF INTEREST; SPEECH ACOUSTICS; SPEECH INVERSION; SPEECH PRODUCTION; VISUAL FEATURE EXTRACTION; VISUAL INFORMATION; VISUAL MODALITIES; VOCAL-TRACTS;

FACE RECOGNITION; FEATURE EXTRACTION; FREQUENCY ESTIMATION; HIDDEN MARKOV MODELS; PIECEWISE LINEAR TECHNIQUES; SPEECH RECOGNITION; VISUAL COMMUNICATION;

AUDIO ACOUSTICS;

EID: 70350574658     PISSN: 15587916     EISSN: None     Source Type: Journal    
DOI: 10.1109/TASL.2008.2008740     Document Type: Article

References (45)

1. G. Potamianos, C. Neti, G. Gravier, and A. Garg, "Recent advances in the automatic recognition of audio-visual speech, " Proc. IEEE, vol. 91, no. 9, pp. 1306-1326, Sep. 2003.
2. G. Bailly, M. Bérar, F. Elisei, and M. Odisio, "Audiovisual speech synthesis, " Int. J. Speech Technol., vol. 6, no. 4, pp. 331-346, Oct. 2003.
3. H. Yehia, P. Rubin, and E. Vatikiotis-Bateson, "Quantitative association of vocal-tract and facial behavior, " Speech Commun., vol. 26, pp. 23-43, 1998.
4. H. Mcgurk, J. Macdonald, Hearing lips and seeing voices, Nature, 264, 746-748, 1976.
5. J. Schroeter and M. Sondhi, "Techniques for estimating vocal-tract shapes from the speech signal, " IEEE Trans. Speech Audio Process., vol. 2, no. 1, pp. 133-150, Jan. 1994.
6. S. King, J. Frankel, K. Livescu, E. McDermott, K. Richmond, and M. Wester, "Speech production knowledge in automatic speech recognition, " J. Acoust. Soc. Amer., vol. 121, no. 2, pp. 723-742, Feb. 2007.
7. J. Schroeter and M. M. Sondhi, "Speech coding based on physiological models of speech production, " in Advances in Speech Signal Processing, S. Furui and M. M. Sondhi, Eds. New York: Marcel Dekker, 1992.
8. O. Engwall, O. Bälter, A.-M. Öster, and H. Sidenbladh- Kjellström, "Designing the user interface of the computer-based speech training system ARTUR based on early user tests, " J. Behavior Inf. Technol., vol. 25, no. 4, pp. 353-365, 2006.
9. S. Ouni and Y. Laprie, "Modeling the articulatory space using a hypercube codebook for acoustic-to-articulatory inversion, " J. Acoust. Soc. Amer., vol. 118, no. 1, pp. 444-460, 2005.
10. K. Richmond, S. King, and P. Taylor, "Modelling the uncertainty in recovering articulation from acoustics, " Comput. Speech Lang., vol. 17, pp. 153-172, 2003.
11. T. Toda, A. W. Black, and K. Tokuda, "Statistical mapping between articulatory movements and acoustic spectrum using a Gaussian mixture model, " Speech Commun., vol. 50, pp. 215-227, 2008.
12. S. Hiroya and M. Honda, "Estimation of articulatory movements from speech acoustics using an HMM-based speech production model, " IEEE Trans. Speech Audio Process., vol. 12, no. 2, pp. 175-185, Mar. 2004.
13. T. Chen, "Audiovisual speech processing, " IEEE Signal Process. Mag., vol. 18, no. 1, pp. 9-21, Jan. 2001.
14. E. Yamamoto, S. Nakamura, and K. Shikano, "Lip movement synthesis from speech based on hidden Markov models, " Speech Commun., vol. 26, pp. 105-115, 1998.
15. G. Englebienne, T. Cootes, and M. Rattray, "A probabilistic model for generating realistic speech movements from speech, " in Proc. Adv. Neural Inf. Process. Syst., 2007.
16. K. Choi, Y. Luo, and J.-N. Hwang, "Hidden Markov model inversion for audio-to-visual conversion in an MPEG-4 facial animation system, " J. VLSI Signal Process., vol. 29, pp. 51-61, 2001.
17. L. Xie and Z.-Q. Liu, "Realistic mouth-synching for speech-driven talking face using articulatory modeling, " IEEE Trans. Multimedia, vol. 9, no. 3, pp. 500-510, Apr. 2007.
18. J. Jiang, A. Alwan, P. A. Keating, E. T. Auer, and L. E. Bernstein, "On the relationship between face movements, tongue movements, and speech acoustics, " EURASIP J. Appl. Signal Process., vol. 11, pp. 1174-1188, 2002.
19. O. Engwall, "Introducing visual cues in acoustic-to-articulatory inversion, " in Proc. Int. Conf. Spoken Lang. Process., 2005, pp. 3205-3208.
20. H. Kjellström, O. Engwall, and O. Bälter, "Reconstructing tongue movements from audio and video, " in Proc. Int. Conf. Spoken Lang. Process., 2006, pp. 2238-2241.
21. A. Katsamanis, G. Papandreou, and P. Maragos, "Audiovisual-to- articulatory speech inversion using HMMS, " in Proc. Int. Workshop Multimedia Signal Process. (MMSP), 2007, pp. 457-460.
22. A. Katsamanis, G. Papandreou, and P. Maragos, "Audiovisual-to- articulatory speech inversion using active appearance models for the face and hidden Markov models for the dynamics, " in Proc. Int. Conf. Acoust., Speech, Signal Process., 2008, pp. 2237-2240.
23. T. F. Cootes, G. J. Edwards, and C. J. Taylor, "Active appearance models, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 6, pp. 681-685, Jun. 2001.
24. P. Viola and M. Jones, "Rapid object detection using a boosted cascade of simple features, " in Proc. IEEE Int. Conf. Comp. Vision Pattern Recog., 2001, vol. I, pp. 511-518.
25. S. Dusan and L. Deng, "Acoustic-to-articulatory inversion using dynamical and phonological constraints, " in Proc. Seminar Speech Production, 2000, pp. 237-240.
26. S. Dupont and J. Luettin, "Audio-visual speech modeling for continuous speech recognition, " IEEE Trans. Multimedia, vol. 2, no. 3, pp. 141-151, Sep. 2000.
27. A. Wrench and W. Hardcastle, "A multichannel articulatory speech database and its application for automatic speech recognition, " in Proc. 5th Seminar Speech Production, Kloster Seeon, Bavaria, 2000, pp. 305-308. .
28. O. Engwall and J. Beskow, "Resynthesis of 3D tongue movements from facial data, " in Proc. Eur. Conf. Speech Commun. Technol., 2003, pp. 2261-2264.
29. K. V. Mardia, J. T. Kent, and J. M. Bibby, Multivariate Analysis. New York: Academic, 1979.
30. L. L. Scharf and J. K. Thomas, "Wiener filters in canonical coordinates for transform coding, filtering, and quantizing, " IEEE Trans. Speech Audio Process., vol. 46, no. 3, pp. 647-654, May 1998.
31. L. Breiman and J. H. Friedman, "Predicting multivariate responses in multiple linear regression, " J. Roy. Statist. Soc. (B), vol. 59, no. 1, pp. 3-54, 1997.
32. C. Bishop, Pattern Recognition and Machine Learning. New York: Springer, 2006.
33. A. Katsamanis, G. Ananthakrishnan, G. Papandreou, P. Maragos, and O. Engwall, "Audiovisual speech inversion by switching dynamical modeling governed by a hidden Markov process, " in Proc. Eur. Signal Process. Conf. (EUSIPCO), 2008, CD-ROM.
34. L. Rabiner and B. Juang, Fundamentals of Speech Recognition. Englewood Cliffs, NJ: Prentice-Hall, 1993.
35. W. DeSarbo and W. Cron, "A maximum likelihood methodology for clusterwise linear regression, " J. Classification, vol. 5, pp. 249-282, 1988.
36. Speechreading by Humans and Machines, D. Stork and M. E. Hennecke, Eds. Berlin, Germany: Springer, 1996.
37. J. Luettin, G. Potamianos, and C. Neti, "Asynchronous stream modeling for large vocabulary audio-visual speech recognition, " in Proc. Int. Conf. Acoust., Speech, Signal Process., 2001, pp. 169-172.
38. G. Papandreou and P. Maragos, "Adaptive and constrained algorithms for inverse compositional active appearance model fitting, " in Proc. IEEE Int. Conf. Comp. Vision and Patern Recog., 2008.
39. A. Fitzgibbon, M. Pilu, and R. Fisher, "Direct least square fitting of ellipses, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 21, no. 5, pp. 476-480, May 1999.
40. M. E. Sargin, Y. Yemez, E. Erzin, and M. Tekalp, "Audiovisual synchronization and fusion using canonical correlation analysis, " IEEE Trans. Multimedia, vol. 9, no. 7, pp. 1396-1403, Nov. 2007.
41. S. Young, G. Evermann, D. Kershaw, G. Moore, J. Odell, D. Ollason, D. Povey, V. Valtchev, and P. Woodland, The HTK Book (for HTK version 3.2) Cambridge Univ. Eng. Dept., Tech. Rep, 2002.
42. F. K. Soong and B.-H. Juang, "Line spectrum pair and speech data compression, " in Proc. Int. Conf. Acoust., Speech Signal Process, 1984, vol. 9, pp. 37-40.
43. C. Qin and M. Carreira-Perpinan, "A comparison of acoustic features for articulatory inversion, " in Proc. Int. Conf. Spoken Lang. Process., 2007, pp. 2469-2472.
44. T. J. Hazen, "Visual model structures and synchrony constraints for audio-visual speech recognition, " IEEE Trans. Speech Audio Process., vol. 14, no. 3, pp. 1082-1089, May 2006.
45. M.-S. Tso, "Reduced-rank regression and canonical analysis, " J. R. Statist. Soc. (B), vol. 43, pp. 183-189, 1981.