Audio/visual mapping with cross-modal hidden Markov models

IEEE Transactions on Multimedia

Volumn 7, Issue 2, 2005, Pages 243-252

  Fu, Shengli ^a   Gutierrez Osuna, Ricardo ^b   Esposito, Anna ^c   Kakumanu, Praveen K ^d   Garcia, Oscar N ^e  

^a UNIVERSITY OF DELAWARE   (United States)

^b TEXAS A AND M UNIVERSITY   (United States)

^c SECOND UNIVERSITY OF NAPLES   (Italy)

^d Wright State University   (United States)

^e UNIVERSITY OF NORTH TEXAS   (United States)

Author keywords

3 D audio video processing; Joint media and multimodal processing; Speech reading and lip synchroization

Indexed keywords

ANIMATION; DATA ACQUISITION; FEATURE EXTRACTION; MAPPING; MARKOV PROCESSES; MATHEMATICAL MODELS; MODAL ANALYSIS; VIDEO SIGNAL PROCESSING;

3-D AUDIO-VIDEO PROCESSING; JOINT MEDIA AND MULTIMODAL PROCESSING; LIP SYNCHRONIZATION; SPEECH READING;

SPEECH COMMUNICATION;

EID: 16244385915     PISSN: 15209210     EISSN: None     Source Type: Journal    
DOI: 10.1109/TMM.2005.843341     Document Type: Review

References (32)

1. I. Pandzic, J. Ostermann, and D. Millen, "User evaluation: Synthetic talking faces for interactive services," Vis. Comput., vol. 15, no. 7-8, pp. 330-340, 1999.
2. M. M. Cohen, J. Beskow, and D. W. Massaro, "Recent developments in facial animation: An inside view," in Proc. AVSP, Terrigal, Australia, 1998, pp. 201-206.
3. L. R. Rabiner and B. H. Juang, Fundamentals of Speech Recognition Englewood Cliffs, NJ, 1993.
4. A. J. Goldschen, "Continuous Automatic Speech Recognition by Lipreading," Ph.D. dissertation, Eng. and App. Sci. Dept., George Washington Univ., Washington, NJ, 1993.
5. S. Morishima, "Real-time talking head driven by voice and its application to communication and entertainment," in Proc. AVSP, Terrigal, Australia, 1998, pp. 195-199.
6. C. Bregler, T. Covell, and M. Slaney, "Video rewrite: Driving visual speech with audio," in Proc. ACM SIGGRAPH'97, 1997, pp. 353-360.
7. S. Morishima and H. Harashima, "A media conversion from speech to facial image for intelligent man-machine interface," IEEE J Select. Areas Commun., vol. 9, no. 4, pp. 594-600, May 1991.
8. F. Lavagetto, "Converting speech into lip movements: A multimedia telephone for hard of hearing people," IEEE Trans. Rehabil. Eng., vol. 3, no. 1, pp. 90-102, Mar. 1995.
9. D. W. Massaro, J. Beskow, M. M. Cohen, C. L. Fry, and T. Rodriguez, "Picture my voice: Audio to visual speech synthesis using artificial neural networks," in Proc. AVSP, D. W. Massaro, Ed., Santa Cruz, CA, 1999, pp. 133-138.
10. P. Hong, Z. Wen, and T. S. Huang, "Real-time speech-driven face animation with expressions using neural networks," IEEE Trans. Neural Netw., vol. 13, no. 4, pp. 916-927, Jul. 2002.
11. T. Chen, "Audiovisual speech processing: Lip reading and lip synchronization," IEEE Signal Process. Mag., vol. 18, no. 1, pp. 9-21, Jan. 2001.
12. K. Choi, Y. Luo, and J.-N. Hwang, "Hidden Markov model inversion for audio-to-visual conversion in an MPEG-4 facial animation system," J. VLSI Signal Process., vol. 29, no. 1-2, pp. 51-61, 2001.
13. M. Brand, "Voice puppetry," in Proc. SIGGRAPH'99, Los Angeles, CA, 1999, pp. 21-28.
14. E. Yamamoto, S. Nakamura, and K. Shikano, "Lip movement synthesis from speech based on hidden Markov models," Speech Commun., vol. 26, no. 1-2, pp. 105-115, 1998.
15. R. R. Rao, T. Chen, and R. M. Mersereau, "Audio-to-visual conversion for multimedia communication," IEEE Trans. Ind. Electron., vol. 45, no. 1, pp. 15-22, Feb. 1998.
16. L. R. Rabiner, "A tutorial on hidden Markov models and selected applications in speech recognition," Proc. IEEE, vol. 77, no. 2, pp. 257-286, Feb. 1989.
17. M. Brand, "Shadow puppetry," in Proc. ICCV'99, Corfu, Greece, Sep. 1999, pp. 1237-1244.
18. B. N. Datta, Numerical Linear Algebra and Applications. Pacific Grove, CA: Brooks/Cole, 1995.
19. S. Fu, "Audio/Visual Mapping Based on Hidden Markov Models," Master's thesis, Dept. Comput. Sci. and Eng., Wright State Univ., Dayton, OH, 2002.
20. K. Choi and J.-N Hwang, "Baum-Welch HMM inversion for audio-to-visual conversion," in Proc. IEEE Int. Workshop Multimedia Signal Processing, 1999, pp. 175-180.
21. S. Moon and J.-N. Hwang, "Robust speech recognition based on joint model and feature space optimization of hidden Markov models," IEEE Tran. Neural Netw., vol. 8, no. 2, pp. 194-204, Mar. 1997.
22. L. E. Baum and G. R. Sell, "Growth functions for transformations on manifolds," Pacific J. Math., vol. 27, no. 2, pp. 211-227, 1968.
23. Y. Li and H.-Y. Shum, "Learning dynamic audio/visual mapping with input-output hidden Markov models," in Proc. 5th Asian Conf. on Computer Vision, Melbourne, Australia, Jan. 2002.
24. M. Brand, "Structure learning in conditional probability models via an entropic prior and parameter extinction," Neural Comput., vol. 11, no. 5, pp. 1155-1182, 1999.
25. F. I. Parke and K. Waters, Computer Facial Animation. Wesley, MA: A.K Peters, 1996.
26. P. K. Kakumanu, "Audio-Visual Processing for Speech Driven Facial Animation," Master, Dept. Comput. Sci. and Eng., Wright State Univ., Dayton, OH, 2002.
27. D. O'Shaughnessy, Speech Communication: Human and Machine, 2nd ed. New York: IEEE, 2000.
28. G. Aversano, A. Esposito, A. Esposito, and M. Marinaro, "A new textindependent method for phoneme segmentation," in Proc. 44th IEEE Midwest Symp. Circuits and Systems, vol. 2, 2001, pp. 516-519.
29. K. Fukunaga, Introduction to Statistical Pattern Recognition, 2nd ed. Boston, MA: Academic, 1990.
30. R. Koenen, "Overview of the MPEG-4 Standard," ISO/IEC JTC1/SC29/WG11, Seoul, South Korea, 1999.
31. R. Gutierrez-Osuna, P. K. Kakumanu, A. Esposito, O. N. Garcia, A. Bojorquez, J. L. Castillo, and I. J. Rudomin, "Speech-driven facial animation with realistic dynamics," IEEE Trans. Multimedia, vol. 7, no. 1, pp. 33-42, Feb. 2005.
32. J. S. Garofolo, Getting Started with the DARPA TIMIT CD-ROM: An Acoustic Phonetic Continuous Speech Database. Gaithersburg, MD: NIST, 1988.