Real-time continuous phoneme recognition system using class-dependent tied-mixture HMM with HBT structure for speech-driven lip-sync

IEEE Transactions on Multimedia

Volumn 10, Issue 7, 2008, Pages 1299-1306

  Park, Junho ^a,b   Ko, Hanseok ^a  

^a Korea University   (South Korea)

^b UNIVERSITY OF CAMBRIDGE   (United Kingdom)

Author keywords

Head body tail HMM; Phoneme recognition; Real time lip sync

Indexed keywords

MARKOV PROCESSES; SPEECH; SPEECH RECOGNITION;

CODEBOOK; CONTINUOUS HIDDEN MARKOV MODELS; GAUSSIANS; HBT MODELS; HEAD-BODY-TAIL HMM; INDEPENDENT MODELS; MODEL PARAMETERS; PHONEME RECOGNITION; PHONEME RECOGNITION SYSTEMS; PHONEME RECOGNITIONS; REAL-TIME LIP-SYNC; RECOGNITION PERFORMANCES; SPEECH SIGNALS; TRANSIENT PARTS; VOWEL RECOGNITIONS;

HIDDEN MARKOV MODELS;

EID: 56549088313     PISSN: 15209210     EISSN: None     Source Type: Journal    
DOI: 10.1109/TMM.2008.2004908     Document Type: Article

References (30)

1. B. E. Koster, R. D. Rodman, and D. Bitzer, "Automated lip-sync: Direct translation of speech-sound to mouth-shape," in Proc. 28th Annu. Asilomar Conf. Signals, 1994, pp. 583-586.
2. D. V. McAllister et al., "Lip synchronization for animation," in Proc. SIGGRAPH 97, Los Angeles, CA, 1997, pp. 225-228.
3. M. Tamura et al., "Visual speech synthesis based on parameter generation from HMM: Speech driven and text-and-speech driven approaches," in Proc. AVSP 98, 1998, pp. 221-226.
4. E. Yamamoto et al., "Lip movement synthesis from speech based on Hidden Markov models," Speech Commun., vol. 26, no. 1-2, pp. 105-115, 1998.
5. S. Morishima, "Real-time talking head driven by voice and its application to communication and entertainment," in Proc. AVSP 98, 1998, pp. 195-200.
6. S. Curinga, F. Lavagetto, and F. Vignoli, "Lip movements synthesis using time delay neural networks," in Proc. EUSIPCO 96 - Systems and Computers, 1996, pp. 36-46.
7. J. Lewis, "Automatic lip-sync: Background and techniques," Visualiz. and Comput. Anim., vol. 2, no. 4, pp. 118-122, 1991.
8. T. Kim, Y. Kang, and H. Ko, "Achieving real-time lip synch via SVM-based phoneme classification and lip shape refinement," in Proc. ICMI'02, 2002, pp. 299-304.
9. W. Chou et al., "Minimum error rate training of inter-word context-dependent acoustic model units in speech recognition," in Proc. ICSLP, 1994, pp. 439-142.
10. M. B. Gandhi and J. Jacob, "Natural number recognition using MCE trained inter-word context-dependent acoustic models," in Proc. ICASSP, 1998, pp. 457-460.
11. J. Strum and E. Sanders, "Modeling phonetic context using head-body-tail models for connected digit recognition," in Proc. ICSLP, 2000, pp. 429-432.
12. J. Park and H. Ko, "A new state-dependent phonetic tied-mixture model with head-body-tail structured HMM for real-time continuous phoneme recognition system," in Proc. INTERSPEECH, 2006, pp. 1583-1586.
13. T. Chen and R. Rao, "Audio-visual integration in multimodal communication," Proc. IEEE (Special Issue on Multimedia Signal Processing), vol. 86, no. 5, pp. 837-852, May 1998.
14. F. J. Huang and T. Chen, "Real-time lip-sync face animation driven by human voice," in Proc. IEEE Workshop on Multimedia Signal Processing, 1998, pp. 352-357.
15. M. Brand, "Voice puppetry," in Proc. SIGGRAPH'99, 1999, pp. 21-28.
16. G. Zoric and I. S. Pandzic, "Automatic lip sync and its use in the new multimedia services for mobile devices," in Proc. 8th Int. Conf. Telecommunications, 2005, vol. 2, pp. 353-358.
17. K. Chio et al., "Hidden Markov model inversion for audio-to-visual conversion in an MPEG-4 facial animation system," J. VLSI Signal Process., vol. 29, pp. 51-61, 2001.
18. E. Cosatto et al., "Lifelike talking faces for interactive services," Proc. IEEE, vol. 91, no. 9, pp. 1406-1429, Sep. 2003.
19. L. Xie and Z. Liu, "Realistic mouth-synching for speech-driven talking face using articulatory modeling," IEEE Trans. Multimedia, vol. 9, no. 3, pp. 500-510, Apr. 2007.
20. R. Andre-Obrecht, "A new statistical approach for the automatic segmentation of continuous speech signals," IEEE Trans. Acoust., Speech, Signal Process., vol. 36, no. 1, pp. 29-40, Jan. 1998.
21. A. Von Brandt, "Detecting and estimating parameters jumps using ladder algorithms and likelihood ratio test," in Proc. ICASSP, 1983, pp. 1017-1020.
22. S. E. G. Ohman, "Co-articulation in VCV utterances: Spectrographic measurements," J. Acoust. Soc. Amer., vol. 31, pp. 151-168, 1966.
23. J. R. Bellegarda and D. Nahamoo, "Tied mixture continuous parameter modeling for speech recognition," IEEE Trans. Acoust. Speech Signal Process., vol. 38, no. 12, pp. 2033-2045, Dec. 1990.
24. J. Park and H. Ko, "Achieving a reliable compact acoustic model for embedded speech recognition system with high confusion frequency model handling," Speech Commun., vol. 48, no. 6, pp. 737-745, 2006.
25. J. Frreiros and J. M. Pardo, "Improving continuous speech recognition in Spanish by phone-class semi continuous HMMs with pausing and multiple pronunciations," Speech Commun., vol. 29, pp. 65-76, 1999.
26. M. M. Cohen and D. W. Massaro, "Modeling coarticulation in synthetic visual speech," in Proc. Computer Animation '93, 1993, pp. 139-156.
27. G. Bailly, "Audiovisual speech synthesis from ground truth to models," in Proc. ICSLP, 2002, pp. 1453-1456.
28. S. J. Young, J. Odell, D. Ollason, V. Valtchev, and P. Woodland, HTK Book/Tech. Rep. Speech Group, Eng. Dept., Cambridge Univ., Cambridge, U.K., 2002.
29. G. Zoric and I. S. Pandzic, "Real-time language independent lip synchronization method using a genetic algorithm," Signal Process., vol. 86, pp. 3644-3656, 2006.
30. H. Ko and D. K. Han, "SVM-based phoneme classification and lip shape refinement in real-time lip-sync system," Int. J. Pattern Recognit. Artific. Intell., vol. 20, no. 7, pp. 1029-1051, Nov. 2006.