Constrained optimization for audio-to-visual conversion

IEEE Transactions on Signal Processing

Volumn 52, Issue 6, 2004, Pages 1783-1790

  Choi, Kyoung Ho ^a   Hwang, Jenq Neng ^b  

^a Electronics and Telecommunications Research Institute (ETRI)   (South Korea)

^b University of Washington   (United States)

Author keywords

Audio to visual conversion; HMM; HMMI; Talking heads

Indexed keywords

ALGORITHMS; COMPUTER SIMULATION; CONSTRAINT THEORY; IMAGE ANALYSIS; LAGRANGE MULTIPLIERS; MARKOV PROCESSES; NEURAL NETWORKS; OPTIMIZATION; PARAMETER ESTIMATION; PROBABILITY DISTRIBUTIONS; VECTOR QUANTIZATION;

AUDIO TO VISUAL CONVERSION; CONSTRAINED OPTIMIZATION; HIDDEN MARKOV MODELS; TALKING HEADS;

SPEECH PROCESSING;

EID: 2942596586     PISSN: 1053587X     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSP.2004.827153     Document Type: Article

References (25)

1. ISO/IEC FDIS 14 496-1 Systems, ISO/IEC JTC1/SC29/WG11 N2501, Nov. 1998.
2. ISO/IEC FDIS 14 496-2 Visual, ISO/IEC JTC1/SC29/WG11 N2502, Nov. 1998.
3. T. Noma, L. Zhao, and N. I. Bradler, "Design of a virtual human presenter," IEEE Comput. Graphics Applicat., vol. 20, no. 4, pp. 79-85, 2000.
4. K. Kiyokawa, H. Takemura, and N. Yokoya, "SeamlessDesign: a face-to-face collaborative virtual/augmented environment for rapid prototyping of geometrically constrained 3-D objects," in Proc. IEEE Int. Conf. Multimedia Comput. Syst., vol. 2, 1999, pp. 447-453.
5. Y.-J. Chang, C.-C. Chen, J.-C. Chou, and Y.-C. Chen, "Implementation of a virtual chat room for multimedia communications," in Proc. IEEE 3rd Workshop Multimedia Signal Process., 1999, pp. 599-604.
6. S. Yura, T. Usaka, and K. Sakamura, "Video avatar: embedded video for collaborative virtual environment," in Proc. IEEE Int. Conf. Multimedia Comput. Syst., vol. 2, 1999, pp. 433-438.
7. S. Morishima and H. Harashima, "A media conversion from speech to facial image for intelligent man-machine interface," IEEE J. Select. Areas Commun., vol. 9, pp. 594-600, May 1991.
8. F. Lavagetto, "Converting speech into lip movement: a multimedia telephone for hard of hearing people," IEEE Trans. Rehab. Eng., vol. 3, pp. 90-102, Jan. 1995.
9. ____, "Time-delay neural networks for estimating lip movements from speech analysis: a useful tool in audio-video synchronization," IEEE Trans. Circuits Syst. Video Technol., vol. 7, pp. 786-800, May 1997.
10. R. R. Rao, T. Chen, and R. M. Mersereau, "Audio-to-visual conversion for multimedia communication," IEEE Trans. Ind. Electron., vol. 45, pp. 15-22, Jan. 1998.
11. S.Satoshi Nakamura and E.Eli Yamamoto, "Speech-to-lip movement synthesis by maximizing audio-visual joint probability based on the EM algorithm," J. VLSI Signal Process., vol. 27, pp. 119-126, 2001.
12. S. Nakamura, E. Yamamoto, and K.Kiyohiro Shikano, "Speech-to-lip movement synthesis maximizing audio-visual joint probability based on EM algorithm," in Proc. IEEE 2nd Workshop Multimedia Signal Process., 1998, pp. 53-58.
13. B. Fleming and D. Dobbs, Animating Facial Features and Expressions. Boston, MA: Charles River Media, 1999, pp. 53-78.
14. S. Y. Moon and J. N. Hwang, "Noisy speech recognition using robust inversion of hidden Markov models," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 1995, pp. 145-148.
15. ____, "Robust speech recognition based on joint model and feature space optimization of hidden Markov models," IEEE Trans. Neural Networks, vol. 8, pp. 194-204, Mar. 1997.
16. B. A. Carlson and M. A. Clements, "A projection-based likelihood measure for speech recognition in noise," IEEE Trans. Speech Audio Processing, vol. 2, pp. 97-102, Jan. 1994.
17. L. R. Rabiner and B. H. Juang, Fundamentals of Speech Recognition. Englewood Cliffs, NJ: Prentice-Hall, 1993, pp. 321-389.
18. D. P. Bertsekas, Constrained Optimization and Lagrange Multiplier Methods, Newe York: Academic, 1982.
19. Y. Li, F. Yu, Y.-Q. Xu, E. Chang, and H.-Y. Shum, "Speech-driven cartoon animation with emotions," in Proc. ACM Multimedia; 9th ACM Int. Multimedia Conf., Ottawa, ON, Canada, Sept. 30th-Oct. 5th 2001.
20. F. Dellaert, T. Polzin, and A. Waibel, "Recognizing emotion in speech," in Proc. IEEE Int. Conf. Spoken Language, vol. 3, 1996, pp. 1970-1973.
21. R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias, W. Fellenz, and J. G. Taylor, "Emotion recognition in human-computer interaction," IEEE Signal Processing Mag., vol. 18, no. 1, pp. 32-80, 2001.
22. L. S. Chen, and T. S. Huang, "Emotional expressions in audiovisual human computer interaction," in Proc. IEEE Int. Conf. Multimedia Expo, vol. 1, 2000, pp. 423-426.
23. [Online] http://htk.eng.cam.ac.uk
24. K. H. Choi and J.-N.Jenq-Neng Hwang, "Creating 3D speech-driven talking heads: a probabilistic approach," in Proc. IEEE Int. Conf. Image Processing, 2002, pp. 984-987.
25. K. H. Choi, J. J. Yoo, T. H. Hwang, J. H. Park, and J. H. Lee, "A probabilistic network for facial feature verification," ETRI J., vol. 25, no. 2, pp. 140-143, 2003.