Real-time prosody-driven synthesis of body language

ACM Transactions on Graphics

Volumn 28, Issue 5, 2009, Pages

  Levine, Sergey ^a   Theobalt, Christian ^a   Koltun, Vladlen ^a  

^a STANFORD UNIVERSITY   (United States)

Author keywords

Control; Data driven animation; Gesture synthesis; Human animation; Nonverbal behavior generation

Indexed keywords

BODY LANGUAGE; BODY MOTIONS; CONTROL DATA; HUMAN ANIMATION; HUMAN COMMUNICATIONS; HUMAN SPEECH; INPUT DATAS; MOTION CAPTURE DATA; MULTI-MODAL; NONVERBAL BEHAVIOR; REAL TIME; SPEECH SIGNALS; SYNTHESIS PHASE; TRAINING PHASE; USER STUDY; VIRTUAL ENVIRONMENTS;

ANIMATION; HIDDEN MARKOV MODELS; INTERACTIVE COMPUTER GRAPHICS; LINGUISTICS; SPEECH COMMUNICATION; VIRTUAL REALITY;

SYNTHESIS (CHEMICAL);

EID: 77749295421     PISSN: 07300301     EISSN: 15577368     Source Type: Journal    
DOI: 10.1145/1661412.1618518     Document Type: Conference Paper

References (52)

1. ADOLPHS, R. 2002. Neural systems for recognizing emotion. Current Opinion in Neurobiology 12, 2, 169-177.
2. ALBRECHT, I., HABER, J., AND PETER SEIDEL, H. 2002. Automatic generation of non-verbal facial expressions from speech. In Computer Graphics International, 283-293.
3. ARIKAN, O., AND FORSYTH, D. A. 2002. Interactive motion generation from examples. ACM Transactions on Graphics 21, 3, 483-490.
4. BARBIČ , J., SAFONOVA, A., PAN, J.-Y., FALOUTSOS, C., HODGINS, J. K., AND POLLARD, N. S. 2004. Segmenting motion capture data into distinct behaviors. In Proceedings of Graphics Interface 2004, 185-194.
5. BESKOW, J. 2003. Talking Heads - Models and Applications for Multimodal Speech Synthesis. PhD thesis, KTH Stockholm.
6. BOERSMA, P. 1993. Accurate short-term analysis of the fundamental frequency and the harmonics-to-noise ratio of a sampled sound. In Proceedings of the Institute of Phonetic Sciences, vol. 17, 97-110.
7. BOERSMA, P. 2001. Praat, a system for doing phonetics by computer. Glot International 5, 9-10, 314-345.
8. BRAND, M. 1999. Voice puppetry. In SIGGRAPH '99: ACM SIGGRAPH 1999 Papers, ACM, New York, NY, USA, 21-28.
9. BREGLER, C., COVELL, M., AND SLANEY, M. 1997. Video rewrite: driving visual speech with audio. In SIGGRAPH '97: ACM SIGGRAPH 1997 Papers, ACM, New York, NY, USA, 353-360.
10. CASSELL, J., PELACHAUD, C., BADLER, N., STEEDMAN, M., ACHORN, B., BECKET, T., DOUVILLE, B., PREVOST, S., AND STONE, M. 1994. Animated conversation: rule-based generation of facial expression, gesture & spoken intonation for multiple conversational agents. In SIGGRAPH '94: ACM SIGGRAPH 1994 Papers, ACM, New York, NY, USA, 413-420.
11. CASSELL, J., VILHJÁLMSSON, H. H., AND BICKMORE, T. 2001. Beat: the behavior expression animation toolkit. In SIGGRAPH '01: ACM SIGGRAPH 2001 Papers, 477-486.
12. CASSELL, J. 2000. Embodied conversational interface agents. Communications of the ACM 43, 4, 70-78.
13. CHUANG, E., AND BREGLER, C. 2005. Mood swings: expressive speech animation. ACM Transactions on Graphics 24, 2, 331-347.
14. EZZAT, T., GEIGER, G., AND POGGIO, T. 2002. Trainable videorealistic speech animation. In SIGGRAPH '02: ACM SIGGRAPH 2002 Papers, ACM, New York, NY, USA, 388-398.
15. FOD, A., MATARIC, M. J., AND JENKINS, O. C. 2002. Automated derivation of primitives for movement classification. Autonomous Robots 12, 39-54.
16. HARTMANN, B., MANCINI, M., AND PELACHAUD, C. 2002. Formational parameters and adaptive prototype instantiation for mpeg-4 compliant gesture synthesis. In Proceedings on Computer Animation, IEEE Computer Society, Washington, DC, USA, 111.
17. JENSEN, C., FARNHAM, S. D., DRUCKER, S. M., AND KOLLOCK, P. 2000. The effect of communication modality on cooperation in online environments. In Proceedings of CHI '00, ACM, New York, NY, USA, 470-477.
18. JU, E., AND LEE, J. 2008. Expressive facial gestures from motion capture data. Computer Graphics Forum 27, 2, 381-388.
19. KENDON, A. 2004. Gesture - Visible Action as Utterance. Cambridge University Press, New York, NY, USA.
20. KIPP, M., NEFF, M., AND ALBRECHT, I. 2007. An annotation scheme for conversational gestures: how to economically capture timing and form. Language Resources and Evaluation 41, 3-4 (December), 325-339.
21. KOPP, S., AND WACHSMUTH, I. 2004. Synthesizing multimodal utterances for conversational agents: Research articles. Computer Animation and Virtual Worlds 15, 1, 39-52.
22. KOVAR, L., GLEICHER, M., AND PIGHIN, F. 2002. Motion graphs. ACM Transactions on Graphics 21, 3, 473-482.
23. KRAUSS, R. M., DUSHAY, R. A., CHEN, Y., AND RAUSCHER, F. 1995. The communicative value of conversational hand gestur. Journal of Experimental Social Psychology 31, 6, 533-552.
24. LEE, J., CHAI, J., REITSMA, P. S. A., HODGINS, J. K., AND POLLARD, N. S. 2002. Interactive control of avatars animated with human motion data. ACM Transactions on Graphics 21, 3, 491-500.
25. LI, Y., AND SHUM, H.-Y. 2006. Learning dynamic audio-visual mapping with input-output hidden markov models. IEEE Transactions on Multimedia 8, 3, 542-549.
26. LI, Y., WANG, T., AND SHUM, H.-Y. 2002. Motion texture: a two-level statistical model for character motion synthesis. In SIGGRAPH '02: ACM SIGGRAPH 2002 Papers, ACM, New York, NY, USA, 465-472.
27. LOEHR, D. 2004. Gesture and Intonation. PhD thesis, Georgetown University.
28. MAERAN, O., PIURI, V., AND STORTI GAJANI, G. 1997. Speech recognition through phoneme segmentation and neural classification. Instrumentation and Measurement Technology Conference, 1997. IMTC/97. Proceedings. 'Sensing, Processing, Networking.', IEEE 2 (May), 1215-1220.
29. MAJKOWSKA, A., ZORDAN, V. B., AND FALOUTSOS, P. 2006. Automatic splicing for hand and body animations. In Proceedings of the 2006 ACM SIGGRAPH/Eurographics symposium on Computer animation, 309-316.
30. MCNEILL, D. 1992. Hand and Mind: What Gestures Reveal About Thought. University Of Chicago Press.
31. MCNEILL, D. 2005. Gesture and Thought. University Of Chicago Press, November.
32. MONTEPARE, J., KOFF, E., ZAITCHIK, D., AND ALBERT, M. 1999. The use of body movements and gestures as cues to emotions in younger and older adults. Journal of Nonverbal Behavior 23, 2, 133-152.
33. MORENCY, L.-P., SIDNER, C., LEE, C., AND DARRELL, T. 2007. Head gestures for perceptual interfaces: The role of context in improving recognition. Artificial Intelligence 171, 8-9, 568-585.
34. MÜLLER, M., RÖDER, T., AND CLAUSEN, M. 2005. Efficient content-based retrieval of motion capture data. In SIGGRAPH '05: ACM SIGGRAPH 2005 Papers, ACM, New York, NY, USA, vol. 24, 677-685.
35. NEFF, M., KIPP, M., ALBRECHT, I., AND SEIDEL, H.-P. 2008. Gesture modeling and animation based on a probabilistic recreation of speaker style. ACM Transactions on Graphics 27, 1, 5:1-24.
36. PARK, J., AND KO, H. 2008. Real-time continuous phoneme recognition system using class-dependent tied-mixture hmm with hbt structure for speech-driven lip-sync. IEEE Transactions on Multimedia 10, 7 (Nov.), 1299-1306.
37. PERLIN, K., AND GOLDBERG, A. 1996. Improv: a system for scripting interactive actors in virtual worlds. In SIGGRAPH '96: ACM SIGGRAPH 1996 Papers, ACM, 205-216.
38. PERLIN, K. 1997. Layered compositing of facial expression. In SIGGRAPH '97: ACM SIGGRAPH 97 Visual Proceedings, ACM, New York, NY, USA, 226-227.
39. RABINER, L. 1989. A tutorial on hidden markov models and selected applications in speech recognition. Proceedings of the IEEE 77, 2, 257-286.
40. RAMSHAW, L. A., AND MARCUS, M. P. 1995. Text chunking using transformation-based learning. In Proceedings of the Third Workshop on Very Large Corpora, 82-94.
41. SARGIN, M. E., ERZIN, E., YEMEZ, Y., TEKALP, A. M., ERDEM, A., ERDEM, C., AND OZKAN, M. 2007. Prosody-driven head-gesture animation. In ICASSP '07: IEEE International Conference on Acoustics, Speech, and Signal Processing.
42. SCHERER, K. R., BANSE, R., WALLBOTT, H. G., AND GOLDBECK, T. 1991. Vocal cues in emotion encoding and decoding. Motivation and Emotion 15, 2, 123-148.
43. SCHRÖDER, M. 2004. Speech and Emotion Research: An overview of research frameworks and a dimensional approach to emotional speech synthesis. PhD thesis, Phonus 7, Research Report of the Institute of Phonetics, Saarland University.
44. SHOEMAKE, K. 1985. Animating rotation with quaternion curves. In SIGGRAPH '85: ACM SIGGRAPH 1985 Papers, ACM, New York, NY, USA, 245-254.
45. SIFAKIS, E., SELLE, A., ROBINSON-M OSHER, A., AND FEDKIW, R. 2006. Simulating speech with a physics-based facial muscle model. In Proc. of ACM SIGGRAPH/Eurographics Symposium on Computer Animation, 261-270.
46. STONE, M., DECARLO, D., OH, I., RODRIGUEZ, C., STERE, A., LEES, A., AND BREGLER, C. 2004. Speaking with hands: creating animated conversational characters from recordings of human performance. In SIGGRAPH '04: ACM SIGGRAPH 2004 Papers, ACM, New York, NY, USA, 506-513.
47. TERKEN, J. 1991. Fundamental frequency and perceived prominence of accented syllables. The Journal of the Acoustical Society of America 89, 4, 1768-1777.
48. TREUILLE, A., LEE, Y., AND POPOVIĆ, Z. 2007. Near-optimal character animation with continuous control. In SIGGRAPH '07: ACM SIGGRAPH 2007 Papers, ACM, New York, NY, USA, 7.
49. WALLBOT, H. G. 1998. Bodily expression of emotion. European Journal of Social Psychology 28, 6, 879-896.
50. WANG, J., AND BODENHEIMER, B. 2008. Synthesis and evaluation of linear motion transitions. ACM Transactions on Graphics 27, 1 (Mar), 1:1-1:15.
51. WARD, J. H. 1963. Hierarchical grouping to optimize an objective function. Journal of the American Statistical Association 58, 301, 236-244.
52. XUE, J., BORGSTROM, J., JIANG, J., BERNSTEIN, L., AND ALWAN, A. 2006. Acoustically-driven talking face synthesis using dynamic bayesian networks. IEEE International Conference on Multimedia and Expo (July), 1165-1168.