Gesture modeling and animation based on a probabilistic re-creation of speaker style

ACM Transactions on Graphics

Volumn 27, Issue 1, 2008, Pages

  Neff, Michael ^a   Kipp, Michael ^b   Albrecht, Irene ^c   Seidel, Hans Peter ^c  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b GERMAN RESEARCH CENTER FOR ARTIFICIAL INTELLIGENCE DFKI   (Germany)

^c MAX PLANCK INSTITUTE FOR INFORMATICS   (Germany)

Author keywords

Character animation; Gesture; Human modeling

Indexed keywords

CHARACTER ANIMATION; GESTURE MODELING; HUMAN MODELING; SPEAKER STYLE;

ALGORITHMS; ANIMATION; GESTURE RECOGNITION; SPEECH ANALYSIS;

COMPUTER GRAPHICS;

EID: 41349085864     PISSN: 07300301     EISSN: 15577368     Source Type: Journal    
DOI: 10.1145/1330511.1330516     Document Type: Article

References (58)

1. ALBRECHT, I., HABER, J., AND SEIDEL, H.-P. 2002. Automatic generation of non-verbal facial expressions from speech. In Proceedings of the Computer Graphics International. 283-293.
2. AXTELL, R. E. 1998. Gestures - The Do 'sand Taboo's of Body Language Around the World. John Wiley & Sons, Inc., New York.
3. BADLER, N. I., PHILLIPS, C. B., AND WEBBER, B. L. 1992. Simulating Humans: Computer Graphics, Animation and Control. Oxford University Press.
4. BOERSMA, P. AND WEENINK, D. 2005. Praat: Doing phonetics by computer (version 4.3.1.4). http://www.praat.org/.
5. CALBRIS, G. 1990. Semiotics of French. Gesture. Indiana University Press, Bloomington, IN.
6. CASSELL, J., NAKANO, Y., BICKMORE, T., SIDNER, C., AND RICH, C. 2001. Non-verbal cues for discourse structure. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. 106-115.
7. CASSELL, J., PELACHAUD, C., BADLER, N., STEEDMAN, M., ACHORN, B., BECKET, T., DOUVILLE, B., PREVOST, S., AND STONE, M. 1994. Animated conversation: Rule-based generation of facial expression, gesture and spoken intonation for multiple conversational agents. In Proceedings of SIGGRAPH. 413-420.
8. CASSELL, J., VILHJÁLMSSON, H., AND BICKMORE, T. 2001. BEAT: The Behavior Expression Animation Toolkit. In Proceedings of SIGGRAPH. 477-486.
9. CHI, D. M., COSTA, M., ZHAO, L., AND BADLER, N. I. 2000. The EMOTE model for effort and shape. In Proceedings of SIGGRAPH. 173-182.
10. COHEN, M. AND MASSARO, D. 1993. Modeling coarticulation in synthetic visual speech. In Models and Techniques in Computer Animation, N. Magnenat-Thalmann and D. Thalmann, Eds. Springer. 139-156.
11. DE RUTTER, J. 2000. The production of gesture and speech. In Language and Gesture: Window into Thought and Action, D. McNeill, Ed. Cambridge University Press, Cambridge, UK. 284-311.
12. FALOUTSOS, P., VAN DE PANNE, M., AND TERZOPOULOS, D. 2001. The virtual stuntman: Dynamic characters with a repertoire of autonomous motor skills. Comput. Graph. 25, 6, 933-953.
13. FINKLER, W. AND NEUMANN, G. 1988. MORPHIX. A Fast Realization of a Classification-Based Approach to Morphology. In Proceedings 4. Österreichische Artificial-Intelligence- Tagung. Wiener Workshop -Wissensbasierte Sprachverarbeitung. H. Trost, Ed. Springer. 11-19.
14. FREY, S. 1999. Die Macht des Bildes: der Einfluß der nonverbalen Kommunikation auf Kultur und Politik. Verlag Hans Huber, Bern.
15. HARTMANN, B., MANCINI, M., BUISINE, S., AND PELACHAUD, C. 2005. Design and evaluation of expressive gesture synthesis for embodied conversational agents. In Proceedings of the 4th International Joint Conference on Autonomous Agents and Multiagent Systems. ACM Press.
16. HARTMANN, B., MANCINI, M., AND PELACHAUD, C. 2002. Formational parameters and adaptive prototype installation for MPEG-4 compliant gesture synthesis. In Proceedings of Computer Animation. 111-119.
17. HARTMANN, B., MANCINI, M., AND PELACHAUD, C. 2006. Implementing expressive gesture synthesis for embodied conversational agents. In Proceedings of Gesture Workshop. Lecture Notes Artificial Intelligence, vol. 3881. Springer. 45-55.
18. HODGINS, J. K., WOOTEN, W. L., BROGAN, D. C., AND O'BRIEN, J. F. 1995. Animating human athletics. In Proceedings of SIGGRAPH. 71-78.
19. HOLLARS, M. G., ROSENTHAL, D. E., AND SHERMAN, M. A. 1994. SD/FAST User's Manual. Symbolic Dynamics Inc.
20. HUENERFAUTH, M., ZHOU, L., GU, E., AND ALLBECK, J. 2007. Design and evaluation of an american sign language generator. In Proceedings of the Workshop on Embodied Language Processing. Association for Computational Linguistics. 51-58.
21. JURAFSKY, D. AND MARTIN, J. H. 2003. Speech and Language Processing. Prentice Hall.
22. KENDON, A. 1980. Gesticulation and speech: Two aspects of the process of utterance. In The Relationship of Verbal and Nonverbal Communication, M. Key, Ed. Mouton Publisher, The Hague, Netherlands. 207-227.
23. KENDON, A. 2004. Gesture - Visible Action as Utterance. Cambridge University Press, Cambridge.
24. KIPP, M. 2001. Anvil - a Generic Annotation Tool for Multimodal Dialogue. In Proceedings of the 7th European Conference on Speech Communication and Technology (Eurospeech). Aalborg, Denmark. 1367-1370.
25. KIPP, M. 2004. Gesture generation by imitation: From human behavior to computer character animation. Dissertation.com., Boca Raton, FL.
26. KIPP, M., NEFF, M., AND ALBRECHT, I. 2006. An Annotation Scheme for Conversational Gestures : How to economically capture timing and form. In Proceedings of the Workshop on Multimodal Corpora (LREC'06). 24-27.
27. KIPP, M., NEFF, M., KIPP, K., AND ALBRECHT, I. 2007. Towards natural gesture synthesis: Evaluating gesture units in a data-driven approach to gesture synthesis. In Proceedings of Intelligent Virtual Agents (IVA'07). Lecture Notes in Artificial Intelligence, vol. 4722, 15-28.
28. KITA, S., VAN GIJN, I., AND VAN DER HULST, H. 1998. Movement phases in signs and co-speech gestures, and their transcription by human coders. In Gesture and Sign Language in Human-Computer Interaction, I. Wachsmuth and M. Fröhlich, Eds. Springer, 23-35.
29. KOPP, S., SOWA, T., AND WACHSMUTH, I. 2004a. Imitation games with an artificial agent: From, mimicking to understanding shape-related iconic gestures. In Proceedings of the Gesture Workshop. Lecture Notes in Artificial Intelligence, vol. 2915. Springer. 436-447.
30. KOPP, S., TEPPER, P., AND CASSELL, J. 2004b. Towards integrated microplanning of language and iconic gesture for multimodal output. In Proceedings of the International Conference on Multimodal Interfaces. 97-104.
31. KOPP, S. AND WACHSMUTH, I. 2004c. Synthesizing multimodal utterances for conversational agents. Comput. Anim. Virt. Worlds 15, 39-52.
32. MANN, W. C. AND THOMPSON, S. A. 1988. Rhetorical Structure Theory: Toward a functional theory of text organization. Text 8, 3, 243-281.
33. MARTELL, C. H. 2004. Form: An extensible, kinematically based gesture annotation scheme. In Natural, Intelligent and Effective Interaction in Multimodal Dialogue Systems. Kluwer Academic Press.
34. MARTIN, J.-C., NIEWIADOMSKI, R., DEVILLERS, L., BUISINE, S., AND PELACHAUD, C. 2006. Multimodal complex emotions: Gesture expressivity and blended facial expressions. J. Huma. Robot. 3, 269-291.
35. MCNEILL, D. 1992. Hand and Mind: What Gestures Reveal about Thought. University of Chicago Press, Chicago, IL.
36. MCNEILL, D. 2005. Gesture and Thought. University of Chicago Press, Chicago, IL.
37. MILLER, G. A., BECKWTTH, R., FELBAITM, C., GROSS, D., AND MILLER, K. 1990. Introduction to WordNet: an online lexical database. Int. J. Lexicog. 3, 4, 235-244.
38. NEFF, M. 2005. Aesthetic exploration and refinement: A computational framework for expressive character animation. Ph.D. Dissertation. Department of Computer Science, University of Toronto.
39. NEFF, M. AND FIUTME, E. 2002. Modeling tension and relaxation for computer animation. In Proceedings of ACM SIGGRAPH Symposium on Computer Animation. 81-88.
40. NEFF, M. AND FIUME, E. 2005. AER: Aesthetic Exploration and Refinement for expressive character animation. In Proceedings of ACM SIGGRAPH/Eurographics Symposium on Computer Animation. 161-170.
41. NEFF, M. AND FIUME, E. 2006. Methods for exploring expressive stance. Graphic. Models 68, 2, 133-157.
42. NEFF, M. AND SEIDEL, H.-P. 2006. Modeling relaxed hand shape for character animation. In Articulated Motion and Deformable Objects (AMDO'06). Lecture Notes in Computer Science, vol. 4069. Springer.
43. NOMA, T., ZHAO, L., AND BADLER, N. 2000. Design of a virtual human presenter. IEEE Comput. Graph. Appl. 20, 4, 79-85.
44. NOOT, H. AND RUTTKAY, Z. 2004. Gesture in style. In Proceedings of Gesture Workshop. Lecture Notes in Artificial Intelligence, vol. 2915. Springer, 324-337.
45. POLLARD, N. S. AND ZORDAN, V. B. 2005. Physically based grasping control from example. In Proceedings of ACM. SIGGRAPH/Eurographics Symposium on Computer Animation. 311-318.
46. POPOVIC, Z. AND WITKIN, A. 1999. Physically based motion transformation. In Proceedings of SIGGRAPH. 11-20.
47. PRESS, W. H., TUKOLSKY, S. A., VETTERLING, W. T., AND FLANNERY, B. P. 1992. Numerical Recipes in C: The Art of Scientific Computing, 2nd ed. Cambridge University Press, Cambridge, UK.
48. SAITZ, R. L. AND CERVENKA, E. J. 1972. Handbook of Gestures: Colombia and the United States, second ed. Mouton, The Hague, The Netherlands.
49. SCHEFLEN, A. E. 1964. The significance of posture in communication systems. Psychiatry 26, 316-331.
50. SCHEGLOFF, E. A. 1984. On some gestures' relation to talk. In Structures of Social Action. J. M. Atkinson and J. Heritage, Eds. Cambridge University Press, Cambridge, UK. 266-296.
51. SHAPIRO, A., FALOUTSOS, P., AND NG-THOW-HING, V. 2005. Dynamic animation and control environment. Graphics Interface, 61-70.
52. STEEDMAN, M. 2000. Information structure and the syntax-phonology interface. Linguist. Inq. 34, 649-689.
53. STONE, M., DECARLO, D., OH, I., RODRIGUEZ, C., STERE, A., LEES, A., AND BREGLER, C. 2004. Speaking with hands: Creating animated conversational characters from recordings of human performance. In Proceedings of SIGGRAPH. 506-513.
54. TAN, R. AND DAVIS, J. 2004. Differential video coding of face and gesture events in presentation videos. Comput. Vision Image Understand. 96, 2, 200-215.
55. WEBB, R. 1997. Linguistic Properties of Metaphoric Gestures. UMI, New York, NY.
56. WITKIN, A. AND KASS, M. 1988. Spacetime constraints. In Proceedings of SIGGRAPH. 159-168.
57. WITKIN, A. AND POPOVIC, Z. 1995. Motion warping. In Proceedings of SIGGRAPH. 105-108.
58. ZORDAN, V. B. AND HODGINS, J. K. 2002. Motion capture-driven simulations that hit and react. In Proceedings of ACM. SIGGRAPH Symposium on Computer Animation, 89-96.