Fully generated scripted dialogue for embodied agents

Artificial Intelligence

Volumn 172, Issue 10, 2008, Pages 1219-1244

  van Deemter, Kees ^a   Krenn, Brigitte ^b   Piwek, Paul ^c   Klesen, Martin ^d   Schröder, Marc ^d   Baumann, Stefan ^e  

^a UNIVERSITY OF ABERDEEN   (United Kingdom)

^b UNIVERSITY OF VIENNA   (Austria)

^c OPEN UNIVERSITY   (United Kingdom)

^d GERMAN RESEARCH CENTER FOR ARTIFICIAL INTELLIGENCE DFKI   (Germany)

^e UNIVERSITY OF COLOGNE   (Germany)

Author keywords

Affective reasoning; Body language; Embodied conversational agents; Emotion modelling; Fully generated scripted dialogue; Multimodal interfaces; Natural language generation; Speech synthesis

Indexed keywords

NATURAL LANGUAGE PROCESSING SYSTEMS; SPEECH PROCESSING; SPEECH SYNTHESIS; TEXT PROCESSING; USER INTERFACES;

BODY LANGUAGE; EMBODIED CONVERSATIONAL AGENTS (ECAS); MULTIMODAL INTERFACES; NATURAL LANGUAGE GENERATION;

INTELLIGENT AGENTS;

EID: 42649139583     PISSN: 00043702     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.artint.2008.02.002     Document Type: Article

References (87)

1. E. André, T. Rist, Presenting through performing: On the use of life-like characters in knowledge-based presentation systems, in: Proceedings IUI '2000: International Conference on Intelligent User Interfaces, 2000
2. André E., Rist T., van Mulken S., Klesen M., and Baldes S. The automated design of believable dialogues for animated presentation teams. In: Cassell J., Sullivan J., Prevost S., and Churchill E. (Eds). Embodied Conversational Agents (2000), MIT Press, Cambridge
3. André E., Klesen M., Gebhard P., Allen S., and Rist T. Integrating models of personality and emotions into lifelike characters. In: Paiva A. (Ed). Affective Interactions: Towards a New Generation of Computer Interfaces. Lecture Notes in Computer Science vol. 1814 (2000), Springer, Berlin
4. Aylett R.S., Figuieredo R., Louchart S., Dias J., and Paiva A. Making it up as you go along-improvising stories for pedagogical purposes. In: Gratch J., Young M., Aylett R., Ballin D., and Olivier P. (Eds). 6th International Conference, IVA 2006. LNAI vol. 4133 (2006), Springer, Berlin 307-315
5. Baumann S., and Grice M. The intonation of accessibility. Journal of Pragmatics 38 10 (2006) 1636-1657
6. S. Baumann, M. Grice, S. Steindamm, Prosodic marking of focus domains-categorical or gradient? in: Proceedings SpeechProsody 2006, Dresden, Germany, 2006, pp. 301-304
7. S. Baumann, M. Grice, Accenting accessible information, in: Proceedings Speech Prosody 2004, Nara, Japan, 2004, pp. 21-24
8. S. Baumann, K. Hadelich, On the perception of intonationally marked givenness after auditory and visual priming, in: Proceedings AAI Workshop "Prosodic Interfaces", Nantes, France, 2003, pp. 21-26
9. M. Bergenstråhle, Feedback gesture generation for embodied conversational agents, Technical Report ITRI-03-22, ITRI, University of Brighton, UK, 2003
10. M. Bulut, S.S. Narayanan, A.K. Syrdal, Expressive speech synthesis using a concatenative synthesiser, in: Proceedings of the 7th International Conference on Spoken Language Processing, Denver, Colorado, USA, 2002
11. S. Busemann, H. Horacek, A flexible shallow approach to text generation, in: Proceedings 9th International Workshop on Natural Language Generation, Canada, 1998, pp. 238-247
12. Callaway Ch.B., and Lester J.C. Narrative prose generation. Artificial Intelligence 139 2 (2002) 213-252
13. R. Carlson, T. Sigvardson, A. Sjölander, Data-driven formant synthesis, Progress Report No. 44, KTH Stockholm, Sweden, 2002
14. In: Cassell J., Sullivan J., Prevost S., and Churchill E. (Eds). Embodied Conversational Agents (2000), MIT Press, Cambridge, MA
15. J. Cassell, M. Stone, H. Yan, Coordination and context-dependence in the generation of embodied conversation, in: Proceedings First International Natural Language Generation Conference (INLG'2000), Mitzpe Ramon, Israel, 2000, pp. 12-16
16. J. Cassell, H. Vilhjálmsson, T. Bickmore, BEAT: The behavior expression animation toolkit, in: Proceedings ACM SIGGRAPH 2001, Los Angeles, USA, 2001, pp. 477-486
17. M. Cavazza, M. Charles, Dialogue generation in character-based interactive storytelling, in: Proceedings AIIDE, 2005
18. Chafe W. Discourse, Consciousness, and Time (1994), University of Chicago Press, Chicago/London
19. Cowie R., and Cornelius R.R. Describing the emotional states that are expressed in speech. Speech Communication 40 1-2 (2003) 5-32 (Special Issue on Speech and Emotion)
20. Cowie R., Douglas-Cowie E., Tsapatsoulis N., Votsis G., Kollias S., Fellenz W., and Taylor J. Emotion recognition in human-computer interaction. IEEE Signal Processing Magazine 18 1 (2001) 32-80
21. A. De Angeli, N. Bianchi-Berthouze (Eds.), Proceedings AVI 2006 Workshop on Gender and Interaction: Real and Virtual Women in a Male World, Venice, Italy, 2006
22. De Carolis B., Pelachaud C., Poggi I., and Steedman M. APML, a mark-up language for believable behavior generation. In: Prendinger H. (Ed). Life-Like Characters. Tools, Affective Functions and Applications (2004), Springer, Berlin
23. Dehn D.M., and Van Mulken S. The impact of animated interface agents: A review of empirical research. Journal of Human-Computer Studies 52 1 (2000) 1-22
24. Douglas-Cowie E., Campbell N., Cowie R., and Roach P. Emotional speech: Towards a new generation of databases. Speech Communication 40 1-2 (2003) 33-60 (Special Issue Speech and Emotion)
25. T. Dutoit, V. Pagel, N. Pierret, F. Bataille, O.V. Vrecken, The mbrola project: Towards a set of high quality speech synthesisers free of use for non-commercial purposes, in: Proceedings 4th International Conference of Spoken Language Processing, Philadelphia, USA, pp. 1393-1396
26. K.R. Echavarria, M. Généreux, D. Arnold, A. Day, J. Glauert, Multilingual virtual city guides, in: Proceedings Graphicon, Novosibirsk, Russia, 2005
27. Elhadad M. FUF/SURGE Homepage. Available from: (19 September 2006). http://www.cs.bgu.ac.il/surge
28. G. Erbach, Profit 1.54 user's guide, University of the Saarland, December 3, 1995
29. P. Gebhard, M. Kipp, M. Klesen, T. Rist, Adding the emotional dimension to scripting character dialogues, in: Proceedings 4th International Working Conference on Intelligent Virtual Agents (IVA'03)
30. P. Gebhard, ALMA-a layered model of affect, in: Proceedings 4th International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS'05), Utrecht, Netherlands, 2005, pp. 29-36
31. P. Gebhard, K.H. Kipp, Are computer-generated emotions and moods plausible to humans? in: Proceedings 6th International Conference on Intelligent Virtual Agents (IVA'06), Marina Del Rey, USA, 2006
32. Grice M., Baumann S., and Benzmüller R. German intonation in autosegmental-metrical phonology. In: Jun A. (Ed). Prosodic Typology. The Phonology of Intonation and Phrasing (2005), Oxford University Press, Oxford 55-83
33. Gstrein E., Schmotzer C., and Krenn B. Report on demonstrator evaluation results, NECA IST report D9e, July 2004. downloadable from. http://www.ofai.at/research/nlu/NECA/publications/publication_docs/d9e.pdf
34. Hall L., Vala M., Hall M., Webster M., Woods S., Gordon A., and Aylett R. FearNot's appearance: Reflecting children's expectations and perspectives. In: Gratch J., Young M., Aylett R., Ballin D., and Olivier P. (Eds). Proceedings 6th International Conference, IVA 2006. LNAI vol. 4133 (2006), Springer, Berlin 407-419
35. Hirschberg J. Pitch accent in context: Predicting intonational prominence from text. Artificial Intelligence 63 (1993) 305-340
36. L. Hiyakumoto, S. Prevost, J. Cassell, Semantic and discourse information for text-to-speech intonation, in: ACL Workshop on Concept-to-Speech Technology, 1997
37. Huang Z., Eliens A., and Visser C. XSTEP: A markup language for embodied agents. Proceedings 16th International Conference on Computer Animation and Social Agents (CASA'2003) (2003), IEEE Press
38. A. Iida, N. Campbell, S. Iga, F. Higuchi, M.A. Yasumura, Speech synthesis system with emotion for assisting communication, in: Proceedings ISCA Workshop on Speech and Emotion, Northern Ireland, 2000, pp. 167-172
39. Joshi A.K., Levy L., and Takahashi M. Tree adjunct grammars. Journal of the Computer and System Sciences 10 (1975) 136-163
40. Kamp H., and Reyle U. From Discourse to Logic (1993), Kluwer, Dordrecht
41. M. Kantrowitz, GLINDA: Natural language text generation in the oz interactive fiction project, Technical Report CMU-CS-90-158, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, 1990
42. Kantrowitz M., and Bates J. Integrated natural language generation systems. Aspects of Automated Natural Language Generation. In: Roesner D., and Stock O. (Eds). NAI vol. 587 (1992), Springer, Berlin
43. H. Kasuya, K. Maekawa, S. Kiritani, Joint estimation of voice source and vocal tract parameters as applied to the study of voice source dynamics, in: Proceedings 14th International Conference of Phonetic Sciences, San Francisco, USA, pp. 2505-2512
44. Kopp S., and Wachsmuth I. Synthesizing multimodal utterances for conversational agents. Computer Animation and Virtual Worlds 15 1 (2004) 39-52
45. Kopp S., Sowa T., and Wachsmuth I. Imitation games with an artificial agent: From mimicking to understanding shape-related iconic gestures. In: Camurri X., and Volpe X. (Eds). Gesture-Based Communication in Human-Computer Interaction. LNAI vol. 2915 (2004), Springer, Berlin 436-447. http://www.techfak.uni-bielefeld.de/%7Eskopp/download/gesture_imitation_GW03.pdf
46. Kopp S., Krenn B., Marsella S., Marshall A., Pelachaud C., Pirker H., Thorisson K., and Vilhjalmsson H. Towards a common framework for multimodal generation in ECAs: The behavior markup language. In: Gratch J., et al. (Ed). Intelligent Virtual Agents 2006. LNAI vol. 4133 (2006), Springer, Berlin 205-217
47. A. Kranstedt, S. Kopp, I. Wachsmuth, MURML: A multimodal utterance representation markup language for conversational agents, in: Proceedings AAMAS'02 Workshop Embodied conversational agents-let's specify and evaluate them! Bologna, Italy, 2002
48. Krenn B., Neumayr B., Gstrein E., and Grice M. Lifelike agents for the Internet: A cross-cultural case study. In: Payr S., and Trappl R. (Eds). Agent Culture: Human-Agent Interaction in a Multicultural World (2004), Lawrence Erlbaum Associates, NJ 197-229
49. B. Krenn, H. Pirker, Defining the gesticon: Language and gesture coordination for interacting embodied agents, in: Proceedings AISB-2004 Symposium on Language, Speech and Gesture for Expressive Characters, University of Leeds, UK, 2004, pp. 107-115
50. Lambrecht K. Information Structure and Sentence Form (1994), Cambridge University Press, Cambridge
51. J.C. Lester, S.A. Converse, S.E. Kahler, S.T. Barlow, B.A. Stone, R.S. Bhoga, The persona effect: Affective impact of animated pedagogical agents, in: Proceedings CHI Conference, Atlanta, Georgia, 1997
52. Levelt W. Speaking: From Intention to Articulation (1989), MIT Press, Cambridge, MA
53. A. Loyall, Believable agents: Building interactive personalities, Ph.D. thesis, CMU, Technical Report CMU-CS-97-123
54. S. Marsella, W.L. Johnson, C. LaBore, Interactive pedagogical drama for health interventions, in: AIED 2003, 11th International Conference on Artificial Intelligence in Education, Australia, 2003
55. M. Mateas, A. Stern, Facade: An experiment in building a fully-realized interactive drama, in: Game Developer's Conference: Game Design Track, San Jose, California, 2003
56. McRoy S., Channarukul S., and Ali S. An augmented template-based approach to text realization. Natural Language Engineering 9 4 (2003) 381-420
57. A. Monaghan, Intonation in a text-to-speech conversion system, Ph.D. thesis, University of Edinburgh, 1991
58. J.M. Montero, J. Gutiérrez-Arriola, J. Colás, E. Enríquez, J.M. Pardo, Analysis and modelling of emotional speech in Spanish, in: Proceedings 14th International Conference of Phonetic Sciences, San Francisco, USA, 1999, pp. 957-960
59. A. Ndiaye, P. Gebhard, M. Kipp, M. Klesen, M. Schneider, W. Wahlster, Ambient intelligence in edutainment: Tangible interaction with life-like exhibit guides, in: Proceedings Conference on INtelligent TEchnologies for interactive entertainment (INTETAIN'05), Madonna di Campiglio, Italy, 2005
60. N. Nicolov, C. Mellish, G. Ritchie, Approximate generation from non-hierarchical representations, in: Proceedings 8th International Workshop on Natural Language Generation, Herstmonceux Castle, UK, 1996
61. Ortony A., Clore G.L., and Collins A. The Cognitive Structure of Emotions (1988), Cambridge University Press, Cambridge, MA
62. Pelachaud C., Badler N.I., and Steedman M. Generating facial expressions for speech. Cognitive Science 20 1 (1996) 1-46
63. P. Piwek, B. Krenn, M. Schröder, M. Grice, S. Baumann, H. Pirker, RRL: A rich representation language for the description of agent behaviour in NECA, in: Proceedings AAMAS Workshop Embodied Conversational Agents-Let's Specify and Evaluate Them! Bologna, Italy, 2002
64. P. Piwek, A flexible pragmatics-driven language generator for animated agents, in: Proceedings of EACL (Research Notes), Budapest, Hungary, 2003
65. P. Piwek, The effect of gestures on the perception of a dialogue between two embodied conversational agents: a pilot study, Technical Report ITRI-03-09, ITRI, University of Brighton, UK, 2003
66. P. Piwek, K. van Deemter, Dialogue as discourse: Controlling global properties of scripted dialogue, in: Proceedings AAAI Spring Symposium on Natural Language Generation in Spoken and Written Dialogue, Stanford, California, 2003
67. P. Piwek, J. Masthoff, M. Bergenstrahle, Reference and gestures in dialogue generation: Three studies with embodied conversational agents, in: Proceedings AISB05 Joint Symposium on Virtual Social Agents Symposium, University of Herfordshire, UK, 2005, pp. 53-60
68. Piwek P., and Van Deemter K. Generating under global constraints: The case of scripted dialogue. Journal of Research on Language and Computation 5 2 (2007) 237-263
69. Prevost S., and Steedman M. Specifying intonation from context for speech synthesis. Speech Communication 15 (1994) 139-153
70. W. Reilly, Believable social and emotional agents, Ph.D. thesis, Carnegie Mellon University, Pittsburgh, 1996
71. Reiter E., and Dale R. Building Natural Language Generation Systems (2000), Cambridge University Press, Cambridge
72. Rickel J., and Johnson W.L. Animated agents for procedural training in virtual reality: Perception, cognition and motor control. Applied Artificial Intelligence 13 (1999) 343-382
73. Y. Schabes, Mathematical and computational aspects of lexicalized grammars, Ph.D. thesis, Computer Science Department, University of Pennsylvania, 1990
74. Schlosberg H. A scale for the judgement of facial expressions. Journal of Experimental Psychology 29 (1941) 497-510
75. M. Schröder, Speech and emotion research: An overview of research frameworks and a dimensional approach to emotional speech synthesis, Ph.D. thesis, Institute of Phonetics, Saarland University (Phonus 7), 2004
76. M. Schröder, M. Grice, Expressing vocal effort in concatenative synthesis, in: Proceedings 15th International Conference of Phonetic Sciences, Barcelona, Spain, 2003
77. Schröder M., and Trouvain J. The German text-to-speech synthesis system MARY: A tool for research, development and teaching. International Journal of Speech Technology 6 (2003) 365-377
78. Schröder M. Expressing degree of activation in synthetic speech. IEEE Transactions on Audio, Speech and Language Processing 14 4 (2006) 1128-1136
79. M. Stone, T. Bleam, C. Doran, M. Palmer, Lexicalized grammar and the description of motion events, in: TAG+: Workshop on Tree-Adjoining Grammar and Related Formalisms, 2000
80. A. Tassa, J.S. Liénard, A new approach to the evaluation of vocal effort by the psola method, in: WEB-SLS, The European Student Journal of Language and Speech, 2000
81. D. Traum, J. Bos, R. Cooper, S. Larsson, I. Lewin, C. Matheson, M. Poesio, A model of dialogue moves and information state revision, Trindi Project Deliverable D2.1, 1999
82. O. Turk, M. Schröder, B. Bozkurt, L.M. Arslan, Voice quality interpolation for emotional text-to-speech synthesis, in: Proceedings Interspeech, Lisbon, Portugal, 2005, pp. 797-800
83. van Deemter K., Krahmer E., and Theune M. Real versus template-based natural language generation: A false opposition?. Computational Linguistics 31 1 (2005) 15-24
84. van Deemter K. What's New? A semantic perspective on sentence accent. Journal of Semantics 11 (1994) 1-31
85. Vallduví E., and Engdahl E. The linguistic realisation of information packaging. Linguistics 34 (1996) 459-519
86. Vilhjalmsson H. Animating conversation in online games. In: Rauterberg M. (Ed). Entertainment Computing ICEC. Lecture Notes in Computer Science vol. 3166 (2004), Springer, Berlin 139-150
87. M. White, M.E. Foster, J. Oberlander, A. Brown, Using facial feedback to enhance turn-taking in a multimodal dialogue system, in: Proceedings HCI International, 2005