Perceptive animated interfaces: First steps toward a new paradigm for human-computer interaction

Proceedings of the IEEE

Volumn 91, Issue 9, 2003, Pages 1391-1404

  Cole, Ronald ^a   Van Vuuren, Sarel ^a   Pellom, Bryan ^a   Hacioglu, Kadri ^a   Ma, Jiyong ^a   Movellan, Javier ^b   Schwartz, Scott ^a   Wade Stein, David ^a   Ward, Wayne ^a   Yan, Jie ^a  

^a UNIVERSITY OF COLORADO   (United States)

^b UNIVERSITY OF SAN DIEGO   (United States)

Author keywords

Animation; Conversational agents; Dialogue system; Electronic books; Human computer interaction; Interactive books; Literacy teaching; Machine perception; Spoken language; Tools for cognitive disabilities; Virtual humans; Vocabulary tutors

Indexed keywords

ANIMATION; COMPUTER VISION; INTERACTIVE COMPUTER SYSTEMS; INTERFACES (COMPUTER); SENSORY PERCEPTION; SPEECH RECOGNITION; SYSTEMS ANALYSIS;

COGNITIVE DISABILITIES; CONVERSATIONAL AGENTS; DIALOGUE SYSTEMS; ELECTRONIC BOOKS; INTERACTIVE BOOKS; LITERACY TEACHING; MACHINE PERCEPTION; SPOKEN LANGUAGES; VIRTUAL HUMANS; VOCABULARY TUTORS;

HUMAN COMPUTER INTERACTION;

EID: 17044367197     PISSN: 00189219     EISSN: None     Source Type: Journal    
DOI: 10.1109/JPROC.2003.817143     Document Type: Conference Paper

References (69)

1. D. Massaro, Perceiving Talking Faces: From Speech Perception to a Behavioral Principle. Cambridge, MA: MIT Press, 1998.
2. D. Massaro and M. Cohen, "Speech perception in hearing-impaired perceivers: Synergy of multiple modalities," J. Speech, Lang., Hear. Sci., vol. 42, pp. 21-41, 1999.
3. L. Barker, "Computer-assisted vocabulary acquisition: The CSLU vocabulary tutor in oral-deaf education," J. Deaf Stud. Deaf Educ., vol. 8, no. 2, pp. 187-198, 2003.
4. F. Thomas and O. Johnston, Disney Animation: The Illusion of Life, New York: Hyperion, 1995.
5. B. Reeves and C. Nash, The Media Equation: How People Treat Computers. Televisions and New Media Like Real People und Places. New York: Cambridge Univ. Press, 1996.
6. J. Allheck and N. Badler, "Toward representing agent behaviors modified by personality and emotion," presented at the 1st Int. Joint Conf. Autonomous Agents and Multi-Agent Systems, Bologna, Italy, 2002.
7. N. Badler, J. Allbeck, L. Zhao, and M. Byun, "Representing and parameterizing agent behaviors," in Proc. Computer Animation, 2002, pp. 133-143.
8. J. Cassell, H. Vilhjalmsson, and T. Bickmore, "Beat: The behavior expression animation toolkit," in Proc. ACM SIGGRAPH, 2001, pp. 477-486.
9. J. Gratch and S. Marsella, "Tears and fears: Modeling emotions and emotional behaviors in synthetic agents," in Proc: Autonomous Agents, 2001, pp. 278-285.
10. A. B. Loyall, "Believable agents: Building interactive personalities," Ph.D. dissertation, Dept. Comput. Sci., Carnegie Mellon Univ., Pittsburgh, PA, 1997.
11. S. Marcella and J. Gratch, "Modeling the interplay of emotions and plans in multi-agent simulations," presented at the 23rd Annu. Conf. Cognitive Science Society, Edinburgh, U.K., 2001.
12. J. Gratch, J. Rickel, E. André, N. Badler, J. Cassell, and E. Petajan, "Creating interactive virtual humans: Some assembly required," IEEE Intell. Syst., vol. 17, pp. 54-63, July/Aug. 2002.
13. W. L. Johnson, J. W. Rickel, and J. C. Lester, "Animated pedagogical agents: Face-to-face interaction in interactive learning environments," Int. J. Artif. Intell. Educ., vol. 11, pp. 47-78, 2000.
14. J. Cassell, T. Stocky, T. Bickmore, Y. Gao, Y. Nakano, K. Ryokai, D. Tversky, C. Vaucelle, and H. Vilhjálmsson, "MACK: Media lab autonomous conversational kiosk," presented at the Imagina: Intelligent Autonomous Agents, Monte Carlo, Monaco, 2002.
15. J. Gustafson, "Developing multimodal spoken dialogue systems-empirical studies of spoken human-computer interactions," Ph.D. dissertation, Dept. Speech, Music and Hearing, Royal Inst. Technol., Stockholm, Sweden, 2002.
16. M. Blomberg, R. Carlson, K. Elenius, B. Granström, J. Gustafson, S. Hunnicutt, R. Lindell, and L. Neovius, "An experimental dialogue system: WAXHOLM," in Proc: Euro-speech, vol. 3, 1993, pp. 1867-1870.
17. J. Bertenstam, M. Blomberg, R. Carlson, K. Elenius, B. Granström, J. Gustafson, S. Hunnicutt, J. Högberg, R. Lindell, L. Neovius, A. de Serpa-Leita, L. Nord, and N. N. Ström. "The Waxholm system - A progress report," in Proc Spoken Dialogue Systems, Vigsø. Denmark, 1995, pp. 81-84.
18. J. Gustafson, M. Lundeberg, and J. Liljencrants, "Experiences from the development of August - a multimodal spoken dialogue system," presented at the IDS'99. Workshop Interactive Dialogue in Multimodal Systems, Kloster Irsee, Germany, 1999.
19. M. Lundeberg and J. Beskow, "Developing a 3D-agent for the August dialogue system," presented at the Audio-Visual Speech Processing Conf., Santa Cruz, CA, 1999.
20. J. Gustafson, L. Bell, J. Boye, J. Edlund, and M. Wiren, "Constraint manipulation and visualization in a multimodal dialogue system," presented at the ISCA Workshop Multi-Modal Dialogue Mobile Environments, Kloster Irsee, Germany, 2002.
21. L. Barker and T. Weston. (2003) Designing, implementing, and assessing Web-based learning modules. [Online]. Available: http://www.colorado.edu/ATLAS/ evaluation/papers_fld/module_pg.htm
22. J. Cassell, "Toward a model of technology and literacy development: Story listening systems," Appl. Development. Psych., to be published.
23. R. Cole, T. Carmell, P. Connors, M. Macon, J. Wouters, J. de Villiers, A. Tarachow, D. Massaro, M. Cohen, J. Beskow, J. Yang, U. Meier, A. Waibel, P. Stone, G. Fortier, A. Davis, and C. Soland, "Intelligent animated agents for interactive language training," presented at the ESCA Workshop Speech Technology in Language Learning, Stockholm, Sweden, 1998.
24. R. Cole, D. Massaro, J. de Villiers, B. Rundle, K. Shobaki, J. Wouters, M. Cohen, J. Beskow, P. Stone, P. Connors, A. Tarachow, and D. Solcher, "New tools for interactive speech and language training: Using animated conversational agents in the classrooms of profoundly deaf children," in Proc. ESCA/SOCRATES Workshop Method and Tool Innovations Speech Science Education, London, U.K., 1999, pp. 45-52.
25. A. Graesser, K. Wiemer-Hastings, P. Wiemer-Hastings, and R. Kreuz, "AutoTutor: A simulation of a human tutor," J. Cognitive Syst. Res., vol. 1, pp. 35-51, 1999, to be published.
26. S. Marsella, "Pedagogical soap," in Socially Intelligent Agents: The Human in the Loop, AAAI Fall 2000 Symp., K. Dautenhahn, Ed., 2000, pp. 107-112.
27. K. Ryokai, C. Vaucelle, and J. Cassell, "Virtual peers as partners in storylelling and literacy learning," J. Cotnput. Assisted Learn., vol. 19, no. 2, pp. 195-208, 2003.
28. P. Stone, "Revolutionizing language instruction in oral deaf education," presented at the Int. Conf. Phonetic Sciences, San Francisco, CA, 1999.
29. B. Bloom, "The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring," Educ. Res., vol. 13, pp. 4-16, 1984.
30. P. Cohen, J. Kulik, and C. Kulik, "Educational outcomes of tutoring: A meta-analysis of findings," Amer. Educ. Res. J., vol. 19, pp. 237-248, 1982.
31. A. Graesser, N. Person, and J. Magliano, "Collaborative dialog patterns in naturalistic one-on-one tutoring," Appl. Cognitive Psychol., vol. 9, pp. 359-387, 1995.
32. D. Dehn and S. van Mulken, "The impact of animated interface agents: A review of empirical research," Int. J. Human-Comput. Stud., vol. 52, pp. 1-22, 2000.
33. A. Graesser, K. Moreno, and J. Marineau, "AutoTutor improves deep learning of computer literacy: Is it the dialog or the talking head?," presented at the 2003 Int. Conf. Artificial Intelligence Education, Sydney, Australia.
34. A. Graesser, M. Ventura, G. Jackson, J. Mueller, X. Hu, and N. Person, "The impact of conversational navigational guides on the learning, use, and perceptions of users of a web site," presented at the AAAI Spring Symp. Agent-Mediated Knowledge Management, Menlo Park, CA, 2003.
35. A. Graesser, K. VanLehn, C. Rose, P. Jordan, and D. Harter, "Intelligent tutoring systems with conversational dialogue," AI Mag., vol. 22, pp. 39-51, 2001.
36. N. Person, A. Graesser, L. Bautista, B. Mathews, and TRG, "Evaluating student learning gains in two versions of AutoTutor," in Artificial Intelligence in Education: AI-ED in the Wired and Wireless Future, J. Moore, C. Redfield, and W. Johnson, Eds. Amsterdam, The Netherlands: OIS, 2001, pp. 286-293.
37. T. Bickmore and J. Cassell, "Social dialogue with embodied conversational agents," in Natural, Intelligent and Effective Interaction with Multimodal Dialogue Systems, J. van Kuppevelt, L. Dybkjaer, and N. Bernsen, Eds. New York: Kluwer, to be published.
38. R. Cole, "Tools for research and education in speech science," presented at the Int. Conf. Phonetic Sciences, San Francisco, CA, 1999.
39. S. Sutton, D. Novick, R. Cole, and M. Fanty, "Building 10000 spoken-dialogue systems," presented at the Int. Conf. Spoken Language Processing, Philadelphia, PA, 1996.
40. R. Cole, S. Sutton, Y. Yan, P. Vermeulen, and M. Fanty, "Accessible technology for interactive systems: A new approach to spoken language research," presented at the Int. Conf. Acoustics, Speech and Signal Processing, Seattle, WA, 1998.
41. S. Sutton and R. Cole, "Universal speech tools: The CSLU toolkit," in Proc. Int. Conf. Spoken Language Processing, 1998, pp. 3221-3224.
42. P. Connors, A. Davis, G. Fortier, K. Gilley, B. Rundle, C. Soland, and A. Tarachow, "Participatory design: Classroom applications and experiences," presented at the Int. Conf. Phonetic Sciences. San Francisco, CA, 1999.
43. J. Paysan. (2001) Look who's, talking. Prime Time Thursday, ABC Television Network. [Online]. Available: http://oak.colorado.edu/~spradhan/ download/Ron-Videos/ABC-Primetime
44. S. Seneff, E. Hurley, R. Lau, C. Pao, P. Schmid, and V. Zue, "Galaxy-II: A reference architecture for conversational system development," in Proc. Int. Conf. Spoken Language Processing, vol. 3, 1997, pp. 931-934.
45. S. Bayer, C. Doran, and B. George, "Exploring speech-enabling dialogue with the Galaxy communicator infrastructure," in Proc: Human Language Technology Conf., 2001, pp. 114-116.
46. W. Ward and B. Pellom, "The CU Communicator system," presented at the IEEE Workshop Automatic Speech Recognition and Understanding, Keystone, CO, 1999.
47. B. Pellom, W. Ward, and S. Pradhan, "The CU Communicator: An architecture for dialogue systems," presented at the Int. Conf. Spoken Language Processing, Beijing, China, 2000.
48. B. Pellom, W. Ward, J. Hansen, K. Hacioglu, J. Zhang, X. Yu, and S. Pradhan, "University of Colorado dialog systems for travel and navigation," presented at the Human Language Technology Conf., San Diego, CA, 2001.
49. (2002) Dialog System for Travel and Navigation. CU Communicator System. [Online]. Available: http://communicator.colorado.edu
50. J. Zhang, W. Ward, B. Pellom, X. Yu, and K. Hacioglu, "Improvements in audio processing and language modeling in the CU Communicator," in Proc. of Eurospeech, Aalborg, Denmark, 2001.
51. J. Zhang, W. Ward, and B. Pellom, "Phone based voice activity delection using online Bayesian adaptation with conjugate normal distributions," in Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing, vol. 1, 2002. pp. 1-321-1-324.
52. B. Pellom, "Sonic: The University of Colorado continuous speech recognizer," Center for Spoken Language Research, Univ. Colorado, Boulder, Tech. Rep. No. TR-CSLR-2001-01, 2001.
53. B. Pellom and K. Hacioglu, "Recent improvements in the Sonic ASR system for noisy speech: The SPINE task." in Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing. 2003, pp. I-4-I-7.
54. O. Salor, B. Pellom, T. Ciloglu, K. Hacioglu, and M. Demirekler, "On developing new text and audio corpora and speech recognition tools for the Turkish language," presented at the Int. Conf. Spoken Language Processing, Denver, CO, 2002.
55. W. Ward, "Extracting information from spontaneous speech," presented at the Int. Conf. Spoken Language Processing, Tokyo, Japan, 1994.
56. P. Taylor, A. Black, and R. Caley, "The architecture of the festival speech synthesis system." in Proc. 3rd ESCA Workshop Speech Synthesis, 1998, pp. 147-151.
57. J. Ma, J. Yan, and R. Cole, "CU animate: Tools for enabling conversations with animated characters," presented at the Int. Conf. Spoken Language Processing, Denver, CO, 2002.
58. J. Movellan, J. Susskind, and J. Hershey, "Global search, real time tracking of non-rigid objects using integral banks of integral images," Machine Perception Lab., Univ. California, San Diego, Tech. Rep. No. TR 2002.04, 2002.
59. H. Rowley, S. Baluja, and T. Kanade, "Neural network-based face detection." IEEE Trans. Pattern Anal. Machine Intell., vol. 20, pp. 23-38, Jan. 1998.
60. J. F. Cohn, A. J. Zlochower, J. J. Lien, and T. Kanade, "Automated face analysis by feature point tracking has high concurrent validity with manual FACS coding," Psychophysiology, vol. 36, pp. 35-43, 1999.
61. Emotion tracker (2002). [Online]. Available: http://markov.ucsd.edu/ ~movellan/mplab
62. Galaxy Communicator download (2002). [Online]. Available: http://communicator.sourceforge.net
63. CU Communicator download (2002). [Online]. Available: http://communicator.colorado.edu
64. ITR reading tutor project (2002). [Online]. Available: http://cslr.colorado.edu/beginweb/reading/reading.html
65. D. Steinhart, "Summary street: An intelligent tutoring system for improving student writing through the use of latent semantic analysis," Ph.D. dissertation, Dept. Psychol., Univ. Colorado, Boulder, 2001.
66. S. Deerwester, S. Dumais, T. Landauer, G. Furnas, and R. Harshman, "Indexing by latent semantic analysis," J. Amer. Soc. Inform. Sci., vol. 41, no. 6, pp. 391-407, 1990.
67. T. Landauer and S. Dumais, "A solution to Plato's problem: The latent semantic analysis theory of acquisition, induction and representation of knowledge," Psychol. Rev., vol. 104, pp. 211-240, 1997.
68. E. Kintsch, D. Steinhart, G. Stahl, C. Matthews, R. Lamb, and LRG, "Developing summarization skills through the use of LSA-based feedback," Interactive Learn. Environ., vol. 8, pp. 87-109, 2000.
69. Colorado literacy tutor (2002). [Online]. Available: http://www.colit.org