Predicting affective states expressed through an emote-aloud procedure from autotutor’s mixed-initiative dialogue

International Journal of Artificial Intelligence in Education

Volumn 16, Issue 1, 2006, Pages 3-28

  D’Mello, Sidney K ^a   Craig, Scotty D ^b   Sullins, Jeremiah ^c   Graesser, Arthur C ^c  

^a University of Memphis   (United States)

^b UNIVERSITY OF PITTSBURGH   (United States)

^c University of Memphis   (United States)

Author keywords

Affect detection; AutoTutor; Dialogue patterns; Intelligent Tutoring Systems

Indexed keywords

EID: 85013610638     PISSN: 15604292     EISSN: 15604306     Source Type: Journal    
DOI: None     Document Type: Article

References (89)

1. Aha, D., & Kibler, D. (1991). Instance-based learning algorithms. Machine Learning, 6, 37-66.
2. Aleven, V., & Koedinger, K.R. (2002). An effective metacognitive strategy: Learning by doing and explaining with a computer based Cognitive Tutor. Cognitive Science, 26, 147-179.
3. Aleven, V., Stahl, E., Schworm, S., Fischer, F., & Wallace, R.M. (2003). Help Seeking and Help Design in Interactive Learning Environments. Review of Educational Research, 73(2), 277-320.
4. Allen, J. (1999). Mixed-initiative interaction. IEEE Intelligent Systems. 14(5), 14-16.
5. Ang, J., Dhillon, R., Krupski, A., Shriberg, E., & Stolcke, A. (2002). Prosody-based automatic detection of annoyance and frustration in human-computer dialog In J. H. L. Hansen & B. Pellom (Eds.) Proceedings of the International Conference on Spoken Language Processing (ICSLP’02) (pp. 2037-2039). Denver, CO, USA.
6. Azevedo, R., & Cromley, J.G. (2004). Does training on self-regulated learning facilitate students’ learning with hypermedia. Journal of Educational Psychology, 96, 523-535.
7. Ball, G., & Breeze, J. (2000). Emotion and personality in a conversational agent In J. Cassel, J. Sullivan, S. Prevost & E. Churchill (Eds.) Embodied Conversational Agents (pp. 189-219). Boston: The MIT Press.
8. Bartlett, F.C. (1932). Remembering: An Experimental and Social Study. Cambridge: Cambridge University Press.
9. Breazeal, C. (2003). Designing sociable robots. Cambridge: MIT Press.
10. Bruner, J.S. (1961). The act of discovery. Harvard Educational Review, 31(1), 21-32.
11. Bull, E.P. (1987) Posture and gesture. New York, Pergamon Press.
12. Chi, M.T.H., Bassok, M., Lewis, M., Reimann, P., & Glaser, R. (1989). Self explanations: How students study and use examples in learning to solve problems. Cognitive Science, 13, 145-182.
13. Chi, M.T.H., Siler, S., Jeong, H., Yamauchi, T., & Hausmann, R.G. (2001). Learning from human tutoring. Cognitive Science, 25, 471-533.
14. Clark, H.H., & Shaefer, E.F. (1989). Contributing to discourse. Cognitive Science, 13, 259-294.
15. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37-46.
16. Conati, C. (2002). Probabilistic assessment of user’s emotions in educational games. Journal of Applied Artificial Intelligence, 16, 555-575.
17. Conati, C. (2004). How to evaluate models of user affect? Proceedings of ADS 04, Tutorial and Research Workshop on Affective Dialogue Systems (pp. 288-300). Kloster Irsee, Germany, June 2004.
18. Conati, C., & Mclaren, H. (2005). Data-driven Refinement of a Probabilistic Model of User Affect In L. Ardissono, P. Brna & A. Mitrovic (Eds.) Proceedings of UM2005 User Modeling, 10th International Conference (pp. 40-49). Berlin, Germany: Springer.
19. Craig, S.D., Graesser, A. C., Sullins, J., & Gholson, B. (2004a). Affect and learning: An exploratory look into the role of affect in learning. Journal of Educational Media, 29, 241-250.
20. Craig, S. D., D’Mello, S.K., Gholson, B., Witherspoon, A., Sullins, J., & Graesser A.C. (2004b). Emotions during learning: The first steps toward an affect sensitive intelligent tutoring system. In J. Nall & R. Robson (Eds.). Proceedings of E-learn 2004: World conference on E-learning in Corporate, Government, Healthcare, & Higher Education (pp. 284-288). Norfolk, VA: Association for the Advancement of Computing in Education.
21. Crutcher, R.J., & Healy, A.F. (1989). Cognitive operations and the generation effect. Journal of Experimental Psychology: Learning, Memory, and Cognition, 15, 669-675.
22. Csikszentmihalyi, M. (1990). Flow: The Psychology of Optimal Experience. Harper-Row: NY.
23. De Vicente, A., & Pain, H. (2002). Informing the detection of students’ motivational state: An empirical study. In S.A. Cerri, G. Gouarderes, & F. Paraguacu (Eds). Proceedings of the Sixth International Conference on Intelligent Tutoring Systems (pp.933-943). Berlin, Germany: Springer.
24. D’Mello, S.K., Craig, S.D., Gholson, B., Franklin, S., Picard, R., & Graesser, A.C. (2005). Integrating affect sensors in an intelligent tutoring system. In Affective Interactions: The Computer in the Affective Loop Workshop at 2005 International Conference on Intelligent User Interfaces (pp. 7-13)New York: AMC Press
25. Earthman, E.A. (1992). Creating the virtual work: Reader’s processes in understanding literary texts. Research in the Teaching of English, 26, 351-384.
26. Ekman, P. (1973). Cross-cultural studies of facial expression In P. Ekman (Ed.) Darwin and facial expression: A century of research in review (pp. 169-222). New York: Academic Press.
27. Ekman, P. (2003). Emotions Revealed: Recognizing Faces and Feelings to Improve Communication and Emotional Life. New York: Henry Holt and Company, LLC.
28. Ekman, P., & Friesen, W.V. (1978). The facial action coding system: A technique for the measurement of facial movement. Palo Alto: Consulting Psychologists Press.
29. Ericsson, K.A., & Simon, H.A. (1993). Protocol analysis: Verbal reports as data. (Rev ed.) Cambridge, MA: The MIT Press.
30. Eva-Wood, A.L. (2004). How think-and feel-aloud instruction influences poetry readers. Discourse Processes, 38, 173-192.
31. Fan, C., Sarrafzadeh, A., Overmyer, S., Hosseini, H. G., Biglari-Abhari, M., & Bigdeli, A. (2003). A fuzzy approach to facial expression analysis in intelligent tutoring systems In A. Méndez-Vilas and J.A. Mesa González (Eds.) Advances in Technology-based Education: Towards a Knowledge-based Society Vol 3. (pp. 1933-1937). Badajoz, Spain: Junta De Extremadura.
32. Fernandez, R., & Picard, R.W. (2005). Classical and Novel Discriminant Features for Affect Recognition from Speech. Interspeech 2005 - Eurospeech 9th European Conference on Speech Communication and Technology. September 4-8, Lisbon Portugal.
33. Foltz, P.W. (1996). Latent semantic analysis for text-based research. Behavior Research Methods, Instruments, and Computers, 28, 197-202.
34. Foltz, P.W., Britt, M.A., & Perfetti, C.A. (1996). Reasoning from multiple texts: An automatic analysis of readers’ situation models. Proceedings of the 18th Annual Conference of the Cognitive Science Society (pp. 110-115). Mahwah, NJ: Erlbaum.
35. Forbes-Riley, K., & Litman, D. (2004). Predicting Emotion in Spoken Dialogue from Multiple Knowledge Sources. Proceedings of the Human Language Technology Conference: 4th Meeting of the North American Chapter of the Association for Computational Linguistics (HLT/NAACL), Boston, MA.
36. Ford, M.E. (1992). Motivating humans: Goals, emotions, and personal agency beliefs. London: Sage.
37. Friedman, J., Hastie T., & Tibshirani, R. (1998). Additive logistic regression: A statistical view of boosting. Stanford University.
38. Goutte, C. (1997). Note on free lunches and cross-validation. Neural Computation, 9, 1211-1215.
39. Graesser, A.C., Chipman, P., Haynes, B.C., & Olney, A. (2005a). AutoTutor: An intelligent tutoring system with mixed-initiative dialogue. IEEE Transactions in Education, 48, 612-618.
40. Graesser, A.C., Lu, S., Olde, B.A., Cooper-Pye, E., & Whitten, S. (2005b). Question asking and eye tracking during cognitive disequilibrium: Comprehending illustrated texts on devices when the devices break down. Memory and Cognition, 33, 1235-1247.
41. Graesser, A.C., & Olde, B. (2003). How does one know whether a person understands a device? The quality of the questions the person asks when the device breaks down. Journal of Educational Psychology, 95, 524-536.
42. Graesser, A.C., & Person, N. K. (1994). Question asking during tutoring. American Educational Research Journal, 31, 104-137.
43. Graesser, A.C., Person, N., Harter, D., & the Tutoring Research Group (2001). Teaching tactics and dialogue in AutoTutor. International Journal of Artificial Intelligence in Education, 12, 257-279.
44. Graesser, A.C., Wiemer-Hastings, P., Wiemer-Hastings, K., Harter, D., Person, N.K., & Tutoring Research Group (2000). Using latent semantic analysis to evaluate the contributions of students in AutoTutor. Interactive Learning Environments, 8, 129-148.
45. Graesser, A.C., Wiemer-Hastings, K., Wiemer-Hastings, P., Kreuz, R., & the Tutoring Research Group (1999). AutoTutor: A simulation of a human tutor. Journal of Cognitive Systems Research, 1, 35-51.
46. Guhe, M., Gray, W.D., Schoelles, M.J., & Ji, Q. (2004). Towards an affective cognitive architecture. Poster session presented at the Cognitive Science Conference, Chicago, IL.
47. Harter, S. (1981). A new self-report scale of intrinsic versus extrinsic orientation in the classroom: Motivation and informational components. Developmental Psychology, 17, 300-312.
48. Hudlicka, E., & McNeese, D. (2002). Assessment of user affective and belief states for interface adaptation: Application to an Air Force pilot task. User Modeling and User-Adapted Interaction, 12(1), 1-47.
49. Izard, C. E. (1971). The face of emotion. New York: Appleton-Century-Crofts.
50. John, G.H., & Langley, P. (1995). Estimating Continuous Distributions in Bayesian Classifiers. In P. Besnard & S. Hanks (Eds.) Proceedings of the Eleventh Conference on Uncertainty in Artificial Intelligence (pp. 338-345). Montreal, Canada: Morgan
51. Kaufmann Kapoor, A., Mota, S., & Picard, R. (2001). Toward a learning companion that recognizes affect. In Proceedings from Emotional and Intelligent II, the tangled knot of social cognition, AAAI Fall Symposium. Menlo Park, CA: AAAI Press.
52. Kim, Y. (2005). Empathetic Virtual Peers Enhanced Learner Interest and Self-Efficacy. Workshop on Motivation and Affect in Educational Software at the 12th International Conference on Artificial Intelligence in Education, Amsterdam, The Netherlands.
53. Kort, B., Reilly, R., & Picard, R. (2001). An affective model of interplay between emotions and learning: Reengineering educational pedagogy-building a learning companion In T. Okamoto, R. Hartley, Kinshuk & J. P. Klus (Eds.) Proceedings IEEE International Conference on Advanced Learning Technology: Issues, Achievements and Challenges (pp. 43-48). Madison, Wisconsin: IEEE Computer Society.
54. Kozma, R., & Freeman, W.J. (2001). Chaotic resonance: Methods and applications for robust classification of noisy and variable patterns. International Journal of Bifurcation & Chaos, 11, 1607-1629.
55. Kozma, R., & Freeman, W.J. (2002). Classification of EEG patterns using Nonlinear Neurodynamics and identifying chaotic phase transitions. Neurocomputing, 44-46, 1107-1112.
56. Landauer, T.K., & Dumais, S.T. (1997). A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104, 211-240.
57. Landauer, T.K., Foltz, P.W., & Laham, D. (1998). An introduction to latent semantic analysis. Discourse Processes, 25, 259-284.
58. Lepper, M.R., & Woolverton, M. (2002). The wisdom of practice: Lessons learned from the study of highly effective tutors In J. Aronson (Ed.) Improving academic achievement: Impact of psychological factors on education (pp. 135-158). Orlando, FL: Academic Press.
59. Lester, J.C., Towns, S.G., FitzGerald, P.J. (1999). Achieving affective impact: visual emotive communication in lifelike pedagogical agents. International Journal of Artificial Intelligence in Education, 10(3-4), 278-291.
60. Linnenbrink, E.A., & Pintrich, P.R. (2002). The role of motivational beliefs in conceptual change. In M. Limon & L. Mason (Eds.) Reconsidering conceptual change: Issues in theory and practice (pp. 115-135). Dordrecht, The Netherlands: Kluwer Academic Publishers.
61. Litman, D.J., & Forbes-Riley, K. (2004). Predicting student emotions in computer-human tutoring dialogues. In Proceedings of the 42nd annual meeting of the association for computational linguistics (pp. 352-359). East Stroudsburg, PA: Association for Computational Linguistics.
62. Litman, D.J., Rose, C.P., Forbes-Riley, K., VanLehn, K., Bhemhe, D., Silliman, S. (2004). Spoken versus typed human and computer dialogue tutoring In J.C. Lester, R.M. Vicari, F. Paraguaçu (Eds.) Proceedings of the Seventh International Conference on Intelligent Tutoring Systems (pp. 368-379). Berlin: Springer Verlag.
63. Litman, D.J., & Silliman, S. (2004). ITSPOKE: An intelligent tutoring spoken dialogue system. In Proceedings of the human language technology conference: 3rd meeting of the North American chapter of the association of computational linguistics (pp. 52-54). Edmonton, Canada: ACL.
64. Louwerse, M. M., Graesser, A.C., Olney, A. & the Tutoring Research Group. (2002). Good computational manners: Mixed-initiative dialog in conversational agents In C. Miller (Ed.) Etiquette for Human-Computer Work (pp. 71-76). Papers from the 2002 Fall Symposium, Technical Report FS-02-02.
65. McNamara, D.S., & Healy, A.F. (1995). A procedural explanation of the generation effect: The use of an operand retrieval strategy for multiplication and addition problems. Journal of Memory and Language, 34, 399-416.
66. Merriam-Webster, Incorporated. (n.d.) Merriam-Webster Online. Retrieved July 7, 2003, from http://www.m-w.com
67. Meyer, D.K., & Turner, J.C. (2002). Discovering emotion in classroom motivation research. Educational Psychologist, 37(2), 107-114.
68. Miserandino, M. (1996). Children who do well in school: Individual differences in perceived competence and autonomy in above-average children. Journal of Educational Psychology, 88, 203-214.
69. Miyake, N., & Norman, D.A. (1979). To ask a question one must know enough to know what is not known. Journal of Verbal Learning and Verbal Behavior, 18, 357-364.
70. Moore, J.D. (1995). Participating in explanatory dialogues. Cambridge: MIT Press.
71. Mota, S., & Picard, R.W. (2003). Automated Posture Analysis for Detecting Learner’s Interest Level. Workshop on Computer Vision and Pattern Recognition for Human-Computer Interaction, CVPR HCI, June, 2003.
72. Newell, A., & Simon, H.A. (1972). Human problem solving. Englewood Cliffs, NJ: Prentice Hall.
73. Otero, J., & Graesser, A.C. (2001). PREG: Elements of a model of question asking. Cognition & Instruction, 19, 143-175.
74. Picard, R.W. (1997). Affective computing. Cambridge: MIT Press.
75. Quinlan, R. (1993). C4.5: Programs for Machine Learning. San Mateo, CA: Morgan Kaufmann Publishers.
76. Robson, C. (1993). Real word research: A resource for social scientist and practitioner researchers. Oxford: Blackwell.
77. Rozin, P., & Cohen, A.B. (2003). High Frequency of facial expressions corresponding to confusion, concentration, and worry in an analysis of naturally occurring facial expressions of Americans. Emotion, 3, 68-75.
78. Schober, M.F., & Clark, H.H. (1989). Understanding by addressees and overhearers. Cognitive Psychology, 21, 211-232.
79. Shafran, I., Riley, M., & Mohri, M. (2003). Voice signatures. In Proc. IEEE Workshop on Automatic Speech Recognition and Understanding (pp. 31-36). Piscataway, NJ: IEEE
80. van Someren, M.W., Barnard, Y.F., & Sandberg, J.A.C. (1994). The think aloud method: A practical guide to modeling cognitive processes. London: Academic Press.
81. Snow, R., Corno, L., & Jackson, D. (1996). Individual differences in affective and cognitive functions In D. C. Berliner & R. C. Calfee (Eds.), Handbook of educational psychology (pp. 243-310). New York: Macmillan.
82. Stipek, D. (1998). Motivation to Learn: From Theory to Practice 3rd edition. Boston: Allyn and Bacon.
83. Trabasso, T., & Magliano, J. (1996). Conscious understanding during comprehension. Discourse Processes, 21, 225-286.
84. VanLehn, K., Jones, R.M., & Chi, M.T.H. (1992). A model of the self- explanation effect. Journal of the Learning Sciences, 2, 1-60.
85. VanLehn, K., Jordan, P., Rosé, C.P., Bhembe, D., Bottner, M., Gaydos, A., et al. (2002). The architecture of Why2-Atlas: A coach for qualitative physics essay writing. In S.A. Cerri, G. Gouarderes, & F. Paraguacu (Eds.) Proceedings of the Sixth International Conference on Intelligent Tutoring Systems (pp.158-167). Berlin: Springer -Verlag.
86. Vavik, L. (1993). Facilitating discovery learning in computer-based simulation learning environments In R.D. Tennyson & A.E. Baron (Eds.) Automating instructional design: Computer-based development and delivery tools (pp. 403-449). Berlin, Germany: Springer-Verlag.
87. Wang, N., Johnson, W.L., Mayer, R., Rizzzo, P., Shaw, E., & Collins, H. (2005). The politeness effect: Pedagogical agents and learning gains In C-K. Looi, G. McCalla, B. Bredeweg & J. Breuker (Eds.) Artificial Intelligence in Education (pp. 686-693). Amsterdam: IOS Press.
88. Wiemer-Hastings, P., Wiemer-Hastings, K., & Graesser, A.C. (1999). Improving an intelligent tutor’s comprehension of students with Latent Semantic Analysis In S.P. Lajoie & M. Vivet (Eds.) Artificial Intelligence in Education (pp. 535-542). Amsterdam: IOS Press.
89. Witten, I. H., & Frank E. (2005). Data Mining: Practical machine learning tools and techniques (2nd ed.). San Francisco, CA: Morgan Kaufmann.