Empirically evaluating the application of reinforcement learning to the induction of effective and adaptive pedagogical strategies

User Modeling and User-Adapted Interaction

Volumn 21, Issue 1-2, 2011, Pages 137-180

  Chi, Min ^a   Vanlehn, Kurt ^b   Litman, Diane ^c   Jordan, Pamela ^c  

^a Carnegie Mellon University   (United States)

^b ARIZONA STATE UNIVERSITY   (United States)

^c UNIVERSITY OF PITTSBURGH   (United States)

Author keywords

Human learning; Machine learning; Pedagogical strategy; Reinforcement learning

Indexed keywords

DISCRETE STEP; E-LEARNING ENVIRONMENT; EMPIRICAL EVALUATIONS; HUMAN LEARNING; LEARNING GAIN; MACHINE LEARNING; MODEL-BASED; NATURAL LANGUAGES; PEDAGOGICAL STRATEGY; POLICY ITERATION; SEQUENTIAL DECISION PROCESS; SMALL TRAINING; TECHNICAL CHALLENGES; TRAINING CORPUS; TUTORING SYSTEM;

E-LEARNING; STUDENTS; TEACHING;

REINFORCEMENT LEARNING;

EID: 79955803320     PISSN: 09241868     EISSN: 15731391     Source Type: Journal    
DOI: 10.1007/s11257-010-9093-1     Document Type: Article

References (71)

1. Ai, H., Litman, D.J.: Knowledge consistent user simulations for dialog systems. In: Proceedings of Interspeech-2007, pp. 2697-2700, Antwerp, Belgium, 2007
2. Aleven, V., Ogan, A., Popescu, O., Torrey, C., Koedinger, K.R.: Evaluating the effectiveness of a tutorial dialogue system for self-explanation. In: Lester, J.C., Vicari, R.M., Paraguaçu, F. (eds.) Intelligent Tutoring Systems, 7th International Conference, ITS 2004, vol. 3220 of Lecture Notes in Computer Science, pp. 443-454, Maceiò, Alagoas, Brazil, 30 August-3 September. Springer, Berlin (2004)
3. Anderson J.R.: The Architecture of Cognition. Harvard University Press, Cambridge (1983)
4. J.R. Anderson A.T. Corbett K.R. Koedinger R. Pelletier 1995 Cognitive tutors: lessons learned J. Learn. Sci. 4 2 167 207 10.1207/s15327809jls0402-2
5. Baker, R.S., Corbett, A.T., Koedinger, K.R.: Detecting student misuse of intelligent tutoring systems. In: Lester, J.C., Vicari, R.M., Paraguaçu, F. (eds.) Intelligent Tutoring Systems, 7th International Conference, ITS 2004, vol. 3220 of Lecture Notes in Computer Science, pp. 531-540, Maceiò, Alagoas, Brazil, 30 August-3 September. Springer, Berlin (2004a)
6. Baker R.S., Corbett A.T., Koedinger K.R., Wagner A.Z.: Off-task behavior in the cognitive tutor classroom: when students "game the system". In: Dykstra-Erickson, E., Tscheligi, M. (eds) CHI, pp. 383-390. ACM, New York (2004b)
7. Barnes T., Stamper J.C.: Toward automatic hint generation for logic proof tutoring using historical student data. In: Woolf, B.P., Aïmeur, E., Nkambou, R., Lajoie, S.P. (eds) Intelligent Tutoring Systems, vol. 5091 of Lecture Notes in Computer Science, pp. 373-382. Springer, Berlin (2008)
8. Beck, J., Woolf, B.P., Beal, C.R.: Advisor: a machine learning architecture for intelligent tutor construction. In: AAAI/IAAI, pp. 552-557. AAAI Press/The MIT Press, Menlo Park/Cambridge (2000)
9. Bernsen N.O., Dybkjaer L.: Designing Interactive Speech Systems: From First Ideas to User Testing. Springer-Verlag New York Inc, Secaucus (1997)
10. Chadés, I., Cros, M.-J., Garcia, F., Sabbadin, R.: Markov decision process (MDP) toolbox v2.0 for MATLAB (2005). http://www.inra.fr/internet/ Departements/MIA/T/MDPtoolbox
11. Chi, M.: Do Micro-level tutorial decisions matter: applying reinforcement learning to induce pedagogical tutorial tactics. PhD thesis, Intelligent Systems Program, University of Pittsburgh, December (2009)
12. M.T.H. Chi N. de Leeuw M.-H. Chiu C. LaVancher 1994 Eliciting self-explanations improves understanding Cogn. Sci. 18 3 439 477
13. Chi, M. Jordan, P.W., VanLehn, K., Hall, M.: Reinforcement learning-based feature selection for developing pedagogically effective tutorial dialogue tactics. In: de Baker, R.S.J., Barnes, T., Beck, J.E. (eds.) The 1st International Conference on Educational Data Mining (EDM), pp. 258-265. Montreal, Québec, Canada (2008). www.educationaldatamining.org
14. Chi M., Jordan P.W., VanLehn K., Litman D.J.: To elicit or to tell: does it matter?. In: Dimitrova, V., Mizoguchi, R., du Boulay, B., Graesser, A.C. (eds) AIED, pp. 197-204. IOS Press, Amsterdam (2009)
15. Chi M., VanLehn K., Litman D.J., Jordan P.W.: Inducing effective pedagogical strategies using learning context features. In: De Bra, P., Kobsa, A., Chin, D.N. (eds) UMAP, vol. 6075 of Lecture Notes in Computer Science, pp. 147-158. Springer, Berlin (2010)
16. A. Collins A. Stevens 1982 Goals and strategies for inquiry teachers Adv. Instr. Psychol. 2 65 119
17. Collins A., Brown J.S., Newman S.E.: Cognitive apprenticeship: teaching the craft of reading, writing and mathematics. In: Resnick, L.B. (eds) Knowing, learning and instruction: essays in honor of Robert Glaser, chap. 14, pp. 453-494. Lawrence Erlbaum Associates, Hillsdale (1989)
18. C. Conati K. VanLehn 2000 Toward computer-based support of meta-cognitive skills: a computational framework to coach self-explanation Int. J. Artif. Intell. Educ. 11 398 415
19. Corbett, A.T., Anderson, J.R.: Locus of feedback control in computer-based tutoring: impact on learning rate, achievement and attitudes. In: CHI, pp. 245-252, Seattle, Washington, USA, 2001
20. S.K. D'Mello A.C. Graesser 2010 Multimodal semi-automated affect detection from conversational cues, gross body language, and facial features User Model. User-Adapt. Interact. 20 2 147 187 10.1007/s11257-010-9074-4
21. S.K. D'Mello S.D. Craig A.M. Witherspoon B. McDaniel A.C. Graesser 2008 Automatic detection of learner's affect from conversational cues User Model. User-Adapt. Interact. 18 1-2 45 80 10.1007/s11257-007-9037-6
22. Forbes-Riley, K., Litman, D.J., Purandare, A., Rotaru, M., Tetreault, J.R.: Comparing linguistic features for modeling learning in computer tutoring. In: Luckin, R., Koedinger, K.R., Greer, J.E. (eds.): Artificial Intelligence in Education, Building Technology Rich Learning Contexts that Work, Proceedings of the 13th International Conference on Artificial Intelligence in Education, AIED 2007, vol. 158 of Frontiers in Artificial Intelligence and Applications, pp. 270-277, Los Angeles, California, USA, July 9-13. IOS Press, Amsterdam (2007)
23. Frampton, M., Lemon, O.: Reinforcement learning of dialogue strategies using the user's last dialogue act. In: Proceedings of the IJCAI Workshop on K&R in Practical Dialogue Systems, pp. 62-67 (2005)
24. Frampton, M., Lemon, O.: Learning more effective dialogue strategies using limited dialogue move features. In: Calzolari, N., Cardie, C., Isabelle, P. (eds.) ACL 2006, 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, Sydney, Australia, pp. 185-192, 17-21 July 2006. The Association for Computational Linguistics, Uppsala (2006)
25. A.C. Graesser N.K. Person J.P. Magliano 1995 Collaborative dialog patterns in naturalistic one-on-one tutoring Appl. Cogn. Psychol. 9 6 422 495 10.1002/acp.2350090604
26. A.C. Graesser K. VanLehn C.P. Rosé P.W. Jordan D. Harter 2001 Intelligent tutoring systems with conversational dialogue AI Magazine 22 4 39 52
27. Hauskrecht, M.: Planning and control in stochastic domains with imperfect information. PhD thesis, MIT (1997) (Available as Technical Report: MIT-LCS-TR-738, 1997)
28. Henderson, J., Lemon, O., Georgila, K.: Hybrid reinforcement/supervised learning for dialogue policies from communicator data. In: IJCAI Workshop on K&R in Practical Dialogue Systems, pp. 68-75, 2005
29. A. Iglesias P. Martínez F. Fernández 2003 An experience applying reinforcement learning in a web-based adaptive and intelligent educational system Infor. Educ. 2 2 223 240
30. A. Iglesias P. Martínez R. Aler F. Fernández 2009 Learning teaching strategies in an adaptive and intelligent educational system through reinforcement learning Appl. Intell. 31 89 106 10.1007/s10489-008-0115-1 10.1007/s10489-008-0115-1
31. Iglesias, A., Martínez, P., Aler, R., Fernández, F.: Reinforcement learning of pedagogical policies in adaptive and intelligent educational systems. Knowledge-Based Syst. 22(4), 266-270 (2009b) (Artificial Intelligence (AI) in Blended Learning)
32. Janarthanam, S., Lemon, O.: User simulations for online adaptation and knowledge-alignment in troubleshooting dialogue systems. In: Proceedings of LonDial the 12th SEMdial Workshop on the Semantics and Pragmatics of Dialogues, pp. 51-58, Stockholm, 2008
33. Jolliffee I.T.: Principal Component Analysis, Springer Series in Statistics, 2nd edn. Springer, New York (2002)
34. Jordan, P.W., Ringenberg, M.A., Hall, B.: Rapidly developing dialogue systems that support learning studies. In: ITS06 Workshop on Teaching with Robots, Agents and NLP, pp. 29-36 (2006). http://facweb.cs.depaul.edu/elulis/ ITS2006RobotsAgentsWorkshop.html
35. Jordan, P.W., Hall, B., Ringenberg, M., Cue, Y., Rosé, C.: Tools for authoring a dialogue agent that participates in learning studies. In: Luckin, R., Koedinger, K.R., Greer, J.E. (eds.) Artificial Intelligence in Education, Building Technology Rich Learning Contexts that Work, Proceedings of the 13th International Conference on Artificial Intelligence in Education, AIED 2007, vol. 158 of Frontiers in Artificial Intelligence and Applications, pp. 43-50, Los Angeles, CA, USA, July 9-13. IOS Press, Amsterdam (2007)
36. L.P. Kaelbling M.L. Littman A.W. Moore 1996 Reinforcement learning: a survey J. Artif. Intell. Res. 4 237 285 (Pubitemid 126646155)
37. S. Katz G. O'Donnell H. Kay 2000 An approach to analyzing the role and structure of reflective dialogue Int. J. Artif. Intell. Educ. 11 3 320 343
38. K.R. Koedinger V. Aleven 2007 Exploring the assistance dilemma in experiments with cognitive tutors Educ. Psychol. Rev 19 3 239 264 10.1007/s10648-007-9049-0
39. K.R. Koedinger J.R. Anderson W.H. Hadley M.A. Mark 1997 Intelligent tutoring goes to school in the big city Int. J. Artif. Intell. Educ. 8 1 30 43
40. Levin, E., Pieraccini, R.: A stochastic model of computer-human interaction for learning dialogue strategies. In: EUROSPEECH 97, pp. 1883-1886, 1997
41. Litman, D.J., Silliman, S.: Itspoke: an intelligent tutoring spoken dialogue system. In: Demonstration Papers at HLT-NAACL 2004, pp. 5-8. Association for Computational Linguistics, Morristown, NJ, USA (2004)
42. Martin, K.N., Arroyo, I.: Agentx: using reinforcement learning to improve the effectiveness of intelligent tutoring systems. In: Lester, J.C., Vicari, R.M., Paraguaçu, F. (eds.) Intelligent Tutoring Systems, 7th International Conference, ITS 2004, vol. 3220 of Lecture Notes in Computer Science, pp. 564-572, Maceiò, Alagoas, Brazil, 30 August-3 September. Springer, Berlin (2004)
43. J. McKendree 1990 Effective feedback content for tutoring complex skills Human-Computer Interaction 5 4 381 413 10.1207/s15327051hci0504-2
44. Moore J.D., Porayska-Pomsta K., Varges S., Zinn C.: Generating tutorial feedback with affect. In: Barr, V., Markov, Z. (eds) FLAIRS Conference, pp. 923-928. Menlo Park, (2004)
45. Newell, A.: Unified Theories of Cognition, Reprint edition. Harvard University Press, Cambridge (1994)
46. Paek, T., Chickering, D.: The Markov assumption in spoken dialogue management. In: 6th SIGDial Workshop on Discourse and Dialogue, pp. 35-44, 2005
47. Pain, H., Porayska-Pomsta, K.: Affect in one-to-one tutoring. In: Ikeda, M., Ashley, K.D., Chan, T.-W. (eds.): Intelligent Tutoring Systems, 8th International Conference, ITS 2006, p. 817, Jhongli, Taiwan, 26-30 June 2006, Proceedings, vol. 4053 of Lecture Notes in Computer Science. Springer, Berlin (2006) (Pubitemid 44117685)
48. Phobun, P., Vicheanpanya, J.: Adaptive intelligent tutoring systems for e-learning systems. Procedia Soc. Behav. Sci. 2(2), 4064-4069 (2010)
49. K. Porayska-Pomsta M. Mavrikis H. Pain 2008 Diagnosing and acting on student affect: the tutor's perspective User Model. User-Adapt. Interact. 18 1-2 125 173 10.1007/s11257-007-9041-x
50. Raux, A., Langner, B., Bohus, D., Black, A.W., Eskenazi, M.: Let's go public! Taking a spoken dialog system to the real world. In: Proceedings of Interspeech (Eurospeech), pp. 885-888, Lisbon Portugal, 2005 (Pubitemid 43908204)
51. Rieser, V., Lemon, O.: Using machine learning to explore human multimodal clarification strategies. In: Calzolari, N., Cardie, C., Isabelle, P. (eds.) ACL 2006, 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, Sydney, Australia, pp. 659-666, 17-21 July 2006. The Association for Computational Linguistics, Uppsala (2006)
52. Ringenberg, M.A., VanLehn, K.: Scaffolding problem solving with annotated, worked-out examples to promote deep learning. In: Ikeda, M., Ashley, K.D., Chan, T.-W. (eds.): Intelligent Tutoring Systems, 8th International Conference, ITS 2006, pp. 625-636, Jhongli, Taiwan, 26-30 June 2006, Proceedings, vol. 4053 of Lecture Notes in Computer Science. Springer, Berlin (2006)
53. Rudnicky, A., Thayer, E., Constantinides, P., Tchou, C., Shern, R., Lenzo, K., Xu, W., Oh, A.: Creating natural dialogs in the Carnegie Mellon communicator system. In: Proceedings of Eurospeech, vol. 4, pp. 1531-1534, 1999
54. Singh S.P., Kearns, M.J., Litman, D.J., Walker, M.A.: Reinforcement learning for spoken dialogue systems. In: Solla S.A., Leen, T.K., Müller, K.-R. (eds.) NIPS, pp. 956-962. The MIT Press, Cambridge (1999)
55. S.P. Singh D.J. Litman M.J. Kearns M.A. Walker A. Marilyn 2002 Optimizing dialogue management with reinforcement learning: Experiments with the NJfun system J. Aritif. Intell. Res. (JAIR) 16 105 133
56. Stamper, J.C., Barnes, T., Croy, M.J.: Extracting student models for intelligent tutoring systems. In: AAAI, pp. 1900-1901, Vancouver, British Columbia, Canada, July 22-26. AAAI Press, Stanford, CA (2007) (Pubitemid 350149863)
57. Sutton R.S., Barto A.G.: Reinforcement Learning. MIT Press Bradford Books, Cambridge (1998)
58. Tetreault, J.R., Litman, D.J.: Comparing the utility of state features in spoken dialogue using reinforcement learning. In: Moore, R.C., Bilmes, J.A., Chu-Carroll, J., Sanderson, M. (eds.) Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pp. 272-279. The Association for Computational Linguistics, New York (2006a)
59. Tetreault, J.R., Litman, D.J.: Using reinforcement learning to build a better model of dialogue state. In: Proceedings 11th Conference of the European Chapter of the Association for Computational Linguistics (EACL), pp. 289-296, Trento, Italy, 2006b
60. Tetreault J.R., Bohus D., Litman D.J.: Estimating the reliability of MDP policies: a confidence interval approach. In: Sidner, C.L., Schultz, T., Stone, M., Zhai, C. (eds) HLT-NAACL, pp. 276-283. The Association for Computational Linguistics, Boston (2007)
61. J.R. Tetreault D.J. Litman 2008 A reinforcement learning approach to evaluating state representations in spoken dialogue systems Speech Commun. 50 8-9 683 696 10.1016/j.specom.2008.05.002
62. K. VanLehn 2006 The behavior of tutoring systems Int. J. Artif. Intell. Educ. 16 3 227 265
63. K. VanLehn A.C. Graesser G.T. Jackson P.W. Jordan A. Olney C.P. Rosé 2007 When are tutorial dialogues more effective than reading? Cogn. Sci. 31 1 3 62 10.1080/03640210709336984
64. VanLehn, K., Jordan, P., Litman, D.: Developing pedagogically effective tutorial dialogue tactics: Experiments and a testbed. In: Proceedings of SLaTE Workshop on Speech and Language Technology in Education ISCA Tutorial and Research Workshop, pp. 17-20, 2007b
65. K. VanLehn C. Lynch K. Schulze J.A. Shapiro R. Shelby L. Taylor D. Treacy A. Weinstein M. Wintersgill 2005 The andes physics tutoring system: lessons learned Int. J. Artif. Intell. Educ. 15 3 147 204
66. Vygotsky, L.S.: Interaction between learning and development. In: Mind and Society, pp. 79-91. Harvard University Press, Cambridge MA (1978)
67. M.A. Walker 2000 An application of reinforcement learning to dialogue strategy selection in a spoken dialogue system for email J. Artif. Intell. Res. 12 387 416 0946.68116
68. Williams, J.D., Poupart, P., Young, S.J.: Factored partially observable Markov decision processes for dialogue management. In: 4th Workshop on Knowledge and Reasoning in Practical Dialog Systems, International Joint Conference on Artificial Intelligence (IJCAI), pp. 76-82, Edinburgh, 2005
69. J.D. Williams S. Young 2007 Partially observable Markov decision processes for spoken dialog systems Comput. Speech Lang. 21 2 231 422 10.1016/j.csl.2006.06.008 (Pubitemid 44709839)
70. J.D. Williams S. Young 2007 Scaling POMDPs for spoken dialog management IEEE Trans. Audio Speech Lang. Process. 15 7 2116 2129 10.1109/TASL.2007.902050
71. Wylie, R., Koedinger, K., Mitamura, T.: Is self-explanation always better? the effects of adding self-explanation prompts to an english grammar tutor. In: Proceedings of the 31st Annual Conference of the Cognitive Science Society, COGSCI 2009, pp. 1300-1305, Amsterdam, The Netherlands, 2009