Leveraging machine-learned detectors of systematic inquiry behavior to estimate and predict transfer of inquiry skill

User Modeling and User-Adapted Interaction

Volumn 23, Issue 1, 2013, Pages 1-39

  Sao Pedro, Michael A ^a   De Baker, Ryan S J ^a   Gobert, Janice D ^a   Montalvo, Orlando ^a   Nakama, Adam ^a  

^a WORCESTER POLYTECHNIC INSTITUTE   (United States)

Author keywords

Bayesian Knowledge Tracing; Behavior detection; Designing and conducting experiments; Exploratory learning environment assessment; Machine learned models; Microworlds; Scientific inquiry; Skill prediction

Indexed keywords

BAYESIAN KNOWLEDGE-TRACING; BEHAVIOR DETECTION; EXPLORATORY LEARNING; MICROWORLDS; SCIENTIFIC INQUIRY;

ESTIMATION; EXPERIMENTS; FORECASTING; SCAFFOLDS;

STUDENTS;

EID: 84872417052     PISSN: 09241868     EISSN: 15731391     Source Type: Journal    
DOI: 10.1007/s11257-011-9101-0     Document Type: Article

References (88)

1. Aleven V.; McLaren B.; Roll I.; Koedinger K.: Toward meta-cognitive tutoring: a model of help seeking with a cognitive tutor. Int. J. Artif. Intell. Educ. 16(2), 101-130 (2006)
2. Alonzo, A.; Aschbacher, P.: Value added? Long assessment of students' scientific inquiry skills. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA. Retrieved December 20, 2010, from the AERA Online Paper Repository (2004)
3. Amershi S.; Conati C.: Combining unsupervised and supervised machine learning to build user models for exploratory learning environments. J. Educ. Data Min. 1(1), 71-81 (2009)
4. Azevedo R.: Theoretical, conceptual, methodological, and instructional issues in research on metacognition and self-regulated learning: a discussion. Metacogn. Learn. 4(1), 87-95 (2009)
5. Baker, R.; de Carvalho, A.: Labeling student behavior faster and more precisely with text replays. In: Baker, R.S.; Barnes, T.; Beck, J.E. (eds.) Proceedings of the 1st International Conference on Educational Data Mining, EDM 2008, Montreal, QC, Canada, pp. 38-47 (2008)
6. Baker R.; Yacef K.: The state of educational data mining in 2009: a review and future visions. J. Educ. Data Min. 1(1), 3-17 (2009)
7. Baker, R.; Corbett, A.; Wagner, A.: Human classification of low-fidelity replays of student actions. In: Proceedings of the Educational Data Mining Workshop held at the 8th International Conference on Intelligent Tutoring Systems, ITS 2006, Jhongli, Taiwan, pp. 29-36 (2006)
8. Baker R.S.; Corbett A.T.; Roll I.; Koedinger K.R.: Developing a generalizable detector of when students game the system. User Model. User-Adapt. Interact. 18(3), 287-314 (2008a)
9. Baker, R.; Corbett, A.; Aleven, V.: Improving contextual models of guessing and slipping with a truncated training set. In: Baker, R.S.; Barnes, T.; Beck, J.E. (eds.) Proceedings of the 1st International Conference on Educational Data Mining, EDM 2008, Montreal, QC, Canada, pp. 67-76 (2008b)
10. Baker, R.; Corbett, A.; Aleven, V.: More accurate student modeling through contextual estimation of slip and guess probabilities in Bayesian knowledge-tracing. In: Woolf, B.; Aimeur, E.; Nkambou, R.; Lajoie, S. (eds.) Proceedings of the 9th International Conference on Intelligent Tutoring Systems, ITS 2008, Montreal, QC, Canada. LNCS 5091, pp. 406-415. Springer (2008c)
11. Baker, R.S.; Mitrovic, A.; Mathews, M.: Detecting gaming the system in constraint-based tutors. In: De Bra, P.; Kobsa, P.; Chin, D. (eds.) Proceedings of the 18th Annual Conference on User Modeling, Adaptation and Personalization, UMAP 2010, Big Island of Hawaii, HI. LNCS 6075, pp. 267-278. Springer (2010a)
12. Baker, R.; Corbett, A.; Gowda, S.; Wagner, A.; MacLaren, B.; Kauffman, L.; et al.: Contextual slip and prediction of student performance after use of an intelligent tutor. In: De Bra, P.; Kobsa, P.; Chin, D. (eds.) Proceedings of the 18th Annual Conference on User Modeling, Adaptation and Personalization, UMAP 2010, Big Island of Hawaii, HI, LNCS 6075, pp. 52-63. Springer (2010b)
13. Baker, R.; Pardos, Z.; Gowda, S.; Nooraei, B.; Heffernan, N.: Ensembling predictions of student knowledge within intelligent tutoring systems. In: Konstan, J.; Conejo, R.; Marzo, J.; Oliver, N. (eds.) Proceedings of the 19th International Conference on User Modeling, Adaptation and Personalization, UMAP 2011, Girona, Spain, LNCS 6787, pp. 13-24. Springer (2011)
14. Beck J.: Engagement tracing: using response times to model student disengagement. In: Looi, C.K.; McCalla, G.; Bredeweg, B.; Breuker, J. (eds) Proceedings of the 12th International Conference on Artificial Intelligence in Education, AIED 2005, pp. 88-95. IOS Press, Amsterdam (2005)
15. Beck J.; Sison J.: Using knowledge tracing in a noisy environment to measure student reading proficiencies. Int. J. Artif. Intell. Educ. 16(2), 129-143 (2006)
16. Ben-David A.: About the relationship between ROC curves and Cohen's kappa. Eng. Appl. Artif. Intell. 21, 874-882 (2008)
17. Bernardini A.; Conati C.: Discovering and recognizing student interaction patterns in exploratory learning environments. In: Aleven, V.; Kay, J.; Mostow, J (eds) Proceedings of the 10th International Conference of Intelligent Tutoring Systems, ITS 2010, Part 1, pp. 125-134. Springer, Pittsburgh, PA (2010)
18. Black P.: Testing: Friend or Foe? Theory and Practice of Assessment and Testing. Falmer Press, New York (1999)
19. Boyd S.; Vandenberghe L.: Convex Optimization. Cambridge University Press, Cambridge (2004)
20. Buckley, B.C.; Gobert, J.D.; Horwitz, P.: Using log files to track students' model-based inquiry. In: Barab, S.; Hay, K.; Hickey, D. (eds.) Proceedings of the 7th International Conference on Learning Sciences, ICLS 2006, Bloomington, IN, pp. 57-63. Lawrence Erlbaum Associates (2006)
21. Buckley B.; Gobert J.; Horwitz P.; O'Dwyer L.: Looking inside the black box: assessments and decision-making in BioLogica. Int. J. Learn. Technol. 5(2), 166-190 (2010)
22. Cetintas S.; Si L.; Xin Y.; Hord C.: Automatic detection of off-task behaviors in intelligent tutoring systems with machine learning techniques. IEEE Trans. Learn. Technol. 3(3), 228-236 (2010)
23. Chen Z.; Klahr D.: All other things being equal: acquisition and transfer of the control of variables strategy. Child Dev. 70(5), 1098-1120 (1999)
24. Cocea M.; Weibelzahl S.: Log file analysis for disengagement detection in e-learning environments. User Model. User-Adapt. Interact. 19, 341-385 (2009)
25. Corbett A.; Anderson J.: Knowledge-tracing: modeling the acquisition of procedural knowledge. User Model. User-Adapt. Interact. 4, 253-278 (1995)
26. de Jong T.: Computer simulations-technological advances in inquiry learning. Science 312(5773), 532-533 (2006)
27. de Jong T.; Beishuizenm J.; Hulshof C.; Prins F.; van Rijn H.; van Someren M. et al.: Determinants of discovery learning in a complex simulation learning environment. In: Gardenfors, P.; Johansson, P. (eds) Cognition, Education and Communication Technology, pp. 257-283. Lawrence Erlbaum Associates, Mahwah (2005)
28. Dean D. Jr.; Kuhn D.: Direct instruction vs. discovery: the long view. Sci. Educ. 91, 384-397 (2006)
29. Dignath C.; Buttner G.: Components of fostering self-regulated learning among students: a meta-analysis on intervention studies at primary and secondary school level. Metacogn. Learn. 3(3), 231-264 (2008)
30. Dragon, T.; Woolf, B.;0 Marshall, D.; Murray, T.: Coaching within a domain independent inquiry environment. In: Ikeda, M.; Ashley, K.D.; Tak-Wai, C. (eds.) Proceedings of the 8th International Conference on Intelligent Tutoring Systems, ITS 2006, Jhongli, Taiwan. LNCS 4053, pp. 144-153. Springer (2006)
31. Efron B.; Gong G.: A leisurely look at the bootstrap, the jackknife, and cross-validation. Am. Stat. 37(1), 36-48 (1983)
32. Feng M.; Heffernan N.; Koedinger K.: Addressing the assessment challenge in an intelligent tutoring system that tutors as it assesses. User Model. User-Adapt. Interact. 19, 243-266 (2009)
33. Ferguson G.: Statistical Analysis in Psychology and Education, 4th edn. McGraw-Hill Inc.; New York (1976)
34. Fogarty, J.; Baker, R.; Hudson, S.: Case studies in the use of ROC curve analysis for sensor-based estimates in human computer interaction. In: Proceedings of Graphics Interface, GI 2005, pp. 129-136. Canadian Human-Computer Communications Society, Victoria (2005)
35. Gertner, A.; VanLehn, K.: Andes: A coached problem solving environment for physics. In: Gauthier, G.; Frasson, C.; VanLehn, K. (eds.) Proceedings of the 5th International Conference on Intelligent Tutoring Systems, ITS 2000, Montreal, QC, Canada. LNCS 1839, pp. 133-142. Springer (2000)
36. Ghazarian A.; Noorhosseini S.M.: Automatic detection of users' skill levels using high-frequency user interface events. User Model. User-Adapt. Interact. 20(2), 109-146 (2010)
37. Glaser R.; Schauble L.; Raghavan K.; Zeitz C.: Scientific reasoning across different domains. In: DeCorte, E.; Linn, M.; Mandl, H.; Verschaffel, L. (eds) Computer-Based Learning Environments and Problem-Solving, pp. 345-371. Springer, Heidelberg (1992)
38. Gobert J.: Leveraging technology and cognitive theory on visualization to promote students' science learning and literacy. In: Gilbert, J. (eds) Visualization in Science Education, pp. 73-90. Springer, Dordrecht (2005)
39. Gobert, J.; Buckley, B.; Levy, S.; Wilensky, U.: Teasing apart domain-specific and domain-general inquiry skills: co-evolution, bootstrapping, or separate paths? Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL. Retrieved April 18, 2011, from the AERA Online Paper Repository (2007a)
40. Gobert, J.; Heffernan, N.; Ruiz, C.; Kim, R.: AMI: ASSISTments Meets Inquiry. Proposal NSF-DRL# 0733286 funded by the National Science Foundation (2007b)
41. Gobert, J.; Heffernan, N.; Koedinger, K.; Beck, J.: ASSISTments Meets Science Learning (AMSL). Proposal (R305A090170) funded by the U.S. Department of Education (2009)
42. Gobert, J.; Raziuddin, J.; Sao Pedro, M.: The influence of learner characteristics on conducting scientific inquiry within microworlds. In: Carlson, L.; Hoelscher, C.; Shipley, T. (eds.) Proceedings of the 33rd Annual Conference of the Cognitive Science Society, Boston, MA, pp. 372-377. Cognitive Science Society (2011)
43. Gong, Y.; Beck, J.; Heffernan, N.: Using multiple Dirichlet distributions to improve parameter plausibility. In: Baker, R.S.; Merceron, A.; Pavlik, P.I.; Jr. (eds.) Proceedings of the 3rd International Conference on Educational Data Mining, EDM 2010, Pittsburgh, PA, pp. 61-70 (2010)
44. Gotwals, A.; Songer, N.: Measuring students' scientific content and inquiry reasoning. In: Barab, S.; Hay, K.; Hickey, D. (eds.) Proceedings of the 7th International Conference of the Learning Sciences, ICLS 2006, Bloomington, IN, pp. 196-202. Lawrence Erlbaum Associates (2006)
45. Graesser A.; Chipman P.; Haynes B.; Olney A.: AutoTutor: an intelligent tutoring system with mixed-initiative dialogue. IEEE Trans. Educ. 48(4), 612-618 (2005)
46. Gu, T.; Wu, Z.; Tao, X.; Pung, H.K.; Lu, J.: epSICAR: an emerging patterns based approach to sequential, interleaved and concurrent activity recognition. In: Proceedings of the 2009 IEEE International Conference on Pervasive Computing and Communications, PERCOM '09, Galveston, TX, pp. 1-9. IEEE Computer Society (2009)
47. Hadwin A.; Nesbit J.; Jamieson-Noel D.; Code J.; Winne P.: Examining trace data to explore self-regulated learning. Metacogn. Learn. 2(2-3), 107-124 (2007)
48. Hanley J.; McNeil B.: The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143, 29-36 (1982)
49. Köck M.; Paramythis A.: Activity sequence modelling and dynamic clustering for personalized e-learning. User Model. User-Adapt. Interact. 21, 51-97 (2011)
50. Koedinger K.; Corbett A.: Cognitive tutors: technology bringing learning sciences to the classroom. In: Sawyer, R. (eds) The Cambridge Handbook of the Learning Sciences, pp. 61-77. Cambridge University Press, New York (2006)
51. Koedinger K.; Suthers D.; Forbus K.: Component-based construction of a science learning space. Int. J. Artif. Intell. Educ. 10, 292-313 (1998)
52. Kuhn D.: Education for Thinking. Harvard University Press, Cambridge (2005)
53. Levy, S.; Wilensky, U.: Emerging knowledge through an emergent perspective: high-school students' inquiry, exploration and learning in the connected chemistry curriculum. Presented at the annual meeting of the American Educational Research Association, San Francisco, CA, 11 April 2006. Retrieved April 18, 2011 from the AERA Online Paper Repository
54. Manlove, S.; Lazonder, A.: Self-regulation and collaboration in a discovery learning environment. Paper presented at the first EARLI metacognition SIG conference, Amsterdam. http://users.edte.utwente.nl/lazonder/homepage/NL/ Publijst.html (2004)
55. Manlove S.; Lazonder A.; Dejong T.: Software scaffolds to promote regulation during scientific inquiry learning. Metacogn. Learn. 2, 141-155 (2007)
56. Massachusetts Department of Education: Massachusetts Science and Technology/Engineering Curriculum Framework. Massachusetts Department of Education, Malden (2006)
57. McElhaney K.; Linn M.: Helping students make controlled experiments more informative. In: Gomez, K.; Lyons, L.; Radinsky, J. (eds) Learning in the Disciplines: Proceedings of the 9th International Conference of the Learning Sciences (ICLS 2010) - Volume 1, Full Papers, pp. 786-793. International Society of the Learning Sciences, Chicago (2010)
58. Mierswa, I.; Wurst, M.; Klinkenberg, R.; Scholz, M.; Euler, T.: YALE: rapid prototyping for complex data mining tasks. In: Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2006, Philadelphia, PA, pp. 935-940. ACM Press (2006)
59. Mitrovic A.: An intelligent SQL tutor on the web. Int. J. Artif. Intell. Educ. 13(2-4), 173-197 (2003)
60. Mitrovic A.; Mayo M.; Suraweera P.; Martin, B.: Constraint-based tutors: a success story. In: Monostori, L.; Vancza, J.; Ali, M (eds) Proceedings of the 14th International Conference on Industrial and Engineering Application of Artificial Intelligence and Expert Systems: Engineering of Intelligent Systems, IEA/AIE-2001. LNCS 2070, pp. 931-940. Springer, Budapest, Hungary (2001)
61. National Research Council: National Science Education Standards. National Academy Press, Washington, DC (1996)
62. Papert S.: Computer-based microworlds as incubators for powerful ideas. In: Taylor, R. (eds) The Computer in the School: Tutor, Tool, Tutee, pp. 201-203. Teacher's College Press, New York (1980)
63. Pardos, Z.; Heffernan, N.: Navigating the parameter space of Bayesian knowledge-tracing models: visualizations of the convergence of the expectation maximization algorithm. In: Baker, R.S.; Merceron, A.; Pavlik, P.I.; Jr. (eds.) Proceedings of the 3rd International Conference on Educational Data Mining, Pittsburgh, PA, pp. 161-170 (2010)
64. Pardos Z.; Heffernan N.; Anderson B.; Heffernan C.: Using fine-grained skill models to fit student performance with Bayesian networks. In: Romero, C.; Ventura, S.; Viola, S.R.; Pechenizkiy, M.; Baker, R.S. (eds) Handbook of Educational Data Mining, pp. 417-426. CRC Press, Boca Raton (2010)
65. Pavlik, P.; Cen, H.; Koedinger, J.: Performance factors analysis - a new alternative to knowledge tracing. In: Dimitrova, V.; Mizoguchi, R.; du Boulay, B.; Graesser, A. (eds.) Proceedings of the 14th International Conference on Artificial Intelligence in Education, AIED 2009, Brighton, UK, pp. 531-540. IOS Press (2009)
66. Pea R.; Kurland D.: On the cognitive effects of learning computer programming. New Ideas Psychol. 2, 137-168 (1984)
67. Pellegrino J.: Rethinking and Redesigning Educational Assessment: Preschool Through Postsecondary. Education Commission of the States, US Department of Education, Denver (2001)
68. Resnick M.: Turtles, Termintes, and Traffic Jams: Explorations in Massively Parallel Microworlds. MIT Press, Cambridge (1997)
69. Reye J.: Student modeling based on belief networks. Int. J. Artif. Intell. Educ. 14(1), 1-33 (2004)
70. Ritter, S.; Harris, T.; Nixon, T.; Dickinson, D.; Murray, R.; Towle, B.: Reducing the knowledge-tracing space. In: Barnes, T.; Desmarais, M.; Romero, C.; Ventura, S. (eds.) Proceedings of the 2nd International Conference on Educational Data Mining, EDM 2009, Cordoba, Spain, pp. 151-160 (2009)
71. Roll, I.; Aleven, V.; Koedinger, K.: The invention lab: using a hybrid of model tracing and constraint-based modeling to offer intelligent support in inquiry environments. In: Aleven, V.; Kay, J.; Mostow, J. (eds.) Proceedings of the 10th International Conference on Intelligent Tutoring Systems, ITS 2010, Pittsburgh, PA, pp. 115-124. Springer (2010)
72. Romero C.; Ventura S.: Educational data mining: a review of the state-of-the-art. IEEE Trans. Syst. Man Cybernet. C 40(6), 601-618 (2010)
73. Rowe J.; Lester J.: Modeling user knowledge with dynamic Bayesian networks in interactive narrative environments. In: Youngblood, C.G.; Bulitko, V (eds) Proceedings of the 6th Annual AI and Interactive Digital Entertainment Conference, AIIDE 2010, pp. 57-62. AAAI Press, Palo Alto, CA (2010)
74. Sao Pedro M.; Gobert J.; Heffernan N.; Beck J.: Comparing pedagogical approaches for teaching the control of variables strategy. In: Taatgen, N.A.; Rijn, H (eds) Proceedings of the 31st Annual Meeting of the Cognitive Science Society, pp. 1294-1299. Cognitive Science Society, Amsterdam, Netherlands (2009)
75. Sao Pedro M.A.; Gobert J.D.; Raziuddin J.: Comparing pedagogical approaches for the acquisition and long-term robustness of the control of variables strategy. In: Gomez, K.; Lyons, L.; Radinsky, J. (eds) Learning in the Disciplines: Proceedings of the 9th International Conference of the Learning Sciences, ICLS 2010, Volume 1, Full Papers, pp. 1024-1031. International Society of the Learning Sciences, Chicago (2010)
76. Schauble L.; Klopfer L.; Raghavan K.: Students' transition from an engineering model to a science model of experimentation. J. Res. Sci. Teach. 28(9), 859-882 (1991)
77. Schraw G.: The use of computer-based environments for understanding and improving self-regulation. Metacogn. Learn. 2(2-3), 169-176 (2007)
78. Schraw G.: A conceptual analysis of five measures of metacognitive monitoring. Metacogn. Learn. 4(1), 33-45 (2009)
79. Schunn C.D.; Anderson J.R.: Scientific discovery. In: Anderson, J.R. (eds) The Atomic Components of Thought, pp. 385-428. Lawrence Erlbaum Associates, Mahwah (1998)
80. Shores, L.; Rowe, J.; Lester, J.: Early prediction of cognitive tool use in narrative-centered learning environments. In: Biswas, G.; Bull, S.; Kay, J.; Mitrovic, A. (eds.) Proceedings of the 15th International Conference on Artificial Intelligence in Education, AIED 2011, Auckland, New Zealand. LNCS 6738, pp. 320-327. Springer (2011)
81. Siler, S.; Klahr, D.; Magaro, C.; Willows, K.; Mowery, D.: Predictors of transfer of experimental design skills in elementary and middle school children. In: Aleven, V.; Kay, J.; Mostow, J. (ed.) Proceedings of the 10th International Conference on Intelligent Tutoring Systems, ITS 2010, Part II, LNCS 6095, Pittsburgh, PA, pp. 198-208. Springer (2010)
82. Stevens, R.; Soller, A.; Cooper, M.; Sprang, M.: Modeling the development of problem solving skills in chemistry with a web-based tutor. In: Lester, J.C.; Vicari, R.M.; Paraguacu, F. (eds.) Proceedings of the 7th International Conference on Intelligent Tutoring Systems, ITS 2004, Maceio, Alagoas, Brazil. LNCS 3220, pp. 580-591. Springer (2004)
83. Strand-Cary M.; Klahr D.: Developing elementary science skills: instructional effectiveness and path independence. Cogn. Dev. 23(4), 488-511 (2008)
84. van der Aalst W.: Process Mining: Discovery, Conformance and Enhancement of Business Processes. Springer, Berlin (2011)
85. Veenman M.; Van Hout-Worters B.; Afflerback P.: Metacognition and learning: conception and methodological considerations. Metacogn. Learn. 1(1), 3-14 (2006)
86. Walonoski, J.; Heffernan, N.: Detection and analysis of off-task gaming behavior in intelligent tutoring systems. In: Ikeda, M.; Ashlay, K.; Chan, T.-W. (eds.) Proceedings of the 8th International Conference on Intelligent Tutoring Systems, ITS 2006, Johngli, Taiwan. LNCS 4053, pp. 382-391. Springer (2006)
87. Winne P.; Hadwin A.: Studying as self-regulated learning. In: Hacker, D.J.; Dunlosky, J.; Graesser, A. (eds) Metacognition in Educational Theory and Practice, pp. 277-304. Lawrence Erlbaum Associates, Hillsdale (1998)
88. Winne P.; Nesbit J.; Kumar V.; Hadwin A.; Lajoie S.; Azevedo R. et al.: Supporting self-regulated learning with gStudy software: the learning kit project. Technol. Instr. Cogn. Learn. J. 3, 105-113 (2006)