Operationalizing and Detecting Disengagement Within Online Science Microworlds

Educational Psychologist

Volumn 50, Issue 1, 2015, Pages 43-57

  Gobert, Janice D ^a   Baker, Ryan S ^b   Wixon, Michael B ^a  

^a WORCESTER POLYTECHNIC INSTITUTE   (United States)

^b Och Spine at New York Presbyterian Hospitals   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84924276802     PISSN: 00461520     EISSN: None     Source Type: Journal    
DOI: 10.1080/00461520.2014.999919     Document Type: Article

References (84)

1. Aleven, V., McLaren, B., Roll, I., & Koedinger, K.R. (2006). Toward meta-cognitive tutoring: A model of help-seeking with a cognitive tutor. International Journal of Artificial Intelligence in Education, 16, 101–130.
2. Annetta, L. A., Minogue, J., Holmes, S. Y., & Cheng, M. T. (2009). Investigating the impact of video games on high school students' engagement and learning about genetics. Computers & Education, 53, 74–85. http://dx.doi.org/10.1016/j.compedu.2008.12.020
3. Appleton, J. J., Christenson, S. L., & Furlong, M. J. (2008). Student engagement with school: Critical conceptual and methodological issues of the construct. Psychology in the Schools, 45, 369–386. http://dx.doi.org/10.1002/pits.20303
4. Appleton, J. J., Christenson, S. L., Kim, D., & Reschly, A. L. (2006). Measuring cognitive and psychological engagement: Validation of the student engagement instrument. Journal of School Psychology, 44, 427–445. http://dx.doi.org/10.1016/j.jsp.2006.04.002
5. Arroyo, I., Ferguson, K., Johns, J., Dragon, T., Meheranian, H., Fisher, D., & Woolf, B. P. (2007). Repairing disengagement with non-invasive interventions. In R. Luckin, K. R. Koedinger, & J. E. Greer (Eds.), Artificial Intelligence in education: Building technology rich learning contexts that work (pp. 195–202). Amsterdam, The Netherlands: IOS.
6. Autor, D., Levy, F., & Murnane, R.J. (2003). The skill content of recent technological change: An empirical exploration. The Quarterly Journal of Economics, 118, 1279–1333. http://dx.doi.org/10.1162/003355303322552801
7. Baker, R. S. J. d. (2007). Modeling and understanding students' off-task behavior in intelligent tutoring systems. In B. Begole, S. Payne, E. Churchill, R. St. Amant, D. Gilmore, & M. B. Rosson (Eds.), Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 1059–1068). New York, NY: ACM.
8. Baker, R., & Carvalho, A. (2008). Labeling student behavior faster and more precisely with text replays. In R. S. J. d. Baker, T. Barnes, & J. E. Beck (Eds.), Educational Data Mining 2008: 1st International Conference on Educational Data Mining, Proceedings (pp. 38–47). Montreal, Quebec, Canada.
9. Baker, R. S. J. d., Carvalho, A. M. J. B. D., Raspat, J., Aleven, V., Corbett, A. T., & Koedinger, K. R. (2009). Educational software features that encourage and discourage “Gaming the System.” In V. Dimitrova, R. Mizoguchi, B. du Boulay, & A. Graesser (Eds.), Proceedings of the 2009 Conference on Artificial Intelligence in Education: Building Learning Systems that Care: From Knowledge Representation to Affective Modelling (pp. 475–482). Amsterdam, The Netherlands: IOS.
10. Baker, R., Corbett, A., & Koedinger, K. (2004). Detecting student misuse of intelligent tutoring systems. In J. Lester, R. Vicari, & F. Paraguaçu (Eds.), Intelligent tutoring systems (vol. 3220, pp. 531–540). Berlin, Germany: Springer.
11. Baker, R. S., Corbett, A. T., Koedinger, K. R., & Wagner, A. Z. (2004). Off-task behavior in the cognitive tutor classroom: When students “Game the System.” In E. Dykstra-Erickson & M. Tscheligi (Eds.), Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 383–390). New York, NY: ACM.
12. Baker, R. S., Corbett, A. T., & Wagner, A. Z. (2006). Human classification of low-fidelity replays of student actions. In Proceedings of the 8th International Conference on Intelligent Tutoring Systems (vol. LNCS 4053, pp. 29–36). Jhongli, Taiwan.
13. Baker, R. S., D'Mello, S. K., Rodrigo, M. M. T., & Graesser, A. C. (2010). Better to be frustrated than bored: The incidence, persistence, and impact of learners’ cognitive-affective states during interactions with three different computer-based learning environments. International Journal of Human-Computer Studies, 68, 223–241. http://dx.doi.org/10.1016/j.ijhcs.2009.12.003
14. Baker, R. S. J. d., Gowda, S. M., Wixon, M., Kalka, J., Wagner, A. Z., Salvi, A., … Rossi, L. (2012). Towards sensor-free automated detection of affect in Cognitive Tutor Algebra. In K. Yacef, O. Zaïane, H. Hershkovitz, M. Yudelson, & J. Stamper (Eds.), Proceedings of the 5th International Conference on Educational Data Mining (pp. 126–133). Chania, Greece.
15. Baker, R. S. J. d., Mitrovic, A., & Mathews, M. (2010). Detecting gaming the system in constraint-based tutors. In P. Bra, A. Kobsa, & D. Chin (Eds.), Proceedings of the 18th International Conference on User Modeling, Adaptation, and Personalization (pp. 267–278). Berlin, Germany: Springer-Verlag.
16. Baker, R., Walonoski, J., Heffernan, N., Roll, I., Corbett, A., & Koedinger, K. (2008). Why students engage in “gaming the system” behavior in interactive learning environments. Journal of Interactive Learning Research, 19, 185–224.
17. Beal, C. R., Qu, L., & Lee, H. (2008). Mathematics motivation and achievement as predictors of high school students' guessing and help‐seeking with instructional software. Journal of Computer Assisted Learning, 24, 507–514. http://dx.doi.org/10.1111/j.1365-2729.2008.00288.x
18. Beck, J. (2005). Engagement tracing: Using response times to model student disengagement. Artificial Intelligence in Education: Supporting Learning Through Intelligent and Socially Informed Technology, 125, 88.
19. Birch, S., & Ladd, G. (1997) The teacher-child relationship and children's early school adjustment. Journal of School Psychology, 35, 61–79. http://dx.doi.org/10.1016/S0022-4405(96)00029-5
20. Blackwell, L., Trzesniewski, K., & Dweck, C. S. (2007). Implicit theories of intelligence predict achievement across an adolescent transition: A longitudinal study and an intervention. Child Development, 78, 246–263. http://dx.doi.org/10.1111/j.1467-8624.2007.00995.x
21. Buckley, B. C., Gobert, J., Horwitz, P., & O'Dwyer, L. (2010). Looking inside the black box: Assessments and decision-making in BioLogica. International Journal of Learning Technologies, 5, 166–190. http://dx.doi.org/10.1504/IJLT.2010.034548
22. Cetintas, S., Si, L., Xin, Y. P., & Hord, C. (2009). Automatic detection of off-task behaviors in intelligent tutoring systems with machine learning techniques. Learning Technologies, IEEE Transactions on, 3, 228–236. http://dx.doi.org/10.1109/TLT.2009.44
23. Christenson, S., Reschly, A. L., & Wylie, C. (2012). Handbook of research on student engagement. New York, NY: Springer. http://dx.doi.org/10.1007/978-1-4614-2018-7
24. Cocea, M., Hershkovitz, A., & Baker, R. S. J. d. (2009). The impact of off-task and gaming behaviors on learning: Immediate or aggregate? In Proceedings of the 14th International Conference on Artificial Intelligence in Education (pp. 507–514).
25. Csíkszentmihályi, M. (1990). Flow: The psychology of optimal experience. New York, NY: Harper & Row.
26. Davis, J., & Goadrich, M. (2006). The relationship between Precision-Recall and ROC curves. In W. Cohen & A. Moore (Chairs), Proceedings of the 23rd International Conference on Machine learning (pp. 233–240). New York, NY: ACM.
27. Duckworth, A. L., Peterson, C., Matthews, M. D., & Kelly, D. R. (2007). Grit: Perseverance and passion for long-term goals. Journal of Personality and Social Psychology, 92, 1087–1101. http://dx.doi.org/10.1037/0022-3514.92.6.1087
28. Efron, B., & Gong, G. (1983). A leisurely look at the bootstrap, the jackknife, and cross-validation. The American Statistician, 37, 36–48.
29. Finn, J. D., & Kasza, K. A. (2009). Disengagement from school. In Engaging young people in learning: Why does it matter and what can we do? (pp. 4–35). Wellington, New Zealand: New Zealand Council for Educational Research.
30. Finn, J. D., & Rock, D. A. (1997). Academic success among students at risk for school failure. Journal of Applied Psychology, 82, 221–234. http://dx.doi.org/10.1037/0021-9010.82.2.221
31. Frank, E., & Witten, I. H. (1998). Generating accurate rule sets without global optimization. In J. W. Shavlik (Eds.), Proceedings of the Fifteenth International Conference on Machine Learning (pp. 144–151). San Francisco, CA: Morgan Kaufmann.
32. Frederick, W., & Walberg, H. (1980). Learning as a function of time. The Journal of Educational Research, 73, 183–194. http://dx.doi.org/10.1080/00220671.1980.10885233
33. Fredricks, J. A., Blumenfeld, P. C., & Paris, A. (2004). School engagement: Potential of the concept: State of the evidence. Review of Educational Research, 74, 59–119. http://dx.doi.org/10.3102/00346543074001059
34. Fredricks, J., McColskey, W., Meli, J., Mordica, J., Montrosse, B., & Mooney, K. (2011). Measuring student engagement in upper elementary through high school: A description of 21 instruments. Issues & Answers Report, REL, 98, 098.
35. Glanville, L., & Wildhagen, T. (2007). The measurement of school engagement: Assessing dimensionality and measurement in variance across race and ethnicity. Educational and Psychological Measurement, 6, 1019–1041. http://dx.doi.org/10.1177/0013164406299126
36. Gobert, J. D., Sao Pedro, M. A., Baker, R. S., Toto, E., & Montalvo, O. (2012). Leveraging educational data mining for real-time performance assessment of scientific inquiry skills within microworlds. JEDM-Journal of Educational Data Mining, 4, 111–143.
37. Gobert, J., Sao Pedro, M., Raziuddin, J., & Baker, R. S. (2013). From log files to assessment metrics for science inquiry using educational data mining. Journal of the Learning Sciences, 22, 521–563. http://dx.doi.org/10.1080/10508406.2013.837391
38. Greene, B. A. (this issue). Measuring cognitive engagement with self-report scales: Reflections from over 20 years of research. Educational Psychologist, 50.
39. Hanley, J. A., & McNeil, B. J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143, 29–36. http://dx.doi.org/10.1148/radiology.143.1.7063747
40. Harmer, A. J., & Ward, M. C. (2007). Designing for learner engagement in middle school science: Technology, inquiry, and the hierarchies of engagement. Computers in the Schools, 24, 1–2, 105–124. http://dx.doi.org/10.1300/J025v24n01_08
41. Hershkovitz, A., Baker, R. S. J. d., Gobert, J., & Nakama, A. (2012). A data-driven path model of student attributes, affect, and engagement in a computer-based science inquiry microworld. In J. van Aalst, K. Thompson, M. J. Jacobson, & P. Reimann (Eds.), The Future of Learning: Proceedings of the 10th International Conference of the Learning Sciences (vol. 1, pp. 167–174). Sydney, NSW, Australia: International Society of the Learning Sciences.
42. Hug, B., Krajcik, J. S., & Marx, R. W. (2005). Using innovative learning technologies to promote learning and engagement in an urban science classroom. Urban Education, 40, 446–472. http://dx.doi.org/10.1177/0042085905276409
43. Karweit, N. L., & Slavin, R. E. (1981). Measurement and modeling choices in studies of time and learning. American Educational Research Journal, 18, 157–171. http://dx.doi.org/10.3102/00028312018002157
44. Klem, A. M., & Connell, J. P. (2004). Relationships matter: Linking teacher support to student engagement and achievement. Journal of School Health, 74, 262–273. http://dx.doi.org/10.1111/j.1746-1561.2004.tb08283.x
45. Kong, Q., Wong, N., & Lam, C. (2003). Student engagement in mathematics: Development of instrument and validation of a construct. Mathematics Education Research Journal, 54, 4–21. http://dx.doi.org/10.1007/BF03217366
46. Lahaderne, H. M. (1968). Attitudinal and intellectual correlates of attention: a study of four sixth-grade classrooms. Journal of Educational Psychology, 59, 320–324. http://dx.doi.org/10.1037/h0026223
47. Liu, M., Horton, L., Olmanson, J., & Toprac, P. (2011). A study of learning and motivation in a new media enriched environment for middle school science. Educational Technology Research and Development, 59, 249–265. http://dx.doi.org/10.1007/s11423-011-9192-7
48. Magnussen, R., & Misfeldt, M. (2004). Player transformation of educational multiplayer games. Proceedings of Other Players, Copenhagan, Denmark.
49. Martin, A. J. (2008). Enhancing student motivation and engagement: The effects of a multidimensional intervention. Contemporary Educational Psychology, 33, 239–269. http://dx.doi.org/10.1016/j.cedpsych.2006.11.003
50. Midgley, C., Maehr, M. L., Hicks, L., Roeser, R., Urdan, T., Anderman, E., & Middleton, M. (1997). Patterns of adaptive learning survey (PALS). Ann Arbor: University of Michigan.
51. Mierswa, I., Wurst, M., Klinkenberg, R., Scholz, M., & Euler, T. (2006, August). Yale: Rapid prototyping for complex data mining tasks. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 935–940). New York, NY: ACM. http://dx.doi.org/10.1145/1150402.1150531
52. Minner, D. D., Levy, A. J., & Century, J. (2010). Inquiry‐based science instruction—What is it and does it matter? Results from a research synthesis years 1984 to 2002. Journal of Research in Science Teaching, 47, 474–496. http://dx.doi.org/10.1002/tea.20347
53. Mitchell, T. (1997). Machine learning. New York, NY: McGraw Hill.
54. National Research Council. (2012). A framework for K-12 science education: Practices, crosscutting concepts, and core ideas. Washington, DC: National Academies Press.
55. Nelson-LeGall, S. (1985). Help-Seeking Behavior in Learning. Review of Research in Education, 12, 55–90.
56. Ocumpaugh, J., Baker, R. S. J. d., & Rodrigo, M. M. T. (2012). Baker-rodrigo observation method protocol (BROMP) 1.0. Training manual version 1.0 (Tech. Rep.). New York, NY: EdLab.
57. Olitsky, S. (2007) Promoting student engagement in science: Interaction rituals and the pursuit of a community of practice. Journal of Research in Science Teaching, 44, 33–56. http://dx.doi.org/10.1002/tea.20128
58. Papastergiou, M. (2009). Digital game-based learning in high school computer science education: Impact on educational effectiveness and student motivation. Computers & Education, 52, 1–12. http://dx.doi.org/10.1016/j.compedu.2008.06.004
59. Pardos, Z. A., Baker, R. S. J. d., San Pedro, M. O. C. Z., Gowda, S. M., & Gowda, S. M. (2013). Affective states and state tests: Investigating how affect throughout the school year predicts end of year learning outcomes. In D. Suthers, K. Verbert, E. Duval, & X. Ochoa (Eds.), Proceedings of the Third International Conference on Learning Analytics and Knowledge (pp. 117–124). New York, NY: ACM.
60. Pekrun, R. (2006). The control-value theory of achievement emotions: Assumptions, corollaries, and implications for educational research and practice. Educational Psychology Review, 18, 315–341. http://dx.doi.org/10.1007/s10648-006-9029-9
61. Pintrich, P. R., Smith, D. A. F., García, T., & McKeachie, W. J. (1991). A manual for the use of the Motivated Strategies for Learning Questionnaire (MSLQ). Ann Arbor: University of Michigan, National Center for Research to Improve Postsecondary Teaching and Learning.
62. Planalp, S. (1996). Varieties of cues to emotion in naturally occurring situations. Cognition & Emotion, 10, 137–154. http://dx.doi.org/10.1080/026999396380303
63. Porayska-Pomsta, K., Mavrikis, M., D'Mello, S., Conati, C., & Baker, R. S. (2013). Knowledge elicitation methods for affect modelling in education. International Journal of Artificial Intelligence in Education, 22, 107–140.
64. Reschly, A., & Christenson, S. L. (2006). Promoting school completion. In G. Bear & K. Minke (Eds.), Children's needs III: Understanding and addressing the developmental needs of children. Bethesda, MD: National Association of School Psychologists.
65. Reyes, L., & Fennema, E. (1981). Classroom processes observer manual. Buffalo, NY: U.S. Department of Education, National Center for Educational Statistics. Retrieved from ERIC database. (ED224793)
66. Rosenthal, R. & Rosnow, R. L. (1991). Essentials of behavioral research: Methods and data analysis (2nd ed.). New York,NY: McGraw-Hill.
67. Ryu, S., & Lombardi, D. (this issue). Coding classroom interactions for collective and individual engagement. Educational Psychologist, 50.
68. Sabourin, J., Rowe, J. P., Mott, B. W., & Lester, J. C. (2011). When off-task is on-task: The affective role of off-task behavior in narrative-centered learning environments. In G. Biswas, S. Bull, J. Kay, & A. Mitrovic (Eds.), Artificial intelligence in education (vol. 6738, pp. 534–536).
69. Sabourin, J. L., Rowe, J. P., Mott, B. W., & Lester, J. C. (2013). Considering alternate futures to classify off-task behavior as emotion self-regulation: A supervised learning approach. Journal of Educational Data Mining, 5, 9–38.
70. San Pedro, M. O. Z., Baker, R. S., Bowers, A. J., & Heffernan, N. T. (2013). Predicting college enrollment from student interaction with an intelligent tutoring system in middle school. Proceedings of the 6th International Conference on Educational Data Mining (pp. 177–184).
71. San Pedro, M. O. Z., Baker, R. S. J. D., & Mercedes, M. M. T. (2014). Carelessness and Affect in an Intelligent Tutoring System for Mathematics. International Journal of Artificial Intelligence in Education, 24, 189–210. http://dx.doi.org/10.1007/s40593-014-0015-y
72. Sao Pedro, M. A. (2013). Real-time assessment, prediction, and scaffolding of middle school students' data collection skills within physical science simulations (Unpublished doctoral dissertation). Worcester Polytechnic Institute, Worcester, MA.
73. Sao Pedro, M. A., Baker, R. S. J. d., Gobert, J., Montalvo, O., & Nakama, A. (2013). Leveraging machine-learned detectors of systematic inquiry behavior to estimate and predict transfer of inquiry skill. User Modeling and User-Adapted Interaction, 23(1), 1–39. http://dx.doi.org/10.1007/s11257-011-9101-0
74. Shea, P., & Bidjerano, T. (2010). Learning presence: Towards a theory of self-efficacy, self-regulation, and the development of a communities of inquiry in online and blended learning environments. Computers & Education, 55, 1721–1731. http://dx.doi.org/10.1016/j.compedu.2010.07.017
75. Shernoff, D., Csikszentmihalyi, M., Schneider, B., & Steele Shernoff, E. (2003). Student engagement in high school classrooms from the perspective of flow theory. School Psychology Quarterly, 18, 158–176. http://dx.doi.org/10.1521/scpq.18.2.158.21860
76. Sinatra, G. M., Heddy, B. C., Lombardi, D. (this issue). The challenges of defining and measuring student engagement in science. Educational Psychologist, 50.
77. Skinner, E. A., Marchand, G., Furrer, C., & Kindermann, T. (2008). Engagement and disaffection in the classroom: Part of a larger motivational dynamic. Journal of Educational Psychology, 100, 765–781. http://dx.doi.org/10.1037/a0012840
78. Skinner, E. A., & Pitzer, J. R. (2012). Developmental dynamics of student engagement, coping, and everyday resilience. In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp. 21–44). New York, NY: Springer. http://dx.doi.org/10.1007/978-1-4614-2018-7_2
79. Tobin, T. J., & Sugai, G. M. (1999). Using sixth-grade school records to predict school violence, chronic discipline problems, and high school outcomes. Journal of Emotional and Behavioral Disorders, 7, 40–53. http://dx.doi.org/10.1177/106342669900700105
80. Tytler, R., & Osborne, J. (2012). Student attitudes and aspirations towards science. In B. J. Fraser, K. Tobin, & C. J. McRobbie (Eds.), Second international handbook of science education (vol. 24, pp. 597–625). Dordrecht, The Netherlands: Springer. http://dx.doi.org/10.1007/978-1-4020-9041-7_41
81. Wigfield, A., Guthrie, J. T., Perencevich, K. C., Taboada, A., Klauda, S. L., McRae, A., & Barbosa, P. (2008). Role of reading engagement in mediating effects of reading comprehension instruction on reading outcomes. Psychology in the Schools, 45, 432–445. http://dx.doi.org/10.1002/pits.20307
82. Witten, I. H., & Frank, E. (2005). Data mining: Practical machine learning tools and techniques (2nd ed.). Series: Morgan Kaufmann Series in Data Management Systems. San Francisco, CA: Morgan Kaufmann.
83. Wixon, M. (2013). Detecting students who are conducting inquiry Without Thinking Fastidiously (WTF) in the Context of Microworld Learning Environments (Unpublished master's thesis). Worcester Polytechnic Institute, Worcester, MA.
84. Wixon, M., Baker, R. S., Gobert, J. D., Ocumpaugh, J., & Bachmann, M. (2012). WTF? detecting students who are conducting inquiry without thinking fastidiously. In User Modeling, Adaptation, and Personalization (pp. 286–296). Berlin, Germany: Springer. http://dx.doi.org/10.1007/978-3-642-31454-4_24