Advanced, Analytic, Automated (AAA) Measurement of Engagement During Learning

Educational Psychologist

Volumn 52, Issue 2, 2017, Pages 104-123

  D'Mello, Sidney ^a   Dieterle, Ed ^b   Duckworth, Angela ^c  

^a UNIVERSITY OF NOTRE DAME   (United States)

^b Summit Consulting LLC   (United States)

^c UNIVERSITY OF PENNSYLVANIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 85013498648     PISSN: 00461520     EISSN: None     Source Type: Journal    
DOI: 10.1080/00461520.2017.1281747     Document Type: Article

References (144)

1. Afzal, S., & Robinson, P., (2011). Natural affect data:Collection and annotation. In R., Calvo & S., D'Mello (Eds.), New perspectives on affect and learning technologies (pp. 44–70). New York, NY:Springer.
2. Arroyo, I., Beal, C. R., Murray, T., Walles, R., & Woolf, B. P., (2004). Web-based intelligent multimedia tutoring for high stakes achievement tests. In J. C., Lester, R. M., Vicari, & F., Paraguaçu (Eds.), Proceedings of the 7th International Conference on Intelligent Tutoring Systems (ITS 2004) (pp. 468–477). Berlin, Germany:Springer.
3. Arroyo, I., Woolf, B., Cooper, D., Burleson, W., Muldner, K., & Christopherson, R., (2009). Emotion sensors go to school. In V., Dimitrova, R., Mizoguchi, B., Du Boulay, & A., Graesser (Eds.), Proceedings of the 14th International Conference on Artificial Intelligence in Education (pp. 17–24). Amsterdam, The Netherlands:IOS Press.
4. Azevedo, R., (2015). Defining and measuring engagement and learning in science:Conceptual, theoretical, methodological, and analytical issues. Educational Psychologist, 50(1), 84–94.
5. Baker, R., & Ocumpaugh, J., (2015). Interaction-based affect detection in educational software. In R., Calvo, S., D'Mello, J., Gratch, & A., Kappas (Eds.), The Oxford handbook of affective computing (pp. 233–245). New York, NY:Oxford University Press.
6. Baker, R., & Rossi, L. M., (2013). Assessing the disengaged behaviors of learners. In R., Sottilare, A., Graesser, X., Hu, & H., Holden (Eds.), Design recommendations for intelligent tutoring systems (pp. 155–165). Orlando, FL:Army Research Laboratory.
7. Baker, R. S., & Ocumpaugh, J., (2014). Cost-effective, actionable engagement detection at scale. In Proceedings of the 7th International Conference on Educational Data Mining (pp. 345–346). Retrieved from http://educationaldatamining.org/EDM2014/uploads/procs2014/posters/21_EDM-2014-Poster.pdf
8. Baker, R. S. J., Kalka, J., Aleven, V., Rossi, L., Gowda, S. M., Wagner, A. Z., … Ocumpaugh, J., (2012). Towards sensor-free affect detection in cognitive tutor algebra. In K., Yacef, O., Zaïane, H., Hershkovitz, M., Yudelson, & J., Stamper (Eds.), Proceedings of the 5th International Conference on Educational Data Mining (pp. 126–133). Retrieved from http://www.educationaldatamining.org/conferences/index.php/EDM/2012/paper/view/967/933
9. Bandura, A., (1986). Social foundations of thought and action:A social cognitive theory. Englewood Cliffs, NJ:Prentice-Hall.
10. Bandura, A., (1997). Self-efficacy:The exercise of control. New York, NY:Freeman.
11. Barrett, L., (2006). Are emotions natural kinds? Perspectives on Psychological Science, 1, 28–58.
12. Barsalou, L. W., (2008). Grounded cognition. Annual Review of Psychology, 59(1), 617–645.
13. Bartlett, M. S., Littlewort, G. C., Frank, M. G., & Lee, K., (2014). Automatic decoding of facial movements reveals deceptive pain expressions. Current Biology, 24, 738–743.
14. Berka, C., Levendowski, D. J., Lumicao, M. N., Yau, A., Davis, G., Zivkovic, V. T., & Craven, P. L., (2007). EEG correlates of task engagement and mental workload in vigilance, learning, and memory tasks. Aviation, space, and environmental medicine, 78(Suppl. 1), B231–B244.
15. Bernardin, H. J., & Buckley, M. R., (1981). Strategies in rater training. Academy of Management Review, 6, 205–212.
16. Bixler, R., & D'Mello, S., (2013). Detecting engagement and boredom during writing with keystroke analysis, task appraisals, and stable traits. In Proceedings of the 2013 International Conference on Intelligent User Interfaces (IUI 2013) (pp. 225–234). New York, NY:ACM.
17. Bixler, R., & D'Mello, S. K., (2016). Automatic gaze-based user-independent detection of mind wandering during computerized reading. User Modeling & User-Adapted Interaction, 26(1), 33–68.
18. Blanchard, N., Bixler, R., Joyce, T., & D'Mello, S. K., (2014). Automated physiological-based detection of mind wandering during learning. In S., Trausan-Matu, K., Boyer, M., Crosby, & K., Panourgia (Eds.), Proceedings of the 12th international conference on intelligent tutoring systems (pp. 55–60). Berlin:Springer.
19. Bosch, N., Chen, H., Baker, R., Shute, V., & D'Mello, S. K., (2015). Accuracy vs. availability heuristic in multimodal affect detection in the wild. Proceedings of the 17th ACM international conference on multimodal interaction (ICMI 2015). New York, NY:ACM.
20. Bosch, N., D'Mello, S., Baker, R., Ocumpaugh, J., & Shute, V., (2016). Using video to automatically detect learner affect in computer-enabled classrooms. ACM Transactions on Interactive Intelligent Systems, 6(2), 17.1–17.31.
21. Bulger, M. E., Mayer, R. E., Almeroth, K. C., & Blau, S. D., (2008). Measuring learner engagement in computer-equipped college classrooms. Journal of Educational Multimedia and Hypermedia, 17, 129–143.
22. Calvo, R., D'Mello, S. K., Gratch, J., & Kappas, A., (Eds.). (2015). The oxford handbook of affective computing. New York, NY:Oxford University Press.
23. Camras, L., & Shutter, J., (2010). Emotional facial expressions in infancy. Emotion Review, 2, 120–129.
24. Camras, L. A., & Witherington, D. C., (2005). Dynamical systems approaches to emotional development. Developmental Review, 25, 328–350. http://dx.doi.org/10.1016/j.dr.2005.10.002
25. Chi, M., & Wylie, R., (2014). The ICAP framework:Linking cognitive engagement to active learning outcomes. Educational Psychologist, 49, 219–243.
26. Christenson, S. L., Reschly, A. L., & Wylie, C., (2012). Handbook of research on student engagement. New York, NY:Springer.
27. Coan, J. A., (2010). Emergent ghosts of the emotion machine. Emotion Review, 2, 274–285.
28. Cohen, J., (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37–46.
29. Conati, C., (2002). Probabilistic assessment of user's emotions in educational games. Applied Artificial Intelligence, 16, 555–575. http://dx.doi.org/10.1080/08839510290030390
30. Cortes, C., & Vapnik, V., (1995). Support-vector networks. Machine Learning, 20, 273–297.
31. Csikszentmihalyi, M., & Larson, R., (1987). Validity and reliability of the experience-sampling method. The Journal of Nervous and Mental Disease, 175, 526–536.
32. D'Mello, S., Craig, S., Witherspoon, A., McDaniel, B., & Graesser, A., (2008). Automatic detection of learner's affect from conversational cues. User Modeling and User-Adapted Interaction, 18(1–2), 45–80.
33. D'Mello, S., Dale, R., & Graesser, A., (2012). Disequilibrium in the mind, disharmony in the body. Cognition & Emotion, 26, 362–374. http://dx.doi.org/10.1080/02699931.2011.613668
34. D'Mello, S., & Graesser, A., (2011). The half-life of cognitive-affective states during complex learning. Cognition & Emotion, 25, 1299–1308.
35. D'Mello, S., & Graesser, A., (2012). Dynamics of affective states during complex learning. Learning and Instruction, 22, 145–157. http://dx.doi.org/10.1016/j.learninstruc.2011.10.001
36. D'Mello, S. K., (2016). On the influence of an iterative affect annotation approach on inter-observer and self-observer reliability. IEEE Transactions on Affective Computing, 7, 136–149.
37. D'Mello, S. K., Dowell, N., & Graesser, A. C., (2013). Unimodal and multimodal human perception of naturalistic non-basic affective states during Human-Computer interactions. IEEE Transactions on Affective Computing, 4, 452–465.
38. D'Mello, S. K., & Graesser, A. C., (2014). Confusion. In R., Pekrun & L., Linnenbrink-Garcia (Eds.), International handbook of emotions in education (pp. 289–310). New York, NY:Routledge.
39. D'Mello, S. K., Kopp, K., Bixler, R., & Bosch, N., (2016). Attending to attention:Detecting and combating mind wandering during computerized reading. In Extended abstracts of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI 2016) (pp. 1661–1669). New York, NY:ACM.
40. D'Mello, S. K., & Kory, J., (2015). A review and meta-analysis of multimodal affect detection systems. ACM Computing Surveys, 47(3), 43:41–43:46.
41. D'Mello, S. K., Taylor, R., Davidson, K., & Graesser, A., (2008). Self versus teacher judgments of learner emotions during a tutoring session with AutoTutor. In B., Woolf, E., Aimeur, R., Nkambou, & S., Lajoie (Eds.), Proceedings of the 9th International Conference on Intelligent Tutoring Systems (pp. 9–18). Berlin, Heidelberg:Springer-Verlag.
42. Darwin, C., (1872). The expression of the emotions in man and animals. London, UK:John Murray.
43. Daschmann, E. C., Goetz, T., & Stupnisky, R. H., (2011). Testing the predictors of boredom at school:Development and validation of the precursors to boredom scales. British Journal of Educational Psychology, 81, 421–440.
44. De Falco, J., Baker, R., & D'Mello, S., (2014). Addressing behavioral disengagement in online learning. In R., Sottilare, A., Graesser, X., Hu, & B., Goldberg (Eds.), Design recommendations for adaptive intelligent tutoring systems:Adaptive instructional strategies (Vol. 2, pp. 49–56). Orlando, FL:Army Research Laboratory.
45. Deci, E. L., & Ryan, R. M., (1985). Intrinsic motivation and self-determination in human behavior. New York, NY:Plenum.
46. Deubel, H., & Schneider, W. X., (1996). Saccade target selection and object recognition:Evidence for a common attentional mechanism. Vision research, 36, 1827–1837.
47. deVega, M., Glenberg, A., & Graesser, A., (Eds.). (2008). Symbols, embodiment, and meaning. Oxford:Oxford University Press.
48. Domingos, P., (2012). A few useful things to know about machine learning. Communications of the ACM, 55(10), 78–87.
49. Drummond, J., & Litman, D., (2010). In the zone:Towards Detecting student zoning out using supervised machine learning. In V., Aleven, J., Kay, & J., Mostow (Eds.), Intelligent tutoring systems (Vol. 6095, pp. 306–308). Berlin, Germany:Springer-Verlag.
50. Duckworth, A. L., & Yeager, D. S., (2015). Measurement matters:Assessing personal qualities other than cognitive ability for educational purposes. Educational Researcher, 44, 237–251.
51. Eastwood, J. D., Frischen, A., Fenske, M. J., & Smilek, D., (2012). The unengaged mind:Defining boredom in terms of attention. Perspectives on Psychological Science, 7, 482–495.
52. Eccles, J., & Wang, M. T., (2012). Part I commentary:So what is student engagement anyway? In S., Christenson, A., Reschly, & C., Wylie (Eds.), Handbook of research on student engagement (pp. 133–145). New York, NY:Springer.
53. Eccles, J. S., & Wigfield, A., (2002). Motivational beliefs, values, and goals. Annual Review of Psychology, 53, 109–132.
54. Ekman, P., (1984). Expression and the nature of emotion. In K., Scherer & P., Ekman (Eds.), Approaches to emotion (pp. 319–344). Hillsdale, NJ:Erlbaum.
55. Ekman, P., (1992). An argument for basic emotions. Cognition & Emotion, 6, 169–200.
56. Ekman, P., (2002, November). Darwin, deception, and facial expression. Paper presented at the conference on Emotions Inside Out, 130 Years after Darwin's The Expression of the Emotions in Man and Animals, New York, NY.
57. Ekman, P., & Friesen, W., (1978). The Facial Action Coding System:A technique for the measurement of facial movement. Palo Alto, CA:Consulting Psychologists Press.
58. Elfenbein, H., & Ambady, N., (2002). On the universality and cultural specificity of emotion recognition:A meta-analysis. Psychological Bulletin, 128, 203–235. http://dx.doi.org/10.1037//0033-2909.128.2.203
59. Eyben, F., Wöllmer, M., & Schuller, B., (2010). Opensmile:The Munich versatile and fast open-source audio feature extractor. Proceedings of the International Conference on Multimedia (pp. 1459–1462). New York, NY:ACM.
60. Faber, M., Bixler, R., & D'Mello, S. K., (in press). An automated behavioral measure of mind wandering rates during computerized reading. Behavior Research Methods.
61. FACET. (2014). Facial Expression Recognition Software. Boston, MA:Emotient. Retrieved from https://imotions.com/emotient/
62. Fiedler, K., & Beier, S., (2014). Affect and cognitive processes in educational contexts. In R., Pekrun & L., Linnenbrink-Garcia (Eds.), International handbook of emotions in education (pp. 36–56). New York, NY:Routledge.
63. Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H., (2004). School engagement:Potential of the concept, state of the evidence. Review of Educational Research, 74(1), 59–109.
64. Fredricks, J. A., & McColskey, W., (2012). The measurement of student engagement:A comparative analysis of various methods and student self-report instruments. In S., Christenson, A., Reschly, & C., Wylie (Eds.), Handbook of research on student engagement (pp. 763–782). New York, NY:Springer.
65. Gobert, J. D., Baker, R. S., & Wixon, M. B., (2015). Operationalizing and detecting disengagement within online science microworlds. Educational Psychologist, 50(1), 43–57.
66. Graesser, A., Chipman, P., Haynes, B., & Olney, A., (2005). AutoTutor:An intelligent tutoring system with mixed-initiative dialogue. IEEE Transactions on Education, 48, 612–618. http://dx.doi.org/10.1109/TE.2005.856149
67. Graesser, A., McDaniel, B., Chipman, P., Witherspoon, A., D'Mello, S. K., & Gholson, B., (2006). Detection of emotions during learning with AutoTutor. In R., Sun & N., Miyake (Eds.), Proceedings of the 28th Annual Conference of the Cognitive Science Society (pp. 285–290). Austin, TX:Cognitive Science Society.
68. Greene, B. A., (2015). Measuring cognitive engagement with self-report scales:Reflections from over 20 years of research. Educational Psychologist, 50(1), 1–17.
69. Hanley, J. A., & McNeil, B. J., (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143(1), 29–36.
70. Hayes, A. F., & Krippendorff, K., (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1(1), 77–89.
71. Heine, S. J., Lehman, D. R., Peng, K., & Greenholtz, J., (2002). What's wrong with cross-cultural comparisons of subjective Likert scales?:The reference-group effect. Journal of Personality and Social Psychology, 82, 903–918.
72. Henrie, C. R., Halverson, L. R., & Graham, C. R., (2015). Measuring student engagement in technology-mediated learning:A review. Computers & Education, 90, 36–53.
73. Hidi, S., & Renninger, K. A., (2006). The four-phase model of interest development. Educational Psychologist, 41, 111–127.
74. Hoffman, J. E., & Subramaniam, B., (1995). The role of visual attention in saccadic eye movements. Attention, Perception, & Psychophysics, 57, 787–795.
75. Hutt, S., Mills, C., Bosch, N., Krasich, K., Brockmole, J., D'Mello, S K. (2017). Out of the fr-“eye”-ing pan:Towards gaze-based models of attention with learning technology in the classroom. Manuscript submitted for publication.
76. Kahneman, D., Krueger, A. B., Schkade, D. A., Schwarz, N., & Stone, A. A., (2004). A survey method for characterizing daily life experience:The day reconstruction method. Science, 306(5702), 1776–1780.
77. Kapoor, A., & Picard, R., (2005). Multimodal affect recognition in learning environments. In Proceedings of the 13th annual ACM International Conference on Multimedia (pp. 677–682). New York, NY:ACM.
78. Kappas, A., (2013). Social regulation of emotion:messy layers. Frontiers in Psychology, 4(51). http://dx.doi.org/10.3389/fpsyg.2013.00051
79. Keltner, D., & Ekman, P., (2000). Facial expression of emotion. In R., Lewis & J. M., Haviland-Jones (Eds.), Handbook of emotions (Vol. 2nd ed, pp. 236–264). New York, NY:Guilford.
80. Kory, J., D'Mello, S. K., & Olney, A., (2015). Motion tracker:Camera-based monitoring of bodily movements using motion silhouettes. PloS ONE, 10(6). doi:10.1371/journal.pone.0130293
81. Krosnick, J. A., (1999). Survey research. Annual Review of Psychology, 50(1), 537–567.
82. Larsen, J., Berntson, G., Poehlmann, K., Ito, T., & Cacioppo, J., (2008). The psychophysiology of emotion. In M., Lewis, J., Haviland-Jones, & L., Barrett (Eds.), Handbook of emotions (3rd ed., pp. 180–195). New York, NY:Guilford.
83. Lehman, B., Matthews, M., D'Mello, S. K., & Person, N., (2008). What are you feeling? Investigating student affective states during expert human tutoring sessions. In B., Woolf, E., Aimeur, R., Nkambou, & S., Lajoie (Eds.), Proceedings of the 9th International Conference on Intelligent Tutoring Systems (pp. 50–59). Berlin, Germany:Springer.
84. Lehr, C. A., Sinclair, M. F., & Christenson, S. L., (2004). Addressing student engagement and truancy prevention during the elementary school years:A replication study of the check & connect model. Journal of Education for Students Placed at Risk, 9, 279–301.
85. Lewis, M. D., (2005). Bridging emotion theory and neurobiology through dynamic systems modeling. Behavioral and Brain Sciences, 28, 169–245.
86. Linnenbrink-Garcia, L., & Pekrun, R., (2011). Students' emotions and academic engagement:Introduction to the special issue. Contemporary Educational Psychology, 36(1), 1–3.
87. Littlewort, G., Whitehill, J., Wu, T., Fasel, I., Frank, M., Movellan, J., & Bartlett, M., (2011). The computer expression recognition toolbox (CERT). In Proceedings of the IEEE International Conference on Automatic Face & Gesture Recognition (pp. 298–305). Washington, DC:IEEE.
88. Loveless, T., (2015). How well are American students learning? Washington, DC:Brookings Institution.
89. Mandler, G., (1990). Interruption (discrepancy) theory:Review and extensions. In S., Fisher & C. L., Cooper (Eds.), On the move:The psychology of change and transition (pp. 13–32). Chichester, UK:Wiley.
90. Marshall, S. P., (2005). Assessing cognitive engagement and cognitive state from eye metrics. In D. D., Schmorrow (Ed.), Foundations of augmented cognition (pp. 312–320). Mahwah, NJ:Erlbaum.
91. Martin, J. P., (2010). Lower Merion district's laptop saga ends with $610,000 settlement, The Philadelphia Inquirer. Retrieved from http://articles.philly.com/
92. Matsumoto, D., Keltner, D., Shiota, M. N., O'Sullivan, M., & Frank, M., (2008). Facial expressions of emotion. In M., Lewis, J., Haviland-Jones, & L., Barrett (Eds.), Handbook of emotions (Vol. 3, pp. 211–234). New York, NY:Guilford Press.
93. Mehl, M. R., Pennebaker, J. W., Crow, D. M., Dabbs, J., & Price, J. H., (2001). The Electronically Activated Recorder (EAR):A device for sampling naturalistic daily activities and conversations. Behavior Research Methods, Instruments, & Computers, 33, 517–523.
94. Mehu, M., & Scherer, K., (2012). A psycho-ethological approach to social signal processing. Cognitive Processing, 13, 397–414.
95. Mesquita, B., & Boiger, M., (2014). Emotions in context:A sociodynamic model of emotions. Emotion Review, 6, 298–302.
96. Miller, B. W., (2015). Using reading times and eye-movements to measure cognitive engagement. Educational Psychologist, 50(1), 31–42.
97. Miller, W. L., Petsche, K., Baker, R. S., Labrum, M. J., & Wagner, A. Z., (2014). Boredom across activities, and across the year, within reasoning mind. Paper presented at the workshop on data mining for educational assessment and feedback (ASSESS 2014), New York, NY.
98. Monkaresi, H., Bosch, N., Calvo, R. A., & D'Mello, S. K., (in press). Automated detection of engagement using video-based estimation of facial expressions and heart rate. IEEE Transactions on Affective Computing.
99. Mota, S., & Picard, R., (2003, June). Automated posture analysis for detecting learner's interest level. Paper presented at the Computer Vision and Pattern Recognition Workshop, Madison, WI.
100. National Student Clearinghouse. (2016). http://www.studentclearinghouse.org/
101. Niedenthal, P. M., (2007). Embodying emotion. Science, 316(5827), 1002–1005. http://dx.doi.org/10.1126/science.1136930
102. Nystrand, M., & Gamoran, A., (1991). Instructional discourse, student engagement, and literature achievement. Research in the Teaching of English, 25, 261–290.
103. Ocumpaugh, J., Baker, R., Gowda, S., Heffernan, N., & Heffernan, C., (2014). Population validity for educational data mining:A case study in affect detection. British Journal of Educational Psychology, 45, 487–501.
104. Ocumpaugh, J., Baker, R. S., & Rodrigo, M. M. T., (2012). Baker-Rodrigo Observation Method Protocol (BROMP) 1.0. Training Manual version 1.0. New York, NY:Teachers College, Columbia University.
105. Ocumpaugh, J., Baker, R. S., & Rodrigo, M. M. T., (2015). Baker Rodrigo Ocumpaugh Monitoring Protocol (BROMP) 2.0 Technical and Training Manual. New York, NY and Manila, Philippines:Teachers College, Columbia University and Ateneo Laboratory for the Learning Sciences.
106. OECD. (2012). PISA 2012 Results:Ready to Learn:Students' Engagement, Drive and Self-Beliefs (Vol. III). OECD Publishing.
107. Olney, A., Risko, E. F., D'Mello, S. K., & Graesser, A. C., (2015). Attention in educational contexts:The role of the learning task in guiding attention. In J., Fawcett, E. F., Risko, & A., Kingstone (Eds.), The handbook of attention (pp. 623–642). Cambridge, MA:MIT Press.
108. Pantic, M., & Patras, I., (2006). Dynamics of facial expression:Recognition of facial actions and their temporal segments from face profile image sequences. IEEE Transactions on Systems, Man, and Cybernetics, Part B., 36, 433–449. http://dx.doi.org/10.1109/tsmcb.2005.859075
109. Pardos, Z., Baker, R. S. J. D., San Pedro, M. O. C. Z., & Gowda, S. M., (2013). Affective states and state tests:Investigating how affect throughout the school year predicts end of year learning outcomes. In D., Suthers, K., Verbert, E., Duval, & X., Ochoa (Eds.), Proceedings of the 3rd international conference on learning analytics and knowledge (pp. 117–124). New York, NY:ACM.
110. Pekrun, R., Goetz, T., Daniels, L., Stupnisky, R. H., & Perry, R., (2010). Boredom in achievement settings:Exploring control–value antecedents and performance outcomes of a neglected emotion. Journal of Educational Psychology, 102, 531–549. http://dx.doi.org/10.1037/a0019243
111. Pekrun, R., & Linnenbrink-Garcia, L., (2012). Academic emotions and student engagement. In S., Christenson, A., Reschly, & C., Wylie (Eds.), Handbook of research on student engagement (pp. 259–282). New York, NY:Springer.
112. Pianta, R. C., Hamre, B. K., & Allen, J. P., (2012). Teacher-student relationships and engagement:Conceptualizing, measuring, and improving the capacity of classroom interactions. In S., Christenson, A., Reschly, & C., Wylie (Eds.), Handbook of research on student engagement (pp. 365–386). New York, NY:Springer.
113. Podsakoff, P. M., MacKenzie, S. B., Lee, J. Y., & Podsakoff, N. P., (2003). Common method biases in behavioral research:A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88, 879–903.
114. Poh, M. Z., McDuff, D. J., & Picard, R. W., (2011). Advancements in noncontact, multiparameter physiological measurements using a webcam. IEEE Transactions on Biomedical Engineering, 58(1), 7–11.
115. Porayska-Pomsta, K., Mavrikis, M., D'Mello, S. K., Conati, C., & Baker, R., (2013). Knowledge elicitation methods for affect modelling in education. International Journal of Artificial Intelligence in Education, 22, 107–140.
116. Rayner, K., (1998). Eye movements in reading and information processing:20 years of research. Psychological Bulletin, 124, 372–422.
117. Razzaq, L., Feng, M., Nuzzo-Jones, G., Heffernan, N. T., Koedinger, K. R., Junker, B., & Choksey, S., (2005). The Assistment project:Blending assessment and assisting. In C., Loi & G., McCalla (Eds.), Proceedings of the 12th International Conference on Artificial Intelligence in Education (pp. 555–562). Amsterdam, The Netherlands:IOS Press.
118. Reeve, J., & Tseng, C. M., (2011). Agency as a fourth aspect of students' engagement during learning activities. Contemporary Educational Psychology, 36, 257–267.
119. Renninger, K. A., & Bachrach, J. E., (2015). Studying triggers for interest and engagement using observational methods. Educational Psychologist, 50(1), 58–69.
120. Reschly, A., & Christenson, S., (2012). Jingle, jangle, and conceptual haziness:Evolution and future directions of the engagement construct. In S., Christenson, A., Reschly, & C., Wylie (Eds.), Handbook of research on student engagement (pp. 3–19). Berlin, Germany:Springer.
121. Roseman, I. J., (2011). Emotional behaviors, emotivational goals, emotion strategies:Multiple levels of organization integrate variable and consistent responses. Emotion Review, 3, 434–443.
122. Ruch, W., (1995). Will the real relationship between facial expression and affective experience please stand up:The case of exhilaration. Cognition & Emotion, 9(1), 33–58.
123. Russell, J., (2003). Core affect and the psychological construction of emotion. Psychological Review, 110, 145–172.
124. Russell, J. A., Bachorowski, J. A., & Fernandez-Dols, J. M., (2003). Facial and vocal expressions of emotion. Annual Review of Psychology, 54, 329–349.
125. Ryan, R., & Deci, E., (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, 55(1), 68–78.
126. Ryu, S., & Lombardi, D., (2015). Coding classroom interactions for collective and individual engagement. Educational Psychologist, 50(1), 70–83.
127. Sabourin, J., Mott, B., & Lester, J., (2011). Modeling learner affect with theoretically grounded dynamic bayesian networks. In S., D'Mello, A., Graesser, B., Schuller, & J., Martin (Eds.), Proceedings of the Fourth International Conference on Affective Computing and Intelligent Interaction (pp. 286–295). Berlin, Germany:Springer-Verlag.
128. San Pedro, M., Baker, R. S., Bowers, A. J., & Heffernan, N. T., (2013). Predicting college enrollment from student interaction with an intelligent tutoring system in middle school. In S., D'Mello, R., Calvo, & A., Olney (Eds.), Proceedings of the 6th international conference on educational data mining (EDM 2013) (pp. 177–184). Retrieved from http://www.educationaldatamining.org/conferences/index.php/EDM/2013/paper/view/1035/1001
129. Schunk, D. H., & Pajares, F., (2005). Competence perceptions and academic functioning. In A. J., Elliot & C. S., Dweck (Eds.), Handbook of competence and motivation (pp. 141–163). New York, NY:Guilford.
130. Scully, C. G., Lee, J., Meyer, J., Gorbach, A. M., Granquist-Fraser, D., Mendelson, Y., & Chon, K. H., (2012). Physiological parameter monitoring from optical recordings with a mobile phone. IEEE Transactions on Biomedical Engineering, 59, 303–306.
131. Sewell, W., & Komogortsev, O., (2010). Real-time eye gaze tracking with an unmodified commodity webcam employing a neural network. In Proceedings of the 2012 ACM annual conference on human factors in computing systems (pp. 3739–3744). New York, NY:ACM.
132. Shibata, K., Watanabe, T., Sasaki, Y., & Kawato, M., (2011). Perceptual learning incepted by decoded fMRI neurofeedback without stimulus presentation. Science, 334(6061), 1413–1415.
133. Shute, V. J., Ventura, M., & Kim, Y. J., (2013). Assessment and learning of qualitative physics in Newton's playground. The Journal of Educational Research, 106, 423–430.
134. Sinatra, G. M., Heddy, B. C., & Lombardi, D., (2015). The challenges of defining and measuring student engagement in science. Educational Psychologist, 50(1), 1–13.
135. Skinner, E. A., & Belmont, M. J., (1993). Motivation in the classroom:Reciprocal effects of teacher behavior and student engagement across the school year. Journal of Educational Psychology, 85(4), 571–581.
136. Smallwood, J., McSpadden, M., & Schooler, J. W., (2008). When attention matters:The curious incident of the wandering mind. Memory & Cognition, 36, 1144–1150. http://dx.doi.org/10.3758/mc.36.6.1144
137. St. John, M., Kobus, D. A., Morrison, J. G., & Schmorrow, D., (2004). Overview of the DARPA augmented cognition technical integration experiment. International Journal of Human–Computer Interaction, 17, 131–149.
138. Stein, N., & Levine, L., (1991). Making sense out of emotion. In W., Kessen, A., Ortony, & F., Kraik (Eds.), Memories, thoughts, and emotions:Essays in honor of George Mandler (pp. 295–322). Hillsdale, NJ:Erlbaum.
139. Turner, J. C., & Meyer, D. K., (2000). Studying and understanding the instructional contexts of classrooms:Using our past to forge our future. Educational Psychologist, 35(2), 69–85.
140. Valstar, M., Mehu, M., Jiang, B., Pantic, M., & Scherer, K., (2012). Meta-analysis of the first facial expression recognition challenge. IEEE Transactions on Systems, Man, and Cybernetics, Part B:Cybernetics, 42, 966–979.
141. Volpe, R. J., DiPerna, J. C., Hintze, J. M., & Shapiro, E. S., (2005). Observing students in classroom settings:A review of seven coding schemes. School Psychology Review, 34, 454–474.
142. Wachowski Brothers. (1999). The matrix [Motion picture]. United States:Warner Brothers.
143. Wang, R., Chen, F., Chen, Z., Li, T., Harari, G., Tignor, S., & Campbell, A. T., (2014). StudentLife:assessing mental health, academic performance and behavioral trends of college students using smartphones. In J., Kientz, J., Scott, & J., Song (Eds.), Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing (ACM UbiComp ′14) (pp. 3–14). New York, NY:ACM.
144. Whitehill, J., Serpell, Z., Lin, Y. C., Foster, A., & Movellan, J., (2014). The faces of engagement:Automatic recognition of student engagement from facial expressions. IEEE Transactions on Affective Computing, 5(1), 86–98.