Assessing Whether Students Seek Constructive Criticism: The Design of an Automated Feedback System for a Graphic Design Task

International Journal of Artificial Intelligence in Education

Volumn 27, Issue 3, 2017, Pages 419-447

  Cutumisu, Maria ^a   Blair, Kristen P ^b   Chin, Doris B ^b   Schwartz, Daniel L ^b  

^a UNIVERSITY OF ALBERTA   (Canada)

^b STANFORD UNIVERSITY   (United States)

Author keywords

Assessment; Choice; Feedback; Game; Learning; Performance

Indexed keywords

DESIGN; FEEDBACK; STUDENTS;

ASSESSMENT; CHOICE; GAME; LEARNING; PERFORMANCE;

EDUCATION;

EID: 85025106648     PISSN: 15604292     EISSN: 15604306     Source Type: Journal    
DOI: 10.1007/s40593-016-0137-5     Document Type: Article

References (50)

1. Aleven, V., & Koedinger, K. R. (2000). Limitations of student control: Do students know when they need help? In International Conference on Intelligent Tutoring Systems (pp. 292–303). Berlin: Springer.
2. Anseel, F., Beatty, A. S., Shen, W., Lievens, F., & Sackett, P. R. (2015). How are we doing after 30 years? A meta-analytic review of the antecedents and outcomes of feedback-seeking behavior. Journal of Management, 41(1), 318–348.
3. Ashford, S. J., & Cummings, L. L. (1983). Feedback as an individual resource: personal strategies of creating information. Organizational Behavior and Human Performance, 32(3), 370–398.
4. Bates, D., Maechler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48.
5. Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education: Principles, Policy and Practice, 5(1), 7–74.
6. Black, P., & Wiliam, D. (2004). The formative purpose: assessment must first promote learning. Yearbook of the National Society for the Study of Education, Blackwell Publishing Ltd, 103(2), 20–50.
7. Black, P., & Wiliam, D. (2009). Developing the theory of formative assessment. Educational Assessment, Evaluation and Accountability, 21(1), 5–31.
8. Bransford, J. D., & Schwartz, D. L. (1999). Rethinking transfer: a simple proposal with multiple implications. Review of Research in Education, 24, 61–100.
9. Clark, R. C., & Mayer, R. E. (2016). E-learning and the science of instruction: Proven guidelines for consumers and designers of multimedia learning. Wiley.
10. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale: Erlbaum.
11. Conati, C., & Vanlehn, K. (2000). Toward computer-based support of meta-cognitive skills: a computational framework to coach self-explanation. International Journal of Artificial Intelligence in Education, 11, 389–415.
12. Conlin, L., Chin, D. B., Blair, K. P., Cutumisu, M., & Schwartz, D. L. (2015). Guardian Angels of Our Better Nature: Finding Evidence of the Benefits of Design Thinking. In Proc. of the 122nd American Society for Engineering Education (ASEE’15), ms. 10 pages, June 14–17, Seattle, WA, USA. Finalist in the Best of Design in Engineering Education (DEED) Division.
13. Corbett, A. T., & Anderson, J. R. (2001). Locus of feedback control in computer-based tutoring: Impact on learning rate, achievement and attitudes. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 245–252). New York: ACM Press.
14. Core Team, R. (2016). R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing. Retrieved from: https://www.R-project.org/.
15. Cutumisu, M., & Schwartz, D. L. (2014). Choosing negative feedback improves learning for students of all ages: A game-based assessment of seeking negative feedback and revising. In Proc. of the London International Conference in Education, London, UK, November 10–12, pp. 171–176.
16. Cutumisu, M., & Schwartz, D. L. (2016). Choosing versus receiving feedback: The impact of feedback valence on learning in an assessment game. In Proc. of the 9th International Conference on Educational Data Mining (EDM’16), June 29 - July 2. 341–346, Raleigh.
17. Cutumisu, M., Chin, D. B., & Schwartz, D. L. (2014). A game-based assessment of students’ choices to seek feedback and to revise. In Proc. of the 11th International Conference on Cognition and Exploratory Learning in Digital Age (CELDA’14), Porto, Portugal, October 25–27, pp. 17–24. Best Paper.
18. Cutumisu, M., Blair, K. P., Chin, D. B., & Schwartz, D. (2015). Posterlet: a game-based assessment of children’s choices to seek feedback and to revise. Journal of Learning Analytics, 2(1), 49–71.
19. Dunning, D. (1995). Trait importance and modifiability as factors influencing self-assessment and self-enhancement motives. Personality and Social Psychology Bulletin, 21, 1297–1306.
20. Dweck, C. S., & Leggett, E. L. (1988). A social-cognitive approach to motivation and personality. Psychological Review, 95(2), 256–273.
21. Ehrlinger, J., Mitchum, A. L., & Dweck, C. S. (2016). Understanding overconfidence: theories of intelligence, preferential attention, and distorted self-assessment. Journal of Experimental Social Psychology, 63, 94–100.
22. Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993). The role of deliberate practice in the acquisition of expert performance. Psychological Review, 100(3), 363–406.
23. Freitas, A. L., Salovey, P., & Liberman, N. (2001). Abstract and concrete self-evaluative goals. Journal of Personality and Social Psychology, 80(3), 410.
24. Hattie, J. (1999). Influences on student learning. New Zealand: Inaugural Lecture: Professor of Education, University of Auckland.
25. Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112.
26. Heck, R. H., Thomas, S., & Tabata, L. N. (2010). Multilevel and longitudinal modeling with IBM SPSS ebook. Routledge Academic.
27. Heck, R. H., Thomas, S., & Tabata, L. N. (2013). Multilevel modeling of categorical outcomes using IBM SPSS. Routledge Academic.
28. Heffernan, N., Heffernan, C., Dietz, K., Soffer, D., Pellegrino, J. W. Goldman, S.R., & Dailey, M. (2012). Improving Mathematical Learning Outcomes Through Automatic Reassessment and Relearning. AERA.
29. Iyengar, S. S., & Lepper, M. R. (1999). Rethinking the value of choice: a cultural perspective on intrinsic motivation. Journal of Personality and Social Psychology, 76(3), 349–366.
30. Klehe, U. C., & Anderson, N. (2007). Working hard and working smart: motivation and ability during typical and maximum performance. Journal of Applied Psychology, 92(4), 978–992.
31. Kluger, A. N., & DeNisi, A. (1996). The effects of feedback interventions on performance: a historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119(2), 254–284.
32. Kluger, A. N., & DeNisi, A. (1998). Feedback interventions: toward the understanding of a double-edged sword. Current Directions in Psychological Science, 7(3), 67–72.
33. Koedinger, K. R., Anderson, J. R., Hadley, W. H., & Mark, M. A. (1997). Intelligent tutoring goes to school in the big city. International Journal of Artificial Intelligence in Education, 8, 30–43.
34. Kulik, J. A., & Kulik, C.-L. C. (1987). Timing of feedback and verbal learning. Review of Educational Research, 58(1), 79–97.
35. , C. E., Bernstein, M. S., & Klemmer, S. R. (2015). PeerStudio: rapid peer feedback emphasizes revision and improves performance. In Proceedings of the Second (2015) ACM Conference on Learning Scale, 75–84. ACM.
36. Liberman, N., & Trope, Y. (1998). The role of feasibility and desirability considerations in near and distant future decisions: a test of temporal construal theory. Journal of Personality and Social Psychology, 75(1), 5–18.
37. Luminance Algorithm. (2016). http://www.w3.org/TR/2008/REC-WCAG20-20081211/#relativeluminancedef.
38. Mangels, J. A., Butterfield, B., Lamb, J., Good, C., & Dweck, C. S. (2006). Why do beliefs about intelligence influence learning success? A social cognitive neuroscience model. Social Cognitive and Affective Neuroscience, 1(2), 75–86.
39. Mitrovic, A., & Ohlsson, S. (2016). Implementing CBM: SQL-tutor after fifteen years. International Journal of Artificial Intelligence in Education, 26(1), 150–159.
40. Nussbaum, A. D., & Dweck, C. S. (2008). Defensiveness vs. remediation: self-theories and modes of self-esteem maintenance. Personality and Social Psychology Bulletin, 34, 127–134.
41. Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (Vol. 1). Sage.
42. Roll, I., Aleven, V., McLaren, B., & Koedinger, K.R. (2011). Improving students’ help-seeking skills using metacognitive feedback in an intelligent tutoring system. Learning and Instruction, 267–280.
43. Roll, I., Baker, R. S., Aleven, V., & Koedinger, K. R. (2014). On the benefits of seeking (and avoiding) help in online problem-solving environments. Journal of the Learning Sciences, 23, 537–560.
44. Schwartz, D. L., & Arena, D. (2013). Measuring what matters most: Choice-based assessments for the digital age. Cambridge: MIT Press.
45. Schwartz, D. L., Tsang, J. M., & Blair, K. P. (2016). The ABCs of How We Learn: 26 Scientifically Proven Approaches, How They Work, and When to Use Them. W. W. Norton & Company.
46. Thorndike, E. L. (1927). The law of effect. American Journal of Psychology, 39, 212–222.
47. Trope, Y., & Neter, E. (1994). Reconciling competing motives in self-evaluation: the role of self-control in feedback seeking. Journal of Personality and Social Psychology, 66, 646–657.
48. Vallacher, R. R., & Wegner, D. M. (1985). A theory of action identification. Hillsdale: Lawrence Erlbaum Associates.
49. Vallacher, R. R., & Wegner, D. M. (1987). What do people think they're doing? Action identification and human behavior. Psychological Review, 94(1), 3–15.
50. Zimmerman, B. J. (1990). Self-regulated learning and academic achievement: an overview. Educational Psychologist, 25(1), 3–17.