Metacognitive scaffolds improve self-judgments of accuracy in a medical intelligent tutoring system

Instructional Science

Volumn 42, Issue 2, 2014, Pages 159-181

  Feyzi Behnagh, Reza ^a   Azevedo, Roger ^a   Legowski, Elizabeth ^b   Reitmeyer, Kayse ^b   Tseytlin, Eugene ^b   Crowley, Rebecca S ^b  

^a MCGILL UNIVERSITY   (Canada)

^b UNIVERSITY OF PITTSBURGH SCHOOL OF MEDICINE   (United States)

Author keywords

Cognitive debiasing; Computer based education; Diagnostic reasoning; Intelligent tutoring systems; Metacognition; Metacognitive judgments

Indexed keywords

EID: 84875905060     PISSN: 00204277     EISSN: 15731952     Source Type: Journal    
DOI: 10.1007/s11251-013-9275-4     Document Type: Article

References (63)

1. Azevedo, R., & Hadwin, A. F. (2005). Scaffolding self-regulated learning and metacognition: Implications for the design of computer-based scaffolds. Instructional Science,33, 367–379.
2. Azevedo, R., & Lajoie, S. (1998). The cognitive basis for the design of a mammography interpretation tutor. International Journal of Artificial Intelligence in Education,9, 32–44.
3. Azevedo, R., Moos, D., Greene, J., Winters, F., & Cromley, J. (2008). Why is externally-facilitated regulated learning more effective than self-regulated learning with hypermedia? Educational Technology Research and Development,56(1), 45–72.
4. Azevedo, R., & Witherspoon, A. M. (2009). Self-regulated use of hypermedia. In A. Graesser, J. Dunlosky, & D. Hacker (Eds.), Handbook of metacognition in education (pp. 319–339). Mahwah, NJ: Erlbaum.
5. Berner, E. S., & Graber, M. L. (2008). Overconfidence as a cause of diagnostic error in medicine. The American Journal of Medicine,121(5A), S2–S23.
6. Bull, S. (2004). Supporting learning with open learner models. In Proceedings of 4th Hellenic conference with international participation: Information and communication technologies in education. Keynote. Athens, Greece.
7. Bull, S., & McKay, M. (2004). An open learner model for children and teachers: Inspecting knowledge level of individuals and peers. In R. Luckin, K. R. Koedinger, & J. E. Greer (Eds.), Intelligent tutoring systems (pp. 232–251). Berlin: Springer.
8. Chi, M. T. H., de Leeuw, N., Chiu, M. H., & LaVancher, C. (1994). Eliciting self-explanations improves understanding. Cognitive Science,18, 439–477.
9. Chi, M. T. H., Siler, S., & Jeong, H. (2004). Can tutors monitor students’ understanding accurately? Cognition and Instruction,22, 363–387.
10. Chi, M. T. H., Siler, S., Jeong, H., Yamauchi, T., & Hausmann, R. (2001). Learning from human tutoring. Cognitive Science,25, 471–534.
11. Clancey, W. (1987). Knowledge-based tutoring: The GUIDON program. Cambridge: MIT Press.
12. Corbett, A. T., Koedinger, K. R., & Hadley, W. H. (2002). Cognitive tutors: From the research classroom to all classrooms. In P. S. Goodman (Ed.), Technology enhanced learning: Opportunities for change (pp. 199–224). Mahwah, NJ: Erlbaum.
13. Croskerry, P., & Norman, G. (2008). Overconfidence in clinical decision making. The American Journal of Medicine,121(5A), S24–S29.
14. Crowley, R., & Gryzbicki, D. (2006). Intelligent medical training systems (guest editorial). Artificial Intelligence in Medicine,38(1), 1–4.
15. Crowley, R. S., Legowski, E., Medvedeva, O., Tseytlin, E., Roh, E., & Jukic, D. (2007). Evaluation of an intelligent tutoring system in pathology: Effects of external representation on performance gains, metacognition, and acceptance. Journal of the American Medical Informatics Association,14(2), 182–190.
16. Crowley, R. S., & Medvedeva, O. (2006). An intelligent tutoring system for visual classification problem solving. Artificial Intelligence in Medicine,36(1), 85–117.
17. Crowley, R. S., Medvedeva, O., & Jukic, D. (2003a). SlideTutor: A model-tracing intelligent tutoring system for teaching microscopic diagnosis. In J. Kay, F. Verdejo, & U. Hoppe (Eds.), Proceedings of the 11th International Conference on Artificial Intelligence in Education, Sydney.
18. Crowley, R. S., Naus, G. J., Stewart, J., & Friedman, C. P. (2003b). Development of visual diagnostic expertise in pathology—An information processing study. Journal of American Medical Information Association,10(1), 39–51.
19. Crowley, R. S., Tseytlin, E., & Jukic, D. (2005). ReportTutor—An intelligent tutoring system that uses a natural language interface. In Proceedings of American Medical Informatics Association Symposium, Austin, TX (pp. 171–175).
20. El Saadawi, G., Azevedo, R., Castine, M., Payne, V., Medvedeva, O., Tseytlin, E., et al. (2010). Factors affecting feeling-of-knowing in a medical intelligent tutoring system: The role of immediate feedback as a metacognitive scaffold. Advances in Health Sciences Education,15(1), 9–30.
21. El Saadawi, G. M., Tseytlin, E., Legowski, E., Jukic, D., Castine, M., Fine, J., et al. (2008). A natural language intelligent tutoring system for training pathologists: Implementation and evaluation. Advances in Health Sciences Education: Theory and Practice,13, 709–722.
22. Feyzi-Behnagh, R., Khezri, Z., & Azevedo, R. (2011). An investigation of accuracy of metacognitive judgments during learning with an intelligent multi-agent hypermedia environment. In L. Carlson, C. Hoelscher, & T. F. Shipley (Eds.), Proceedings of the 33rd Annual Conference of the Cognitive Science Society (pp. 96–101). Austin, TX: Cognitive Science Society.
23. Freidman, C. P., Gatti, G. G., Franz, T. M., Murphy, G. C., Wolf, F. M., Heckerling, P. S., et al. (2005). Do physicians know when their diagnoses are correct? Implications for decision support and error reduction. Journal of General Internal Medicine,20, 334–339.
24. Goodman, L. A., & Kruskal, W. H. (1954). Measures of association for cross classifications. Journal of the American Statistical Association,49(268), 732–764.
25. Graber, M. L., Franklin, N., & Gordon, R. (2005). Diagnostic error in internal medicine. Archives of Internal Medicine,165(13), 1493–1499.
26. Graesser, A. C., Bowers, C. A., Hacker, D. J., & Person, N. K. (1997). An anatomy of naturalistic tutoring. In K. Hogan & M. Presley (Eds.), Effective scaffolding of instruction (pp. 145–184). Cambridge, MA: Brookline Books.
27. Graesser, A. C., Wiemer-Hastings, K., Wiemer-Hastings, P., Kruez, R., & The Tutoring Research Group. (2000). AutoTutor: A simulation of a human tutor. Journal of Cognitive Systems Research,1, 35–51.
28. Johnson, A. M., Azevedo, R., & D’Mello, S. K. (2011). The temporal and dynamic nature of self-regulatory processes during independent and externally assisted hypermedia learning. Cognition and Instruction,29(4), 471–504.
29. Juslin, P., Olsson, N., & Winman, A. (1996). Calibration and diagnosticity of confidence in eyewitness identification: Comments on what can be inferred from the low confidence–accuracy correlation. Journal of Experimental Psychology: Learning, Memory, and Cognition,22(5), 1304–1316.
30. Katz, J. (1984). Why doctors don’t disclose uncertainty. The Hastings Center Report,14(1), 35–44.
31. Kelemen, W. L., Frost, P. J., & Weaver, C. A. (2000). Individual differences in metacognition: Evidence against a general metacognitive ability. Memory & Cognition,28(1), 92–107.
32. Koedinger, K., & Aleven, V. (2007). Exploring the assistance dilemma in experiments with cognitive tutors. Educational Psychology Review,19(3), 239–264.
33. Koedinger, K., & Corbett, A. (2006). Cognitive tutors: Technology bringing learning science to the classroom. In R. K. Sawyer (Ed.), The Cambridge Handbook of the Learning Sciences (pp. 61–78). New York: Cambridge University Press.
34. Koriat, A., Sheffer, L., & Ma’ayan, H. (2002). Comparing objective and subjective learning curves: Judgments of learning exhibit increased underconfidence with practice. Journal of Experimental Psychology: General,131, 147–162.
35. Lajoie, S. P. (2009). Developing professional expertise with a cognitive apprenticeship model: Examples from avionics and medicine. In K. A. Ericsson (Ed.), Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments (pp. 61–83). New York: Cambridge University Press.
36. Lajoie, S. P., & Azevedo, R. (2006). Teaching and learning in technology-rich environments. In P. Alexander & P. Winne (Eds.), Handbook of educational psychology (2nd ed., pp. 803–821). Mahwah, NJ: Erlbaum.
37. Lepper, M. R., Drake, M. F., & O’Donnell-Johnson, T. (1997). Scaffolding techniques of expert human tutors. In K. Hogan & M. Pressley (Eds.), Scaffolding student learning: Instructional approaches and issues (pp. 108–144). Cambridge, MA: Brookline.
38. Maki, R. H., Shields, M., Wheeler, A. E., & Zacchilli, T. L. (2005). Individual differences in absolute and relative metacomprehension accuracy. Journal of Educational Psychology,97(4), 723.
39. Mann, D. (1993). The relationship between diagnostic accuracy and confidence in medical students. Atlanta, GA: American Educational Research Association.
40. Maries, A., & Kumar, A. (2008). The effect of student model on learning. In I. Aedo, N. Chen Kinshuk, D. Sampson, & L. Zaitseva (Eds.), ICALT’09, 8th IEEE International Conference on Advanced Learning Technologies (pp. 877–881). Piscataway, NJ: IEEE Computer Society Press.
41. McGinnis, K. S., Lessin, S. R., Elder, D. E., DuPont Guerry, I. V., Schuchter, L., Ming, M., et al. (2002). Pathology review of cases presenting to a multidisciplinary pigmented lesion clinic. Archives of Dermatology,138(5), 617–621.
42. Medvedeva, O., Chavan, G., & Crowley, R. S. (2005). A data collection framework for capturing ITS data based on an agent communication standard. In Proceedings of the 20th Annual Meeting of the Association for the Advancement of Artificial Intelligence (AAAI) (pp. 23–30). Pittsburgh, PA: AAAI Press.
43. Metcalf, J., & Dunlosky, J. (2008). Metamemory. In H. Roediger (Ed.), Cognitive psychology of memory (Vol. 2, pp. 349–362). Oxford: Elsevier.
44. Nelson, T. O. (1984). A comparison of current measures of the accuracy of feeling-of-knowing predictions. Psychological Bulletin, 95, 109–133.
45. Nelson, S. B. (1996). Why study? How reasons for learning influence strategy selection. Educational Psychology Review, 8(4), 335–355.
46. Nietfeld, J. L., Enders, C. K., & Schraw, G. (2006). A Monte Carlo comparison of measures of relative and absolute monitoring accuracy. Educational and Psychological Measurement,66(2), 258–271. doi:10.1177/0013164404273945.
47. Obradovich, J. H., Smith, P. J., Guerlain, S., Rudmann, S., & Smith, J. (2000). Field evaluation of an intelligent tutoring system for teaching problem-solving skills in transfusion medicine. In Proceedings of the IEA 2000/HFES 2000 Congress. 44th Annual Meeting of the Human Factors and Ergonomics Society, San Diego, CA. London: Taylor & Francis.
48. Pintrich, P. R. (2000). The role of goal orientation in self-regulated learning. In M. Boekaerts, P. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 451–502). San Diego, CA: Academic Press.
49. Rogers, E. (1995). VIA-RAD: A blackboard-based system for diagnostic radiology. Artificial Intelligence in Medicine,7, 343–360.
50. Schraw, G. (2009). A conceptual analysis of five measures of metacognitive monitoring. Metacognition and Learning,4(1), 33–45. doi:10.1007/s11409-008-9031-3.
51. Sharples, M., Jeffery, N., du Boulay, B., Teather, B., Teather, D., & du Boulay, G. (2000). Structured computer-based training in the interpretation of neuroradiological images. International Journal of Medical Informatics,60, 263–280.
52. Shute, V. J., & Zapata-Rivera, D. (2012). Adaptive educational systems. In P. Durlach (Ed.), Adaptive technologies for training and education (pp. 7–27). New York: Cambridge University Press.
53. Smith, P., Obradovich, J., Heintz, P., et al. (1998). Successful use of an expert system to teach diagnostic reasoning for antibody identification. In B. P. Goettl, H. M. Halff, C. L. Redfield, & V. J. Shute (Eds.), Proceedings of the 4th International Conference on Intelligent Tutoring Systems, ITS ‘98 (pp. 54–63). San Antonio, TX: Springer.
54. Troxel, D. B. (2006). Medicolegal aspects of error in pathology. Archives of Pathology and Laboratory Medicine,130(5), 617–619.
55. VanLehn, K. (2006). The behavior of tutoring systems. International Journal of Artificial Intelligence in Education,16(3), 227–265.
56. VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist,46(4), 197–221.
57. Voytovich, A. E., Rippey, R. M., & Suffredini, A. (1985). Premature conclusion in diagnostic reasoning. Journal of Medical Education,60, 302–307.
58. Winne, P. H. (2001). Self-regulated learning viewed from models of information processing. In J. Douglas, J. D. Hacker, & A. C. Graesser (Eds.), Self-regulated learning and academic achievement: Theoretical perspective (pp. 153–190). Mahwah, NJ: Erlbaum.
59. Winne, P., & Hadwin, A. (1998). Studying as self-regulated learning. In D. Hacker, J. Dunlosky, & A. Graesser (Eds.), Motivation in educational theory and practice (pp. 227–304). Mahwah, NJ: Erlbaum.
60. Winne, P., & Hadwin, A. (2008). The weave of motivation and self-regulated learning. In D. Schunk & B. Zimmerman (Eds.), Motivation and self-regulated learning: Theory, research, and application (pp. 297–314). Mahwah, NJ: Erlbaum.
61. Woo, C. W., Evens, M. W., Freedman, R., et al. (2006). An intelligent tutoring system that generates a natural language dialogue using dynamic multi-level planning. Artificial Intelligence in Medicine,38(1), 25–46.
62. Woolf, B. P. (2009). Building intelligent interactive tutors: Student-centered strategies for revolutionizing e-learning. Boston: Morgan Kaufmann/Elsevier.
63. Zimmerman, B. (2006). Development and adaptation of expertise: The role of self-regulatory processes and beliefs. In K. A. Ericsson, P. Charness, P. Feltovich, & R. Hoffman (Eds.), The Cambridge handbook of expertise and expert performance (pp. 705–722). New York: Cambridge University Press.