Intelligent tutoring systems and learning outcomes: A meta-analysis

Journal of Educational Psychology

Volumn 106, Issue 4, 2014, Pages 901-918

  Ma, Wenting ^a   Adesope, Olusola O ^b   Nesbit, John C ^a   Liu, Qing ^a  

^a SIMON FRASER UNIVERSITY   (Canada)

^b WASHINGTON STATE UNIVERSITY   (United States)

Author keywords

Effect size; Intelligent tutoring system; Meta analysis; Student model

Indexed keywords

EID: 84925386655     PISSN: 00220663     EISSN: 19392176     Source Type: Journal    
DOI: 10.1037/a0037123     Document Type: Article

References (142)

1. Abu-Naser, S. S. (2009). Evaluating the effectiveness of the CPP-Tutor, an Intelligent Tutoring System for students learning to program in C++. Journal of Applied Sciences Research, 5, 109-114.
2. Adesope, O. O., Lavin, T., Thompson, T., & Ungerleider, C. (2010). A systematic review and meta-analysis on the cognitive correlates of bilingualism. Review of Educational Research, 80, 207-245. doi:10.3102/0034654310368803
3. Adesope, O. O., & Nesbit, J. C. (2012). Verbal redundancy in multimedia learning environments: A meta-analysis. Journal of Educational Psychology, 104, 250-263. doi:10.1037/a0026147
4. Aist, G., Mostow, J., Tobin, B., Burkhead, P., Cuneo, A., Junker, B., & Sklar, M. B. (2001). Computer-assisted oral reading helps third graders learn vocabulary better than a classroom control-About as well as one-on-one human-assisted oral reading. In J. D. Moore, C. L. Redfield, & W. L. Johnson (Eds.), Artificial intelligence in education: AI-ED in the wired and wireless future (pp. 267-277). San Antonio, TX: IOS Press.
5. Albacete, P. L., & VanLehn, K. A. (2000). Evaluating the effectiveness of a cognitive tutor for fundamental physics concepts. In L. R. Gleitman & A. K. Joshi (Eds.), Proceedings of the 22nd Annual Meeting of the Cognitive Science Society (pp. 25-30). Mahwah, NJ: Erlbaum.
6. Anderson, J. R. (1993). Rules of the mind. Hillsdale, NJ: Erlbaum.
7. Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995). Cognitive tutors: Lessons learned. Journal of the Learning Sciences, 4, 167-207. doi:10.1207/s15327809jls0402_2
8. Anderson, J. R., & Lebière, C. (1998). The atomic components of thought. Mahwah, NJ: Erlbaum.
9. Arbuckle, W. J. (2005). Conceptual understanding in a computer assisted Algebra I classroom (Unpublished doctoral dissertation). Retrieved from ProQuest database. (UMI No. 320318)
10. Argotte, L., Arroyo-Figueroa, G., & Noguez, J. (2011). SI-APRENDE: An intelligent learning system based on SCORM learning objects for training power systems operators. In K. G. Mehrotra, C. Mohan, J. C. Oh, P. K. Varshney, & M. Ali (Eds.), Developing concepts in applied intelligence (Studies in Computational Intelligence, Vol. 363, pp. 33- 38). doi:10.1007/978-3-642-21332-8_5
11. Arnott, E., Hastings, P., & Allbritton, D. (2008). Research methods tutor: Evaluation of a dialogue-based tutoring system in the classroom. Behavior Research Methods, 40, 694-698. doi:10.3758/BRM.40.3.694
12. Arroyo, I., Royer, J. M., & Woolf, B. P. (2011). Using an intelligent tutor and math fluency training to improve math performance. International Journal of Artificial Intelligence in Education, 21, 135-152.
13. Azevedo, R., & Bernard, R. M. (1995). A meta-analysis of the effects of feedback in computer-based instruction. Journal of Educational Computing Research, 13, 111-127. doi:10.2190/9LMD-3U28-3A0G-FTQT
14. Baker, R. S., Corbett, A. T., Koedinger, K. R., Evenson, S., Roll, I., Wagner, A. Z., . . . Beck, J. E. (2006). Adapting to when students game an intelligent tutoring system. In M. Ikeda, K. D. Ashley, & T. Chan (Eds.), Intelligent tutoring systems (pp. 392-401). Berlin, Germany: Springer Berlin-Heidelberg.
15. Barr, A., Beard, M., & Atkinson, R. C. (1976). The computer as a tutorial laboratory: The Stanford BIP Project. International Journal of Man-Machine Studies, 8, 567-596. doi:10.1016/S0020-7373(76)80021-1
16. Beal, C. R., Arroyo, I., Cohen, P. R., & Woolf, B. P. (2010). Evaluation of AnimalWatch: An Intelligent Tutoring System for arithmetic and fractions. Journal of Interactive Online Learning, 9, 64-77.
17. Beal, C. R., Walles, R., Arroyo, I., & Woolf, B. P. (2007). On-line tutoring for math achievement testing: A controlled evaluation. Journal of Interactive Online Learning, 6, 43-55.
18. Bloom, B. S. (1984). The 2-sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Educational Researcher, 13, 4-16. doi:10.3102/0013189X013006004
19. Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. doi:10.1002/9780470743386
20. Brown, J. S., & Burton, R. R. (1978). Diagnostic models for procedural bugs in basic mathematical skills. Cognitive Science, 2, 155-192. doi: 10.1207/s15516709cog0202_4
21. Cabalo, J. V., Ma, B., & Jaciw, A. (2007). Comparative effectiveness of Carnegie Learning's Cognitive Tutor Bridge to Algebra Curriculum: A report of a randomized experiment in the Maui School District. Palo Alto, CA: Empirical Education.
22. Carbonell, J. (1970). AI in CAI: An artificial intelligence approach to computer aided instruction. Science, 167, 190-202.
23. Carnegie Learning. (2001). Cognitive Tutor results report: Freshman Academy, Canton City Schools, Canton, OH, 2001. Retrieved from http://www.carnegielearning.com/static/web_docs/OH-01-01.pdf
24. Chambers, B., Abrami, P., Tucker, B., Slavin, R. E., Madden, N. A., Cheung, A., & Gifford, R. (2008). Computer assisted tutoring in success for all: Reading outcomes for first graders. Journal of Research on Educational Effectiveness, 1, 120-137. doi:10.1080/19345740801941357
25. Chen, C. M. (2008). Intelligent web-based learning system with personalized learning path guidance. Computers & Education, 51, 787-814. doi:10.1016/j.compedu.2007.08.004
26. Chen, C. M., Lee, H. M., & Chen, Y. H. (2005). Personalized e-learning system using item response theory. Computers & Education, 44, 237-255. doi:10.1016/j.compedu.2004.01.006
27. Chen, L. H. (2011). Enhancement of student learning performance using personalized diagnosis and remedial learning system. Computers & Education, 56, 289-299. doi:10.1016/j.compedu.2010.07.015
28. Chien, T. C., Yunus, A. S. M., Ali, W. Z. W., & Bakar, A. R. (2008). The effect of an Intelligent Tutoring System (ITS) on student achievement in algebraic expression. International Journal of Instruction, 1, 25-38.
29. Chin, D. B., Dohmen, I. M., Cheng, B. H., Oppezzo, M. A., Chase, C. C., & Schwartz, D. L. (2010). Preparing students for future learning with Teachable Agents. Educational Technology Research and Development, 58, 649-669. doi:10.1007/s11423-010-9154-5
30. Cohen, P. A., & Dacanay, L. S. (1992). Computer-based instruction and health professions education: A meta-analysis of outcomes. Evaluation & the Health Professions, 15, 259 -281. doi:10.1177/016327879201500301
31. Conati, C. (2009, July). Intelligent tutoring systems: New challenges and directions. Paper presented at the Twenty-First International Joint Conference on Artificial Intelligence (IJCAI-09), Pasadena, CA.
32. Conati, C., Gertner, A., & VanLehn, K. (2002). Using Bayesian networks to manage uncertainty in student modeling. User Modeling and User-Adapted Interaction, 12, 371-417. doi:10.1023/A:1021258506583
33. Conati, C., & VanLehn, K. (1999). Teaching meta-cognitive skills: Implementation and evaluation of a tutoring system to guide selfexplanation while learning from examples. In S. P. Lajoie & M. Vivet (Eds.), Artificial intelligence in education (pp. 297-304). Amsterdam, the Netherlands: IOS Press.
34. Conati, C., & Zhao, X. (2004). Building and evaluating an intelligent pedagogical agent to improve the effectiveness of an educational game. In N. J. Nunes & C. Rich (Eds.), Proceedings of the International Conference on Intelligent User Interfaces (IUI-04) (pp. 6-13). New York, NY: ACM Press.
35. Cooper, H., Hedges, L. V., & Valentine, J. C. (Eds.). (2009). The handbook of research synthesis and meta-analysis (2nd ed.). New York, NY: Russell Sage Foundation.
36. Corbalan, G., Kester, L., & Van Merriënboer, J. J. G. (2006). Towards a personalized task selection model with shared instructional control. Instructional Science, 34, 399-422. doi:10.1007/s11251-005-5774-2
37. Corbett, A. (2001). Cognitive computer tutors: Solving the two-sigma problem. In M. Bauer, P. J. Gmytrasiewicz, & J. Vassileva (Eds.), User modeling: Proceedings of the Eighth International Conference, UM 2001 (pp. 137-147). Berlin, Germany: Springer-Verlag Berlin-Heidelberg.
38. Corbett, A. T., Koedinger, K. R., & Anderson, J. R. (1997). Intelligent Tutoring Systems. In M. G. Helander, T. K. Landauer, & P. V. Prabhu (Eds.), Handbook of human-computer interaction (pp. 849-874). Amsterdam, the Netherlands: Elsevier.
39. Cumming, G. (2012). Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis. New York, NY: Routledge.
40. Dede, C. (1986). A review and synthesis of recent research in intelligent computer-assisted instruction. International Journal of Man-Machine Studies, 24, 329-353. doi:10.1016/S0020-7373(86)80050-5
41. Denson, N. (2009). Do curricular and co-curricular diversity activities influence racial bias? A meta-analysis. Review of Educational Research, 79, 805-838. doi:10.3102/0034654309331551
42. Fossati, D., Di Eugenio, B., Brown, C., & Ohlsson, S. (2008). Learning linked lists: Experiments with the iList system. In B. P. Woolf, E. Aimeur, R. Nkambou, & S. P. Lajoie (Eds.), Intelligent Tutoring Systems: 9th International Conference, ITS 2008 (pp. 80-89). Berlin, Germany: Springer.
43. Fossati, D., Di Eugenio, B., Ohlsson, S., Brown, C., Chen, L., & Cosejo, D. (2009, July). I learn from you, you learn from me: How to make iList learn from students. Paper presented at the AIED 2009: The 14thInternational Conference on Artificial Intelligence in Education, Brighton, United Kingdom.
44. Graesser, A. C., Chipman, P., Haynes, B., & Olney, A. (2005). AutoTutor: An Intelligent Tutoring System with mixed-initiative dialogue. IEEE Transactions on Education, 48, 612-618. doi:10.1109/TE.2005.856149
45. Graesser, A. C., Jackson, G. T., Mathews, E. C., Mitchell, H. H., Olney, A., Ventura, M., . . . TRG. (2003). Why/AutoTutor: A test of learning gains from a physics tutor with natural language dialog. In R. Alterman & D. Hirsh (Eds.), Proceedings of the 25th Annual Conference of the Cognitive Science Society (pp. 1-5). Boston, MA: Cognitive Science Society.
46. Graesser, A. C., Lu, S., Jackson, G. T., Mitchell, H., Ventura, M., Olney, A., & Louwerse, M. M. (2004). AutoTutor: A tutor with dialogue in natural language. Behavior Research Methods, Instruments, & Computers, 36, 180-192. doi:10.3758/BF03195563
47. Graff, M., Mayer, P., & Lebens, M. (2008). Evaluating a web based Intelligent Tutoring System for mathematics at German lower secondary schools. Education and Information Technologies, 13, 221-230. doi:10.1007/s10639-008-9062-z
48. Grubišíc, A., Stankov, S., & Hrepic, Z. (2008, November). Comparing the effectiveness of learning content management systems to intelligent tutoring systems. Paper presented at the IASK International Conference, Madrid, Spain.
49. Hagerty, G., & Smith, S. (2005). Using the web-based interactive software ALEKS to enhance college algebra. Mathematics and Computer Education, 39, 183-194.
50. Halpern, D. F., Millis, K., Graesser, A. C., & Butler, H. (2012). Operation ARA: A computerized learning game that teaches critical thinking and scientific reasoning. Thinking Skills and Creativity, 7, 93-100. doi:10.1016/j.tsc.2012.03.006
51. Han, K., Lee, E., & Lee, Y. (2010). The impact of a peer-learning agent based on pair programming in a programming course. IEEE Transactions on Education, 53, 318-327. doi:10.1109/TE.2009.2019121
52. Hannum, W. (1986). Techniques for creating computer-based instructional text: Programming languages, authoring languages, and authoring systems. Educational Psychologist, 21, 293-314. doi:10.1207/s15326985ep2104_4
53. Hartley, J. R., & Sleeman, D. H. (1973). Towards more intelligent teaching systems. International Journal of Man-Machine Studies, 5, 215-236. doi:10.1016/S0020-7373(73)80033-1
54. Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77, 81-112. doi:10.3102/003465430298487
55. Hayes-Roth, B., Saker, R., & Amano, K. (2010). Automating individualized coaching and authentic role-play practice for brief intervention training. Methods of Information in Medicine, 49, 406 - 411. doi:10.3414/ME9311
56. Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.
57. Hedges, L. V., & Vevea, J. L. (1998). Fixed and random-effects models in meta-analysis. Psychological Methods, 3, 486-504. doi:10.1037/1082- 989X.3.4.486
58. Heffernan, N. T. (2003). Web-based evaluations showing both cognitive and motivational benefits of the Ms. Lindquist Tutor. Paper presented at the 11th International Conference on Artificial Intelligence in Education, Sydney, Australia.
59. Hu, X., Luellen, J. K., Okwumabua, T. M., Xu, Y., & Mo, L. (2007, April). Observational findings from a web-based Intelligent Tutoring System: Elimination of racial disparities in an undergraduate behavioral statistics course. Paper presented at the 2007 Annual Meeting of the American Educational Research Association (AERA), Chicago, IL.
60. Hughes, M. G., Day, E., Wang, X., Schuelke, M. J., Arsenault, M. L., Harkrider, L. N., & Cooper, O. D. (2013). Learner-controlled practice difficulty in the training of a complex task: Cognitive and motivational mechanisms. Journal of Applied Psychology, 98, 80-98. doi:10.1037/a0029821
61. Hwang, G. J., Tseng, J. C. R., & Hwang, G. H. (2008). Diagnosing student learning problems based on historical assessment records. Innovations in Education and Teaching International, 45, 77- 89. doi:10.1080/14703290701757476
62. International Journal of Artificial Intelligence in Education. (n.d.). Journal scope and standards. Retrieved from http://ijaied.org/journal/scope/
63. Jeremic, Z., Jovanovic, J., & Gasevic, D. (2009). Evaluating an Intelligent Tutoring System for design patterns: The DEPTHS experience. Educational Technology & Society, 12, 111-130.
64. Johnson, B. G., Phillips, F., & Chase, L. G. (2009). An Intelligent Tutoring System for the accounting cycle: Enhancing textbook homeworkwith artificial intelligence. Journal of Accounting Education, 27, 30-39. doi:10.1016/j.jaccedu.2009.05.001
65. Kodaganallur, V., Weitz, R. B., & Rosenthal, D. (2005). A comparison of model-tracing and constraint-based intelligent tutoring paradigms. International Journal of Artificial Intelligence in Education, 15, 117-144.
66. Koedinger, K. R. (2002). Toward evidence for instructional design principles: Examples from Cognitive Tutor Math 6. In D. S. Mewborn, P. Sztajn, D. Y. White, H. G. Wiegel, R. L. Bryant, & K. Nooney (Eds.), Proceedings of the Annual Meeting of the North American Chapter ofthe International Group for the Psychology of Mathematics Education (Vol. 1, pp. 21-49). Columbus, OH: ERIC Clearinghouse for Science, Mathematics, and Environmental Education.
67. Koedinger, K. R., Anderson, J. R., Hadley, W. H., & Mark, M. (1997). Intelligent tutoring goes to school in the big city. International Journal of Artificial Intelligence in Education, 8, 30-43.
68. Kozierkiewicz-Hetmánska, A., & Nguyen, N. T. (2011). A method for learning scenario determination and modification in Intelligent Tutoring Systems. International Journal of Applied Mathematics and Computer Science, 21, 69-82. doi:10.2478/v10006-011-0005-2
69. Kumar, A. N. (2002). Model-based reasoning for domain modeling in a web-based Intelligent Tutoring System to help students learn to debug C ++ programs. In S. A. Cerri, G. Gouarderes, & F. Paraguacu (Eds.), Proceedings of Intelligent Tutoring Systems (ITS-2002) (LNCS 2363, pp. 792-801). Berlin, Germany: Springer-Verlag Berlin-Heidelberg.
70. Landauer, T. K., Foltz, P. W., & Laham, D. (1998). An introduction to latent semantic analysis. Discourse Processes, 25, 259 -284. doi:10.1080/01638539809545028
71. Lane, H. C., & VanLehn, K. (2005). Teaching the tacit knowledge of programming to novices with natural language tutoring. Computer Science Education, 15, 183-201. doi:10.1080/08993400500224286
72. Lanzilotti, R., & Roselli, T. (2007). An experimental evaluation of Logiocando, an intelligent tutoring hypermedia system. International Journal of Artificial Intelligence in Education, 17, 41-56.
73. Lesta, L., & Yacef, K. (2002). An intelligent teaching-assistant system for logic. In S. Cerri & F. Paraguo (Eds.), Proceedings of International Conference on Intelligent Tutoring Systems, ITS'02 (pp. 421-431). Biarritz, France: Springer-Verlag.
74. Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage.
75. Martin, F., Klein, J. D., & Sullivan, H. (2007). The impact of instructional elements in computer-based instruction. British Journal of Educational Technology, 38, 623-636. doi:10.1111/j.1467-8535.2006.00670.x
76. McLaren, B. M., & Isotani, S. (2011). When is it best to learn with all worked examples? In G. Biswas, S. Bull, J. Kay & A. Mitrovic (Eds.), AIED 2011 (LNAI 6738, pp. 222-229). Berlin, Germany: Springer- Verlag Berlin-Heidelberg.
77. McNamara, D. S., O'Reilly, T. P., Best, R. M., & Ozuru, Y. (2006). Improving adolescent students reading comprehension with iSTART. Journal of Educational Computing Research, 34, 147-171. doi:10.2190/1RU5-HDTJ-A5C8-JVWE
78. Millán, E., Loboda, T., & Pérez-de-la-Cruz, J. L. (2010). Bayesian networks for student model engineering. Computers & Education, 55, 1663-1683. doi:10.1016/j.compedu.2010.07.010
79. Mills-Tettey, G. A., Mostow, J., Dias, M. B., Sweet, T. M., Belousov, S. M., Dias, M. F., & Gong, H. (2010). Improving child literacy in Africa: Experiments with an automated reading tutor. Information Technologies and International Development, 6, 1-19.
80. Mitrovic, A. (2003). An intelligent SQL tutor on the web. International Journal of Artificial Intelligence in Education, 13(2- 4), 171-195.
81. Mitrovic, A., & Djordjevic-Kajan, S. (1995). Interactive reconstructive student modeling: A machine-learning approach. International Journal of Human-Computer Interaction, 7, 385- 401. doi:10.1080/10447319509526132
82. Mitrovic, A., Martin, B., & Suraweera, P. (2007). Intelligent tutors for all: Constraint-based modeling methodology, systems and authoring. IEEE Intelligent Systems, 22, 38-45. doi:10.1109/MIS.2007.74
83. Mitrovic, A., Martin, B., Suraweera, P., Zakharov, K., Milik, N., Holland, J., & McGuigan, N. (2009). ASPIRE: An authoring system and deployment environment for constraint-based tutors. International Journal of Artificial Intelligence in Education, 19, 155-188.
84. Mitrovic, A., & Ohlsson, S. (1999). Evaluation of a constraint-based tutor for a database language. International Journal on Artificial Intelligence in Education, 10(3- 4), 238-256.
85. Mitrovic, A., Williamson, C., Bebbington, A., Mathews, M., Suraweera, P., Martin, B., . . . Holland, J. (2011). Thermo-Tutor: An intelligent tutoring system for thermodynamics. In Global Engineering Education Conference (EDUCON), 2011 IEEE (pp. 378-385). Amman, Jordan: IEEE.
86. Morgan, P., & Ritter, S. (2002). An experimental study of the effect of Cognitive Tutor Algebra I on student knowledge and attitude. Retrieved from the Carnegie Learning website: http://carnegielearning.com/web_docs/morgan_ritter_2002.pdf
87. Mostow, J., Aist, G., Bey, J., Burkhead, P., Cuneo, A., Junker, B., . . .Wilson, S. (2002). Independent versus computer-assisted reading: Equal-time comparison of sustained silent reading to an automated reading tutor that listens. Paper presented at the 9th Annual Meeting of the Society for the Scientific Study of Reading, Chicago, IL.
88. Myneni, L. S., Narayanan, N. H., Rebello, S., Rouinfar, A., & Pumtambekar, S. (2013). An interactive and intelligent learning system for physics education. IEEE Transactions on Learning Technologies, 6, 228-239. doi:10.1109/TLT.2013.26
89. National Research Council. (1992). Combining information: Statistical issues and opportunities for research. Washington, DC: National Academy Press.
90. Nesbit, J. C., & Adesope, O. O. (2006). Learning with concept and knowledge maps: A meta-analysis. Review of Educational Research, 76, 413-448. doi:10.3102/00346543076003413
91. Ohlsson, S. (1994). Constraint based user modeling. In J. E. Greer & G. McCalla (Eds.), Student modeling: The key to individualized knowledgebased instruction (pp. 167-189). doi:10.1007/978-3-662-03037-0_7
92. Orwin, R. G. (1983). A fail-safe N for effect size in meta-analysis. Journal of Educational Statistics, 8, 157-159. doi:10.2307/1164923
93. Pane, J. F., McCaffrey, D. F., Slaughter, M. E., Steele, J. L., & Ikemoto, G. S. (2010). An experiment to evaluate the efficacy of Cognitive Tutor Geometry. Journal of Research on Educational Effectiveness, 3, 254-281. doi:10.1080/19345741003681189
94. Patel, K. A., & Russell, D. (2000). A multi-institutional evaluation of Intelligent Tutoring Tools in Numeric Disciplines. Educational Technology and Society, 3, 66-74.
95. Pearl, J. (1988). Probabilistic reasoning in intelligent systems: Networks of plausible inference. San Mateo, CA: Morgan Kaufmann.
96. Person, N. K., Graesser, A. C., Bautista, L., Mathews, E. C., & The Tutoring Research Group. (2001). Evaluating student learning gains in two versions of AutoTutor. In J. D. Moore, C. L. Redfield, & W. L. Johnson (Eds.), Artificial intelligence in education: AI-ED in the wired and wireless future (pp. 286-293). Amsterdam, the Netherlands: IOS Press.
97. Phillips, F., & Johnson, B. G. (2011). Online homework versus Intelligent Tutoring Systems: Pedagogical support for transaction analysis and recording. Issues in Accounting Education, 26, 87-97. doi:10.2308/iace.2011.26.1.87
98. Pinkwart, N., Ashley, K., Lynch, C., & Aleven, V. (2009). Evaluating an Intelligent Tutoring System for making legal arguments with hypotheticals. International Journal of Artificial Intelligence in Education, 19, 401-424.
99. Poulsen, R. (2004). Tutoring bilingual students with an automated reading tutor that listens: Results of a two-month pilot study (Unpublished master's thesis). DePaul University, Chicago, IL.
100. Radwan, Z. R. (1997). Evaluation of the effectiveness of a computer assisted intelligent tutor system model developed to improve specific learning skills of special needs student (Unpublished doctoral dissertation). Retrieved from ProQuest Dissertations and Theses database. (UMI No. 9729551)
101. Ramadhan, H. A. (2000). DISCOVER: An intelligent discovery programming system. Cybernetics & Systems: An International Journal, 31, 87-114. doi:10.1080/019697200124937
102. Reif, F., & Scott, L. A. (1999). Teaching scientific thinking skills: Students and computers coaching each other. American Journal of Physics, 67, 819-831. doi:10.1119/1.19130
103. Ritter, S., Kulikowich, J., Lei, P. W., McGuire, C. L., & Morgan, P. (2007). What evidence matters? A randomized field trial of Cognitive Tutor Algebra I. In T. Hirashima, U. Hoppe, & S. S. C. Young (Eds.), Frontiers in artificial intelligence and applications, 162: Supporting learning flow through integrative technologies (pp. 13-20). Amsterdam, the Netherlands: IOS Press.
104. Roll, I., Baker, R. S., Aleven, V., & Koedinger, K. (2004). A metacognitive ACT-R model of students' learning strategies in Intelligent Tutoring Systems. In J. C. Lester, R. M. Vicari, & F. Paraguacu (Eds.), Proceedings of International Conference on Intelligent Tutoring Systems (pp. 854-856). Berlin, Germany: Springer-Verlag.
105. Rosé, C. P., Bhembe, D., Siler, S., Srivasteva, R., & VanLehn, K. (2003). Exploring the effectiveness of knowledge construction dialogues. In U. Hoppe, F. Verdejo, & J. Kay (Eds.), Artificial intelligence in education: Shaping the future of learning through intelligent technologies (pp. 497-499). Amsterdam, the Netherlands: IOS Press.
106. Rosenthal, R. (1979). The "file drawer problem" and tolerance for null results. Psychological Bulletin, 86, 638-641. doi:10.1037/0033-2909.86.3.638
107. Rosenthal, R. (1995). Critiquing Pygmalion: A 25-year perspective. Current Directions in Psychological Science, 4, 171-172. doi:10.1111/ 1467-8721.ep10772607
108. Rowe, N. C., & Schiavo, S. (1998). An intelligent tutor for intrusion detection on computer systems. Computers and Education, 31, 395-404. doi:10.1016/S0360-1315(98)00049-9
109. Santopietro, G. (2011). Effects of computerized formative assessment on student learning in economics. Journal of Learning in Higher Education, 7, 7-15.
110. Schulze, K. G., Shelby, R. N., Treacy, D. J., Wintersgill, M. C., VanLehn, K., & Gertner, A. (2000). Andes: An active learning Intelligent Tutoring System for Newtonian physics. Themes in Education, 1, 115-136.
111. Shneyderman, A. (2001). Evaluation of the Cognitive Tutor Algebra I program. Unpublished manuscript, Miami-Dade County Public Schools, Office of Evaluation and Research, Miami-Dade County Public Schools, Miami, FL.
112. Shute, V. J., & Glasser, R. (1990). A large-scale evaluation of an intelligent discovery world: Smithtown. Interactive Learning Environments, 1, 51-77. doi:10.1080/1049482900010104
113. Shute, V. J., Hansen, E. G., & Almond, R. G. (2007). An assessment for learning system called ACED: Designing for learning effectiveness and accessibility (ETS Research Report, RR-07-26). Retrieved from http://www.ets.org/research/policy_research_reports/publications/report/2007/hsmy
114. Shute, V. J., & Psotka, J. (1996). Intelligent Tutoring Systems: Past, present, and future. In D. Jonassen (Eds.), Handbook of research for educational communications and technology (pp. 570-600). New York, NY: Macmillan.
115. Sleeman, D. H., & Brown, J. S. (Eds.). (1979). Intelligent tutoring systems [Special issue]. International Journal of Man-Machine Studies, 11, 1-3. doi:10.1016/S0020-7373(79)80002-4
116. Sleeman, D. H., & Brown, J. S. (Eds.). (1982). Intelligent Tutoring Systems. London, England: Academic Press.
117. Smith, J. E. (2001). The effect of the Carnegie Algebra Tutor on student achievement and attitude in introductory high school algebra. (Unpublished doctoral dissertation). Virginia Polytechnic Institute and State University, Blacksburg.
118. Sosa, G. W., Berger, D. E., Saw, A. T., & Mary, J. C. (2011). Effectiveness of computer-assisted instruction in statistics: A meta-analysis. Review of Educational Research, 81, 97-128. doi:10.3102/0034654310378174
119. Sottilare, R., Graesser, A., Hu, X., & Holden, H. (Eds.). (2013). Design recommendations for Intelligent Tutoring Systems. Orlando, FL: U.S. Army Research Laboratory.
120. Stankov, S., Glaviníc, V., & Grubišíc, A. (2004). What is our effect size: Evaluating the educational influence of a web-based intelligent authoring shell? Paper presented at IEEE International Conference on Intelligent Engineering Systems 2004 (INES, 2004), Cluj-Napoca, Romania.
121. Stankov, S., Rosíc, M., Žitko, B., & Grubišíc, A. (2008). TEx-Sys model for building Intelligent Tutoring Systems. Computers & Education, 51, 1017-1036. doi:10.1016/j.compedu.2007.10.002
122. Steenbergen-Hu, S., & Cooper, H. (2013, September 9). A meta-analysis of the effectiveness of Intelligent Tutoring Systems on K-12 students' mathematical learning. Journal of Educational Psychology. Advance online publication. doi:10.1037/a0032447
123. Steenbergen-Hu, S., & Cooper, H. (2014). A meta-analysis of the effectiveness of Intelligent Tutoring Systems (ITS) on college students' academic learning. Journal of Educational Psychology, 106, 331-347. doi:10.1037/a0034752
124. Suraweera, P., & Mitrovic, A. (2002). Kermit: A constraint-based tutor for database modeling. In S. A. Cerri, G. Gouarderes & F. Paraguacu (Eds.), Intelligent Tutoring Systems: 6th International Conference (pp. 377- 387). Berlin, Germany: Springer-Verlag.
125. Suraweera, P., & Mitrovic, A. (2004). An Intelligent Tutoring System for entity relationship modelling. International Journal of Artificial Intelligence in Education (IJAIED), 14(3- 4), 375-417.
126. Tabachnick, B. G., & Fidell, L. S. (2013). Using multivariate statistics (6th ed.). Boston, MA: Allyn & Bacon.
127. Tamim, R. M., Bernard, R. M., Borokhovski, E., Abrami, P. C., & Schmid, R. F. (2011). What forty years of research says about the impact of technology on learning: A second-order meta-analysis and validation study. Review of Educational Research, 81, 4 -28. doi:10.3102/0034654310393361
128. Tsiriga, V., & Virvou, M. (2004). Evaluating the intelligent features of a web-based intelligent computer assisted language learning system. International Journal on Artificial Intelligence Tools, 13, 411-425. doi: 10.1142/S0218213004001600
129. VanLehn, K. (2011). The relative effectiveness of human tutoring, Intelligent Tutoring Systems, and other tutoring systems. Educational Psychologist, 46, 197-221. doi:10.1080/00461520.2011.611369
130. VanLehn, K., Graesser, A. C., Jackson, G. T., Jordan, P., Olney, A., & Carolyn, P. (2007). When are tutorial dialogues more effective than reading? Cognitive Science, 31, 3-62. doi:10.1080/03640210709336984
131. VanLehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R., Taylor, L., . . . Wintersgill, M. (2005). The Andes Physics Tutoring System: Lessons learned. International Journal of Artificial Intelligence in Education, 15, 1-47.
132. VanLehn, K., Van de Sande, B., Shelby, R., & Gershman, S. (2010). The Andes Physics Tutoring System: An experiment in freedom. Studies in Computational Intelligence, 308, 421- 443. doi:10.1007/978-3-642-14363-2_21
133. Veermans, K. H., de Jong, T., & van Joolingen, W. R. (2000). Promoting self-directed learning in simulation based discovery learning environments through intelligent support. Interactive Learning Environments, 8, 229-255. doi:10.1076/1049-4820(200012)8:3;1-D;FT229
134. Veldkamp, B. P., Matteucci, M., & Eggen, T. J. H. M. (2011). Computerized adaptive testing in computer assisted learning? In S. De Wannemacker, G. Clarebout, & P. De Causmaecker (Eds.), Interdisciplinary approaches to adaptive learning. A look at the neighbours (Vol. 126, pp. 28-39). doi:10.1007/978-3-642-20074-8_3
135. Wang, H. C., Rosé, C. P., & Chang, C. Y. (2011). Agent-based dynamic support for learning from collaborative brainstorming in scientific inquiry. International Journal of Computer-Supported Collaborative Learning, 6, 371-395. doi:10.1007/s11412-011-9124-x
136. Wang, N., Johnson, W. L., Mayer, R. E., Rizzo, P., Shaw, E., & Collins, H. (2008). The politeness effect: Pedagogical agents and learning outcomes. International Journal of Human-Computer Studies, 66, 98-112. doi:10.1016/j.ijhcs.2007.09.003
137. Wheeler, J. L., & Regian, J. W. (1999). The use of a cognitive tutoring system in the improvement of the abstract reasoning component of word problem solving. Computers in Human Behavior, 15, 243-254. doi:10.1016/S0747-5632(99)00021-7
138. Wijekumar, K. K., Meyer, B. J. F., & Lei, P. (2012). Large-scale randomized controlled trial with 4th graders using intelligent tutoring of the structure strategy to improve nonfiction reading comprehension. Educational Technology Research and Development, 60, 987-1013. doi:10.1007/s11423-012-9263-4
139. Winne, P. H., & Baker, R. S. J. D. (2013). The potentials of educational data mining for researching metacognition, motivation and selfregulated learning. Journal of Educational Data Mining, 5, 1-8.
140. Wisher, R. A., MacPherson, D. H., Abramson, L. J., Thorndon, D. M., & Dees, J. J. (2001). The virtual sand table: Intelligent tutoring for field artillery training (ARI Research Report No. 1768). Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.
141. Woo, C. W., Evens, W. M. W., Freedman, R., Glass, M., Seop Shim, L., Zhang, Y., Michael, J. (2006). An Intelligent Tutoring System that generates a natural language dialogue using dynamic multi-level planning. Artificial Intelligence in Medicine, 38, 25-46. doi:10.1016/j.artmed.2005.10.004
142. Wu, L., & Looi, C. K. (2012). Agent prompts: Scaffolding for productive reflection in an intelligent learning environment. Journal of Educational Technology & Society, 15, 339-353.