A meta-analysis of the effectiveness of intelligent tutoring systems on college students' academic learning

Journal of Educational Psychology

Volumn 106, Issue 2, 2014, Pages 331-347

  Steenbergen Hu, Saiying ^a   Cooper, Harris ^a  

^a DUKE UNIVERSITY   (United States)

Author keywords

Academic learning; College students; Effectiveness; Intelligent tutoring systems; Meta analysis

Indexed keywords

EID: 84908446791     PISSN: 00220663     EISSN: 19392176     Source Type: Journal    
DOI: 10.1037/a0034752     Document Type: Article

References (90)

1. *Aberson, C. L., Berger, D. E., Healy, M. R., Kyle, D. J., Romero, V. L. (2000). Evaluation of an interactive tutorial for teaching the central limit theorem. Teaching of Psychology, 27, 289-291. doi:10.1207/S15328023TOP2704_08
2. *Aberson, C. L., Berger, D. E., Healy, M. R., Romero, V. L. (2002). An interactive tutorial for teaching statistical power. Journal of Statistics Education, 10(3). Retrieved from www.amstat.org/publications/jse/v10n3/aberson.html
3. *Aberson, C. L., Berger, D. E., Healy, M. R., Romero, V. L. (2003). Evaluation of an interactive tutorial for teaching hypothesis testing concepts. Teaching of Psychology, 30, 75-78. doi:10.1207/S15328023TOP3001_12
4. Aleven, V., Koedinger, K. R. (2002). An effective metacognitive strategy: Learning by doing and explaining with a computer-based Cognitive Tutor. Cognitive Science, 26, 147-179. doi:10.1207/s15516709cog2602_1
5. Aleven, V., McLaren, B., Roll, I., Koedinger, K. (2006). Toward meta-cognitive tutoring: A model of help seeking with a cognitive tutor. International Journal of Artificial Intelligence in Education, 16, 101-128.
6. Anderson, J. R., Corbett, A. T., Koedinger, K. R., Pelletier, R. (1995). Cognitive tutors: Lessons learned. Journal of the Learning Sciences, 4, 167-207.
7. *Arnott, E., Hastings, P., Allbritton, D. (2008). Research Methods Tutor: Evaluation of a dialogue-based tutoring system in the classroom. Behavior Research Methods, 40, 694-698. doi:10.3758/BRM.40.3.694
8. Beal, C. R., Arroyo, I. M., Cohen, P. R., Woolf, B. P. (2010). Evaluation of AnimalWatch: An intelligent tutoring system for arithmetic and fractions. Journal of Interactive Online Learning, 9, 64-77.
9. *Bliwise, N. G. (2005). Web-based tutorials for teaching introductory statistics. Journal of Educational Computing Research, 33, 309-325. doi:10.2190/0D1J-1CE1-5UXY-3V34
10. Bloom, B. S. (1984). The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Educational Researcher, 13(6), 4-16. doi:10.3102/0013189X013006004
11. Borenstein, M., Hedges, L. V., Higgins, J. P. T., Rothstein, H. R. (2006). Comprehensive Meta-Analysis (Version 2.2.027) [Computer software]. Englewood, NJ: Biostat.
12. Campuzano, L., Dynarski, M., Agodini, R., Rall, K. (2009). Effectiveness of reading and mathematics software products: Findings from two student cohorts (ED-01CO0039/0007). Retrieved from http://ies.ed.gov/ncee/pubs/20094041/pdf/20094042.pdf
13. *Chang, K.-E., Sung, Y.-T., Wang, K.-Y., Dai, C.-Y. (2003). Web_Soc: A Socratic-dialect-based collaborative tutoring system on the World Wide Web. IIEE Transactions on Education, 46, 69-78.
14. Cherniavsky, J. C., Vanderputten, E. (2003, April). The controversy of technology in education. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.
15. Cheung, A., Slavin, R. E. (2012). How features of educational technology applications affect student reading outcomes: A metaanalysis. Educational Research Review, 7, 198-215. doi:10.1016/j.edurev.2012.05.002
16. Chi, M. T. H., de Leeuw, N., Chiu, M., LaVancher, C. (1994). Eliciting self-explanation improves understanding. Cognitive Science, 18, 439-477.
17. Chi, M., VanLehn, K. (2010). Meta-cognitive strategy instruction in intelligent tutoring systems: How, when, and why. Educational Technology & Society, 13, 25-39.
18. Cohen, J. (1977). Statistical power analysis for the behavioral sciences (rev. ed.). New York, NY: Academic Press.
19. Conati, C. (2009, July). Intelligent tutoring systems: New challenges and directions. Paper presented at the Twenty-First International Joint Conference on Artificial Intelligence (IJCAI-09), Pasadena, CA.
20. *Conati, C., Muldner, K., Carenini, G. (2006). From example studying to problem solving via tailored computer-based meta-cognitive scaffolding: Hypotheses and design. Technology, Instruction, Cognition and Learning, 4, 1-52.
21. *Conati, C., VanLehn, K. (2000). Further results from the evaluation of an intelligent computer tutor to coach self-explanation. In G. Gauthier, C. Frasson, K. VanLehn (Eds.), Intelligent tutoring systems: 5th International Conference (pp. 304-313). Montreal, Quebec, Canada. doi:10.1007/3-540-45108-0_34
22. Cooper, H. (2010). Research synthesis and meta-analysis: A step-by-step approach (4th ed.). Thousand Oaks, CA: Sage.
23. Cooper, H., Hedges, L. V., Valentine, J. C. (Eds.). (2009). The handbook of research synthesis and meta-analysis (2nd ed.). New York, NY: Russell Sage Foundation.
24. *Corbett, A. (2001). Cognitive computer tutors: Solving the two-sigma problem. In M. Bauer, P. J. Gmytrasiewicz, J. Vassileva (Eds.), User modeling: Proceedings of the Eighth International Conference, UM 2001 (pp. 137-147). Berlin, Germany: Springer-Verlag.
25. Cumming, G., Finch, S. (2001). A primer on the understanding, use and calculation of confidence intervals that are based on central and noncentral distributions. Educational and Psychological Measurement, 61, 532-574.
26. Doignon, J. P., Falmagne, J. C. (1999). Knowledge spaces. Berlin, Germany: Springer. doi:10.1007/978-3-642-58625-5
27. Duval, S., Tweedie, R. (2000). Trim and fill: A simple funnel-plot- based method of testing and adjusting for publication bias in metaanalysis. Biometrics, 56, 455-463. doi:10.1111/j.0006-341X.2000.00455.x
28. Dynarski, M., Agodini, R., Heaviside, S., Novak, T., Carey, N., Campuzano, L., Sussex, W. (2007). Effectiveness of reading and mathematics software products: Findings from the first student cohort (ED-01-CO-0039/0007). Retrieved from http://ies.ed.gov/ncee/pdf/20074005.pdf
29. Glass, G. V., McGaw, B., Smith, M. L. (1981). Meta-analysis in social research. Beverly Hills, CA: Sage.
30. Graesser, A. C., Conley, M., Olney, A. (2011). Intelligent tutoring systems. In K. R. Harris, S. Graham, T. Urdan (Eds.), APA educational psychology handbook: Vol. 3. Applications to learning and teaching (pp. 451-473). Washington, DC: American Psychological Association.
31. *Graesser, A. C., Jackson, G. T., Mathews, E. C., Mitchell, H. H., Olney, A., Ventura, M., . . . Tutoring Research Group. (2003). Why/AutoTutor: A test of learning gains from a physics tutor with natural language dialog. In R. Alterman & D. Hirsh (Eds.), Proceedings of the Twenty- Fifth Annual Conference of the Cognitive Science Society (pp. 1-6). Mahwah, NJ: Erlbaum.
32. *Graesser, A. C., Moreno, K. N., Marineau, J. C., Adcock, A. B., A. M., Person, N. K. (2003). AutoTutor improves deep learning of computer literacy: Is it the dialog or the talking head? In U. Hoppe, F. Verdejo & J. Kay (Eds.), Artificial intelligence in education: Shaping the future of learning through intelligent technologies (pp. 47-54). Amsterdam, the Netherlands: IOS Press. Olney
33. Graesser, A. C., Wiemer-Hastings, K., Wiemer-Hastings, P., Kreuz, R., Tutoring Research Group, University of Memphis. (1999). AutoTutor: A simulation of a human tutor. Journal of Cognitive Systems Research, 1, 35-51. doi:10.1016/S1389-0417(99)00005-4
34. Grubbs, F. E. (1950). Sample criteria for testing outlying observations. Annals of Mathematical Statistics, 21(1), 27-58.
35. *Grubišić, A., Stankov, S., Hrepic, Z. (2008, November). Comparing the effectiveness of learning content management systems to intelligent tutoring systems. Paper presented at the IASK International Conference, Madrid, Spain.
36. *Grubišić, A., Stankov, S., Rosić, M., Žitko, B. (2009). Controlled experiment replication in evaluation of e-learning system's educational influence. Computers & Education, 53, 591- 602. doi:10.1016/j.compedu.2009.03.014
37. *Grubišić, A., Stankov, S., Žitko, B. (2006, September). An approach to automatic evaluation of educational influence. Paper presented at the 6th WSEAS International Conference on Distance Learning and Web Engineering, Lisbon, Portugal.
38. *Hagerty, G., Smith, S. (2005). Using the Web-based interactive software ALEKS to enhance college algebra. Mathematics and Computer Education, 39, 183-194.
39. *Hampikian, J., Guarino, J., Chyung, S. Y., Gardner, J., Moll, A., Pyke, P., Schrader, C. (2007). Benefits of a tutorial mathematics program for engineering students enrolled in precalculus: A template for assessment. Retrieved from http://www.icee.usm.edu/ICEE/conferences/asee2007/papers/1998_BENEFITS_OF_A_TUTORIAL_MATHEMATICS_PROGR.pdf
40. Haslam, M. B., White, R. N., Klinge, A. (2006). Improving student literacy: READ 180 in the Austin Independent School District, 2004-05. Washington, DC: Policy Studies Associates.
41. *Heyden, D. C. (1990). A DSM-III-R computer tutorial for abnormal psychology. Teaching of Psychology, 13, 203-206. doi:10.1207/s15328023top1703_21
42. Hsu, Y. C. (2003). The effectiveness of computer-assisted instruction in statistics education: A meta-analysis (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses database. (UMI No. 3089963)
43. *Hu, X., Luellen, J. K., Okwumabua, T. M., Xu, Y., Mo, L. (2007, April). Observational findings from a Web-based intelligent tutoring system: Elimination of racial disparities in an undergraduate behavioral statistics course. Paper presented at the annual meeting of the American Educational Research Association (AERA), Chicago, IL.
44. *Johnson, B. G., Phillips, F., Chase, L. G. (2009). An intelligent tutoring system for the accounting cycle: Enhancing textbook homework with artificial intelligence. Journal of Accounting Education, 27, 30-39. doi:10.1016/j.jaccedu.2009.05.001
45. *Koch, C., Gobell, J. (1999). A hypertext-based tutorial with links to the Web for teaching statistics and research methods. Behavior Research Methods, Instruments & Computers, 31, 7-13. doi:10.3758/BF03207686
46. Koedinger, K. R., Anderson, J. R., Hadley, W. H., Mark, M. A. (1997). Intelligent tutoring goes to school in the big city. International Journal of Artificial Intelligent in Education, 8, 30-43.
47. Lane, H. C. (2006, August). Intelligent tutoring systems: Prospects for guided practice and efficient learning. Paper presented at the Army's Science of Learning Workshop, Hampton, VA.
48. *Lane, H. C., VanLehn, K. (2005). Teaching the tacit knowledge of programming to novices with natural language tutoring. Computer Science Education, 15, 183-201. doi:10.1080/08993400500224286
49. *Livergood, N. D. (1994). A study of the effectiveness of a multimedia intelligent tutoring system. Journal of Educational Technology Systems, 22, 337-344.
50. McNamara, D. S., Levinstein, I. B., Boonthum, C. (2004). iSTART: Interactive strategy training for active reading and thinking. Behavior Research Methods, Instruments & Computers, 36, 222-233. doi: 10.3758/BF03195567
51. Mitrovic, A. (2012). Fifteen years of constraint-based tutors: What we have achieved and where we are going. User Modeling and User-Adapted Interaction, 22, 39-72. doi:10.1007/s11257-011-9105-9
52. *Mitrovic, A., Ohlsson, S. (1999). Evaluation of a constraint-based tutor for a database language. International Journal of Artificial Intelligence in Education, 10, 238-256.
53. *Morris, E. (2001). The design and evaluation of Link: A computer-based learning system for correlation. British Journal of Educational Technology, 32, 39-52. doi:10.1111/1467-8535.00175
54. Ohlsson, S. (1992). Constraint-based student modeling. International Journal of Artificial Intelligence in Education, 3, 429-447.
55. Ohlsson, S. (1996). Learning from performance errors. Psychological Review, 103, 241-262. doi:10.1037/0033-295X.103.2.241
56. *Phillips, F., Johnson, B. G. (2011). Online homework versus intelligent tutoring systems: Pedagogical support for transaction analysis and recording. Issues in Accounting Education, 26, 87-97. doi:10.2308/iace.2011.26.1.87
57. Pintrich, P. R. (2000). The role of goal orientation in self-regulated learning. In M. Boekaerts, P. Pintrich, M. Zeidner (Eds.), Handbook of self-regulation (pp. 451-502). New York, NY: Academic Press. doi: 10.1016/B978-012109890-2/50043-3
58. Pintrich, P. R. (2004). A conceptual framework for assessing motivation and self-regulated learning in college students. Educational Psychology Review, 16, 385-407. doi:10.1007/s10648-004-0006-x
59. *Reif, F., Scott, L. A. (1999). Teaching scientific thinking skills: Students and computers coaching each other. American Journal of Physics, 67, 819-831. doi:10.1119/1.19130 Ritter, S., Kulikowich, J., Lei, P., McGuire, C. L., Morgan, P. (2007). What evidence matters? A randomized field trial of Cognitive Tutor Algebra I. Supporting Learning Flow Through Integrative Technologies, 162(1), 13-20.
60. Ritter, S., Kulikowich, J., Lei, P., McGuire, C. L., Morgan, P. (2007). What evidence matters? A randomized field trial of Cognitive Tutor Algebra I. Supporting Learning Flow Through Integrative Technologies, 162(1), 13-20.
61. *Rosé, C. P., Bhembe, D., Siler, S., Srivasteva, R., VanLehn, K. (2003). Exploring the effectiveness of knowledge construction dialogues. In U. Hoppe, F. Verdejo, J. Kay (Eds.), Artificial intelligence in education: Shaping the future of learning through intelligent technologies (pp. 497-499). Amsterdam, the Netherlands: IOS Press.
62. Rowley, K., Carlson, P., Miller, T. (1998). A cognitive technology to teach composition skills: Four studies with the R-WISE writing tutor. Journal of Educational Computing Research, 18, 259-296. doi:10.2190/ KW4V-FJKD-L7J1-EFK0
63. Schenker, J. D. (2007). The effectiveness of technology use in statistics instruction in higher education: A meta-analysis using hierarchical linear modeling (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses database. (UMI No. 3286857)
64. *Shute, V. J., Glasser, R. (1990). A large-scale evaluation of an intelligent discovery world: Smithtown. Interactive Learning Environments, 1, 51-77. doi:10.1080/1049482900010104
65. Shute, V. J., Zapata-Rivera, D. (2007). Adaptive technologies (ETS Research Report RR-07-05). Princeton, NJ: Educational Testing Service.
66. Sleeman, D., Brown, J. S. (Eds.). (1982). Intelligent tutoring systems. Orlando, FL: Academic Press.
67. Sosa, G. W., Berger, D. E., Saw, A. T., Mary, J. C. (2011). Effectiveness of computer-assisted instruction in statistics: A meta-analysis. Review of Educational Research, 81, 97-127. doi:10.3102/0034654310378174
68. *Stankov, S., Glavinić, V., Grubišić, A. (2004, April). What is our effect size: Evaluating the educational influence of a Web-based intelligent authoring shell? Paper presented at the IEEE International Conference on Intelligent Engineering Systems 2004-INES 2004, Cluj-Napoca, Romania.
69. Steenbergen-Hu, S., Cooper, H. (2013). A meta-analysis of the effectiveness of intelligent tutoring systems on K-12 students' mathematical learning. Journal of Educational Psychology, 105, 970-987.
70. *Stylianou, D. A., Shapiro, L. (2002). Revitalizing algebra: The effect of the use of a cognitive tutor in a remedial course. Journal of Educational Media, 27, 147-171. doi:10.1080/1358165022000081404
71. Tamim, R. M., Bernard, R. M., Borokhovski, E., Abrami, P. C., Schmid, R. F. (2011). What forty years of research says about the impact of technology on learning: A second-order meta-analysis and validation study. Review of Educational Research, 81, 4 -28. doi:10.3102/0034654310393361
72. Tamim, R. M., Lowerison, G., Schmid, R. F., Bernard, R. M., Abrami, P. C. (2011). A multi-year investigation of the relationship between pedagogy, computer use and course effectiveness in postsecondary education. Journal of Computing in Higher Education, 23, 1-14. doi: 10.1007/s12528-010-9041-4
73. Thompson, B. (2002). What future quantitative social science research could look like: Confidence intervals for effect sizes. Educational Researcher, 31(3), 25-31. doi:10.3102/0013189X031003025
74. Thompson, B. (2006). Research synthesis: Effect sizes. In J. Green, G. Camilli, P. B. Elmore (Eds.), Complementary methods for research in education (pp. 583-603). Mahwah, NJ: Erlbaum.
75. VanLehn, K. (2006). The behavior of tutoring systems. International Journal of Artificial Intelligence in Education, 16, 227-265.
76. VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist, 46, 197-221. doi:10.1080/00461520.2011.611369
77. *VanLehn, K., Graesser, A. C., Jackson, G. T., Jordan, P., Olney, A., Carolyn, P. (2007). When are tutorial dialogues more effective than reading? Cognitive Science, 31, 3-62. doi:10.1080/03640210709336984
78. VanLehn, K., Jordan, P., Rosé, C. P., Behmbe, C., Böttner, D., Gaydos, M., Srivastava, R. (2002). The architecture of Why2-Atlas: A coach for qualitative physics essay writing. In S. A. Cerri, G. Gouarderes, F. Paraguacu (Eds.), Intelligent tutoring systems: 6th International Conference (pp. 158-167). Berlin, Germany: Springer.
79. *VanLehn, K., Van de Sande, B., Shelby, R., Gershman, S. (2010). The Andes Physics Tutoring System: An experiment in freedom. Studies in Computational Intelligence, 308, 421- 443. doi:10.1007/978-3-642-14363-2_21
80. *Wang, H.-C., Li, T.-Y., Chang, C.-Y. (2006). A Web-based tutoring system with styles-matching strategy for spatial geometric transformation. Interacting With Computers, 18, 331-355. doi:10.1016/j.intcom.2005.11.002
81. What Works Clearinghouse. (2004, December). What Works Clearinghouse topic report: Curriculum-based interventions for increasing K-12 math achievement-middle school. Retrieved from http://www.eric.ed.gov/PDFS/ED485395.pdf
82. What Works Clearinghouse. (2007, May). WWC intervention report middle school math: Cognitive Tutor Algebra I. Retrieved from http://www.aea9.k12.ia.us/documents/filelibrary/pdf/cognitive_tutor/WWC_Cognitive_Tutor_052907_3B8688D14AA44.pdf
83. What Works Clearinghouse. (2009, July). WWC intervention report middle school math: Cognitive Tutor Algebra I. Retrieved from http://www.aea9.k12.ia.us/documents/filelibrary/pdf/cognitive_tutor/WWC_CogTutor_Report_July2009_B2A3C279D0481.pdf
84. What Works Clearinghouse. (2010a, August). WWC intervention report high school math: Carnegie Learning Curricula and Cognitive Tutor software. Retrieved from http://ies.ed.gov/ncee/wWc/pdf/intervention_reports/wwc_cogtutor_083110.pdf
85. What Works Clearinghouse. (2010b, March). WWC intervention report middle school math: Plato Achieve Now. Retrieved from http://ies.ed.gov/ncee/wwc/interventionreport.aspx?sid=378
86. What Works Clearinghouse. (2013). What Works Clearinghouse: Procedures and standards handbook (Version 3.0). Retrieved from http://ies.ed.gov/ncee/wwc/pdf/reference:resources/wwc_procedures_v3_0_draft_standards_handbook.pdf
87. Wilson, D. B., Lipsey, M. W. (2001). The role of method in treatment effectiveness research: Evidence from meta-analysis. Psychological Methods, 6, 413-429. doi:10.1037/1082-989X.6.4.413
88. Woolf, B. P. (2009). Building intelligent interactive tutors: Studentcentered strategies for revolutionizing e-learning. Burlington MA: Morgan Kaufman Publishers.
89. Woolf, B. P., Cunningham, P. A. (1987). Multiple knowledge sources in intelligent teaching systems. IEEE Expert, 2(2), 41-54. doi:10.1109/MEX.1987.4307063
90. Xu, Y. J., Meryer, K. A., Morgan, D. D. (2009). A mixed-methods assessment of using an online commercial tutoring system to teach introductory statistics. Journal of Statistics Education, 17, 1-17. Retrieved from http://www.amstat.org/publications/jse/v17n2/xu.pdf