The ASSISTments ecosystem: Building a platform that brings scientists and teachers together for minimally invasive research on human learning and teaching

International Journal of Artificial Intelligence in Education

Volumn 24, Issue 4, 2014, Pages 470-497

  Heffernan, Neil T ^a   Heffernan, Cristina Lindquist ^a  

^a Worcester Polytechnic Institute   (United States)

Author keywords

Authoring tool; Controlled experiment; Formative assessment; Intelligent tutoring System; RCT

Indexed keywords

COMPUTER AIDED INSTRUCTION; CURRICULA; DATA MINING; ECOSYSTEMS; EDUCATION COMPUTING; STUDENTS;

AUTHORING TOOL; CONTROLLED EXPERIMENT; FORMATIVE ASSESSMENT; IMMEDIATE FEEDBACKS; INTELLIGENT TUTORING SYSTEM; MINIMALLY INVASIVE; PLATFORM FOR LEARNING; RANDOMIZED CONTROLLED TRIAL;

TEACHING;

EID: 84908665792     PISSN: 15604292     EISSN: 15604306     Source Type: Journal    
DOI: 10.1007/s40593-014-0024-x     Document Type: Article

References (81)

1. Aleven, V. (2010). Rule-based cognitive modeling for intelligent tutoring systems. In R. Nkambou, J. Bourdeau, R. Mizoguchi (Eds.), Studies in computational intelligence (Advances in intelligent tutoring systems, Vol. 308, pp. 33-62). Berlin: Springer. doi: 10.1007/978-3-642-14363-2-3.
2. Aleven, V.; McLaren, B. M.; Sewall, J. (2009a). Scaling up programming by demonstration for intelligent tutoring systems development: an open-access website for middle-school mathematics learning. IEEE Transactions on Learning Technologies, 2(2), 64-78.
3. Aleven, V.; McLaren, B. M.; Sewall, J.; Koedinger, K. R. (2006). The cognitive tutor authoring tools (CTAT): preliminary evaluation of efficiency gains. In M. Ikeda, K. D. Ashley, T. W. Chan (Eds.), Proceedings of the 8th international conference on intelligent tutoring systems. Taiwan: Jhongli.
4. Aleven, V.; McLaren, B. M.; Sewall, J.; Koedinger, K. R. (2009b). A new paradigm for intelligent tutoring systems: example-tracing tutors. International Journal of Artificial Intelligence in Education, 19(2), 105-154.
5. Anderson, J. R.; Pelletier, R. (1991). A developmental system for model-tracing tutors. In B. Lawrence (Ed.), The international conference on the learning sciences (pp. 1-8). Charlottesville: Association for the Advancement of Computing in Education.
6. Anderson, J. R. (1993). Rules of the mind. Mahwah: Lawrence Erlbaum.
7. Anderson, J. R.; Corbett, A. T.; Koedinger, K. R.; Pelletier, R. (1995). Cognitive tutors: lessons learned. The Journal of the Learning Sciences, 4(2), 167-207.
8. Baker, R. S. J. D.; D'Mello, S. K.; Rodrigo, M. M. T.; Graesser, A. C. (2010). Better to be frustrated than bored: the incidence, persistence, and impact of learners' cognitive-affective states during interactions with three different computer-based learning environments. International Journal of Human Computer Studies, 68(4), 223-24.
9. th Florida Artificial Intelligence Research Society Conference (FLAIRS-19). Press. pp. 497-502.
10. Bloom, B. (1984). The 2 sigma problem: the search for methods of group instruction as effective as one-to-one tutoring. Educational Researcher, 13(6), 4-16.
11. Broderick, Z.; O'Connor, C.; Mulcahy, C.; Heffernan, N.; Heffernan, C. (2012). Increasing parent engagement in student learning using an intelligent tutoring system. Journal of Interactive Learning Research, 22(4), 523-550.
12. Brusilovsky, P.; Schwarz, E.; Weber, G. (1996). ELM-ART: an intelligent tutoring system on World Wide Web. In C. Frasson, G. Gauthier, A. Lesgold (Eds.), Intelligent tutoring systems (Lecture Notes in Computer Science) (Vol. 1086, pp. 261-269). Berlin: Springer Verlag.
13. Chi, M.; VanLehn, K.; Litman, D.; Jordan, P. (2011). An evaluation of pedagogical tutorial tactics for a natural language tutoring system: a reinforcement learning approach. International Journal of Artificial Intelligence in Education, 21(1-2), 83-113. doi: 10.3233/JAI-2011-014.
14. Chen, W.; Aist, G.; Mostow, J. (2011). Recognizing young readers' spoken questions. International Journal of Artificial Intelligence and Education, 21(4).
15. Corbett, A. (2001). Cognitive computer tutors: solving the two-sigma problem. In M. Bauer, P. J. Gmytrasiewicz, J. Vassileva (Eds.), Proceedings of the 2001 international conference on user modeling (pp. 137-147). Berlin: Springer.
16. D'Mello, S. K.; Lehman, B.; Person, N. (2010). Expert tutors feedback is immediate, direct, and discriminating. Proceedings of the 23rd Florida Artificial Intelligence Research Society Conference (FLAIRS-23). AAAI Press. pp. 595-604.
17. du Boulay, B.; Avramides, K.; Luckin, R.; Martínez-Mirón, E.; Méndez, G. R. Carr, A. (2010). Towards systems that care: a conceptual framework based on motivation, metacognition and affect. International Journal of Artificial Intelligence in Education 20(3).
18. Feng, M. Heffernan, N. (2010). Can We get better assessment from a tutoring system compared to traditional paper testing? Can we have our Cake (Better Assessment) and eat it too (Student Learning During the Test) In Baker, R.S.J.d.; Merceron, A.; Pavlik, P.I. Jr. (Eds.) Proceedings of the 3rd International Conference on Educational Data Mining, Pittsburgh, PA. (pp 41-50).
19. Feng, M.; Heffernan, N. T.; Koedinger, K. R. (2009). Addressing the assessment challenge in an intelligent tutoring system that tutors as it assesses. The Journal of User Modeling and User Adapted Interaction.; 19, 243-266.
20. Feng, M.; Roschelle, J.; Heffernan, N.; Fairman, J. Murphy, R. (2014) Implementation of an intelligent tutoring system for online homework support in an efficacy trial. In Stefan Trausan-Matu, et al. (Eds) The Proceeding of the International Conference on Intelligent Tutoring 2014. LNCS 8474. pp 561-566.
21. Gong, Y.; Beck, J. Heffernan, N. (2012). WEBsistments: enabling an intelligent tutoring system to excel at explaining why other than showing how. Proceedings of the 11th International Conference on Intelligent Tutoring Systems, Greece.
22. Heffernan, N. T (2001). Intelligent tutoring systems have forgotten the tutor: adding a cognitive model of human tutors dissertation. Computer Science Department, School of Computer Science, Carnegie Mellon University.
23. Heffernan. N. (2014a). ASSISTments: as a researcher's tool: studies retrieved July 28th 2014 from https://sites.google.com/site/assistmentsstudies/home
24. Heffernan, N. (2014b). Adding research accounts to the ASSISTments platform:randomized controlled studies with thousands of students. An National Science Foundation grant abstract. Retrieved July 28th, 2014 from http://goo.gl/TmlxQv
25. Heffernan, N. T. (2003). Web-based evaluations showing both cognitive and motivational benefits of the Ms. Lindquist tutor In F. Verdejo and U. Hoppe (Eds) 11th International conference artificial intelligence in education. Sydney, Australia. pp.115-122.
26. Heffernan, N. T. Croteau, E. (2004). Web-based evaluations showing differential learning for tutorial strategies employed by the Ms. Lindquist Tutor. In James C. Lester, Rosa Maria Vicari, Fábio Paraguaçu (Eds.) Proceedings of 7th annual intelligent Tutoring Systems Conference, Maceio, Brazil. (pp. 491-500).
27. Heffernan, N. Heffernan, C. (2014) Application process to by outsiders submit a study. Retrieved August 7, 2014 from https://sites.google.com/site/assistmentstestbed/creating-a-study/getting-started
28. Heffernan, N.; Heffernan, C.; Dietz, K.; Soffer, D.; Pellegrino, J. W.; Goldman, S. R. Dailey, M. (2012). Improving mathematical learning outcomes through automatic reassessment and relearning. Proceedings of AERA 2012. Vancouver, BC, Canada.
29. Heffernan, N.; Heffernan, C.; D'Mello, S.; Williams, J. J. and Stoelinga, T. (2014) A webinar explaining how WPI offers to be the midwife to others researchers' ideas. Retrieved August 9, 2014 from https://sites.google.com/site/neilheffernanscv/webinar
30. Heffernan, N. T.; Koedinger, K. R. (2000). Intelligent tutoring systems are missing the tutor: Building a more strategic dialog-based tutor. In C.P. Rose R. Freedman (Eds.) Proceedings of the AAAI fall symposium on building dialogue systems for tutorial applications. Menlo Park, CA. pp. 14-19.
31. Heffernan, N.; Militello, M.; Heffernan, C.; Decoteau, M. (2012b). Effective and meaningful use of educational technology: three cases from the classroom. In C. Dede J. Richards (Eds.), Digital teaching platforms (pp. 88-102). Columbia: Teachers College Press.
32. Johnson, W. L. (2010). Serious use of a serious game for language learning. International Journal of Artificial Intelligence in Education, 20, 175-195.
33. Kehrer, P.; Kelly, K. Heffernan, N. (2013). Does immediate feedback while doing homework improve learning. In Boonthum-Denecke, Youngblood(Eds) Proceedings of the twenty-sixth international Florida artificial intelligence research society conference, FLAIRS 2013, St. Pete Beach, Florida. May 22-24, 2013. AAAI Press 2013. p 542-545.
34. Kelly, K.; Heffernan, N.; D'Mello, S.; Namias, J.; Strain, A. (2013a). Adding teacher-created motivational video to an ITS. Florida artificial intelligence research society (FLAIRS 2013). pp. 503-508.
35. Kelly, K.; Heffernan, N.; Heffernan, C.; Goldman, S.; Pellegrino, J.; Goldstein, D. S. (2013b). Estimating the Effect of Web-Based Homework. In Lane, Yacef, Motow Pavlik (Eds) The Artificial Intelligence in Education Conference. Springer-Verlag. pp. 824-827. (Archived material is at http://web.cs.wpi.edu/~nth/PublicScienceArchive/Kelly.htm).
36. A longer version is available as Kelly, K.; Heffernan, N.; Heffernan, C.; Goldman, S.; Pellegrino, J.; Goldstein, D. S. (2013). Estimating the effect of web-based homework. in formative feedback in interactive learning environments (FFILE) Workshop. Held at the artificial intelligence in education conference, Memphis.
37. Kim, R, Weitz, R.; Heffernan, N. Krach, N. (2009). Tutored problem solving vs. "Pure": worked examples In N. A. Taatgen H. van Rijn (Eds.), Proceedings of the 31st annual conference of the cognitive science society (pp. 3121-3126). Austin, TX: cognitive science society.
38. Koedinger, K. R.; Aleven, V.; Heffernan. T.; McLaren, B. Hockenberry, M. (2004). Opening the door to non-programmers: authoring intelligent tutor behavior by demonstration. In James C. Lester, Rosa Maria Vicari, Fábio Paraguaçu (Eds.) Proceedings of 7th annual intelligent tutoring systems conference, Maceio, Brazil. (pp.162-173).
39. Koedinger, K. R.; Anderson, J. R.; Hadley, W. H.; Mark, M. A. (1997). Intelligent tutoring goes to school in the big city. International Journal of Artificial Intelligence in Education, 8, 30-43.
40. Koedinger, K. R.; Booth, J. L.; Klahr, D. (2013). Instructional complexity and the science to constrain it. Science, 342(6161), 935-937. doi: 10.1126/science.1238056.
41. Koedinger, K.; Martineua, J. (2010) Technology capabilities to assess student cognition and learning in next generation classroom formative assessments. Retrieved from http://www.ctb-e.com/emails/pdfs/Asst-Webinar-11-30-10-final.pdf Slide 40.
42. Koedinger, K.; McLaughlin, E.; Heffernan, N. (2010b). A quasi-experimental evaluation of an on-line formative assessment and tutoring system. Journal of Educational Computing Research Baywood Publishing, 4, 489-510.
43. Lebeau, J.; Fortin, M.; Paquette, L.; Mayers, A. (2009) From cognitive to pedagogical knowledge models in problem-solving ITS frameworks. In Dimitrova, Mizoguchi, du Boulay and Graesser (Eds.) Proceedings of the conference on artificial intelligence in education. pp. 731-733
44. Lester, J.; Rowe, J.; Mott, B. (2013). Narrative-centered learning environments: a story-centric approach to educational games. In C. Mouza N. Lavigne (Eds.), Emerging technologies for the classroom: a learning sciences perspective (pp. 223-238). Manhattan: Springer US.
45. Li, S.; Xiong, X.; Beck, J. (2013). Modeling student retention in an environment with delayed testing. The sixth international conference on educational data mining. Memphis, TN. pp. 328-329
46. Miller, G.; Zheng, Y.; Means, B.; Van Brunt Miller, B. (2013). Next generation learning challenges wave II, Retrieved on August 1st 2014 from http://goo.gl/V6pGUo.
47. National Science Foundation (NSF, 2008). Partnership implementing math and science education: assisting middle school use of tutoring technology. A Grant written by Heffernan funded by the NSF. Abstract retrieved on Jan 2013 from http://www.nsf.gov/awardsearch/showAward?AWD-ID = 0742503.
48. McArthur, D.; Stasz, C.; Zmuidzinas, M. (1990). Tutoring techniques in algebra. Cognition and instruction. 7 pp. 197-244.
49. Mendicino, M.; Razzaq, L.; Heffernan, N. T. (2009). Comparison of traditional homework with computer supported homework. Journal of Research on Technology in Education, 41(3), 331-359.
50. Mitrovic, A.; Ohlsson, S. (1999). Evaluation of a constraint-based tutor for a database language. International Journal of Artificial Intelligence in Education, 10(3), 238-256.
51. Murray, T. (1999). Authoring intelligent tutoring systems: an analysis of the state of the art. International Journal of Artificial Intelligence in Education, 10, 98-129.
52. Ohlsson, S. (1992). Constraint-based student modeling. Journal of Artificial Intelligence in Education, 3(4), 429-447.
53. Ostrow, K.; Heffernan, N. T. (2014) Testing the multimedia principle in the real world: a comparison of video vs. Text feedback in authentic middle school math assignments. In John Stamper et al. (eds) Proceedings of the 7th international conference on educational data mining. pp. 296-299.
54. Ostrow, K.; Heffernan, C. (2014) How to create controlled experiments in ASSISTments. A manual for researchers. Retrieved on August 9, 2014 from https://sites.google.com/site/assistmentstestbed/
55. Pardos, Z.; Dailey, M.; Heffernan, N. (2011). Learning what works in ITS from non-traditional randomized controlled trial data. International Journal of Artificial Intelligence in Education, 21, 47-63.
56. Pardos, Z. Heffernan, N. (2012). Tutor modeling vs. Student modeling. Proceedings of the twenty-fifth international Florida artificial intelligence research society conference. Invited talk. Florida Artificial Intelligence Research Society (FLAIRS 2012). pp 420-425.
57. Paul, A. M. (2012). The Machines are Taking over. New York Times, Education Section. Retrieved on Sept 12, 2014 from http://www.nytimes.com/2012/09/16/magazine/how-computerized-tutors-are-learning-to-teachhumans.html?pagewanted=all&-moc.semityn.www&-r=1&
58. Person, N. K, Graesser, A. C.; Kreuz, R. J.; Pomeroy, V. Tutoring Research Group. (2001). simulating human tutor dialog moves in autotutor. International Journal of Artificial Intelligence in Education, 12.
59. Razzaq, L.; Feng, M.; Nuzzo-Jones, G.; Heffernan, N.T.; Koedinger, K. R.; Junker, B.; Ritter, S.; Knight, A.; Aniszczyk, C.; Choksey, S.; Livak, T.; Mercado, E.;Turner, T.E.; Upalekar. R, Walonoski, J.A.; Macasek. M.A. Rasmussen, K.P. (2005). The assistment project: blending assessment and assisting. In C.K.Looi, G. McCalla, B. Bredeweg, J. Breuker (Eds.) Proceedings of the 12th artificial intelligence in education, Amsterdam: ISO Press. pp. 555-562.
60. Razzaq, L. Heffernan, N.T. (2006). Scaffolding vs. hints in the assistment system. In Ikeda, Ashley Chan (Eds.). Proceedings of the eighth international conference on intelligent tutoring systems. Springer-Verlag: Berlin. pp. 635-644.
61. Razzaq, L.; Heffernan, N.T. (2009). To tutor or not to tutor: that is the question. In Dimitrova, Mizoguchi, du Boulay and Graesser (Eds.) Proceedings of the Conference on Artificial Intelligence in Education. pp. 457-464.
62. Razzaq, L. Heffernan, N. (2010). Hints: is it better to give or wait to be asked? In Aleven, V.; Kay, J Mostow, J. (Eds) Proceedings of the 10th international conference on intelligent tutoring systems (ITS2010) Part 1. Springer. Pages 349-358.
63. Razzaq, L.; Heffernan, N. T.; Lindeman, R. W. (2007). What level of tutor interaction is best? In Luckin Koedinger (Eds.) Proceedings of the 13th conference on artificial intelligence in education. IOS Press. pp 222-229.
64. Razzaq, L.; Mendicino, M.; Heffernan, N. (2008). Comparing classroom problem-solving with no feedback to web-based homework assistance. In A. Woolf L. Nkambou (Eds.), Proceeding of the 9th international conference on intelligent tutoring systems (pp. 426-437). Berlin: Springer.
65. Razzaq, L.; Patvarczki, J.; Almeida, S. F.; Vartak, M.; Feng, M.; Heffernan, N. T.; Koedinger, K. (2009b). The ASSISTment builder: supporting the life-CYCLE of ITS content creation. IEEE Transactions on Learning Technologies Special Issue on Real World Applications of Intelligent Tutoring Systems, 2(2), 157-166.
66. San Pedro, M.; Baker, R.; Gowda, S.; Heffernan, N. (2013). Towards an understanding of affect and knowledge from student interaction with an intelligent tutoring system. In Lane, Yacef, Motow Pavlik (Eds) The artificial intelligence in education conference. Springer-Verlag. pp. 41-50.
67. Sao Pedro, M.; Gobert, J.; Heffernan, N. Beck, J. (2009). In N.A. Taathen H. van Rjin (Eds.) Comparing pedagogical approaches for teaching the control of variables strategy. Proceedings of the 31st annual conference of the cognitive science society Austin, TX.
68. Shrestha, P.; Wei, X.; Maharjan, A.; Razzaq, L.; Heffernan, N.T.; Heffernan, C.; (2009). Are worked examples an effective feedback mechanism during problem solving? In N. A. Taatgen H. van Rijn (Eds.), Proceedings of the 31st annual conference of the cognitive science society (pp. 1294-1299). Austin, TX: Cognitive Science Society.
69. Singh, R.; Saleem, M.; Pradhan, P.; Heffernan, C.; Heffernan, N.; Razzaq, L. Dailey, M. O'Connor, C.Mulchay, C. (2011). Feedback during web-based homework: the role of hints In Biswas et al. (Eds) Proceedings of the artificial intelligence in education conference 2011. pp. 328-336.
70. Soffer, D.; Das, V.; Pellegrino, G.; Goldman, S.; Heffernan, N.; Heffernan, C.;Dietz, K. (2014) Improving long-term retention of mathematical knowledge through automatic reassessment and relearning. American educational research association (AERA 2014) Conference. Division C - learning and instruction/section 1c: mathematics. PDF, poster nominated for the best poster of the session.
71. Suraweera, S. Mitrovic, A. (2004). An intelligent tutoring system for entity relationship modelling. International Journal of Artificial Intelligence in Education, 14(3).
72. VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist, 46(4), 197-221.
73. VanLehn, K.; Lynch, C.; Schulze, K.; Shapiro, J.A.; Shelby, R.; Taylor, L.; Treacy, D.; Weinstein, A.; and Wintersgill, M. (2005). The andes physics tutoring system: lessons learned. International Journal of Artificial Intelligence and Education, 15 (3).
74. VanLehn, K.; Siler, S.; Murray, C. (2003). Why do only some events cause learning during human tutoring? Cognition and Instruction, 21(3), 209-249.
75. Walonoski, J.; Heffernan, N. (2006). Prevention of off-task gaming behavior in intelligent tutoring systems. In A. Ikeda Chan (Eds.), Proceedings of the 8th international conference on intelligent tutoring systems (pp. 722-724). Berlin: Springer.
76. Wang, H. C.; Rose, C. P.; Chang, C. Y. (2011). Agent-based dynamic support for learning from collaborative brainstorming in scientific inquiry. International Journal of Computer Supported Collaborative Learning, 6(3), 371-396.
77. Whorton, S. (2013). Can a computer adaptive assessment system determine, better than traditional methods, whether students know mathematics skills? Masters thesis, computer science department. Retrieved on Jan 10, 2014 from http://www.wpi.edu/Pubs/ETD/Available/etd-041913-095912/
78. Woolf, B. P. (2009) Building Intelligent Tutoring Systems. Morgan Kaufman.
79. Woolf, B.P.; Arroyo, I.; Muldner, K.; Burleson, W.; Cooper, D.; Dolan, R.; Christopherson, R.M. (2010). The effect of motivational learning companions on low-achieving students and students with learning disabilities. Proceedings of the international conference on intelligent tutoring systems, Pittsburgh, PA.
80. WPI IRB (2014) Terms of use of using the platform to to run a study. Worcester polytechnic institute's institutional review board. Retrieved Augusts 7th, 2014 from https://sites.google.com/site/assistmentstestbed/terms-of-use
81. Xiong, X. B0eck, J. (2014) A study of exploring different schedules of spacing and retrieval interval on mathematics skills in ITS environment. In Stefan Trausan-Matu, et al. (Eds) The Proceedings of the International Conference on Intelligent Tutoring 2014. LNCS 8474. Pages 504-509.