BEETLE II: Deep natural language understanding and automatic feedback generation for intelligent tutoring in basic electricity and electronics

International Journal of Artificial Intelligence in Education

Volumn 24, Issue 3, 2014, Pages 284-332

  Dzikovska, Myroslava ^a   Steinhauser, Natalie ^b   Farrow, Elaine ^a   Moore, Johanna ^a   Campbell, Gwendolyn ^b  

^a UNIVERSITY OF EDINBURGH   (United Kingdom)

^b Naval Air Warfare Center Training Systems Division   (United States)

Author keywords

Intelligent tutoring systems; Natural language processing; Tutorial dialogue

Indexed keywords

CIRCUIT SIMULATION; COMPUTER AIDED INSTRUCTION; CURRICULA; ELECTRIC GENERATORS; ELECTRICITY; EXPERIMENTS; QUALITY CONTROL; STUDENTS;

ACTIVE EXPERIMENTATION; EFFECTIVE TEACHING METHODS; INTELLIGENT TUTORING SYSTEM; NATURAL LANGUAGE PROCESSING; NATURAL LANGUAGE UNDERSTANDING; RESEARCH AND DEVELOPMENT; SIMULATION ENVIRONMENT; TUTORIAL DIALOGUE;

NATURAL LANGUAGE PROCESSING SYSTEMS;

EID: 84903699960     PISSN: 15604292     EISSN: 15604306     Source Type: Journal    
DOI: 10.1007/s40593-014-0017-9     Document Type: Article

References (94)

1. Aleven, V., Ogan, A., Popescu, O., Torrey, C., Koedinger, K. (2004). Evaluating the effectiveness of a tutorial dialogue system for self-explanation. In Lester, J. C., Vicari, R. M., Paraguaçu, F. (Eds.), Proceedings of the international conference on intelligent tutoring systems (Vol. 3220, pp. 443-454). Berlin: Springer.
2. Aleven, V., Popescu, O., Koedinger, K.R. (2002). Pilot-testing a tutorial dialogue system that supports self-explanation. In Proceedings of the 6th international conference on intelligent tutoring systems (pp. 344-354). Springer.
3. Allen, J., Dzikovska, M., Manshadi, M., Swift, M. (2007). Deep linguistic processing for spoken dialogue systems. In Proceedings of the ACL-07 workshop on deep linguistic processing (pp. 49-56). Association for Computational Linguistics.
4. Bell, P., Dzikovska, M., Isard, A. (2012). Designing a spoken language interface for a tutorial dialogue system. In Proceedings of 13th annual conference of the international speech communication association (INTERSPEECH 2012). ISCA.
5. Bohus, D., & Rudnicky, A. (2005). Sorry, I didn't catch that! - An investigation of nonunderstanding errors and recovery strategies. In Proceedings of 6th SIGdial workshop on discourse and dialogue. Lisbon: Association for Computational Linguistics.
6. Byron, D.K. (2002). Resolving pronominal reference to abstract entities (Unpublished doctoral dissertation). University of Rochester.
7. Callaway, C., Dzikovska, M., Farrow, E., Marques-Pita, M., Matheson, C., Moore, J.D. (2007). The Beetle and BeeDiff tutoring systems. In Proceedings of the SLaTE workshop on Speech and Language Technology in Education (SLaTE'07).
8. Callaway, C., Dzikovska, M., Matheson, C., Moore, J., Zinn, C. (2006). Using dialogue to learn math in the LeActiveMath project. In Proceedings of the ECAI workshop on languageenhanced educational technology (pp. 1-8).
9. Campbell, G.C., Steinhauser, N.B., Dzikovska, M.O., Moore, J.D., Callaway, C.B., Farrow, E. (2009). The DeMAND coding scheme: A "common language" for representing and analyzing student discourse. In Proceedings of 14th International Conference on Artificial Intelligence in Education (AIED), poster session.
10. Carey, S. (1986). Cognitive science and science education. American Psychologist, 10, 1123-1130.
11. Chi, M., Jordan, P., VanLehn, K., Litman, D. (2009). To elicit or to tell: Does it matter? In Proceedings 14th International Conference on Artificial Intelligence in Education (AIED). Brighton, UK.
12. Chi, M., VanLehn, K., Litman, D., Jordan, P. (2011). Empirically evaluating the application of reinforcement learning to the induction of effective and adaptive pedagogical strategies. User Modeling and User-Adapted Interaction, 21, 137-180.
13. Chi, M.T.H., de Leeuw, N., Chiu, M.-H., LaVancher, C. (1994). Eliciting self-explanations improves understanding. Cognitive Science, 18(3), 439-477.
14. Chinn, C.A., & Brewer, W.F. (1993). The role of anomalous data in knowledge acquisition: a theoretical framework and implications for science instruction. Review of Educational Research, 63(1), 1-49.
15. Cho, B.-I. (2000). Dynamic planning models to support curriculum planning and multiple tutoring protocols in intelligent tutoring systems (Unpublished doctoral dissertation). Illinois Institute of Technology.
16. Clark, P., & Porter, B. (1999). KM (1.4): Users manual [Computer software manual]. Retrieved from http://www.cs.utexas.edu/users/mfkb/km.
17. de Jong, T., & van Joolingen, W.R. (1998). Scientific discovery learning with computer simulations of conceptual domains. Review of Educational Research, 68(2), 179-201.
18. Dietterich, T.G. (1998). Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation, 10(7), 1895-1923.
19. Duffy, T.M., & Jonassen, D.H. (1992). Constructivism and the technology of instruction: A conversation. Hillsdale: Lawrence Erlbaum Associates.
20. Duit, R., & Treagust, D.F. (2003). Conceptual change: A powerful framework for improving science teaching and learning. International Journal of Science Education, 25(6), 671-688.
21. Dzikovska, M.O., Allen, J.F., Swift, M.D. (2008). Linking semantic and knowledge representations in a multi-domain dialogue system. Journal of Logic and Computation, 18(3), 405-430.
22. Dzikovska, M.O., Bell, P., Isard, A., Moore, J.D. (2012). Evaluating language understanding accuracy with respect to objective outcomes in a dialogue system. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics (pp. 471-481). Avignon: Association for Computational Linguistics.
23. Dzikovska, M.O., Callaway, C.B., Farrow, E. (2006). Interpretation and generation in a knowledge-based tutorial system. In Proceedings of EACL-06 workshop on Knowledge and Reasoning for Language Processing (KRAQ'06). Trento: European Association for Computational Linguistics.
24. Dzikovska, M.O., Callaway, C.B., Farrow, E., Moore, J.D., Steinhauser, N.B., Campbell, G.E. (2009). Dealing with interpretation errors in tutorial dialogue. In Proceedings of the SIGDIAL 2009 Conference: The 10th annual meeting of the special interest group on discourse and dialogue (pp. 38-45). Association for Computational Linguistics.
25. Dzikovska, M.O., Campbell, G.E., Callaway, C.B., Steinhauser, N.B., Farrow, E., Moore, J.D.,., Matheson, C. (2008). Diagnosing natural language answers to support adaptive tutoring. In Proceedings of the 21st International FLAIRS Conference. Coconut Grove, Florida.
26. Dzikovska, M.O., Farrow, E., Moore, J. (2013a). Improving interpretation robustness in a tutorial dialogue system. In Proceedings of the 8th workshop on innovative use of NLP for Building Educational Applications (BEA-8). Atlanta: Association for Computational Linguistics.
27. Dzikovska, M.O., Farrow, E., Moore, J.D. (2013b). Combining semantic interpretation and statistical classification for improved explanation processing in a tutorial dialogue system. In Proceedings of the 16th international conference on Artificial Intelligence in Education (AIED 2013). Memphis, TN, USA.
28. Dzikovska, M.O., Moore, J.D., Steinhauser, N., Campbell, G. (2010). The impact of interpretation problems on tutorial dialogue. In Proceedings of the 48th annual meeting of the Association for Computational Linguistics(ACL-2010) (pp. 43-48). Association for Computational Linguistics.
29. Dzikovska, M.O., Moore, J.D., Steinhauser, N., Campbell, G. (2011). Exploring user satisfaction in a tutorial dialogue system. In Proceedings of the SIGDIAL 2011 conference (pp. 162-172). Portland: Association for Computational Linguistics.
30. Dzikovska, M.O., Nielsen, R., Brew, C., Leacock, C., Giampiccolo, D., Bentivogli, L.,., Dang, H.T. (2013). Semeval-2013 task 7: The joint student response analysis and 8th recognizing textual entailment challenge. In Proceedings of the 6th International Workshop on Semantic Evaluation (SEMEVAL-2013). Atlanta: Association for Computational Linguistics.
31. Dzikovska, M.O., Nielsen, R.D., Brew, C. (2012). Towards effective tutorial feedback for explanation questions: A dataset and baselines. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT12) (pp. 200-210). Association for Computational Linguistics.
32. Dzikovska, M.O., Steinhauser, N.B., Moore, J.D., Campbell, G.E., Harrison, K.M., Taylor, L.S. (2010). Content, social, and metacognitive statements: An empirical study comparing human-human and human-computer tutorial dialogue. In Sustaining TEL: From innovation to learning and practice - 5th European Conference on Technology Enhanced Learning (EC-TEL 2010) (pp. 93-108). Barcelona, Spain.
33. Dzikovska, M.O., Swift, M.D., Allen, J.F. (2004). Building a computational lexicon and ontology with FrameNet. In Proceedings of LREC workshop on building lexical resources from semantically annotated corpora. Lisbon, Portugal.
34. Elhadad, M., & Robin, J. (1992). Controlling content realization with functional unification grammars In Dale, R., Hovy, E., Rösner, D., Stock, O. (Eds.), Proceedings of the sixth international workshop on natural language generation, (pp. 89-104). Berlin: Springer.
35. Engelhardt, P.V., & Beichner, R.J. (2004). Students' understanding of direct current resistive electrical circuits. American Journal of Physics, 72(1), 98-115.
36. Eylon, B.S., & Linn, M.C. (1988). Learning and instruction: An examination in four research perspectives in science education. Review of Educational Research, 58(3), 251-301.
37. Farnham-Diggory, S. (1994). Paradigms of knowledge and instruction. Review of Educational Research, 64(3), 463-477.
38. Forbes-Riley, K., & Litman, D.J. (2011). Designing and evaluating a wizarded uncertainty-adaptive spoken dialogue tutoring system. Computer Speech & Language, 25(1), 105-126.
39. Glass, M.S. (2000). Processing language input in the CIRCSIM-Tutor intelligent tutoring system. In Papers from the 2000 AAAI fall symposium (pp. 74-79). Available as AAAI technical report FS-00-01.
40. Glass, M.S., & Evens, M.W. (2008). Extracting information from natural language input to an intelligent tutoring system. Far Eastern Journal of Experimental and Theoretical Artificial Intelligence, 1(2).
41. Graesser, A.C., Wiemer-Hastings, K., Wiemer-Hastings, P., Kreuz, R. (1999). Autotutor: A simulation of a human tutor. Cognitive Systems Research, 1, 35-51.
42. Halloun, I.A., & Hestenes, D. (1985a). Common sense concepts about motion. American Journal of Physics, 53(11), 1056-1065.
43. Halloun, I.A., & Hestenes, D. (1985b). The initial knowledge state of college physics students. American Journal of Physics, 53(11), 1043-1055.
44. Hockey, B.A., Lemon, O., Campana, E., Hiatt, L., Aist, G., Hieronymus, J.,., Dowding, J. (2003). Targeted help for spoken dialogue systems: intelligent feedback improves naive users' performance. In Proceedings of the tenth conference of European chapter of the association for computational linguistics (pp. 147-154). Association for Computational Linguistics.
45. Jackson, G.T., Graesser, A.C., McNamara, D.S. (2009). What students expect may have more impact than what they know or feel. In Proceedings of the 14th international conference on Artificial Intelligence in Education (AIED). Brighton, UK.
46. Jordan, P., Albacete, P., VanLehn, K. (2005). Taking control of redundancy in scripted tutorial dialogue. In Proceedings of international conference on Artificial Intelligence in Education (AIED-2005) (pp. 314-321).
47. Jordan, P., Hall, B., Ringenberg, M., Cue, Y., Rosé, C. (2007). Tools for authoring a dialogue agent that participates in learning studies. In Proceedings of the 2007 Conference on Artificial Intelligence in Education: Building technology rich learning contexts that work (pp. 43-50). IOS Press.
48. Jordan, P., Litman, D., Lipschultz, M., Drummond, J. (2009). Evidence of misunderstandings in tutorial dialogue and their impact on learning. In Proceedings of the 2009 conference on artificial intelligence in education: building learning systems that care: from knowledge representation to affective modelling (pp. 125-132). IOS Press.
49. Jordan, P., Makatchev, M., Pappuswamy, U., VanLehn, K., Albacete, P. (2006). A natural language tutorial dialogue system for physics. In Proceedings of the 19th International FLAIRS Conference (pp. 521-527).
50. Katz, S., Allbritton, D., Connelly, J. (2003). Going beyond the problem given: how human tutors use post-solution discussions to support transfer. International Journal of Artificial Intelligence in Education, 13(1), 79-116.
51. Katz, S., Connelly, J., Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in andes. In Proceedings of the 2007 Conference on Artificial Intelligence in Education: Building technology rich learning contexts that work (pp. 425-432). Amsterdam: IOS Press.
52. Khuwaja, R.A., Evens, M.W., Michael, J.A., Rovick, A.A. (1994). Architecture of CIRCSIM-tutor (v.3): A smart cardiovascular physiology tutor. In Proceedings of the 7th annual IEEE computer-based medical systems symposium (pp. 158-163). IEEE Computer Society Press.
53. King, P.M. (1997). The reflective judgment model: transforming assumptions about knowing. College Student Development and Academic Life: Psychological, Intellectual, Social, and Moral Issues, 4, 141.
54. Kolb, D.A. (1984). Experiential learning: Experience as the source of learning and development (Vol. 1). Englewood Cliffs: Prentice-Hall.
55. Larsson, S., & Traum, D. (2000). Information state and dialogue management in the TRINDI dialogue move engine toolkit. Natural Language Engineering, 6(3-4), 323-340.
56. Lee, A.Y., & Hutchison, L. (1998). Improving learning from examples through reflection. Journal of Experimental Psychology: Applied, 4(3), 187.
57. Litman, D., & Forbes-Riley, K. (2005). Speech recognition performance and learning in spoken dialogue tutoring. In Proceedings of EUROSPEECH-2005 (p. 1427).
58. Litman, D., Moore, J., Dzikovska, M., Farrow, E. (2009). Using natural language processing to analyze tutorial dialogue corpora across domains and modalities. In Proceedings of 14th International Conference on Artificial Intelligence in Education (AIED). Brighton, UK.
59. Litman, D., & Silliman, S. (2004). ITSPOKE: an intelligent tutoring spoken dialogue system. In Demonstration papers at HLT-NAACL 2004 (pp. 5-8). Boston: Association for Computational Linguistics.
60. McCarthy, P.M., Rus, V., Crossley, S.A., Graesser, A.C., McNamara, D.S. (2008). Assessing forward-, reverse-, and average-entailment indices on natural language input from the intelligent tutoring system, iSTART. In Wilson, D., Sutcliffe, G. (Eds.), Proceedings of the 21st International Florida Artificial Intelligence Research Society Conference (pp. 165-170).
61. McDermott, L.C., Shaffer, P.S., Rosenquist, C.M., Physics Education Group Univ. Washington (1995). Physics by inquiry: An introduction to physics and the physical sciences (Vol. 2). New York: Wiley.
62. Mestre, J.P., Docktor, J.L., Strand, N.E., Ross, B.H. (2011). Conceptual problem solving in physics. In Mestre, J.P., Ross, B.H. (Eds.), Cognition in education (Vol. 55, p. 269-298). Academic Press.
63. Mller, S., Smeele, P., Boland, H., Krebber, J. (2007). Evaluating spoken dialogue systems according to de-facto standards: a case study. Computer Speech & Language, 21(1), 26-53.
64. Narciss, S. (2004). The impact of informative tutoring feedback and self-efficacy on motivation and achievement in concept learning. Experimental Psychology, 51(3), 214-228.
65. Narciss, S., & Huth, K. (2004). How to design informative tutoring feedback for multimedia learning. In Niegemann, D.L.H., & Brunken, R. (Eds.), Instructional design for multimedia learning, (pp. 181-195). Waxmann: Muenster.
66. Nielsen, R.D., Ward, W., Martin, J.H. (2008). Learning to assess low-level conceptual understanding. In Proceedings 21st International FLAIRS Conference (pp. 427-432). Coconut Grove, Florida.
67. Osborne, R.J., & Wittrock, M.C. (1983). Learning science: a generative process. Science Education, 67, 489-508.
68. Papadopoulos, P.M., Demetriadis, S.N., Stamelos, I. (2009). The impact of prompting in technology-enhanced learning as moderated by students motivation and metacognitive skills. In Cress, U., Dimitrova, V., Specht, M., (Eds.), Learning in the synergy of multiple disciplines, proceedings of the 4th european conference on technology enhanced learning, (EC-TEL 2009) (Vol. 5794, pp. 535-548). Springer.
69. Pon-Barry, H., Clark, B., Bratt, E.O., Schultz, K., Peters, S. (2004). Evaluating the effectiveness of SCoT: A spoken conversational tutor. In Mostow, J., Tedesco, P. (Eds.), Proceedings of the ITS 2004 workshop on dialog-based intelligent tutoring systems (pp. 23-32).
70. Pon-Barry, H., Clark, B., Schultz, K., Bratt, E.O., Peters, S. (2004). Advantages of spoken language interaction in dialogue-based intelligent tutoring systems. In Lester, J.C., Vicari, R.M., Paraguaçu, F. (Eds.), Proceedings of Intelligent Tutoring Systems Conference (ITS-2004) (Vol. 3220, pp. 390-400). Springer.
71. Purandare, A., & Litman, D. (2008). Content-learning correlations in spoken tutoring dialogs at word, turn and discourse levels. In Proceedings of the 21st International FLAIRS Conference.
72. Redish, E.F., Saul, J.M., Steinberg, R.M. (1997). On the effectiveness of activeengagement microcomputer-based laboratories. American Journal of Physics, 65, 45-54.
73. Ros, C.P., Torrey, C., Aleven, V., Robinson, A., Wu, C., Forbus, K. (2004). CycleTalk: Toward a dialogue agent that guides design with an articulate simulator. In Lester, J. C., Vicari, R. M., Paraguaçu, F. (Eds.), Proceedings of the intelligent tutoring systems conference (Vol. 3220, pp. 401-411). Berlin: Springer.
74. Rotaru, M., & Litman, D.J. (2009). Discourse structure and performance analysis: beyond the correlation. In Proceedings of the SIGDIAL 2009 Conference: The 10th annual meeting of the special interest group on discourse and dialogue (pp. 178-187). Association for Computational Linguistics.
75. Sagot, B., & de La Clergerie, E. (2006). Error mining in parsing results. In Proceedings of the 21st international conference on computational linguistics and 44th annual meeting of the association for computational linguistics (pp. 329-336). Sydney: Association for Computational Linguistics.
76. Scott, P.H., Asoko, H.M., Driver, R. (1992). Teaching for conceptual change: A review of strategies. In Duit, R., Goldberg, F., Niedderer, H. (Eds.), Research in physics learning: Theoretical issues and empirical studies, (pp. 310-329). Kiel: IPN.
77. Seeskin, K. (1987). Dialogue and discovery: A study in socratic method. SUNY Press.
78. Shute, V.J. (2008). Focus on formative feedback. Review of educational research, 78(1), 153-189.
79. Sokoloff, D.R. (1996). Teaching electric circuit concepts using microcomputer-based current/voltage probes. In Tinker, R. F. (Ed.), Microcomputer-based labs: Educational research and standards (Vol. 156, pp. 129-146). Berlin: Springer.
80. Steinhauser, N.B., Butler, L.A., Campbell, G.E. (2007). Simulated tutors in immersive learning environments: Empirically-derived design principles. In Proceedings of the 2007 interservice/industry training, simulation and education conference. Orlando, FL.
81. Steinhauser, N.B., Campbell, G.E., Harrison, K.M., Taylor, L.S., Dzikovska, M.O., Moore, J.D. (2010). Comparing human-human and human-computer tutorial dialogue. In Proceedings of the 32nd Annual Conference of the Cognitive Science Society poster session.
82. Steinhauser, N.B., Campbell, G.E., Taylor, L.S., Caine, S., Scott, C., Dzikovska, M.O., Moore, J.D. (2011). Talk like an electrician: Student dialogue mimicking behavior in an intelligent tutoring system. In Proceedings of the 15th International Conference on Artificial Intelligence in Education (AIED-2011) (pp. 361-368). Berlin: Springer.
83. Swift, M. (2005). Towards automatic verb acquisition from VerbNet for spoken dialog processing. In Proceedings of the interdisciplinary workshop on the identification and representation of verb features and verb classes. Saarbrucken, Germany.
84. Tallant, D. (1993). A review of misconceptions of electricity and electrical circuits. In Novak, J. (Ed.), Proceedings of the third international seminar on misconceptions and educational strategies in science and mathematics, (pp. 111-120). New York: Cornell University.
85. Thacker, B., Kim, E., Trefz, K., Lea, S.M. (1994). Comparing problem solving performance of physics students in inquiry-based and traditional introductory physics courses. American Journal of Physics, 62, 627-633.
86. VanLehn, K. (2006). The behavior of tutoring systems. International Journal of Artificial Intelligence in Education, 16(3), 227-265.
87. VanLehn, K., Graesser, A.C., Jackson, G.T., Jordan, P., Olney, A., Ros, C.P. (2007). When are tutorial dialogues more effective than reading?Cognitive Science, 31(1), 3-62.
88. VanLehn, K., Jordan, P., Litman, D. (2007). Developing pedagogically effective tutorial dialogue tactics: Experiments and a testbed. In Proceedings of SLaTE Workshop on Speech and Language Technology in Education.
89. van Noord, G. (2004). Error mining for wide-coverage grammar engineering. In Proceedings of the 42nd Meeting of the Association for Computational Linguistics (ACL04) (pp. 446-453). Barcelona, Spain.
90. Ventura, M.J., Franchescetti, D.R., Pennumatsa, P., Graesser, A.C., Jackson, G.T., Hu, X., Cai, Z. (2004). Combining computational models of short essay grading for conceptual physics problems. In Lester, J. C., Vicari, R. M., Paraguaçu, F. (Eds.), Proceedings of the intelligent tutoring systems conference (Vol. 3220, pp. 423-431). Berlin: Springer.
91. Vosniadou, S., & Brewer, W.F. (1987). Theories of knowledge restructuring in development. Review of Educational Research, 57(1), 51-67.
92. Walker, M.A., Kamm, C.A., Litman., D.J. (2000). Towards developing general models of usability with PARADISE. Natural Language Engineering, 6(3).
93. Ward, A., & Litman, D. (2006). Cohesion and learning in a tutorial spoken dialog system. In Proceedings of 19th International FLAIRS (Florida Artificial Intelligence Research Society) Conference.
94. Yeh, A. (2000). More accurate tests for the statistical significance of result differences. In Proceedings of the 18th International Conference on Computational Linguistics (COLING 2000) (pp. 947-953). Association for Computational Linguistics.