Computer-based assessment of student-constructed responses

Behavior Research Methods

Volumn 44, Issue 3, 2012, Pages 608-621

  Magliano, Joseph P ^a   Graesser, Arthur C ^b  

^a NORTHERN ILLINOIS UNIVERSITY   (United States)

^b UNIVERSITY OF MEMPHIS   (United States)

Author keywords

Computer based assessment; Natural language processing; Student constructed responses

Indexed keywords

ADOLESCENT; ADULT; ALGORITHM; ARTICLE; ARTIFICIAL INTELLIGENCE; COMPREHENSION; COMPUTER PROGRAM; EDUCATION; HEALTH CARE PLANNING; HUMAN; HYBRID COMPUTER; LEARNING; METHODOLOGY; NATURAL LANGUAGE PROCESSING; TEACHING;

ADOLESCENT; ALGORITHMS; ARTIFICIAL INTELLIGENCE; COMPREHENSION; COMPUTER-ASSISTED INSTRUCTION; COMPUTERS, HYBRID; EDUCATIONAL MEASUREMENT; HUMANS; LEARNING; NATURAL LANGUAGE PROCESSING; REGIONAL HEALTH PLANNING; SOFTWARE; TEACHING; YOUNG ADULT;

EID: 84865624001     PISSN: 1554351X     EISSN: 15543528     Source Type: Journal    
DOI: 10.3758/s13428-012-0211-3     Document Type: Article

References (74)

1. Allen, J. (1995). Natural language understanding (2nd ed.). Amsterdam: Benjamin Cummngs.
2. Attali, Y., & Burstein, J. (2006). Automated essay scoring with e-rater R V. 2. Journal of Technology, Learning and Assessment, 4, 1-30.
3. Biber, D., Conrad, S., & Reppen, R. (1998). Corpus linguistics: Investigating language structure and use. Cambridge: Cambridge University Press.
4. Bråten, I., & Strømsø, H. I. (2003). A longitudinal think-aloud study of spontaneous strategic processing during the reading of multiple expository texts. Reading and Writing, 16, 195-218.
5. Britt, M. A., Wiemer, K., Millis, K. K., Magliano, J. P., Wallace, P., & Hastings, P. (2012). Understanding and reasoning with text. In P. McCarthy & C. Boonthum (Eds.), Applied natural language processing and content analysis: Identification, investigation, and resolution (pp. 133-154). Hershey, PA: IGI Global.
6. Britt, M. A., Wiemer-Hastings, P., Larson, A. A., & Perfetti, C. A. (2004). Using intelligent feedback to improve sourcing and integration in students' essays. International Journal of Artificial Intelligence in Education, 14, 359-374.
7. Burgess, C., Livesay, K., & Lund, K. (1998). Explorations in context space: Words, sentences, and discourse. Discourse Processes, 25, 211-257.
8. Burstein, J., Marcu, D., & Knight, K. (2003). Finding the WRITE stuff: Automatic identification of discourse structure in student essays. IEEE Intelligent Systems, 18, 32-39.
9. Cai, Z., Graesser, A. C., Forsyth, C., Burkett, C., Millis, K. K., Wallace, P. Butler, H. (2011). Trialog in ARIES: User input assessment in an intelligent tutoring system. In W. Chen & S. Li (Eds.), Proceedings of the 3rd IEEE International Conference on Intelligent Computing and Intelligent Systems (pp. 429-433). Guangzhou: IEEE Press.
10. Calfee, R. (2000). To grade or not to grade. IEEE Intelligent Systems, 15, 35-37.
11. Chi, M. T. H., Bassok, M., Lewis, M. W., Reimann, R., & Glaser, R. (1989). Self-explanation: How students study and use examples in learning to solve problems. Cognitive Science, 13, 145-182.
12. Chi, M. T. H., de Leeuw, N., Chiu, M., & LaVancher, C. (1994). Eliciting self-explanations improves understanding. Cognitive Science, 18, 439-477.
13. Chi, M. T. H., Siler, S. A., & Jeong, H. (2004). Can tutors monitor students' understanding accurately? Cognition and Instruction, 22, 363-387.
14. Christian, P. (1998). Soundex-can it be improved? Computers in Genealogy, 6, 215-221.
15. Coté, N., Goldman, S. R., & Saul, E. U. (1998). Students making sense of informational text: Relations between processing and representation. Discourse Processes, 25, 1-53.
16. D'Mello, S., Dowell, N., & Graesser, A. C. (2011). Does it really matter whether students' contributions are spoken versus typed in an intelligent tutoring system with natural language? Journal of Experimental Psychology. Applied, 17, 1-17.
17. Elliott, S. (2003). IntelliMetric: From here to validity. In M. D. Shermis & J. Burstein (Eds.), Automated essay scoring: A cross-disciplinary perspective (pp. 71-86). Hillsdale, NJ: Erlbaum.
18. Ericsson, P. F., & Haswell, R. (Eds.). (2006). Machine scoring of student essays: Truth and consequences. Logan, UT: Utah State University Press.
19. Foltz, P. W., Gilliam, S., & Kendall, S. (2000). Supporting content-based feedback in on-line writing evaluation with LSA. Interactive Learning Environments, 8, 111-127.
20. Foltz, P. W., Kintsch, W., & Landuaer, T. (1998). The measurement of textual coherence with latent semantic analysis. Discourse Processes, 25, 285-307.
21. Franzke, M., Kintsch, E., Caccamise, D., Johnson, N., & Dooley, S. (2005). Summary street: Computer support for comprehension and writing. Journal of Educational Computing Research, 33, 53-80.
22. Gilliam, S., Magliano, J. P., Millis, K. K., Levinstein, I., & Boonthum, C. (2007). Assessing the format of the presentation of text in developing a Reading Strategy Assessment Tool (R-SAT). Behavior Research Methods, Instruments, & Computers, 39, 199-204.
23. Graesser, A. C., D'Mello, S. K., & Cade, W. (in press). Instruction based on tutoring. In R. E. Mayer & P. A. Alexander (Eds.), Handbook of research on learning and instruction (pp. 408-426). New York: Routledge.
24. Graesser, A. C., D'Mello, S., & Person, N. K. (2009). Metaknowledge in tutoring. In D. Hacker, J. Donlosky, & A. C. Graesser (Eds.), Handbook of metacognition in education (pp. 361-382). New York: Taylor & Francis.
25. Graesser, A. C., Jeon, M., & Dufty, D. (2008). Agent technologies designed to facilitate interactive knowledge construction. Discourse Processes, 45, 298-322.
26. Graesser, A. C., Jeon, M., Yang, Y., & Cai, Z. (2007). Discourse cohesion in text and tutorial dialogue. Information Design Journal, 15, 199-213.
27. Graesser, A. C., Lu, S., Jackson, G. T., Mitchell, H., Ventura, M., Olney, A., & Louwerse, M. M. (2004). AutoTutor: A tutor with dialogue in natural language. Behavioral Research Methods, Instruments, & Computers, 36, 180-193.
28. Graesser, A. C., & McNamara, D. S. (2010). Self-regulated learning in learning environments with pedagogical agents that interact in natural language. Educational Psychologist, 45, 234-244.
29. Graesser, A. C., & McNamara, D. S. (2011). Computational analyses of multilevel discourse comprehension. Topics in Cognitive Sciences, 3, 371-398.
30. Graesser, A. C., & McNamara, D. S. (2012). Automated analysis of essays and open-ended verbal responses. In H. Cooper & A. T. Panter (Eds.), APA handbook of research methods in psychology. Washington, DC: American Psychological Association.
31. Graesser, A. C., McNamara, D. S., & Kulikowich, J. (2011). Coh-Metrix: Providing multilevel analyses of text characteristics. Educational Researcher, 40, 223-234.
32. Graesser, A. C., McNamara, D. S., Louwerse, M. M., & Cai, Z. (2004). Coh-metrix: Analysis of text on cohesion and language. Behavior Research Methods, Instruments, & Computers, 36, 193-202.
33. Graesser, A. C., Ozuru, Y., & Sullins, J. (2009). What is a good question? In M. G. McKeown & L. Kucan (Eds.), Threads of coherence in research on the development of reading ability (pp. 112-141). New York: Guilford.
34. Graesser, A. C., Penumatsa, P., Ventura, M., Cai, Z., & Hu, X. (2007). Using LSA in AutoTutor: Learning through mixed initiative dialogue in natural language. In T. Landauer, D. McNamara, S. Dennis, & W. Kintsch (Eds.), Handbook of latent semantic analysis (pp. 243-262). Mahwah, NJ: Erlbaum.
35. Graesser, A. C., Person, N. K., & Magliano, J. P. (1995). Collaborative dialogue patterns in naturalistic one-to-one tutoring. Applied Cognitive Psychology, 9, 495-522.
36. Graesser, A. C., Singer, M., & Trabasso, T. (1994). Constructing inferences during narrative text comprehension. Psychological Review, 101, 371-395.
37. Jackson, G. T., Guss, R. H., & McNamara, D. S. (2010). Assessing cognitively complex strategy use in an untrained domain. Topics in Cognitive Science, 2, 127-137.
38. Jones, M. N., Kintsch, W., & Mewhort, D. J. K. (2006). High-dimensional semantic space accounts of priming. Journal of Memory and Language, 55, 534-552.
39. Jurafsky, D., & Martin, J. H. (2008). Speech and language processing: An introduction to natural language processing, computational linguistics, and speech recognition. Upper Saddle River, NJ: Prentice-Hall.
40. Kintsch, E., Caccamise, D., Franzke, M., Johnson, N., & Dooley, S. (2007). Summary Street ®: Computer-guided summary writing. In T. K. Landauer, D. M. McNamara, S. Dennis, & W. Kintsch (Eds.), Latent semantic analysis (pp. 263-277). Mahwah, NJ: Erlbaum.
41. Kintsch, W. (1998). Comprehension: A paradigm for cognition. New York: Cambridge University Press.
42. Landauer, T. K., & Dumais, S. T. (1997). A solution to Plato's problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104, 211-240.
43. Landauer, T. K., Laham, D., & Foltz, P. W. (2003). Automatic essay assessment. Assessment in Education: Principles, Policy & Practice, 10, 295-308.
44. Landauer, T. K., McNamara, D. S., Dennis, S., & Kintsch, W. (Eds.). (2007). Handbook of latent semantic analysis. Mahwah, NJ: Erlbaum.
45. Leacock, C., & Chodorow, M. (2003). C-rater: Automated scoring of short-answer questions. Computers and the Humanities, 37, 389-405.
46. Lehmann, W. P. (Ed.). (1985) Natural language processing. Special issue of Computers and the Humanities, 19.
47. Litman, D. J., Rose, C. P., Forbes-Riley, K., VanLehn, K., Bhembe, D., & Silliman, S. (2006). Spoken versus typed human and computer dialogue tutoring. International Journal of Artificial Intelligence in Education, 16, 145-170.
48. Magliano, J. P. (1999). Revealing inference processes during text comprehension. In S. R. Goldman, A. C. Graesser, & P. van den Broek (Eds.), Narrative comprehension, causality, and coherence: Essays in honor of Tom Trabasso (pp. 55-75). Mahwah, NJ: ErIbaum.
49. Magliano, J. P., & Millis, K. K. (2003). Assessing reading skill with a think-aloud procedure. Cognition and Instruction, 21, 251-283.
50. Magliano, J. P., Millis, K. K., the RSAT Development Team, Levinstein, I., & Boonthum, C. (2011). Assessing comprehension during reading with the Reading Strategy Assessment Tool (RSAT). Metacognition and Learning, 6, 131-154.
51. Magliano, J. P., Millis, K. K., Ozuru, Y., & McNamara, D. S. (2007). A multidimensional framework to evaluate assessment tools. In D. S. McNamara (Ed.), Reading comprehension strategies: Theories, interventions, and technologies (pp. 107-136). Mahwah, NJ: Erlbaum.
52. McCarthy, P., & Boonthum-Denecke, C. (Eds.). (2012). Applied natural language processing and content analysis: Identification, investigation, and resolution. Hershey, PA: IGI Global.
53. McNamara, D. S., Boonthum, C., Levinstein, I., & Millis, K. K. (2007a). Evaluating self-explanations with word-based and LSA algorithms. In T. Landauer, D. S. McNamara, S. Dennis, & W. Kintsch (Eds.), The handbook of latent semantic analysis (pp. 227-241). Mahwah, NJ: Erlbaum.
54. McNamara, D. S., Crossley, S. A., & McCarthy, P. M. (2010a). The linguistic features of quality writing. Written Communication, 27, 57-86.
55. McNamara, D. S., Levinstein, I. B., & Boonthum, C. (2004). iSTART: Interactive strategy training for active reading and thinking. Behavioral Research Methods, Instruments, & Computers, 36, 222-233.
56. McNamara, D. S., Louwerse, M. M., McCarthy, P. M., & Graesser, A. C. (2010b). Coh-metrix: Capturing linguistic features of cohesion. Discourse Processes, 47, 292-330.
57. McNamara, D. S., & Magliano, J. P. (2009). Towards a comprehensive model of comprehension. In B. Ross (Ed.), The psychology of learning and motivation (Vol. 51, pp. 297-384). New York: Elsevier Science.
58. McNamara, D. S., O'Reilly, T., Rowe, M., Boonthum, C., & Levinstein, I. B. (2007b). iSTART: A web-based tutor that teaches self-explanation and metacognitive reading strategies. In D. S. McNamara (Ed.), Reading comprehension strategies: Theories, interventions, and technologies (pp. 397-421). Mahwah, NJ: Erlbaum.
59. McNamara, D. S., Raine, R., Roscoe, R., Crossley, S., Jackson, G. T., Dai, J Graesser, A. C. (2012). The Writing-Pal: Natural language algorithms to support intelligent tutoring on writing strategies. In P. M. McCarthy & C. Boonthum (Eds.), Applied natural language processing and content analysis: Identification, investigation, and resolution (pp. 298-311). Hershey, PA: IGI Global.
60. Millis, K. K., Forsyth, C., Butler, H., Wallace, P., Graesser, A., & Halpern, D. (in press) Operation ARIES! A serious game for teaching scientific inquiry. In M. Ma, A. Oikonomou, & J. Lakhmi (Eds.), Serious games and edutainment applications. London: Springer.
61. Millis, K. K., Magliano, J. P., Todaro, S., & McNamara, D. S. (2007). Assessing and improving comprehension with latent semantic analysis. In T. Landauer, D. S., McNamara, S. Dennis, & W. Kintsch (Eds.), The Handbook of Latent Semantic Analysis (pp. 207-225). Mahwah, NJ: Erlbaum.
62. Mislevy, R. J. (2007). Validity by design. Educational Researcher, 36, 463-469.
63. Muñoz, B., Magliano, J. P., Sheridan, R., & McNamara, D. S. (2006). Typing versus thinking aloud when reading: Implications for computer-based assessment and training tools. Behavior Research Methods, Instruments, & Computers, 38, 211-217.
64. Pellegrino, J. W., & Chudowsky, N. (2003). The foundations of assessment. Interdisciplinary Research and Perspectives, 1, 103-148.
65. Pennebaker, J. W., Booth, R. J., & Francis, M. E. (2007). Linguistic inquiry and word count: LIWC 2007 [Computer program]. Austin, TX: LIWC. net. Retrieved from www. liwc. net.
66. Person, N. K., Graesser, A. C., & the Tutoring Research Group (TRG). (2002). Human or computer? AutoTutor in a bystander Turing test. In S. A. Cerri, G. Gouarderes, & F. Paraguaçu (Eds.), Intelligent tutoring systems 2002 (pp. 821-830). Berlin: Springer.
67. Rudner, L. M., Garcia, V., & Welch, C. (2006). An evaluation of the IntelliMetric essay scoring system. Journal of Technology, Learning and Assessment, 4, 1-22.
68. Rus, V., McCarthy, P. M., McNamara, D. S., & Graesser, A. C. (2008). A study of textual entailment. International Journal on Artificial Intelligence Tools, 17, 659-685.
69. Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78, 153-189.
70. Shermis, M. D., & Burstein, J. (2003). Automated essay scoring: A cross disciplinary perspective. Mahwah, NJ: Erlbaum.
71. Shermis, M. D., Burstein, J., Higgins, D., & Zechner, K. (2010). Automated essay scoring: Writing assessment and instruction. In E. Baker, B. McGaw, & N. S. Petersen (Eds.), International encyclopedia of education (3rd ed., pp. 20-26). Oxford: Elsevier.
72. Trabasso, T., & Magliano, J. P. (1996). Conscious understanding during text comprehension. Discourse Processes, 21, 255-288.
73. VanderVeen, A., Huff, K., Gierl, M., McNamara, D. S., Louwerse, M., & Graesser, A. C. (2007). Developing and validating instructionally relevant reading competency profiles measured by the critical reading sections of the SAT. In D. S. McNamara (Ed.), Theories of text comprehension: The importance of reading strategies to theoretical foundations of reading comprehension (pp. 137-172). Mahwah, NJ: Erlbaum.
74. VanLehn, K., Graesser, A. C., Jackson, G. T., Jordan, P., Olney, A., & Rose, C. P. (2007). When are tutorial dialogues more effective than reading? Cognitive Science, 31, 3-62.