Using latent semantic analysis to grade brief summaries: Some proposals

International Journal of Continuing Engineering Education and Life-Long Learning

Volumn 21, Issue 2-3, 2011, Pages 192-209

  Olmos, Ricardo ^a   León, José A ^a   Escudero, Inmaculada ^b   Jorge Botana, Guillermo ^a  

^a UNIVERSIDAD AUTÓNOMA DE MADRID   (Spain)

^b UNIVERSIDAD NACIONAL DE EDUCACIÓN A DISTANCIA   (Spain)

Author keywords

Assessment summaries; Automated scoring of summaries; Brief summary; Expository text; Latent semantic analysis; LSA; Narrative text; ROUGE N; Vector length

Indexed keywords

BEHAVIORAL RESEARCH; SEMANTICS;

ASSESSMENT SUMMARIES; AUTOMATED SCORING OF SUMMARIES; BRIEF SUMMARY; EXPOSITORY TEXT; LATENT SEMANTIC ANALYSIS; NARRATIVE TEXT;

QUALITY CONTROL;

EID: 79957824836     PISSN: 15604624     EISSN: None     Source Type: Journal    
DOI: 10.1504/IJCEELL.2011.040198     Document Type: Article

References (45)

1. Bransford, J.D., Brown, A.L. and Cocking, R.R. (2000) How People Learn: Brain, Mind, Experience, and School, National Research Council Commission on Behavioral and Social Sciences and Education, National Academy Press, Washington, DC.
2. Brown, A.L., Bransford, J.D., Ferrara, R.A. and Campione, J.C. (1983) 'Learning, remembering, and understanding', in Flavell, J. and Markman, E.M. (Eds.): Handbook of Child Psychology Cognitive Development, 4th edition, Vol. 3, pp.515-629, Wiley, New York.
3. Burstein, J. (2003) T'he e-rater® scoring engine: Automated essay scoring with natural language processing', in anonymous (Eds.): Automated Essay Scoring: A Cross-Disciplinary Perspective, Lawrence Erlbaum Associates, Inc., Hillsdale, NJ.
4. Burstein, J., Chodorow, M. and Leacock, C. (2003) 'CriterionSM: online essay evaluation: an application for automated evaluation of student essays', in Proceedings of the Fifteenth Annual Conference on Innovative Applications of Artificial Intelligence, Acapulco, Mexico.
5. Deerwester, S., Dumais, S.T., Furnas, G.W., Landauer, T.K. and Harshman, R. (1990) 'Indexing by latent semantic analysis', Journal of the American Society for Information Science', Vol. 41, pp.391-407.
6. Denhière, G., Lemaire, B., Bellissens, C. and Jhean-Larose, S. (2007) 'A semantic space modeling children's semantic memory', in Landauer, T.K., McNamara, D., Dennis, S. and Kintsch, W. (Eds.): The Handbook of Latent Semantic Analysis, pp.143-167, Erlbaum, Mahwah, NJ.
7. Dessus, P. and Lemaire, B. (2002) 'Using production to assess learning: an ILE that fosters selfregulated learning', International Conference on Intelligent Tutoring Systems (ITS'2002), Springer Verlag, Berlin, LNCS 2363, pp.772-781.
8. Dikli, S. (2006) 'An overview of automated scoring of essays', Journal of Technology, Learning, and Assessment, Vol. 5, No. 1, pp.3-35, available at http://www.jtla.org.
9. Elliott, S. (2003) 'Intellimetric: from here to validity', in Shermis, M. and Burstein, J. (Eds.): Automated Essay Scoring: A Cross-Disciplinary Perspective, Lawrence Erlbaum Associates, Hillsdale, NJ.
10. Foltz, P. W. (1996) 'Latent semantic analysis for text-based research', Behavior Research Methods, Instruments and Computers, Vol. 28, No. 2, pp.197-202. (Pubitemid 126454813)
11. Foltz, P.W., Gilliam, S. and Kendall, S. (2000) 'Supporting content-based feedback in on-line writing evaluation with LSA', Interactive Learning Environments, Vol. 8, pp.111-128.
12. Foltz, P.W., Laham, D. and Landauer, T.K. (1999) 'The intelligent essay assessor: applications to educational technology', Interactive Multimedia Electronic Journal of Computer-Enhanced Learning, Vol. 1, available at http://knowledge-technologies.com (accessed on 29 June 2004).
13. Graesser, A.C., Chipman, P., Haynes, B.C. and Olney, A. (2005) 'AutoTutor: an intelligent tutoring system with mixed-initiative dialogue', IEEE Transactions in Education, Vol. 48, pp.612-618. (Pubitemid 41723893)
14. Graesser, A.C., Lu, S., Jackson, G.T., Mitchell, H, Ventura, M., Olney, A. and Lowerse, M.M. (2004) 'AutoTutor: a tutor with dialogue in natural language', Behaviour Research Methods, Instruments, and Computers, Vol. 36, pp.180-193.
15. Graesser, A.C., Penumatsa, P., Ventura, M., Cai, Z. and Hu, X. (2007) 'Using LSA in AutoTutor: learning through mixed-initiative dialogue in natural language', in Landauer, T.K., McNamara, D., Dennis, S. and Kintsch, W. (Eds.): The Handbook of Latent Semantic Analysis, pp.243-262, Erlbaum, Mahwah, NJ.
16. Higgins, D., Burstein, J., Marcu, D. and Gentile, C. (2004) 'Evaluating multiple aspects of coherence in student essays', in Proceedings of the Annual Meeting of the North American Chapter of the Association for Computational Linguistics, Boston, MA.
17. Hu, X., Cai, Z., Wiemer-Hastings, Graesser, A.C. and McNamara, D.S. (2007) 'Strengths, limitations, and extensions of LSA', in Landauer, T.K., McNamara, D., Dennis, S. and Kintsch, W. (Eds.): The Handbook of Latent Semantic Analysis, pp.401-426, Erlbaum, Mahwah, NJ.
18. Jorge-Botana, G., León, J.A., Olmos, R. and Escudero, I. (2010) 'Latent semantic analysis parameters for essay evaluation using small-scale corpora', Journal of Quantitative Linguistics, Vol. 17, No.1, pp.1-29.
19. Jorge-Botana, G., Olmos, R. and León, J.A. (2009) 'Using LSA and the predication algorithm to improve extraction of meanings from a diagnostic corpus', Spanish Journal of Psychology. Vol. 12, No. 2, pp.424-440.
20. Kintsch, E., Steinhart, D., Stahl, G. and LSA Research Group (2000) 'Developing summarization skills through the use of LSA-based feedback', Interactive Learning Environments, Vol. 8, No. 2, pp.87-109.
21. Kintsch, W. (2000) 'Metaphor comprehension: a computational theory', Psychonomic Bulletin & Review, Vol. 7, No. 2, pp.257-266.
22. Kintsch, W. (2001) 'Predication' Cognitive Science, Vol. 25, pp.173-202.
23. Kintsch, W. (2002) 'On the notions of theme and topic in psychological process models of text comprehension', in Louwerse, M. and van Peer, W. (Eds.): Thematics: Interdisciplinary Studies, pp.157-170, Benjamins, Amsterdam.
24. Kintsch, W. (2007) 'Meaning in context', in Landauer, T.K., McNamara, D., Dennis, S. and Kintsch, W. (Eds.): The Handbook of Latent Semantic Analysis, pp.89-106, Erlbaum, Mahwah, NJ.
25. Kintsch, W. (2008) 'Symbol systems and perceptual representations', in de Vega, M., Glenberg, A. and Graesser, A. (Eds.): Symbols, Embodiment and Meaning, pp.145-164, University Press, Oxford.
26. Landauer, T.K. (1998) 'Learning and representing verbal meaning: the latent semantic analysis theory', Current Directions in Psychological Science, Vol. 7, pp.161-164. (Pubitemid 128188367)
27. Landauer, T.K. and Dumais, S.T. (1997) 'A solution to Plato's problem: the latent semantic analysis theory of the acquisition, induction and representation of knowledge', Psychological Review, Vol. 104, pp.211-240.
28. Landauer, T.K., Foltz, P.W. and Laham, D. (1998a) 'Introduction to latent semantic analysis', Discourse Processes, Vol. 25, pp.259-284.
29. Landauer, T.K., Laham, D. and Foltz, P.W. (1998b) 'Computer-based grading of the conceptual content of essays', unpublished manuscript.
30. Larkey, L. (1998) 'automatic essay grading using text categorization techniques' in Proceedings of the 21st ACM-SIGIR Conference on Research and Development in Information Retrieval, Melbourne, Australia, pp.90-95.
31. León, J.A., Olmos, R., Escudero, I., Cañas, J.J. and Salmerón, L. (2006) 'assessing short summaries with human judgments procedure and latent semantic analysis in narrative and expository texts', Behavior Research Methods, Instruments and Computers, Vol. 38, No. 4, pp.616-627.
32. Lin, C.Y. (2004) 'Looking for a few good metrics: ROUGE and its evaluation', in Proceedings of NTCIR Workshop 2004, Tokyo, Japan.
33. Mani, I. (2001) Automatic Summarization, John Benjamins Publishing Co.
34. McNamara, D., Boonthum, C., Levinstein, I. and Millis, K.K. (2007) 'Evaluating self-explanations in iSTART: comparing word-based and LSA algorithms', in Landauer, T.K., McNamara, D., Dennis, S. and Kintsch, W. (Eds.): The Handbook of Latent Semantic Analysis, pp.227-241, Erlbaum, Mahwah, NJ.
35. McNamara, D.S., Levinstein, I.B. and Boonthum, C. (2004) 'iSTART: interactive strategy training for active reading and thinking', Behaviour Research Methods, Instruments, and Computers, Vol. 36, pp.222-233. (Pubitemid 39196120)
36. Millis, K.K., Magiano, J.P., Wiemer-Hastings, K., Todaro, S. and McNamara, D. (2007) 'Assesing and improving comprehension with latent semantic analysis', in Landauer, T.K., McNamara, D., Dennis, S. and Kintsch, W. (Eds.): The Handbook of Latent Semantic Analysis, pp.207-225, Erlbaum, Mahwah, NJ.
37. Olmos, R., León, J.A., Jorge-Botana, G. and Escudero, I. (2009) 'New algorithms assessing short summaries in expository texts using latent semantic analysis', Behaviour Research Methods, Instruments, and Computers, Vol. 41, pp.944-950.
38. Over, P. and Yen. J. (2003) 'An introduction to DUC 2003 - intrinsic evaluation of generic news text summarization systems'.
39. Palincsar, A.S. and Brown, A.L. (1984) 'Reciprocal teaching of comprehension-fostering and comprehension-monitoring activities', Cognition & Instruction, Vol. 1, pp.117-175.
40. Rehder, B., Schreiner, M.E., Wolfe, B.W., Laham, D., Landauer, T.K. and Kintsch, W. (1998) 'Using latent semantic analysis to assess knowledge: some technical considerations', Discourse Processes, Vol. 25, pp.337-354.
41. Sherrard, C. (1985) 'The psychology of summary writing: applying text linguistics', Journal of Technical Writing and Communication, Vol. 15, No. 3, pp.247-258.
42. Sherrard, C. (1989) 'Teaching students to summarize: applying text linguistics', System, Vol. 17, No. 1, pp.1-11.
43. Steyvers, M. and Griffiths, T. (2007) 'Probabilistic topic models', in Landauer, T.K., McNamara, D., Dennis, S. and Kintsch, W. (Eds.): The Handbook of Latent Semantic Analysis, pp.427-448, Erlbaum, Mahwah, NJ.
44. Van Dijk, T.A. and Kintsch, W. (1983) Strategies of Discourse Comprehension, Academic Press, New York.
45. Wade-Stein, D. and Kintsch, E. (2004) 'Summary street: interactive computer support for writing.', Cognition and Instruction, Vol. 22, No. 3, pp.333-362. (Pubitemid 39203188)