Automated scoring of constructed-response science items: Prospects and obstacles

Educational Measurement: Issues and Practice

Volumn 33, Issue 2, 2014, Pages 19-28

  Liu, Ou Lydia ^a   Brew, Chris ^b   Blackmore, John ^a   Gerard, Libby ^c   Madhok, Jacquie ^c   Linn, Marcia C ^c  

^a EDUCATIONAL TESTING SERVICE   (United States)

^b NUANCE COMMUNICATIONS   (United States)

^c UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Automated scoring; C rater ; Constructed response items; Science assessment

Indexed keywords

EID: 84903310579     PISSN: 07311745     EISSN: 17453992     Source Type: Journal    
DOI: 10.1111/emip.12028     Document Type: Article

References (49)

1. Attali, Y., & Powers, D. (2008). Effect of immediate feedback and revision on psychometric properties of open-ended GRE® subject test items. GRE Board Research Rep. No. 04-05; ETS RR-08-21. Princeton, NJ: Educational Testing Service.
2. Attali, Y., Powers, D., Freedman, M., Harrison, M., & Obetz, S. (2008). Automated scoring of short-answer open-ended GRE subject test items.ETS GRE Board Research Report No. 04-02. Princeton, NJ: Educational Testing Service.
3. Azevedo, R., & Brenard, R. M. (1995). A meta-analysis of the effects of feedback in computer-based instruction. Journal of Educational Computing Research, 13, 11-127.
4. Bennett, R. E., & Sebrechts, M. M. (1996). The accuracy of expert-system diagnoses of mathematical problem solutions. Applied Measurement in Education, 9, 133-150.
5. Bernstein, J., Van Moere, A., & Cheng, J. (2010). Validating automated speaking tests. Language Testing, 27, 355-377.
6. Burstein, J., & Marcu, D. (2002). Automated evaluation of discourse structure in student essays. In M. D. Shermis & J. Burstein (Eds.), Automated essay scoring: A cross-disciplinary perspective (pp. 200-219). Mahwah, NJ: Lawrence Erlbaum.
7. Burstein, J., Marcu, D., & Knight, K. (2003). Finding the WRITE stuff: Automatic identification of discourse structure in student essays. IEEE Intelligent Systems: Special Issue on Advances in Natural Language Processing, 18(1), 32-39.
8. Butler, D. L., & Winne, P. H. (1995). Feedback and self-regulated learning: A theoretical synthesis. Review of Educational Research, 65, 245-281.
9. Cohen, J. (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70, 213-220.
10. Dzikovska, M.O., Moore, J. D., Steinhauser, N., Campbell, G., Farrow, E., & Calloway, C. (2010). Beetle II: A system for tutoring and computational linguistics experimentation. In J. Hajic, S. Carberry, & S. Clark (Eds.), ACL 2010: Proceedings of the 48th annual meeting of the Association for Computational Linguistics (pp. 13-18). Stroudsburg, PA: Association for Computational Linguistics.
11. Dzikovska, M. O., Nielsen, R. D., & Brew, C. (2012). Towards effective tutorial feedback for explanation questions: A dataset and baselines. In T. Chandra (Ed.), Proceedings of the 2012 conference of the North American Chapter of the Association for Computational Linguistics: Human language technologies (pp. 200-201). Stroudsburg, PA: Association for Computational Linguistics.
12. Fleiss, J. L., & Cohen, J. (1973). The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement, 33, 613-619.
13. Foltz, P. W., Laham, D., & Landauer, T. K. (1999). The intelligent essay assessor: Applications to educational technology. Interactive Multimedia Education Journal of Computer Enhanced Learning, 1(2). Retrieved from http://imej.wfu.edu/articles/1999/2/04/
14. Graesser, A. C. (2011). Learning, thinking, and emoting with discourse teachers technologies. American Psychologist, 66, 743-757.
15. Graesser, A. C., Rus, V., D'Mello, S. K., & Jackson, G. T. (2008). AutoTutor: Learning through natural language dialogue that adapts to the cognitive and affective states of the learner. In D. H. Robinson & G. Schraw (Eds.), Recent innovations in educational technology that facilitate student learning (pp. 95-125). Charlotte, NC: Information Age.
16. Hattie, J., & Temperley, H. (2007). The power of feedback. Review of Educational Research, 77, 81-112.
17. Higgins, D., Zechner, K., Xi, X., & Williamson, D. (2011). A three-stage approach to the automated scoring of spontaneous spoken responses. Computer Speech and Language, 25, 282-306.
18. Kintsch, E., Steinhart, D., Stahl, G., Matthews, C., & Lamb, R. (2000). Developing summarization skills through the use of LSA-based feedback. Interactive Learning Environments, 8, 87-109.
19. Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159-174.
20. Lane, S. (2004). Validity of high-stakes assessment: Are students engaged in complex thinking? Educational Measurement: Issues and Practice, 23(3), 6-14.
21. Leacock, C., & Chodorow, M. (2003). C-rater: Automated scoring of short-answer questions. Computers and the Humanities, 37, 389-405.
22. Lee, H. S., & Liu, O. L. (2010). Assessing learning progression of energy concepts across middle school grades: The knowledge integration perspective. Science Education, 94, 665-688.
23. Lee, H. S., Varma, K., Linn, M. C., & Liu, O. L. (2010). Impact of visualization-based inquiry science experience on classroom learning. Journal of Research in Science Teaching, 47, 71-90.
24. Linn, M. C., & Eylon, B.-S. (2011). Science learning and instruction: Taking advantage of technology to promote knowledge integration. New York, NY: Routledge.
25. Linn, M. C., & Liu, O. L. (2013, April). NCME invited session on big data: New opportunities for measurement and data analysis. Paper accepted for presentation at the 2013 conference of the National Council on Measurement in Education, San Francisco, CA.
26. Liu, O. L., Lee, H. S., Hoftstetter, C., & Linn, M. C. (2008). Assessing knowledge integration in science: Construct, measures, and evidence. Educational Assessment, 13, 33-55.
27. Liu, O. L., Lee, H. S., & Linn, M. C. (2010). Evaluating inquiry-based science modules using a hierarchical linear model. Educational Assessment, 15(2), 69-86.
28. Liu, O. L., Lee, H. S., & Linn, M. C. (2011). A comparison among multiple-choice, constructed-response and explanation multiple-choice items. Educational Assessment, 16, 164-184.
29. Matthews, K., Janicki, T., He, L., & Patterson, L. (2012). Implementation of an automated grading system with an adaptive learning component to affect student feedback and response time. Journal of Information Systems Education, 23(1), 71-83.
30. Meyer, B. J. F., Wijekumar, K., Middlemiss, W., Higley, K., Lei, P., Meier, C., & Spielvogel, J. (2010). Web-based tutoring of the structure strategy with or without elaborated feedback or choice for fifth- and seventh-grade readers. Reading Research Quarterly, 45, 62-92.
31. Mitchell, T., Russell, T., Broomhead, P., & Aldridge, N. (2002). Towards robust computerized marking of free-text responses. In Proceedings of the Sixth International Computer Assisted Assessment Conference (pp. 233-249). Loughborough, UK: Loughborough University.
32. Nielsen, R. D., Ward, W., & Martin, J. H. (2008). Classification errors in a domain-independent assessment system. In Proceedings of the Third Workshop on Innovative Use of NLP for Building Educational Applications (pp. 10-18). Stroudsburg, PA: Association for Computational Linguistics.
33. National Research Council. (2012). A framework for K-12 science education: Practices, crosscutting concepts, and core ideas. Washington, DC: National Academies Press.
34. National Council of Teachers of English. (2008). Statement on class size and teacher workload: Secondary. Retrieved November 14, 2013, from http://www.ncte.org/positions/statements/classsizesecondary
35. Pulman, S., & Sukkarieh, J. (2005). Automatic short answer marking. In The second workshop on building educational applications using NLP: Proceedings (pp. 9-16). New Brunswick, NJ: Association for Computational Linguistics. Retrieved November 14, 2013, from http://www.comlab.oxford.ac.uk/people/publications/date/Stephen.Pulman.html
36. Quellmalz, E., & Pelligrino, J. (2009). Technology and learning. Science, 323, 75-79.
37. Sandene, B., Horkay, N., Bennett, R., Braswell, J., & Oranje, A. (2005). Online assessment in mathematics and writing: Reports from the NAEP Technology-Based Assessment Project, Research and Development series. NCES 2005-457. Washington, DC: U.S. Government Printing Office.
38. Sargeant, J., Wood, M. M., & Anderson, S. M. (2004, July). A human-computer collaborative approach to the marking of free text answers. Paper presented at the 8th Annual Computer-Assisted Assessment Conference, (pp. 361-370), Loughborough, UK.
39. Shepard, L. A. (2000). The role of assessment in a learning culture. Educational Researcher, 29(7), 4-14.
40. Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78, 153-189.
41. Sukkarieh, J. Z., & Blackmore, J. (2009, May). c-Rater: Automatic content scoring for short constructed responses. In H. C. Lane & H. W. Guesgen (Eds.), Proceedings of the Twenty-Second International Florida Artificial Intelligence Research Society Conference (pp. 290-295). Menlo Park, CA: AAAI Press.
42. Sukkarieh, J. Z., & Bolge, E. (2010). Building a textual entailment suite for evaluating content scoring technologies. In Proceedings of the Seventh Conference on International Language Resources and Evaluation (pp. 3149-3156). Paris, France: European Language Resources Association.
43. Sukkarieh, J. Z., & Pulman, S. G. (2005). Information extraction and machine learning: Auto-marking short free-text responses for science questions. In Proceedings of the Twelfth International Conference on Artificial Intelligence in Education (pp. 629-637). Amsterdam, The Netherlands: IOS Press.
44. Sukkarieh, J. Z., Pulman, S. G., & Raikes, N. (2003, October). Auto-marking: Using computational linguistics to score short, free text responses. Paper presented at the Twenty-Ninth Annual Conference of the International Association for Educational Assessment (IAEA), Manchester, UK.
45. VanLehn, K., Graesser, A. C., Jackson, G. T., Jordan, P., Olney, A., & Rose, C. P. (2007). When are tutorial dialogues more effective than reading? Cognitive Science, 31, 3-62.
46. Wainer, H., & Thissen, D. (1993). Combining multiple-choice and constructed response test scores: Toward a Marxist theory of test construction. Applied Measurement in Education, 6, 103-118.
47. Wang, H.-C., Chang, C.-Y., & Li, T.-Y. (2005). Automated scoring for creative problem-solving ability with ideation-explanation modeling. In Proceedings of the Thirteenth International Conference on Computers in Education (pp. 522-529). Singapore: IOS Press.
48. Wiemer-Hastings, P., Arnott, E., & Allbritton, D. (2005). Initial results and mixed directions for research methods tutor. In Supplementary Proceedings of the Twelfth International Conference on Artificial Intelligence in Education. Amsterdam, The Netherlands: IOS Press.
49. Williamson, D., Xi, X., & Breyer, J. (2012). A framework for evaluation and use of automated scoring. Educational Measurement: Issues and Practice, 31(1), 2-13.