Validation of automated scoring of science assessments

Journal of Research in Science Teaching

Volumn 53, Issue 2, 2016, Pages 215-233

  Liu, Ou Lydia ^a   Rios, Joseph A ^a   Heilman, Michael ^a   Gerard, Libby ^b   Linn, Marcia C ^b  

^a EDUCATIONAL TESTING SERVICE   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

automated scoring; c rater ML; science assessment

Indexed keywords

EDUCATION;

AUTOMATED SCORING; EDUCATIONAL TESTING; REFLECTIVE EXPERIENCE; SCIENCE ASSESSMENTS; SCIENCE INQUIRIES;

AUTOMATION;

EID: 84954103223     PISSN: 00224308     EISSN: 10982736     Source Type: Journal    
DOI: 10.1002/tea.21299     Document Type: Article

References (45)

1. Attali, Y., &, Powers, D., (2008). Effect of immediate feedback and revision on psychometric properties of open-ended GRE R subject test items (GRE Board Research Rep. No. 04-05; ETS RR-08-21). Princeton, NJ: Educational Testing Service.
2. Beggrow, E. P., Ha, M., Nehm, R. H., Pearl, D., &, Boone, W. J., (2014). Assessing scientific practices using machine-learning methods: How closely do they match clinical interview performance ?. Journal of Science Education and Technology, 23, 160-182.
3. Bennett, R. E., &, Sebrechts, M. M., (1996). The accuracy of expert-system diagnoses of mathematical problem solutions. Applied Measurement in Education, 9, 133-150.
4. Bridgeman, B., Trapani, C., &, Attali, Y., (2012). Comparison of human and machine scoring of essays: Differences by gender, ethnicity, and country. Applied Measurement in Education, 25, 27-40.
5. Burstein, J., &, Chodorow, M., (1999). Automated essay scoring for nonnative English speakers (PDF). In Proceedings of the ACL99 Workshop on Computer-Mediated Language Assessment and Evaluation of Natural Language Processing. College Park, MD.
6. Burstein, J., &, Marcu, D., (2002). Automated evaluation of discourse structure in student essays. In M. D. Shermis & J. Burstein (Eds.), Automated essay scoring: A cross-disciplinary perspective (pp. 200-219). Mahwah, NJ: Lawrence Erlbaum.
7. Bernstein, J., Van Moere, A., &, Cheng, J., (2010). Validating automated speaking tests. Language Testing, 27, 355-377.
8. Chodorow, M., &, Burstein, J., (2004). Beyond essay length: Evaluating e-rater's performance on TOEFL essays (TOEFL Research Report No. RR-73; ETS RR-04-04). Princeton, NJ: Educational Testing Service.
9. Cohen, J., (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70, 213-220.
10. Cohen, J., (1988). Statistical power analysis for the behavioral sciences. 2nd ed. Hillsdale, NJ: Erlbaum.
11. Dzikovska, M., Moore, J. D., Steinhauser, N., Campbell, G., Farrow, E., &, Callway, C. B,. (2010). Beetle II: A system for tutoring and computational linguistics experimentation. Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (pp. 13-18). Uppsala, Sweden: Association for Computational Linguistics.
12. Dzikovska, M. O., Nielsen, R. D., Brew, C., Leacock, C., Giampicollo, D., Bentivogli, L.,..., Dang, H. T., (2013). SemEval-2013 Task 7: The joint student response analysis and 8th recognizing textual entailment challenge. Proceedings of the Seventh International Workshop on Semantic Evaluation: Vol. 2. Second Joint Conference on Lexical and Computational Semantics (pp. 263-274). Atlanta, GA: Association for Computational Linguistics.
13. Fleiss, J. L., &, Cohen, J., (1973). The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement, 33, 613-619.
14. Foltz, P. W., Laham, D., &, Landauer, T. K., (1999). The intelligent essay assessor: Applications to educational technology. Interactive Multimedia Education Journal of Computer Enhanced Learning, 1 (2). Retrieved from http://imej.wfu.edu/articles/1999/2/04/
15. Gibbs, G., &, Simpson, C., (2004). Conditions under which assessment supports students' learning. Learning and Teaching in Higher Education, 1, 3-31.
16. Ha, M., Nehm, R. H., Urban-Lurain, M., &, Merrill, J. E., (2011). Applying computerized-scoring models of written biological explanations across courses and colleges: Prospects and limitations. CBE-Life Sciences Education, 10, 379-393.
17. Haudek, K. C., Prevost, L. B., Moscarella, R. A., Merrill, J., &, Urban-Lurain, M., (2012). What are they thinking? Automated analysis of student writing about acid-base chemistry in introductory biology. CBE-Life Sciences Education, 11 (3), 283-293.
18. Heilman, M., &, Madnani, N., (2013). ETS: Domain adaptation and stacking for short answer scoring. Proceedings of the 2nd joint conference on lexical and computational semantics: Vol. 2. Seventh International Workshop on Semantic Evaluation (pp. 275-279). Atlanta, GA: Association for Computational Linguistics.
19. Higgins, D., Zechner, K., Xi, X., &, Williamson, D., (2011). A three-stage approach to the automated scoring of spontaneous spoken responses. Computer Speech and Language, 25, 282-306.
20. Kuo, C. Y., &, Wu, H. K., (2013). Toward an integrated model for designing assessment systems: An analysis of the current status of computer-based assessments in science. Computer & Education, 68, 388-403.
21. Landis, J. R., &, Koch, G. G., (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159-174.
22. Leacock, C., &, Chodorow, M., (2003). C-rater: Automated scoring of short-answer questions. Computers and the Humanities, 37, 389-405.
23. Lee, H. S., Liu, O. L., &, Linn, M. C., (2011). Validating measurement of knowledge integration in science using multiple-choice and explanation items. Applied Measurement in Education, 24, 115-136.
24. Linn, M. C., &, Eylon, B- S., (2006). Science education: Integrating views of learning and instruction. In P. A. Alexander & P. H. Winne (Eds.), Handbook of educational psychology (pp. 511-544). Mahwah, NJ: Lawrence Erlbaum Associates.
25. Linn, M. C., Gerard, L., Ryoo, K., McElhaney, K., Liu, O. L., &, Rafferty, A. N., (2014). Computer-guided inquiry to improve science learning. Science, 344 (6180), 155-156.
26. Liu, O. L., Brew, C., Blackmore, J., Gerard, L., Madhok, J., &, Linn, M. C., (2014). Automated scoring of constructed-response science items: Prospects and obstacles. Educational Measurement: Issues and Practice, 33, 19-28.
27. Liu, O. L., Lee, H. S., Hoftstetter, C., &, Linn, M. C., (2008). Assessing knowledge integration in science: Construct, measures, and evidence. Educational Assessment, 13, 33-55.
28. Liu, O. L., Lee, H. S., &, Linn, M. C., (2011). Measuring knowledge integration: Validation of four-year assessments. Journal of Research in Science Teaching, 48, 1079-1107.
29. Liu, O. L., Ryoo, K., Sato, E., Svihla, V., &, Linn, M. C., (2015). Designing assessment to measure cumulative learning of energy topics. International Journal of Science Education. DOI: 10.1080/09500693.2015.1016470
30. Mayfield, E., &, Rosé, C., (2010). An interactive tool for supporting error analysis for text mining. Proceedings of the demonstration session at the International Conference of the North American Association for Computational Linguistics (pp. 25-28). Los Angeles, CA: Association for Computational Linguistics.
31. Mitchell, T., Russell, T., Broomhead, P., &, Aldridge, N., (2002). Towards robust computerized marking of free-text responses. Proceedings of the International Computer Assisted Assessment Conference (pp. 233-249). Loughborough, UK: Loughborough University.
32. Moharreri, K., Ha, M., &, Nehm, R., (2014). EvoGrader: An online formative assessment tool for automatically evaluating written evolutionary explanations. Evolution: Education and Outreach, 7 (1), 1-15. Retrieved from http://www.evolution-outreach.com/content/7/1/15
33. Mohler, M., Bunescu, R., &, Mihalcea, R., (2011). Learning to grade short answer questions using semantic similarity measures and dependency graph alignments. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Vol. 1. Human Language Technologies (pp. 752-762). Portland, OR: Association for Computational Linguistics.
34. National Council of Teachers of English. (2008). Statement on class size and teacher workload: Secondary. Retrieved from http://www.ncte.org/positions/statements/classsizesecondary.
35. Nehm, R. H., Ha, M., &, Mayfield, E., (2011). Transforming biology assessment with machine learning: Automated scoring of written evolutionary explanations. Journal of Science Education and Technology, 21 (1), 183-196. doi: 10.1007/s10956-011-9300-9
36. Nehm, R. H., &, Schonfeld, I. S., (2008). Measuring knowledge of natural selection: A comparison of the CINS, an open-response instrument, and an oral interview. Journal of Research in Science Teaching, 45, 1131-1160.
37. Nehm, R. H., &, Haertig, H., (2012). Human vs. computer diagnosis of students' natural selection knowledge: Testing the efficacy of text analytic software. Journal of science education and technology, 21 (1), 56-73.
38. Nielsen, R. D., Ward, W., &, Martin, J. H., (2008). Classification errors in a domain-independent assessment system. Proceedings of the Third Workshop on Innovative Use of NLP for Building Educational Applications (pp. 10-18). Stroudsburg, PA: Association for Computational Linguistics.
39. Sandene, B., Horkay, N., Bennett, R., Braswell, J., &, Oranje, A., (2005). Online assessment in mathematics and writing: Reports from the NAEP technology-based assessment project, research and development series (NCES 2005-457). Washington, DC: U.S. Government Printing Office.
40. Smola, A. J., &, Schölkopf, B., (2004). A tutorial on support vector regression. Journal of Statistics and Computing, 14 (3), 199-222.
41. Stemler, S. E., (2004). A comparison of consensus, consistency, and measurement approaches to estimating interrater reliability. Practical Assessment, Research & Evaluation, 9 (4), 66-78.
42. Sukkarieh, J. Z., &, Bolge, E., (2010). Building a textual entailment suite for evaluating content scoring technologies. Proceedings of the Seventh Conference on International Language Resources and Evaluation (pp. 3149-3156). Paris, France: European Language Resources Association.
43. Sukkarieh, J. Z., &, Blackmore, J., (2009). c-Rater: Automatic content scoring for short constructed responses. In H. C. Lane & H. W. Guesgen (Eds.), Proceedings of the twenty-second international Florida artificial intelligence research society conference (pp. 290-295). Menlo Park, CA: Association for the Advancement of Artificial Intelligence Press.
44. Weston, M., Parker, J.M., &, Urban-Lurain, M., (2013). Comparing formative feedback reports: Human and automated text analysis of constructed response questions in biology. Paper presented at the Annual Conference of the National Association on Research in Science Teaching, Rio Grande, Puerto Rico.
45. Williamson, D., Xi, X., &, Breyer, J., (2012). A framework for evaluation and use of automated scoring. Educational Measurement: Issues and Practice, 31 (1), 2-13.