Transforming Biology Assessment with Machine Learning: Automated Scoring of Written Evolutionary Explanations

Journal of Science Education and Technology

Volumn 21, Issue 1, 2012, Pages 183-196

  Nehm, Ross H ^a   Ha, Minsu ^a   Mayfield, Elijah ^b  

^a OHIO STATE UNIVERSITY   (United States)

^b Carnegie Mellon University   (United States)

Author keywords

Assessment; Computers; Evolution; Explanation; Machine learning; SIDE; Text analysis

Indexed keywords

EID: 84856612191     PISSN: 10590145     EISSN: 15731839     Source Type: Journal    
DOI: 10.1007/s10956-011-9300-9     Document Type: Article

References (43)

1. Alberts B (2010) Reframing science standards. Science 329(5991): 491.
2. Arora S, Nyberg E (2009) Interactive annotation learning with indirect feature voting. In: Paper in the proceedings of student research symposium at NAACL-HLT 2009, Boulder, Colorado, USA. Accessed online at: http://www. cs. cmu. edu/%7Eshilpaa/NAACL_SRW_IAL. pdf.
3. Bejar II (1991) A methodology for scoring open-ended architectural design problems. J Appl Psychol 76(4): 522-532.
4. Bishop B, Anderson C (1990) Student conceptions of natural selection and its role in evolution. J Res Sci Teach 27: 415-427.
5. Burstein J (2003) The e-rater scoring engine: automated essay scoring with natural language processing. In: Shermis MD, Burstein J (eds) Automated essay scoring: a cross-disciplinary perspective. Lawrence Erlbaum Associates, Inc, Mahwah, pp 113-122.
6. Chung GKWK, Baker EL (2003) Issues in the reliability and validity of automated scoring of constructed responses. In: Shermis MD, Burstein J (eds) Automated essay scoring: a cross-disciplinary perspective. Erlbaum, Mahwah, pp 23-40.
7. Clough EE, Driver R (1986) A study of consistency in the use of students' conceptual frameworks across different task contexts. Sci Educ 70: 473-496.
8. Demastes SS, Good RG, Peebles P (1995) Students' conceptual ecologies and the process of conceptual change in evolution. Sci Educ 79(6): 637-666.
9. Donmez P, Rosé C, Stegmann K, Weinberger A, Fischer F (2005) Supporting CSCL with automatic corpus analysis technology. In: Paper in proceedings of the international conference on computer support for collaborative learning (CSCL), Taipei, Taiwan.
10. Endler JA (1992) Natural selection: current usages. In: Keller EF, Lloyd EA (eds) Keywords in evolutionary biology. Harvard, Cambridge, pp 220-224.
11. Galt K (2008) SPSS text analysis for surveys 2. 1 and qualitative and mixed methods analysis. J Mixed Meth Res 2(3): 284-286.
12. Gitomer DH, Duschl RA (2007) Establishing multilevel coherence in assessment. In: Moss PA (ed) Evidence and decision making. The 106th yearbook of the National Society for the Study of Education, Part I. National Society for the Study of Education, Chicago, pp 288-320.
13. Krippendorff K (1980) Content analysis: an introduction to its methodology, 1st edn. Sage Publications, Thousand Oaks.
14. Krippendorff K (2004) Content analysis: an introduction to its methodology, 2nd edn. Sage Publications, Thousand Oaks, London.
15. Kumar R, Rosé C, Wang YC, Joshi M, Robinson A (2007) Tutorial dialogue as adaptive collaborative learning support. In: Paper in proceedings of the international conference on artificial intelligence in education, Los Angeles, USA.
16. Landauer TK, Laham D, Foltz PW (2001) The intelligent essay assessor: putting knowledge to the test. In: Paper presented at the Association of Test Publishers Computer-Based Testing: Emerging Technologies and Opportunities for Diverse Applications conference, Tucson, AZ.
17. Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33: 159-174.
18. Lewontin R (1978) Adaptation. Sci Am 239: 212-228.
19. Liu OL, Lee HS, Hofstetter C, Linn MC (2008) Assessing knowledge integration in science: construct, measures, and evidence. Educ Assess 13(1): 33-55.
20. Markoff J (2011) Computer wins on 'jeopardy!': trivial, it's not. New York Times, 16 Feb.
21. Mayfield E, Rosé C (2010) An interactive tool for supporting error analysis for text mining. In: Paper in proceedings of the demonstration session at the international conference of the North American Association for Computational Linguistics (NAACL), Los Angeles, USA.
22. McLaren B, Scheuer O, de Laat M, Hever R, de Groot R, Rosé C (2007) Using machine learning techniques to analyze and support mediation of student e-discussions. In: Paper in proceedings of the international conference on artificial intelligence in education, Los Angeles, USA.
23. National Research Council (2001) Knowing what students know: the science and design of educational assessment. National Academy Press, Washington, D. C.
24. National Research Council (2007) Taking science to school: learning and teaching science in grades K-8. National Academy Press, Washington, D. C.
25. National Research Council (2008) Rising above the gathering storm: energizing and employing America for a brighter economic future. National Academy Press, Washington, D. C.
26. Nehm RH (2010) Understanding undergraduates' problem solving processes. J Biol Microbiol Educ 11(2): 119-122.
27. Nehm RH, Ha M (2011) Item feature effects in evolution assessment. J Res Sci Teach 48(3): 237-256.
28. Nehm RH, Haertig H (2011) Human vs. computer diagnosis of students' natural selection knowledge: testing the efficacy of text analytic software. J Sci Educ Technol. doi: 10. 1007/s10956-011-9282-7.
29. Nehm RH, Reilly L (2007) Biology majors' knowledge and misconceptions of natural selection. Bioscience 57(3): 263-272.
30. Nehm RH, Schonfeld IS (2008) Measuring knowledge of natural selection: a comparison of the CINS, an open-response instrument, and an oral interview. J Res Sci Teach 45(10): 1131-1160.
31. Nehm RH, Schonfeld IS (2010) The future of natural selection knowledge measurement: a reply to Anderson et al. J Res Sci Teach 47(3): 358-362.
32. Nehm RH, Ha M, Rector M, Opfer J, Perrin L, Ridgway J, Mollohan K (2010) Scoring guide for the open response instrument (ORI) and evolutionary gain and loss test (EGALT). Technical Report of National Science Foundation REESE Project 0909999. Accessed online 10 Jan 2011 at: http://evolutionassessment. org.
33. Page EB (1966) The imminence of grading essays by computers. Phi Delta Kappan 47: 238-243.
34. Patterson C (1978) Evolution. Cornell University Press, Ithaca.
35. Pigliucci M, Kaplan J (2006) Making sense of evolution: the conceptual foundations of evolutionary biology. University of Chicago Press, Chicago.
36. Rose C, Donmez P, Gweon G, Knight A, Junker B, Cohen W, Koedinger K, Heffernan N (2005) Automatic and semi-automatic skill coding with a view towards supporting on-line assessment. In: Paper in proceedings of the international conference on artificial intelligence in education, Amsterdam, The Netherlands.
37. Rose CP, Wang YC, Cui Y, Arguello J, Stegmann K, Weinberger A, Fischer F (2008) Analyzing collaborative learning processes automatically: exploiting the advances of computational linguistics in computer-supported collaborative learning. Int J Comput Support Collab Learn 3(3): 237-271.
38. Shermis MD, Burstein J (2003) Automated essay scoring: a cross-disciplinary perspective. Lawrence Erlbaum Associates, Inc, Mahwah.
39. Sukkarieh J, Bolge E (2008) Leveraging c-rater's automated scoring capability for providing instructional feedback for short constructed responses. In: Woolf BP, Aimeur E, Nkambou R, Lajoie S (eds) Lecture notes in computer science: vol. 5091. Proceedings of the 9th international conference on intelligent tutoring systems, ITS 2008, Montreal, Canada, June 23-27, 2008. Springer, New York, pp 779-783.
40. The Conference Board, Corporate Voices for Working Families, the Partnership for 21st Century Skills, and the Society for Human Resource Management (2007) Are they really ready to work? Employers' perspectives on the basic knowledge and applied skills of new entrants to the 21st century workforce. Accessed online 22 Mar 2011 at: http://www. p21. org/index. php?option=com_content&task=view&id=250&Itemid=64.
41. Wagner T (2008) The global achievement gap. Basic Books, New York.
42. Witten IH, Frank E (2005) Data mining, 2nd edn. Elsevier, Amsterdam.
43. Yang Y, Buckendahl CW, Juszkiewicz PJ, Bhola DS (2002) A review of strategies for validating computer automated scoring. Appl Meas Educ 15(4): 391-412.