Applying computerized-scoring models ofwritten biological explanations across courses and colleges: Prospects and limitations

CBE Life Sciences Education

Volumn 10, Issue 4, 2011, Pages 379-393

  Ha, Minsu ^a   Nehm, Ross H ^a   Urban Lurain, Mark ^b   Merrill, John E ^b  

^a OHIO STATE UNIVERSITY   (United States)

^b MICHIGAN STATE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; BIOLOGY; COMPUTER PROGRAM; EDUCATION; HUMAN; INFORMATION PROCESSING; LEARNING; METHODOLOGY; WRITING;

AUTOMATIC DATA PROCESSING; BIOLOGY; EDUCATIONAL MEASUREMENT; HUMANS; LEARNING; SOFTWARE; WRITING;

EID: 82755186234     PISSN: None     EISSN: 19317913     Source Type: Journal    
DOI: 10.1187/cbe.11-08-0081     Document Type: Article

References (53)

1. American Association for the Advancement of Science (1994). Benchmarks for Science Literacy, New York, NY: Oxford University Press.
2. American Association for the Advancement of Science (2011). Vision and Change in Undergraduate Biology Education, Washington, DC: http://visionandchange.org.
3. Batistta BT, Hanegan N, Sudweeks R, Cates R (2010). Using item response theory to conduct a distracter analysis on conceptual inventory of natural selection. Int J Sci Math Educ 8, 845-868.
4. Bejar II (1991). A methodology for scoring open-ended architectural design problems. J Appl Psychol 76, 522-532.
5. Bishop B, Anderson C(1990). Student conceptions of natural selection and its role in evolution. J Res Sci Teach 27, 415-427.
6. Braaten M, Windschitl M(2011). Working toward a stronger conceptualization of scientific explanation for science education. Sci Educ 95, 639-669.
7. Burstein J (2003). The e-rater scoring engine: automated essay scoring with natural language processing. In: Automated Essay Scoring: A Cross-Disciplinary Perspective, ed. MD Shermis and J Burstein, Mahwah, NJ: Lawrence Erlbaum, 113-122.
8. Butler AC, Marsh EJ, Goode MK, Roediger HL (2006). When additional multiple-choice lures aid versus hinder later memory. Appl Cogn Psychol 20, 941-956.
9. Chi MTH, Slotta JD, de Leeuw N (1994). From things to processes: a theory of conceptual change for learning science concepts. Learn Instr 4, 27-43.
10. Clough EE, Driver R (1986). A study of consistency in the use of students' conceptual frameworks across different task contexts. Sci Educ 70, 473-496.
11. Corcoran TB, Mosher FA, Rogat A (2009). Learning Progressions in Science: An Evidence-Based Approach to Reform, New York: Center on Continuous Instructional Improvement.
12. Duschl RA, Schweingruber HA, Shouse AW(2007). Taking Science to School: Learning and Teaching Science in Grades K-8, Washington, DC: National Academy Press.
13. Evans EM, Lane JD (2011). Contradictory or complementary? Creationist and evolutionist explanations of the origin(s) of species. Hum Dev 54, 144-159.
14. Galt K (2008). SPSS text analysis for surveys 2.1 and qualitative and mixed methods analysis. J Mixed Meth Res 2, 284-286.
15. Gregory TR (2009). Understanding natural selection: essential concepts and common misconceptions. Evol Educ Outreach 2, 156-175.
16. Ha M, Nehm RH (2011). Comparative efficacy of two computerassisted scoring tools for evolution assessment. Paper presented at the National Association of Research in Science Teaching, Orlando, FL, April 2011.
17. Haudek KC, Kaplan JJ, Knight J, Long T, Merrill JE, Munn A, Nehm RH, Smith M, Urban-Lurain M(2011). Harnessing technology to improve formative assessment of student conceptions in STEM: forging a national network. CBE Life Sci Educ 10, 149-155.
18. Kang SHK, McDermott KB, Roediger HL (2007). Test format and corrective feedback modify the effect of cross-validation on longterm retention. Eur J Cogn Psychol 19, 528-558.
19. Kelemen D, Rosset E (2009). The human function compunction: teleological explanation in adults. Cognition 111, 138-143.
20. Landauer TK, Laham D, Foltz PW (2001). The intelligent essay assessor: putting knowledge to the test. Paper presented at the Association of Test Publishers Computer-Based Cross-validation: Emerging Technologies and Opportunities for Diverse Applications conference, Tucson, AZ, February 2001.
21. Landis JR, Koch GG (1977). The measurement of observer agreement for categorical data. Biometrics 33, 159-174.
22. Mandler G, Rabinowitz JC (1981). Appearance and reality: does a recognition test really improve subsequent recall and recognition?. J Exp Psychol Learn 7, 79-90.
23. Mayfield E, Rosé C (2010a). An interactive tool for supporting error analysis for text mining. Proceedings of the North American Association for Computational Linguistics (NAACL) HLT 2010: Demonstration Session, Los Angeles, CA, June 2010, 25-28.
24. Mayfield E, Rosé C (2010b). SIDE: The Summarization IDE (User's Manual). www.cs.cmu.edu/-cprose/SIDE.html (accessed 13 October 2011).
25. National Research Council (1996). National Science Education Standards, Washington, DC: National Academy Press.
26. National Research Council (2001). Knowing What Students Know: The Science and Design of Educational Assessment, Washington, DC: National Academies Press.
27. Nehm RH (2010). Understanding undergraduates' problem solving processes. J Biol Microbiol Educ 11, 119-122.
28. Nehm RH, Beggrow E, Opfer J, Ha M. Reasoning about natural selection: diagnosing contextual competency using the ACORNS instrument. Am Bio Teacher (in press).
29. Nehm RH, Ha M(2011). Item feature effects in evolution assessment. J Res Sci Teach 48, 237-256.
30. Nehm RH, Ha M, Mayfield E. Transforming biology assessmentwith machine learning: automated scoring of written evolutionary explanations. J Sci Educ Technol (in press).
31. Nehm RH, Ha M, Rector M, Opfer J, Perrin L, Ridgway J, Mollohan K (2010a). Scoring Guide for the Open Response Instrument (ORI) and Evolutionary Gain and Loss Test (EGALT). Technical Report of National Science Foundation REESE Project 0909999 http://evolutionassessment.org (accessed 10 January 2011).
32. Nehm RH, Haertig H Human vs. computer diagnosis of students' natural selection knowledge: cross-validation the efficacy of text analytic software. J Sci Educ Technol (in press).
33. Nehm RH, Kim SY, Sheppard K (2009a). Academic preparation in biology and advocacy for teaching evolution: Biology versus non biology teachers. Sci Educ 93, 1122-1146.
34. Nehm RH, Poole TM, Lyford ME, Hoskins SG, Carruth L, Ewers BE, Colberg PJS (2009b). Does the segregation of evolution in biology textbooks and introductory courses reinforce students' faulty mental models of biology and evolution? Evol Educ Outreach 2, 527-532.
35. Nehm RH, Rector M, Ha M (2010b). "Force-talk" in evolutionary explanation: metaphors and misconceptions. Evol Educ Outreach 3, 605-613.
36. Nehm RH, Reilly L (2007). Biology majors' knowledge and misconceptions of natural selection. Bioscience 57, 263-272.
37. Nehm RH, Schonfeld IS (2007). Does increasing biology teacher knowledge of evolution and the nature of science lead to greater preference for the teaching of evolution in schools? J Sci Teach Educ 18, 699-723.
38. Nehm RH, Schonfeld IS (2008). Measuring knowledge of natural selection: a comparison of the CINS, an open-response instrument, and an oral interview. J Res Sci Teach 45, 1131-1160.
39. Nehm RH, Schonfeld IS (2010). The future of natural selection knowledge measurement: a reply to Anderson et al. J Res Sci Teach 47, 358-362.
40. Neumann I, Neumann K, Nehm RH (2011). Evaluating instrument quality in science education: Rasch based analyses of a nature of science test. Int J Sci Educ 33, 1373-1405.
41. Opfer J, Ridgway J, Perrin L, Mollohan K, Nehm RH (2011). Applying cognitive science to assessment of evolution education. Paper presented at the National Association of Research in Science Teaching, Orlando, FL, April 2011.
42. Page EB (1966). The imminence of grading essays by computers. Phi Delta Kappan 47, 238-243.
43. Powers DE, Burstein JC, Chodorow M, Fowles ME, Kukich K (2002). Stumping E-rater: challenging the validity of automated essay scoring. Comput Hum Behav 18, 103-134.
44. Roediger HL, Marsh EJ (2005). The positive and negative consequences of multiple-choice cross-validation. J Exp Psychol Learn 31, 1155-1159.
45. Shermis MD, Burstein J (2003). Automated Essay Scoring: A Cross-Disciplinary Perspective, Mahwah, NJ: Lawrence Erlbaum.
46. Smith MK, Wood WB, Knight JK (2008). The genetics concept assessment: a new concept inventory for gauging student understanding of genetics. CBE Life Sci Educ 7, 422-430.
47. Sukkarieh J, Bolge E (2008). Leveraging C-rater's automated scoring capability for providing instructional feedback for short constructed responses. In: Lecture Notes in Computer Science, vol. 5091, ed. BP Woolf, E Aimeur, R Nkambou, and S Lajoie, New York: Springer, 779-783.
48. Urban-Lurain M, Moscarella RA, Haudek KC, Giese E, Merrill JE, Sibley DF (2010). Insight into student thinking in STEM: lessons learned from lexical analysis of student writing. Paper presented at the National Association for Research in Science Teaching Annual International Conference, Philadelphia, PA, March 2010.
49. Vosniadou S (2008). The framework theory approach to the problem of conceptual change. In: International Handbook of Research on Conceptual Change, ed. S Vosniadou, New York: Routledge, 3-34.
50. Williamson DM (2009). A framework for implementing automated scoring. Paper presented at the American Educational Research Association, San Diego, CA, April 2009.
51. Witten IH, Frank E (2005). Data Mining, 2nd ed., Amsterdam: Elsevier.
52. Wood WB (2004). Clickers: a teaching gimmick that works. Dev Cell 7, 796-798.
53. Yang Y, Buckendahl CW, Juszkiewicz PJ, Bhola DS (2002). A review of strategies for validating computer automated scoring. Appl Meas Educ 15, 391-412.