Investigating the impact of automated feedback on students’ scientific argumentation

International Journal of Science Education

Volumn 39, Issue 12, 2017, Pages 1648-1668

  Zhu, Mengxiao ^a   Lee, Hee Sun ^b   Wang, Ting ^a   Liu, Ou Lydia ^a   Belur, Vinetha ^a   Pallant, Amy ^b  

^a EDUCATIONAL TESTING SERVICE   (United States)

^b Concord Consortium ^*  (United States)

Author keywords

automated scoring and feedback; climate change; Log data analysis; scientific argumentation

Indexed keywords

EID: 85023762344     PISSN: 09500693     EISSN: 14645289     Source Type: Journal    
DOI: 10.1080/09500693.2017.1347303     Document Type: Article

References (51)

1. Allchin, D., (2012). Teaching the nature of science through scientific errors. Science Education, 96(5), 904–926. doi:10.1002/sce.21019
2. Anderson, D., Magill, R. A., & Sekiya, H., (2001). Motor learning as a function of KR schedule and characteristics of task intrinsic feedback. Journal of Motor Behavior, 33, 59–66. doi:10.1080/00222890109601903
3. Aufschnaiter, C. V., Erduran, S., Osborne, J., & Simon, S., (2008). Arguing to learn and learning to argue:Case studies of how students’ argumentation relates to their scientific knowledge. Journal of Research in Science Teaching, 45, 101–131. doi:10.1002/tea.20213
4. Azevedo, R., & Brenard, R. M., (1995). A meta-analysis of the effects of feedback in computer-based instruction. Journal of Educational Computing Research, 13, 111–127. doi:10.2190/9LMD-3U28-3A0G-FTQT
5. Berland, L. K., & Reiser, B. J., (2009). Making sense of argumentation and explanation. Science Education, 93(1), 26–55. doi:10.1002/sce.20286
6. Black, P., Harrison, C., Lee, C., Marshall, B., & Wiliam, D., (2003). Assessment for learning:Putting it into practice. Berkshire:McGraw-Hill Education.
7. Bricker, A., & Bell, P., (2008). Conceptualizations of argumentation from science studies and the learning sciences and their implications for the practices of science education. Science Education, 92, 473–498. doi:10.1002/sce.20278
8. Chin, C., & Osborne, J., (2010). Students’ questions and discursive interaction:Their impact on argumentation during collaborative group discussions in science. Journal of Research in Science Teaching, 47(7), 883–908. doi:10.1002/tea.20385
9. Clarke, S., (2003). Enriching feedback in the primary classroom. London:Hodder and Stoughton.
10. Cope, B., Kalantzis, M., Abd-El-Khalick, F., & Bagley, E., (2013). Science in writing:Learning scientific argument in principle and practice. E-Learning and Digital Media, 10(4), 420–441. doi:10.2304/elea.2013.10.4.420
11. Dihoff, R. E., Brosvic, G. M., & Epstein, M. L., (2003). The role of feedback during academic testing:The delay retention effect revisited. The Psychological Record, 53, 533–548. doi:10.1007/BF03395451
12. Duschl, R. A., & Osborne, J., (2002). Supporting and promoting argumentation discourse in science education. Studies in Science Education, 38, 39–72. doi:10.1080/03057260208560187
13. Dzikovska, M., Nielsen, R., Brew, C., Leacock, C., Giampiccolo, D., Bentivogli, L., … Dang, H. T., (2013). SemEval-2013 Task 7:The joint student response analysis and 8th recognizing textual entailment challenge. Second joint conference on lexical and computational semantics (*SEM), Volume 2:Proceedings of the seventh international workshop on semantic evaluation (SemEval 2013) (pp. 263–274). Retrieved from http://www.aclweb.org/anthology/S13-2045
14. Erduran, S., Simon, S., & Osborne, J., (2004). TAPping into argumentation:Developments in the application of toulmin’s argument pattern for studying science discourse. Science Education, 88, 915–933. doi:10.1002/sce.20012
15. Glorfeld, L. W., (1995). An improvement on horn’s parallel analysis methodology for selecting the correct number of factors to retain. Educational and Psychological Measurement, 55(3), 377–393. doi:10.1177/0013164495055003002
16. Gonzales, P., Williams, T., Jocelyn, L., Roey, S., Kastberg, D., & Brenwald, S., (2008). Highlights from TIMSS 2007:Mathematics and science achievement of U.S. Fourth- and eighth-grade students in an international context (NCES 2009-001 revised). Washington, DC:National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education. Retrieved from https://nces.ed.gov/pubs2009/2009001.pdf
17. Graesser, A. C., (2011). Learning, thinking, and emoting with discourse technologies. American Psychologist, 66(8), 746–757. doi:10.1037/a0024974
18. Ha, M., Nehm, R. H., Urban-Lurain, M., & Merrill, J. E., (2011). Applying computerized-scoring models of written biological explanations across courses and colleges:Prospects and limitations. CBE Life Sciences Education, 10(4), 379–393. doi:10.1187/cbe.11-08-0081
19. Hattie, J., (2009). Visible learning:A synthesis of over 800 meta-analyses relating to achievement. New York, NY:Routledge.
20. Heilman, M., & Madnani, N., (2013). ETS:Domain adaptation and stacking for short answer scoring. Second joint conference on lexical and computational semantics (*SEM), Volume 2:Proceedings of the seventh international workshop on semantic evaluation (SemEval 2013), 2(SemEval) (pp. 275–279). Retrieved from http://www.aclweb.org/anthology/S13-2046
21. Horn, J. L., (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30, 179–185. doi:10.1007/BF02289447
22. Johnson, W., & Valente, A., (2009). Tactical language and culture training systems:Using artificial intelligence to teach foreign languages and cultures. AI Magazine, 30(2), 72–83. Retrieved from https://www.researchgate.net/profile/Andre_Valente/publication/221606431_Tactical_Language_and_Culture_Training_Systems_Using_Artificial_Intelligence_to_Teach_Foreign_Languages_and_Cultures/links/0046352964ed029fb0000000.pdf doi:10.1609/aimag.v30i2.2240
23. Kuhn, D., (2010). Teaching and learning science as argument. Science Education, 94(5), 810–824. doi:10.1002/sce.20395
24. Kuhn, D., & Udell, W., (2003). The development of argument skills. Child Development, 74(5), 1245–1260. doi:10.1111/1467-8624.00605
25. Lane, S., (2005). Validity of high-stakes assessment:Are students engaged in complex thinking? Educational Measurement:Issues and Practice, 23(3), 6–14. doi:10.1111/j.1745-3992.2004.tb00160.x
26. Leacock, C., & Chodorow, M., (2003). C-rater:Automated scoring of short-answer questions. Computers and the Humanities, 37(4), 389–405. doi:10.1023/A:1025779619903
27. Lee, H.-S., Liu, O. L., & Linn, M. C., (2011). Validating measurement of knowledge integration in science using multiple-choice and explanation items. Applied Measurement in Education, 24(2), 115–136. doi:10.1080/08957347.2011.554604
28. Lee, H.-S., Liu, O. L., Pallant, A., Roohr, K. C., Pryputniewicz, S., & Buck, Z. E., (2014). Assessment of uncertainty-infused scientific argumentation. Journal of Research in Science Teaching, 51(5), 581–605. doi:10.1002/tea.21147
29. Lee, H.-S., Pallant, A., Lord, T., & Liu, O. L., (2017). Articulating uncertainty attribution as part of critical epistemic practice of scientific argumentation. Proceedings of the computer-supported collaborative learning conference (CSCL ‘2017). Philadelphia, PA:International Society of the Learning Sciences.
30. Litman, D. J., & Silliman, S., (2004). ITSPOKE:An intelligent tutoring spoken dialogue system. Proceedings of the human language technology conference:4th meeting of the North American chapter of the association for computational linguistics (HLT/NAACL) (pp. 233–236). Retrieved from http://dl.acm.org/citation.cfm?id=1614027
31. Liu, O. L., Brew, C., Blackmore, J., Gerard, L., Madhok, J., & Linn, M. C., (2014). Automated scoring of constructed-response science items:Prospects and obstacles. Educational Measurement:Issues and Practice, 33(2), 19–28. doi:10.1111/emip.12028
32. Liu, O. L., Rios, J. A., Heilman, M., Gerard, L., & Linn, M. C., (2016). Validation of automated scoring of science assessments. Journal of Research in Science Teaching, 53(2), 215–233. doi:10.1002/tea.21299
33. Mao, L., Liu, O. L., Roohr, K., Belur, V., Mulhollan, M., Lee, H.-S., & Pallant, A., (2016). Validation of automated scoring for a formative assessment of students’ scientific argumentation. Manuscript submitted for publication.
34. Mayfield, E., & Penstein Rosé, C., (2010). An interactive tool for supporting error analysis for text mining. Proceedings of the NAACL HLT 2010:Demonstration Session (pp. 25–28). Retrieved from http://www.aclweb.org/anthology/N/N10/N10-2007
35. McNeill, K. L., & Pimentel, D. S., (2010). Scientific discourse in three urban classrooms:The role of the teacher in engaging high school students in argumentation. Science Education, 94(2), 203–229.
36. Mitchell, T., Russell, T., Broomhead, P., & Aldridge, N., (2002). Towards robust computerised marking of free-text responses. Proceedings of the 6th CAA international computer assisted assessment conference, Loughborough.
37. Moharreri, K., Ha, M., & Nehm, R. H., (2014). Evograder:An online formative assessment tool for automatically evaluating written evolutionary explanations. Evolution:Education and Outreach, 7(15). doi:10.1186/s12052-014-0015-2
38. National Research Council. (2012). A framework for K-12 science education:Practices, crosscutting concepts, and core ideas. Washington, DC:The National Academies Press.
39. NGSS Lead States. (2013). Next generation science standards:For states, by states. Washington, DC:The National Academies Press.
40. Pallant, A., Pryputniewicz, S., Lord, T., & Lee, H.–S., (2016). Uncertainty-infused scientific argumentation rubrics. Concord, MA:The Concord Consortium.
41. Quitana, C., Reiser, B. J., Davis, E. A., Krajcik, J., Fretz, E., Duncan, R. G., … Soloway, E., (2004). A scaffolding design framework for software to support science inquiry. The Journal of the Learning Sciences, 13(3), 337–386. doi:10.1207/s15327809jls1303_4
42. Sampson, V., & Clark, D. B., (2008). Assessment of the ways students generate arguments in science education:Current perspectives and recommendations for future directions. Science Education, 92(3), 447–472. doi:10.1002/sce.20276
43. Sandoval, W. A., & Reiser, B. J., (2004). Explanation-driven inquiry:Integrating conceptual and epistemic scaffolds for scientific inquiry. Science Education, 88, 345–372. doi:10.1002/sce.10130
44. Shepard, L. A., (2009). Commentary:Evaluating the validity of formative and interim assessment. Educational Measurement:Issues and Practice, 28(3), 32–37. doi:10.1111/j.1745-3992.2009.00152.x
45. Shute, V. J., (2008). Focus on formative feedback. Review of Educational Research, 78, 153–189. doi:10.3102/0034654307313795
46. Smith, E. V., (2000). Metric development and score reporting in Rasch measurement. Journal of Applied Measurement, 1(3), 303–326. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/12029173
47. Staley, K. W., (2014). Experimental knowledge in the face of theoretical error. In M., Boumans, G., Hon, & A. C., Petersen (Eds.), Error and uncertainty in scientific practice:History and philosophy of technoscience (pp. 39–56). London:Routledge.
48. Toulmin, S., (1958). The uses of argument. New York, NY:Cambridge University Press.
49. Wainer, H., & Thissen, D., (1993). Combining multiple-choice and constructed-response test scores:Toward a Marxist theory of test construction. Applied Measurement in Education, 6(2), 103–118. doi:10.1207/s15324818ame0602_1
50. Williamson, D. M., Xi, X., & Breyer, F. J., (2012). A framework for evaluation and use of automated scoring. Educational Measurement:Issues and Practice, 31(1), 2–13. doi:10.1111/j.1745-3992.2011.00223.x
51. Zwick, W. R., & Velicer, W. F., (1986). Comparison of five rules for determining the number of components to retain. Psychological Bulletin, 99(3), 432–442. doi:10.1037/0033-2909.99.3.432