Automated text scoring and real-time adjustable feedback: Supporting revision of scientific arguments involving uncertainty

Science Education

Volumn 103, Issue 3, 2019, Pages 590-622

  Lee, Hee Sun ^a   Pallant, Amy ^a   Pryputniewicz, Sarah ^a   Lord, Trudi ^a   Mulholland, Matthew ^b   Liu, Ou Lydia ^b  

^a The Concord Consortium ^*  (United States)

^b EDUCATIONAL TESTING SERVICE   (United States)

Author keywords

automated text scoring; formative feedback; scientific argumentation; uncertainty

Indexed keywords

EID: 85062954961     PISSN: 00368326     EISSN: 1098237X     Source Type: Journal    
DOI: 10.1002/sce.21504     Document Type: Article

References (88)

1. Allchin, D. (2012). Teaching the nature of science through scientific errors. Science Education, 96(5), 904–926.
2. Azevedo, R., & Bernard, R. M. (1995). A meta-analysis of the effects of feedback in computer-based instruction. Journal of Educational Computing Research, 13(2), 111–127.
3. Beggrow, E. P., Ha, M., Nehm, R. H., Pearl, D., & Boone, W. J. (2014). Assessing scientific practices using machine-learning methods: How closely do they match clinical interview performance? Journal of Science Education and Technology, 23(1), 160–182.
4. Bell, P., & Linn, M. C. (2000). Scientific arguments as learning artifacts: Designing for learning from the web with KIE. International Journal of Science Education, 22(8), 797–817.
5. Berland, L. K., & McNeill, K. L. (2010). A learning progression for scientific argumentation: Understanding student work and designing supportive instructional contexts. Science Education, 94(5), 765–793.
6. Berland, L. K., & Reiser, B. J. (2009). Making sense of argumentation and explanation. Science Education, 93(1), 26–55.
7. Black, P., Harrison, C., Lee, C., Marshall, B., & William, D. (2003). Assessment for learning: Putting it into practice. Berkshire, England: McGraw-Hill Education.
8. Blumenfeld, P. C., Marx, R. W., Soloway, E., & Krajcik, J. (1996). Learning with peers: From small group cooperation to collaborative communities. Educational Researcher, 25(8), 37–40.
9. Bond, T. G., & Fox, C. M. (2007). Applying the Rasch model. Mahwah, NJ: Lawrence Erlbaum Associates.
10. Boumans, M., & Hon, G. (2014). Introduction. In M. Boumans, G. Hon, & A. C. Petersen (Eds.), Error and uncertainty in scientific practice: History and philosophy of technoscience (pp. 1–12). London: Routledge.
11. Bridgeman, B., Trapani, C., & Attali, Y. (2012). Comparison of human and machine scoring of essays: Differences by gender, ethnicity, and country. Applied Measurement in Education, 25, 27–40.
12. Buck, Z. E., Lee, H. -S., & Flores, J. (2014). I am sure there may be a planet there: Student articulation of uncertainty in argumentation tasks. International Journal of Science Education, 36(14), 2391–2420.
13. Cavagnetto, A. R. (2010). Argument to foster scientific literacy: A review of argument interventions in K−12 science contexts. Review of Educational Research, 80(3), 336–371.
14. Cazden, C. B. (1988). Classroom discourse: The language of teaching and learning. Portsmouth, NH: Heinemann.
15. Chalmers, R. P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1–29.
16. Chin, C., & Osborne, J. (2010). Students’ questions and discursive interaction: Their impact on argumentation during collaborative group discussions in science. Journal of Research in Science Teaching, 47(7), 883–908.
17. Chinn, C. A., & Hmelo-Silver, C. E. (2002). Authentic inquiry: Introduction to the special section. Science Education, 86, 171–174.
18. Clark, D. B., & Sampson, V. (2008). Assessing dialogic argumentation in online environments to relate structure, grounds, and conceptual quality. Journal of Research in Science Teaching, 45, 293–321.
19. Clark, D. B., Sampson, V., Weinberger, A., & Erkens, G. (2007). Analytic frameworks for assessing dialogic argumentation in online learning environments. Educational Psychology Review, 19, 343–374.
20. Davis, E. A., & Linn, M. C. (2000). Scaffolding students’ knowledge integration: Prompts for reflection in KIE. International Journal of Science Education, 22(8), 819–837.
21. Desmarais, M. C., & Baker, R. S. J. (2012). A review of recent advances in learner and skill modeling in intelligent learning environments. User Modeling and User-Adapted Interaction, 22(1–2), 9–38. https://doi.org/10.1007/s11257-011-9106-8
22. de Vries, E., Lund, K., & Baker, M. (2002). Computer-mediated epistemic dialogue: Explanation and argumentation as vehicles for understanding scientific notions. The Journal of the Learning Sciences, 11(1), 63–103.
23. Dihoff, R. E., Brosvic, G. M., & Epstein, M. L. (2003). The role of feedback during academic testing: The delay retention effect revisited. The Psychological Record, 53, 533–548.
24. Duschl, R. A., & Osborne, J. (2002). Supporting and promoting argumentation discourse in science education. Studies in Science Education, 38, 39–72.
25. Erduran, S., Simon, S., & Osborne, J. (2004). TAPping into argumentation: Developments in the application of Toulmin's argument pattern for studying science discourse. Science Education, 88, 915–933.
26. Finley, F. N., Nam, Y., & Oughton, J. (2011). Earth systems science: An analytic framework. Science Education, 95(6), 1066–1085.
27. Fleer, M. (1992). Identifying teacher-child interaction which scaffolds scientific thinking in young children. Science Education, 76(4), 373–397.
28. Guzdial, M. (1994). Software-realized scaffolding to facilitate programming for science learning. Interactive Learning Environments, 4(1), 1–44.
29. Hattie, J. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. New York, NY: Routledge.
30. Heilman, M., & Madnani, N. (2013a). ETS: Domain adaptation and stacking for short answer scoring. Seventh International Workshop on Semantic Evaluation, 2(SemEval), 275–279. Retrieved from. http://www.aclweb.org/anthology/S13-2046
31. Heilman, M., & Madnani, N. (2013b). HENRY-CORE: Domain adaptation and stacking for text similarity. Second Joint Conference on Lexical and Computational Semantics (*SEM), 1, 96–102. Retrieved from. http://www.aclweb.org/anthology-new/S/S13/S13-1013
32. Kelly, G. J., & Takao, A. (2002). Epistemic levels in argument: An analysis of university of oceanography students’ use of evidence in writing. Science Education, 86, 314–342.
33. M. S. Khine (Ed.) (2012). Perspectives on scientific argumentation: Theory, practice and research. New York, NY: Springer.
34. Kluger, A. N., & DeNisi, A. (1998). Feedback interventions: Toward the understanding of a double-edged sword. Current Directions in Psychological Science, 7, 67–72.
35. Kuhn, D. (1993). Science as argument: Implications for teaching and learning scientific thinking. Science Education, 77(3), 319–337.
36. Kuhn, D. (2010). Teaching and learning science as argument. Science Education, 94(5), 810–824.
37. Kulik, J. A., & Fletcher, J. D. (2016). Effectiveness of intelligent tutoring systems: A meta-analytic review. Review of Educational Research, 86(1), 42–78.
38. Kuo, C. Y., & Wu, H.-K. (2013). Toward an integrated model for designing assessment systems: An analysis of the current status of computer-based assessments in science. Computer and Education, 68, 388–403.
39. Lee, H.-S., & Butler, N. (2003). Making authentic science accessible to students. International Journal of Science Education, 25(8), 923–948.
40. Lee, H.-S., Liu, O. L., Pallant, A., Roohr, K. C., Pryputniewicz, S., & Buck, Z. E. (2014). Assessment of uncertainty-infused scientific argumentation. Journal of Research in Science Teaching, 51(5), 581–605.
41. Lee, H.-S., Pallant, A., Pryputniewicz, S., Lord, T., & Liu, O. L. (2017). Articulating uncertainty attribution as part of critical epistemic practice of scientific argumentation. In B. K. Smith, M. Borge, E. Mercier, & K. Y. Lim (Eds.), 12th International conference on computer supported collaborative learning (CSCL) (pp. 135–142). Philadelphia, PA.
42. Lee, M.-H., Wu, Y.-T., & Tsai, C.-C. (2009). Research and trends in science education from 2003 to 2007: A content analysis of publications in selected journals. International Journal of Science Education, 31(15), 1999–2020.
43. Linn, M. C., Gerard, L., Ryoo, K., McElhaney, K., Liu, O. L., & Rafferty, A. N. (2014). Computer- guided inquiry to improve science learning. Science, 344(6180), 155–156.
44. Liu, O. L., Brew, C., Blackmore, J., Gerard, L., Madhok, J., & Linn, M. C. (2014). Automated scoring of constructed-response science items: Prospects and obstacles. Educational Measurement: Issues and Practice, 33(2), 19–28.
45. Liu, O. L., Rios, J. A., Heilman, M., Gerard, L., & Linn, M. C. (2016). Validation of automated scoring of science assessments. Journal of Research in Science Teaching, 53(2), 215–233. https://doi.org/10.1002/tea.21299
46. Madnani, N., Loukina, A., & Cahill, A., (2017). A large scale quantitative exploration of modeling strategies for contentscoring. In Proceedings of the 12th workshop on innovative use of nlp for building educational applications (pp. 457–467). Retrieved from http://www.aclweb.org/anthology/W17-5052.
47. Manz, E. (2015). Representing student argumentation as functionally emergent from scientific activity. Review of Educational Research, 85(4), 553–590.
48. Mao, L., Liu, O. L., Roohr, K., Belur, V., Mulholland, M., Lee, H., & Pallant, A. (2018). Validation of automated scoring for a formative assessment that employs scientific argumentation. Educational Assessment, 23(2), 121–138. https://doi.org/10.1080/10627197.2018.1427570
49. Martin, T., & Sherin, B. (2013). Learning analytics and computational techniques for detecting and evaluating patterns in learning: An introduction to the special issue. Journal of the Learning Sciences, 22(4), 511–520.
50. Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149–173.
51. McNeill, K. L., & Krajcik, J. (2007). Middle school students’ use of appropriate and inappropriate evidence in writing scientific explanations. In M. Lovett, & P. Shah (Eds.), Thinking with data (pp. 233–265). New York, NY: Taylor & Francis Group, LLC.
52. McNeill, K. L., Lizotte, D. J., Krajcik, J., & Marx, R. W. (2006). Supporting students’ construction of scientific explanations by fading scaffolds in instructional materials. Journal of the Learning Sciences, 15(2), 153–191.
53. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (pp. 13–103). Phoenix, AZ: The Oryx Press.
54. Metz, K. E. (2004). Children's understanding of scientific inquiry: Their conceptualization of uncertainty in investigations of their own design. Cognition and Instruction, 22(2), 219–290.
55. Mitrovic, A., Ohlsson, S., & Barrow, D. K. (2013). The effect of positive feedback in a constraint-based intelligent tutoring system. Computers and Education, 60(1), 264–272.
56. National Research Council (2012). A framework for K–12 science education: Practices, crosscutting concepts, and core ideas. Washington, DC: National Academies Press.
57. Nehm, R. H., Ha, M., & Mayfield, E. (2012). Transforming biology assessment with machine learning: Automated scoring of written evolutionary explanations. Journal of Science Education and Technology, 21(1), 183–196.
58. Pallant, A., & Lee, H. -S. (2015). Constructing scientific arguments using evidence from computational models. Journal of Science Education and Technology, 24(2/3), 378–395.
59. Pellegrino, J. W., Chudowsky, N., & Glaser, R. (2001). Knowing what students know. Washington, DC: National Academic Press.
60. Quintana, C., Reiser, B. J., Davis, E. A., Krajcik, J., Fretz, E., Duncan, R. G., … Soloway, E. (2004). A scaffolding design framework for software to support science inquiry. The Journal of the Learning Sciences, 13(3), 337–386.
61. Rasch, G. (1966). Probabilistic models for some intelligence and attainment tests. Chicago, IL: University of Chicago Press.
62. Sadler, D. R. (1989). Formative assessment and the design of instructional systems. Instructional Science, 18, 119–144.
63. Sadler, T. D., & Fowler, S. R. (2006). A threshold model of content knowledge transfer for socioscientific argumentation. Science Education, 90, 986–1004.
64. Sampson, V., & Clark, D. (2009). The impact of collaboration on the outcomes of scientific argumentation. Science Education, 93, 448–484.
65. Sampson, V., & Clark, D. B. (2008). Assessment of the ways students generate arguments in science education: Current perspectives and recommendations for future directions. Science Education, 92, 447–472.
66. Sampson, V., Grooms, J., & Walker, J. P. (2010). Argument-driven inquiry as a way to help students learn how to participate in scientific argumentation and craft written arguments: An exploratory study. Science Education, 95, 217–257.
67. Sandoval, W. A. (2003). Conceptual and epistemic aspects of students’ scientific explanations. The Journal of the Learning Sciences, 12(1), 5–51.
68. Sandoval, W. A. (2006). Understanding students’ practical epistemologies and their influence on learning through inquiry. Science Education, 89, 634–656.
69. Sandoval, W. A., & Millwood, K. A. (2005). The quality of students’ use of evidence in written scientific explanations. Cognition and Instruction, 23(1), 23–55.
70. Sandoval, W. A., & Reiser, B. J. (2004). Explanation-driven inquiry: Integrating conceptual and epistemic scaffolds for scientific inquiry. Science Education, 88, 345–372.
71. Shavelson, R. J., Young, D. B., Ayala, C. C., Brandon, P. R., Furtak, E. M., Ruiz-Primo, M. A., … Yin, Y. (2008). On the impact of curriculum-embedded formative assessment on learning: A collaboration between curriculum and assessment developers. Assessment in Education: Principles, Policy and Practice, 31(2), 59–75.
72. Sherin, B. (2013). A computational study of commonsense science: An exploration in the automated analysis of clinical interview data. Journal of the Learning Sciences, 22, 600–638.
73. Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153–189.
74. Smith, L. A., & Petersen, A. C. (2014). Predictions to climate policy. In M. Boumans, G. Hon, & A. C. Petersen (Eds.), Error and uncertainty in scientific practice (pp. 137–156). New York, NY: Routledge.
75. Staley, K. W. (2014). Experimental knowledge in the face of theoretical error. In M. Boumans, G. Hon, & A. C. Petersen (Eds.), Error and uncertainty in scientific practice: History and philosophy of technoscience (pp. 39–56). London: Routledge.
76. Sterman, J. D. (2002). All models are wrong: Reflections on becoming a systems scientist. System Dynamics Review, 18(4), 501–531.
77. Toulmin, S. (1958). The uses of argument. New York, NY: Cambridge University Press.
78. Von Aufschnaiter, C., Erduran, S., Osborne, J., & Simon, S. (2008). Arguing to learn and learning to argue: Case studies of how students’ argumentation relates to their scientific knowledge. Journal of Research in Science Teaching, 45, 101–131.
79. Vygotsky, L. S. (1978). Mind in society. Cambridge, MA: Harvard University Press.
80. Walton, D., Reed, C., & Macagno, F. (2008). Argumentation schemes. New York, NY: Cambridge University Press.
81. White, B. Y., & Frederiksen, J. R. (1998). Inquiry, modeling, and metacognition: Making science accessible to all students. Cognition and Instruction, 16(1), 3–118.
82. Williamson, D. M., Xi, X., & Breyer, F. J. (2012). A framework for evaluation and use of automated scoring. Educational Measurement: Issues and Practice, 31(1), 2–13.
83. Wilson, M. (2005). Constructing measures: An item response modeling approach. Mahwah, NJ: Lawrence Erlbaum Associates.
84. Wood, D., Bruner, J. S., & Ross, G. (1976). The role of tutoring in problem solving. Journal of Child Psychology and Psychiatry, 17, 89–100.
85. Wright, K., Pallant, A., & Lee, H.-S. (2017). Appropriating a climate science discourse about uncertainty in online lessons. In B. K. Smith, M. Borge, E. Mercier, & K. Y. Lim (Eds.), 12th International conference on computer supported collaborative learning (CSCL) (pp. 577–580). Philadelphia, PA: International Society of the Learning Sciences.
86. Yen, W. M. (1993). Scaling performance assessments: Strategies for managing local item dependence. Journal of Educational Measurement, 30(3), 187–213.
87. Zhu, M., Lee, H.-S., Wang, T., Liu, O. L., Belur, V., & Pallant, A. (2017). Investigating the impact of automated feedback on students’ scientific argumentation. International Journal of Science Education, 39(12), 1648–1668.
88. Zohar, A., & Nemet, F. (2002). Fostering students’ knowledge and argumentation skills through dilemmas in human genetics. Journal of Research in Science Teaching, 39(1), 35–62.