Scaling short-answer grading by combining peer assessment with algorithmic scoring

L@S 2014 - Proceedings of the 1st ACM Conference on Learning at Scale

Volumn , Issue , 2014, Pages 99-108

  Kulkarni, Chinmay ^a   Socher, Richard ^a   Bernstein, Michael S ^a   Klemmer, Scott R ^b  

^a STANFORD UNIVERSITY   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Assessment; Automated assessment; Online learning; Peer learning

Indexed keywords

ALGORITHMS; STUDENTS;

ASSESSMENT; AUTOMATED ASSESSMENT; DETAILED FEEDBACKS; LEARNING EXPERIENCES; MACHINE GRADING; ONLINE LEARNING; PEER ASSESSMENT; PEER LEARNING;

GRADING;

EID: 84899689260     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2556325.2566238     Document Type: Conference Paper

References (29)

1. Professionals against machine scoring of student essays in high-stakes assessment (www.humanreaders.com).
2. Bernstein, M. S., Little, G., Miller, R. C., Hartmann, B., and Ackerman., M. S., Karger, D. R., Crowell, D., and Panovich, K. Soylent: a word processor with a crowd inside. In Proceedings of the 23nd annual ACM symposium on User interface software and technology, ACM (2010), 313-322.
3. Boud, D., and Brew, A. Enhancing learning through self assessment, Vol. 1. Kogan Page London, 1995.
4. Burstein, J. The e-rater scoring engine: Automated essay scoring with natural language processing. Automated essay scoring: A cross-disciplinary perspective (2003), 113-121.
5. Burstein, J., Chodorow, M., and Leacock, C. Automated essay evaluation: the criterion online writing service. AI Magazine 25, 3 (2004), 27.
6. Carlson, P., and Berry, F. Calibrated peer review and assessing learning outcomes. In Frontiers in Education Conference, Vol. 2, STIPES (2003).
7. Chen, H., and He, B. Automated essay scoring by maximizing human-machine agreement.
8. Chodorow, M., and Leacock, C. An unsupervised method for detecting grammatical errors. In Proceedings of the 1st North American chapter of the Association for Computational Linguistics conference, Association for Computational Linguistics (2000), 140-147.
9. Fast, E., Lee, C., Aiken, A., Bernstein, M., Koller, D., Smith, E., and Institute, K. Crowd-scale interactive formal reasoning and analytics.
10. Foltz, P. W., Laham, D., and Landauer, T. K. The intelligent essay assessor: Applications to educational technology. Interactive Multimedia Electronic Journal of Computer-Enhanced Learning 1, 2 (1999).
11. Grimes, D., and Warschauer, M. Utility in a fallible tool: A multi-site case study of automated writing evaluation. The Journal of Technology, Learning and Assessment 8, 6 (2010).
12. Heimerl, K., Gawalt, B., Chen, K., Parikh, T., and Hartmann, B. Communitysourcing: engaging local crowds to perform expert work via physical kiosks. In Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems (2012), 1539-1548.
13. Hirschman, L., Breck, E., Light, M., Burger, J. D., and Ferro, L. Automated grading of short-answer tests. Intelligent Systems and their Applications, IEEE (2000), 31-37.
14. Hirschman, L., Light, M., Breck, E., and Burger, J. D. Deep read: A reading comprehension system. In Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics (1999), 325-332.
15. Kahneman, D., Lovallo, D., and Sibony, O. Before you make that big decision. Harvard Business Review 89, 6 (2011), 50-60.
16. Karger, D. R., Oh, S., and Shah, D. Iterative learning for reliable crowdsourcing systems. In Advances in neural information processing systems (2011), 1953-1961.
17. Kulkarni, C., and Wei., K. P., Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller, D., and Klemmer, S. R. Peer and self assessment in massive online classes. ACM Trans. on Computer-Human Interaction 20 (2013), Preprint.
18. Lintott, C. J., Schawinski, K., Slosar, A., Land, K., Bamford, S., Thomas, D., Raddick, M. J., Nichol, R. C., Szalay, A., Andreescu, D., et al. Galaxy zoo: morphologies derived from visual inspection of galaxies from the sloan digital sky survey. Monthly Notices of the Royal Astronomical Society 389, 3 (2008), 1179-1189.
19. Lovallo, D., and Sibony, O. The case for behavioral strategy. McKinsey Quarterly (2010), 30-43.
20. Nielsen, J. Usability inspection methods. In Conference companion on Human factors in computing systems, ACM (1994), 413-414.
21. Peng Dai, M. D., and Weld, S. Decision-theoretic control of crowd-sourced workflows. In In the 24th AAAI Conference on Artificial Intelligence AAAI10 (2010).
22. Piech, C., Huang, J., Chen, Z., Do, C., Ng, A., and Koller, D. Tuning peer grading. In Proceedings of the 6th International Conference on Educational Data Mining (2013).
23. Shermis, M. D., and Garvan., C. W., and Diao, Y. The impact of automated essay scoring on writing outcomes. Online Submission (2008).
24. Sommers, N. Revision strategies of student writers and experienced adult writers. College composition and communication 31, 4 (1980), 378-388.
25. Wetzel, C. G., and Wilson., T. D., and Kort, J. The halo effect revisited: Forewarned is not forearmed. Journal of Experimental Social Psychology (1981).
26. Winerip, M. Facing a robo-grader? just keep obfuscating mellifluously. New York Times (2013).
27. Yang, B., Sun, J.-T., Wang, T., and Chen, Z. Effective multi-label active learning for text classification. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, ACM (2009), 917-926.
28. Yannakoudakis, H., Briscoe, T., and Medlock, B. A new dataset and method for automatically grading esol texts. In ACL (2011), 180-189.
29. Zhang, M. Contrasting automated and human scoring of essays. R & D Connections (2013).