Automated Scoring of Teachers’ Open-Ended Responses to Video Prompts: Bringing the Classroom-Video-Analysis Assessment to Scale

Educational and Psychological Measurement

Volumn 74, Issue 6, 2014, Pages 950-974

  Kersting, Nicole B ^a   Sherin, Bruce L ^b   Stigler, James W ^c  

^a UNIVERSITY OF ARIZONA   (United States)

^b NORTHWESTERN UNIVERSITY   (United States)

^c UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

automated scoring; classroom video analysis assessment; na ve Bayes; short answer items; teacher knowledge; text classification

Indexed keywords

EID: 84911415972     PISSN: 00131644     EISSN: 15523888     Source Type: Journal    
DOI: 10.1177/0013164414521634     Document Type: Article

References (31)

1. Apte C.,Damerau F.,Weiss S. M.Automated learning of decision rules for text categorization.ACM Transaction on Information Systems. 1994;12:233-251
2. Attali Y.Automated subscores for TOEFL iBT® independent essays. Princeton, NJ: ETS; 2011:
3. Attali Y.Handbook of automated essay evaluation: Current applications and new directions. Shermis M. D.Burstein J., ed. New York, NY: Routledge; 2013:181-199.
4. Berry M. W.,Dumais S. T.,O’Brien G. W.Using linear algebra for intelligent information retrieval.SIAM Review. 1995;37:573-595
5. Brew C.,Leacock C.Handbook of automated essay evaluation: Current applications and new directions. Shermis M. D.Burstein J., ed. New York, NY: Routledge; 2013:136-153.
6. Cohen D. K.,Raudenbush S. W.,Ball D. L.Resources, instruction, and research.Educational Evaluation and Policy Analysis. 2003;25:119-142
7. Darling-Hammond L.,Baratz-Snowden J.A good teacher in every classroom. Preparing the highly qualified teachers our children deserve. Hoboken, NJ: Jossey-Bass; 2007:
8. Deerwester S.,Dumais S. T.,Furnas G.,Landauer T.,Harshman R.Indexing by latent semantic analysis.Journal of the American Society for Information Science. 1990;41:391-407
9. Dumais S.,Platt J.,Heckerman D.,Sahami M..Presented at the Seventh International Conference; 1998New York, NY: ACM Press; 1998. 148.
10. Foltz P. W.,Gilliam S.,Kendall S.Supporting content-based feedback in on-line writing evaluation with LSA.Interactive Learning Environments. 2000;8:111-127
11. FoltzP. W.LahamD.LandauerT. (1999). The intelligent essay assessor: Applications to educational technology. Interactive Multimedia Electronic Journal of Computer-Enhanced Learning, 1(2). Retrieved from imej.wfu.edu/articles/1999/2/04/index.asp
12. Hill H. C.,Ball D. L.Learning mathematics for teaching: Results from California’s Mathematics Professional Development Institutes.Journal of Research in Mathematics Education. 2004;35:330-351
13. Hill H. C.,Ball D. L.,Sleep L.,Lewis J. M.Handbook for research on mathematics education. 2007. Assessing teachers’ mathematical knowledge: What knowledge matters and what evidence counts? In LesterF. (Ed.), Handbook for research on mathematics education (2nd ed., pp. 111-155). Charlotte, NC: Information Age.Lester F., ed. Charlotte, NC: Information Age; 2007:111-155.
14. Hill H. C.,Schilling S. G.,Ball D. L.Developing measures of teachers’ mathematics knowledge for teaching.Elementary School Journal. 2004;105:11-30
15. Kersting N.Using Video Clips as Item Prompts to Measure Teachers’ Knowledge of Teaching Mathematics.Educational and Psychological Measurement. 2008;68 (5): 845-861
16. Kersting N. B.,Givvin K.,Sotelo F.,Stigler J. W.Teacher’s Analysis of Classroom Video Predicts Student Learning of Mathematics: Further Explorations of a Novel Measure of Teacher Knowledge.Journal of Teacher Education. 2010;61 (1-2): 172-181
17. Kersting N. B.,Givvin K. B.,Thompson B.,Santagata R.,Stigler J.Developing Measures of Usable Knowledge: Teachers’ Analyses of Mathematics Classroom Videos Predict Teaching Quality and Student Learning.American Educational Research Journal. 2012;49 (3): 568-590
18. KohaviR. (1996). Scaling up the accuracy of naive-Bayes classifiers: A decision-tree hybrid. Retrieved from http://www.sc.ehu.es/ccwbayes/docencia/mmcc/docs/lecturas-clasificacion/NBTree.pdf
19. Landauer T.Automatic essay assessment.Assessment in Education: Principles, Policy and Practice. 2003;10:295-308
20. Landis J.R.,Koch G.G.The measurement of observer agreement for categorical data.Biometrics. 1977;33 (1): 159-174
21. Lewis D. D.Multiple values selected. Berlin, Germany: Springer-Verlag; 1998:4-15.
22. Ng A. Y.,Jordan M. I.On discriminative vs. generative classifiers: A comparison of logistic regression and naive Bayes.Advances in NIPS. 2002;14:841-848
23. Pauli C.,Reusser K.Expertise in mathematics instruction: An international perspective. Li Y.Kaiser G., ed. New York: Springer; 2011:85-107.
24. Sebastiani F.Machine learning in automated text categorization.ACM computing surveys (CSUR). 2002;34 (1): 1-47
25. Handbook of automated essay evaluation: Current applications and new directions. Shermis M. D.Burstein J., ed. New York, NY: Routledge; 2013:
26. Shermis M. D.,Burstein J.,Higgins D.,Zechner K.Automated essay scoring: Writing assessment and instruction.International Encyclopedia of Education. 2010;4:20-26
27. ShermisM. D.HammerB. (2012). Contrasting state-of-the-art automated scoring of essays: Analysis. Retrieved from http://dl.dropboxusercontent.com/u/44416236/NCME%202012%20Paper3_29_12.pdf
28. StreeterL.BernsteinJ.FoltzP.DeLandD. (2011). Pearson’s automated scoring of writing, speaking, and mathematics (White Paper). Retrieved from http://kt.pearsonassessments.com/download/PearsonAutomatedScoring-WritingSpeakingMath-051911.pdf
29. Topol B.,Olson J.,Roeber E.The cost of new higher quality assessments: A comprehensive analysis of the potential costs for future state assessments. Stanford, CA: Stanford University, Stanford Center for Opportunity Policy in Education; 2010:
30. ZhangH. (2004). The optimality of naive Bayes. In BarrV.MarkovZ. (Eds.), Proceedings of the 17th International FLAIRS conference (FLAIRS2004) (pp. 568-573). Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.61.5794&rep=rep1&type=pdf
31. ZhangM. (2013). Contrasting automated and human scoring (ETS Research Report No. RDC-21). Princeton, NJ: ETS. Retrieved from http://www.ets.org/Media/Research/pdf/RD_Connections_21.pdf