Trends in Classroom Observation Scores

Educational and Psychological Measurement

Volumn 75, Issue 2, 2015, Pages 311-337

  Casabianca, Jodi M ^a   Lockwood, J R ^b   McCaffrey, Daniel F ^b  

^a UNIVERSITY OF TEXAS AT AUSTIN   (United States)

^b EDUCATIONAL TESTING SERVICE   (United States)

Author keywords

B splines; classroom assessment; Classroom Assessment Scoring System Secondary (CLASS S); generalizability theory; rater drift; scoring trend; teaching quality

Indexed keywords

EID: 84924955859     PISSN: 00131644     EISSN: 15523888     Source Type: Journal    
DOI: 10.1177/0013164414539163     Document Type: Article

References (36)

1. Attali Y.Sequential effects in essay ratings.Educational and Psychological Measurement. 2011;71:68-79
2. BatesD.MaechlerM.BolkerB. (2013). lme4: Linear mixed-effects models using S4 classes. R package version 0.999999-2. Retrieved from http://CRAN.R-project.org/package=lme4
3. Bell C. A.,Qi Y.,Croft A. J.,Leusner D.,Gitomer D. H.,McCaffrey D. F.,Pianta R. C.Designing Teacher Evaluation Systems: New Guidance from the Measures of Effective Teaching Project. Kane T.Kerr K.Pianta R., ed. San Francisco, CA: Jossey-Bass; 2014:
4. Bill and Melinda Gates Foundation. (2012). Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains. Seattle, WA: Author. Retrieved from http://www.metproject.org/downloads/MET_Gathering_Feedback_Research_Paper.pdf
5. Braun H. I.Understanding score reliability: Experiments in calibrating essay readers.Journal of Educational Statistics. 1988;13:1-18
6. Brennan R. L.Generalizability theory. New York, NY: Springer-Verlag; 2001:
7. Casabianca J. M.,McCaffrey D. F.,Gitomer D.,Bell C.,Hamre B. K.,Pianta R. C.Effect of observation mode on measures of secondary mathematics teaching.Educational and Psychological Measurement. 2013;73:757-783
8. Congdon P. J.,McQueen J.The stability of rater severity in large-scale assessment programs.Journal of Educational Measurement. 2000;37:163-178
9. Danielson C.Enhancing professional practice: A framework for teaching. Alexandria, VA: Association for Supervision and Curriculum Development; 2007:
10. Engelhard G.,Myford C. M.Monitoring faculty consultant performance in the Advanced Placement English Literature and Composition Program with a many-faceted Rasch model. Princeton, NJ: Educational Testing Service; 2003:
11. GitomerD. H.BellC. A.QiY.McCaffreyD. F.HamreB. K.PiantaR. C. (2014). The instructional challenge in improving teaching quality: Lessons from a classroom observation protocol. Teachers College Record, 116(6). Retrieved from http://www.tcrecord.org/library/abstract.asp?contentid=17460
12. GoeL.BellC. A.LittleO. (2008). Approaches to evaluating teacher effectiveness: A research synthesis. Washington, DC: National Comprehensive Center for Teacher Quality. Retrieved from http://www.tqsource.org/publications/EvaluatingTeachEffectiveness.pdf
13. GrossmanP.LoebS.CohenJ.HammernessK.WyckoffJ.BoydD.LankfordH. (2010). Measure for measure: The relationship between measures of instructional practice in middle school English language arts and teachers’ value-added scores (NBER Working Paper No. 16015). Retrieved from http://www.nber.org/papers/w16015
14. Harik P.,Clauser B. E.,Grabovsky I.,Nungester R. J.,Swanson D.,Nandakumar R.An examination of rater drift within a generalizability theory framework.Journal of Educational Measurement. 2009;46:43-58
15. Hastie T.,Tibshirani R.,Friedman J.The elements of statistical learning: Data mining, inference, and prediction. 2009 The elements of statistical learning: Data mining, inference, and prediction (2nd ed.). New York, NY: Springer Science+Business.New York, NY: Springer Science+Business; 2009:
16. Hill H. C.,Blunk M.,Charalambous C.,Lewis J.,Phelps G. C.,Sleep L.,Ball D. L.Mathematical knowledge for teaching and the mathematical quality of instruction: An exploratory study.Cognition and Instruction. 2008;26:430-511
17. Hill H. C.,Charalambous C. Y.,Kraft M. A.When rater reliability is not enough: Teacher observation systems and a case for the generalizability study.Educational Researcher. 2012;41:56-64
18. Hoskens M.,Wilson M.Real-time feedback on rater drift in constructed response items: An example from the Golden State Examination.Journal of Educational Measurement. 2001;38:121-146
19. Jawahar I. M.,Williams C. R.Where all the children are above average: The performance appraisal purpose effect.Personnel Psychology. 1997;50:905-925
20. Kingsbury F. A.Analyzing ratings and training raters.Journal of Personnel Research. 1922;1:377-383
21. Lord R.,Cole D.Principal bias in rating teachers.Journal of Educational Research. 1961;55:33-35
22. Martínez J. F.,Borko H.,Stecher B. M.Measuring instructional practice in science using classroom artifacts: Lessons learned from two validation studies.Journal of Research in Science Teaching. 2012;49:38-67
23. McKinley D.,Boulet J. R.Detecting score drift in a high-stakes performance-based assessment.Advances in Health Sciences Education. 2004;9:29-38
24. McQueen J.,Congdon P. J..Paper presented at the annual meeting of the American Educational Research Association; 1997Chicago, IL; 1997.
25. Meyer J. P.,Cash A. H.,Mashburn A.Occasions and the reliability of classroom observations: Alternative conceptualizations and methods of analysis.Educational Assessment. 2012;16:227-243
26. Myford C. M.,Wolfe E. M.Monitoring rater performance over time: A framework for detecting differential accuracy and differential scale use.Journal of Educational Measurement. 2009;46:371-389
27. Pianta R. C.,Hamre B. K.,Haynes N. J.,Mintz S. L.,Paro K. M.Classroom Assessment Scoring System (CLASS), secondary manual. Charlottesville: University of Virginia Center for Advanced Study of Teaching and Learning; 2007:
28. Raudenbush S. W.,Bryk A. S.Hierarchical linear models: Applications and data analysis methods. 2002 Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage.Thousand Oaks, CA: Sage; 2002:
29. Rowan B.,Harrison D. M.,Hayes A.Using instructional logs to study mathematics curriculum and teaching in the early grades.Elementary School Journal. 2004;105:103-127
30. Saal F. E.,Downey R. G.,Lahey M. A.Rating the ratings: Assessing the psychometric quality of rating data.Psychological Bulletin. 1980;88:413-428
31. Schwarz G. E.Estimating the dimension of a model.Annals of Statistics. 1978;6:461-464
32. Searle S. R.Linear models. New York, NY: Wiley; 1971:
33. Thorndike E. L.A constant error in psychological ratings.Journal of Applied Psychology. 1920;4:25-29
34. Wang Z.,von Davier A. A.Monitoring of scoring using e–rater® and human raters on a writing test. Princeton, NJ: Educational Testing Service; 2014:
35. WeisbergD.SextonS.MulhernJ.KeelingD. (2009). The widget effect. Brooklyn, NY: The New Teacher Project. Retrieved from http://widgeteffect.org/downloads/TheWidgetEffect.pdf
36. Wilson M.,Case H.Objective measurement: Theory into practice. Wilson M.Engelhard G., ed. Stamford, CT: Ablex; 2000:113-134.