The Effect of Observation Length and Presentation Order on the Reliability and Validity of an Observational Measure of Teaching Quality

Educational and Psychological Measurement

Volumn 74, Issue 3, 2014, Pages 400-422

  Mashburn, Andrew J ^a   Meyer, J Patrick ^b   Allen, Joseph P ^b   Pianta, Robert C ^b  

^a PORTLAND STATE UNIVERSITY   (United States)

^b UNIVERSITY OF VIRGINIA   (United States)

Author keywords

generalizability theory; observational measures; reliability; validity

Indexed keywords

EID: 84900466139     PISSN: 00131644     EISSN: 15523888     Source Type: Journal    
DOI: 10.1177/0013164413515882     Document Type: Article

References (33)

1. Allen J.,Pianta R.,Gregory A.,Mikami A.,Lun J.An interaction-based approach to enhancing secondary school instruction and student achievement.Science. 2011;333:1034-1037.
2. Bell C. A.,Gitomer D. H.,McCaffrey D. F.,Hamre B. K.,Pianta R. C.,Qi Y.An argument approach to observation protocol validity.Educational Assessment. 2012;17:62-87.
3. Bill and Melinda Gates Foundation. (2012). Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains. Seattle, WA: Author. Retrieved from http://www.metproject.org/downloads/MET_Gathering_Feedback_Practioner_Brief.pdf.
4. Brennan R. L.Generalizability theory: An NCME instructional module.Educational Measurement: Issues and Practice. 1992;11:27-34.
5. Casabianca J.,Lockwood J. R. Paper presented at the annual meeting of the Society for Research on Educational Effectiveness; 2013Washington, DC; 2013.
6. Congdon P. J.,McQueen J.The stability of rater severity in large-scale assessment programs.Journal of Educational Measurement. 2000;37:163-178.
7. Cronbach L. J.,Gleser G. C.,Nanda H.,Rajaratnam N.The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York, NY: Wiley; 1972:.
8. Danielson C.The framework for teaching evaluation instrument. Princeton, NJ: Danielson Group; 2011:.
9. Ebbinghaus H.On memory: A contribution to experimental psychology. New York, NY: Teacher's College Press; 1913:.
10. Erlich O.,Borich G.Occurrence and generalizability of scores on a classroom interaction instrument.Journal of Educational Measurement. 1979;16:11-18.
11. FritzM.ChenK. (2013, February). Federal grants, rise of charter schools expand teacher evaluations. Retrieved from http://www.pbs.org/newshour/rundown/2013/02/teacher-evaluations.html.
12. Grossman P.,Loeb S.,Cohen J.,Hammerness K.,Wyckoff J.,Boyd D.,Lankford H.Measure for measure: The relationship between measures of instructional practice in middle school English language arts and teachers' value-added scores. Cambridge, MA: National Bureau of Economic Research; 2010:.
13. Hamre B.,Pianta R.,Downer J.,Hamagami F.,Mashburn A.,Jones S., Brackett M.Teaching through interactions: Testing a developmental framework of teacher effectiveness in over 4,000 classrooms.Elementary School Journal. 2013;113 (4): 461-487.
14. HeitinL. (2011, October26). Study: States' teacher-evaluation policies are A-changin'. Education Week. Retrieved from http://blogs.edweek.org/edweek/teacherbeat/2011/10/study_states_teacher_evaluatio.html.
15. Hill H. C.,Blunk M.,Charalambous C.,Lewis J.,Phelps G. C.,Sleep L.,Ball D. L.Mathematical knowledge for teaching and the mathematical quality of instruction: An exploratory study.Cognition and Instruction. 2008;26:430-511.
16. Hill H. C.,Charalambous C. Y.,Kraft M. A.When rater reliability is not enough: Teacher observation systems and a case for the generalizability study.Educational Researcher. 2012;41:56-64.
17. Hintze J. M.Psychometrics of direct observation.School Psychology Review. 2005;34:507-519.
18. Ho A. D.,Kane T. J.The reliability of teaching observations by school personnel. Seattle, WA: Bill and Melinda Gates Foundation; 2013:.
19. Kane M. T.A sampling model for validity.Applied Psychological Measurement. 1982;6:125-160.
20. Kane M. T.The errors of our ways.Journal of Educational Measurement. 2011;48:12-30.
21. Leventhal L.,Turcotte S. J.,Abrami P. C.,Perry R. P.Primacy/recency effects in student ratings of instruction: A reinterpretation of gain-loss effects.Journal of Educational Psychology. 1983;75:692-704.
22. Malmberg L.,Hagger H.,Burn K.,Mutton T.,Colls H.Observed classroom quality during teacher education and two years of professional practice.Journal of Educational Psychology. 2010;102:916-932.
23. Marcoulides G.The application of generalizability analysis to observational studies.Quality & Quantity. 1989a;23:115-127.
24. Marcoulides G. A.Performance appraisal: Issues of validity.Performance Improvement Quarterly. 1989b;2:3-12.
25. Mashburn A. J.,Downer J. D.,Hamre B. K.,Justice L. M.,Pianta R. C.Teaching consultation and children's language and literacy development during pre-kindergarten.Applied Developmental Science. 2010;14:179-198.
26. Mashburn A. J.,Downer J. T.,Rivers S. E.,Brackett M. A.,Martinez A.Improving the power of an efficacy study of a social and emotional learning program: Application of generalizability theory to the measurement of classroom-level outcomes.Prevention Science. 2013;:.
27. Mashburn A. J.,Pianta R. C.,Hamre B. K.,Downer J. T.,Barbarin O.,Bryant D,. Howes C.Measures of pre-k quality and children's development of academic, language and social skills.Child Development. 2008;79:732-749.
28. McCaffrey D.,Yuan K.,Savitsky T.,Lockwood J. R.,Edelen M.O. Paper presented at the annual meeting of the Society for Research on Educational Effectiveness; 2013Washington, DC; 2013.
29. Meyer J. P.,Henry A. E.,Mashburn A. J.Occasions and the reliability of teaching observations: Alternative conceptualizations and methods of analysis.Educational Assessment Journal. 2011;16:227-243.
30. Mikami A.,Gregory A.,Allen J. P.,Pianta R. C.,Lun J.Effects of a teacher professional development intervention on peer relationships in secondary classrooms.School Psychology Review. 2011;40:367-385.
31. Pianta R. C.,Hamre B.,Hayes N.,Mintz S.,LaParo K. M.Classroom Assessment Scoring System-Secondary (CLASS-S). Charlottesville, VA: University of Virginia; 2008:.
32. Savitsky T. T.,McCaffrey D. F. Paper presented at the annual meeting of the Society for Research on Educational Effectiveness; 2013Washington, DC; 2013.
33. U.S. Department of Education. (2009). Race to the top executive summary. Washington, DC: Author. Retrieved from http://www2.ed.gov/programs/racetothetop/executive-summary.pdf.