Rater calibration when observational assessment occurs at large scale: Degree of calibration and characteristics of raters associated with calibration

Early Childhood Research Quarterly

Volumn 27, Issue 3, 2012, Pages 529-542

  Cash, Anne H ^a   Hamre, Bridget K ^a   Pianta, Robert C ^a   Myers, Sonya S ^b  

^a UNIVERSITY OF VIRGINIA   (United States)

^b VANDERBILT UNIVERSITY   (United States)

Author keywords

Observation; Rater calibration; Teacher quality

Indexed keywords

EID: 84864023287     PISSN: 08852006     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.ecresq.2011.12.006     Document Type: Article

References (59)

1. American Educational Research Association (AERA), American Psychological Association (APA), National Council on Measurement in Education (NCME) Standards for educational and psychological testing 1999, AERA, Washington, DC.
2. Arnett J. Caregivers in day-care centers: Does training matter. Journal of Applied Developmental Psychology 1989, 10(4):541-552. doi:10. 1016/0193-3973(89)90026-9.
3. Barnett W.S., Epstein D.J., Friedman A.H., Boyd J.S., Hustedt J.T. The state of preschool 2008: State preschool yearbook 2008, National Institute for Early Education Research, Rutgers University, New Brunswick, NJ, Retrieved from http://nieer.org/yearbook2008/.
4. Burchinal M., Howes C., Pianta R.C., Bryant D., Early D.M., Clifford R.M., et al. Predicting child outcomes at the end of kindergarten from the quality of pre-kindergarten teacher-child interactions and instruction. Applied Developmental Science 2008, 12(3):140-153. 10.1080/10888690802199418.
5. Burchinal M., Vandergrift N., Pianta R., Mashburn A. Threshold analysis of association between child care quality and child outcomes for low-income children in pre-kindergarten programs. Early Childhood Research Quarterly 2010, 25:166-176. 10.1016/j.ecresq.2009.10.004.
6. Clifford R.M., Barbarin O., Chang F., Early D., Bryant D., Howes C., et al. What is pre-kindergarten? Characteristics of public pre-kindergarten programs. Applied Developmental Science 2005, 9(3):126-143. doi:10.1207/s1532480xads0903_1.
7. Cohen J. Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin 1968, 70(4):213-220. doi:10.1037/h0026256. PMID 19673146.
8. Colvin G., Flannery K.B., Sugai G., Monegan J. Using observational data to provide performance feedback to teachers: A high school case study. Preventing School Failure 2009, 53(2):95-104. doi: 10.3200/PSFL.53.2.95-104.
9. Curby T.W., LoCasale-Crouch J., Konold T.R., Pianta R.C., Howes C., Burchinal M., et al. The relations of observed pre-K classroom quality profiles to children's achievement and social competence. Early Education & Development 2009, 20(2):346-372. doi: 10.1080/10409280802581284.
10. Dickinson D.K., Caswell L. Building support for language and early literacy in preschool classrooms through in-service professional development: Effects of the Literacy Environment Enrichment Program (LEEP). Early Childhood Research Quarterly 2007, 22(2):243-260. doi:10.1016/j.ecresq.2007.03.001.
11. Downer J.T., Hamre B.K. Beliefs about intentional instruction [Survey] 2010, Center for Advanced Study of Teaching and Learning, Charlottesville, VA, Unpublished instrument.
12. Fleiss J.L., Cohen J. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Pyschological Measurement 1973, 33:613-619. doi:10.1177/001316447303300309.
13. Frank Porter Graham Child Development Institute Levels of training on the environment rating scales 2009, Retrieved from http://www.fpg.unc.edu/~ecers/training_levels.htm.
14. Graham J.W. Missing data analysis: Making it work in the real world. Annual Review of Psychology 2009, 60:6.1-6.28. doi:10.1146/annurev.psych.58.110405.085530.
15. Halle T., Vick J. Quality in early childhood care and education settings: A compendium of measures 2007, Child Trends for the Office of Planning, Research and Evaluation, Administration for Children and Families, US Department of Health and Human Services, Washington, DC, Retrieved from http://www.childtrends.org/Files/Child_Trends-2010_03_10_FR_QualityCompendium.pdf.
16. Hamre B.K., Pianta R.C. Can instructional and emotional support in the first-grade classroom make a difference for children at risk of school failure?. Child Development 2005, 76(5):949-967. doi: 10.1111/j.1467-8624.2005.00889.x.
17. Hamre B.K., Pianta R.C., Mashburn A.J., Downer J.T. Building a science of classrooms: Application of the CLASS framework in over 4,000 US early childhood and elementary classrooms 2008, Retrieved from http://www.fcd-us.org/resources/resources_show.htm?doc_id=507559.
18. Harms T., Clifford R.M. The Early Childhood Environment Rating Scale 1980, Teachers College Press, New York, NY. 1st ed.
19. Harms T., Clifford R.M., Cryer D. The Early Childhood Environment Rating Scale 1998, Teachers College Press, New York, NY. Rev. ed.
20. Hintze J.M. Psychometrics of direct observation. School Psychology Review 2005, 34(4):507-519.
21. Howes C., Burchinal M., Pianta R., Bryant D., Early D., Clifford R., et al. Ready to learn? Children's pre-academic achievement in pre-Kindergarten programs. Early Childhood Research Quarterly 2008, 23(1):27-50. doi:10.1016/j.ecresq.2007.05.002.
22. Johnson R.L., Penny J.A., Gordon B. Assessing performance: Designing, scoring, and validating performance tasks 2008, Guilford Press, New York.
23. Landis J.R., Koch G.G. The measurement of observer agreement for categorical data. Biometrics 1977, 33(1):159-174. doi:10.2307/2529310.
24. LoCasale-Crouch, J., Downer, J. T., & Hamre, B.K. (2011). Assessing teacher beliefs about intentional instruction. Unpublished manuscript.
25. LoCasale-Crouch J., Konold T., Pianta R., Howes C., Burchinal M., Bryant D., et al. Observed classroom quality profiles in state-funded pre-kindergarten programs and associations with teacher, program, and classroom characteristics. Early Childhood Research Quarterly 2007, 22(1):3-17. 10.1016/j.ecresq.2006.05.001.
26. Mashburn A.J., Pianta R.C., Hamre B.K., Downer J.T., Barbarin O., Bryant D., et al. Measures of classroom quality in prekindergarten and children's development of academic, language, and social skills. Child Development 2008, 79(3):732-749. doi: 10.1111/j.1467-8624.2008.01154.x.
27. Merrell K.W. Behavioral, social, and emotional assessment of children and adolescents 1999, Erlbaum, Mahwah, NJ.
28. MET Project Working with teachers to develop fair and reliable measures of effective teaching 2010, Retrieved from http://www.gatesfoundation.org/highschools/Documents/met-framing-paper.pdf.
29. MET Project Learning about teaching: Initial findings from the Measures of Effective Teaching project 2011, Retrieved from http://www.metproject.org/downloads/Preliminary_Findings-Research_Paper.pdf.
30. MET Project Measures of Effective Teaching (MET) 2011, Retrieved from http://www.gatesfoundation.org/united-states/Pages/measures-of-effective-teaching-fact-sheet.aspx.
31. Montgomery D. Helping teachers develop through classroom observation 2002, David Fulton, London, England.
32. Moon T.R., Hughes K.R. Training and scoring issues involved in large-scale writing assessments. Educational Measurement: Issues and Practice 2005, 21(2):15-19. doi:10.1111/j.1745-3992.2002.tb00088.x.
33. National Association of Child Care Resource and Referral Agencies Comparison of Quality Rating and Improvement Systems (QRIS) with Department of Defense standards for quality (No. 724-0714) 2009, Retrieved from http://www.naccrra.org/publications/naccrra-publications/comparison-qris-dod-quality-standards-2009.php.
34. National Child Care Information and Technical Assistance Center Child Care Bulletin Issue 32 2007, Child Care Bureau, Administration for Children and Families, U.S. Department of Health and Human Services, Fairfax, VA, Retrieved from http://nccic.acf.hhs.gov/ccb/issue32.pdf.
35. National Child Care Information and Technical Assistance Center Child care and development fund report of state and territory plans FY 2008-2009 2009, Retrieved from http://nccic.acf.hhs.gov/pubs/stateplan2008-09/index.html.
36. NICHD ECCRN A day in third grade: A large-scale study of classroom quality and teacher and student behavior. Elementary School Journal 2005, 105(3):305-323. doi: 10.1086/428746.
37. NICHD ECCRN Child-care effect sizes for the NICHD Study of Early Child Care and Youth Development. American Psychologist 2006, 61(2):99-116. doi: 10.1037/0003-066X.61.2.99.
38. Penny J.A. Reading high stakes writing samples: My life as a reader. Assessing Writing 2003, 8(3):192-215. 10.1016/j.asw.2003.08.001.
39. Pianta R.C., Allen J.P. Building capacity for positive youth development in secondary school classrooms: Changing teachers' interactions with students. Toward positive youth development: Transforming schools and community programs 2008, 21-40. Oxford University Press, New York, NY. M. Shinn, H. Yoshikawa (Eds.).
40. Pianta R.C., Hamre B.K. Conceptualization, measurement, and improvement of classroom processes: Standardized observation can leverage capacity. Educational Researcher 2009, 38(2):109. doi:10.3102/0013189X09332374.
41. Pianta R.C., Howes C., Burchinal M., Bryant D., Clifford R., Early D., et al. Features of pre-kindergarten programs, classrooms, and teachers: Do they predict observed classroom quality and child-teacher interactions?. Applied Developmental Science 2005, 9(3):144-159. doi:10.1207/s1532480xads0903_2.
42. Pianta R.C., La Paro K.M., Hamre B.K. Classroom Assessment Scoring System 2008, Brookes, Baltimore, MD.
43. Preschool Curriculum Evaluation Research Consortium Effects of preschool curriculum programs on school readiness (NCER No. 2008-2009) 2008, Institute of Education Sciences, National Center for Education Research, Washington, DC, Retrieved from http://ncer.ed.gov.
44. Raudenbush S.W., Bryk A.S. Hierarchical linear models: Applications and data analysis methods 2002, Sage, London, UK.
45. Raudenbush S.W., Sadoff S. Statistical inference when classroom quality is measured with error. Journal of Research on Educational Effectiveness 2008, 1(2):138-154. doi:10.1080/19345740801982104.
46. Rimm-Kaufman S.E., Curby T.W., Grimm K.J., Nathanson L., Brock L.L. The contribution of children's self-regulation and classroom quality to children's adaptive behaviors in the kindergarten classroom. Developmental Psychology 2009, 45(4):958-972. doi: 10.1037/a0015861.
47. Schaefer E.S., Edgerton M. Parent and child correlates of parental modernity. Parental belief systems 1985, 83-105. Erlbaum, Hillsdale, NJ. I.E. Sigel (Ed.).
48. Schafer W.D., Gagné P., Lissitz R.W. Resistance to confounding style and content in scoring constructed-response items. Educational Measurement: Issues and Practice 2005, 24(2):22-28. doi: 10.1111/j.1745-3992.2005.00008.
49. Singer J.D. Using SAS PROC MIXED to fit multilevel models, hierarchical models, and individual growth models. Journal of Educational and Behavioral Statistics 1998, 23(4):323. doi: 10.3102/10769986023004323.
50. Smith M., Dickinson D.K., Sangeorge A., Anastasopoulos L. Early language and literacy classroom observation (ELLCO) toolkit 2002, Brookes, Baltimore, MD.
51. Stoney L. Financing quality rating systems: Lessons learned 2004, Retrieved from http://www.wkkflearninglabs.org/upload_main/docs/UWFinancingQRSLessonsLrnd.pdf.
52. Teachstone How do I prepare for observer recertification? 2011, Retrieved from http://www.teachstone.org/certification/.
53. Teachstone Train-the-Trainer 2011, Retrieved from http://www.teachstone.org/training-programs/.
54. Tout K., Starr R., Soli M., Moodie S., Kirby G., Boller K. The Child Care Quality Rating System (QRS) Assessment: Compendium of quality rating systems and evaluations 2010, Child Trends, Washington, DC, Retrieved from http://www.acf.hhs.gov/programs/opre/cc/childcare_quality/compendium_qrs/qrs_compendium_final.pdf.
55. Tout K., Zaslow M., Halle T., Forry N. Issues for the next decade of Quality Rating and Improvement Systems (Issue Brief No. 3) 2009, Prepared by Child Trends for the Office of Planning, Research, and Evaluation, Administration for Children and Families, US Department of Health and Human Services, Washington, DC, Retrieved from http://www.childtrends.org/Files//Child_Trends-2009_5_19_RB_QualityRating.pdf.
56. U.S. Department of Health and Human Services, Administration for Children and Families, Office of Head Start Classroom Assessment Scoring System (Information Memorandum No. ACF-IM-HS-08-11) 2008, Brown, Patricia E, Washington, DC, Retrieved from http://www.acf.hhs.gov/programs/ohs/policy/im2008/acfimhs_08_11.html.
57. Zellman G.L., Perlman M. Child-care Quality Rating and Improvement Systems in five pioneer states: Implementation issues and lessons learned 2008, RAND Corporation, Arlington, VA, Retrieved from http://www.rand.org/pubs/monographs/2008/RAND_MG795.pdf.
58. Zill N., Resnick G., Kim K., McKey R.H., Clark C., Pai-Samant S., et al. Head Start FACES: Longitudinal findings on program performance (Progress Report 3) 2001, Retrieved from http://eric.ed.gov/PDFS/ED453969.pdf.
59. Zill N., Sorongon A., Kim K., Clark C., Woolverton M. FACES 2003 research brief 2006, U.S. Department of Health and Human Services, Administration for Children and Families, Washington, DC, Retrieved from www.acf.hhs.gov/programs/opre/hs/faces/index.html.