Validating Arguments for Observational Instruments: Attending to Multiple Sources of Variation

Educational Assessment

Volumn 17, Issue 2-3, 2012, Pages 88-106

  Hill, Heather C ^a   Charalambous, Charalambos Y ^b   Blazar, David ^a   McGinn, Daniel ^a   Kraft, Matthew A ^a   Beisiegel, Mary ^c   Humez, Andrea ^d   Litke, Erica ^a   Lynch, Kathleen ^a  

^a HARVARD GRADUATE SCHOOL OF EDUCATION   (United States)

^b UNIVERSITY OF CYPRUS   (Cyprus)

^c OREGON STATE UNIVERSITY   (United States)

^d BOSTON COLLEGE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84866657910     PISSN: 10627197     EISSN: None     Source Type: Journal    
DOI: 10.1080/10627197.2012.715019     Document Type: Article

References (36)

1. Allen, J. P., Pianta, R. C., Gregory, A., Mikami, A. and Lun, J. 2011. An interaction-based approach to enhancing secondary school instruction and student achievement. Science, 333: 1034-1037.
2. American Educational Research Association/American Psychological Association/National Council on Measurement in Education. 1999. Standards for educational and psychological testing Washington, DC: AERA.
3. Ball, D. L., Thames, M. H. and Phelps, G. 2008. Content knowledge for teaching: What makes it special?. Journal of Teacher Education, 59: 389-407.
4. Bangert, A. W. 2009. Building a validity argument for the community of inquiry survey instrument. The Internet and Higher Education, 12: 104-111.
5. Bell, C., Gitomer, D., McCaffrey, D., Hamre, B., Pianta, R. and Qi, Y. 2012/this issue. An argument approach to observation protocol validity. Educational Assessment, 17: 62-87.
6. Brennan, R. L. 2001. Generalizability theory New York, NY: Springer-Verlag.
7. Chapelle, C. A., Enright, M. K. and Jamieson, J. M., eds. 2007. Building a validity argument for the Test of English as a Foreign Language™ New York, NY: Routledge.
8. Cohen, D. K., Raudenbush, S. and Ball, D. L. 2003. Resources, instruction and research. Educational Evaluation and Policy Analysis, 25: 119-142.
9. Cronbach, L. J. 1988. "Five perspectives on validity argument.". In Test validity Edited by: Wainer, H. and Braun, H. 3-17. Hillsdale, NJ: Erlbaum.
10. Fisher, R. A. 1921. On the "probable error" of a coefficient of correlation deduced from a small sample. Metron, 1(4): 3-32.
11. Grossman, P., Loeb, S., Cohen, J., Hammerness, K., Wyckoff, J., Boyd, D. and Lankford, H. May 2010. Measure for measure: The relationship between measures of instructional practice in middle school English Language Arts and teachers' value-added scores. May, Cambridge, MA: National Bureau of Economic Research. (NBER Working Paper 16015)
12. Hawkins, R. E., Margolis, M. J., Dunning, S. J. and Norcini, J. J. 2010. Constructing a validity argument for the mini-clinical evaluation exercise: A review of the research. Academic Medicine, 85: 1453-1461.
13. Hill, H. C., Blunk, M., Charalambous, C., Lewis, J., Phelps, G. C., Sleep, L. and Ball, D. L. 2008. Mathematical knowledge for teaching and the mathematical quality of instruction: An exploratory study. Cognition and Instruction, 26: 430-511.
14. Hill, H. C., Charalambous, C. Y. and Kraft, M. 2012. When rater reliability is not enough: Observational systems and a case for the G-study. Educational Researcher, 41(2): 56-64.
15. Hill, H. C. and Herlihy, C. 2011. "Prioritizing teaching quality in a new system of teacher evaluation.". In Education Outlook Retrieved fromhttp://www.aei.org/outlook/101089
16. Hill, H. C., Kapitula, L. R. and Umland, K. L. 2011. A validity argument approach to evaluating value-added scores. American Educational Research Journal, 48: 794-831.
17. Hill, H. C., Umland, K. U., Litke, E. and Kapitula, L. 2012. Teacher quality and quality teaching: Examining the relationship of a teacher assessment to practice. American Journal of Education, 118: 489-519.
18. Kane, M. T. 2001. Current concerns in validity theory. Journal of Educational Measurement, 38: 319-342.
19. Kane, M. T. 2004. Certification testing as an illustration of argument-based validation. Measurement: Interdisciplinary Research and Perspectives, 2: 135-170.
20. Kane, M. T. 2006. "Validation.". In Educational measurement, 4th ed. Edited by: Brennan, R. L. 17-64. Westport, CT: American Council on Education/Praeger.
21. Kane, T. J. and Staiger, D. O. 2012. Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains. Seattle, WA: Bill & Melinda Gates Foundation. Retrieved fromhttp://www.metproject.org/reports.php
22. Kane, T. J., Taylor, E. S., Tyler, J. and Wooten, A. March 2010. Identifying effective classroom practices using student achievement data March, Cambridge, MA: National Bureau of Economic Research. (NBER Working Paper 15803)
23. Kennedy, M. M. 2010. Attribution error and the quest for teaching quality. Educational Researcher, 39: 591-598.
24. Koretz, D., Stecher, B., Klein, S. and McCaffrey, D. 1994. The Vermont Portfolio Assessment Program: Findings and implications. Educational Measurement: Issues and Practice, 13(3): 5-16.
25. Lane, S., Liu, M., Ankenmann, R. D. and Stone, C. A. 1996. Generalizability and validity of a mathematics performance assessment. Journal of Educational Measurement, 3: 71-92.
26. Llosa, L. 2008. Building and supporting a validity argument for a standards-based classroom assessment of English proficiency based on teacher judgments. Educational Measurement: Issues and Practice, 27(3): 32-42.
27. Matsumura, L. C., Garnier, H. E., Slater, S. C. and Boston, M. D. 2008. Toward measuring instructional interactions "at-scale.". Educational Assessment, 13: 267-300.
28. McCollum, J. A., Hemmeter, M. L. and Hsieh, W. Coaching teachers for emergent literacy instruction using performance-based feedback. Topics in Early Childhood Education., (in press)
29. McGaghie, W. C., Cohen, E. R. and Wayne, D. B. 2011. Are United States medical licensing exam step 1 and 2 scores valid measures for postgraduate medical residency selection decisions?. Academic Medicine, 86(1): 48-52.
30. Newton, X. 2010. Developing indicators of classroom practice to evaluate the impact of a district mathematics reform initiative: A generalizability analysis. Studies in Educational Evaluation, 36: 1-13.
31. Rosenbaum, P. R. 2002. Observational studies, 2nd ed. New York, NY: Springer-Verlag.
32. Rowland, T., Turner, F., Thwaites, A. and Huckstep, P. 2009. Developing primary mathematics teaching: Reflecting on practice with the Knowledge Quartet London, UK: Sage.
33. Sawchuck, S. 2009. New teacher-evaluation systems face obstacles: Stimulus funds require districts to revamp teacher yardsticks. Education Week, Retrieved fromhttp://www.edweek.org/ew/articles/2009/12/11/15evaluate.h29.html
34. Schilling, S. G. and Hill, H. C. 2007. Assessing measures of Mathematical Knowledge for Teaching: A validity argument approach. Measurement: Interdisciplinary Research and Perspectives, 5: 70-80.
35. Shavelson, R. J. and Webb, N. M. 1991. Generalizability theory: A primer Newbury Park, CA: Sage.
36. Taylor, E. S. and Tyler, J. H. March 2011. The effect of evaluation on performance: Evidence from longitudinal student achievement data of mid-career teachers March, Cambridge, MA: National Bureau of Economic Research. (NBER Working Paper 16877)