A validity argument approach to evaluating teacher value-added scores

American Educational Research Journal

Volumn 48, Issue 3, 2011, Pages 794-831

  Hill, Heather C ^a   Kapitula, Laura ^b   Umland, Kristin ^c  

^a HARVARD GRADUATE SCHOOL OF EDUCATION   (United States)

^b CALVIN COLLEGE   (United States)

^c UNIVERSITY OF NEW MEXICO   (United States)

Author keywords

accountability; educational policy; reliability; teacher assessment; validity

Indexed keywords

EID: 79955844752     PISSN: 00028312     EISSN: None     Source Type: Journal    
DOI: 10.3102/0002831210387916     Document Type: Article

References (63)

1. Standards for educational and psychological testing. Washington, DC: American Educational Research Association; 1999:.
2. Amrein-BeardsleyA. Methodological concerns about the education value-added assessment system. Educational Researcher. 2008;37 (2): 65-75.
3. BallD. L.,ForzaniF. M. What makes education research "educational"?. Educational Researcher. 2007;36 (9): 529-540.
4. BallD. L.,ThamesM. H.,PhelpsG. Content knowledge for teaching: What makes it special?. Journal of Teacher Education. 2008;59 (5): 389-407.
5. BallouD.,SandersW.,WrightP. Controlling for student background in value-added assessment of teachers. Journal of Educational and Behavioral Statistics. 2004;29 (1): 37-65.
6. BoydD.,LankfordR. H.,LoebS.,WycoffJ. School finance and teacher quality: American Education Finance Association 2003 yearbook. PleckiM. L. MonkD. H., ed. Larchmont, NY: Eye on Education; 2003:55-84.
7. Buddin, R., & Zamarro, G. (2008). Teacher quality, teacher licensure tests and studentachievement (RAND working paper). Retrieved from http://www.rand.org/pubs/working_papers/2008/RAND_WR555.pdfValidity Argument Approach to Evaluating Teacher Scores827
8. Center for Educator Compensation Reform. (2007). The evolution of performance pay inFlorida. Retrieved from http://www.cecr.ed.gov/guides/summaries/FloridaCaseSummary.pdf
9. Center for Educator Compensation Reform. (2010). Compensation reform in action.Retrieved from http://cecr.ed.gov/reformInAction/
10. ClotfelterC.,LaddH. F.,VigdorJ. Teacher quality and minority achievement gaps. Durham, NC: Terry Sanford Institute of Public Policy; 2004:.
11. CohenD. K.,RaudenbushS.,BallD. L. Resources, instruction and research. Educational Evaluation and Policy Analysis. 2003;25: 119-142.
12. Colorado teacher-evaluation bill enacted. (2010, May 18). Education Week. Retrievedfrom http://www.edweek.org/ew/articles/2010/05/19/32colorado.h29.htm
13. Dillon, S. (2009, November 10). States compete for federal school dollars. New YorkTimes. Retrieved from http://www.nytimes.com/2009/11/11/education/11educ.html?_r=1&sq=colorado%20education&st=cse&adxnnl=1&scp=5&adxnnlx=1277474473-7wHNM/8mH25t3oRTa27+vg
14. Dillon, S. (2010, May 31). States create flood of education bills. New York Times.Retrieved from http://www.nytimes.com/2010/06/01/education/01educ.html?scp=1&sq=colorado%20education&st=cse
15. Duncan, A. (2009). The race to the top begins: Remarks by Secretary Arne Duncan.Retrieved from http://www.ed.gov/news/speeches/2009/07/07242009.html
16. Editorial: The New Haven model. (2009, October 28). New York Times. Retrieved fromhttp://www.nytimes.com/2010/05/03/opinion/03mon3.html?_r=1&scp=1&sq=Editorial:%20The%20New%20Haven%20Model&st=cse
17. GordonR.,KaneT. J.,StaigerD. O. Identifying effective teachers using performance on the job. Washington, DC: Brookings Institution 2006:.
18. GrubbW. N. Multiple resources, multiple outcomes: Testing the "improved" school finance with NELS88. American Educational Research Journal. 2008;45 (1): 104-144.
19. HanushekE. A. The single salary schedule and other issues of teacher pay. Peabody Journal o Education. 2007;82 (4): 574-586.
20. Hanushek, E. A., Kain, J. F., O'Brien, D. M., & Rivkin, S. G. (2005). The market for teacher quality. Unpublished manuscript.
21. HillH. C. Mathematical knowledge of middle school teachers: Implications for the No Child Left Behind Policy initiative. Educational Evaluation and Policy Analysis. 2007;29: 95-114.
22. HillH. C.,BlunkM.,CharalambousC.,LewisJ.,PhelpsG. C.,SleepL.,BallD. L. Mathematical knowledge for teaching and the mathematical quality of instruction: An exploratory study. Cognition and Instruction. 2008;26 (4): 430-511.
23. HillH. C.,RowanB.,BallD. L. Effects of teachers' mathematical knowledge for teaching on student achievement. American Educational Research Journal. 2005;42 (2): 371-406.
24. IshiiJ.,RivkinS. G. Impediments to the estimation of teacher value added. Education Finance and Policy. 2009;4 (4): 520-536.
25. JacobB. A.,LefgrenL. Principals as agents: Subjective performance measurement in education. Cambridge, MA: Harvard University, John F. Kennedy School of Government; 2005:.
26. KaneM. Current concerns in validity theory. Journal of Educational Measurement. 2001;38 (4): 319-342.
27. KaneM. Certification testing as an illustration of argument-based approach validation. Measurement: Interdisciplinary Research and Perspectives. 2004;2 (3): 135-170.
28. KaneM. Educational measurement. 4th ed. BrennanR. L., ed. Westport, CT: Praeger; 2006:18-64.
29. Kane, T. J., Rockoff, J. E., & Staiger, D. O. (2006). What does certification tell us about teacher effectiveness? Evidence from New York City. Unpublished manuscript, Harvard University.
30. KimballS. M.,WhiteB.,MilanowskiA. T. Examining the relationship between teacher evaluation and student assessment results in Washoe County. Peabody Journal of Education. 2004;79 (4): 54-78.
31. Kane, T. J., Rockoff, J. E., & Staiger, D. O. (2006). What does certification tell us about teacher effectiveness? Evidence from New York City. Unpublished manuscript, Harvard University.
32. KoedelC.,BettsJ. R. Does student sorting invalidate value-added models of teacher-effectiveness? An extended analysis of the Rothstein critique. Education Finance and Policy.:.
33. KoretzD. Limitations in the use of achievement tests. Journal of Human Resources. 2008;27 (4): 752-777.
34. KupermintzH. Teacher effects and teacher effectiveness: A validity investigation of the Tennessee value added assessment system. Educational Evaluation and Policy Analysis. 2003;25: 287-298.
35. Lewin, T. (2010). School chief dismisses 241 teachers in Washington. New York Times. Retrieved from www.nytimes.com/2010/07/24/education/24teachers.html
36. LinnR. L. Methodological issues in achieving school accountability. Journal of Curriculum Studies. 2008;40 (6): 699-711.
37. LockwoodJ. R.,LouisT. A.,McCaffreyD. F. Uncertainty in rank estimation: Implications for value-added modeling accountability systems. Journal of Educational and Behavioral Statistics. 2002;27 (3): 255-270.
38. LockwoodJ. R.,McCaffreyD. F.,HamiltonL. S.,StecherB.,LeV.,MartinezJ. F. The sensitivity of value-added teacher effect estimates to different mathematics achievement measures. Journal of Educational Measurement. 2007;44 (1): 47-67.
39. LockwoodJ. R.,McCaffreyD. F.,MarianoL. T.,SetodjiC. M. Bayesian methods for scalable multivariate value-added assessment. Journal of Educational and Behavioral Statistics. 2007;32 (2): 125-150.
40. McCaffreyD. F.,KoretzD.,LockwoodJ. R.,HamiltonL. S. Evaluating value-added models for teacher accountability. Santa Monica, CA: RAND; 2003:.
41. McCaffreyD. F.,LockwoodJ.,KoretzD.,LouisT. A.,HamiltonL. Models for value-added modeling of teacher effects. Journal of Educational and Behavioral Statistics. 2004;29 (1): 67-101.
42. McCaffreyD. F.,SassT. R.,LockwoodJ. R.,MihalyK. The intertemporal variability of teacher effect estimates. Education, Finance and Policy. 2009;4 (4): 572-606.
43. MedleyD. M.,CokerH. How valid are principals' judgments of teacher effectiveness?. Phi Delta Kappan. 1987;69: 138-140.
44. Messick, S. (1988). The once and future issues of validity. Assessing the meaning and consequences of measurement. In H. Wainer & H. Braun (Eds.), Test validity (pp. 33-45). Hillsdale, NJ: Lawrence Erlbaum.
45. MessickS. Educational measurement. 3rd ed. LinnR. L., ed. New York: American Council on Education and Macmillan; 1989:13-103.
46. MilanowskiA. The relationship between teacher performance evaluation scores and student achievement: Evidence from Cincinnati. Peabody Journal of Education. 2004;79 (4): 33-53.
47. NyeB.,KonstantopoulosS.,HedgesL. V. How large are teacher effects?. Educational Evaluation and Policy Analysis. 2004;26: 237-257.
48. PapayJ. P. Different tests, different answers: The stability of teacher value-added estimates across outcome measures. American Educational Research Journal.:.
49. PiantaR. C.,BelskyJ.,VandergriftN.,HoutsR.,MorrisonF. J. Classroom effects on children's achievement trajectories in elementary school. American Educational Research Journal. 2008;45 (2): 365-397.
50. RockoffJ. E. The impact of individual teachers on student achievement: Evidence from panel data. American Economic Review. 2004;94 (2): 247-252.
51. Rockoff, J. E., Staiger, D. O., Kane, T. J., & Taylor, E. S. (2010). Information and employee evaluation: Evidence from a randomized intervention in public schools (Working paper 16240). Cambridge, MA: National Bureau of Economic Research. Retrieved from http://www.nber.org/papers/w16240
52. RowanB.,CorrentiR.,MillerR. J. What large-scale, survey research tells us about teacher effects on student achievement: Insights from the "Prospects" study of elementary schools. Teachers College Record. 2002;104 (8): 1525-1567.
53. Sanders, W. L. (2006, October). Comparisons among various educational assessment value-added models. Paper presented at the Power of Two-National Value- Added Conference, Columbus, OH.
54. SandersW. L.,RiversJ. C. Cumulative and residual effects of teachers on future student academic achievement. Knoxville: University of Tennessee, Value-Added Research and Assessment Center; 1996:.
55. SandersW. L.,SaxtonA. M.,HornS. P. Grading teachers, grading schools: Is student achievement a valid evaluation measure?. MillmanJ., ed. Thousand Oaks, CA: Corwin Press; 1997:137-162.
56. Sanders, W. L., & Wright, S. P. (2008). A response to Amrein-Beardsley (2008) "Methodological concerns about the education value-added assessment system." Retrieved from http://www.sas.com/govedu/edu/services/Sanders_Wright_ response_to_Amrein-Beardsley_4_14_2008.pdf
57. SchacterJ.,ThumY. M. Paying for high and low-quality teaching. Economics of Education Review. 2004;23: 411-430.
58. TekweC. D.,CarterR. L.,MaC.,AlginaJ.,LucasM. E.,RothJ.,ResnickM. B. An empirical comparison of statistical models for value-added assessment of school performance. Journal of Educational and Behavioral Statistics. 2004;29 (1): 11-35.
59. Weerasinghe, D. (2008). How to compute school and classroom effective indices: The value-added model implemented in Dallas Independent School District. Dallas, TX: Office of Institutional Research Dallas Independent School District. Retrieved July 16, 2010, from https://mydata.dallasisd.org/docs/CEI/SEI_CEI_ Research.pdf
60. WeingartenR. How to fix No Child Left Behind. New York Times. 2007;: WK7.
61. WeisbergD.,SextonS.,MulhernJ.,KeelingD. The widget effect: Our national failure to acknowledge and act on differences in teacher effectiveness. Brooklyn, NY: New Teacher Project; 2009:.
62. Wiley, E. W., Spindler, E. R., & Subert, A. N. (2010). Denver ProComp: An Outcomes Evaluation of Denver's Alternative Teacher Compensation System 2010 Report. Boulder, CO: University of Boulder at Bolder, School of Education. Retrieved from http://static.dpsk12.org/gems/newprocomp/ProCompOutcomesEvaluation April2010final.pdf
63. WrightS. P.,HornS. P.,SandersW. L. Teacher and classroom context effects on student achievement: Implications for teacher evaluation. Journal of Personnel Evaluation in Education. 1997;1 (1): 57-67.