The SAS education value-added assessment system (SAS® EVAAS®) in the Houston independent school district (HISD): Intended and unintended consequences;El sas sistema de evaluación de la educación de valor agregado (sas® evaas®): Algunos efectos intencionales y no intencionales en un sistema escolar urbano de gran tamaño

Education Policy Analysis Archives

Volumn 20, Issue , 2012, Pages 1-31

  Amrein Beardsley, Audrey ^a   Collins, Clarin ^a  

^a ARIZONA STATE UNIVERSITY   (United States)

Author keywords

Accountability; Teacher effectiveness; Teacher evaluation; Teacher quality; Value added models (VAMs)

Indexed keywords

EID: 84860904353     PISSN: None     EISSN: 10682341     Source Type: Journal    
DOI: 10.14507/epaa.v20n12.2012     Document Type: Article

References (82)

1. American Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME). (2000). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
2. American Educational Research Association (AERA) (2000). Position statement on high-stakes testing in PreK-12 education. Retrieved from http://www.aera.net/AboutAERA/AERARulesPolicies/AERAPolicyStatements/PositionStatementonHighStakesTesting/tabid/11083/Default.aspx
3. Amrein-Beardsley, A. (2008). Methodological concerns about the Education Value-Added Assessment System (EVAAS). Educational Researcher, 37(2), 65-75. doi: 10.3102/0013189X08316420
4. Amrein-Beardsley, A. Haladyna, T, & Polasky, S. A. (2012). Imagining a new system: Using multiple measures and inter-indicator consistency to measure value-added. Paper presented at the annual convention of the American Association of Colleges for Teacher Education (AACTE), Chicago, IL.
5. Au, W. (2010, Winter). Neither fair nor accurate: Research-based reasons why high-stakes tests should not be used to evaluate teachers. Rethinking Schools. Retrieved from http://www.rethinkingschools.org/archive/25_02/25_02_au.shtml
6. Baeder, J. (2010, December 21). Gates' measures of effective teaching study: More value-added madness. Education Week. Retrieved from http://blogs.edweek.org/edweek/on_performance/2010/12/gates_measures_of_effective_teaching_study_more_value-added_madness.html
7. Baker, B. D. (2012, March 31). Firing teachers based on bad (VAM) versus wrong (SGP) measures of effectiveness: Legal note. School Finance 101. Retrieved from http://schoolfinance101.wordpress.com/2012/03/31/firing-teachers-based-on-bad-vam-versus-wrong-sgp-measures-of-effectiveness-legal-note/?utm_source=feedburner&utm_medium=email&utm_campaign=Feed%3A+GLCWorthARead+%28Worth+A+Read%29&blogsub=confirming#subscribe-blog
8. Baker, E. L., Barton, P. E., Darling-Hammond, L., Haertel, E., Ladd, H. F., Linn, R. L., Ravitch, D., Rothstein, R., Shavelson, R. J., F Shepard, L. A. (2010). Problems with the use of student test scores to evaluate teachers. Washington, DC: Economic Policy Institute. Retrieved from http://www.epi.org/publications/entry/bp278
9. Ballou, D., Sanders, W. L., & Wright, P. (2004). Controlling for student background in value-added assessment of teachers. Journal of Educational and Behavioral Statistics, 29(1), 37-66. doi:10.3102/10769986029001037
10. Banchero, S. & Kesmodel, D. (2011, September 13). Teachers are put to the test. The Wall Street Journal. Retrieved from
11. http://online.wsj.com/article/SB10001424053111903895904576544523666669018.html?mod=googlenews_wsj
12. Braun, H. I. (2005). Using student progress to evaluate teachers: A primer on value-added models. Princeton, NJ: Educational Testing Service. Retrieved from http://www.ets.org/Media/Research/pdf/PICVAM.pdf
13. Brophy, J. (1973). Stability of teacher effectiveness. American Educational Research Journal, 10(3), 245-252. doi:10.2307/1161888
14. Campbell, D. (1975). Degrees of freedom and the case study. Comparative Political Studies, 8, 178-185.
15. Campbell, D. T. & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. Chicago, IL: Rand McNally & Company.
16. CCSSO. (2010, October). Teacher evaluation literature review. Washington, DC: Council of Chief State School Officers.
17. Cody, C. A., McFarland, J., Moore, J. E., & Preston, J. (2010, August). The evolution of growth models. Public Schools of North Carolina. Retrieved from http://www.dpi.state.nc.us/docs/intern-research/reports/growth.pdf
18. ®) from teachers' perspectives in the Houston Independent School District. (Unpublished doctoral dissertation). Arizona State University, Tempe.
19. Corcoran, S. P. (2010). Can teachers be evaluated by their students' test scores? Should they be? The use of value- added measures of teacher effectiveness in policy and practice. Providence, RI: Annenberg Institute for School Reform. Retrieved from http://www.annenberginstitute.org/products/Corcoran.php
20. Creswell, J. W. (2008). Educational research: Planning, conducting, and evaluating. Quantitative and qualitative research. (3rd ed.). Upper Saddle River, NJ: Merrill Prentice Hall.
21. Day, C., Sammons, P., & Gu, Q. (2008). Combining qualitative and quantitative methodologies in research on teachers' lives, work, and effectiveness: From integration to synergy. Educational Researcher, 37(6), 330-342. doi:10.3102/0013189X08324091
22. Derringer, P. (2010, August). RTT in Tennessee: Assessment done right. Technology and Learning, 31(1), 40. Retrieved from http://www.techlearning.com/article/31572
23. Eckert, J. M., & Dabrowski, J. (2010, May). Should value-added measures be used for performance pay? Phi Delta Kappan, 91(8), 88-92.
24. Feagin, J., & Orum, A. (1991). A case for the case study. Chapel Hill, NC: The University of North Carolina Press.
25. Flyvbjerg, B. (2011). Five misunderstandings about case-study research. Qualitative Inquiry, 12(2), 219- 245. doi:10.1177/1077800405284363
26. Garland, S. (2012, February 7). Tennessee teacher evaluation systems have rough road ahead. The Huffington Post. Retrieved from http://www.huffingtonpost.com/2012/02/07/tennessee-teacher-evaluat_n_1260790.html?page=1
27. Gerring, J. (2004). What is a case study and what is it good for? The American Political Science Review, 98(2), 341-354. doi:10.1017/S0003055404001182
28. Glaser, B., & Strauss, A. (1967). The discovery of grounded theory: Strategies for qualitative research. Chicago, IL: Aldine.
29. Greene, J. C., Caracelli, V. J., & Graham, W. D. (1989). Toward a conceptual framework for mixedmethod evaluation designs. Educational Evaluation and Policy Analysis, 11(3), 255-274.
30. Guba, E. G., & Lincoln, Y. S. (1981). Effective evaluation: Improving the usefulness of evaluation results through responsive and naturalistic approaches. San Francisco, CA: Jossey-Bass.
31. Haertel, E. (2011). Using student test scores to distinguish good teachers from bad. Paper presented at the Annual Conference of the American Educational Research Association (AERA), New Orleans, LA.
32. Harris, D. N. (2009). Teacher value-added: Don't end the search before it starts. Journal of Policy Analysis and Management, 28(4), 693-700. doi:10.1002/pam.20464
33. Harris, D. N. (2011). Value-added measures in education: What every educator needs to know. Cambridge, MA: Harvard Education Press.
34. Hill, H. C. (2009). Teacher value-added: Don't end the search before it starts. Journal of Policy Analysis and Management, 28(4), 700-709.
35. Hill, H. C., Kapitula, L., & Umland, K. (2011, June). A validity argument approach to evaluating teacher value-added scores. American Educational Research Journal, 48(3), 794-831. doi:10.3102/0002831210387916
36. Ho, A. D., Lewis, D. M., & Farris, J. L. (2009). The dependence of growth-model results on proficiency cut scores. Educational Measurement: Issues and Practice, 28(4), 15-26. doi:10.1111/j.1745-3992.2009.00159.x
37. Houston Independent School District (HISD). (2010). Accelerating Student Progress and Increasing Results and Expectations (ASPIRE) at HISD: A guide for parents and our community. Retrieved from www.houstonisd.org/.../PDF/ASPIRE_ParentGuide_2010114web.pdf
38. Houston Independent School District (HISD). (2011). Board of education workshop 2011-12 budget update [PowerPoint slides]. Retrieved from www.houstonisd.org/.../Home/.../BudgetUpdate_April212011.ppt
39. Ishii, J., & Rivkin, S. G. (2009). Impediments to the estimation of teacher value added. Education Finance and Policy, 4(4), 520-536. doi:10.1162/edfp.2009.4.4.520
40. Jacob, B. A., & Lefgren, L. (2007, June). Can principals identify effective teachers? Evidence on subjective performance evaluation in education. Retrieved from econ.byu.edu/faculty/Lefgren/Assets/papers/principals.pdf
41. Johnson, R. B. & Onwuegbuzie, A. J. (2004) Mixed methods research: A research paradigm whose time has come. Educational Researcher, 33(7), 14-26. doi:10.3102/0013189X033007014
42. Kane, T. J., & Staiger, D. O. (2008, March). Are teacher-level value-added estimates biased? An experimental validation of non-experimental estimates. Paper presented at the National Conference on Value- Added Modeling, Madison, WI.
43. Kennedy, M. M. (2010). Attribution error and the quest for teacher quality. Educational Researcher, 39(8), 591-598. doi:10.3102/0013189X10390804
44. Koedel, C., & Betts, J. R. (2007, April). Re-examining the role of teacher quality in the educational production function. Working Paper No. 2007-03. Nashville, TN: National Center on Performance Initiatives.
45. ® make Wake schools better? Raleigh Public Record. Retrieved from http://www.raleighpublicrecord.org/news/2011/06/01/will-evaas-make-wake-schools-better-part-ii/
46. Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Newbury Park, CA: Sage.
47. Linn, R. L. (2008). Methodological issues in achieving school accountability. Journal of Curriculum Studies, 40, 699-711.
48. Lowrey, A. (2012, January 6). Big study links good teachers to lasting gain. The New York Times. Retrieved from http://www.nytimes.com/2012/01/06/education/big-study-links-good-teachers-to-lasting-gain.html?emc=eta1#comments
49. McCaffrey, D. F., Lockwood, J. R., Koretz, D. M., & Hamilton, L. S. (2003). Evaluating value-added models for teacher accountability. Santa Monica, CA: Rand.
50. McCaffrey, D. F., Lockwood, J. R., Koretz, D., Louis, T. A., & Hamilton, L. (2004a). Let's see more empirical studies on value-added modeling of teacher effects: A reply to Raudenbush, Rubin, Stuart and Zanutto, and Reckase. Journal of Educational and Behavioral Statistics, 29(1), 139-143.
51. McCaffrey, D. F., Lockwood, J. R., Koretz, D., Louis, T. A., & Hamilton, L. (2004b). Models for value-added modeling of teacher effects. Journal of Educational and Behavioral Statistics, 29(1), 67-101. doi:10.3102/10769986029001067
52. Mellon, E. (2010, January 14). HISD moves ahead on dismissal policy: In the past, teachers were rarely let go over poor performance, data show. The Houston Chronicle. Retrieved from http://www.chron.com/disp/story.mpl/metropolitan/6816752.html
53. Milanowski, A., Kimball, S. M., & White, B. (2004). The relationship between standards-based teacher evaluation scores and student achievement: Replication and extensions at three sites. CPRE-UW Working Paper Series TC-04-01, Madison, WI: University of Wisconsin-Madison, Wisconsin. Center for Education Research, Consortium for Policy Research in Education.
54. Monk, David H. (1987). Assigning elementary pupils to their teachers. Elementary School Journal, 88(2), 167-187. doi:10.1086/461531
55. Nelson, F. H. (2011, April). A guide for developing growth models for teacher development and evaluation. Paper presented at the Annual Conference of the American Educational Research Association (AERA), New Orleans, LA.
56. Newton, X., Darling-Hammond, L., Haertel, E., & Thomas, E. (2010) Value-added modeling of teacher effectiveness: An exploration of stability across models and contexts. Education Policy Analysis Archives, 18(23). Retrieved from http://epaa.asu.edu/ojs/article/view/810
57. Otterman, S. (2010, December 26). Hurdles emerge in rising effort to rate teachers. New York Times. Retrieved from http://www.nytimes.com/2010/12/27/nyregion/27teachers.html
58. Papay, J. P. (2010). Different tests, different answers: The stability of teacher value-added estimates across outcome measures. American Educational Research Journal, 48(1), 163-193. doi:10.3102/0002831210362589
59. Patton, M. Q. (2001). Qualitative evaluation and research methods (3rd ed.). Thousand Oaks, CA: Sage Publications, Inc.
60. PDAS. (2004). Professional Development Appraisal System (PDAS). Austin, TX. Education Service Center Region XIII. Retrieved from http://www5.esc13.net/pdas/
61. Race to the Top. (2009, November). Race to the Top program: Executive summary. Washington, DC: US Department of Education. Retrieved from http://www2.ed.gov/programs/racetothetop/executive-summary.pdf
62. Ragin, C. C. & Becker, H. S. (2000). Cases of what is a case? In C. C. Ragin & H. S Becker, What is a case? Exploring the foundations of social inquiry, pp. 1-17. Cambridge, UK: The Press Syndicate of The University of Cambridge.
63. Ravitch, D. (2012, February 21). No student left untested. The New York Review of Books. http://www.nybooks.com/blogs/nyrblog/2012/feb/21/no-student-left-untested/
64. Robelen, E. W. (2012, January 9). Yardsticks vary by nation in calling education to account. Education Week. Retrieved from http://www.edweek.org/ew/articles/2012/01/12/16testing.h31.html?tkn=ZRXFgiQ5krPVo%2FsHmf1v%2Bh33GqSq%2ByE1LBEQ&cmp=ENL-EU-NEWS1&intc=EW-QC12-ENL
65. Rothstein, J. (2009, January 11). Student sorting and bias in value-added estimation: Selection on observables and unobservables. Cambridge, MA: The National Bureau of Economic Research. Retrieved from http://www.nber.org/papers/w14607
66. Sanders, W. L. (2000). Value-added assessment from student achievement data: Opportunities and hurdles. Journal of Personnel Evaluation in Education, 14(4), 329-339.
67. Sanders, W. L., & Horn, S. P. (1994). The Tennessee value-added assessment system (TVAAS): Mixed-model methodology in educational assessment. Journal of Personnel Evaluation in Education, 8, 299-311.
68. Sanders, W. L., & Horn, S. P. (1998). Research findings from the Tennessee value-added assessment system (TVAAS) database: Implications for educational evaluation and research. Journal of Personnel Evaluation in Education, 12(3), 247-256.
69. Sanders, W. L., Saxton, A. M., & Horn, S. P. (1997). The Tennessee Value-Added Accountability System: A quantitative, outcomes-based approach to educational assessment. In J. Millman (Ed.), Grading teachers, grading schools: Is student achievement a valid evaluation measure? (pp. 137-162). Thousand Oaks, CA: Corwin Press.
70. Sanders, W. L., & Wright, S. P. (2008, April 14). A response to Amrein-Beardsley (2008):"Methodological concerns about the Education Value-Added Assessment System." Available: www.sas.com/govedu/edu/services/Sanders_Wright_response_to_Amrein-Beardsley_4_14_2008.pdf
71. ®. Retrieved from http://www.sas.com/resources/asset/Response_to_Criticisms_of_SAS_EVAAS_11-13-09.pdf
72. SAS. (2007). Resource guide for value-added reporting. Cary, NC. SAS Institute. Retrieved from www.dpi.state.nc.us/docs/evaas/guide/resourceguide.pdf
73. ® for K-12: Assess and predict student performance with precision and reliability. Retrieved from http://www.sas.com/govedu/edu/k12/evaas/index.html
74. Schochet, P. Z. & Chiang, H. S. (2010, July). Error rates in measuring teacher and school performance based on student test score gains. U.S. Department of Education. Retrieved from http://ies.ed.gov/ncee/pubs/20104004/
75. Shaw, L. H. & Bovaird, J. A. (2011, April). The impact of latent variable outcomes on value-added models of intervention efficacy. Paper presented at the Annual Conference of the American Educational Research Association (AERA), New Orleans, LA.
76. Sparks, S. D. (2011, November 15). Value-added formulas strain collaboration. Education Week. Retrieved from http://www.edweek.org/ew/articles/2011/11/16/12collab-changes.h31.html?tkn=OVMFb8PQXxQi4wN6vpelNIr7%2BNhOFCbi71mI&intc=es
77. Tekwe C. D., Carter, R. L., Ma, C., Algina, J., Lucas, M. E., Roth, J., et al. (2004). An empirical comparison of statistical models for value-added assessment of school performance. Journal of Educational and Behavioral Statistics, 29(1), 11-36. doi:10.3102/10769986029001011
78. Thomas, G. (2011a) A typology for the case study in social science following a review of definition, discourse, and structure. Qualitative Inquiry, 17(6), 511-521. doi:10.1177/1077800411409884
79. Wilson, M., Hallman, P. J., Pecheone, R., & Moss, P. (2007, October). Using student achievement test scores as evidence of external validity for indicators of teacher quality: Connecticut's Beginning Educator Support and Training [BEST] Program. Retrieved from http://edpolicy.stanford.edu/pages/pubs/pubs.html
80. Wright, S. P., Horn, S. P., & Sanders, W. L. (1997). Teacher and classroom context effects on student achievement: Implications for teacher evaluation. Journal of Personal Evaluation in Education, 11, 57-67.
81. ® statistical models. Retrieved from http://www.sas.com/resources/asset/SAS-EVAAS-Statistical-Models.pdf
82. Yin, R. (1994). Case study research: Design and methods (2nd ed.). Beverly Hills, CA: Sage Publishing.