Distorting value added: The use of longitudinal, vertically scaled student achievement data for growth-based, value-added accountability

Journal of Educational and Behavioral Statistics

Volumn 31, Issue 1, 2006, Pages 35-62

  Martineau, Joseph A ^a,b  

^a Michigan Department of Education   (United States)

^b Office of Educational Assessment and Accountability   (United States)

Author keywords

Accountability; Measurement of change; Multidimensionality; School effects; Teacher effects; Validity; Value added models; Vertical scaling

Indexed keywords

EID: 33646413144     PISSN: 10769986     EISSN: None     Source Type: Journal    
DOI: 10.3102/10769986031001035     Document Type: Review

References (47)

1. American Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME). (1985). Standards for educational and psychological testing. Washington, DC: American Psychological Association (APA).
2. American Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME). (1999). Standards/or educational and psychological testing. Washington, DC: American Educational Research Association (AERA).
3. Angoff, W. H. (1971). Scales, norms, and equivalent scores. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 508-600). Washington, DC: American Council on Education.
4. Baker, E. L., Linn, R. L., Herman, J. L., & Koretz, D. (2002, Winter). Standards for educational accountability systems. The CRESST Line, 1-4.
5. Ballou, D., Sanders, W., & Wright, P. (2004). Controlling for student background in valueadded assessment for teachers. Journal of Educational and Behavioral Statistics, 29(1), 37-66.
6. Barnard, J. J. ( 1996, September). In search for equity in educational measurement: Traditional versus modern equating methods. Paper presented at the National Conference of the Association for the Study of Evaluation in Education in South Africa, Pretoria, South Africa.
7. Brennan, R. L. ( 1998). Misconceptions at the intersection of measurement theory and practice. Educational Measurement: Issues and Practice, 17(1), 5-9.
8. Bryk, A. S., Thum, Y. M., Easton, J. Q., & Luppescu, S. (1998). Assessing school academic productivity: The case of Chicago school reform. Social Psychology of Education, 2, 103-142.
9. Camilli, G. (1999). Measurement error, multidimensionality, and scale shrinkage: A reply to Yen and Burket. Journal of Educational Measurement, 36(1). 73-78.
10. Cliff, N. ( 1991 ). Ordinal methods in the assessment of change. In L. M. Collins & J. L. Horn (Eds.), Best methods for the analysis of change: Recent advances, unanswered questions, future directions (pp. 34-46). Washington, DC: American Psychological Association.
11. Crocker. L.. & Algina, J. ( 1986). Introduction to classical & modern test theory. New York: Holt, Rinehart, & Winston.
12. Fuhrman, S. H., & Elmore, R. F. (2004). Redesigning accountability systems for Education. New York: Teachers College Press.
13. Goertz, M. E., Duffy, M. C., & Le Floch, K. C. (2001 ). Assessment and accountability systems in the 50 states: 1999-2000 (No. RR-046). Philadelphia, PA: Consortium for Policy Research in Education.
14. Herman, J. L., Brown, R. S., & Baker, E. L. (2000). Student assessment and performance in the California public school svstem. Los Angeles: Center for the Study of Evaluation (CSE), National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
15. Koretz, D. M., McCaffrey, D. F., & Hamilton, L. S. (2001). Toward a framework for validating gains under high stakes conditions (Technical Report 551). Los Angeles: Center for the Study of Evaluation, UCLA.
16. Lewis, A. (2001). 2000 CRESST Conference proceedings: Educational accountability in the 21st century (No. CSE Technical Report 549). Los Angeles: Center for the Study of Evaluation (CSE), National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
17. Linn, R. L. (1993). Linking results of distinct assessments. Applied Measurement in Education, 6(1), 83-102.
18. Linn, R. L. (2001 ). The design and evaluation of educational assessment and accountability systems (No. CSE Technical Report 539). Los Angeles: Center for the Study of Evaluation (CSE), National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
19. Luecht, R. M. (1996). Multidimensional computerized adaptive testing in a certification or licensure context. Applied Psychological Measurement, 20, 389-404.
20. McCaffrey, D. F., Lockwood, J. R., Koretz, D. M., & Hamilton, L. S. (2003). Evaluating value-added models for teacher accountability. Santa Monica: The RAND Corporation.
21. Michigan State University Education Policy Center. (2002). State accountability frameworks.
22. Retrieved August 19, 2004, from http://www.epc.msu.edu/publications/ accountability/ StateAccountability.pdf
23. Millman, J. (Ed.). (1997). Grading teachers, grading schools: Is student achievement a valid evaluation measure? Thousand Oaks, CA: Corwin Press.
24. Mislevy, R. J. (1992). Linking educational assessments: Concepts, issues, methods, and practice. Princeton, NJ: Educational Testing Service Policy Information Center.
25. No Child Left Behind Act of 2001, Pub. L. No. 107-110, 115 Stat. 1425, (2002).
26. Oison, L. (2002, November 20). Education scholars finding new "value" in student test data. Education Week, 1-14.
27. Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage Publications.
28. Reckase, M. D. (1989a, August). Controlling the psychometric snake: Or, how I learned to love multidimensionality. Paper presented at the Annual Meeting of the American Psychological Association, New Orleans, LA.
29. Reckase, M. D. ( 1989b, March). Similarity of the multidimensional space defined by parallel forms of a mathematics test. Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco, CA.
30. Reckase, M. D. ( 1998, October). Investigating assessment instrument parallelism in a high dimensional latent space. Paper presented at the Annual Meeting of the Society of Multivariate Experimental Psychology, Woodcliff Lake, NJ.
31. Reckase, M. D., Ackerman, T. A., & Carlson, J. E. (1988). Building a unidimensional test using multidimensional items. Journal of Educational Measurement, 25(3), 193-203.
32. Reckase, M. D., & McKinley, R. L. (1991). The discriminating power of items that measure more than one dimension. Applied Psychological Measurement, 15(4), 361-373.
33. Sanders, W. L., & Horn, S. P. (1994). The Tennessee value-added assessment system (TVAAS): Mixed model methodology in educational assessment. Journal of Personnel Evaluation in Education, 8(3), 299-311.
34. Sanders, W. L., Saxon, A. M., & Horn, S. P. (1997). The Tennessee value-added assessment system: A quantitative, outcomes-based approach to educational assessment. In J. Millman (Ed.), Grading teachers, grading schools: Is student achievement a valid evaluation measure? (pp. 137-162). Thousand Oaks, CA: Corwin Press.
35. Schacter, J. (2001). Teacher performance-based accountability: Why, what and how. Santa Monica: Milken Family Foundation.
36. Schulz, E. M., & Nicewander, W. A. (1997). Grade equivalent and IRT representations of growth. Journal of Educational Measurement, 34(4), 315-331.
37. Segall, D. O. (1996). Multidimensional adaptive testing. Psychometrika, 61, 331-354.
38. Segall, D. O. (2000). Principles of multidimensional adaptive testing. In W. J. van der Linden, & C. A. W. Glas (Eds.), Computerized adaptive testing: Theory and practice (pp. 53-57). Dordrecht, The Netherlands: Kluwer Academic Publishers.
39. Stout, W. (2002). Psychometrics: From practice to theory and back. 15 Years of nonparametric multidimensional IRT, DIF/Test equity, and skills diagnostic assessment. Psychometrika, 67(4), 485-518.
40. Thum, Y. M. (2002). Measuring progress towards a goal: Estimating teacher productivity using a multivariate multilevel model for value-added analysis. Santa Monica: Milken Family Foundation.
41. Thum, Y. M. (2003). No Child Left Behind: Methodological challenges & recommendations for measuring adequate yearly progress (No. 590). Los Angeles: Center for the Study of Evaluation, University of California, Los Angeles (CSE).
42. van der Linden, W. J. (1999). Multidimensional adaptive testing with a minimum errorvariance criterion. Journal of Educational and Behavioral Statistics, 24, 398-412.
43. Webster, W. J., Mendro, R. L., & Almaguer, T. D. (1994). Effectiveness indices: A "valueadded" approach to measuring school effect. Studies in Educational Evaluation, 20, 1135.
44. Westat & Policy Studies Associates. (2001). The longitudinal evaluation of school change and performance (LESCP) in Title I schools, Vol. 1: Executive summary. Washington, IXT: U.S. Department of Education, Office of the Deputy Secretary, Planning and Evaluation Service.
45. Williams, V. S., Pommerich, M., & Thissen, D. (1998). A comparison of developmental scales based on Thurstone methods and item response theory. Journal of Educational Measurement, 35(2). 93-107.
46. Yen, W. M. ( 1986). The choice of scale for educational measurement: An IRT perspective. Journal of Educational Measurement, 23(4), 399-325.
47. Zwick, R. (1992). Statistical and psychometric issues in the measurement of educational achievement trends: Examples from the national assessment of educational progress. Journal of Educational Statistics, 17(2), 205-218.