Measuring Progress Toward a Goal: Estimating Teacher Productivity Using a Multivariate Multilevel Model for Value-Added Analysis

Sociological Methods and Research

Volumn 32, Issue 2, 2003, Pages 153-207

  Thum, Yeow Meng ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Accountability models; Bayesian effect size; Covariate adjustment; Gain score; Markov chain Monte Carlo; Multivariate multilevel modeling

Indexed keywords

EID: 0242678279     PISSN: 00491241     EISSN: None     Source Type: Journal    
DOI: 10.1177/0049124103257073     Document Type: Article

References (95)

1. Allison, P. 1990. "Change Scores as Dependent Variables in Regression Analysis." Pp. 93-114 in Sociological Methodology, edited by C. Clogg. San Francisco: Jossey-Bass.
2. Berk, R. A. 1988. "Fifty Reasons Why Student Achievement Gain Does Not Mean Teacher Effectiveness." Journal of Personnel Evaluation in Education 1:345-63.
3. Bock, R. D. 1975. Multivariate Statistical Methods in Behavioral Research. New York: McGraw-Hill.
4. Box, G. E. P. 1976. "Science and Statistics." Journal of the American Statistical Association 71:791-9.
5. Brooks, S. P. and A. Gelman. 1998. "Alternative Methods for Monitoring Convergence of Iterative Simulations." Journal of Computational and Graphical Statistics 7:434-55.
6. Brooks, S. P. and G. O. Roberts. 1998. "Assessing Convergence of Markov Chain Monte Carlo Algorithms." Statistics and Computing 8:319-35.
7. Bryk, A. S. and S. W. Raudenbush. 1992. Hierarchical Linear Models: Applications and Data Analysis Methods. Newbury Park, CA: Sage.
8. Bryk, A. S., Y. M. Thum, J. Q. Easton, and S. Luppescu. 1998. "Assessing School Academic Productivity: The Case of Chicago School Reform." Social Psychology of Education 2:103-42.
9. Bryk, A. S. and H. I. Weisberg. 1976. "Value-Added Analysis: A Dynamic Approach to the Estimation of Treatment Effects." Journal of Educational Statistics 1:127-55.
10. Burket, G. R. 1984. "Response to Hoover." Educational Measurement: Issues and Practices 3:15-16.
11. California Department of Education (CDE). 1999. Public Schools Accountability Act (PSAA) [Legislation]. California Department of Education.
12. Campbell, D. T. and D. A. Kenny. 1999. A Primer on Regression Artifacts. New York: Guilford.
13. Carlin, B. P. 1996. "Hierarchical Longitudinal Modelling." Pp. 303-19 in Markov Chain Monte Carlo in Practice, edited by W. R. Gilks, S. Richardson, and D. J. Spiegelhalter. London: Chapman & Hall.
14. Carlin, B. P. and T. A. Louis. 1996. Bayes and Empirical Bayes Methods for Data Analysis. London: Chapman & Hall.
15. Casella, G. and E. I. George. 1992. "Explaining the Gibbs Sampler." Journal of the American Statistical Association 46:167-74.
16. Coleman, J. S. and N. L. Karweit. 1972. Information Systems and Performance Measures in Schools. Englewood Cliffs, NJ: Educational Technology Publications.
17. Collins, L. M. 1996. "Is Reliability Obsolete? A Commentary on 'Are Simple Gain Scores Obsolete?' " Applied Psychological Measurement 20:289-92.
18. Cowles, M. K. and B. P. Carlin. 1996. "Markov Chain Monte Carlo Diagnostics: A Comparative Review." Journal of the American Statistical Association 91:883-904.
19. Cronbach, L. J. and L. Furby. 1970. "How Should We Measure 'Change' - or Should We?" Psychological Bulletin 74:68-80.
20. Draper, D. 1995. "Inference and Hierarchical Modeling in the Social Sciences." Journal of Educational and Behavioral Statistics 20:115-47.
21. Eissenberg, T. E. and L. M. Rudner. 1988. "Explaining Test Results to Parents." Practical Assessment, Research & Evaluation 1:1.
22. Fitz-Gibbon, C. T. 1997. "The Value-Added National Project: Final Report. Feasibility Studies for a National System of Value-Added Indicators." Report, SCAA, London.
23. Fleishman, A. I. 1980. "Confidence Intervals for Correlation Ratios." Educational and Psychological Measurement 40:659-70.
24. Gelfand, A. E. 1996. "Model Determination Using Sampling-Based Methods." Pp. 145-61 in Markov Chain Monte Carlo in Practice, edited by W. R. Gilks, S. Richardson, and D. J. Spiegelhalter, London: Chapman & Hall.
25. Gelman, A., J. C. Carlin, H. Stern, and D. B. Rubin. 1995. Bayesian Data Analysis. London: Chapman & Hall.
26. Gilks, W. R., S. Richardson, and D. J. Spiegelhalter, eds. 1996. Markov Chain Monte Carlo in Practice. London: Chapman & Hall.
27. Glass, G. and K. Hopkins. 1984. Statistical Methods in Education and Psychology. 2d ed. Boston: Allyn & Bacon.
28. Glass, G. V. 1990. "Using Student Test Scores to Evaluate Teachers," Pp. 229-40 in The New Handbook of Teacher Evaluation, edited by L. Millman and L. Darling-Hammond. Newbury Park, CA: Sage.
29. Goldstein, H. 1995. Multilevel Statistical Models. London: Edward Arnold.
30. Goldstein, H. and D. J. Spiegelhalter. 1996. "League Tables and Their Limitations: Statistical Issues in Comparisons of Institutional Performance." Journal of the Royal Statistical Society, Series A 159:384-443.
31. Good, T. L., B. J. Biddle, and J. E. Brody. 1975. Teachers Make a Difference. Lanham, MD: University Press of America.
32. Gray, J., D. Jesson, H. Goldstein, K. Hedger, and J. Rabash. 1995, "A Multi-Level Analysis of School Improvement: Changes in School's Performance Over Time." School Effectiveness and School Improvement 6:97-114.
33. Haertel, E. 1986. "The Valid Use of Student Achievement Measures for Teacher Evaluation." Educational Evaluation and Policy Analysis 8:45-60.
34. Harker, R. and R. Nash. 1996. "Academic Outcomes and School Effectiveness: Type 'A' and Type 'B' Effects." New Zealand Journal of Educational Studies 32:143-70.
35. Harris, C. W. 1963. Problems in the Measurement of Change. Madison: University of Wisconsin Press.
36. Hills, J. R. 1984. "Interpreting NCE Scores." Educational Measurement: Issues and Practices 3:25-26, 31.
37. Hoeting, J. A., D. Madigan, A. E. Raftery, and C. T. Volinsky. 1999. "Bayesian Model Averaging: A Tutorial." Statistical Science 14:382-417.
38. Hoover, H. D. 1984a. "The Most Appropriate Scores for Measuring Educational Development in Elementary Schools: GE's." Educational Measurement: Issues and Practices 3:8-14.
39. _. 1984b. "Rejoinder to Burket." Educational Measurement; Issues and Practices 3:16-18.
40. Jaeger, R. M. 1993. Statistics: A Spectator Sport. 2d ed. Newbury Park, CA: Sage.
41. Kalaian, H. and S. W. Raudenbush. 1996. "A Multivariate Mixed Linear Model for Meta-Analysis." Psychological Methods 3:227-35.
42. Kaplan, D. and P. R. Elliott. 1997. "A Model-Based Approach to Validating Education Indicators Using Multilevel Structural Equation Modeling." Journal of Educational and Behavioral Statistics 22:323-48.
43. Kass, R. E. and R. E. Raftery. 1995. "Bayes Factors." Journal of the American Statistical Association 90:773-95.
44. Laird, N. M. and J. H. Ware. 1982. "Random-Effects Models for Longitudinal Data." Biometrika 65:581-90.
45. Lindley, D. V. and A. F. M. Smith. 1972. "Bayesian Estimates for the Linear Model." Journal of the Royal Statistical Society, Series B 34:1-41.
46. Linn, R. L. 2000. "Assessments and Accountability." Educational Researcher 29:4-16.
47. _. 2001. "The Design and Evaluation of Educational Assessment and Accountability Systems." CSE Technical Report No. 539, University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
48. Linn, R. L. and J. A. Slinde. 1977. "The Determination of the Significance of Change Between Pre- and Post-Periods." Review of Educational Research 47:121-50.
49. Littell, R. C., G. A. Milliken, W. W. Stroup, and R. D. Wolfinger. 1996. SAS System for Mixed Models. Cary, NC: SAS Institute.
50. Maassen, G. H. 2000. "Kelley's Formula as a Basis for the Assessment of Reliable Change." Psychometrika 65:187-97.
51. Maris, E. 1998. "Covariance Adjustment Versus Gain Score-Revisited." Psychological Methods 3:309-27.
52. Mellenbergh, G. J. 1999. "A Note on Simple Gain Score Precision." Applied Psychological Measurement 23:87-89.
53. Meyer, R. H. 1996. "Value-Added Indicators of School Performance." Pp. 197-223 in Improving America's Schools: The Role of Incentives, edited by E. A. Hanushek and D. W. Jorgenson. Washington, DC: National Academic Press.
54. Millman, J. 1981. "Student Achievement as a Measure of Teacher Competence." Pp. 146-66 in Handbook of Teacher Evaluation, edited by J. Millman. Beverly Hills, CA: Sage.
55. No Child Left Behind Act (2001), Pub. L. No. 107-110, 115 Stat. 1425.
56. O'Hagan, A. 1994. Bayesian Inference. Vol. 2B. London: Arnold.
57. O'Hagan, A., J. W. Stevens, and J. Montmartin. 2000. "Inference for the Cost-Effectiveness Acceptability Curve and the Cost-Effectiveness Ratio." Pharmocoeconomics 17:339-49.
58. _. 2001. "Bayesian Cost-Effectiveness Analysis From Clinical Trial Data." Statistics in Medicine 20:733-53.
59. Raudenbush, S. W., R. T. Brennan, and R. C. Barnett. 1995. "A Multivariate Hierarchical Model for Studying Psychological Change Within Married Couples." Journal of Family Psychology 9:161-76.
60. Raudenbush, S. W., R. P. Fotiu, and Y. F. Cheong. 1999. "Synthesizing Results From the Trial State Assessment." Journal of Educational and Behavioral Statistics 24:413-38.
61. Raudenbush, S. W. and X.-F. Liu. 2000. "Statistical Power and Optimal Design for Multisite Randomized Trials." Psychological Methods 5:199-213.
62. Raykov, T. 1999. "Are Simple Gain Scores Obsolete? On an Approach to the Study of Correlates and Predictors of Ability Growth." Applied Psychological Measurement 23:120-26.
63. Rogosa, D. R. 1995. "Myths and Methods: Myths About Longitudinal Research Plus Supplemental Questions." Pp. 3-66 in The Analysis of Change, edited by J. M. Gottman. Mahwah, NJ: Lawrence Erlbaum.
64. Rogosa, D. R., D. Brandt, and M. Zimowski. 1982. "A Growth Curve Approach to the Measurement of Change." Psychological Bulletin 90:726-48.
65. Sanders, W. L. and S. P. Horn. 1994. "The Tennessee Value-Added Assessment System (TVAAS): Mixed-Model Methodology in Educational Assessment." Journal of Personnel Evaluation 8:299-311.
66. Sanders, W. L. and J. C. Rivers. 1996. "Cummulative and Residual Effects of Teachers on Future Student Academic Achievement." Research progress report, University of Tennessee, Value-Added Research and Assessment Center, Knoxville, TN.
67. Saunders, L. 1999. "A Brief History of Educational 'Value Added': How Did We Get to Where We Are?" School Effectiveness and School Improvement 10:233-56.
68. Schacter, J. 2001. Teacher Advancement Program Performance-Based Accountability. Santa Monica, CA: Milken Family Foundation.
69. Schulz, E. M. and W. A. Nicewander. 1997. "Grade Equivalent and IRT Representations of Growth." Journal of Educational Measurement 34:315-32.
70. Schuster, C. and A. von Eye. 1998. "Determining the Meaning of Parameters in Multilevel Models for Longitudinal Data." International Journal of Behavioral Development 22:475-91.
71. Searle, S. R. 1971. Linear Models. New York: John Wiley.
72. Seltzer, M., K. C. Choi, and Y. M. Thum. 2001. "Examining Relationships Between Where Students Start and How Rapidly They Progress: Implications for Conducting Analyses That Help Illuminate the Distribution of Achievement Within Schools." CSE Technical Report No. 560, University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
73. Seltzer, M., K. Frank, and A. S. Bryk. 1994. "The Metric Matters: The Sensitivity of Conclusions About Growth in Student Achievement to Choice of Metric." Educational Evaluation and Policy Analysis 16:41-49.
74. Shinkfield, A. J. and D. L. Stufflebeam. 1995. Teacher Evaluation: Guide to Effective Practice. Kalamazoo, MI: Center for Research on Educational Accountability and Teacher Evaluation.
75. Slavin, R. E. and N. A. Madden. 1991. "Modifying Chapter 1 Program Improvement Guidelines to Reward Appropriate Practices." Educational Evaluation and Policy Analysis 13:369-79.
76. Smith, A. F. M. and A. E. Gelman. 1992. "Bayesian Statistics Without Tears: A Sampling-Resampling Perspective." Journal of the American Statistical Association 46:84-88.
77. Snijders, T. and R. Bosker. 1999. Multilevel Analysis. Thousand Oaks, CA: Sage.
78. Spiegelhalter, D. J., N. G. Best, B. P. Carlin, and A. van der Linde. 2002. "Bayesian Measures of Model Complexity and Fit (With Discussion)." Journal of the Royal Statistical Society, Series B 64:583-640.
79. Spiegelhalter, D. J., A. Thomas, and N. G. Best. 1999. "WinBUGS: Bayesian Inference Using Gibbs Sampling, Version 1.3" [Computer program]. Cambridge, UK: MRC Biostatistics Unit.
80. Standard Achievement Test Series Technical Data Report. 1997. 9th ed. San Antonio, TX: Harcourt-Brace Educational Measurement.
81. Steiger, J. H. and R. T. Fouladi. 1997. "Non-Centrality Interval Estimation and the Evaluation of Statistical Models." Pp. 221-57 in What If There Were No Significance Tests? edited by L. L. Harlow, S. A. Mulaik, and J. H. Steiger. Mahwah, NJ: Lawrence Erlbaum.
82. Stone, J. E. 1999. "Value-Added Assessment: An Accountability Revolution." Pp. 239-49 in Better Teachers, Better Schools, edited by M. Kanstoroom and C. E. Finn Jr. Washington, DC: Thomas B. Fordham Foundation.
83. Thomas, S. and H. Goldstein. 1995. "Questionable Value." Education 185:17.
84. Thompson, B. 2002. "What Future Quantitative Social Science Research Could Look Like: Confidence Intervals for Effect Sizes." Educational Researcher 31:25-32.
85. Thum, Y. M. 1994. "Analysis of Individual Variation: A Multivariate Hierarchical Linear Model for Behavioral Data." Doctoral dissertation, University of Chicago.
86. _. 1997. "Hierarchical Linear Models for Multivariate Outcomes." Journal of Educational and Behavioral Statistics 22:77-108.
87. _. 2002. "Measuring Student and School Progress with the California API." CSE Technical Report No. 578, University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
88. _. 2003. "No Child Left Behind: Methodological Challenges and Recommendations for Measuring Adequate Yearly Progress." CSE Technical Report No. 590, University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
89. Tymms, P. 1990. "Can an Indicator System Improve the Effectiveness of Science and Mathematics Education? The Case of the UK." Evaluation and Research in Education 4:61-73.
90. Webster, W. J. and R. L. Mendro. 1997. "The Dallas Value-Added Accountability System." Pp. 81-99 Grading Teachers, Grading Schools: Is Student Achievement a Valid Evaluation Measure? edited by J. Millman. Thousand Oaks, CA: Corwin Press.
91. Willett, J. B. 1988. "Questions and Answers in the Measurement of Change." Review of Research in Education 15:345-422.
92. Williams, R. H. and D. W. Zimmerman. 1996. "Are Gain Scores Obsolete?" Applied Psychological Measurement 20:59-69.
93. Willms, D. 1992. Monitoring School Performance: A Guide for Educators. Washington, DC: Falmer.
94. Willms, D. J. and S. W. Raudenbush. 1989. "A Longitudinal Hierarchical Linear Models for Estimating School Effects and Their Stability." Journal of Educational Measurement 26:209-32.
95. Yen, W. M. 1986. "The Choice of Scale for Educational Measurement: An IRT Perspective." Journal of Educational Measurement 23:299-325.