Uses, applications and problems of educational value-added models;Usos, aplicaciones y problemas de los modelos de valor añadido en educación

Revista de Educacion

Volumn 348, Issue , 2009, Pages 217-250

  Arias, Rosario Martínez ^a  

^a UNIVERSIDAD COMPLUTENSE DE MADRID   (Spain)

Author keywords

School accountability; School improvement; Statistical issues of value added models (VAMs); Value added models; Vertical scaling

Indexed keywords

EID: 70349482826     PISSN: 00348082     EISSN: 1988592X     Source Type: Journal    
DOI: None     Document Type: Article

References (121)

1. AERA, APA & NACIONAL COUNCIL ON MEASUREMENT IN EDUCATION (1999). Standards for Educational and Psychological Tests. Washington, DC: American Psychological Association.
2. AMREIN-BEARDSLEY, A. (2008). Methodological concerns about the education value-added assessment system. Educational researcher, 37, 65-75.
3. BALLOU, D. (2002). Sizing up test scores. Education Next, 2, 10-15.
4. BALLOU, D. (2005). Value-added assessment: Lessons from Tennessee. En R. LISSITZ (ed.), Value-Added models in education: Theory and Applications (pp. 272-297). Mapple Grove, MN: JAM Press
5. BALLOU, D., SANDERS, W. & WRIGHT, P. (2004). Controlling for student background in value-added assessment of teachers. Journal of Educational and Behavioral Statistics, 29, 37-65.
6. BOCK, R. D., WOLFE, R. & FISHER, T. H. (1996). A review and analysis of the Tennessee value-added assessment system. Summary and recommendations. Nashville, TN: Tennessee Office of Education Accountability.
7. BRAUN, H. (2005a). Value-Added Modeling: What Does Due Diligence Require? En R. LISSITZ (ed.), Value-Added models in education: Theory and Applications (19-39). Mapple Grove, MN: JAM Press.
8. BRAUN, H. (2005b). Using Student Progress to Evaluate Teachers: A Primer on Value-Added Models. Princeton, NJ: Educational Testing Service.
9. BRAUN, H. & WAINER H. (2007). Value-added modelling. En S. SINHARAY & C. RAO (eds.), Handbook of statistics, 26: Psychometrics (pp. 867-891). Amsterdam: North Holland.
10. BRYK, S. A., THUM, M. Y., EASTON, Q. J. & LUPPESCU, S. (1998). Academic productivity of Chicago public elementary schools. Chicago, IL: Consortium on Chicago School Research.
11. CAMILLI, G. (1988). Scale shrinkage and the estimation of latent distribution parameters. Journal of Educational Statistics, 13, 227-241.
12. CAMPBELL, D. T. (1975). Assessing the impact of planned social change. En G. LYONS (ed.), Social research and public policies: The darmouth/OECD Conference (pp. 3-45). Hanover. NH: Darthmouth College. The Public Affairs Center.
13. CASTRO, M., RUÍZ DE MIGUEL, C. Y LÓPEZ, E. (2008). Forma básica del crecimiento en los modelos de valor añadido: vías para la supresión del efecto de regresión y funciones de crecimiento no lineales. Revista de Educación, 348.
14. CHOI, K., GOLDSCHMIDT, P. & YAMASHIRO, K. (2006). Exploring models of school performance: From theory to practice (CSE Rep: No. 673). Los Angeles, CA: National Center for Research on Evaluation, Standards, and Student Testing.
15. CIZEK, J. (ed.). (2005). Adapting testing technology to serve accountability aims: The Case of vertically-moderated standard setting. A Special Issue of Applied Measurement in Education, 18, 1-9.
16. CIZEK, G. J. & BUNCH, M. B. (2007). Standard setting: a guide to establishing and evaluating performance standards on tests. Thousand Oaks, CA: Sage.
17. COHEN, J., JOHNSON, E. & ANGELES, J. (2000). Variance estimation when sampling in two dimensions via the jackknife with application to the national assessment of educational progress. Washington, DC: American Institute for Research.
18. DORAN, H. C. (2004). Value-Added Models and Adequate Yearly Progress: Combining Growth and Adequacy in a Standards-based Environment. Paper presented at the Annual CCSSO Large-Scale Assessment Conference, Boston, Massachusetts.
19. DORAN, H. C. & COHEN, J. (2005). The confounding effect of linking bias on gains estimated from value-added models. En R. LISSITZ (ed.), Value-added models in education: Theory and applications (pp. 80-110). Maple Grove, MN: JAM Press.
20. DORAN, H. C. & FLEISCHMAN, S. (2005). Research matters/Challenges of value-added assessment. Eductional Leadership, 63, 85-87.
21. DORAN, H. C. & IZUMI, L. T. (2004). Putting Education to the Test: A Value-Added Model for California. San Francisco, CA: Pacific Research Institute.
22. DORAN, H. C. & JIANG, T. (2006). The impact of linking error in longitudinal análisis: an empirical demonstration. En R. LISSITZ (ed.), Longitudinal and value added models of student performance (pp. 210-229). Maple Grove, MN: JAM Press.
23. DORAN, H. C. & LOCKWOOD, J. R. (2006). Fitting value-added models in R. Journal of Educational and Behavioral Statistics, 31, 205-230.
24. DORANS, N. J., POMMERICH, M. & HOLLAND, P. W. (eds.). (2007). Linking and aligning scores and scales. New York: Springer.
25. DRURY, D. & DORAN, H. (2003). The Value of Value-Added Analysis. NSBA Policy Research Brief, 3, 1-4.
26. FERRARA, S., PHILLIPS, G. W., WILLIAMS, P. L., LEINWAND, S., MAHONEY, S. & AHADI, S. (2007). Vertically articulated performance standards: An exploratory study of inferences about achievement and growth. En R. LISSITZ (ed.), Assessing and modeling cognitive development in school (pp. 31-63). Maple Grove, MN: JAM Press.
27. FIELDING, A., YANG, M. & GOLDSTEIN, H. (2003). Multilevel ordinal models for examination grades. Statistical Modelling, 3, 127-153.
28. FITZ-GIBBON, C. T. (1997). The value-added national project: Final report: feasibility studies for a national system of value added indicators. London: School Curriculum and Assessment Authority.
29. FUCHS, T. & WOSSMANN, L. (2007). What accounts for international differences in student performance? A re-examination using PISA data. Empirical Economics, 32, 433-464.
30. GAVIRIA, J. L., BIENCINTO, M. C. Y NAVARRO, E. (2008). Invarianza de la estructura de covarianzas de las medidas de rendimiento académicos en estudios longitudinales en la transición de la educación primaria a secundaria. Revista de Educación, 348.
31. GLENN, C. & DE GROOF, J. (2005). Balancing Freedom, Autonomy and Accoutnability in Education. Nijmegan NL: Wolf Legal Publishers.
32. GOLDSCHMIDT, P. & CHOI, K. (2007). The practical benefits of growth models for accountability and the limitations under NCLB. Policy Brief 9. Los Angeles: National Center for Research on Evaluation, Standards, and Student Testing.
33. GOLDSCHMIDT, P., CHOI, K. & MARTINEZ, F. (2004). Using Hierarchical Growth Models To Monitor School Performance Over Time: Comparing NCE to Scale Score Results (CSE Report 618). Los Angeles, CA: Center for Research on Evaluation Standards and Student Testing.
34. GOLDSTEIN, H. & HEALY M. J. R. (1995). The Graphical Presentation of a Collection of Means. Journal of the Royal Statistical Society, 581, 175-177.
35. GOLDSTEIN, H. (2003). Multilevel models. London: Arnold.
36. GOLDSTEIN, H., BURGESS, S. & MCCONNELL, B. (2007). Modelling the effect of pupil mobility on school differences in educational achievement. Journal of The Royal Statistical Society, A, 170, 941-954.
37. GRAY, J., JESSON, D., GOLDSTEIN, H., HEDGER, K. & RASBASH, J. (1995). A multilevel analysis of school improvement: changes in schools' performance over time. School Effectiveness and School Improvement, 10, 97-114.
38. GULLIKSEN, H. (1950). Test theory. New York. Wiley.
39. HAERTEL, E. H. (2004). The behavior of linking items in test equating (CSE Report No 630). Los Angeles, CA: Center for Research on Evaluation Standards and Student Testing.
40. HANSON, B. A. & BEGUIN, A. A. (2002). Obtaining a common scale for IRT item parameters using separate versus concurrent estimation in the common item non-equivalent groups equating design. Applied Psychological Measurement, 21, 3-24.
41. HARRIS, D. J. (2007). Practical issues in vertical scaling. En N. J. DORANS, M. POMERICH, & P. W. HOLLAND (eds.), Linking and aligning scores and scales (pp. 233-251). New York: Springer.
42. HILL, R., SCOTT, M., DE PASCALE, C., DUNN, J. & SIMPSON, M. A. (2006). Using value tables to explicitly value student growth. En R. LISSITZ (ed.), Longitudinal and value added models of student performance (pp. 255-290). Maple Grove, MN: JAM Press.
43. HOLLAND, P. W. (2007). A framework and history for score linking. En N. J. DORANS, M. POMERICH & P. W. HOLLAND (eds.), Linking and aligning scores and scales (pp. 5-30). New York: Springer.
44. HOXBY, C. (2000). Does competition among public schools benefit students and taxpayers. American Economic Review, 90, 1. 209-1. 238.
45. HOXBY, C. (2003). School choice and school competition: Evidence from the United States. Swedish Economic Policy Review, 10, 9-65.
46. HUYHN, H. & SCHNEIDER, C. (2005). Vertically moderated standards: Background, assumptions, and practice. Applied Measurement in Education, 18, 99-114.
47. JACOB, B. (2002). Accountability, Incentives and Behavior: The Impact of High-Stakes Testing in the Chicago Public Schools. Cambridge, MA.: NBER Working Paper No. 8968.
48. KANE, T. J. & STAIGER, D. O. (2002). Volatility in school test scores: Implications for test-based accountability systems. En D. Ravith (ed.), Brooking papers on education policy (pp. 235-260). Washington, DC: Brookings Institution.
49. KIM, S. & COHEN, A. S. (1998). A comparison of linking and concurrent calibration under item response theory. Applied Psychological Measurement, 22, 131-143.
50. KOHN, A. (2000). The case against standardized testing: Raising the scores, ruining the schools. Portsmouth, NH: Heineman.
51. KOLEN, M. J. (2006). Scaling and norming. En R. L. BRENNAN (ed.). Educational Measurement (4a ed.). (pp. 155-186). Westport, CT: American Council on Education.
52. KOLEN, M. & BRENNAN, R. (2004). Test equating, scaling, and linking: Methods and practices (2a ed.). New York: Springer-Verlag.
53. KORETZ, D. M. (2005). Alignment, high stakes and the inflation of test scores. En J. L. HERMAN & E. H. HAERTEL (eds.), Uses and misuses of data in accountability testing. Yearbook of the National Society for the Study of Education. vol. 104 Part 2 (pp. 99-118). Malden, MA: Blackwell.
54. KORETZ, D. M. & HAMILTON, L. S. (2006). Testing for accountability in K-12. En R. L. BRENNAN (ed.), Educational Measurement (4a ed.), (pp. 531-578). Westport, CT: Praeger.
55. KUPERMINTZ, H. (2003). Teacher Effects and Teacher Effectiveness: A Validity Investigation of the Tennessee Value Added Assessment System. Educational Evaluation and Policy Analysis, 25(3).
56. LADD, H. F. & WALSH, R. P. (2002). Implementing value-added measures of school effectiveness: Getting the incentives right. Economics of Education Review, 21, 1-17.
57. LINN, R. L. (2000). Assessments and Accountability. Educational Researcher, 29, 4-14.
58. LINN, R. L. (2006). Validity of inferences from test-based educational accountability systems. Journal of Personnel Evaluation in Education, 19, 5-15.
59. LINN, R. L. & HAUG, C. (2002). Stability of school building accountability scores and gains. Educational Evaluation and Policy Analysis, 24, 29-36.
60. LISSITZ, R. (ed.). (2005). Value added models in education: Theory and applications. Maple Grove, MN: JAM Press.
61. LISSITZ, R. (ed.) (2006). Longitudinal and Value added models of student performance. Maple Grove, MN: JAM Press.
62. LISSITZ, R. W. & HUYHN, H. (2003). Vertical equating for state assessments: issues and solutions in determination of adequate yearly progress and school accountability. Practical Assessment, Research, and Evaluation, 8, 10.
63. LITTLE R. J. A. & RUBIN D. B. (2002). Statistical Analysis with Missing Data (2a ed.) New York: Wiley.
64. LIZASOAIN, L. Y JUARISTI, L. (2008). Análisis de la dimensionalidad en los modelos de valor añadido: estudio de pruebas de matemáticas empleando técnicas factoriales y métodos no paramétricos basados en TRI. Revista de Educación, 348.
65. LOCKWOOD, J. R. & MCCAFFREY, D. F. (2007). Controling for individual heterogeneity in longitudinal models with applications to student achievement. Electronic Journal of Statistics, 1, 223-252.
66. MADAUS, G. & CLARKE, M. (2001). The adverse impact of high-stakes testing on minority students: Evidence from one hundred years of test data. En G. ORFIELD & M. L. KORNHABER (eds.), Raising standards or raising barriers? Inequality and high-stakes testing in public education (pp. 85-106). New York: Century Foundation Press.
67. MARCHESI, A., MARTÍN, E., MARTÍNEZ ARIAS, R., TIANA, A. & MORENO, A. (2002). An evaluation network for educational change. Studies in Educational Evaluation, 29, 43-56.
68. MARCHESI, A., MARTÍNEZ ARIAS, R. Y MARTÍN, E. (2004). Estudio longitudinal sobre la influencia del nivel sociocultural en el aprendizaje de los alumnos de la ESO. Infancia y Aprendizaje, 27, 307-323.
69. MARTINEAU, J.A. (2004). The effects of construct shift on growth and accountability models. Unpublished Doctoral Dissertation. Michigan State University.
70. MARTINEAU, J. A. (2006). Distorting value-added: The use of longitudinal, vertically scaled student achievement data for growth-based value-added accountability. Journal of Educational and Behavioral Statistics, 31, 35-62.
71. MARTINEAU, J. A., SUBEDI, D. R., WARD, K. H., LI, T., LU, Y., DIAO, Q., PANG, F. H., DRAKE, S., SONG, T., KAO, S. C., ZHENG, Y. & LI, X. (2007). Non-linear trajectories through multidimensional content spaces: An examination of psychometric claims of unidimensionality, linearity, and interval measurement. En R. LISSITZ (ed.), Assessing and modeling cognitive development in school (pp. 96-142). Maple Grove, MN: JAM Press.
72. MARTÍNEZ ARIAS, R., GAVIRIA, J. L. Y CASTRO, M. (2008). Concepto y evolución de los modelos de valor añadido en educación. Revista de Educación, 348.
73. MARTÍNEZ ARIAS, R., HERNÁNDEZ LLOREDA, V. Y HERNÁNDEZ LLOREDA, M. J. (2006). Psicometría. Madrid: Alianza.
74. MCCAFFREY, D. F., LOCKWOOD, J. R., KORETZ, D. M. & HAMILTON, L. S. (2003). Evaluating Value-Added Models for Teacher Accountability. Santa Mónica, CA: RAND Corporation.
75. MCCAFFREY, D. M., LOCKWOOD, J. R., KORETZ, D., LOUIS, T. A. & HAMILTON, L. (2004). Models for value-added modeling of teacher effects. Journal of Educational and Behavioral Statistics, 29, 67-101.
76. MCCALL, M. & HAUSER, C. (2007). Item response theory and longitudinal modeling: The real world is less complicated than we fear. En R. LISSITZ (ed.), Assessing and modeling cognitive development in school (pp. 143-174). Maple Grove, MN: JAM Press.
77. MICHAELIDES, M. P., & HAERTEL, E. H. (2004, May). Sampling of common items: An unrecognized source of error in test equating (Technical Report). Los Angeles: Center for the Study of Evaluation y National Center for Research on Evaluation, Standards, and Student Testing.
78. O'DAY, J. (2002). Complexity, Accountability, and School Improvement. Harvard Educational Review, 72, 293-329.
79. PATZ, R. J. (2007). Vertical scaling in standards-based educational assessment and accountability systems. Washington, DC: The Council of Chief State School Officers.
80. PATZ, R. J. & YAO, L. (2007a). Vertical scaling: Statistical models for measuring growth and achievement. En S. SINHARAY & C. RAO (eds.), Handbook of statistics, 26: Psychometrics (pp. 955-975). Amsterdam: North Holland.
81. PATZ, R. J. & YAO, L. (2007b). Methods and models for vertical scaling. En N. J. DORANS, M. POMMERICH & P. W. HOLLAND (eds.), Linking and aligning scores and scales (pp. 253-272). New York: Springer.
82. PETERSEN, N. S., KOLEN, M. J. & HOOVER, H. D. (1989). Scaling, norming and equating. En R. L. LINN (ed.), Educational Measurement. (3a ed.). (221-262). New York: Macmillan.
83. POMPLUM, M., OMAR, M. H. & CUSTER, M. (2004). A comparison of Winstep and Bilog-MG for vertical scaling with the Rasch model. Applied Psychological measurement, 28, 247-273.
84. PONISCIAK, S. M. & BRYK, A. S. (2005). Value-added analysis of the Chicago public schools: An application of hierarchical models. En R. LISSITZ (ed.). Value-Added models in education: Theory and applications (pp. 40-79). Mapple Grove, MN: JAM Press.
85. RAUDENBUSH, S. (2004a). Schooling, statistics, and poverty: Can we measure school improvement? Princeton, NJ: Educational Testing service.
86. RAUDENBUSH, S. (2004b). What are value-added models estimating and what does this imply for statistical practice? Journal of Educational and Behavioral Statistics, 29, 121-129.
87. RAUDENBUSH, S. W. & WILLMS, J. D. (1995). The estimation of school effects. Journal of Educational and Behavioral Statistics, 20, 307-335.
88. RAY, A., EVANS, H. & MCCORMACK, T. (2008). The use of national value added models for school improvement in English schools. Revista de Educación, 348.
89. RECKASE, M. D. (2004). The real world is more complicated than we would like. Journal of Educational and Behavioral Statistics, 29, 117-120.
90. RECKASE, M. D. & LI, T. (2007). Estimating gain in achievement when content specifications change: A multidimensional item response theory approach. En R. LISSITZ (ed.), Assessing and modeling cognitive development in school (pp. 189-204). Maple Grove, MN: JAM Press.
91. RECKASE, M. D. & MARTINEAU, J. (2004). The vertical scaling of science achievement tests. Paper commissioned by the Committee on Test Design for K-12 Science Achievement. Washington, DC: National Research Council.
92. ROBERTS, J. S. & MA, Q. (2006). IRT models for the assessment of change across repeated measurements. En R. LISSITZ (ed.), Longitudinal and value added models of student performance (100-129). Maple Grove, MN: JAM Press.
93. ROGOSA, D. (1995). Myths about longitudinal research. En J. M. GOTTMAN (ed.), The analysis of change (pp. 3-66). Mahwah, NJ: Erlbaum.
94. ROSENBAUM, P. R. (2002). Observational studies. New York: Springer.
95. ROSE, L. C. & GALLUP, A. M. (2007). The 39th Annual Phi Delta Kappa/Gallup poll of the public's attitudes toward the public schools. Phi, Delta, Kappan, 89, 33-48.
96. RUBIN, D. B., STUART, E. A. & ZANUTTO, E. E. (2004). A potencial outcomes view of value-added assessment in education. Journal of Educational and behavioural Statistics, 29, 103-116.
97. SAMMONS, P., THOMAS, S. & MORTIMORE, P. (1997). Forging links: Effective schools and effective departments. London: Chapman y Hall.
98. SANDERS, W. & HORN, S. (1994). The Tennessee value-added assessment system (TVAAS): Mixed model methodology in educational assessment. Journal of Personnel Evaluation, 9, 299-311.
99. SANDERS, W. L., SAXTON, A. M. & HORN, S. P. (1997). The Tennessee value-added assessment system: a quantitative, outcomes-based approach to educational assessment. En J. MILLMAN (ed.), Grading teachers, grading schools: Is student achievement a valid evaluation measure? (pp. 137-162). Thousand Oaks, CA: Corwin Press.
100. SAUNDERS, L. (2000). Understanding schools' use of value-aded data. The psychology and sociology of numbers. Research Papers in Education: Policy and Practice, 15, 241-258.
101. SCHMIDT, W. H., HOUANG, R. T. & MCKNIGHT, C. C. (2005). Value-added research: Right idea but wrong solution? En R. LISSITZ (ed.), Value-added models in education: Theory and applications (pp. 145-165). Maple Grove, MN: JAM Press.
102. SCHAFER, W. D., YEN, S. J. & RAHMAN, T. (2000). School effects indices: Stability of one-and-two level formulations. Journal of Experimental Education, 68, 239-250.
103. SCHATZ, C. J., VONSECKER, C. E. & ALBAN, T. R. (2005). Balancing accountabilit and improvement: Introducing value-added models to a large school system. En R. LISSITZ (ed.), Value added models in education: Theory and applications (pp. 1-18). Maple Grove, MN: JAM Press.
104. SMITH, R. L. & YEN, W. M. (2006). Models for evaluating grade-to-grade growth. En R. W. LISSITZ (ed.), Longitudinal and value added modeling of student performance (pp. 82-99). Maple Grove, MN: JAM Press.
105. STEVENS, J. (2005). The study of school effectiveness as a problem in research design. En R. LISSITZ (ed.), Value-Added models in education: Theory and applications (pp. 166-208). Maple Grove, MN: JAM Press.
106. STUART, E. A. (2007). Estimating causal effects using school-level datasets. Educational Researcher, 36 (4), 187-198.
107. SUPOVITZ, J. A. & KLEIN, V. (2003). Mapping a course for improved student learning. Philadelphia: Consortium for Policy Research in Education.
108. TEKWE, C. D., CARTER, R. L., MA C. - X., ALGINA, J., LUCAS, M., ROTH, J. et al. (2004). An empirical comparison of statistical models for value-added assessment of school performance. Journal of Educational and Behavioral Statistics, 29, 11-36.
109. THUM, Y. M. (2006). Designing gross productivity indicators: A proposal for connecting accountability goals, data and analysis. En R. LISSITZ (ed.), Longitudinal and value added models of student performance (pp. 436-479). Maple Grove, MN: JAM Press.
110. TONG, Y. & KOLEN, M. J. (2007). Comparison of methodologies and results in vertical scaling for educational achievement tests. Applied Measurement in Education, 20, 227-253.
111. WANG, L., BECKETT, G. H. & BROWN, L. (2006). Controversies of standardized assessment in school accountability reform: A critical synthesis of multidisciplinary research evidence. Applied Measurement in Education. 19, 305-328.
112. WEBSTER, W. J. (2005). The Dallas school level accountability model: The marriage of status and value-added approaches. En R. LISSITZ (ed.), Value added models in education: Theory and applications (pp. 233-271). Maple Grove, MN: JAM Press.
113. WEBSTER, W. J. & MENDRO, R. L. (1997). The Dallas value-added accountability system. En J. MILLMAN (ed.), Grading teachers, grading schools: Is student achievement a valid evaluation measure? (81-99). Thousand Oaks, CA: Corwin Press.
114. WRIGHT, S. P. (2004). Advantages of a multivariate longitudinal approach to educational value-added assessment without imputation. Paper presented at the 2004 National Evaluation Institute, July 8-10, Colorado Springs, Colorado.
115. WRIGHT, S. P., SANDERS, W. L. & RIVERS, J. C. (2006). Measurement of academic growth of individual students toward variable and meaningful academic standards. En R. LISSITZ (ed.), Longitudinal and value added models of student performance (pp. 385-406). Maple Grove, MN: JAM Press.
116. YANG, M., GOLDSTEIN, H., RATH, T. & HILL, N. (1999). The use of assessment data for school improvement purposes. Oxford Review of Education, 25, 469-483.
117. YEN, W. M. (1986). The choice of scale for educational measurement: An IRT perspective. Journal of Educational Measurement, 23, 299-325.
118. YEN, W. M. (2007). Vertical scaling and Noo Child Left Behind. En N. J. DORANS, M. POMMERICH & P. W. HOLLAND (eds.), Linking and aligning scores and scales (pp. 273-283). New York: Springer.
119. YEN, W. M. & BURKET, G. R. (1997). Comparison of item response theory and Thurstone methods of vertical scaling. Journal of Educational Measurement, 34, 293-313.
120. YEN, W. M. & FITZPATRICK, A. R. (2006). Item response theory. En R. L. BRENNAN (ed.), Educational Measurement (4a ed.). (pp. 111-154). Westport, CT: Praeger.
121. ZVOCH, K. & STEVENS, J. J. (2005). Sample exclusion and student attrition effects in the longitudinal study of middle school mathematics performance. Educational Assessment, 10, 105-123.