Performance status and change - measuring education system effectiveness with data from PISA 2000-2009

Educational Research and Evaluation

Volumn 20, Issue 2, 2014, Pages 146-174

  Lenkeit, Jenny ^a   Caro, Daniel H ^b  

^a UNIVERSITY OF AMSTERDAM   (Netherlands)

^b UNIVERSITY OF OXFORD   (United Kingdom)

Author keywords

education system effectiveness; educational policymaking; performance trends; PISA

Indexed keywords

EID: 84897821418     PISSN: 13803611     EISSN: 17444187     Source Type: Journal    
DOI: 10.1080/13803611.2014.891462     Document Type: Article

References (91)

1. Baird, J.-A., Isaacs, T., Johnson, S., Stobart, G., Yu, G., Sprague, T., & Daugherty, R. (2011). Policy effects of PISA. Oxford: Oxford University Centre for Educational Assessment.
2. Baker, D. P., Goesling, B., & LeTendre, G. K. (2002). Socioeconomic status, school quality, and national economic development: A cross-national analysis of the "Heyneman-Loxley Effect" on mathematics and science achievement. Comparative Education Review, 46, 291-312. doi: 10.1086/341159
3. Ballou, D., Sanders, W., & Wright, P. (2004). Controlling for student background in value-added assessment of teachers. Journal of Educational and Behavioral Statistics, 29, 37-65. doi: 10.3102/10769986029001037
4. Bank, V., & Heidecke, B. (2009). Gegenwind für PISA. Ein systematisierender Überblick über kritische Schriften zur internationalen Vergleichsmessung [Head wind for PISA. A systemizing overview of critical papers on international comparative assessments]. Vierteljahresschrift für Wissenschaftliche Pädagogik, 85, 361-372.
5. Barone, C., & Van de Werfhorst, H. G. (2011). Education, cognitive skills and earnings in comparative perspective. International Sociology, 26, 483-502. doi: 10.1177/0268580910393045
6. Bechger, T. M., Van den Wittenboer, G., Hox, J. J., & De Glopper, C. (1999). The validity of comparative educational studies. Educational Measurement: Issues and Practice, 18(3), 18-26. doi: 10.1111/j.1745-3992.1999.tb00269.x
7. Breakspear, S. (2012). The policy impact of PISA: An exploration of the normative effects of international benchmarking in school system performance (OECD Education Working Papers, No. 71). Paris: OECD Publishing.
8. Bryk, A. S., & Raudenbush, S. W. (1987). Application of hierarchical linear models to assessing change. Psychological Bulletin, 101, 147-158. doi: 10.1037/0033-2909.101.1.147
9. Buchmann, C. (2002). Measuring family background in international studies of education: Conceptual issues and methodological challenges. In National Research Council (Ed.), Methodological advances in cross-national surveys of educational achievement (pp. 150-197). Washington, DC: National Academy Press.
10. Byrne, B. M., & Van de Vijver, F. J. R. (2010). Testing for measurement and structural equivalence in large-scale cross-cultural studies: Addressing the issue of nonequivalence. International Journal of Testing, 10, 107-132. doi: 10.1080/15305051003637306
11. Caro, D. H., & Lenkeit, J. (2012). An analytical approach to study educational inequalities: 10 hypothesis tests in PIRLS 2006. International Journal of Research & Method in Education, 35, 3-30. doi: 10.1080/1743727X.2012.666718
12. Caro, D. H., Sandoval-Hernández, A., & Lüdtke, O. (2013). Cultural, social, and economic capital constructs in international assessments: An evaluation using exploratory structural equation modeling. School Effectiveness and School Improvement. Advance online publication. doi:10.1080/09243453.2013.812568
13. Chiu, M. M. (2007). Families, economies, cultures, and science achievement in 41 countries: Country-, school-, and student-level analyses. Journal of Family Psychology, 21, 510-519. doi: 10.1037/0893-3200.21.3.510
14. Chudgar, A., & Luschei, T. F. (2009). National income, income inequality, and the importance of schools: A hierarchical cross-national comparison. American Educational Research Journal, 46, 626-658. doi: 10.3102/0002831209340043
15. Chudgar, A., Luschei, T. F., Fagioli, L. P., & Lee, C. (2012, April). Socio-economic status (SES) measures using Trends in International Mathematics and Science Studies (TIMSS) data. Paper presented at the Annual Meeting of the American Educational Research Association, Vancouver, Canada.
16. Coe, R., & Fitz-Gibbon, C. T. (1998). School effectiveness research: Criticisms and recommendations. Oxford Review of Education, 24, 421-438. doi: 10.1080/0305498980240401
17. Creemers, B. P. M. (2006). The importance and perspectives of international studies in educational effectiveness. Educational Research and Evaluation, 12, 499-511. doi: 10.1080/13803610600873978
18. Creemers, B. P. M., & Kyriakides, L. (2008). The dynamics of educational effectiveness: A contribution to policy, practice and theory in contemporary schools. London: Routledge.
19. Creemers, B. P. M., & Kyriakides, L. (2010). Explaining stability and changes in school effectiveness by looking at changes in the functioning of school factors. School Effectiveness and School Improvement, 21, 409-427. doi: 10.1080/09243453.2010.512795
20. Doolaard, S. (2002). Stability and change in results of schooling. British Educational Research Journal, 28, 773-787. doi: 10.1080/0141192022000019053
21. Duru-Bellat, M., & Suchaut, B. (2005). Organisation and context, efficiency and equity of educational systems: What PISA tells us. European Educational Research Journal, 4, 181-194. doi: 10.2304/eerj.2005.4.3.3
22. Ercikan, K. (2002). Disentangling sources of differential item functioning in multilanguage assessments. International Journal of Testing, 2, 199-215. doi: 10.1080/15305058.2002.9669493
23. Frenette, M. (2008). The returns to schooling on academic performance: Evidence from large samples around school entry cut-off dates (Analytical Studies Research Paper Series - No. 317). Ottawa: Statistics Canada.
24. Fuchs, T., & Wößmann, L. (2007). What accounts for international differences in student performance? A re-examination using PISA data. Empirical Economics, 32, 433-464. doi: 10.1007/s00181-006-0087-0
25. Gebhardt, E., & Adams, R. J. (2007). The influence of equating methodology on reporting trends in PISA. Journal of Applied Measurement, 8, 305-322.
26. Giambona, F., Vassallo, E., & Vassiliadis, E. (2011). Educational systems efficiency in European Union countries. Studies in Educational Evaluation, 37, 108-122. doi: 10.1016/j.stueduc.2011.05.001
27. Goodnow, W. E. (1982). The contingency theory of education. International Journal of Lifelong Education, 1, 341-352. doi: 10.1080/0260137820010404
28. Grek, S. (2009). Governing by numbers: The PISA "effect" in Europe. Journal of Education Policy, 24, 23-37. doi: 10.1080/02680930802412669
29. Grisay, A., Gonzalez, E., & Monseur, C. (2009). Equivalence of item difficulties across national versions of the PIRLs and PIsA reading assessments. IERI Monograph Series: Issues and Methodologies in Large-Scale Assessments, 2, 63-83.
30. Hanushek, E. A. (2013). Economic growth in developing countries: The role of human capital. Economics of Education Review, 37, 204-212. doi: 10.1016/j.econedurev.2013.04.005
31. Hargreaves, A., & Goodson, I. (2006). Educational change over time? The sustainability and nonsustainability of three decades of secondary school change and continuity. Educational Administration Quarterly, 42, 3-41. doi: 10.1177/0013161X05277975
32. Hofman, W. H. A., Hofman, R. H., & Gray, J. M. (2010). Institutional contexts and international performances in schooling: Comparing patterns and trends over time in international surveys. European Journal of Education, 45, 153-173. doi: 10.1111/j.1465-3435.2009.01420.x
33. Honaker, J., King, G., & Blackwell, M. (2011). Amelia II: A program for missing data. Journal of Statistical Software, 45(7), 1-47.
34. Hopkins, D., Harris, A., Stoll, L., & Mackay, T. (2011, January). School and system improvement: State of the art review. Keynote speech presented at the Annual Meeting of the International Congress of School Effectiveness and Improvement, Limassol, Cyprus.
35. Kamens, D. H., & McNeely, C. L. (2010). Globalization and the growth of international educational testing and national assessment. Comparative Education Review, 54, 5-25. doi: 10.1086/648471
36. Kelly, S., & Monczunski, L. (2007). Overcoming the volatility in school-level gain scores: A new approach to identifying value added with cross-sectional data. Educational Researcher, 36, 279-287. doi: 10.3102/0013189X07306557
37. Klein, R. (2011). A reanalysis of PISA results: Comparability problems. Avaliação e Políticas Públicas em Educação, 19, 743-768.
38. Kreiner, S., & Christensen, K. B. (2013). Analyses of model fit and robustness. A new look at the PISA scaling model underlying ranking of countries according to reading literacy. Psychometrika. Advance online publication. doi:10.1007/S11336-013-9347-Z
39. Kyriakides, L., & Creemers, B. P. M. (2008). A longitudinal study on the stability over time of school and teacher effects on student outcomes. Oxford Review of Education, 34, 521-545. doi: 10.1080/03054980701782064
40. Lenkeit, J. (2012). How effective are educational systems? A value- added approach to measure trends in PIRLS. Journal for Educational Research Online, 4(2), 143-173.
41. Leung, K., & Van de Vijver, F. J. R. (2008). Strategies for strengthening causal inferences in cross cultural research: The consilience approach. International Journal of Cross Cultural Management, 8, 145-169. doi: 10.1177/1470595808091788
42. Levin, H. M. (2012). More than just test scores. Prospects, 42, 269-284. doi: 10.1007/s11125-012-9240-z
43. Liegmann, A. B., & Van Ackeren, I. (2012). The impact of PIRLS in 12 countries: A comparative summary. In K. Schwippert & J. Lenkeit (Eds.), Progress in reading literacy in national and international context. The impact of PIRLS 2006 in 12 countries (pp. 228-252). Münster, Germany: Waxmann.
44. Luyten, H. (1994). Stability of school effects in dutch secondary education: The impact of variance across subjects and years. International Journal of Educational Research, 21, 197-216. doi: 10.1016/0883-0355(94)90032-9
45. Luyten, H., Peschar, J., & Coe, R. (2008). Effects of schooling on reading performance, reading engagement, and reading activities of 15-year-olds in England. American Educational Research Journal, 45, 319-342. doi: 10.3102/0002831207313345
46. Martin, M. O., Mullis, I. V. S., Foy, P., Brossman, B., & Stanco, G. M. (2012). Estimating linking error in PIRLS. IERI Monograph Series: Issues and Methodologies in Large-Scale Assessments, 5, 35-47.
47. Martineau, J. A. (2006). Distorting value added: The use of longitudinal, vertically scaled student achievement data for growth-based, value-added accountability. Journal of Educational and Behavioral Statistics, 31, 35-62. doi: 10.3102/10769986031001035
48. Meyer, J. W. (2010). World Society, institutional theories, and the actor. Annual Review of Sociology, 36, 1-20. doi: 10.1146/annurev.soc.012809.102506
49. Mislevy, R. J. (1995). What can we learn from iInternational assessments? Educational Evaluation and Policy Analysis, 17, 419-437.
50. Mitter, W. (2004). Rise and decline of education systems: A contribution to the history of the modern state. Compare: A Journal of Comparative and International Education, 34, 351-369. doi: 10.1080/0305792042000294788
51. Mullis, I. V. S., Martin, M. O., Kennedy, A. M., & Foy, P. (2007). PIRLS 2006 international report: IEA's Progress in International Reading Literacy Study in primary schools in 40 countries. Chestnut Hill, MA: Boston College.
52. Nachtigall, C., Kröhne, U., Enders, U., & Steyer, R. (2008). Causal effect and fair comparison: Considering the influence of context variables on student competencies. In J. Hartig, E. Klieme, & D. Leutner (Eds.), Assessment of competencies in educational contexts (pp. 315-335). Göttingen, Germany: Hogrefe & Huber.
53. Newton, X. A., Darling-Hammond, L., Haertel, E., & Thomas, E. (2010). Value-added modeling of teacher effectiveness: An exploration of stability across models and contexts. Education Policy Analysis Archives, 18(23), 1-27.
54. Organisation for Economic Co-operation and Development. (2001). Knowledge and skills for life. First results from the OECD Programme for International Student Assessment (PISA) 2000. Paris: Author.
55. Organisation for Economic Co-operation and Development. (2008). Measuring improvements in learning outcomes. Best practices to assess the value-added of schools. Paris: Author.
56. Organisation for Economic Co-operation and Development. (2010a). PISA 2009 assessment framework. Key competencies in reading, mathematics and science. Paris: Author.
57. Organisation for Economic Co-operation and Development. (2010b). PISA 2009 results: Learning trends (Vol. V). Paris: Author.
58. Organisation for Economic Co-operation and Development. (2010c). PISA 2009 results: Overcoming social background (Vol. II). Paris: Author.
59. Organisation for Economic Co-operation and Development. (2010d). PISA 2009 results: What students know and can do. Student performance in reading, mathematics and science (Vol. I). Paris: Author.
60. Organisation for Economic Co-operation and Development. (2010e). TALIS 2008: Technical report. Paris: Author.
61. Organisation for Economic Co-operation and Development. (2011). Lessons from PISA for the United States. Strong performers and successful reformers in education. Paris: Author.
62. Organisation for Economic Co-operation and Development. (2012). PISA 2009 technical report. Paris: Author.
63. Postlethwaite, T. N., & Ross, K. N. (1992). Effective schools in reading. Implications for educational planners. Hamburg, Germany: International Association for the Evaluation of Educational Achievement.
64. Reynolds, D. (2006). World Class Schools: Some methodological and substantive findings and implications of the International School Effectiveness Research Project (ISERP). Educational Research and Evaluation, 12, 535-560. doi: 10.1080/13803610600874026
65. Reynolds, D., Sammons, P., De Fraine, B., Townsend, T., & Van Damme, J. (2011, January). Educational effectiveness research (EER): A state of the art review. Paper presented at the Annual Meeting of the International Congress for School Effectiveness and Improvement, Limassol, Cyprus.
66. Ringarp, J., & Rothland, M. (2010). Is the grass always greener? The effect of the PISA results on education debates in Sweden and Germany. European Educational Research Journal, 9, 422-430. doi: 10.2304/eerj.2010.9.3.422
67. Robitaille, D. F., Beaton, A. E., & Plomp, T. (2000). The impact of TIMSS on teaching and learning mathematics and science. Vancouver: Pacific Educational Press.
68. Robitzsch, A. (2010). TIMSS 1995 und 2007: Trend der mathematischen Kompetenzen in Österreich [TIMSS 1995 and 2007: Trends of mathematic competences in Austria]. In B. Suchan, C. Wallner-Paschon, & C. Schreiner (Eds.), TIMSS 2007. Österreichischer Expertenbericht (pp. 56-63). Graz, Austria: Leykam.
69. Rogosa, D. R., & Willett, J. B. (1985). Understanding correlates of change by modelling individual differences in growth. Psychometrika, 50, 203-258. doi: 10.1007/BF02294247
70. Rutkowski, D., & Rutkowski, L. (2013). Measuring socioeconomic background in PISA: One size might not fit all. Research in Comparative and International Education, 8, 259-278. doi: 10.2304/rcie.2013.8.3.259
71. Sahlberg, P. (2007). Education policies for raising student learning: The Finnish approach. Journal of Education Policy, 22, 147-171. doi: 10.1080/02680930601158919
72. Sahlberg, P. (2011). PISA in Finland: An education miracle or an obstacle to change? CEPS Journal, 1(3), 119-140.
73. Sammons, P. (2006). The contribution of international studies on Educational Effectiveness: Current and future directions. Educational Research and Evaluation, 12, 583-593. doi: 10.1080/13803610600874091
74. Schriewer, J., & Martinez, C. (2004). Constructions of internationality in education. In G. Steiner-Khamsi (Ed.), The global politics of educational borrowing and lending (pp. 29-53). New York, NY: Teachers College Press.
75. Sellar, S., & Lingard, B. (2013). Looking east: Shanghai, PISA 2009 and the reconstitution of reference societies in the global education policy field. Comparative Education, 49, 464-485. doi: 10.1080/03050068.2013.770943
76. Slater, R. O., & Teddlie, C. (1992). Toward a theory of school effectiveness and leadership. School Effectiveness and School Improvement, 3, 247-257. doi: 10.1080/0924345920030402
77. Solano-Flores, G., & Nelson-Barber, S. (2001). On the cultural validity of science assessments. Journal of Research in Science Teaching, 38, 553-573. doi: 10.1002/tea.1018
78. Sun, H. (2003). National contexts and effective school improvement (Unpublished doctoral dissertation). Rijksuniversiteit Groningen, Groningen, The Netherlands.
79. Tan, J. B. Y., & Yates, S. M. (2007). A Rasch analysis of the academic self-concept questionnaire. International Education Journal, 8(2), 470-484.
80. Thomas, S. (1998). Value-added measures of school effectiveness in the United Kingdom. Prospects, 18, 91-108. doi: 10.1007/BF02737782
81. Van Ackeren, I. (2007). Comparative synthesis. In K. Schwippert (Ed.), Progress in reading literacy. The impact of PIRLS 2001 in 13 countries (pp. 243-264). Münster, Germany: Waxmann.
82. Van Damme, J., Liu, H., Vanhee, L., & Pustjens, H. (2010). Longitudinal studies at the country level as a new approach to educational effectiveness: Explaining change in reading achievement (PIRLS) by change in age, socio-economic status and class size. Effective Education, 2, 53-84. doi: 10.1080/19415531003616888
83. Van de Ven, A. H., Ganco, M., & Hinings, C. R. (2013). Returning to the frontier of Contingency Theory of organizational and institutional designs. The Academy of Management Annals, 7, 393-440. doi: 10.1080/19416520.2013.774981
84. Von Davier, M. (2013). Introduction: What are trends and why are they important for PISA? In M. Prenzel, M. Kobarg, K. Schöps, & S. Rönnebeck (Eds.), Research on PISA. Research outcomes of the PISA Research Conference 2009 (pp. 160-164). Dordrecht, The Netherlands: Springer.
85. Waldow, F. (2009). What PISA did and did not do: Germany after the "PISA-shock". European Educational Research Journal, 8, 476-483. doi: 10.2304/eerj.2009.8.3.476
86. Waldow, F. (2010). Der Traum vom "skandinavisch schlau Werden". Drei Thesen zur Rolle Finnlands als Projektionsfläche in der gegenwärtigen Bildungsdebatte [The dream of becoming clever the Scandinavian way. Three theses on the role of Finland as a projection surface in the current educational debate]. Zeitschrift für Pädagogik, 56, 497-511.
87. Watermann, R., & Klieme, E. (2002). Reporting results of large-scale assessment in psychologically and educationally meaningful terms. European Journal of Psychological Assessment, 18, 190-203. doi: 10.1027//1015-5759.18.3.190
88. Willms, J. D., & Raudenbush, S. W. (1989). A longitudinal hierarchical linear model for estimating school effects and their stability. Journal of Educational Measurement, 26, 209-232. doi: 10.1111/j.1745-3984.1989.tb00329.x
89. Wößmann, L. (2002). Cross-country evidence on human capital and the level of economic developmenr: The role of measurement issues in education. Historical Social Research/Historische Sozialforschung, 27(4), 47-76.
90. Wolf, A. (2004). Education and economic performance: Simplistic theories and their policy consequences. Oxford Review of Economic Policy, 20, 315-333. doi: 10.1093/oxrep/grh018
91. Zvoch, K., & Stevens, J. J. (2008). Measuring and evaluating school performance: an investigation of status and growth-based achievement indicators. Evaluation Review, 32, 569-595. doi: 10.1177/0193841X08320398