Lessons Learned from PISA: A Systematic Review of Peer-Reviewed Articles on the Programme for International Student Assessment

Scandinavian Journal of Educational Research

Volumn 62, Issue 3, 2018, Pages 333-353

  Hopfenbeck, Therese N ^a   Lenkeit, Jenny ^a   El Masri, Yasmine ^a   Cantrell, Kate ^a   Ryan, Jeanne ^a   Baird, Jo Anne ^a  

^a UNIVERSITY OF OXFORD   (United Kingdom)

Author keywords

critiques; PISA; policy impact; secondary analysis

Indexed keywords

EID: 85011298902     PISSN: 00313831     EISSN: 14701170     Source Type: Journal    
DOI: 10.1080/00313831.2016.1258726     Document Type: Article

References (134)

1. Afonso, N., & Costa, E., (2009). A influência do PISA na decisão política em Portugal: o caso das políticas educativas do XVII Governo Constitucional Português [The influence of PISA on the political decision in Portugal: the case of the educational policies of the XVII Portuguese Constitutional Government]. Sísifo. Revista de Ciências da Educação, 10, 53–64.
2. Akour, M., Sabah, S., & Hammouri, H., (2015). Net and global differential item functioning in PISA polytomously scored science items: Application of the differential step functioning framework. Journal of Psychoeducational Assessment, 33(2), 166–176. Retrieved from http://jpa.sagepub.com/cgi/content/abstract/33/2/166 doi: 10.1177/0734282914541337
3. Arffman, I., (2010). Equivalence of translations in international Reading literacy studies. Scandinavian Journal of Educational Research, 54 (1), 37–59. doi: 10.1080/00313830903488460
4. Baird, J., Johnson, S., Hopfenbeck, T. N., Isaacs, T., Sprague, T., Stobart, G., & Yu, G., (2016). On the supranational spell of PISA in policy. Educational Research, 58(2), 121–138. Special Issue: International Policy Borrowing and Evidence-based Educational Policy Making: Relationships and Tensions. doi: 10.1080/00131881.2016.1165410
5. Baumert, J., & Demmrich, A., (2001). Test motivation in the assessment of student skills: The effects of incentives on motivation and performance. European Journal of Psychology of Education, 16, 441–462. doi: 10.1007/BF03173192
6. Baumert, J., Ludtke, O., Trautwein, U., & Brunner, M., (2009). Large-scale student assessment studies measure the results of processes of knowledge acquisition: Evidence in support of the distinction between intelligence and student achievement. Educational Research Review, 4, 165–176. doi: 10.1016/j.edurev.2009.04.002
7. Benítez, I., & Padilla, J.-L., (2014). Analysis of nonequivalent assessments across different linguistic groups using a mixed methods approach: Understanding the causes of differential item functioning by cognitive interviewing. Journal of Mixed Methods Research, 8(1), 52–68. Retrieved from http://mmr.sagepub.com/cgi/content/abstract/8/1/52 doi: 10.1177/1558689813488245
8. Bieber, T., & Martens, K., (2011). The OECD PISA study as a soft power in education? Lessons from Switzerland and the USA. European Journal of Education, Research, Development and Policy, 46(1), 101–116.
9. Bonnet, G., (2002). Reflections in a critical eye: On the pitfalls of international assessment. Assessment in Education: Principles, Policy & Practice, 9(3), 387–399. http://dx.doi.org/10.1080/0969594022000027690a
10. Bosker, R. J., (2007). The different levels of the g-factor. European Journal of Personality, 21, 712–714.
11. Breakspear, S., (2012). The policy impact of PISA: An exploration of the normative effects of international benchmarking in school system performance. OECD Education Working Papers. doi:10.1787/5k9fdfqffr28-en.
12. Brunner, M., & Martin, R., (2007). Not every g is g. European Journal of Personality, 21, 714–716.
13. Butler, J., & Adams, R. J., (2007). The impact of differential investment of student effort on the outcomes of international studies. Journal of Applied Measurement, 8 (3), 279–304.
14. Bybee, R., McCrae, B., & Laurie, R., (2009). PISA 2006: An assessment of scientific literacy. Journal of Research in Science Teaching, 46 (8), 865–883. doi: 10.1002/tea.20333
15. Caro, D. H., Sandoval-Hernández, A., & Lüdtke, O., (2013). Cultural, social, and economic capital constructs in international assessments: An evaluation using exploratory structural equation modeling. School Effectiveness and School Improvement, 25(3), 433–450. doi:10.1080/09243453.2013.812568
16. Çetin, B., (2010). Cross-cultural structural parameter invariance on PISA 2006 student questionnaires. Eurasian Journal of Educational Research, 38, 71–89.
17. Chen, Y.-F., & Jiao, H., (2014). Exploring the utility of background and cognitive variables in explaining latent differential item functioning: An example of the PISA 2009 Reading assessment. Educational Assessment, 19(2), 77–96. Retrieved from http://www.tandfonline.com/doi/abs/10.1080/10627197.2014.903650
18. Chmielewski, A. K., Dumont, H., & Trautwein, U., (2013). Tracking effects depend on tracking type: An international comparison of students’ mathematics self-concept. American Educational Research Journal, 50(5), 925–957. doi:10.3102/0002831213489843
19. Çıkrıkçı Demirtaşlı, N., & Uluştaş, S., (2015). A study on detecting of differential item functioning of PISA 2006 science literacy items in Turkish and American samples. Eurasian Journal of Educational Research, 58, 41–60.
20. Dall, A., (2011). Is PISA counter-productive to building successful educational systems? Social Alternatives, 30(4), 10–14.
21. Dancis, J., (2014). What does the international PISA math test really tell Us? AASA Journal of Scholarship and Practice, 10(4), 31–42.
22. De Lange, J., (2006). Mathematical literacy for living from OECD-PIA perspective. Tsukuba Journal of Educational Study in Mathematics, 25, 13–35.
23. De Lange, M., Dronkers, J., & Wolbers, M. H. J., (2014). Single-parent family forms and children’s educational performance in a comparative perspective: Effects of school’s share of single-parent families. School Effectiveness and School Improvement, 25(3), 329–350. doi:10.1080/09243453.2013.809773
24. Di Fuccia, D., Witteck, T., Markic, S., & Eilks, I., (2012). Trends in practical work in German science education. Eurasia Journal of Mathematics, Science and Technology Education, 8(1), 59–72.
25. Domínguez, M., Vieira, M.-J., & Vidal, J., (2012). The impact of the programme for international student assessment on academic journals. Assessment in Education: Principles, Policy and Practice, 19(4), 393–409. doi: 10.1080/0969594X.2012.659175
26. Duit, R., (2007). Science education research internationally: Conceptions, research methods, domains of research. Eurasia Journal of Mathematics Science and Technology Education, 3(1), 3–15. doi: 10.12973/ejmste/75369
27. Duru-Bellat, M., & Suchaut, B., (2005). Organisation and context, efficiency and equity of educational systems: What PISA tells us. European Educational Research Journal, 4(3), 181–194. doi: 10.2304/eerj.2005.4.3.3
28. Eivers, E., (2010). PISA: Issues in implementation and interpretation. Irish Journal of Education/Iris Eireannach an Oideachais, 38, 94–118. Retrieved from http://www.jstor.org/stable/20789130
29. Eklöf, H., Hopfenbeck, T. N., & Kjærnsli, M., (2012). Hva vet vi om elevers testmotivasjon? Erfaringer fra internasjonale og nasjonale undersøkelser i Norge og Sverige, p. 84–96 [What do we know about students’ test motivation? Experiences of international and national tests in Norway and Sweden]. In T. N., Hopfenbeck, M., Kjaernsli, & R. V., Olsen (Eds.), Kvalitet i norsk skole. Internasjonale og nasjonale undersøkelser av læringsutbytte og undervisning [Quality in the Norwegian school. International and national tests of learning outcomes and teaching] (pp. 84–96). Oslo: Universitetsforlaget.
30. Ercikan, K., Roth, W.-M., & Asil, M., (2015). Cautions about inferences from international assessments: The case of PISA 2009. Teachers College Record, 117, 1–28.
31. Ertl, H., (2006). Educational standards and the changing discourse on education: The reception and consequences of the PISA study in Germany. Oxford Review of Education, 32 (5), 619–634. doi: 10.1080/03054980600976320
32. Feniger, Y., & Lefstein, A., (2014). How not to reason with PISA data: An ironic investigation. Journal of Education Policy, 29 (6), 845–855. doi: 10.1080/02680939.2014.892156
33. Ferrini-Mundy, J., & Schmidt, W. H., (2005). International comparative studies in mathematics education: Opportunities for collaboration and challenges for researchers. Journal for Research in Mathematics Education, 36(3), 164–175.
34. Flynn, J., (2007). What lies behind g(I) and g(ID). European Journal of Personality, 21, 722–724.
35. Frey, A., & Seitz, N.-N., (2011). Hypothetical use of multidimensional adaptive testing for the assessment of student achievement in the programme for international student assessment. Educational and Psychological Measurement, 71(3), 503–522. Retrieved from http://epm.sagepub.com/cgi/doi/10.1177/0013164410381521
36. Galloway, N., & Rose, H., (2015). Introducing global Englishes. New York: Routledge.
37. Goldstein, H., (2004). Measuring educational standards. Significance, 1, 103–105. doi:10.1111/j.1740-9713.2004.00039.x
38. Goldstein, H., Bonnet, G., & Rocher, T., (2007). Multilevel structural equation models for the analysis of comparative data on educational performance. Journal of Educational and Behavioral Statistics, 32 (3), 252–286. doi: 10.3102/1076998606298042
39. Goldstein, H., & Thomas, S. M., (2008). Reflections on the international comparative surveys debate. Assessment in Education: Principles, Policy & Practice, 15(3), 215–222. doi: 10.1080/09695940802417368
40. Gorard, S., & Smith, E., (2004). An international comparison of equity in education systems. Comparative Education, 40(1), 15–28. doi:10.1080/0305006042000184863
41. Gough, D., Oliver, S., & Thomas, J., (2012). An introduction to systematic reviews. London: Sage.
42. Grek, S., (2009). Governing by the numbers: The PISA “effect” in Europe. Journal of Education Policy, 24(1), 23–37. doi: 10.1080/02680930802412669
43. Grek, S., (2010). International organisations and the shared construction of policy “problems’: Problematisation and change in education governance in Europe. European Educational Research Journal, 9(3), 396–406. doi: 10.2304/eerj.2010.9.3.396
44. Grisay, A., Gonzalez, E., & Monseur, C., (2009). Equivalence of item difficulties across national versions of the PIRLs and PISA Reading assessments. IERI Monograph Series: Issues and Methodologies in Large-Scale Assessments, 2, 63–83.
45. Grisay, A., de Jong, J. H. A. L., Gebhardt, E., Breezier, A., & Halleux-Monseur, B., (2007). Translation equivalence across PISA countries. Journal of Applied Measurement, 8(3), 249–266.
46. Grisay, A., & Monseur, C., (2007). Measuring the equivalence of item difficulty in the various versions of an international test. Studies in Educational Evaluation, 33(1), 69–86. doi: 10.1016/j.stueduc.2007.01.006
47. Hambleton, R. K., Merenda, P. F., & Spielberger, C. D., (2005). Adapting educational and psychological tests for cross-cultural assessment. Mahwah, NJ: Lawrence Erlbaum Associates.
48. Hampden-Thompson, G., (2009). Are two better than one? A comparative study of achievement gaps and family structure. Compare: A Journal of Comparative and International Education, 39(4), 517–534. doi:10.1080/03057920802366372
49. Hampden-Thompson, G., Guzman, L., & Lippman, L., (2013). A cross-national analysis of parental involvement and student literacy. International Journal of Comparative Sociology, 54(3), 246–266. doi:10.1177/0020715213501183
50. Hartong, S., (2012). Overcoming resistance to change: PISA, school reform in Germany and the example of lower Saxony. Journal of Education Policy, 27 (6), 747–760. doi: 10.1080/02680939.2012.672657
51. Hatzinikita, V., Dimopoulos, K., & Christidou, V., (2008). PISA test items and school textbooks related to science: A textual comparison. Science Education, 92(4), 664–687. doi: 10.1002/sce.20256
52. Hendrickson, K. A., (2012). Learning from Finland: Formative assessment. The Mathematics Teacher, 105(7), 488–489. doi: 10.5951/mathteacher.105.7.0488
53. Hopfenbeck, T. N., & Maul, A., (2011). Examining evidence for the validity of PISA learning strategy scales based on student response processes. International Journal of Testing, 11(2), 95–121. doi:10.1080/15305058.2010.529977
54. Huang, X., Wilson, M., & Wang, L., (2016). Exploring plausible causes of differential item functioning in the PISA science assessment: Language, curriculum or culture. Educational Psychology: An International Journal of Experimental Educational Psychology, 36(2), 378–390. doi:10.1080/01443410.2014.946890
55. Jerrim, J., (2012). The socio-economic gradient in teenagers’ reading skills: How does England compare with other countries? Fiscal Studies, 33(2), 159–184. doi: 10.1111/j.1475-5890.2012.00157.x
56. Kaplan, D., & McCarty, A. T. (2013). Data fusion with international large scale assessments: A case study using the OECD PISA and TALIS surveys. Large-scale Assessments in Education, 1(6). Retrieved from http://www.largescaleassessmentsineducation.com/content/1/1/6
57. Kankaraš, M., & Moors, G., (2014). Analysis of cross-cultural comparability of PISA 2009 scores. Journal of Cross-Cultural Psychology, 45(3), 381–399. Retrieved from http://jcc.sagepub.com/cgi/doi/10.1177/0022022113511297
58. Kind, P. M., (2013). Conceptualizing the science curriculum: 40 years of developing assessment frameworks in three large-scale assessments. Science Education, 97(5), 671–694. doi: 10.1002/sce.21070
59. Klemm, K., (2003). If no reform then at least no rollback. European Education, 35(4), 34–43. doi: 10.2753/EUE1056-4934350434
60. Kreiner, S., & Christensen, K. B., (2014). Analyses of model fit and robustness: A new look at the PISA scaling model underlying ranking of countries according to Reading literacy. Psychometrika, 79(2), 210–231. doi: 10.1007/s11336-013-9347-z
61. Lafontaine, D., & Monseur, C., (2009). Gender gap in comparative studies of reading comprehension: To what extent do the test characteristics make a difference? European Educational Research Journal, 8(1), 69–79. doi: 10.2304/eerj.2009.8.1.69
62. Lau, K.-C., (2009). A critical examination of PISA’s assessment on scientific literacy. International Journal of Science and Mathematics Education, 7, 1061–1088. doi: 10.1007/s10763-009-9154-2
63. Le Donne, N., (2014). European variations in socioeconomic inequalities in students’ cognitive achievement: The role of educational policies. European Sociological Review, 30(3), 329–343. doi:10.1093/esr/jcu040
64. Lenkeit, J., Chan, J., Hopfenbeck, T. N., & Baird, J.-A., (2015). A review of the representation of PIRLS related research in scientific journals. Educational Research Review, 16, 102–115. doi: 10.1016/j.edurev.2015.10.002
65. Leung, F. K. S., (2014). What can and should we learn from international studies of mathematics achievement? Mathematics Education Research Journal, 26, 579–605. doi: 10.1007/s13394-013-0109-0
66. Lie, S., Linnakylä, P., & Roe, A., (Eds.). (2003). Northern lights on PISA: Unity and diversity in the Nordic countries. Oslo: Department of Teacher Education and School Development, University of Oslo.
67. Lietz, P., (2006). A meta-analysis of gender differences in reading achievement at the secondary school level. Studies in Educational Evaluation, 32(4), 317–344. doi:10.1016/j.stueduc.2006.10.002
68. Lindblad, S., Pettersson, D., & Popkewitz, T. S., (2015). International comparisons of school results: A systematic review of research on large scale assessments in education. Stockholm: Swedish Research Council.
69. Linneweber-Lammerskitten, H., & Walti, B., (2005). Is the definition of mathematics as used in the PISA assessment framework applicable to the HarmoS project? ZDM, 37(5), 402–407.
70. Lüdtke, O., Robitzsch, A., Trautwein, U., Kreuter, F., & Ihme Marten, J., (2007). Are there test administrator effects in large-scale educational assessments? Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 3(4), 149–159. doi: 10.1027/1614-2241.3.4.149
71. Marks, G. N., (2006). Are between- and within-school differences in student performance largely due to socio-economic background? Evidence from 30 countries. Educational Research, 48(1), 21–40. doi:10.1080/00131880500498396
72. Martins, L., & Veiga, P., (2010). Do inequalities in parents’ education play an important role in PISA student mathematics achievement test score disparities? Economics of Education Review, 29(6), 1016–1033. doi:10.1016/j.econedurev.2010.05.001
73. Mauranen, A., (2003). The corpus of English as lingua franca in academic settings. TESOL Quarterly, 37 (3), 513–527. doi: 10.2307/3588402
74. McConney, A., & Perry, L. B., (2010). Science and mathematics achievement in Australia: The role of school socioeconomic composition in educational equity and effectiveness. International Journal of Science and Mathematics Education, 8(3), 429–452. doi: 10.1007/s10763-010-9197-4
75. McGaw, B., (2008). The role of the OECD in international comparative studies of achievement. Assessment in Education: Principles, Policy & Practice, 15(3), 223–243. Retrieved from http://www.tandfonline.com/doi/abs/10.1080/09695940802417384
76. McQueen, J., & Mendelovits, J., (2003). PISA Reading: Cultural equivalence in a cross-cultural study. Language Testing, 20 (2), 208–224. doi: 10.1191/0265532203lt253oa
77. Meyer, H. D., (2014). The OECD as pivot of the emerging global educational accountability regime: How accountable are the accountants?. Teachers College Record, 116(9), 1–20.
78. Meyer, H.-D., & Benavot, A., (2013). PISA, power, and policy: The emergence of global educational governance. Didcot, UK: Symposium Books.
79. Micklewright, J., Schnepf, S. V., & Skinner, C., (2012). Non-response biases in surveys of schoolchildren: The case of the English programme for international student assessment (PISA) samples. Journal of the Royal Statistical Society: Series A (Statistics in Society), 175(4), 915–938. doi: 10.1111/j.1467-985X.2012.01036.x
80. Monseur, C., & Adams, R., (2009). Plausible values: How to deal with their limitations. Journal of Applied Measurement, 10(3), 320–334.
81. Møller, J., & Skedsmo, G., (2013). Modernising education: New public management reform in the Norwegian education system. Journal of Educational Administration and History, 45(4), 336–353. doi: 10.1080/00220620.2013.822353
82. Nardi, E., (2008). Cultural biases: A non-Anglophone perspective. Assessment in Education: Principles, Policy & Practice, 15(3), 259–266. doi: 10.1080/09695940802417467
83. Neumann, E., Kiss, A., Fejes, I., Bajomi, I., Berenyi, E., Biro, Z. A., & Vida, J., (2012). The hard work of interpretation: The national politics of PISA reception in Hungary and Romania. European Educational Research Journal, 11(2), 227–242. doi: 10.2304/eerj.2012.11.2.227
84. Nonoyama-Tarumi, Y., (2008). Cross-national estimates of the effects of family background on student achievement: A sensitivity analysis. International Review of Education, 54(1), 57–82. doi:10.1007/s11159-007-9069-5
85. Nyborg, H., (2007). Do recent large-scale cross-national student assessment studies neglect general intelligence g for political reasons? European Journal of Personality, 21, 739–741.
86. OECD. (1999). Measuring student knowledge and skills: A new framework for assessment. Paris: OECD.
87. OECD. (2000). Measuring student knowledge and skills: The PISA 2000 assessment of Reading, mathematical and scientific literacy. Paris: OECD.
88. OECD. (2009a). PISA 2006 technical report. Paris: OECD.
89. OECD. (2009b). PISA data analysis manual. Paris: OECD.
90. OECD. (2011). Strong performers and successful reformers in education lessons from PISA for the United States. Paris: OECD.
91. OECD. (2013). PISA
92. 2015 draft questionnaire framework. Paris: OECD.
93. Oliveri, M. E., & Ercikan, K., (2011). Do different approaches to examining construct comparability in multilanguage assessments lead to similar conclusions? Applied Measurement in Education, 24, 349–366. doi: 10.1080/08957347.2011.607063
94. Oliveri, M. E., Olson, B. F., Ercikan, K., & Zumbo, B. D., (2012). Methodologies for investigating item- and test-level measurement equivalence in international large-scale assessments. International Journal of Testing, 12(3), 203–223. doi: 10.1080/15305058.2011.617475
95. Olsen, R. V., (2004). The search for descriptions of students’ thinking and knowledge: Exploring nominal cognitive variables by correspondence and homogeneity analysis. Scandinavian Journal of Educational Research, 48(3), 325–341. doi:10.1080/00313830410001695763
96. O’Mara, J., (2014). Closing the emergency facility: Moving schools from literacy triage to better literacy outcomes. English Teaching: Practice and Critique, 13(1), 8–23.
97. Oppedisano, V., & Turati, G., (2015). What are the causes of educational inequality and of its evolution over time in Europe? Evidence from PISA. Education Economics, 23(1), 3–24. doi:10.1080/09645292.2012.736475
98. Osborne, J., (2013). The 21st century challenge for science education: Assessing scientific reasoning. Thinking Skills and Creativity, 10, 265–279. doi: 10.1016/j.tsc.2013.07.006
99. Ozga, J., (2012). Introduction. Assessing PISA. European Educational Research Journal, 11(2), 166–171. doi: 10.2304/eerj.2012.11.2.166
100. Pegg, J., & Panizzon, D., (2007). Inequities in student achievement for literacy: Metropolitan versus rural comparisons. Australian Journal of Language and Literacy, 30(3), 177–190.
101. Perera, L. D. H., (2014). Parents’ attitudes towards science and their children’s science achievement. International Journal of Science Education, 36(18), 3021–3041. doi:10.1080/09500693.2014.949900
102. Perry, L. B., & McConney, A., (2010). Does the SES of the school matter? An examination of socioeconomic status and student achievement using PISA 2003. Teachers College Record, 112(4), 1137–1162.
103. Polidano, C., Hanel, B., & Buddelmeyer, H., (2013). Explaining the socio-economic status school completion Gap. Education Economics, 21(3), 230–247. doi:10.1080/09645292.2013.789482
104. Prais, S. J., (2003). Cautions on OECD’s recent educational survey (PISA). Oxford Review of Education, 29, 139–163. doi: 10.1080/0305498032000080657
105. Prais, S. J., (2004). Cautions on OECD’s recent educational survey (PISA): Rejoinder to OECD’s response. Oxford Review of Education, 30(4), 569–573. doi: 10.1080/0305498042000303017
106. Prais, S. J., (2007). Two recent (2003) international surveys of schooling attainments in mathematics: England’s problems. Oxford Review of Education, 33(1), 33–46. doi: 10.1080/03054980601119557
107. Rindermann, H., (2007). The g-factor of international cognitive ability comparisons: The homogeneity of results in PISA, TIMSS, PIRLS and IQ tests across nations. European Journal of Personality, 21, 667–706. doi: 10.1002/per.634
108. Roth, W., Ercikan, K., Simon, M., & Fola, R., (2015). The assessment of mathematical literacy of linguistic minority students: Results of a multi-method investigation. The Journal of Mathematical Behavior, 40, 88–105. doi: 10.1016/j.jmathb.2015.01.004
109. Rutkowski, L., & Rutkowski, D., (2010). Getting it “better”: The importance of improving background questionnaires in international large-scale assessment. Journal of Curriculum Studies, 42(3), 411–430. doi: 10.1080/00220272.2010.487546
110. Schuelka, M. J., (2013). Excluding students with disabilities from the culture of achievement: The case of the TIMSS, PIRLS, and PISA. Journal of Education Policy, 28(2), 216–230. doi: 10.1080/02680939.2012.708789
111. Segeritz, M., & Pant, H. A., (2013). Do they feel the same way about math? Testing measurement invariance of the PISA “students’ approaches to learning” instrument across immigrant groups within Germany. Educational and Psychological Measurement, 73(4), 601–630. Retrieved from http://epm.sagepub.com/cgi/doi/10.1177/0013164413481802
112. Sellar, S., & Lingard, B., (2013). The OECD and global governance in education. Journal of Education Policy, 28(5), 710–725. doi: 10.1080/02680939.2013.779791
113. Sellar, S., & Lingard, B., (2014). The OECD ad the expansion of PISA: New global modes of governance in education. British Educational Research Journal, 40(6), 917–936. doi: 10.1002/berj.3120
114. Shapira, M., (2012). An exploration of differences in mathematics attainment among immigrant pupils in 18 OECD countries. European Educational Research Journal, 11(1), 68. doi:10.2304/eerj.2012.11.1.68
115. Shiel, G., & Eivers, E., (2009). International comparisons of reading literacy: What can they tell us? Cambridge Journal of Education, 39(3), 345–360. doi: 10.1080/03057640903103736
116. Shrout, P. E., (1998). Measurement reliability and agreement in psychiatry. Statistical Methods in Medical Research, 7(3), 301–317. doi: 10.1177/096228029800700306
117. Shrout, P. E., & Fleiss, J. L., (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428. doi: 10.1037/0033-2909.86.2.420
118. Sireci, S., (2015). Beyond ranking of nations: Innovative research on PISA. Teachers College Record, 117, 1–8.
119. Sirin, S. R., (2005). Socioeconomic status and academic achievement: A meta-analytic review of research. Review of Educational Research, 75(3), 417–453. doi: 10.3102/00346543075003417
120. Strakova, J., (2007). The impact of the structure of the education system on the development of educational inequalities in the Czech Republic. Sociologicky Casopis-Czech Sociological Review, 43(3), 589–610.
121. Sullivan, K., Perry, L. B., & McConney, A., (2013). How do school resources and academic performance differ across Australia’s rural, regional and metropolitan communities? Australian Educational Researcher, 40(3), 353–372. doi: 10.1007/s13384-013-0100-5
122. Sünker, H., (2004). Education and reproduction of social inequality: German politics and sociology of education. Policy Futures in Education, 2(3-4), 593–606. doi: 10.2304/pfie.2004.2.3.12
123. Tienken, C. H., (2014). PISA problems. AASA Journal of Scholarship and Practice, 10(4), 4–18.
124. Turmo, A., (2004). Scientific literacy and socio-economic background among 15-year-olds: A nordic perspective. Scandinavian Journal of Educational Research, 48(3), 287–305. doi:10.1080/00313830410001695745
125. Vidal-Abarca, E., Gilabert, R., Ferrer, A., Ávila, V., Martínez, T., Mañá, A., Llorens, A.-C., Gil, L., Cerdán, R., Ramos, L., & Serrano, M.-A., (2014). TuinLEC, an intelligent tutoring system to improve reading literacy skills / TuinLEC, un tutor inteligente para mejorar la competence lector. Journal for the Study of Education and Development, 37(1), 25–56.
126. Wagemaker, H., (2008). Choices and trade-offs: Reply to McGaw. Assessment in Education: Principles, Policy & Practice, 15(3), 267–278. Retrieved from http://www.tandfonline.com/doi/abs/10.1080/09695940802417491
127. Williams, J. H., (2005). Cross-national variations in rural mathematics achievement: A descriptive overview. Journal of Research in Rural Education, 20(5), 1–18.
128. Wu, M., (2005). The role of plausible values in large-scale surveys. Studies in Educational Evaluation, 31, 114–128. doi: 10.1016/j.stueduc.2005.05.005
129. Xie, Y., & Wilson, M., (2008). Investigating DIF and extensions using an LLTM approach and also an individual differences approach: An international testing context. Psychology Science Quarterly, 50(3), 403–416.
130. Xu, J., & Hampden-Thompson, G., (2012). Cultural reproduction, cultural mobility, cultural resources, or trivial effect? A comparative approach to cultural capital and educational performance. Comparative Education Review, 56(1), 98–124. doi: 10.1086/661289
131. Yildirim, H. H., & Berberoĝlu, G., (2009). Judgmental and statistical DIF analyses of the PISA-2003 mathematics literacy items. International Journal of Testing, 9(2), 108–121. doi: 10.1080/15305050902880736
132. Yildirim, H. H., & Yildirim, S., (2011). Correlates of communalities as matching variables in differential item functioning analyses. Hacettepe University Journal of Education, 40, 386–396.
133. Yildirim, H. H., Yildirim, S., & Verhelst, N., (2014). Profile analysis as a generalized differential item functioning analysis method. Education and Science, 39(172), 49–64.
134. Zamir, S., & Sabo, H., (2012). PISA assessment: The problematic issue of administrating PISA science literacy survey to ultra-orthodox pupils in Israel, 2006. Acta Didactica Napocensia, 5(3), 43–48.