Cautions on OECD's recent educational survey (PISA): Rejoinder to OECD's response

Oxford Review of Education

Volumn 30, Issue 4, 2004, Pages 569-573

  Prais, S J ^a  

^a NATIONAL INSTITUTE OF ECONOMIC AND SOCIAL RESEARCH   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 10844250269     PISSN: 03054985     EISSN: None     Source Type: Journal    
DOI: 10.1080/0305498042000303017     Document Type: Review

References (8)

1. S. J. Prais (2003) 'Cautions on OECD's recent educational survey', Oxford Review of Education, 29(2), pp. 139-163.
2. R. J. Adams (2003) 'Response to Cautions on OECD's recent educational survey', Oxford Review of Education, 29(3), pp. 377-390.
3. I thought it better to concentrate here, as positively as possible, on these five main points. I fear it would try the reader's patience if I took up space in a point-by-point, courtroom-style rebuttal of the many small niggling points made in the OECD response.
4. The point was, however, well understood in the article by D. Elliott (1993) referred to by the OECD Rejoinder in their note 18 (but notice the year of publication is not 1933 as they state): 'substitution of non-respondents' with replacements does 'not in any way reduce[e] non-response bias' (Elliott, p. 11). I agree that this worry applied also to previous IEA studies.
5. Such a measured failure rate is not necessarily the precise complement of success rates, whether measured by average attainments or top attainments (some English secondary schools have low-attaining sets of 15-year-olds with large absence rates of pupils, while a neighbouring school with the same average attainments may not be plagued in this way (i.e. it may have the same average, but lower variability).
6. H. Goldstein, 'International comparisons of student attainment: some issues arising from the PISA study', Assessment in Education (forthcoming 2004) ; the author is Professor of Statistical Methods at the University of London's Institute of Education, and is an expert in IRT.
7. I was able, with the help of my Institute's computing experts, to download the country-averages for each question (as mentioned in my original Cautious); but it was not easy. Following the latest reference in the OECD Response (foot of p. 368), we tried again at end-March 2004 and still found it difficult (the pages were being revised, and not available). Occasional availability on the internet is clearly not quite the same as publication in book form. In any event, the more important point is that systematic comment on different countries' success in responding to different questions is not to be found in the PISA publications (the reader may like to consider again my observation that Greek and Portuguese girls-in contrast to Swiss, German and Australian boys-did particularly poorly in a multiple-choice question on racing-track circuits-so poorly that those girls would have done better to tick randomly; Cautions, note 11, p. 156).
8. D. Foxman, Learning Mathematics and Science, National Foundation for Educational Research, Slough (1992), p. 74 (curiously, there is no reference to this 1990 ETS survey in the OECD's Response). Of course, nothing stands still in this world: English pupils may have improved in the subsequent decade, and Swiss pupils worsened; but nothing that our teams have seen with their own eyes on successive visits would be consistent with a change as large as implied by PISA.