Rickety numbers: Volatility of university rankings and policy implications

Research Policy

Volumn 40, Issue 1, 2011, Pages 165-177

  Saisana, Michaela ^a   D'Hombres, Béatrice ^a   Saltelli, Andrea ^a  

^a JOINT RESEARCH CENTRE   (Italy)

Author keywords

Composite indicator; Ranking; Sensitivity analysis; Uncertainty analysis; University

Indexed keywords

ALTERNATIVE RANKING; COMPOSITE INDICATORS; CONCEPTUAL FRAMEWORKS; HIGHER EDUCATION; MULTI-MODELLING; POLICY IMPLICATIONS; RANKING; ROBUSTNESS ANALYSIS; SOCIAL ACTORS; STATISTICAL INFERENCE; UNIVERSITY;

SENSITIVITY ANALYSIS;

UNCERTAINTY ANALYSIS;

EID: 78650690464     PISSN: 00487333     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.respol.2010.09.003     Document Type: Article

References (51)

1. G. Abramo, C.A. d'Angelo, and A. Caprasecca Allocative efficiency in public research funding: can biblometrics help? Research Policy 38 2009 206 215
2. D. Acemoglu, P. Aghion, and F. Zilibotti Distance to frontier, selection, and economic growth Journal of the European Economic Association 4 1 2006 37 74
3. Aghion, P.; Dewatripont, M.; Hoxby, C.; Sapir, A.; 2008. Higher aspirations: an agenda for reforming European universities, Bluegel Blueprint Series N.5.
4. Billaut, J.C.; Bouyssou, D.; Vincke P.; 2009. Should you believe in the Shanghai ranking? http://hal.archives-ouvertes.fr/docs/00/40/39/93/PDF/Shanghai- JCB-DB-PV.pdf.
5. F. Booysen An overview and evaluation of composite indices of development Social Indicators Research 59 2 2002 115 151
6. Borda, J.C.de, 1784. Mémoire sur le elections au scrutiny, Histoire de l'Académie Royale des Sciences, Paris.
7. D.A. Brand, M. Saisana, L.A. Rynn, F. Pennoni, and A.B. Lowenfels Comparative analysis of alcohol control policies in 30 countries PLoS Medicine 4 4 2007 752 759
8. L. Cherchye, C.A. Knox Lovell, W. Moesen, and T. Van Puyenbroeck One market, one number? A composite indicator of EU internal market dynamics European Economic Review 51 2007 749 779
9. L. Cherchye, W. Moesen, N. Rogge, T. Van Puyenbroeck, M. Saisana, A. Saltelli, R. Liska, and S. Tarantola Creating composite indicators with DEA and robustness analysis: the case of the Technology Achievement Index Journal of Operational Research Society 59 2008 239 251
10. Condorcet, M.de, 1785. Essai sur l'application de l'analyse la probabilité des décisions rendues la probabilité des voix, De l' Imprimerie Royale, Paris.
11. C. Dehon, A. McCathie, and V. Verardi Uncovering excellence in academic rankings: a closer look at the Shanghai ranking Scientometrics 83 2 2009 515 524
12. P. Ederer, P. Schuller, and S. Willms University Systems Ranking: Citizens and Society in the Age of Knowledge 2007 Lisbon Council Policy Brief Brussels
13. B. Efron Bootstrap methods: another look at the jackknife The Annals of Statistics 7 1 1979 1 26
14. M. Enserink Who ranks the rankers Science 317 2007 1026 1028
15. European Commission, 2008. Progress towards the Lisbon Objectives in Education and Training, Indicators and Benchmarks, Commission Staff Working Document, Brussels.
16. D. Hand Measurement Theory and Practice: The World through Quantification 2004 Hodder Arnold Publisher
17. E. Hazelkorn Impact and influence of league tables and ranking systems on higher education decision-making Higher Education Management and Policy 19 2 2007 87 110
18. D.D. Hendel, and I. Stolz A comparative analysis of higher ranking systems in Europe Tertiary Education and management 14 3 2008 173 189
19. B. Jacobs, and F. van der Ploeg Guide to reform of higher education: a European perspective Economic Policy, CEPR, CES, MSH 21 47 2006 535 592
20. J. Kemeny Mathematics without numbers Daedalus 88 1959 571 591
21. P. Kennedy A Guide to Econometrics fifth ed. 2007 Blackwell
22. E. Leamer Let's take the con out of econometrics, and sensitivity analysis would help C. Granger, Modelling Economic Series 1990 Clarendon Press Oxford
23. Le Figaro, 2008. La France, mauvaise élve du classement de Shanghaï, 06 August 2008. http://www.lefigaro.fr/.
24. Le Monde, 2008. Comment sauver l'université franaise, 14 August 2008, http://www.lemonde.fr.
25. Le Monde, 2008. La France veut hisser ses universités dans les classements mondiaux, 3 July 2008, http://www.lemonde.fr.
26. Les Echos, 2008. Classement de Shanghai: les universités franaises la traine, 7 August 2008, http://www.lesechos.fr/info/france/4759189.htm.
27. S. Marginson Global university rankings: implications in general and for Australia Journal of Higher Education Policy and Management 29 2 2007 131 142
28. G. Munda Social Multi-criteria Evaluation for a Sustainable Economy 2008 Springer Berlin
29. Nature News Academics strike back at spurious rankings Nature 447 May 2007 514 515
30. Nicoletti, G.; Scarpetta, S.; Boylaud, O.; 2000. Summary indicators of product market regulation with an extension to employment protection legislation, OECD, Economics department working papers No. 226, ECO/WKP(99)18.
31. OECD Handbook on Constructing Composite Indicators. Methodology and User Guide 2008 OECD Paris
32. Saisana, M.; Annoni, P.; Nardo, M.; 2009. A robust model to measure governance in African countries, Report 23773, European Commission, JRC-IPSC, Italy.
33. Saisana, M.; 2008. The 2007 Composite Learning Index: Robustness Issues and Critical Assessment, Report 23274, European Commission, JRC-IPSC, Italy.
34. Saisana M.; D'Hombres B.; 2008. Higher Education Rankings: Robustness Issues and Critical Assessment, Report 23487, European Commission, JRC-IPSC, Italy.
35. Saisana, M.; Munda G.; 2008. Knowledge Economy: measures and drivers, Report 23486, European Commission, JRC-IPSC, Italy.
36. M. Saisana, A. Saltelli, and S. Tarantola Uncertainty and sensitivity analysis techniques as tools for the analysis and validation of composite indicators Journal of the Royal Statistical Society A 168 2 2005 307 323
37. Saisana, M.; Saltelli, A.; 2010. Uncertainty and Sensitivity Analysis of the 2010 Environmental Performance Index, Report 24269, European Commission, Joint Research Centre, Italy.
38. J. Salmi The Challenge of Establishing World Class Universities 2009 World Bank Publications Washington 136 pp.
39. A. Saltelli Composite indicators between analysis and advocacy Social Indicators Research 81 1 2007 65 77
40. A. Saltelli, M. Ratto, T. Andres, F. Campolongo, J. Cariboni, D. Gatelli, M. Saisana, and S. Tarantola Global Sensitivity Analysis 2008 The Primer, John Wiley & Sons England
41. A. Sapir, P. Aghion, G. Bertola, M. Hellwig, J. Pisani-Ferry, D. Rosati, J. Viñals, and H. Wallace An Agenda for a Growing Europe: The Sapir Report 2004 Oxford University Press
42. Stiglitz, J.E.; Sen, A.; Fitoussi JP, 2009, Report by the Commission on the Measurement of Economic Performance and Social Progress, www.stiglitz-sen-fitoussi.fr.
43. P. Taylor, and R. Braddock International university ranking systems and the idea of university excellence Journal of Higher Education Policy and Management 29 3 2007 245 260
44. A. Usher, and M. Savino A global survey of university ranking and league tables Higher Education in Europe 32 1 2007 5 15
45. A.F.J. van Raan Fatal Attraction: conceptual and methodological problems in the ranking of universities by bibliometric methods Scientometrics 62 1 2005 133 143
46. A.F.J. van Raan Challenges in the Ranking of Universities J. Sadlak, L.N. Cai, The World-Class University and Ranking: Aiming Beyond Status 2007 UNESCO-CEPES Bucharest
47. R. Veugelers, and F. van der Ploeg Towards evidence-based reform of European universities CESifo Economic Studies 54 2007 99 120
48. R. Williams Broadening the criteria: lessons from the australian rankings J. Sadlak, L.N. Cai, The World-Class University and Ranking: Aiming Beyond Status 2007 UNESCO-CEPES Bucharest
49. Y-H.B. Wong, and J.E. Beasley Restricting weight flexibility in data envelopment analysis Journal of the Operational Research Society 47 1990 136 150
50. H.P. Young, and A. Levenglick A consistent extension of Condorcet's election principle SIAM Journal on Applied Mathematics 35 1978 285 300
51. M. Zitt, and G. Filliatreau Big is (made) beautiful - some comments about Shanghai-ranking of world-class universities J. Sadlak, L.N. Cai, The World-Class University and Ranking: Aiming Beyond Status 2007 UNESCO-CEPES Bucharest