Ratings and rankings: Voodoo or science?

Journal of the Royal Statistical Society. Series A: Statistics in Society

Volumn 176, Issue 3, 2013, Pages 609-634

  Paruolo, Paolo ^a   Saisana, Michaela ^b   Saltelli, Andrea ^b  

^a UNIVERSITY OF INSUBRIA   (Italy)

^b JOINT RESEARCH CENTRE   (Italy)

Author keywords

Composite indicators; Linear aggregation; Modelling; Pearson's correlation ratio; Weights

Indexed keywords

EID: 84877801828     PISSN: 09641998     EISSN: 1467985X     Source Type: Journal    
DOI: 10.1111/j.1467-985X.2012.01059.x     Document Type: Article

References (52)

1. Agrast, M. D., Botero, J. C. and Ponce, A. (2010) Rule of law index 2010. Technical Report. World Justice Project, Washington DC. (Available from http://worldjusticeproject.org/.)
2. Balinski, M. and Laraki, R. (2010) Majority Judgment: Measuring, Ranking and Electing. Cambridge: MIT Press.
3. Bandura, R. (2008) A survey of composite indices measuring country performance: 2008 update. Technical Report. United Nations Development Programme, Office of Development Studies, New York.
4. Billaut, J. C., Bouyssou, D. and Vincke, P. (2010) Should you believe in the Shanghai ranking? Scientometrics, 84, 237-263.
5. Bird, S. M., Cox, D., Farewell, V. T., Goldstein, H., Holt, T. and Smith, P. C. (2005) Performance indicators: good, bad, and ugly. J. R. Statist. Soc. A, 168, 1-27.
6. Bowman, A. W. and Azzalini, A. (1997) Applied Smoothing Techniques for Data Analysis: a Kernel Approach with S-Plus Illustrations. Oxford: Clarendon.
7. Boyssou, D., Marchant, T., Pirlot, M. and Tsoukiàs, A. (2006) Evaluation and Decision Models with Multiple Criteria: Stepping Stones for the Analyst. Berlin: Springer.
8. Center for World-Class Universities (2008) Academic ranking of world universities-2008. Technical Report. Center for World-Class Universities, Institute of Higher Education, Shanghai Jiao Tong University, Shanghai. (Available from http://www.arwu.org/.)
9. Decancq, K. and Lugo, M. (2010) Weights in multidimensional indices of well-being: an overview. Econmetr. Rev., to be published, doi 10.1080/07474938.2012.69041.
10. Freedom House (2011) Freedom of the press 2011. Technical Report. Freedom House, Washington DC. (Available from http://www.freedomhouse.org.)
11. Hacking, I. (1990) The Taming of Chance. Cambridge: Cambridge University Press.
12. Hand, D. (2009) Measurement Theory and Practice: the World through Quantification. Chichester: Wiley.
13. Hendrik, W., Howard, C. and Maximilian, A. (2008) Consequences of data error in aggregate indicators: evidence from the human development index. Technical Report. Department of Agricultural and Resource Economics, University of California at Berkeley, Berkeley.
14. van de Kerk, G. and Manuel, A. R. (2008) A comprehensive index for a sustainable society: the SSI, Sustainable Society Index. J. Ecol. Econ., 66, 228-242.
15. Leckie, G. and Goldstein, H. (2009) The limitations of using school league tables to inform school choice. J. R. Statist. Soc. A, 172, 835-851.
16. Li, G., Rabitz, H., Yelvington, P., Oluwole, O., Bacon, F., Kolb, C. and Schoendorf, J. (2010) Global sensitivity analysis for systems with independent and/or correlated inputs. J. Phys. Chem., 114, 6022-6032.
17. Munda, G. (2008) Social Multi-criteria Evaluation for a Sustainable Economy. Berlin: Springer.
18. Munda, G. and Nardo, M. (2009) Non-compensatory/non-linear composite indicators for ranking countries: a defensible setting. Appl. Econ., 41, 1513-1523.
19. Nardo, M. (2009) Product market regulation: robustness and critical assessment 1998-2003-2007-how much confidence can we have on PMR ranking?Technical Report EUR 23667. Joint Research Centre, European Commission, Ispra.
20. Nicoletti, G., Scarpetta, S. and Boylaud, O. (2000) Summary indicators of product market regulation with an extension to employment protection legislation. Working Paper 226 Economics Department, Organisation for Economic Co-operation and Development, Paris. (Available from http://dx.doi.org/10.1787/215182844604.)
21. Organisation for Economic Co-operation and Development (2008) Handbook on Constructing Composite Indicators: Methodology and User Guide. Paris: Organisation for Economic Co-operation and Development.
22. Pearson, K. (1905) Mathematical Contributions to the Theory of Evolution, Drapers' Company Research Memoirs, vol.XIV, On the General Theory of Skew Correlation and Non-linear Regression London: Dulau.
23. Plischke, E. (2010) An effective algorithm for computing global sensitivity indices (EASI). Reliab. Engng Syst. Safty, 95, 354-360.
24. Ratto, M. and Pagano, A. (2010) Recursive algorithms for efficient identification of smoothing spline anova models. Adv. Statist. Anal., 94, 367-388.
25. Ratto, M., Pagano, A. and Young, P. (2007) State dependent parameter metamodelling and sensitivity analysis. Comput. Phys. Communs, 177, 863-876.
26. Ravallion, M. (2010) Troubling tradeoffs in the human development index. Policy Research Working Paper 5484 Development Research Group, World Bank, Washington DC.
27. Reporters Sans Frontières (2011) Press freedom index. Technical Report Reporters Without Borders, Paris. (Available from http://en.rsf.org/pressfreedomindex2010, 1034.html.)
28. Rotberg, R. and Gisselquist, R. (2008) Strengthening African governance: Ibrahim index of African governance; results and rankings 2008. Technical Report. Kennedy School of Government, Harvard University, Boston.
29. Ruppert, A., Sheather, S. J. and Wand, M. P. (1995) An effective bandwidth selector for local least squares regression. J. Am. Statist. Ass., 90, 1257-1270.
30. Ruppert, D. and Wand, M. P. (1994) Multivariate locally weighted least squares regression. Ann. Statist., 22, 1346-1370.
31. Saaty, T. (1980) The Analytic Hierarchy Process. New York: McGraw-Hill.
32. Saaty, T. L. (1987) The analytic hierarchy process: what it is and how it is used. Math. Modllng, 9, 161-176.
33. Saisana, M., Annoni, P. and Nardo, M. (2009) A robust model to measure governance in African countries. Technical Report EUR 23773. Joint Research Centre, European Commission, Ispra.
34. Saisana, M., d'Hombres, B. and Saltelli, A. (2011) Rickety numbers: volatility of university rankings and policy implications. Res. Poly, 40, 165-177.
35. Saisana, M., Saltelli, A. and Tarantola, S. (2005) Uncertainty and sensitivity analysis techniques as tools for the quality assessment of composite indicators. J. R. Statist. Soc. A, 168, 307-323.
36. Saltelli, A. (2002) Making best use of model valuations to compute sensitivity indices. Comput. Phys. Communs, 145, 280-297.
37. Saltelli, A. (2007) Composite Indicators between analysis and advocacy. Socl Indic. Res., 81, 65-77.
38. Saltelli, A., Annoni, P., Azzini, I., Campolongo, F., Ratto, M. and Tarantola, S. (2010) Variance based sensitivity analysis of model output: design and estimator for the total sensitivity index. Comput. Phys. Communs, 181, 259-270.
39. Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D., Saisana, M. and Tarantola, S. (2008) Global Sensitivity Analysis-the Primer. Chichester: Wiley.
40. Saltelli, A. and Tarantola, S. (2002) On the relative importance of input factors in mathematical models: safety assessment for nuclear waste disposal. J. Am. Statist. Ass., 97, 702-709.
41. Sobol', I. (1993) Sensitivity analysis for non-linear mathematical models. Math. Modlng Computnl Expt, 1, 407-414 (Engl. transl.).
42. Stanley, J. and Wang, M. (1968) Differential weighting: a survey of methods and empirical studies. Technical Report. Johns Hopkins University, Baltimore.
43. Stiglitz, J. E., Sen, A. and Fitoussi, J. (2009) Report by the Commission on the Measurement of Economic Performance and Social Progress. Technical Report Commission on the Measurement of Economic Performance and Social Progress, Paris. (Available from www.stiglitzsenfitoussi.fr.)
44. Tarantola, S., Gatelli, D. and Mara, T. (2006) Random balance designs for the estimation of first order global sensitivity indices. Reliab. Engng Syst. Safty, 91, 717-727.
45. Times Higher Education Supplement (2008) World university rankings. Technical Report. Times Higher Education Supplement, London. (Available from http://www.timeshighereducation.co.uk.)
46. Toulmin, S. (1990) Cosmopolis-the Hidden Agenda of Modernity. Chicago: University of Chicago Press.
47. United Nations Development Programme (2009) Human development report 2009. Technical Report. United Nations Development Programme, New York. (Available from http://hdr.undp.org/en/reports/.)
48. United Nations Development Programme (2010) Human development report 2010. Technical Report. United Nations Development Programme, New York. (Available from http://hdr.undp.org/en/reports/.)
49. Wang, M. W. and Stanley, J. C. (1970) Differential weighting: a review of methods and empirical studies. Rev. Educ. Res., 40, 663-705.
50. Wooldridge, J. M. (2010) Econometric Analysis of Cross Section and Panel Data, 2nd edn. Cambridge: MIT Press.
51. World Economic Forum (2010) The Global Competitiveness Report 2010-2011. Technical Report. World Economic Forum, Geneva. (Available from http://www.weforum.org/.)
52. Xu, C. and Gertner, G. (2011) Understanding and comparisons of different sampling approaches for the fourier amplitudes sensitivity test (fast). Computnl Statist. Data Anal., 55, 184-198.