The use of percentiles and percentile rank classes in the analysis of bibliometric data: Opportunities and limits

Journal of Informetrics

Volumn 7, Issue 1, 2013, Pages 158-165

  Bornmann, Lutz ^a   Leydesdorff, Loet ^b   Mutz, Rüdiger ^c  

^a MAX PLANCK INSTITUTE   (Germany)

^b UNIVERSITY OF AMSTERDAM   (Netherlands)

^c ETH ZURICH   (Switzerland)

Author keywords

Citation impact; Normalized citation impact; Percentile; Percentile rank class; Reference set

Indexed keywords

EID: 84870509676     PISSN: 17511577     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.joi.2012.10.001     Document Type: Article

References (36)

1. Blom G. Statistical estimates and transformed beta-variables 1958, John Wiley & Sons, New York, NY, USA.
2. Bornmann, L. How to analyze percentile impact data meaningfully in bibliometrics? The statistical analysis of distributions, percentile rank classes and top-cited papers. Journal of the American Society for Information Science and Technology, in press-a.
3. Bornmann, L. The problem of percentile rank scores used with small reference sets. Journal of the American Society of Information Science and Technology, in press-b.
4. Bornmann L., Daniel H.-D. What do citation counts measure? A review of studies on citing behavior. Journal of Documentation 2008, 64(1):45-80. 10.1108/00220410810844150.
5. Bornmann L., de Moya Anegón F., Leydesdorff L. The new Excellence Indicator in the World Report of the SCImago Institutions Rankings 2011. Journal of Informetrics 2012, 6(2):333-335. 10.1016/j.joi.2011.11.006.
6. Bornmann L., Leydesdorff L. Which cities produce more excellent papers than can be expected? A new mapping approach-Using Google Maps-Based on statistical significance testing. Journal of the American Society of Information Science and Technology 2011, 62(10):1954-1962.
7. Bornmann L., Mutz R. Further steps towards an ideal method of measuring citation performance: The avoidance of citation (ratio) averages in field-normalization. Journal of Informetrics 2011, 5(1):228-230.
8. Bornmann L., Mutz R., Marx W., Schier H., Daniel H.-D. A multilevel modelling approach to investigating the predictive validity of editorial decisions: Do the editors of a high-profile journal select manuscripts that are highly cited after publication?. Journal of the Royal Statistical Society - Series A (Statistics in Society) 2011, 174(4):857-879. 10.1111/j.1467-985X.2011.00689.x.
9. Cox, N. J. (2005). Calculating percentile ranks or plotting positions. Retrieved May 30, from http://www.stata.com/support/faqs/stat/pcrank.html.
10. Cunnane C. Unbiased plotting positions - A review. Journal of Hydrology 1978, 37(3-4):205-222. 90017-3. 10.1016/0022-1694(78).
11. Egghe L., Rousseau R. An informetric model for the Hirsch-index. Scientometrics 2006, 69(1):121-129.
12. Glänzel W., Thijs B., Schubert A., Debackere K. Subfield-specific normalized relative indicators and a new generation of relational charts: Methodological foundations illustrated on the assessment of institutional research performance. Scientometrics 2009, 78(1):165-188.
13. Gringorten I.I. A plotting rule for extreme probability paper. Journal of Geophysical Research 1963, 68(3):813.
14. Guerrero-Bote, V. P., & de Moya-Anegon, F. (2012). A further step forward in measuring journals' scientific prestige: The SJR2 indicator. Retrieved July 3, from http://arxiv.org/abs/1201.4639.
15. Harter H.L. Another look at plotting positions. Communications in Statistics - Theory and Methods 1984, 13(13):1613-1633.
16. Hazen A. Storage to be provided in impounding reservoirs for municipal water supply. Transactions of American Society of Civil Engineers 1914, 77:1539-1640.
17. Hintze J.L., Nelson R.D. Violin plots: A box plot-density trace synergism. The American Statistician 1998, 52(2):181-184.
18. Kohler U., Kreuter F. Data analysis using Stata 2009, Stata Press, Stata Corporation, College Station, TX, USA. 2nd ed.
19. Leydesdorff, L. Accounting for the uncertainty in the evaluation of percentile ranks. Journal of the American Society for Information Science and Technology, available at , in press. http://arxiv.org/abs/1204.1894.
20. Leydesdorff L., Bornmann L. Testing differences statistically with the Leiden ranking. Scientometrics 2012, 92(3):781-783.
21. Leydesdorff L., Bornmann L., Mutz R., Opthof T. Turning the tables in citation analysis one more time: Principles for comparing sets of documents. Journal of the American Society for Information Science and Technology 2011, 62(7):1370-1381.
22. Moed H.F., Debruin R.E., Van Leeuwen T.N. New bibliometric tools for the assessment of national research performance - Database description, overview of indicators and first applications. Scientometrics 1995, 33(3):381-422.
23. National Science Board Science and engineering indicators 2012 2012, National Science Foundation (NSB 12-01), Arlington, VA, USA.
24. Opthof T., Leydesdorff L. Caveats for the journal and field normalizations in the CWTS (Leiden) evaluations of research performance. Journal of Informetrics 2010, 4(3):423-430.
25. Rousseau R. Basic properties of both percentile rank scores and the I3 indicator. Journal of the American Society for Information Science and Technology 2012, 63(2):416-420. 10.1002/asi.21684.
26. Schreiber, M. (2012). Inconsistencies of recently proposed citation impact indicators and how to avoid them. Retrieved February 20, from http://arxiv.org/abs/1202.3861.
27. Schubert A., Braun T. Relative indicators and relational charts for comparative assessment of publication output and citation impact. Scientometrics 1986, 9(5-6):281-291.
28. StataCorp. Stata statistical software: Release 12 2011, Stata Corporation, College Station, TX, USA.
29. Tijssen R., van Leeuwen T. Centres of research excellence and science indicators. Can 'excellence' be captured in numbers?. Ninth international conference on science and technology indicators 2006, 146-147. Katholieke Universiteit Leuven, Leuven, Belgium. W. Glänzel (Ed.).
30. Tijssen R., Visser M., van Leeuwen T. Benchmarking international scientific excellence: Are highly cited research papers an appropriate frame of reference?. Scientometrics 2002, 54(3):381-397.
31. Vinkler P. The evaluation of research by scientometric indicators 2010, Chandos Publishing, Oxford, UK.
32. Vinkler P. The case of scientometricians with the absolute relative impact indicator. Journal of Informetrics 2012, 6(2):254-264. 10.1016/j.joi.2011.12.004.
33. Waltman, L., Calero-Medina, C., Kosten, J., Noyons, E. C. M., Tijssen, R. J. W., van Eck, et al. (2012). The Leiden Ranking 2011/2012: Data collection, indicators, and interpretation. Retrieved February 24, from http://arxiv.org/abs/1202.3941.
34. Waltman, L., Calero-Medina, C., Kosten, J., Noyons, E. C. M., Tijssen, R. J. W., van Eck, N.J., et al. The Leiden Ranking 2011/2012: Data collection, indicators, and interpretation. Journal of the American Society for Information Science and Technology, in press.
35. Waltman, L., & Schreiber, M. (2012). On the calculation of percentile-based bibliometric indicators. Retrieved May 4, from http://arxiv.org/abs/1205.0646.
36. Waltman L., van Eck N.J., van Leeuwen T.N., Visser M.S., van Raan A.F.J. Towards a new crown indicator: Some theoretical considerations. Journal of Informetrics 2011, 5(1):37-47. 10.1016/j.joi.2010.08.001.