How to analyze percentile citation impact data meaningfully in bibliometrics: The statistical analysis of distributions, percentile rank classes, and top-cited papers

Journal of the American Society for Information Science and Technology

Volumn 64, Issue 3, 2013, Pages 587-595

  Bornmann, Lutz ^a  

^a MAX PLANCK INSTITUTE   (Germany)

Author keywords

bibliometrics

Indexed keywords

BIBLIOMETRIC; BIBLIOMETRICS; CITATION IMPACT; EVALUATION STUDY; SAMPLE DATA; SCIENTIFIC PERFORMANCE;

INFORMATION SCIENCE;

PUBLISHING;

EID: 84874223207     PISSN: 15322882     EISSN: 15322890     Source Type: Journal    
DOI: 10.1002/asi.22792     Document Type: Article

References (43)

1. Albarrán, P., Crespo, J., Ortuño, I., &, Ruiz-Castillo, J., (2011). The skewness of science in 219 sub-fields and a number of aggregates. Scientometrics, 88 (2), 385-397. doi: 10.1007/s11192-011-0407-9.
2. Bornmann, L., (2010). Towards an ideal method of measuring research performance: some comments to the Opthof and Leydesdorff (2010) paper. Journal of Informetrics, 4 (3), 441-443.
3. Bornmann, L., (in press). The problem of percentile rank scores used with small reference sets. Journal of the American Society of Information Science and Technology.
4. Bornmann, L., &, Daniel, H.-D., (2007). What do we know about the h index? Journal of the American Society for Information Science and Technology, 58 (9), 1381-1385. doi: 10.1002/asi.20609.
5. Bornmann, L., &, Daniel, H.-D., (2008). What do citation counts measure? A review of studies on citing behavior. Journal of Documentation, 64 (1), 45-80.
6. Bornmann, L., de Moya Anegón, F., &, Leydesdorff, L., (2012). The new Excellence Indicator in the World Report of the SCImago Institutions Rankings 2011. Journal of Informetrics, 6 (2), 333-335.
7. Bornmann, L., &, Mutz, R., (2011). Further steps towards an ideal method of measuring citation performance: The avoidance of citation (ratio) averages in field-normalization. Journal of Informetrics, 5 (1), 228-230.
8. Bornmann, L., Mutz, R., Marx, W., Schier, H., &, Daniel, H.-D., (2011). A multilevel modelling approach to investigating the predictive validity of editorial decisions: Do the editors of a high-profile journal select manuscripts that are highly cited after publication? Journal of the Royal Statistical Society-Series A (Statistics in Society), 174 (4), 857-879.
9. Bornmann, L., Mutz, R., Neuhaus, C., &, Daniel, H.-D., (2008). Use of citation counts for research evaluation: Standards of good practice for analyzing bibliometric data and presenting and interpreting results. Ethics in Science and Environmental Politics, 8, 93-102.
10. Bornmann, L., Schier, H., Marx, W., &, Daniel, H.-D., (2011). Is interactive open access publishing able to identify high-impact submissions? A study on the predictive validity of Atmospheric Chemistry and Physics by using percentile rank classes. Journal of the American Society for Information Science and Technology, 62 (1), 61-71.
11. Bornmann, L., Schier, H., Marx, W., &, Daniel, H.-D., (2012). What factors determine citation counts of publications in chemistry besides their quality? Journal of Informetrics, 6, 11-18.
12. Boyack, K. W., (2004). Mapping knowledge domains: Characterizing PNAS. Proceedings of the National Academy of Sciences of the United States of America, 101, 5192-5199.
13. Cohen, J., (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates, Publishers.
14. Cox, N. J., (2004). Speaking Stata: Graphing categorical and compositional data. The Stata Journal, 4 (2), 190-215.
15. Hardin, J., &, Hilbe, J., (2012). Generalized linear models and extensions. College Station, TX: Stata Corporation.
16. Hirsch, J. E., (2005). An index to quantify an individual's scientific research output. Proceedings of the National Academy of Sciences of the United States of America, 102 (46), 16569-16572.
17. Hosmer, D. W., &, Lemeshow, S., (2000). Applied logistic regression (2nd ed.). Chichester, UK: John Wiley & Sons, Inc.
18. Kohler, U., &, Kreuter, F., (2009). Data analysis using Stata (2nd ed.). College Station, TX: Stata Press, Stata Corporation.
19. Leydesdorff, L., (2012). Accounting for the uncertainty in the evaluation of percentile ranks. Journal of the American Society for Information Science and Technology, 63 (11), 2349-2350.
20. Leydesdorff, L., &, Bornmann, L., (2011). Integrated Impact Indicators (I3) compared with Impact Factors (IFs): An alternative research design with policy implications. Journal of the American Society of Information Science and Technology, 62 (11), 2133-2146.
21. Leydesdorff, L., Bornmann, L., Mutz, R., &, Opthof, T., (2011). Turning the tables in citation analysis one more time: Principles for comparing sets of documents. Journal of the American Society for Information Science and Technology, 62 (7), 1370-1381.
22. Maurer, H., (2012). A linear ordering of a multi-parameter universe is usually nonsense. Theoretical Computer Science, 429 (0), 222-226.
23. Mitchell, M. N., (2012). Interpreting and visualizing regression models using Stata. College Station, TX: Stata Corporation.
24. Mutz, R., &, Daniel, H.-D., (2012). Skewed citation distributions and bias factors: Solutions to two core problems with the journal impact factor. Journal of Informetrics, 6 (2), 169-176.
25. National Science Board. (2012). Science and engineering indicators 2012. Arlington, VA: National Science Foundation (NSB 12-01).
26. Prathap, G., (2012). Energy indicators and percentile ranking normalization. Scientometrics, 91 (3), 997-1003.
27. Rabe-Hesketh, S., &, Everitt, B., (2004). A handbook of statistical analyses using Stata. Boca Raton, FL: Chapman & Hall/CRC.
28. Radicchi, F., &, Castellano, C., (2011). Rescaling citations of publications in physics. Physical Review E, 83 (4).
29. Rousseau, R., (2012). Basic properties of both percentile rank scores and the I3 indicator. Journal of the American Society for Information Science and Technology, 63 (2), 416-420.
30. Schreiber, M., (2012a). Inconsistencies of recently proposed citation impact indicators and how to avoid them. Retrieved from http://arxiv.org/abs/ 1202.3861
31. Schreiber, M., (2012b). Uncertainties and ambiguities in percentiles and how to avoid them. Retrieved from http://arxiv.org/abs/1205.3588
32. Sheskin, D., (2007). Handbook of parametric and nonparametric statistical procedures (4th ed.). Boca Raton, FL: Chapman & Hall/CRC.
33. StataCorp. (2011). Stata statistical software: Release 12. College Station, TX: Stata Corporation.
34. Tijssen, R., &, van Leeuwen, T., (2006). Centres of research excellence and science indicators. Can "excellence" be captured in numbers? In, W. Glänzel, (Ed.), Ninth International Conference on Science and Technology Indicators (pp. 146-147). Leuven, Belgium: Katholieke Universiteit Leuven.
35. Tijssen, R., Visser, M., & van Leeuwen, T., (2002). Benchmarking international scientific excellence: Are highly cited research papers an appropriate frame of reference? Scientometrics, 54 (3), 381-397.
36. Wagner, C. S., &, Leydesdorff, L., (2012). An Integrated Impact Indicator (I3): A new definition of "impact" with policy relevance. Retrieved from http://arxiv.org/abs/1205.1419
37. Waltman, L., Calero-Medina, C., Kosten, J., Noyons, E. C.M., Tijssen, R. J. W., van, Eck, N. J.,., Wouters, P., (2012 a). The Leiden Ranking 2011/2012: data collection, indicators, and interpretation. Retrieved from http://arxiv.org/abs/1202.3941
38. Waltman, L., Calero-Medina, C., Kosten, J., Noyons, E. C. M., Tijssen, R. J. W., van, Eck, N. J.,., Wouters, P., (2012 b). The Leiden ranking 2011/2012: Data collection, indicators, and interpretation. Journal of the American Society for Information Science and Technology, 63 (12), 2419-2432.
39. Waltman, L., &, Schreiber, M., (2012). On the calculation of percentile-based bibliometric indicators. Retrieved from http://arxiv.org/abs/ 1205.0646
40. Waltman, L., &, van Eck, N. J., (2012). The inconsistency of the h-index. Journal of the American Society for Information Science and Technology, 63 (2), 406-415.
41. Waltman, L., van Eck, N. J., van Leeuwen, T. N., Visser, M. S., & van Raan, A. F. J., (2011). Towards a new crown indicator: Some theoretical considerations. Journal of Informetrics, 5 (1), 37-47.
42. Williams, R., (2012). Using the margins command to estimate and interpret adjusted predictions and marginal effects. The Stata Journal, 12 (2), 308-331.
43. Zitt, M., &, Lahatte, A., (2009). Performance and over-ranking measures on citation profiles. In Proceedings of the 12th Conference of the International Society for Scientometrics and Informetrics, Rio de Janeiro, Brazil.