Turning the tables on citation analysis one more time: Principles for comparing sets of documents

Journal of the American Society for Information Science and Technology

Volumn 62, Issue 7, 2011, Pages 1370-1381

  Leydesdorff, Loet ^a   Bornmann, Lutz ^b   Mutz, Rüdiger ^c   Opthof, Tobias ^d,e  

^a UNIVERSITY OF AMSTERDAM   (Netherlands)

^b MAX PLANCK INSTITUTE   (Germany)

^c ETH ZURICH   (Switzerland)

^d ACADEMIC MEDICAL CENTER   (Netherlands)

^e UNIVERSITY MEDICAL CENTER UTRECHT   (Netherlands)

Author keywords

[No Author keywords available]

Indexed keywords

CITATION ANALYSIS; CITATION DISTRIBUTION; CITATION IMPACT; CITATION SCORE; DATA SETS; EVALUATION CRITERIA; IMPACT INDICATORS; MEDICAL CENTER; PRINCIPAL INVESTIGATORS; REFERENCE SET; THREE DEGREES OF FREEDOM; UNIVERSITY OF AMSTERDAM;

EID: 79959373072     PISSN: 15322882     EISSN: 15322890     Source Type: Journal    
DOI: 10.1002/asi.21534     Document Type: Article

References (66)

1. Albarrán, P., &, Ruiz-Castillo, J., (2011). References made and citations received by scientific articles. Journal of the American Society for Information Science and Technology, 62 (1), 40-49.
2. Bornmann, L., (2010). Towards an ideal method of measuring research performance: Some comments to the Opthof and Leydesdorff (2010) paper. Journal of Inormetrics, 4 (3), 441- 443.
3. Bornmann, L., (2011). Scientific peer review. Annual Review of Information Science and Technology, 45, 199-245.
4. 0 at the micro level using data from chemistry. Journal of the American Society for Information Science and Technology, 60 (8), 1664-1670.
5. Bornmann, L., de Moya-Anegón, F., &, Leydesdorff, L., (2010). Do scientific advancements lean on the shoulders of giants? A bibliometric investigation of the Ortega hypothesis. PLoS ONE, 5 (10), e11344.
6. Bornmann, L., Leydesdorff, L., &, Van den Besselaar, P., (2010). A meta-evaluation of scientific research proposals: Different ways of comparing rejected to awarded applications. Journal of Informetrics, 4 (3), 211-220.
7. Bornmann, L., &, Mutz, R., (2011). Further steps towards an ideal method of measuring citation performance: The avoidance of citation (ratio) averages in field-normalization. Journal of Informetrics, 5 (1), 228-230.
8. Bornmann, L., Mutz, R., Marx, W., Schier, H., &, Daniel, H.-D., (2011). A multilevel modelling approach to investigating the predictive validity of editorial decisions: Do the editors of a high-profile journal select manuscripts that are highly cited after publication ? Journal of the Royal Statistical Society, Series A (Statistics in Society), 174 (4), 1 -23.
9. Bornmann, L., Mutz, R., Neuhaus, C., &, Daniel, H.-D., (2008). Use of citation counts for research evaluation: Standards of good practice for analyzing bibliometric data and presenting and interpreting results. Ethics in Science and Environmental Politics, 8, 93-102. (Pubitemid 351836496)
10. Bornmann, L., Schier, H., Marx, W., &, Daniel, H.-D., (2011). Is interactive open access publishing able to identify high-impact submissions? A study on the predictive validity of atmospheric chemistry and physics by using percentile rank classes. Journal of the American Society for Information Science and Technology, 62 (1), 61-71.
11. Boyack, K.W., (2004). Mapping knowledge domains: Characterizing PNAS. Proceedings of the National Academy of Sciences of the United States of America, 101, 5192-5199. (Pubitemid 38469126)
12. Boyack, K.W., Klavans, R., &, Börner, K., (2005). Mapping the backbone of science. Scientometrics, 64 (3), 351-374. (Pubitemid 41152144)
13. Cohen, J., (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.
14. Colliander, C., &, Ahlgren, P., (2011). The effects and their stability of field normalization baseline on relative performance with respect to citation impact: A case study of 20 natural science departments. Journal of Informetrics, 5 (1), 101-113.
15. Corder, G.W., &, Foreman, D.I., (2009). Nonparametric statistics for non-statisticians. Hoboken, NJ: John Wiley & Sons.
16. Cramér, H., (1980). Mathematical methods of statistics. Princeton, NJ: Princeton University Press.
17. CWTS. (2008). AMC-specifieke CWTS-analyse 1997-2006 (access via AMC intranet; unpublished, confidential). Leiden, The Netherlands: CWTS.
18. CWTS. (2010). AMC-specifieke CWTS-analyse 1997-2008 (access via AMC intranet; unpublished, confidential). Leiden, The Netherlands: CWTS.
19. Evidence Ltd. (2007). The use of bibliometrics to measure research quality in UK higher education institutions. London: Universities UK.
20. Garfield, E., (1971). The mystery of the transposed journal lists - wherein Bradford's Law of Scattering is generalized according to Garfield's Law of Concentration. Current Contents, 3 (33), 5-6.
21. Garfield, E., (1979). Is citation analysis a legitimate evaluation tool ? Scientometrics, 1 (4), 359-375.
22. Gingras, Y., &, LariviÃre, V., (2011). There are neither "king" nor "crown" in scientometrics: Comments on a supposed "alternative" method of normalization. Journal of Informetrics, 5 (1), 226-227.
23. Glänzel, W., (1992). Publication dynamics and citation impact: A multi-dimensional approach to scientometric research evaluation. In, P. Weingart, R. Sehringer, &, M. Winterhagen, (Eds.), Representations of science and technology. Proceedings of the International Conference on Science and Technology Indicators, Bielefeld, 10-12 June 1990 (pp. 209-224). Leiden, the Netherlands: DSWO / Leiden University Press.
24. Glänzel, W., (2010). On reliability and robustness of scientometrics indicators based on stochastic models. An evidence-based opinion paper. Journal of Informetrics, 4 (3), 313-319.
25. Glänzel, W., &, Schubert, A., (2003). A new classification scheme of science fields and subfields designed for scientometric evaluation purposes. Scientometrics, 56 (3), 357-367. (Pubitemid 41444803)
26. Glänzel, W., Thijs, B., Schubert, A., &, Debackere, K., (2009). Subfield-specific normalized relative indicators and a new generation of relational charts: Methodological foundations illustrated on the assessment of institutional research performance. Scientometrics, 78 (1), 165-188.
27. Goldstein, H., &, Healy, M.J.R., (1995). The graphical presentation of a collection of means. Journal of the Royal Statistical Society. Series A (Statistics in Society), 158 (1), 175-177.
28. Hirsch, J.E., (2005). An index to quantify an individual's scientific research output. Proceedings of the National Academy of Sciences of the United States of America, 102 (46), 16569-16572. (Pubitemid 41688817)
29. Jackson, C., (1996). Understanding psychological testing. Leicester, UK: British Psychological Society.
30. Joint Committee on Quantitative Assessment of Research. (2008). Citation statistics. A report from the International Mathematical Union (IMU) in cooperation with the International Council of Industrial and Applied Mathematics (ICIAM) and the Institute of Mathematical Statistics (IMS). Berlin, Germany: International Mathematical Union (IMU).
31. Kraemer, H.C., Morgan, G.A., Leech, N.L., Gliner, J.A., Vaske, J.J., &, Harmon, R.J., (2003). Measures of clinical significance. Journal of the American Academy of Child and Adolescent Psychiatry, 42 (12), 1524-1529. (Pubitemid 39585985)
32. Kurtz, M.J., &, Henneken, E.A., (2007). Open access does not increase citations for research articles from the Astrophysical Journal. Retrieved from
33. Kvam, P.H., &, Vidakovic, B., (2007). Nonparametric statistics with applications to science and engineering. Hoboken, NJ: John Wiley & Sons, Inc.
34. Levine, G., (1991). A guide to SPSS for analysis of variance. Hillsdale, NJ: Lawrence Erlbaum.
35. Leydesdorff, L., (2008). Caveats for the use of citation indicators in research and journal evaluation. Journal of the American Society for Information Science and Technology, 59 (2), 278-287. (Pubitemid 351205033)
36. Leydesdorff, L., &, Bornmann, L., (2011). How fractional counting affects the impact factor: Normalization in terms of differences in citation potentials among fields of science. Journal of the American Society for Information Science and Technology, 62 (2), 217-229
37. Leydesdorff, L., &, Opthof, T., (2010). Normalization at the field level: Fractional counting of citations. Journal of Informetrics, 4 (4), 644-646.
38. Leydesdorff, L., &, Opthof, T., (2011). Remaining problems with the "New Crown Indicator" (MNCS) of the CWTS. Journal of Informetrics, 5 (1), 224-225.
39. Leydesdorff, L., &, Rafols, I., (2009). A Global Map of Science Based on the ISI subject categories. Journal of the American Society for Information Science and Technology, 60 (2), 348-362.
40. Lundberg, J., (2007). Lifting the crown - citation z-score. Journal of Informetrics, 1 (2), 145-154.
41. Martin, B., &, Irvine, J., (1983). Assessing basic research: Some partial indicators of scientific progress in radio astronomy. Research Policy, 12, 61-90.
42. Moed, H.F., (2010). Measuring contextual citation impact of scientific journals. Journal of Informetrics, 4 (3), 265-277.
43. National Science Board (2010). Science and engineering indicators 2010, appendix tables. Arlington, VA, USA: National Science Foundation (National Science Board 10-01).
44. Opthof, T., &, Leydesdorff, L., (2010). Caveats for the journal and field normalizations in the CWTS ("Leiden") evaluations of research performance. Journal of Informetrics, 4 (3), 423-430.
45. Opthof, T., &, Leydesdorff, L., (in press). Citation analysis cannot legitimate the strategic selection of excellence Scientometrics.
46. Pudovkin, A.I., &, Garfield, E., (2002). Algorithmic procedure for finding semantically related journals. Journal of the American Society for Information Science and Technology, 53 (13), 1113-1119. (Pubitemid 35359180)
47. Pudovkin, A.I., &, Garfield, E., (2009). Percentile rank and author superiority indexes for evaluating individual journal articles and the author's overall citation performance. CollNet Journal of Scientometrics and Information Management, 3 (2), 3-10.
48. Radicchi, F., Fortunato, S., &, Castellano, C., (2008). Universality of citation distributions: Toward an objective measure of scientific impact. Proceedings of the National Academy of Sciences, 105 (45), 17268-17272.
49. Rafols, I., &, Leydesdorff, L., (2009). Content-based and algorithmic classifications of journals: Perspectives on the dynamics of scientific communication and indexer effects. Journal of the American Society for Information Science and Technology, 60 (9), 1823-1835.
50. Rons, N., &, Amez, L., (2008). Impact vitality - a measure for excellent scientists. In, J. Gorraiz, &, E. Schiebel, (Eds.), Excellence and emergence. A new challenge for the combination of quantitative and qualitative approaches. 10th International Conference on Science and Technology Indicators (pp. 211-213). Vienna: Austrian Research Centers (ARC).
51. Ross, S.M., (2007). Introduction to probability models. London: Elsevier.
52. Rousseau, R., &, Leydesdorff, L., (2011). Non-consistency, non-cited items, and the impact factor: A consequence of the arithmetic. ISSI Newsletter, 7 (1), 10-14.
53. Schubert, A., &, Braun, T., (1993). Reference standards for citation based assessments. Scientometrics, 26 (1), 21-35.
54. Schubert, A., &, Braun, T., (1996). Cross-field normalization of scientometric indicators. Scientometrics, 36 (3), 311-324. (Pubitemid 126449886)
55. Schubert, A., &, Glänzel, W., (1983). Statistical reliability of comparisons based on the citation impact of scientific publications. Scientometrics, 5 (1), 59-73.
56. Seglen, P.O., (1992). The skewness of science. Journal of the American Society for Information Science, 43 (9), 628-638.
57. Sheskin, D., (2007). Handbook of parametric and nonparametric statistical procedures (4th ed.). Boca Raton, FL, USA: Chapman & Hall/CRC.
58. Spaan, J.A.E., (2010). The danger of pseudoscience in informetrics. Journal of Informetrics 4 (3), 439-440.
59. Van Raan, A.F.J., (2005). Measurement of central aspects of scientific research: performance, interdisciplinarity, structure. Measurement, 3 (1), 1-19.
60. Van Raan, A.F.J., Van Eck, N.J., Van Leeuwen, T.N., Visser, M.S., &, Waltman, L., (2010b, September). The new set of bibliometric indicators of CWTS. Paper presented at the 11th International Conference on Science and Technology Indicators, Leiden, the Netherlands.
61. Van Raan, A.F.J., Van Leeuwen, T.N., Visser, M.S., Van Eck, N.J., &, Waltman, L., (2010). Rivals for the crown: Reply to Opthof and Leydesdorff. Journal of Informetrics, 4 (3), 431-435.
62. Van Raan, T., (2010). Bibliometrics: Measure for measure. Nature, 468 (7325), 763-763.
63. Waltman, L., Van Eck, N.J., Van Leeuwen, T.N., Visser, M.S., &, Van Raan, A.F.J., (2011). Towards a new crown indicator: Some theoretical considerations. Journal of Informetrics, 5 (1), 37-47.
64. Waltman, L., Van Eck, N.J., Van Leeuwen, T.N., Visser, M.S., &, Van Raan, A.F.J., (in press). Towards a new crown indicator: An empirical analysis. Scientometrics. Arxiv preprint arXiv:1004.1632.
65. Zitt, M., (2010). Citing-side normalization of journal impact: A robust variant of the audience factor. Journal of Informetrics, 4 (3), 392-406.
66. Zitt, M., &, Small, H., (2008). Modifying the journal impact factor by fractional citation weighting: The audience factor. Journal of the American Society for Information Science and Technology, 59 (11), 1856-1860.