How to evaluate universities in terms of their relative citation impacts: Fractional counting of citations and the normalization of differences among disciplines

Journal of the American Society for Information Science and Technology

Volumn 62, Issue 6, 2011, Pages 1146-1155

  Leydesdorff, Loet ^a   Shin, Jung C ^b  

^a UNIVERSITY OF AMSTERDAM   (Netherlands)

^b Seoul National University   (South Korea)

Author keywords

[No Author keywords available]

Indexed keywords

CITATION DATA; CITATION IMPACT; MULTI-DISCIPLINARY RESEARCH; RESEARCH UNIVERSITIES; SCIENTOMETRICS;

PUBLISHING; SOCIETIES AND INSTITUTIONS;

BEHAVIORAL RESEARCH;

EID: 79956141391     PISSN: 15322882     EISSN: 15322890     Source Type: Journal    
DOI: 10.1002/asi.21511     Document Type: Article

References (64)

1. Altbach, P.G., (2009). Peripheries and centers: research universities in developing countries. Asia Pacific Education Review, 10 (1), 15-27.
2. Anderson, J., Collins, P.M.D., Irvine, J., Isard, P.A., Martin, B.R., Narin, F., &, Stevens, K., (1988). On-line approaches to measuring national scientific output: A cautionary tale. Science and Public Policy 15, 153-161.
3. Bensman, S.J., &, Leydesdorff, L., (2009). Definition and identification of journals as bibliographic and subject entities: Librarianship vs. ISI Journal Citation Reports (JCR) methods and their effect on citation measures. Journal of the American Society for Information Science and Technology, 60 (6), 1097-1117.
4. Billaut, J.-C., Bouyssou, D., &, Vincke, P., (2010). Should you believe in the Shanghai ranking ? Scientometrics, 84 (1), 237-263.
5. Bornmann, L., Mutz, R., Neuhaus, C., &, Daniel, H.D., (2008). Citation counts for research evaluation: standards of good practice for analyzing bibliometric data and presenting and interpreting results. Ethics in Science and Environmental Politics(ESEP), 8 (1), 93-102. (Pubitemid 351836496)
6. Boyack, K.W., Klavans, R., &, Börner, K., (2005). Mapping the Backbone of Science. Scientometrics, 64 (3), 351-374. (Pubitemid 41152144)
7. Braun, T., Glänzel, W., &, Schubert, A., (1991). The bibliometric assessment of UK scientific performance-some comments on Martin's reply. Scientometrics, 20, 359-362.
8. Chan, L.M., (1999). A guide to the library of congress classification (5th ed.). Englewood, CO: Libraries Unlimited.
9. Foray, D., (2004). The economics of knowledge. Cambridge, MA/London: MIT Press.
10. Foray, D., &, Lundvall, B.-A., (1996). The knowledge-based economy: From the economics of knowledge to the learning economy. In employment and growth in the knowledge-based economy (pp. 11-32). Paris: OECD.
11. Galam, S., (2010). Integrating multiple coauthorship in the quantitative evaluation of individual's scientific records. Retrieved from
12. Garfield, E., (1972). Citation analysis as a tool in journal evaluation. Science 178 (4060), 471-479.
13. Garfield, E., (1979). Is citation analysis a legitimate evaluation tool ? Scientometrics, 1 (4), 359-375.
14. Gingras, Y., &, Larivière, V., (2011). There are neither "king" nor "crown" in scientometrics: Comments on a supposed "alternative" method of normalization. Journal of Informetrics, 5 (1), 226-227.
15. Guns, R., &, Rousseau, R., (2010). New journal impact indicators take references into account: A comparison. ISSI Newsletter, 6 (1), 9-14.
16. Halffman, W., &, Leydesdorff, L., (2010). Is inequality among universities increasing? Gini coefficients and the elusive rise of elite universities. Minerva, 48 (1), 55-72.
17. Laudel, G., &, Origgi, G., (2006). Introduction to a special issue on the assessment of interdisciplinary research. Research Evaluation, 15 (1), 2-4. (Pubitemid 44544937)
18. Levine, G., (1991). A guide to SPSS for analysis of variance. Hillsdale, NJ: Lawrence Erlbaum.
19. Leydesdorff, L., (1988). Problems with the 'measurement' of national scientific performance. Science and Public Policy, 15, 149-152.
20. Leydesdorff, L., (2006). Can scientific journals be classified in terms of aggregated journal-journal citation relations using the Journal Citation Reports ? Journal of the American Society for Information Science & Technology, 57 (5), 601-613.
21. Leydesdorff, L., (2008). Caveats for the use of citation indicators in research and journal evaluation. Journal of the American Society for Information Science and Technology, 59 (2), 278-287. (Pubitemid 351205033)
22. Leydesdorff, L., (2010). The knowledge-based economy and the triple helix model. Annual Review of Information Science and Technology, 44, 367-417.
23. Leydesdorff, L., &, Bornmann, L., (2011). How fractional counting affects the impact factor: Normalization in terms of differences in citation potentials among fields of science. Journal of the American Society for Information Science and Technology, 62 (2), 217-229.
24. Leydesdorff, L., &, Opthof, T., (2010a). Normalization at the field level: fractional counting of citations. Journal of Informetrics, 4 (4), 644-646.
25. Leydesdorff, L., &, Opthof, T., (2010b). Scopus' source normalized impact per paper (SNIP) versus the Journal Impact Factor based on fractional counting of citations. Journal of the American Society for Information Science and Technology, 61 (11), 2365-2396.
26. Leydesdorff, L., &, Opthof, T., (2011). Scopus' SNIP indicator: Reply to Moed. Journal of the American Society for Information Science and Technology, 62 (1), 214-215.
27. Liu, N.C., Cheng, Y., &, Liu, L., (2004). Academic ranking of world universities using scientometrics-A comment to the "fatal attraction". Scientometrics, 64 (1), 101-109.
28. Lundberg, J., (2007). Lifting the crown-Citation z-score. Journal of informetrics, 1 (2), 145-154.
29. Marginson, S., (2009). Global university rankings: Some potentials. In, Barbara M. Kehm, &, Bjorn Stensaker, (Eds.), University rankings, diversity, and the new landscape of higher education. Sense Publishers, 85-96.
30. McAllister, P.R., Narin, F., &, Corrigan, J.G., (1983). Programmatic evaluation and comparison based on standardized citation scores. IEEE Transactions on Engineering Management, 30 (4), 205-211. (Pubitemid 14488808)
31. Moed, H.F., (2010a). Measuring contextual citation impact of scientific journals. Journal of Informetrics, 4 (3), 265-277.
32. Moed, H.F., (2010b). CWTS crown indicator measures citation impact of a research group's publication oeuvre. Journal of Informetrics, 3 (3), 436-438.
33. Moed, H.F., (2011). The Source-Normalized Impact per Paper (SNIP) is a valid and sophisticated indicator of journal citation impact. Journal of the American Society for Information Science and Technology, 62 (1) 211-213.
34. Narin, F., (1976). Evaluative bibliometrics: The use of publication and citation analysis in the evaluation of scientific activity. Washington, DC: National Science Foundation.
35. Narin, F., Carpenter, M., &, Berlt, N.C., (1972). Interrelationships of scientific journals. Journal of the American Society for Information Science, 23, 323-331.
36. National Science Board. (2010). Science and engineering indicators. Washington DC: National Science Foundation. Retrieved from
37. Neufeld, J., &, Von Ins, M., (in press). Informed peer review and uninformed bibliometrics? Research Evaluation.
38. Opthof, T., &, Leydesdorff, L., (2010). Caveats for the journal and field normalizations in the CWTS ("Leiden") evaluations of research performance. Journal of Informetrics, 4 (3), 423-430.
39. Park, H.W., Hong, H.D., &, Leydesdorff, L., (2005). A comparison of the knowledge-based innovation systems in the economies of South Korea and the Netherlands using triple helix indicators. Scientometrics, 65 (1), 3-27. (Pubitemid 41244717)
40. Park, H.W., &, Leydesdorff, L., (2010). Longitudinal trends in networks of university-industry-government relations in South Korea: The role of programmatic incentives. Research Policy, 39 (5), 640-649.
41. Persson, O., Glänzel, W., &, Danell, R., (2004). Inflationary bibliometric values: The role of scientific collaboration and the need for relative indicators in evaluative studies. Scientometrics, 60 (3), 421-432. (Pubitemid 38926350)
42. Price, D., J. de Solla, &, deBeaver, D., (1966). Collaboration in an invisible college. American Psychologist, 21 (11), 1011-1018.
43. Pudovkin, A.I., &, Garfield, E., (2002). Algorithmic procedure for finding semantically related journals. Journal of the American Society for Information Science and Technology, 53 (13), 1113-1119. (Pubitemid 35359180)
44. Rafols, I., &, Leydesdorff, L., (2009). Content-based and algorithmic classifications of journals: Perspectives on the dynamics of scientific communication and indexer effects. Journal of the American Society for Information Science and Technology, 60 (9), 1823-1835.
45. Rafols, I., Leydesdorff, L., O'Hare, A., Nightingale, P., &, Stirling, A., (in press). How rankings suppress interdisciplinarity. The case of innovation studies vs. business and management.
46. Rafols, I., Porter, A., &, Leydesdorff, L., (2010). Science overlay maps: A new tool for research policy and library management. Journal of the American Society for Information Science and Technology, 61 (9), 1871-1887.
47. Ramanana-Rahary, S., Zitt, M., &, Rousseau, R., (2009). Aggregation properties of relative impact and other classical indicators: Convexity issues and the Yule-Simpson paradox. Scientometrics, 79 (2), 311-327.
48. Shin, J.C., (2009a). Classifying higher education institutions in Korea: A performance-based approach. Higher Education, 57 (2), 247-266.
49. Shin, J.C., (2009b). Building world-class research university: The Brain Korea 21 project. Higher Education, 58, 669-688.
50. Shin, J., &, Cummings, W., (2010). Multilevel analysis of academic publishing across disciplines: Research preference, collaboration, and time on research. Scientometrics. doi:10.1007/s11192-010-0236-2.
51. Simon, H.A., (1973). The Organization of Complex Systems. In, H.H. Pattee, (Ed.), Hierarchy Theory: The Challenge of Complex Systems (pp. 1-27). New York: George Braziller Inc.
52. Small, H., &, Sweeney, E., (1985). Clustering the Science Citation Index using co-citations I. A comparison of methods, Scientometrics, 7, 391-409.
53. Spaan, J.A.E., (2010). The danger of pseudo science in informetrics. Journal of Informetrics, 4 (3), 439-440.
54. Van Noorden, R., (2010). A profusion of measures. Nature, 465 (7300, 17 June 2010), 864-866.
55. Van Raan, A.F.J., (2005). Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods. Scientometrics, 62 (1), 133-143. (Pubitemid 40397597)
56. Van Raan, A.F.J., Van Leeuwen, T.N., Visser, M.S., Van Eck, N.J., &, Waltman, L., (2010). Rivals for the crown: Reply to Opthof and Leydesdorff. Journal of Informetrics, 4 (3), 431-435.
57. Wagner, C., Roessner, J.D., Bobb, K., Klein, J., Boyack, K., Keyton, J., &, Börner, K., (2011). Approaches to understanding and measuring Interdisciplinary Scientific Research (IDR): A review of the literature. Journal of Informetrics, 5 (1), 14-26.
58. Waltman, L., &, Van Eck, N.J., (2010, September). A general source normalized approach to bibliometric research performance assessment. Paper presented at the 11th International Conference on Science and Technology Indicators, Leiden, the Netherlands.
59. Waltman, L., Van Eck, N.J., Van Leeuwen, T.N., Visser, M.S., &, Van Raan, A.F.J., (2010). Towards a new crown indicator: An empirical analysis. Preprint at arXiv:1004.1632.
60. Waltman, L., Van Eck, N.J., Van Leeuwen, T.N., Visser, M.S., &, Van Raan, A.F.J., (2011). Towards a new crown indicator: Some theoretical considerations. Journal of Informetrics, 5 (1), 37-47.
61. Wouters, P., (1999). The citation culture. (Unpublished doctoral dissertation). University of Amsterdam.
62. Zhou, P., &, Leydesdorff, L., (in press). Fractional counting of citations in research evaluation: A cross-and interdisciplinary assessment of Tsinghua University in Beijing. Journal of Informetrics. doi:10.1016/j.joi.2011. 01.010
63. Zitt, M., (2010). Citing-side normalization of journal impact: A robust variant of the Audience Factor. Journal of Informetrics, 4 (3), 392-406.
64. Zitt, M., &, Small, H., (2008). Modifying the journal impact factor by fractional citation weighting: The audience factor. Journal of the American Society for Information Science and Technology, 59 (11), 1856-1860.