How can journal impact factors be normalized across fields of science? An assessment in terms of percentile ranks and fractional counts

Journal of the American Society for Information Science and Technology

Volumn 64, Issue 1, 2013, Pages 96-107

  Leydesdorff, Loet ^a   Zhou, Ping ^b   Bornmann, Lutz ^c  

^a UNIVERSITY OF AMSTERDAM   (Netherlands)

^b INSTITUTE OF SCIENTIFIC AND TECHNICAL INFORMATION OF CHINA   (China)

^c MAX PLANCK INSTITUTE   (Germany)

Author keywords

citation indexes; information science; journals

Indexed keywords

CITATION DISTRIBUTION; CITATION INDEXES; IMPACT FACTOR; JOURNALS; NATIONAL SCIENCE FOUNDATIONS; SCIENCE CITATION INDEX;

ARTIFICIAL INTELLIGENCE; INFORMATION SCIENCE;

SOFTWARE ENGINEERING;

EID: 84871180922     PISSN: 15322882     EISSN: 15322890     Source Type: Journal    
DOI: 10.1002/asi.22765     Document Type: Article

References (61)

1. Althouse, B.M., West, J.D., Bergstrom, C.T., &, Bergstrom, T., (2009). Differences in impact factor across fields and over time. Journal of the American Society for Information Science and Technology, 60 (1), 27-34.
2. Beirlant, J., Glänzel, W., Carbonez, A., &, Leemans, H., (2007). Scoring research output using statistical quantile plotting. Journal of Informetrics, 1 (3), 185-192.
3. Bensman, S.J., (1996). The structure of the library market for scientific journals: The case of chemistry. Library Resources & Technical Services, 40, 145-170.
4. Bensman, S.J., (2007). Garfield and the impact factor. Annual Review of Information Science and Technology, 41 (1), 93-155.
5. Bornmann, L., (2011). Scientific peer review. Annual Review of Information Science and Technology, 45, 199-245.
6. Bornmann, L., &, Daniel, H.-D., (2008). What do citation counts measure? A review of studies on citing behavior. Journal of Documentation, 64 (1), 45-80.
7. Bornmann, L., &, Mutz, R., (2011). Further steps towards an ideal method of measuring citation performance: The avoidance of citation (ratio) averages in field-normalization. Journal of Informetrics, 5 (1), 228-230.
8. Bornmann, L., Mutz, R., Neuhaus, C., &, Daniel, H.D., (2008). Citation counts for research evaluation: Standards of good practice for analyzing bibliometric data and presenting and interpreting results. Ethics in Science and Environmental Politics (ESEP), 8 (1), 93-102. (Pubitemid 351836496)
9. Boyack, K.W., &, Klavans, R., (2011). Multiple dimensions of journal specificity: Why journals can't be assigned to disciplines. In, E. Noyons, P. Ngulube, &, J. Leta, (Eds.), The 13th Conference of the International Society for Scientometrics and Informetrics (Vol. I, pp. 123-133). Durban, South Africa: ISSI, Leiden University and the University of Zululand.
10. Cameron, A.C., &, Trivedi, P.K., (1998). Regression analysis of count data. Cambridge, UK: Cambridge University Press.
11. Carpenter, M.P., &, Narin, F., (1973). Clustering of Scientific Journals. Journal of the American Society for Information Science, 24, 425-436.
12. Cohen, J., (1977). Statistical power analysis for the behavioral sciences. Orlando, FL: Acdemic Press.
13. de Andrés, A., (2011). Evaluating research using impact and Hirsch factors. Europhysics News, 42 (2), 29-31.
14. Garfield, E., (1972). Citation analysis as a tool in journal evaluation. Science 178 (4060), 471-479.
15. Garfield, E., (1979). Is citation analysis a legitimate evaluation tool? Scientometrics, 1 (4), 359-375.
16. Gingras, Y., &, Larivière, V., (2011). There are neither "king" nor "crown" in scientometrics: Comments on a supposed "alternative" method of normalization. Journal of Informetrics, 5 (1), 226-227.
17. Glänzel, W., (2010). On reliability and robustness of scientometrics indicators based on stochastic models. An evidence-based opinion paper. Journal of Informetrics, 4 (3), 313-319.
18. Glänzel, W., &, Schubert, A., (2003). A new classification scheme of science fields and subfields designed for scientometric evaluation purposes. Scientometrics, 56 (3), 357-367. (Pubitemid 41444803)
19. Glänzel, W., Schubert, A., Thijs, B., &, Debackere, K., (2011). A priori vs. a posteriori normalisation of citation indicators. The case of journal ranking. Scientometrics, 87 (2), 415-424.
20. Leydesdorff, L., (2008). Caveats for the use of citation indicators in research and journal evaluation. Journal of the American Society for Information Science and Technology, 59 (2), 278-287. (Pubitemid 351205033)
21. Leydesdorff, L., (2009). How are new citation-based journal indicators adding to the bibliometric toolbox? Journal of the American Society for Information Science and Technology, 60 (7), 1327-1336.
22. Leydesdorff, L., (2012). Alternatives to the journal impact factor: I3 and the top-10% (or top-25%?) of the most-highly cited papers. Scientometrics, 92 (2), 355-365.
23. Leydesdorff, L., (in press-a). An evaluation of impacts in "nanoscience & nanotechnology": Steps towards standards for citation analysis. Scientometrics.
24. Leydesdorff, L., (in press-b). Accounting for the uncertainty in the evaluation of percentile ranks. Journal of the American Society for Information Science and Technology. Retrieved from: http://arxiv.org/ftp/arxiv/papers/1204/ 1204.1894.pdf
25. Leydesdorff, L., &, Bornmann, L., (2011a). How fractional counting affects the impact factor: Normalization in terms of differences in citation potentials among fields of science. Journal of the American Society for Information Science and Technology, 62 (2), 217-229.
26. Leydesdorff, L., &, Bornmann, L., (2011b). Integrated impact indicators (I3) compared with impact factors (IFs): An alternative design with policy implications. Journal of the American Society for Information Science and Technology, 62 (11), 2133-2146.
27. Leydesdorff, L., &, Bornmann, L., (2012). Percentile Ranks and the Integrated Impact Indicator (I3). Journal of the American Society for Information Science and Technology, 63 (9), 1901-1902.
28. Leydesdorff, L., Bornmann, L., Mutz, R., &, Opthof, T., (2011). Turning the tables in citation analysis one more time: Principles for comparing sets of documents. Journal of the American Society for Information Science and Technology, 62 (7), 1370-1381.
29. Leydesdorff, L., &, Opthof, T., (2010a). Normalization at the field level: Fractional counting of citations. Journal of Informetrics, 4 (4), 644-646.
30. Leydesdorff, L., &, Opthof, T., (2010b). Scopus' Source Normalized Impact per Paper (SNIP) versus the Journal Impact Factor based on fractional counting of citations. Journal of the American Society for Information Science and Technology, 61 (11), 2365-2396.
31. Leydesdorff, L., &, Shin, J.C., (2011). How to evaluate universities in terms of their relative citation impacts: Fractional counting of citations and the normalization of differences among disciplines. Journal of the American Society for Information Science and Technology, 62 (6), 1146-1155.
32. Lundberg, J., Fransson, A., Brommels, M., Skår, J., &, Lundkvist, I., (2006). Is it better or just the same? Article identification strategies impact bibliometric assessments. Scientometrics, 66 (1), 183-197. (Pubitemid 43099462)
33. Martyn, J., &, Gilchrist, A., (1968). An evaluation of British scientific journals. London: Aslib.
34. Moed, H.F., (2010). Measuring contextual citation impact of scientific journals. Journal of Informetrics, 4 (3), 265-277.
35. Moed, H.F., &, Van Leeuwen, T.N., (1996). Impact factors can mislead. Nature, 381 (6579), 186.
36. National Science Board. (2012). Science and engineering indicators. Washington DC: National Science Foundation. Retrieved from: http://www.nsf.gov/ statistics/seind12/
37. Price, D.J. de Solla,. (1965). Networks of scientific papers. Science, 149 (3683), 510-515.
38. Price, D.J. de Solla,. (1970). Citation measures of hard science, soft science, technology, and nonscience. In, C.E. Nelson, &, D.K. Pollock, (Eds.), Communication among scientists and engineers (pp. 3-22). Lexington, MA: Heath.
39. Pudovkin, A.I., &, Garfield, E., (2002). Algorithmic procedure for finding semantically related journals. Journal of the American Society for Information Science and Technology, 53 (13), 1113-1119. (Pubitemid 35359180)
40. Pudovkin, A. I., &, Garfield, E., (2009). Percentile Rank and Author Superiority Indexes for Evaluating Individual Journal Articles and the Author's Overall Citation Performance. CollNet Journal of Scientometrics and Information Management, 3 (2), 3-10.
41. Rabe-Hesketh, S., &, Skrondal, A., (2008). Multilevel and longitudinal modeling using Stata. College Station, TX: Stata Press.
42. Radicchi, F., &, Castellano, C., (2012). Testing the fairness of citation indicators for comparison across scientific domains: The case of fractional citation counts. Journal of Informetrics, 6 (1), 121-130.
43. Radicchi, F., Fortunato, S., &, Castellano, C., (2008). Universality of citation distributions: Toward an objective measure of scientific impact. Proceedings of the National Academy of Sciences, 105 (45), 17268-17272.
44. Rafols, I., &, Leydesdorff, L., (2009). Content-based and algorithmic classifications of journals: Perspectives on the dynamics of scientific communication and indexer effects. Journal of the American Society for Information Science and Technology, 60 (9), 1823-1835.
45. Rafols, I., Leydesdorff, L., O'Hare, A., Nightingale, P., &, Stirling, A., (2012). How journal rankings can suppress interdisciplinary research: A comparison between innovation studies and business & management. Research Policy, 41 (7), 1262-1282.
46. Rafols, I., Porter, A., &, Leydesdorff, L., (2010). Science overlay maps: A new tool for research policy and library management. Journal of the American Society for Information Science and Technology, 61 (9), 1871-1887.
47. Rousseau, R., (2006). Timelines in citation research. Journal of the American Society for Information Science and Technology, 57 (10), 1404-1405. (Pubitemid 44187278)
48. Rousseau, R., (2012). Basic properties of both percentile rank scores and the I3 indicator. Journal of the American Society for Information Science and Technology, 63 (2), 416-420.
49. Schreiber, M., (in press-a). Inconsistencies of recently proposed citation impact indicators and how to avoid them. Journal of the American Society for Information Science and Technology, preprint available at arXiv:1202.3861.
50. Schreiber, M., (in press-b). Uncertainties and ambiguities in percentiles and how to avoid them. Journal of the American Society for Information Science and Technology.
51. Seglen, P.O., (1992). The skewness of science. Journal of the American Society for Information Science, 43 (9), 628-638.
52. Seglen, P.O., (1997). Why the impact factor of journals should not be used for evaluating research. British Medical Journal, 314, 498-502. (Pubitemid 27081238)
53. Sher, I.H., &, Garfield, E., (1965). New tools for improving and evaluating the effectiveness of research. Paper presented at the second conference on Research Program Effectiveness, July 27-29, Washington, DC.
54. Sirtes, D., (2012). Finding the Easter eggs hidden by oneself: Why Radicchi and Castellano's (2012) fairness test for citation indicators is not fair. Journal of Informetrics 6 (3), 448-450.
55. Small, H., &, Sweeney, E., (1985). Clustering the Science Citation Index using co-citations I. A comparison of methods. Scientometrics 7 (3-6), 391-409.
56. Vanclay, J. K., (2012). Impact Factor: Outdated artefact or stepping-stone to journal certification? Scientometrics, 92 (2), 211-238.
57. Waltman, L., &, Van Eck, N.J., (2010). A general source normalized approach to bibliometric research performance assessment. Paper presented at the 11th Conference on Science & Technology Indicators (STI), Leiden, September 8-11.
58. Waltman, L., van Eck, N.J., &, van Raan, A.F.J., (2012). Universality of citation distributions revisited. Journal of the American Society for Information Science and Technology, 63 (1), 72-77.
59. White, H., (1980). A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica, 48, 817-830.
60. Zhou, P., &, Leydesdorff, L., (2011). Fractional counting of citations in research evaluation: A cross- and interdisciplinary assessment of the Tsinghua University in Beijing. Journal of Informetrics, 5 (3), 360-368.
61. Zitt, M., &, Small, H., (2008). Modifying the journal impact factor by fractional citation weighting: The audience factor. Journal of the American Society for Information Science and Technology, 59 (11), 1856-1860.