Grim tales about the impact factor and the h-index in the Web of Science and the Journal Citation Reports databases: Reflections on Vanclay's criticism

Scientometrics

Volumn 92, Issue 2, 2012, Pages 325-354

  Jacso, Peter ^a  

^a University of Hawai'i at Manoa   (United States)

Author keywords

Google Scholar; h index; Journal Citation Reports; Journal impact factors; Research assessment; Web of Science

Indexed keywords

EID: 84864289665     PISSN: 01389130     EISSN: None     Source Type: Journal    
DOI: 10.1007/s11192-012-0769-7     Document Type: Article

References (92)

1. Abraham, P. (2000). Duplicate and salami publications. Journal of Postgraduate Medicine, 46(2), 67-69.
2. Alonso, S., Cabrerizo, F., Herrera-Viedma, E., & Herrera, F. (2009). H-index: a review focused in its variants, computation and standardization for different scientific fields. Journal of Informetrics, 3(4), 273-289.
3. Balaban, A. T. (2012). Positive and negative aspects of citation indices and journal impact factors. Scientometrics. doi: 10. 1007/s11192-012-0637-5.
4. Bar-Ilan, J. (2008). Which h-index?-A comparison of WoS, Scopus and Google Scholar. Scientometrics, 74(2), 257-271. Accessed April 3, 2012, from http://150. 214. 190. 154/hindex/pdf/Bar-Ilan2008. pdf.
5. Bar-Ilan, J. (2010). Ranking of information and library science journals by JIF and by h-type indices. Journal of Informetrics, 4(2), 141-147.
6. Bensman, S. J. (1998). Scientific and technical serials holdings optimization in an inefficient market: A LSU serials redesign project exercise. Library Resources & Technical Services, 42(3), 147-242.
7. Bensman, S. J. (2007). Garfield and the impact factor: The creation, utilization, and validation of a citation measure. Annual Review of Information Science and Technology, 41(1), 93-155.
8. Bensman, S. J. (2011). The impact factor: its place in Garfield's thought in science evaluation and in library collection. Scientometrics. doi: 10. 1007/s11192-011-0601-9.
9. Bollen, J., Rodriquez, M. A., & van De Sompel, H. (2006). Journal status. Scientometrics, 69(3), 669-687.
10. Bornmann, L., Mutz, R., Neuhaus, C., & Daniel, H. D. (2008). Citation counts for research evaluation: Standards of good practice for analyzing bibliometric data and presenting and interpreting results. Ethics in Science and Environmental Politics, 8(1), 93-102.
11. Bornmann, L., Rüdiger, M., & Daniel, H.-D. (2009). Do we need the h index and its variants in addition to standard bibliometric measures? Journal of the American Society for Information Science and Technology, 60(6), 1286-1289.
12. Braun, T. (2007). The impact factor of scientific and scholarly journals: Its use and misuse in research evaluation. Akadémiai Kiadó, Budapest. ISBN 963-05-8438-7.
13. Braun, T., & Glänzel, W. (1995). On a source of errors in computing journal impact factors. Chemical Intelligencer, 1, 31-32.
14. Braun, T., Glänzel, W., & Schubert, A. (2005). A Hirsch-type index for journals. The Scientist, 19(11), 8.
15. Brumback, R. A. (2009a). Impact factor wars: Episode V.: The empire strikes back. Journal of Child Neurology, 24(3), 260-262.
16. Brumback, R. A. (2009b). Impact factor: Let's be unreasonable! Epidemiology, 20(6), 932-933.
17. Buchanan, R. A. (2006). Accuracy of cited references: The role of citation databases. College and Research Libraries, 67, 292-303.
18. Butler, L. (2011). The devil is in the detail: Concerns about Vanclay's analysis of Australian journal rankings. Journal of Informetrics, 5(4), 693-694.
19. Campanario, J. M. (2011). Empirical study of journal impact factors obtained using the classical two-year citation window versus a five-year citation window. Scientometrics, 87(1), 189-204.
20. Dong, P., Loh, M., & Mondry, A. (2005). The "impact factor" revisited. Biomedical Digital Libraries, 2(7), 1-8.
21. Falagas, M. E., & Alexiou, V. G. (2008). The top-ten in journal impact factor manipulation. Archivum Immunologiae et Therapiae Experimentalis, 56(4), 223-226.
22. Garfield, E. (1994). The Thomson Reuters impact factor. Accessed April 3, 2012, from http://thomsonreuters. com/products_services/science/free/essays/impact_factor/.
23. Garfield, E. (1996). How can impact factors be improved? British Medical Journal, 313, 411-413.
24. Garfield, E. (1999). Journal impact factor: A brief review. Canadian Medical Association Journal, 161(8), 979-980.
25. Garfield, E. (2005). The agony and the ecstasy-The history and meaning of the journal impact factor. Chicago: International Congress on Peer Review and Biomedical Publication.
26. Garfield, E. (2006). The history and meaning of the journal impact factor. Journal of the American Medical Association, 295(1), 90-93.
27. Glänzel, W. (2009). The multi-dimensionality of journal impact. Scientometrics, 78(2), 355-374.
28. Gorman, G. E. (2008). They can't read, but they sure can count-Flawed rules of the journal rankings game. Online Information Review, 32(6), 705-708.
29. Harnad, S. (2008). Validating research performance metrics against peer rankings. Ethics in Science and Environmental Politics, 8(1), 103-107.
30. Hirsch, J. E. (2005). An index to quantify an individual's scientific research output. Proceedings of the National Academies of Science, 102(46), 16569-16572.
31. Hirst, G. (1978). Discipline impact factors: A method for determining core journal lists. Journal of the American Society for Information Science, 29(4), 171-172.
32. Jacso, P. (2000). The number game. Online Information Review, 24(2), 180-183.
33. Jacso, P. (2001). A deficiency in the algorithm for calculating the impact factor of scholarly journals: The journal impact factor. Cortex, 37(4), 590-594.
34. Jacso, P. (2004). WorldCat, Sociological Abstracts on CSA. e-Psyche, 28(2), 54-58.
35. Jacso, P. (2005a). Google Scholar (Redux). Gale Reference Reviews (online), June 2005. Accessed March 31, 2012, from http://www. gale. cengage. com/reference/peter/index. htm.
36. Jacso, P. (2005b). Google Scholar: The pros and the cons. Online Information Review, 29(2), 208-214.
37. Jacso, P. (2006). Deflated, inflated and phantom citation counts. Online Information Review, 30(3), 297-309.
38. Jacso, P. (2007a). How big is a database versus how is a database big. Online Information Review, 31(4), 533-536.
39. Jacso, P. (2007b). Software issues related to cited references. Online Information Review, 31(6), 892-905.
40. Jacso, P. (2007c). The dimensions of cited reference enhanced database subsets. Online Information Review, 31(5), 695-705.
41. Jacso, P. (2007d). Trends in professional and academic online information services. (Keynote Address) at INFORUM 13th conference on professional information resources prague May 22-24, 2007. Accessed April 3, 2012, from http://www. inforum. cz/pdf/2007/jacso-peter. pdf.
42. Jacso, P. (2008a). Google Scholar revisited. Online Information Review, 32(1), 102-114. Accessed March 31, 2012, from http://www. jacso. info/OIR/2008-01-jacso-Google-Scholar-revisited. doc (pre-print version).
43. Jacso, P. (2008b). Testing the calculation of a realistic h-index in Google Scholar, Scopus and Web of Science for F. W. Lancaster. Library Trends, 56(4), 784-815.
44. Jacso, P. (2008c). The plausibility of computing the h-index of scholarly productivity and impact using reference-enhanced databases. Online Information Review, 32(2), 266-283. Accessed April 3, from, http://www. jacso. info/PDFs/jacso-plausibility-of-computing-the-h-index-jav. pdf.
45. Jacso, P. (2008d). The pros and cons of computing the h-index using Google Scholar. Online Information Review, 32(3), 437-452. Accessed March 31, 2012, from http://www. jacso. info/PDFs/jacso-GS-for-h-index-unabridged. pdf (unabridged manuscript).
46. Jacso, P. (2008e). The pros and cons of computing the h-index using Web of Science. Online Information Review, 32(5), 673-688. Accessed March 31, 2012, from http://www. jacso. info/OIR/2008-05-jacso-pros-and-cons-of-computing-the-h-index-using-Web-of-Science. doc (pre-print version).
47. Jacso, P. (2009a). Calculating the h-index and other bibliometric and scientometric indicators from Google Scholar with the Publish or Perish software. Online Information Review, 33(6), 1189-1200. Accessed March 31, 2012, from http://www. jacso. info/OIR/2009-06-jacso-calculating-H-index-and-other-bibliometric-and-scientometric-indicators. doc (pre-print version).
48. Jacso, P. (2009b). Database source coverage: hypes, vital signs and reality checks. Online Information Review, 33(5), 997-1007. Accessed March 31, 2012, fromhttp://www. jacso. info/OIR/2009-05-jacso-database-source-coverage-hypes-and-reality-checks. doc (pre-print version).
49. Jacso, P. (2009c). Errors of omission and their implication for computing scientometric measures in evaluating the publishing productivity and impact of countries. Online Information Review, 33(2), 376-385. Accessed March 31, 2012, from http://www. jacso. info/OIR/2009-02-jacso-errors-of-omission-and-their-implications-for-computing-scientometrics-measures. doc (pre-print version).
50. Jacso, P. (2009d). Five-year impact factor data in the Journal Citation Reports. Online Information Review, 33(3), 603-614. Accessed February 5, 2012, from http://www. jacso. info/OIR/2009-03-five-year-impact-factor-data-in-the-Journal-Citation-Reports. doc (pre-print version).
51. Jacso, P. (2009e). Google Scholar's ghost authors. Library Journal, 134(18), 26-27. Accessed April 3, 2012, from http://www. libraryjournal. com/article/CA6698580. html?q=jacso.
52. Jacso, P. (2009f). The h-index for countries in Web of Science and Scopus. Online Information Review, 33(4), 831-837. Accessed March 31, 2012, from http://www. jacso. info/OIR/2009-04-jacso-h-index-for-countries-in-WoS-and-Scopus. doc (pre-print version).
53. Jacso, P. (2009g). Web of Science. Gale Reference Reviews (online), July 2009. Accessed March 31, 2012, from http://www. gale. cengage. com/reference/peter/200907/science. html.
54. Jacso, P. (2010a). Eigenfactor and Article Influence Score in the Journal Citation Reports. Online Information Review, 34(2), 339-348. Accessed March 31, 2012, from http://www. jacso. info/jcr-eigenfactor/.
55. Jacso, P. (2010b). Metadata mega mess in Google Scholar. Online Information Review, 34(1), 175-191. Accessed March 31, 2012, from http://www. jacso. info/OIR/2010-01-jacso-metadata-mega-mess-in-Google-Scholar. doc.
56. Jacso, P. (2011). Google Scholar duped and deduped-the aura of 'robometrics'. Online Information Review, 35(1), 154-160. Accessed March, 31, 2012 fromhttp://www. jacso. info/OIR/2011-01-jacso-Google-Scholar-duped-deduped. doc (pre-print version).
57. Jacso, P. (2012a). Google Scholar author citation tracker: Is it too little, too late? Online Information Review, 36(1), 126-141. Accessed March 31, 2012, from http://www. jacso. info/OIR/2012-01-jacso-Google-Scholar-author-citation-tracker. doc (pre-print version).
58. Jacso, P. (2012b). Using Google Scholar for journal impact factors and the h-index in nationwide publishing assessments in academia-Siren songs and air-raid sirens. Online Information Review, 36(3), (in press).
59. Martin, B. R. (1996). The use of multiple indicators in the assessment of basic research. Scientometrics, 36(3), 343-362.
60. Mc Veigh, M., & Mann, S. J. (2009). The journal impact factor denominator: Defining citable (countable) items. Journal of the American Medical Association, 302(10), 1107-1109.
61. Meho, L. I., & Yang, K. (2007). Impact of data sources on citation counts and rankings of LIS faculty: Web of Science versus Scopus and Google Scholar. Journal of the American Society for Information Science and Technology, 58(13), 2105-2125.
62. Moed, H. F. (2010). Measuring contextual citation impact of scientific journals. Journal of Informetrics, 4(3), 265-277. Accessed April 3, 2012, from http://arxiv. org/ftp/arxiv/papers/0911/0911. 2632. pdf (pre-print version).
63. Moed, H. F., & Plume, A. (2011). The multi-dimensional research assessment matrix. Research Trends, 23, 5-7.
64. Moed, H. F., & van Leeuwen, T. N. (1995). Improving the accuracy of (the) Institute for Scientific Information's journal impact factors. Journal of the American Society for Information Science and Technology, 46(6), 461-467.
65. Moed, H. F., van Leeuwen, T. N., & Reedijk, J. (1996). A critical analysis of the journal impact factors of Angewandte Chemie and the Journal of the American Chemical Society: Inaccuracies in published impact factors based on overall citations only. Scientometrics, 37(1), 105-116.
66. Moed, H. F., van Leeuwen, T. N., & Reedijk, J. (1999). Towards appropriate indicators of journal impact. Scientometrics, 46(3), 575-589.
67. Nisonger, T. E. (1994). A methodological issue concerning the use of Social Sciences Citation Index Journal Citation Reports impact factor data for journal ranking. Library Collections, Acquisition and Technical Services, 18(4), 447-458.
68. Nisonger, T. E. (2004). The benefits and drawbacks of impact factor for journal collection management in libraries. Serials Librarian, 47(1-2), 57-75.
69. Nunberg, G. (2009). Google's book search: A disaster for scholars. The Chronicle of Higher Education, August 31. Accessed April 3, 2012, from http://chronicle. com/article/Googles-Book-Search-A/48245/.
70. Oppenheim, C. (1996). Do citations count? Citation indexing and the Research Assessment Exercise (RAE). Serials: The Journal for the Serials Community, 9(2), 155-161.
71. Opthof, T. (1999). Submission acceptance rate rapid review system and impact factor. Cardiovascular Research, 41(1), 1-4.
72. Pendlebury, D. A. (2009). The use and misuse of journal metrics and other citation indicators. Archivum Immunologiae Et Therapiae Experimentalis, 57(1), 1-11.
73. Pendlebury, D. A., & Adams, J. (2012). Comments on a critique of the Thomson Reuters journal impact factor. Scientometrics,. doi: 10. 1007/s11192-012-0689-6.
74. Pinski, G., & Narin, F. (1976). Citation influence for journal aggregates of scientific publications: Theory with application to the literature of physics. Information Processing and Management, 12(5), 297-312.
75. Pudovkin, A. I., & Garfield, E. (2004). Rank-normalized impact factor: A way to compare journal performance across subject categories. Proceedings of the ASIST Annual Meeting, 41, 507-515.
76. Pudovkin, A. I., & Garfield, E. (2012). Rank normalization of impact factors will resolve Vanclay's dilemma with TRIF. Scientometrics. doi: 10. 1007/s11192-012-0634-8.
77. Rousseau, R. (2001). Journal evaluation: Technical and practical issues. Library Trends, 50(3), 418-439.
78. Rousseau, R. (2005). Median and percentile impact factors: A set of new indicators. Scientometrics, 63(3), 431-441.
79. Schubert, A., & Braun, T. (1996). Cross-field normalization of scientometric indicators. Scientometrics, 36(3), 311-324.
80. Schubert, A., & Glänzel, W. (1983). Statistical reliability of comparisons based on the citation impact of scientific publications. Scientometrics, 5(1), 59-73.
81. Seglen, P. O. (1997). Why the impact factor of journals should not be used for evaluating research. British Medical Journal, 314, 497-502.
82. Sombatsompop, N., Markpin, T., & Premkamolnetr, N. (2004). A modified method for calculating the impact factors of journals in ISI journal citation reports: Polymer science category in 1997-2001. Scientometrics, 60(2), 217-235.
83. Vanclay, J. K. (2008a). Gauging the impact of journals. Forest Ecology and Management, 256(4), 507-509. Accessed April 3, 2012, from http://epubs. scu. edu. au/cgi/viewcontent. cgi?article=1321&context=esm_pubs.
84. Vanclay, J. K. (2008b). Ranking forestry journals using the h-index. Journal of Informetrics, 2(4), 326-334. Accessed March 31, 2012, from http://arxiv. org/ftp/arxiv/papers/0712/0712. 1916. pdf.
85. Vanclay, J. K. (2009). Bias in the journal impact factor. Scientometrics, 78(1), 3-12. Accessed March 31, 2012, from http://arxiv. org/ftp/cs/papers/0612/0612091. pdf.
86. Vanclay, J. K. (2011). An evaluation of the Australian Research Council's journal ranking. Journal of Informetrics, 5(2), 265-274. Accessed April 3, 2012, from http://arxiv. org/ftp/arxiv/papers/1009/1009. 3359. pdf.
87. Vanclay, J. K. (2012). Impact factor: Outdated artefact or stepping-stone to journal certification? Scientometrics, (online). Accessed March 31, 2012, from http://arxiv. org/ftp/arxiv/papers/1201/1201. 3076. pdf.
88. Williams, G. (2007). Should we ditch impact factors? British Medical Journal, 334(7593), 568. Accessed April 3, 2012, from http://www. ncbi. nlm. nih. gov/pmc/articles/PMC1828313/.
89. Wilson, C. S. (1999). Informetrics. Annual Review of Information Science and Technology, 34, 107-247.
90. Yin, C.-Y., Aris, M. J., & Chen, X. (2010). Combination of Eigenfactor™ and h-index to evaluate scientific journals. Scientometrics, 84(3), 639-648.
91. Yu, G., & Wang, L. (2007). The self-cited rate of scientific journals and the manipulation of their impact factors. Scientometrics, 73(3), 321-330.
92. Yu, G., Wang, X.-H., & Yu, D.-R. (2005). The influence of publication delays on impact factors. Scientometrics, 64(2), 235-246.