Can the journal impact factor be used as a criterion for the selection of junior researchers? A large-scale empirical study based on ResearcherID data

Journal of Informetrics

Volumn 11, Issue 3, 2017, Pages 788-799

  Bornmann, Lutz ^a   Williams, Richard ^b  

^a MAX PLANCK INSTITUTE   (Germany)

^b UNIVERSITY OF NOTRE DAME   (United States)

Author keywords

Journal impact factor; Journal metrics; Normalized citation scores; ResearcherID; Web of science

Indexed keywords

STATISTICAL TESTS;

CITATION SCORE; IMPACT FACTOR; JOURNAL METRICS; RESEARCHERID; WEB OF SCIENCE;

PUBLISHING;

EID: 85021441154     PISSN: 17511577     EISSN: 18755879     Source Type: Journal    
DOI: 10.1016/j.joi.2017.06.001     Document Type: Article

References (59)

1. Abramo, G., D'Angelo, C.A., A farewell to the MNCS and like size-independent indicators. Journal of Informetrics 10:2 (2016), 646–651.
2. Abramo, G., D'Angelo, C.A., A farewell to the MNCS and like size-independent indicators: Rejoinder. Journal of Informetrics 10:2 (2016), 679–683, 10.1016/j.joi.2016.01.011.
3. Abramo, G., D'Angelo, C.A., Costa, F.D., Testing the trade-off between productivity and quality in research activities. Journal of the American Society for Information Science and Technology 61:1 (2010), 132–140.
4. Abramo, G., D'Angelo, C.A., Soldatenkova, A., An investigation on the skewness patterns and fractal nature of research productivity distributions at field and discipline level. Journal of Informetrics 11:1 (2017), 324–335, 10.1016/j.joi.2017.02.001.
5. Acock, A.C., A gentle introduction to Stata. 5 ed., 2016, Stata Press, College Station, TX, USA.
6. Allison, P.D., Stewart, J.A., Productivity differences among scientists – evidence for accumulative advantage. American Sociological Review 39:4 (1974), 596–606, 10.2307/2094424.
7. Amara, N., Landry, R., Halilem, N., What can university administrators do to increase the publication and citation scores of their faculty members?. Scientometrics 103:2 (2015), 489–530, 10.1007/s11192-015-1537-2.
8. American Psychological Association, Publication manual of the american psychological association. 6. ed., 2010, American Psychological Association (APA), Washington, DC, USA.
9. Bornmann, L., Marx, W., How to evaluate individual researchers working in the natural and life sciences meaningfully? A proposal of methods based on percentiles of citations. Scientometrics 98:1 (2014), 487–509, 10.1007/s11192-013-1161-y.
10. Bornmann, L., Marx, W., Methods for the generation of normalized citation impact scores in bibliometrics: Which method best reflects the judgements of experts?. Journal of Informetrics 9:2 (2015), 408–418.
11. Bornmann, L., Wallon, G., Ledin, A., Does the committee peer review select the best applicants for funding? An investigation of the selection process for two European Molecular Biology Organization programmes. PUBLIC LIBRARY OF SCIENCE, 3(10), 2008, e3480.
12. Bornmann, L., Stefaner, M., de Moya Anegón, F., Mutz, R., What is the effect of country-specific characteristics on the research performance of scientific institutions? Using multi-level statistical models to rank and map universities and research-focused institutions worldwide. Journal of Informetrics 8:3 (2014), 581–593.
13. Bornmann, L., Scientific peer review. Annual Review of Information Science and Technology 45 (2011), 199–245.
14. Bornmann, L., The Hawthorne effect in journal peer review. Scientometrics 91:3 (2012), 857–862, 10.1007/s11192-011-0547-y.
15. Boyack, K.W., Klavans, R., Sorensen, A.A., Ioannidis, J.P.A., A list of highly influential biomedical researchers, 1996–2011. European Journal of Clinical Investigation 43:12 (2013), 1339–1365, 10.1111/eci.12171.
16. Claveau, F., There should not be any mystery: A comment on sampling issues in bibliometrics. Journal of Informetrics 10:4 (2016), 1233–1240, 10.1016/j.joi.2016.09.009.
17. Cohen, J., Statistical power analysis for the behavioral sciences. 2nd ed., 1988, Lawrence Erlbaum Associates, Publishers, Hillsdale, NJ, USA.
18. Cole, J.R., Cole, S., Social stratification in science. 1973, The University of Chicago Press, Chicago, MA, USA.
19. Costas, R., Bordons, M., van Leeuwen, T.N., van Raan, A.F.J., Scaling rules in the science system: Influence of field-specific citation characteristics on the impact of individual researchers. Journal of the American Society for Information Science and Technology 60:4 (2009), 740–753.
20. D'Angelo, C.A., Abramo, G., Publication rates in 192 research fields. Salah, A., Tonta, Y., Salah, A.A.A., Sugimoto, C., (eds.) Proceedings of the 15th international society of scientometrics and informetrics conference – (ISSI – 2015), Istanbul: Bogazici University Printhouse., 2015, 909–919.
21. de Rijcke, S., Rushforth, A., To intervene, or not to intervene; is that the question? On the role of scientometrics in research evaluation. Journal of the Association for Information Science and Technology 66:9 (2015), 1954–1958.
22. Diem, A., Wolter, S.C., The use of bibliometrics to measure research performance in education sciences. Research in Higher Education 54:1 (2013), 86–114, 10.1007/s11162-012-9264-5.
23. Fanelli, D., Larivière, V., Researchers? Individual publication rate has not increased in a century. PUBLIC LIBRARY OF SCIENCE, 11(3), 2016, e0149504, 10.1371/journal.pone.0149504.
24. Fox, M.F., Publication productivity among scientists: A critical review. Social Studies of Science 13:2 (1983), 285–305.
25. Garfield, E., The history and meaning of the Journal Impact Factor. Journal of the American Medical Association 295:1 (2006), 90–93.
26. Garwood, J., Time to change. Labor Journal 5 (2013), 18–23.
27. Glänzel, W., Schöpflin, U., A bibliometric study on aging and reception processes of scientific literature. Journal of Information Science 21:1 (1995), 37–53, 10.1177/016555159502100104.
28. Hemlin, S., Research on research evaluations. Social Epistemology 10:2 (1996), 209–250.
29. Hirsch, J.E., An index to quantify an individual's scientific research output. Proceedings of the National Academy of Sciences of the United States of America 102:46 (2005), 16569–16572, 10.1073/pnas.0507655102.
30. Hirsch, J.E., Does the h index have predictive power?. Proceedings of the National Academy of Sciences of the United States of America 104:49 (2007), 19193–19198, 10.1073/pnas.0707962104.
31. Hutchins, B.I., Yuan, X., Anderson, J.M., Santangelo, G.M., Relative Citation Ratio (RCR): A new metric that uses citation rates to measure influence at the article level. PLoS Biology, 14(9), 2016, e1002541.
32. Ioannidis, J.P.A., Boyack, K.W., Klavans, R., Estimates of the continuously publishing core in the scientific workforce. PUBLIC LIBRARY OF SCIENCE, 9(7), 2014, e101698, 10.1371/journal.pone.0101698.
33. Kline, R.B., Beyond significance testing: Reforming data analysis methods in behavioral research. 2004, American Psychological Association, Washington, DC, USA.
34. Koski, T., Sandström, E., Sandström, U., Towards field-adjusted production: Estimating research productivity from a zero-truncated distribution. Journal of Informetrics 10:4 (2016), 1143–1152, 10.1016/j.joi.2016.09.002.
35. Larivière, V., Costas, R., How many is too many? on the relationship between research productivity and impact?. PUBLIC LIBRARY OF SCIENCE, 11(9), 2016, e0162709, 10.1371/journal.pone.0162709.
36. Laurance, W.F., Useche, D.C., Laurance, S.G., Bradshaw, C.J.A., Predicting publication success for biologists. Bioscience 63:10 (2013), 817–823, 10.1525/bio.2013.63.10.9.
37. Leydesdorff, L., Wouters, P., Bornmann, L., Professional and citizen bibliometrics: Complementarities and ambivalences in the development and use of indicators—a state-of-the-art report. Scientometrics 109:3 (2016), 2129–2150, 10.1007/s11192-016-2150-8.
38. Liu, W.S., Hu, G.Y., Gu, M.D., The probability of publishing in first-quartile journals. Scientometrics 106:3 (2016), 1273–1276, 10.1007/s11192-015-1821-1.
39. Merton, R.K., The Matthew effect in science. Science 159:3810 (1968), 56–63.
40. Miguel, S., Chinchilla-Rodriguez, Z., de Moya-Anegón, F., Open access and Scopus: A new approach to scientific visibility from the standpoint of access. Journal of the American Society for Information Science and Technology 62:6 (2011), 1130–1145, 10.1002/asi.21532.
41. Parker, J., Allesina, S., Lortie, C., Characterizing a scientific elite (B): Publication and citation patterns of the most highly cited scientists in environmental science and ecology. Scientometrics 94:2 (2013), 469–480, 10.1007/s11192-012-0859-6.
42. Paulus, F.M., Rademacher, L., Schäfer, T.A.J., Müller-Pinzler, L., Krach, S., Journal impact factor shapes scientists? Reward signal in the prospect of publication. PUBLIC LIBRARY OF SCIENCE, 10(11), 2015, e0142537, 10.1371/journal.pone.0142537.
43. Piro, F.N., Rørstad, K., Aksnes, D.W., How does prolific professors influence on the citation impact of their university departments?. Scientometrics 107:3 (2016), 941–961, 10.1007/s11192-016-1900-y.
44. Pudovkin, A.I., Garfield, E., Rank-normalized impact factor: a way to compare journal performance across subject categories. Bryans, J.B., (eds.) ASIST 2004: Proceedings of the 67th Asis&T annual meeting, vol 41, 2004: Managing and enhancing information: Cultures and conflicts, 41, 2004, Information Today Inc., Medford, 507–515.
45. Reich, E.S., Science publishing: The golden club. Nature 502:7471 (2013), 291–293.
46. Riffenburgh, R.H., Statistics in medicine. 3. ed., 2012, Elsevier, Oxford, UK.
47. Rijcke, Sarah de, Wouters, Paul F., Rushforth, Alex D., Franssen, Thomas P., Hammarfelt, Björn, Evaluation practices and effects of indicator use—a literature review. Research Evaluation 25:2 (2016), 161–169, 10.1093/reseval/rvv038.
48. Ruiz-Castillo, J., Costas, R., The skewness of scientific productivity. Journal of Informetrics 8:4 (2014), 917–934, 10.1016/j.joi.2014.09.006.
49. Rushforth, A., de Rijcke, S., Accounting for impact? The journal impact factor and the making of biomedical research in the Netherlands. Minerva 53:2 (2015), 117–139, 10.1007/s11024-015-9274-5.
50. Seglen, P.O., The skewness of science. Journal of the American Society for Information Science 43:9 (1992), 628–638.
51. Sinatra, R., Wang, D., Deville, P., Song, C., Barabási, A.-L., Quantifying the evolution of individual scientific impact. Science, 354(6312), 2016, 10.1126/science.aaf5239.
52. van den Besselaar, P., Sandström, U., Does quantity make a difference?. Salah, A.A., Tonta, Y., Salah, A.A.A., Sugimoto, C., Al, U., (eds.) The 15th conference of the international society for scientometrics and informetrics, 2015, ISSI, Boaziçi University Printhouse, Istanbul, Turkey, 577–583.
53. van Dijk, D., Manor, O., Carey, L.B., Publication metrics and success on the academic job market. Current Biology 24:11 (2014), R516–R517, 10.1016/j.cub.2014.04.039.
54. Waltman, L., Traag, V.A., Use of the journal impact factor for assessing individual articles need not be wrong. 2017 Retrieved June 6 2017 https://arxiv.org/abs/1703.02334.
55. Waltman, L., van Eck, N., van Leeuwen, T., Visser, M., van Raan, A., Towards a new crown indicator: An empirical analysis. Scientometrics 87:3 (2011), 467–481, 10.1007/s11192-011-0354-5.
56. Waltman, L., A review of the literature on citation impact indicators. Journal of Informetrics 10:2 (2016), 365–391.
57. Williams, R., Bornmann, L., Sampling issues in bibliometric analysis. Journal of Informetrics 10:4 (2016), 1253–1257.
58. Wouters, P., Thelwall, M., Kousha, K., Waltman, L., de Rijcke, S., Rushforth, A., Franssen, T., The metric tide: Literature review (Supplementary report I to the independent review of the role of metrics in research assessment and management)., 2015, Higher Education Funding Council for England. (HEFCE), London UK.
59. Yang, B., Rousseau, R., Huang, S.Q., Yan, S.L., Do first rate scientists work at first rate organizations?. Malaysian Journal of Library & Information Science 20:1 (2015), 47–60.