Proposals of standards for the application of scientometrics in the evaluation of individual researchers working in the natural sciences;Vorschläge für standards zur anwendung der szientometrie bei der evaluation von einzelnen wissenschaftler(inne)n im bereich der naturwissenschaften

Zeitschrift fur Evaluation

Volumn 12, Issue 1, 2013, Pages 103-127

  Bornmann, Lutz ^a   Marx, Werner ^b  

^a MAX PLANCK INSTITUTE   (Germany)

^b MAX PLANCK INSTITUT FÜR FESTKÖRPERFORSCHUNG   (Germany)

Author keywords

Citations; Percentiles; Publications; Scientometrics

Indexed keywords

EID: 84877825304     PISSN: 16195515     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Article

References (93)

1. Abramo, Giovanni/Cicero, Tindaro/D'Angelo, Ciriaco A. (2011): Assessing the Varying Level of Impact Measurement Accuracy as a Function of the Citation Window Length. In: Journal of Informetrics, 5(4), S. 659-667. doi: 10.1016/j.joi.2011.06.004.
2. Abramo, Giovanni/D'Angelo, Ciriaco A./Di Costa, Flavia (2010): Testing the Trade-Off Between Productivity and Quality in Research Activities. In: Journal of the American Society for Information Science and Technology, 61(1), S. 132-140.
3. Abramo, Giovanni/D'Angelo, Ciriaco A. (2011): Evaluating Research: from Informed Peer Review to Bibliometrics. In: Scientometrics, 87(3), S. 499-514. doi: 10.1007/s11192-011-0352-7.
4. Aksnes, Dag W. (2003): A Macro Study of Self-Citation. In: Scientometrics, 56(2), S. 235-246.
5. Albarrán, Pedro/Ruiz-Castillo, Javier (2011): References Made and Citations Received by Scientific Articles. In: Journal of the American Society for Information Science and Technology, 62(1), S. 40-49. doi: 10.1002/asi.21448.
6. Alonso, Sergio/Cabrerizo, Francisco J./Herrera-Viedma, Enrique/Herrera, Francisco (2009): H-Index: A Review Focused in its Variants, Computation and Standardization for Different Scientific Fields. In: Journal of Informetrics, 3(4), S. 273-289. doi: 10.1016/j.joi.2009.04.001.
7. American Psychological Association (2009): Publication Manual of the American Psychological Association. Washington: American Psychological Association (APA) (6. Aufl.).
8. Andres, Ana (2011): Measuring Academic Research: How to Undertake a Bibliometric Study. New York: Neal-Schuman Publishers.
9. Azoulay, Pierre/Graff Zivin, Joshua S./Manso, Gustavo (2009): Incentives and Creativity: Evidence from the Academic Life Sciences (NBER Working Paper No. 15466). Cambridge, MA, USA: National Bureau of Economic Research (NBER).
10. Bornmann, Lutz (2011): Scientific Peer Review. In: Annual Review of Information Science and Technology, 45, S. 199-245.
11. Bornmann, Lutz (in Druck): How to Analyse Percentile Citation Impact Data Meaningfully in Bibliometrics: The Statistical Analysis of Distributions, Percentile Rank Classes and Top-Cited Papers. In: Journal of the American Society for Information Science and Technology. Bornmann, Lutz/Bowman, Benjamin F./Bauer, Johann/Marx, Werner/Schier, Hermann/Palzenberger
12. Margit (2012): Standards für die Anwendung der Bibliometrie bei der Evaluation von Forschungsinstituten im Bereich der Naturwissenschaften. In: Zeitschrift für Evaluation, 11(12), S. 233-260.
13. Bornmann, Lutz/Daniel, Hans-Dieter (2007): What Do We Know about the h Index? In: Journal of the American Society for Information Science and Technology, 58(9), S. 1381-1385. doi: 10.1002/ asi.20609.
14. Bornmann, Lutz/Daniel, Hans-Dieter (2008): What Do Citation Counts Measure? A Review of Studies on Citing Behavior. In: Journal of Documentation, 64(1), S. 45-80. doi: 10.1108/0022041081084 4150.
15. Bornmann, Lutz/Daniel, Hans-Dieter (2009): The State of h Index Research. Is the h Index the Ideal Way to Measure Research Performance? In: EMBO Reports, 10(1), S. 2-6. doi: 10.1038/embor. 2008.233.
16. Bornmann, Lutz/de Moya-Anegon, Felix/Leydesdorff, Loet (2010): Do Scientific Advancements Lean on the Shoulders of Giants? A Bibliometric Investigation of the Ortega Hypothesis. In: PLoS ONE, 5(10), e11344.
17. Bornmann, Lutz/de Moya Anegón, Felix/Leydesdorff, Loet (2012): The New Excellence Indicator in the World Report of the SCImago Institutions Rankings 2011. In: Journal of Informetrics, 6(2), S. 333-335. doi: 10.1016/j.joi.2011.11.006.
18. Bornmann, Lutz/Marx, Werner/Gasparyan, Armen Y./Kitas, George D. (2012): Diversity, Value and Limitations of the Journal Impact Factor and Alternative Metrics. In: Rheumatology International (Clinical and Experimental Investigations), 32(7), 1861-1867.
19. Bornmann, Lutz/Marx, Werner/Schier, Hermann/Rahm, Erhard/Thor, Andreas/Daniel, Hans-Dieter (2009): Convergent Validity of Bibliometric Google Scholar Data in the Field of Chemistry. Citation Counts for Papers that were Accepted by Angewandte Chemie International Edition or Rejected but Published Elsewhere, Using Google Scholar, Science Citation Index, Scopus, and Chemical Abstracts. In: Journal of Informetrics, 3(1), S. 27-35. doi: 10.1016/j.joi.2008.11.001.
20. Bornmann, Lutz/Mutz, Rüdiger/Daniel, Hans-Dieter (2008): Are There Better Indices for Evaluation Purposes than the h Index? A Comparison of Nine Different Variants of the h Index Using Data from Biomedicine. In: Journal of the American Society for Information Science and Technology, 59(5), S. 830-837. doi: 10.1002/asi.20806.
21. Bornmann, Lutz/Mutz, Rüdiger/Hug, Sven E./Daniel, Hans-Dieter (2011): A Meta-analysis of Studies Reporting Correlations Between the h Index and 37 Different h Index Variants. In: Journal of Informetrics, 5(3), S. 346-359. doi: 10.1016/j.joi.2011.01.006.
22. Bornmann, Lutz/Mutz, Rüdiger/Marx, Werner/Schier, Hermann/Daniel, Hans-Dieter (2011): A Multilevel Modeling Approach to Investigating the Predictive Validity of Editorial Decisions: Do the Editors of a High-Profile Journal Select Manuscripts that Are Highly Cited After Publication? In: Journal of the Royal Statistical Society-Series A (Statistics in Society), S. 174(4), 857-879. doi: 10.1111/j.1467-985X.2011.00689.x.
23. Bornmann, Lutz/Ozimek, Adam (2012): Stata Commands for Importing Bibliometric Data and Processing Author Address Information. In: Journal of Informetrics, 6(4), S. 505-512. doi: 10.1016/ j.joi.2012.04.002.
24. Boyack, Kevin W. (2004): Mapping Knowledge Domains: Characterizing PNAS. In: Proceedings of the National Academy of Sciences of the United States of America, 101, S. 5192-5199.
25. Chang, Chia-Lin/McAleer, Michael/Oxley, Les (2011): How are Journal Impact, Prestige and Article Influence Related? An Application to Neuroscience. In: Journal of Applied Statistics, 38(11), S. 2563-2573.
26. Cole, Stephen (1992): Making Science: Between Nature and Society. Cambridge, MA, USA: Harvard University Press.
27. Coleman, B. Jay/Bolumole, Yemisi A./Frankel, Robert (2012): Benchmarking Individual Publication Productivity in Logistics. In: Transportation Journal, 51(2), S. 164-196.
28. Costas, Rodrigo/van Leeuwen, Thed N./Bordons, Mario (2010): A Bibliometric Classificatory Approach for the Study and Assessment of Research Performance at the Individual Level: The Effects of Age on Productivity and Impact. In: Journal of the American Society for Information Science and Technology, 61(8), S. 1564-1581.
29. Council of Canadian Academies (2012): Informing Research Choices: Indicators and Judgment: The Expert Panel on Science Performance and Research Funding. Ottawa, Canada: Council of Canadian Academies.
30. Cronin, Blaise/Meho, Lokman I. (2007): Timelines of Creativity: A Study of Intellectual Innovators in Information Science. In: Journal of the American Society for Information Science and Technology, 58(13), S. 1948-1959. doi: 10.1002/Asi.20667.
31. D'Angelo, Ciriaco A./Giuffrida, Cristiano/Abramo, Giovanni (2011): A Heuristic Approach to Author Name Disambiguation in Bibliometrics Databases for Large-scale Research Assessments. In: Journal of the American Society for Information Science and Technology, 62(2), S. 257-269. doi: 10.1002/asi.21460.
32. Danell, Rickard (2011): Can the Quality of Scientific Work Be Predicted Using Information on the Author's Track Record? In: Journal of the American Society for Information Science and Technology, 62(1), S. 50-60. doi: 10.1002/asi.21454.
33. De Bellis, Nicola (2009): Bibliometrics and Citation Analysis: From the Science Citation Index to Cybermetrics. Lanham: Scarecrow Press.
34. De Moya-Anegon, Felix/Guerrero-Bote, Vicente P./Bornmann, Lutz/Moed, Henk F. (in Vorbereitung): The Main Contributors of Scientific Papers and the Output Counting: A Promising New Approach.
35. Doane, David P./Tracy, Ronald L. (2000): Using Beam and Fulcrum Displays to Explore Data. In: American Statistician, 54(4), S. 289-290. doi: 10.2307/2685780.
36. Duffy, Ryan/Jadidian, Alex/Webster, Gregory D./Sandell, Kyle J. (2011): The Research Productivity of Academic Psychologists: Assessment, Trends, and Best Practice Recommendations. In: Scientometrics, 89(1), S. 207-227. doi: 10.1007/s11192-011-0452-4.
37. Egghe, Leo (2006): Theory and Practise of the g-Index. In: Scientometrics, 69(1), S. 131-152. doi: 10.1007/s11192-006-0144-7.
38. Egghe, Leo (2010): The Hirsch Index and Related Impact Measures. In: Annual Review of Information Science and Technology, 44, S. 65-114.
39. Franceschini, Fiorenzo/Galetto, Maurizio/Maisano, Domenico/Mastrogiacomo, Luca (2012): The Success-Index: An Alternative Approach to the h-Index for Evaluating an Individual's Research Output. In: Scientometrics, 92(3), S. 621-641. doi: 10.1007/s11192-011-0570-z. Froghi, Saied/Ahmed, Kamran/Finch, Adam/Fitzpatrick, John M./Khan, Mohammed S./Dasgupta
40. Froghi, Saied/Ahmed, Kamran/Finch, Adam/Fitzpatrick, John M./Khan, Mohammed S./Dasgupta, Prokar (2012): Indicators for Research Performance Evaluation: An Overview. In: BJU International, 109(3), S. 321-324. doi: 10.1111/j.1464-410X.2011.10856.x.
41. García-Pérez, Miguel A. (2010): Accuracy and Completeness of Publication and Citation Records in the Web of Science, PsycINFO, and Google Scholar: A Case Study for the Computation of h Indices in Psychology. In: Journal of the American Society for Information Science and Technology, 61(10), S. 2070-2085. doi: 10.1002/asi.21372.
42. Garfield, Eugene (1979): Citation Indexing-Its Theory and Application in Science, Technology, and Humanities. New York: John Wiley & Sons Ltd.
43. Garfield, Eugene (2002): Highly Cited Authors. In: Scientist, 16(7), S. 10.
44. Glänzel, Wolfgang/Debackere, Koenraad/Thijs, Bart/Schubert, András (2006): A Concise Review on the Role of Author Self-Citations in Information Science, Bibliometrics and Science Policy. In: Scientometrics, 67(2), S. 263-277.
45. Grupp, Hariolf/Mogee, Mary Ellen (2004): Indicators for National Science and Technology Policy: Their Development, Use, and Possible Misuse. In: Moed, Henk F./Glänzel, Wolfgang/Schmoch, Ulrich (Hg.): Handbook of Quantitative Science and Technology Research. The Use of Publication and Patent Statistics in Studies of S&T-Systems. Dordrecht, The Netherlands: Kluwer Academic Publishers, S. 75-94.
46. Haslam, Nick/Laham, Simon M. (2010): Quality, Quantity, and Impact in Academic Publication. In: European Journal of Social Psychology, 40(2), S. 216-220. doi: 10.1002/ejsp.727.
47. Hemlin, Sven (1996): Research on Research Evaluations. In: Social Epistemology, 10(2), S. 209-250.
48. Hirsch, Jorge E. (2005): An Index to Quantify an Individual's Scientific Research Output. In: Proceedings of the National Academy of Sciences of the United States of America, 102(46), S. 16569-16572. doi: 10.1073/pnas.0507655102.
49. Jacso, Peter (2009): Google Scholar's Ghost Authors. In: Library Journal, 134(18), S. 26-27.
50. Jacso, Peter (2010): Metadata Mega Mess in Google Scholar. In: Online Information Review, 34(1), S. 175-191. doi: 10.1108/14684521011024191.
51. Kosmulski, Marek (2011): Successful Papers: A New Idea in Evaluation of Scientific Output. In: Journal of Informetrics, 5(3), S. 481-485. doi: 10.1016/j.joi.2011.03.001.
52. Kosmulski, Marek (2012): Modesty-Index. In: Journal of Informetrics, 6(3), S. 368-369. doi: 10.1016/ j.joi.2012.02.004.
53. Kreiman, Gabriel/Maunsell, John H. R. (2011): Nine Criteria for a Measure of Scientific Output. In: Frontiers in Computational Neuroscience, 5, S. 1-6. doi: 10.3389/fncom.2011.00048.
54. Lehmann, Sune/Jackson, Andrew/Lautrup, Benny (2008): A Quantitative Analysis of Indicators of Scientific Performance. In: Scientometrics, 76(2), S. 369-390. doi: 10.1007/s11192-007-1868-8.
55. Lewison, Grant/Thornicroft, Graham/Szmukler, George I./Tansella, Michele (2007): Fair Assessment of the Merits of Psychiatric Research. In: British Journal of Psychiatry, 190, S. 314-318. doi: 10.1192/bjp.bp.106.024919.
56. Leydesdorff, Loet/Bornmann, Lutz/Mutz, Rüdiger/Opthof, Tobias (2011): Turning the Tables in Citation Analysis One More Time: Principles for Comparing Sets of Documents. In: Journal of the American Society for Information Science and Technology, 62(7), 1370-1381.
57. Martin, Ben R./Irvine, John (1983): Assessing Basic Research-Some Partial Indicators of Scientific Progress in Radio Astronomy. In: Research Policy, 12(2), 61-90.
58. Marx, Werner (2011): Bibliometrie in der Forschungsbewertung: Aussagekraft und Grenzen. In: Forschung & Lehre, 11(11), 858-860.
59. Marx, Werner/Bornmann, Lutz (2012): Der Journal Impact Factor: Aussagekraft, Grenzen und Alternativen in der Forschungsevaluation. In: Beiträge zur Hochschulforschung, 34(2), 50-66.
60. Meho, Lokman I./Spurgin, Kristina M. (2005): Ranking the Research Productivity of Library and Information Science Faculty and Schools: An Evaluation of Data Sources and Research Methods. In: Journal of the American Society for Information Science and Technology, 56(12), S. 1314-1331.
61. Merton, Robert K. (1957): Priorities in Scientific Discovery: A Chapter in the Sociology of Science. In: American Sociological Review, 22(6), 635-659. doi: 10.2307/2089193.
62. Merton, Robert K. (1980): Auf den Schultern von Riesen-Ein Leitfaden durch das Labyrinth der Gelehrsamkeit. Frankfurt am Main: Syndikat.
63. Moed, Henk F. (2005): Citation Analysis in Research Evaluation. Dordrecht: Springer.
64. Moed, Henk F./van Leeuwen, Thed N./Reedijk, John (1996): A Critical Analysis of the Journal Impact Factors of Angewandte Chemie and the Journal of the American Chemical Society-Inaccuracies in Published Impact Factors Based on Overall Citations Only. In: Scientometrics, 37(1), S. 105-116.
65. Norris, Michael/Oppenheim, Charles (2010): The h-Index: A Broad Rreview of a New Bibliometric Indicator. In: Journal of Documentation, 66(5), 681-705. doi: 10.1108/00220411011066790.
66. Nosek, Brian A./Graham, Jesse/Lindner, Nicole M./Kesebir, Selin/Hawkins, Carlee B./Hahn, Cheryl/ Tenney, Elizabeth R. (2010): Cumulative and Career-Stage Citation Impact of Social-Personality Psychology Programs and their Members. In: Personality and Social Psychology Bulletin, 36(10), 1283-1300. doi: 10.1177/0146167210378111.
67. Opthof, Tobias/Wilde, Arthur A. M. (2011): Bibliometric Data in Clinical Cardiology Revisited. The Case of 37 Dutch Professors. In: Netherlands Heart Journal, 19(5), 246-255. doi: 10.1007/s12471-011-0128-y.
68. Panaretos, John/Malesios, Chrisovaladis (2009): Assessing Scientific Research Performance and Impact with Single Indices. In: Scientometrics, 81(3), 635-670. doi: 10.1007/s11192-008-2174-9.
69. Pendlebury, David A. (2008): Using Bibliometrics in Evaluating Research. Philadelphia: Research Department, Thomson Scientific.
70. Pendlebury, David A. (2009): The Use and Misuse of Journal Metrics and Other Citation Indicators. In: Archivum Immunologiae Et Therapiae Experimentalis, 57(1), 1-11. doi: 10.1007/s00005-009-0008-y.
71. Pinski, Gabriel/Narin, Francis (1976): Citation Influence for Journal Aggregates of Scientific Publications-Theory, with Application to Literature of Physics. In: Information Processing & Management, 12(5), 297-312.
72. Research Evaluation and Policy Project (2005): Quantitative Indicators for Research Assessment-A Literature Review (REPP discussion paper 05/1). Canberra: Research Evaluation and Policy Project, Research School of Social Sciences, The Australian National University.
73. Retzer, Vroni/Jurasinski, Gerald (2009): Towards Objectivity in Research Evaluation Using Bibliometric Indicators: A Protocol for Incorporating Complexity. In: Basic and Applied Ecology, 10(5), 393-400. doi: 10.1016/j.baae.2008.09.001.
74. Ruiz-Castillo, Javier (2012): The Evaluation of Citation Distributions. In: SERIEs: Journal of the Spanish Economic Association, 3(1), 291-310. doi: 10.1007/s13209-011-0074-3.
75. Sahel, José-Alain (2011): Quality Versus Quantity: Assessing Individual Research Performance. In: Science Translational Medicine, 3(84). doi: 84cm1310.1126/scitranslmed.3002249.
76. Schubert, András/Braun, Tibor (1993): Reference Standards for Citation Based Assessments. In: Scientometrics, 26(1), 21-35.
77. Schubert, András/Braun, Tibor (1996): Cross-Field Normalization of Scientometric Indicators. In: Scientometrics, 36(3), 311-324.
78. Smith, Andy T./Eysenck, Michael (2002): The Correlation Between RAE Ratings and Citation Counts in Psychology. London: Department of Psychology, Royal Holloway, University of London, UK.
79. StataCorp. (2011): Stata Statistical Software: Release 12. College Station, TX, USA: Stata Corporation.
80. Strotmann, Andreas/Zhao, Dangzhi (2012): Author Name Disambiguation: What Difference Does it Make in Author-Based Citation Analysis? In: Journal of the American Society for Information Science and Technology, 63(9), 1820-1833. doi: 10.1002/asi.22695.
81. Sugimoto, Cassidy R./Cronin, Blaise (2012): Biobibliometric Profiling: An Examination of Multifaceted Approaches to Scholarship. In: Journal of the American Society for Information Science and Technology, 63(3), 450-468. doi: 10.1002/asi.21695.
82. Taylor, Jim (2011): The Assessment of Research Quality in UK Universities: Peer Review or Metrics? In: British Journal of Management, 22(2), 202-217. doi: 10.1111/j.1467-8551.2010.00722.x.
83. Thompson, Dennis F./Callen, Eric C./Nahata, Milap C. (2009): New Indices in Scholarship Assessment. In: American Journal of Pharmaceutical Education, 73(6).
84. Tijssen, Robert/van Leeuwen, Thed N. (2006): Centres of Research Excellence and Science Indicators. Can 'Excellence' Be Captured in Numbers? In: Glänzel, Wolfgang (Hg.): Ninth International Conference on Science and Technology Indicators. Leuven: Katholieke Universiteit Leuven, S. 146-147.
85. Tijssen, Robert/Visser, Martijn/van Leeuwen, Thed N. (2002): Benchmarking International Scientific Excellence: Are Highly Cited Research Papers an Appropriate Frame of Reference? In: Scientometrics, 54(3), S. 381-397.
86. van Raan, Anthony F. J. (1996): Advanced Bibliometric Methods as Quantitative Core of Peer Review Based Evaluation and Foresight Exercises. In: Scientometrics, 36(3), 397-420.
87. van Raan, Anthony F. J. (2005): Measurement of Central Aspects of Scientific Research: Performance, Interdisciplinarity, Structure. In: Measurement, 3(1), 1-19.
88. van Raan, Anthony F. J. (2008): Bibliometric Statistical Properties of the 100 Largest European Research Universities: Prevalent Scaling Rules in the Science System. In: Journal of the American Society for Information Science and Technology, 59(3), 461-475. doi: 10.1002/asi.20761.
89. Vinkler, Peter (2010): The Evaluation of Research by Scientometric Indicators. Oxford: Chandos Publishing. Waltman, Ludo/Calero-Medina, Clara/Kosten, Joost/Noyons, Ed C. M./Tijssen, Robert J. W./van Eck
90. Nees J./Wouters, Paul (2012): The Leiden Ranking 2011/2012: Data Collection, Indicators, and Interpretation. In: Journal of the American Society for Information Science and Technology, 63(12), 2419-2432.
91. Waltman, Ludo/van Eck, Nees J. (2012): The Inconsistency of the h-Index. In: Journal of the American Society for Information Science and Technology, 63(2), 406-415. doi: 10.1002/asi.21678.
92. Yin, Chun-Yang (2011): Do Impact Factor, h-Index and Eigenfactor (TM) of Chemical Engineering Journals Correlate Well with Each Other and Indicate the Journals' Influence and Prestige? In: Current Science, 100(5), 648-653.
93. Zhang, Lin/Glänzel, Wolfgang (2012): Where Demographics Meets Scientometrics: Towards a Dynamic Career Analysis. In: Scientometrics, 91(2), 617-630. doi: 10.1007/s11192-011-0590-8.