Professional and citizen bibliometrics: complementarities and ambivalences in the development and use of indicators—a state-of-the-art report

Scientometrics

Volumn 109, Issue 3, 2016, Pages 2129-2150

  Leydesdorff, Loet ^a   Wouters, Paul ^b   Bornmann, Lutz ^c  

^a UNIVERSITY OF AMSTERDAM   (Netherlands)

^b LEIDEN UNIVERSITY   (Netherlands)

^c MAX PLANCK INSTITUTE   (Germany)

Author keywords

Boundary object; Evaluative bibliometrics; Scientometric indicators; Validity

Indexed keywords

EID: 84989880462     PISSN: 01389130     EISSN: 15882861     Source Type: Journal    
DOI: 10.1007/s11192-016-2150-8     Document Type: Article

References (92)

1. Abramo, G., & d’Angelo, C. A. (2016). A farewell to the MNCS and like size-independent indicators. Journal of Informetrics, 10(2), 646–651.
2. Aguillo, I. F. (2011). Is Google Scholar useful for bibliometrics? A webometric analysis. Scientometrics, 91(2), 343–351.
3. Ahlgren, P., Persson, O., & Rousseau, R. (2014). An approach for efficient online identification of the top-k percent most cited documents in large sets of Web of Science documents. ISSI Newsletter, 10(4), 81–89.
4. Azrout, R., van Spanje, J., & de Vreese, C. (2013). A threat called Turkey: Perceived religious threat and support for EU entry of Croatia, Switzerland and Turkey. Acta Politica, 48(1), 2–21. doi:10.1057/ap.2012.20.
5. Baumgartner, S. E., & Leydesdorff, L. (2014). Group-based trajectory modeling (GBTM) of citations in scholarly literature: Dynamic qualities of “transient” and “sticky knowledge claims”. Journal of the Association for Information Science and Technology, 65(4), 797–811. doi:10.1002/asi.23009.
6. Bensman, S. J. (2007). Garfield and the impact factor. Annual Review of Information Science and Technology, 41, 93–155.
7. Bornmann, L., & Leydesdorff, L. (2013). Macro-indicators of citation impacts of six prolific countries: InCites data and the statistical significance of trends. PLoS ONE, 8(2), e56768.
8. Bornmann, L., Leydesdorff, L., & Mutz, R. (2013). The use of percentiles and percentile rank classes in the analysis of bibliometric data: Opportunities and limits. Journal of Informetrics, 7(1), 158–165.
9. Bornmann, L., & Marx, W. (2014). How to evaluate individual researchers working in the natural and life sciences meaningfully? A proposal of methods based on percentiles of citations. Scientometrics, 98(1), 487–509. doi:10.1007/s11192-013-1161-y.
10. Bornmann, L., & Mutz, R. (2011). Further steps towards an ideal method of measuring citation performance: The avoidance of citation (ratio) averages in field-normalization. Journal of Informetrics, 5(1), 228–230.
11. Bornmann, L., Mutz, R., & Daniel, H.-D. (2011a). A reliability-generalization study of journal peer reviews—A multilevel meta-analysis of inter-rater reliability and its determinants. PLoS ONE, 5(12), e14331.
12. Bornmann, L., Mutz, R., Hug, S. E., & Daniel, H.-D. (2011b). A meta-analysis of studies reporting correlations between the h index and 37 different h index variants. Journal of Informetrics, 5(3), 346–359. doi:10.1016/j.joi.2011.01.006.
13. Bornmann, L., Thor, A., Marx, W., & Schier, H. (2016). The application of bibliometrics to research evaluation in the humanities and social sciences: An exploratory study using normalized Google Scholar data for the publications of a research institute. Journal of the Association for Information Science and Technology. doi:10.1002/asi.23627.
14. Bornmann, L. & Wohlrabe, K. (in preparation). Normalization of citation impact in economics.
15. Casti, J. (1989). Alternate realities. New York: Wiley.
16. Coleman, B. J., Bolumole, Y. A., & Frankel, R. (2012). Benchmarking individual publication productivity in logistics. Transportation Journal, 51(2), 164–196.
17. Council of Canadian Academies. (2012). Informing research choices: Indicators and judgment: The expert panel on science performance and research funding. Ottawa: Council of Canadian Academies.
18. Dahler-Larsen, P. (2011). The evaluation society. Stanford, CA: Stanford University Press.
19. de Solla Price, D. (1965). Networks of scientific papers: The pattern of bibliographic references indicates the nature of the scientific research front. Science, 149(3683), 510–515.
20. de Solla Price, D. J. (1970). Citation measures of hard sciences, soft science, technology and non-science. In C. E. Nelson & D. K. Polloc (Eds.), Communication among scientists and engineers (pp. 3–22). Lexington, MA: Heath.
21. Delgado López-Cózar, E., Robinson-García, N., & Torres-Salinas, D. (2014). The Google scholar experiment: How to index false papers and manipulate bibliometric indicators. Journal of the Association for Information Science and Technology, 65(3), 446–454. doi:10.1002/asi.23056.
22. Egghe, L. (2006). Theory and practise of the g-index. Scientometrics, 69(1), 131–152.
23. El Emam, K., Arbuckle, L., Jonker, E., & Anderson, K. (2012). Two h-index benchmarks for evaluating the publication performance of medical informatics researchers. Journal of Medical Internet Research. doi:10.2196/jmir.2177.
24. Elkana, Y., Lederberg, J., Merton, R. K., Thackray, A., & Zuckerman, H. (1978). Toward a metric of science: The advent of science indicators. New York, NY: Wiley.
25. Garfield, E. (1972). Citation analysis as a tool in journal evaluation: Journals can be ranked by frequency and impact of citations for science policy studies. Science, 178(4060), 471–479.
26. Garfield, E. (1979a). Citation indexing—Its theory and application in science, technology, and humanities. New York, NY: Wiley.
27. Garfield, E. (1979b). Is citation analysis a legitimate evaluation tool? Scientometrics, 1(4), 359–375.
28. Gibbons, M., Limoges, C., Nowotny, H., Schwartzman, S., Scott, P., & Trow, M. (1994). The new production of knowledge: The dynamics of science and research in contemporary societies. London: Sage.
29. Gieryn, T. F. (1983). Boundary-work and the demarcation of science from non-science—Strains and interests in professional ideologies of scientists. American Sociological Review, 48(6), 781–795. doi:10.2307/2095325.
30. Gilbert, G. N., & Mulkay, M. J. (1984). Opening Pandora’s Box. A sociological analysis of scientists’ discourse. Cambridge: Cambridge University Press.
31. Griliches, Z. (1994). Productivity, R&D and the Data constraint. American Economic Review, 84(1), 1–23.
32. Hargens, L. L., & Schuman, H. (1990). Citation counts and social comparisons: Scientists’ use and evaluation of citation index data. Social Science Research, 19(3), 205–221. doi:10.1016/0049-089X(90)90006-5.
33. Harzing, A. W. (2007). Publish or Perish. Retrieved June 27, 2016, from http://www.harzing.com/pop.htm.
34. Harzing, A.-W. (2012). A preliminary test of Google Scholar as a source for citation data: A longitudinal study of Nobel Prize winners. Scientometrics, 94(3), 1057–1075. doi:10.1007/s11192-012-0777-7.
35. Harzing, A.-W. (2014). A longitudinal study of Google Scholar coverage between 2012 and 2013. Scientometrics, 98(1), 565–575. doi:10.1007/s11192-013-0975-y.
36. Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., & Rafols, I. (2015). Bibliometrics: The Leiden Manifesto for research metrics. Nature, 520(7548), 429–431.
37. Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences of the United States of America, 102(46), 16569–16572. doi:10.1073/pnas.0507655102.
38. Ioannidis, J. P., Boyack, K., & Wouters, P. F. (2016). Citation Metrics: A primer on how (not) to normalize. PLoS Biology, 14(9), e1002542.
39. Jacso, P. (2012a). Google Scholar Author Citation Tracker: Is it too little, too late? Online Information Review, 36(1), 126–141. doi:10.1108/14684521211209581.
40. Jacso, P. (2012b). Google Scholar metrics for publications: The software and content features of a new open access bibliometric service. Online Information Review, 36(4), 604–619. doi:10.1108/14684521211254121.
41. Kosten, J. (2016). A classification of the use of research indicators. Scientometrics, 108(1), 457–464. doi:10.1007/s11192-016-1904-7.
42. Kostoff, R. N., & Martinez, W. L. (2005). Is citation normalization realistic? Journal of Information Science, 31(1), 57–61.
43. Kreft, G. G., & de Leeuw, E. (1988). The see-saw effect: A multilevel problem? Quality & Quantity, 22(2), 127–137.
44. Larivière, V., Archambault, É., Gingras, Y., & Vignola-Gagné, É. (2006). The place of serials in referencing practices: Comparing natural sciences and engineering with social sciences and humanities. Journal of the American Society for Information Science and Technology, 57(8), 997–1004.
45. Larivière, V., Kiermer, V., MacCallum, C. J., McNutt, M., Patterson, M., Pulverer, B., et al. (2016). A simple proposal for the publication of journal citation distributions. Biorxiv. doi:10.1101/062109.
46. Latour, B. (1987). Science in action: How to follow scientists and engineers through society. Cambridge, MA: Harvard University Press.
47. Leydesdorff, L., & Bornmann, L. (2011). Integrated impact indicators (I3) compared with impact factors (IFs): An alternative research design with policy implications. Journal of the American Society of Information Science and Technology, 62(11), 2133–2146.
48. Leydesdorff, L., & Bornmann, L. (2016). The operationalization of “fields” as WoS subject categories (WCs) in evaluative bibliometrics: The cases of “library and information science” and “science & technology studies”. Journal of the Association for Information Science and Technology, 67(3), 707–714. doi:10.1002/asi.23408.
49. Leydesdorff, L., Bornmann, L., Comins, J., & Milojević, S. (2016). Citations: Indicators of quality? The impact fallacy. Frontiers in Research Metrics and Analytics. doi:10.3389/frma.2016.00001.
50. Leydesdorff, L., Bornmann, L., Mutz, R., & Opthof, T. (2011). Turning the tables in citation analysis one more time: Principles for comparing sets of documents. Journal of the American Society for Information Science and Technology, 62(7), 1370–1381.
51. Leydesdorff, L., & Opthof, T. (2011). Scopus’ SNIP indicator: Reply to Moed. Journal of the American Society for Information Science and Technology, 62(1), 214–215.
52. Leydesdorff, L., & Opthof, T. (2013). Citation analysis with Medical Subject Headings (MeSH) using the web of knowledge: A new routine. Journal of the American Society for Information Science and Technology, 64(5), 1076–1080.
53. Martyn, J., & Gilchrist, A. (1968). An evaluation of British Scientific Journals. London: Aslib.
54. Marx, W. (2011). Special features of historical papers from the viewpoint of bibliometrics. Journal of the American Society for Information Science and Technology, 62(3), 433–439.
55. Mingers, J. (2014). Problems with SNIP. Journal of Informetrics, 8(4), 890–894.
56. Mingers, J., & Leydesdorff, L. (2015). A review of theory and practice in scientometrics. European Journal of Operational Research, 246(1), 1–19. doi:10.1016/j.ejor.2015.04.002.
57. Moed, H. F. (2010). Measuring contextual citation impact of scientific journals. Journal of Informetrics, 4(3), 265–277.
58. Moed, H. F., De Bruin, R. E., & Van Leeuwen, T. N. (1995). New bibliometric tools for the assessment of national research performance: Database description, overview of indicators and first applications. Scientometrics, 33(3), 381–422.
59. Nagin, D. S. (2005). Group-based modeling of development. Cambridge, MA: Harvard University Press.
60. Nederhof, A. J. (2006). Bibliometric monitoring of research performance in the social sciences and the humanities: A review. Scientometrics, 66(1), 81–100.
61. Neuhaus, C., & Daniel, H.-D. (2009). A new reference standard for citation analysis in chemistry and related fields based on the sections of Chemical Abstracts. Scientometrics, 78(2), 219–229.
62. OECD. (1963, 1976). The measurement of scientific and technical activities: “Frascati Manual”. Paris: OECD.
63. Oswald, A. J. (2007). An examination of the reliability of prestigious scholarly journals: Evidence and implications for decision-makers. Economica, 74(293), 21–31. doi:10.1111/j.1468-0335.2006.00575.x.
64. Plomp, R. (1990). The significance of the number of highly cited papers as an indicator of scientific prolificacy. Scientometrics, 19(3), 185–197.
65. Prins, A. A. M., Costas, R., van Leeuwen, T. N., & Wouters, P. F. (2016). Using Google Scholar in research evaluation of humanities and social science programs: A comparison with Web of Science data. Research Evaluation. doi:10.1093/reseval/rvv049.
66. Pudovkin, A. I., & Garfield, E. (2002). Algorithmic procedure for finding semantically related journals. Journal of the American Society for Information Science and Technology, 53(13), 1113–1119.
67. Rafols, I., & Leydesdorff, L. (2009). Content-based and algorithmic classifications of journals: Perspectives on the dynamics of scientific communication and indexer effects. Journal of the American Society for Information Science and Technology, 60(9), 1823–1835.
68. Rafols, I., Leydesdorff, L., O’Hare, A., Nightingale, P., & Stirling, A. (2012). How journal rankings can suppress interdisciplinary research: A comparison between Innovation Studies and Business & Management. Research Policy, 41(7), 1262–1282. doi:10.1016/j.respol.2012.03.015.
69. Robinson, W. D. (1950). Ecological correlations and the behavior of individuals. American Sociological Review, 15, 351–357.
70. Rotolo, D., & Leydesdorff, L. (2015). Matching Medline/PubMed data with Web of Science: A routine in R language. Journal of the Association for Information Science and Technology. doi:10.1002/asi.23385.
71. Ruiz-Castillo, J., & Waltman, L. (2015). Field-normalized citation impact indicators using algorithmically constructed classification systems of science. Journal of Informetrics, 9(1), 102–117.
72. Rushforth, A., & de Rijcke, S. (2015). Accounting for impact? The journal impact factor and the making of biomedical research in the Netherlands. Minerva, 53(2), 117–139. doi:10.1007/s11024-015-9274-5.
73. Schubert, A., & Braun, T. (1986). Relative indicators and relational charts for comparative assessment of publication output and citation impact. Scientometrics, 9(5), 281–291.
74. Seglen, P. O. (1992). The skewness of science. Journal of the American Society for Information Science, 43(9), 628–638.
75. Seglen, P. O. (1997). Why the impact factor of journals should not be used for evaluating research. British Medical Journal, 314, 498–502.
76. Simon, H. A. (1973). The organization of complex systems. In H. H. Pattee (Ed.), Hierarchy theory: The challenge of complex systems (pp. 1–27). New York: George Braziller Inc.
77. Simon, H. A. (2002). Near decomposability and the speed of evolution. Industrial and Corporate Change, 11(3), 587–599.
78. Spaan, J. A. E. (2010). The danger of pseudoscience in informetrics. Journal of Informetrics, 4, 439–440.
79. Star, S. L., & Griesemer, J. R. (1989). Institutional ecology, translations and boundary objects—Amateurs and professionals in Berkeleys-Museum-of-Vertebrate-Zoology, 1907–39. Social Studies of Science, 19(3), 387–420. doi:10.1177/030631289019003001.
80. Stirling, A. (2007). ‘Opening up’ and ‘closing down’: Power, participation, and pluralism in the social appraisal of technology. Science, Technology and Human Values, 33(2), 262–294. doi:10.1177/0162243907311265.
81. Tijssen, R., Visser, M., & van Leeuwen, T. (2002). Benchmarking international scientific excellence: Are highly cited research papers an appropriate frame of reference? Scientometrics, 54(3), 381–397.
82. van den Daele, W., & Weingart, P. (1975). Resistenz und Rezeptivität der Wissenschaft-zu den Entstehungsbedingungen neuer Disziplinen durch wissenschaftliche und politische Steuerung. Zeitschrift fuer Soziologie, 4(2), 146–164.
83. Van Eck, N. J., Waltman, L., van Raan, A. F., Klautz, R. J., & Peul, W. C. (2013). Citation analysis may severely underestimate the impact of clinical research as compared to basic research. PLoS ONE, 8(4), e62395.
84. Waltman, L. (2016). A review of the literature on citation impact indicators. Journal of Informetrics, 10(2), 365–391.
85. Waltman, L., Calero-Medina, C., Kosten, J., Noyons, E. C. M., Tijssen, R. J. W., van Eck, N. J., et al. (2012). The Leiden Ranking 2011/2012: Data collection, indicators, and interpretation. Journal of the American Society for Information Science and Technology, 63(12), 2419–2432.
86. Waltman, L., & van Eck, N. J. (2012a). The inconsistency of the h-index. Journal of the American Society for Information Science and Technology, 63(2), 406–415. doi:10.1002/asi.21678.
87. Waltman, L., & van Eck, N. J. (2012b). A new methodology for constructing a publication-level classification system of science. Journal of the American Society for Information Science and Technology, 63(12), 2378–2392.
88. Waltman, L., & van Eck, N. J. (2013). A systematic empirical comparison of different approaches for normalizing citation impact indicators. Journal of Informetrics, 7(4), 833–849.
89. Waltman, L., van Eck, N. J., van Leeuwen, T. N., & Visser, M. S. (2013). Some modifications to the SNIP journal impact indicator. Journal of Informetrics, 7(2), 272–285.
90. Wang, J. (2013). Citation time window choice for research impact evaluation. Scientometrics, 94(3), 851–872.
91. Wilsdon, J., Allen, L., Belfiore, E., Campbell, P., Curry, S., & Hill, S., et al. (2015). The metric tide: Report of the independent review of the role of metrics in research assessment and management. London: HEFCE. doi:10.13140/RG.2.1.4929.1363.
92. Zitt, M., Ramanana-Rahary, S., & Bassecoulard, E. (2005). Relativity of citation performance and excellence measures: From cross-field to cross-scale effects of field-normalisation. Scientometrics, 63(2), 373–401.