Perverse effects of output-based research funding? Butler's Australian case revisited

Journal of Informetrics

Volumn 11, Issue 3, 2017, Pages 905-918

  van den Besselaar, Peter ^a   Heyman, Ulf ^b   Sandström, Ulf ^c  

^a VU UNIVERSITY AMSTERDAM   (Netherlands)

^b UPPSALA UNIVERSITY   (Sweden)

^c ROYAL INSTITUTE OF TECHNOLOGY   (Sweden)

Author keywords

[No Author keywords available]

Indexed keywords

FINANCE; TIME SERIES ANALYSIS;

AUSTRALIA; JOURNAL PUBLICATION; PERFORMANCE BASED; RESEARCH EVALUATION; RESEARCH FUNDING; RESEARCH OUTPUTS; RESEARCH QUALITY; SCIENCE POLICIES;

QUALITY CONTROL;

EID: 85023638982     PISSN: 17511577     EISSN: 18755879     Source Type: Journal    
DOI: 10.1016/j.joi.2017.05.016     Document Type: Article

References (45)

1. Bourke, P., Butler, L., A crisis for Australian science? Canberra: Research school of social sciences, ANU – monograph series. 1993, Australian National University [Performance Indicators Project; no.1].
2. Broad, W.J., The publishing game: Getting more for less. Science 211 (1981), 1137–1139, 10.1126/science.7008199.
3. Butler, L., A list of published papers is no measure of value: The present system rewards quantity, not quality—But hasty changes could be as bad. Nature, 419, 2002, 877.
4. Butler, L., Explaining Australia's increased share of ISI publications—The effects of a funding formula based on publication counts. Research Policy 32 (2003), 143–155.
5. Butler, L., Modifying publication practices in response to funding formulas. Research Evaluation 12:1 (2003), 39–46.
6. Butler, L., What happens when funding is linked to publication counts?. Moed, H.F., Glanzel, W., Schmoch, U., (eds.) Handbook of quantitative science and technology, 2004, Kluwer Academic Publishers, London, 340–389.
7. Butler L. (2010) Impacts of performance-based research funding systems: A review of the concerns and the evidence. In OECD (2010), Performance-based Funding for Public Research in Tertiary Education Institutions: Workshop Proceedings, pp. 127–165.
8. Cronin, B., Sugimoto, C.R., Beyond bibliometrics: Harnessing multidimensional indicators of scholarly impact. 2015, MIT Press, Cambridge, MA.
9. de Jong, S., Barker, K., Cox, D., Sveinsdottir, T., Van den Besselaar, P., Understanding societal impact through studying productive interactions. Research Evaluation 23:2 (2014), 89–102.
10. de Rijcke, S., Wouters, P.F., Rushforth, A.D., Franssen, T.P., Hammarfelt, B., Evaluation practices and effects of indicator use—A literature review. Research Evaluation 25:2 (2015), 161–169.
11. The evaluation of scientific research. Evered, D., Harnett, S., (eds.) Ciba foundation conference, 1988, John Wiley, Chichester.
12. Geuna, A., Martin, B. (2001). University Research Evaluation and Funding: An International Comparison. SPRU Electronic Working Paper Series No. 71.
13. Geuna, A., Martin, B., University research evaluation and funding: an international comparison. Minerva 41 (2003), 277–304.
14. Gläser, J., Laudel, G., Hinze, S. & Butler, L. (2002). Impact of evaluation-based funding on the production of scientific knowledge: What to worry about, and how to find out. (Expertise for the German Ministry of Education and Research). [Available from Research Gate accessed 2016-04-06].
15. Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., Rafols, I., Bibliometrics: the leiden manifesto for research metrics. Nature, 22, 2015, 2015.
16. Hicks, D., Evolving regimes of multi-university research evaluation. Higher Education 57 (2009), 393–404.
17. Hicks, D., Performance-based university research funding systems. Research Policy 41 (2012), 251–261.
18. Hicks, D., One size doesn't fit all: on the co-evolution of national evaluation systems and social science publishing. Confero: essays on Education. Philosophy and Politics 1:1 (2013), 67–90 [(reprinted in (Eds.) Cronin & Sugimoto, Scholarly Metrics under the Microscope. New Yersey: Information Today Inc. 2015, pp. 661–677).
19. Larivière, V., Costas, R., How many is too many? On the relationship between output and impact in research. PLoS One, 11(9), 2015, e0162709, 10.1371/journal.pone.0162709.
20. Lozano, G.A., Larivière, V., Gingras, Y., The weakening relationship between the impact factor and papersè citations in the digital age. Journal of the American Society for Information Science and Technology 63:11 (2012), 2140–2145, 10.1002/asi.22731.
21. MacRoberts, M.H., MacRoberts, B.R., Problems of citation analysis—A critical review. JASIS 40:5 (1989), 342–349.
22. Marinova, D., Newman, P., The changing research funding regime in Australia and academic productivity. Mathematics and Computers in Simulation 78:2–3 (2008), 283–291.
23. Martin, B.R., Irvine, J., Assessing basic research—Some partial indicators of scientific progress in radio astronomy. Research Policy 12:2 (1983), 61–90.
24. Narin, F., Evaluative bibliometrics: The use of publication and citation analysis in the evaluation of scientific activity. 1976, Computer Horizons Inc. [456 pp].
25. Nederhof, A.J., van Raan, A.F.J., A bibliometric analysis of 6 economic research groups—A comparison with peer review. Research Policy 22:4 (1993), 353–368.
26. OECD, Performance-based funding for public research in tertiary education institutions: Workshop proceedings. 2010, OECD.
27. Osuna, C., Cruz-Castro, L., Sanz-Menendez, L., Overturning some assumptions about the effects of evaluation systems on publication performance. Scientometrics 86 (2011), 575–592.
28. Pelz, D.C., Andrews, F.M., Scientists in organizations: Productive climates for research and development. 1966, Wiley, New York.
29. Phelan, T.J., A compendium of issues for citation analysis. Scientometrics 45:1 (1999), 117–136.
30. Roche, T., Smith, D.L., Frequency of citations as criterion for the ranking of departments, journals and individuals. Sociological Inquiry 48 (1978), 49–57.
31. Sandström, U., van den Besselaar, P., Quantity and/or quality? The importance of publishing many papers. PLoS One, 11(11), 2016, e0166149, 10.1371/journal.pone.0166149.
32. Sandström, U., Heyman, U., van den Besselaar, P., The complex relationship between competitive funding and performance. Conference: Science technology indicators, Leiden, The Netherlands, 2014.
33. Schneider, J.W., Aagaard, K., Bloch, C.W., What happens when national research funding is linked to differentiated publication counts? A comparison of the Australian and Norwegian publication-based funding models. Research Evaluation 25:3 (2016), 244–256.
34. Seglen, P., Why the impact factor of journals should not be used for evaluating research. BMJ, 314, 1997, 497.
35. Simonton, D.K., Creativity in science: Chance, logic genius, and zeitgeist. 2004, N.Y. Cambridge Univ Press [Reprinted 2008].
36. Stephan, P., Perverse incentives. Nature 484:212 (2012), 29–231.
37. Van Arensbergen, P., Van der Weijden, I., van den Besselaar, P., Different views on scholarly talent—What are the talents we are looking for in science?. Research Evaluation 23:4 (2014), 273–284.
38. Van Raan, A.F.J., Advanced bibliometric methods as quantitative core of peer review based evaluation and foresight exercises. Scientometrics 36:3 (1996), 397–420.
39. Van den Besselaar, P., Sandström, U., Does quantity make a difference?. Salah, A., Sugimoto, S., Al, U., (eds.) Proc. ISSI 2015 conference, Istanbul, 2015, 577–583.
40. Van den Besselaar P., SandstroÅNm U., Counterintuitive effects of incentives? Research Evaluation (Forthcoming).
41. Van den Besselaar, P., Sandström, U., van der Weijden, I., The independence indicator. Archambault, E., Gingras, Y., Lariviere, V., (eds.) Science & Technology Indicators, 2012, OST & Science Metrix, Montreal, 131–141.
42. Van der Weijden, I., Verbree, M., van den Besselaar, P., From bench to bedside: The societal orientation of research leaders. The case of biomedical and health research in the Netherlands. Science & Public Policy 39 (2012), 285–303.
43. Van der Weijden, I., Belder, R., van Arensbergen, P., van den Besselaar, P., How do young tenured professors benefit from a mentor? Effects on management, motivation and performance. Higher Education 69 (2015), 275–287.
44. Waltman, L., Van Eck, N.J., Field-normalized citation impact indicators and the choice of an appropriate counting method. Journal of Informetrics 9:4 (2015), 872–894.
45. Wilsdon, J. et al. (2015). The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management. 10.13140/RG.2.1.4929.1363.