Exploring the epistemic impacts of academic performance indicators in the life sciences

Research Evaluation

Volumn 26, Issue 3, 2017, Pages 157-168

  Müller, Ruth ^a   De Rijcke, Sarah ^b  

^a TECHNICAL UNIVERSITY OF MUNICH   (Germany)

^b LEIDEN UNIVERSITY   (Netherlands)

Author keywords

indicators; quality; research evaluation; responsible metrics

Indexed keywords

EID: 85024126146     PISSN: 09582029     EISSN: 14715449     Source Type: Journal    
DOI: 10.1093/reseval/rvx023     Document Type: Article

References (80)

1. Aksnes, D. W., and Rip, A. (2009) 'Researchers' Perceptions of Citations', Research Policy, 38/6: 895-905.
2. Alldred, P., and Miller, T. (2007) 0Measuring What0s Valued or Valuing What's Measured Knowledge Production and the Research Assessment Exercise', in Gillies V. and Lucey H. (eds), Power, Knowledge and the Academy: The Institutional Is Political, pp. 147-67; Basingstoke: Palgrave.
3. Benderly, B. L. (2005) 'Not Your Father's Postdoc', Science, 308/5722: 717-8.
4. Benedictus, R., and Miedema, F. (2016) 'Fewer Numbers, Better Science', Nature, 538/7626: 3.
5. Biagioli, M. (1998) 'The Instability of Authorship: Credit and Responsibility in Contemporary Biomedicine', FASEB J, 12/1: 3-16.
6. Biagioli, M. & Galison, P. (eds.) Scientific Authorship: Credit and Intellectual Property in Science. Galison P. (ed.). New York: Routledge.
7. Blumer, H. (1954) 'What is Wrong with Social Theory', American Sociological Review, 18/1: 3-10.
8. Blumer, H. (1969) Symbolic Interactionism: Perspective and Method. Berkeley: University of California Press 224.
9. Boltanski, L., and Thévenot, L. (2006) On Justification: Economies of Worth. Princeton, NJ: Princeton University Press.
10. Buela-Casal, G., and Zych, I. (2012) 'What do the Scientists Think About the Impact Factor', Scientometrics, 92/2: 281-92.
11. Burrows, R. (2012) 'Living with the h-index Metric Assemblages in the Contemporary Academy', The Sociological Review, 60/2: 355-72.
12. Butler, L. (2003) 'Explaining Australia's Increased Share of ISI Publications-The Effects of a Funding Formula Based on Publication Counts', Research Policy, 32/1: 143-55.
13. Butler, L. (2005) 0What Happens When Funding is Linked to Publication Counts0, in Moed H., Glanzel W. and Schmoch U. (eds), Handbook of Quantitative Science and Technology Research, pp. 389-405. Dordrecht, Netherlands: Springer.
14. Butler, L. (2007) 'Assessing University Research: A Plea for a Balanced Approach', Science and Public Policy, 34/8: 565-74.
15. Callon, M. (1986) 0Elements of a Sociology of Translation: Domestication of the Scallops and the Fishermen of St-Brieuc Bay0, in Law J. (ed.), Power, Action and Belief: A New Sociology of Knowledge, pp. 196-233. London: Routledge.
16. Casati, R. (2010) 'On Publishing', Social Epistemology, 24/3: 191-200.
17. Colwell, R. et al. (2012) Informing Research Choices: Indicators and Judgment. The Expert Panel on Science Performance and Research Funding. Council of Canadian Academies: Ottawa.
18. Cronin, B. (2013) 'Slow Food for Thought', Journal of the American Society for Information Science and Technology, 64/1: 1.
19. Davies, S. R., and Horst, M. (2015) 'Crafting the Group: Care in Research Management', Social Studies of Science, 45/3: 371-93.
20. de Rijcke, S. et al. (2015) 0Comparing Comparisons. On Rankings and Accounting in Hospitals and Universities0, in Guggenheim M., Deville J. and Hrdlickova Z. (eds), Practising Comparison: Logics, Relations, Collaborations. Manchester, UK: Mattering Press.
21. de Rijcke, S. et al. (2015) 'Evaluation Practices and Effects of Indicator Use-A Literature Review', Research Evaluation, 25: 161-9.
22. Derrick, G., and Gillespie, J. (2013) "A number you just can't get away from": Characteristics of Adoption and the Social Construction of Metric Use by Researchers', in 18th International Conference on Science and Technology Indicators, Berlin.
23. Espeland, W. N., and Sauder, M. (2007) 'Rankings and Reactivity: How Public Measures Recreate Social Worlds', American Journal of Sociology, 113/1: 1-40.
24. European Commission (2007) 'The European Researche Area: New Perspectives. Green Paper. Directorate-General for Research and Innovation (ed.). Brussels: European Commission.
25. European Commission (2011) Horizon 2020. European Commission.
26. Felt, U. (2009) 0Creating and Inhabiting Epistemic Living Spaces: Concluding Reflections0, in Felt U. (ed.), Knowing and Living in Academic Research. Convergence and Heterogeneity in Research Cultures in the European Context, pp. 231-42. Prague: Institute of Sociology of the Academy of Sciences of the Czech Republic.
27. Felt, U., and Fochler, M. (2010) 0Riskante Verwicklungen des Epistemischen, Strukturellen und Biographischen: Governance-Strukturen und deren mikropolitische Implikationen für das akademische Leben0, in Biegelbauer P. (ed.), Steuerung von Wissenschaft Die Governance des österreichischen Innovationssystems. Innovationsmuster in der österreichischen Wirtschafts geschichte, Band 7, pp. 297-327. Innsbruck, Austria: Studien-Verlag.
28. Fochler, M. (2016) 'Beyond and between Academia and Business: How Austrian Biotechnology Researchers Describe High-Tech Startup Companies as Spaces of Knowledge Production', Social Studies of Science, 46/2: 259-281.
29. Fochler, M., Felt, U., and Müller, R. (2016) 'Unsustainable Growth, Hyper-Competition, and Worth in Life Science Research: Narrowing Evaluative Repertoires in Doctoral and Postdoctoral Scientists' Work and Lives', Minerva, 54/2: 175-200.
30. Fujimura, J. H. (1987) 'Constructing Do-able' Problems in Cancer Research: Articulating Alignment', Social Studies of Science, 17/2: 257-93.
31. Gezondheidsraad (2016) 0Onderzoek waarvan je beter wordt. Een herorientatie op umc-onderzoek0, in Gezondheidsraad (ed.), Onderzoek waarvan je beter wordt, 2016/14. DenHaag: Gezondheidsraad.
32. Glaser, J. (2013) How does Governance Change Research Content On the Possibility of a Sociological Middle-Range Theory Linking Science Policy Studies to the Sociology of Scientific Knowledge. Berlin: The Technical University Technology Studies Working Papers
33. Hackett, E. J. (1990) 'Science as a Vocation in the 1990s: The Changing Organizational Culture of Academic Science', The Journal of Higher Education, 61/3: 241-79.
34. Hammarfelt, B., and de Rijcke, S. (2015) 'Accountability in Context: Effects of Research Evaluation Systems on Publication Practices, Disciplinary Norms, and Individual Working Routines in the Faculty of Arts at Uppsala University', Research Evaluation, 24/1: 63-77.
35. Hare, P. G. (2003) 'The United Kingdom's Research Assessment Exercise', Higher Education Management and Policy, 15/2: 43-62.
36. Heuts, F., and Mol, A. (2013) 'What Is a Good Tomato A Case of Valuing in Practice', Valuation Studies, 1/2: 125-46.
37. Hicks, D. (2012) 'Performance-based University Research Funding Systems', Research Policy, 41/2: 251-61.
38. Hicks, D. et al. (2015) 'The Leiden Manifesto for Research Metrics', Nature News, 520: 429-31.
39. Ingwersen, P., and Larsen, B. (2014) 'Influence of a Performance Indicator on Danish Research Production and Citation Impact 2000-12', Scientometrics, 101/2: 1325-44.
40. Jimenez-Contreras, E., Anegon, F. D., and Lopez-Cozar, E. D. (2003) 'The Evolution of Research Activity in Spain-The Impact of the National Commission for the Evaluation of Research Activity (CNEAI)', Research Policy, 32/1: 123-42.
41. Kaltenbrunner, W., and de Rijcke, S. (2017) 'Quantifying 'Output' for Evaluation: Administrative Knowledge Politics and Changing Epistemic Cultures in Dutch Law Faculties', Science and Public Policy, 44: 284-93.
42. Keenoy, T. (2005) 'Facing Inwards and Outwards at Once: The Liminal Temporalities of Academic Perfomativity', Time & Society, 14/2-3: 303-21.
43. Kjellberg, H. et al. (2013) 'Valuation Studies Our Collective Two Cents', Valuation Studies, 1/1: 11-30.
44. Lamont, M. (2012) 'Toward a Comparative Sociology of Valuation and Evaluation', Annual Review of Sociology, 38/1: 201-21.
45. Laudel, G., and Glaser, J. (2006) 'Tensions between Evaluations and Communication Practices', Journal of Higher Education Policy and Management, 28/3: 289-95.
46. Laudel, G., and Glaser, J. (2014) 'Beyond Breakthrough Research: Epistemic Properties of Research and Their Consequences for Research Funding', Research Policy, 43/7: 1204-16.
47. Lemke, T. (2002) 'Foucault, Governmentality, and Critique', Rethinking Marxism, 14/3: 49-64.
48. Merton, R. K. (1940) 0Bureaucratic Structure and Personality0, in Social Theory and Social Structure, pp. 195-206. Glencoe, IL: Free Press.
49. Meyer, E. T., and Schroeder, R. (2015) Knowledge Machines: Digital Transformations of the Sciences and Humanities. Cambridge, MA: MIT Press.
50. Moed, H. F. (2008) 'UK Research Assessment Exercises: Informed Judgments on Research Quality or Quantity', Scientometrics, 74/1: 153-61.
51. Müller, R. (2012) 'Collaborating in Life Science Reserach Groups: The Question of Authorship', Higher Education Policy, 25/3: 289-311.
52. Müller, R. (2014) 'Postdoctoral Life Scientists and Supervision Work in the Contemporary University: A Case Study of Changes in the Cultural Norms of Science', Minerva, 52/3: 329-49.
53. Müller, R. (2014) 'Racing forWhat Anticipation and Acceleration in the Work and Career Practices of Academic Life Science Postdocs', Forum Qualitative Sozialforschung/Forum: Qualitative Social Research, 15/3: Art. 15.
54. Opthof, T., and Wilde, A. A. M. (2011) 'Bibliometric Data in Clinical Cardiology Revisited. The Case of 37 Dutch Professors', Netherlands Heart Journal, 19/5: 246-55.
55. Ossenblok, T. L. B., Engels, T. C. E., and Sivertsen, G. (2012) 'The Representation of the Social Sciences and Humanities in the Web of Science-A Comparison of Publication Patterns and Incentive Structures in Flanders and Norway (2005-9)', Research Evaluation, 21/4: 280-90.
56. Regalado, A. (1995) 'Scientific Publishing-Multiauthor Papers on the Rise', Science, 268/5207: 25-25.
57. RFTE (2005) Strategie für die Entwicklung der Life Sciences in Osterreich. Rat für Forschung und Technologieentwicklung.
58. Rushforth, A. D., and de Rijcke, S. (2015) 'Accounting for Impact The Journal Impact Factor and the Making of Biomedical Research in the Netherlands', Minerva, 53/2: 117-39.
59. Sa, C. M., Kretz, A., and Sigurdson, K. (2013) 'Accountability, Performance Assessment, and Evaluation: Policy Pressures and Responses from Research Councils', Research Evaluation, 22/2: 105-17.
60. Sauder, M., and Espeland, W. N. (2009) 'The Discipline of Rankings: Tight Coupling and Organizational Change', American Sociological Review, 74/ 1: 63-82.
61. Schneider, J. W. (2009) 'An Outline of the Bibliometric Indicator Used for Performance-based Funding of Research Institutions in Norway', European Political Science, 8/3: 364-78.
62. Sigl, L. (2016) 'On the Tacit Governance of Research by Uncertainty: How Early Stage Researchers Contribute to the Governance of Life Science Research', Science, Technology&Human Values, 41: 347-74.
63. Sousa, S. B., and Brennan, J. (2014) 0The UK Research Excellence Framework and the transformation of research production0, in Musselin C. and Teixeira P. (eds), Reforming Higher Education: Public Policy Design and Implementation, pp. 65-80; Dordrecht, Netherlands: Springer.
64. Stark, D. (2009) The Sense of Dissonance: Accounts of Worth in Economic Life. Princeton, NJ: Princeton University Press.
65. Stephan, P. (2012) How Economics Shapes Science. Cambridge, MA: Havard University Press.
66. Strange, K. (2008) 'Authorship: Why Not Just Toss a Coin', American Journal of Physiology-Cell Physiology, 295/3: C567-C75.
67. Strauss, A. L., and Corbin, J. M. (1998) Basics of Qualitative Research. Techniques and Procedures forDevelopingGrounded Theory, 2nd edn. London: Sage.
68. Tijdink, J. K. et al. (2014) 'Publicatiedrang en Citatiestress: De Invloed van Prestatie-indicatoren op Wetenschapsbeoefening', Nederlands Tijdschrift Voor Geneeskunde, 158: A7147.
69. Tijdink, J. K. et al. (2016) 'How do Scientists Perceive the Current Publication Culture A Qualitative Focus Group Interview Study Among Dutch Biomedical Researchers', BMJ Open, 6/2
70. Tscharntke, T. et al. (2007) 'Author Sequence and Credit for Contributions in Multiauthored Publications', PLoS Biol, 5/1: 13-14.
71. van Dalen, H., and Henkens, K. (2012) 'Intended and Unintended Consequences of a Publish-or-Perish Culture: A Worldwide Survey', Journal of the Association for Information Science&Technology, 63/7: 1282-93.
72. Vatin, F. (2009). Evaluer et Valoriser: Une Sociologie Economique de la Mesure. Toulouse, France: Presses universitaires du Mirail.
73. Vatin, F. (2013) 'Valuation as Evaluating and Valorizing', Valuation Studies, 1/1: 31-50.
74. Whitley, R. (2000). The Intellectual and Social Organisation of the Sciences, 2nd edn. Oxford: Oxford University Press.
75. Whitley, R. (2007) 0Changing Governance of the Public Sciences0, in Whitley R. and Glaser J. (eds), The Changing Governance of the Sciences. The Advent of Research Evaluation Systems vol. 26 of Sociology of the Sciences Yearbook, pp. 3-27. Dordrecht, Netherlands: Springer.
76. Whitley, R., Glaser, J., and., and Engwall, L. (2010) Reconfiguring Knowledge Production: Changing Authority Relationships in the Sciences and their Consequences for Intellectual Innovation, p. 387. Oxford: Oxford University Press.
77. Willmott, H. (2011) 'Journal List Fetishism and the Perversion of Scholarship: Reactivity and the ABS List', Organization, 18/4: 429-42.
78. Wouters, P. (2014) 0The Citation: From Culture to Infrastructure0, in Cronin B. and Sugimoto C. R. (eds), Beyond Bibliometrics: Harnessing Multidimensional Indicators of Scholarly Impact, pp. 47-66. Cambridge, MA: MIT Press.
79. Wouters, P. et al. (2008) 0Messy Shapes of Knowledge-STS Explores Informatization, New Media, and Academic Work0, in Hackett E. J. et al. (eds), The Handbook of Science and Technology Studies, 3rd edn, pp. 319-52. Cambridge, MA: MIT Press.
80. Zetterström, R. (2004) 'The Number of Authors of Scientific Publications', Acta Paediatrica, 93/5: 581-82.