Evaluation practices and effects of indicator use-a literature review

Research Evaluation

Volumn 25, Issue 2, 2016, Pages 161-169

  De Rijcke, Sarah ^a   Wouters, Paul F ^a   Rushforth, Alex D ^a   Franssen, Thomas P ^a   Hammarfelt, Björn ^a,b  

^a LEIDEN UNIVERSITY   (Netherlands)

^b UNIVERSITY OF BORÅS   (Sweden)

Author keywords

Assessments; Evaluation; Indicators; Metrics; Strategic responses

Indexed keywords

EID: 84984685509     PISSN: 09582029     EISSN: 14715449     Source Type: Journal    
DOI: 10.1093/reseval/rvv038     Document Type: Review

References (116)

1. Aagaard, K. (2015), 'How Incentives Trickle Down: Local Use of a National Bibliometric Indicator System', Science and Public Policy, 1-13.
2. Aagaard, K., Bloch, C. and Schneider, J. W. (2015), 'Impacts of Performance- Based Research Funding Systems: The Case of the Norwegian Publication Indicator', Research Evaluation, 1-12.
3. Aksnes, D. W. and Rip, A. (2009), 'Researchers' Perceptions of Citations', Research Policy, 38/6: 895-905.
4. Alldred, P. and Miller, T. (2007), 'Measuring What's Valued or Valuing What's Measured? Knowledge Production and the Research Assessment Exercise', in V. Gillies and H. Lucey (eds) Power, Knowledge and the Academy: The Institutional Is Political, pp. 147-67. Basingstoke: Palgrave.
5. Archambault, É . and Larivière, V. (2009), 'History of the Journal Impact Factor: Contingencies and Consequences', Scientometrics, 79/3: 635-49.
6. Ström, F., and Hansson, J. (2013), 'How Implementation of Bibliometric Practice Affects the Role of Academic Libraries', Journal of Librarianship and Information Science, 45/4: 316-22.
7. Boltanski, L. and Thévenot, L. (2006), On Justification: Economies of Worth. Princeton, NJ: Princeton University Press.
8. Braun, T. (2012), 'Editorial', Scientometrics, 92/2: 207-8.
9. Buela-Casal, G. and Zych, I. (2012), 'What do the Scientists Think About the Impact Factor?', Scientometrics, 92/2: 281-92.
10. Butler, L. (2003a), 'Explaining Australia's Increased Share of ISI Publications-the Effects of a Funding Formula Based on Publication Counts', Research Policy, 32/1: 143-55.
11. Butler, L. (2003b), 'Modifying Publication Practices in Response to Funding Formulas', Research Evaluation, 12/1: 39-46.
12. Butler, L. (2005), 'What happens when funding is linked to publication counts?', in H. Moed, W. Glänzel and U. Schmoch (eds) Handbook of Quantitative Science and Technology Research, pp. 389-405. Dordrecht: Springer.
13. Butler, L. (2007), 'Assessing University Research: A Plea for a Balanced Approach', Science and Public Policy, 34/8: 565-74.
14. Butler, L. (2010), 'Impacts of Performance-based Research Funding Systems: A Review of the Concerns and the Evidence', in OECD, Performance-based Funding for Public Research in Tertiary Education Institutions: Workshop Proceedings. OECD Publishing.
15. Burrows, R. (2012), 'Living with the h - index? Metric Assemblages in the Contemporary Academy', The Sociological Review, 60/2: 355-72.
16. Chandler, J., Barry, J., and Clark, H. (2002), 'Stressing Academe: The Wear and Tear of the New Public Management', Human Relations, 55/9: 1051-69.
17. Chew, M., Villanueva, E. V., and Van Der Weyden, M. B. (2007), 'Life and Times of the Impact Factor: Retrospective Analysis of Trends for Seven Medical Journals (1994-2005) and their Editors' Views', Journal of the Royal Society of Medicine, 100/3: 142-50.
18. Collini, S. (2015), 'Defending Universities: Argument and Persuasion', Power and Education, 7/1: 29-33.
19. Colwell, R. et al. (2012), Informing Research Choices: Indicators and Judgment. The Expert Panel on Science Performance and Research Funding. Ottawa: Council of Canadian Academics.
20. Cozzens, S., and Melkers, J. (1997), 'Use and Usefulness of Performance Measurement in State Science and Technology Programs', Policy Studies Journal, 25/3: 425-35.
21. Craig, R., Amernic, J., and Tourish, D. (2014), 'Perverse Audit Culture and Accountability of the Modern Public University', Financial Accountability and Management, 30/1: 1-24.
22. Cronin, B. and Sugimoto, C. (eds) (2014), Beyond Bibliometrics: Harnessing Multidimensional Indicators of Scholarly Impact. Cambridge, MA: MIT Press.
23. Dahler-Larsen, P. (2012), The Evaluation Society. Stanford CA: Stanford Business Books.
24. Dahler-Larsen, P. (2014), 'Constitutive Effects of Performance Indicator Systems', Public Management Review, 16/7: 969-86.
25. De Boer, H. F., Enders, J., and Leisyte, L. (2007), 'Public Sector Reform in Dutch Higher Education: The Organisational Transformation of the University', Public Administration, 85/1: 27-46.
26. De Boer, H., Enders, J. and Schimank, U. (2007), On the Way Towards New Public Management? The Governance of University Systems in England, the Netherlands, Austria, and Germany. Netherlands: Springer.
27. Deem, R., Hillyard, S., and Reed, M. (2007), Knowledge, Higher Education and the New Managerialism: The Changing Management of UK Universities. Oxford: Oxford University Press.
28. Demsar, F. and Juznic, P. (2013), 'Transparency of Research Policy and the Role of Librarians', Journal of Librarianship and Information Science, 46: 139-47.
29. de Rijcke, S., and Rushforth, A. D. (2015), 'To Intervene, or Not to Intervene, is that the Question? On the Role of Scientometrics in Research Evaluation', Journal of the Association for Information Science and Technology, 66:1954-58. Article first published online: 13 May 2015, DOI:10.1002/asi.23382.
30. de Rijcke, S., Wallenburg, I., Bal, R., and Wouters, P. (in press), 'Comparing Comparisons. On Rankings and Accounting in Hospitals and Universities', in M. Guggenheim, J. Deville and Z. Hrdlickova (eds) Practising Comparison: Logics, Relations, Collaborations. Manchester: Mattering Press, doi:10.1093/reseval/rvv040.
31. Derrick, G. E. and Gillespie, J. (2013), '"A Number You Just Can't Get Away from": Characteristics of Adoption and the Social Construction of Metrics Use by Researchers', in S. Hinze and A. Lottmann (eds) Proceedings of the 18th International Conference on Science and Technology Indicators, pp. 104-16. Berlin.
32. Derrick, G. E. and Pavone, V. (2013), 'Democratising Research Evaluation: Achieving Greater Public Engagement with Bibliometrics-informed Peer Review', Science and Public Policy, 40/5: 563-75.
33. Espeland, W. N. and Sauder, M. (2007), 'Rankings and Reactivity: How Public Measures Recreate Social Worlds', American Journal of Sociology, 113/1: 1-40.
34. Espeland, W. N. and Stevens, M. L. (1998), 'Commensuration as a Social Process', Annual review of sociology, 24: 313-43.
35. Fealing, K., Lane, J., Marburger, J. III, and Shipp, S. (eds) (2011), The Science of Science Policy. A Handbook. Stanford, CA: Stanford University Press.
36. Feller, I. (2009), 'Performance Measurement and the Governance of American Academic Science', Minerva, 47/3: 323-44.
37. Garfield, E. (2006), 'The History and Meaning of the Journal Impact Factor', JAMA, 295/1: 90-3.
38. Georghiou, L. et al. (2000), Impact of the Research Assessment Exercise and the Future of Quality Assurance in the Light of Changes in the Research Landscape. Manchester: Policy Research in Engineering, Science and Technology. accessed 29 November 2014.
39. Gill, R. (2009), 'Breaking the Silence: TheHidden Injuries of Neo-liberal Academia', in Secrecy and Silence in the Research Process: Feminist reflections, pp. 228-44. London: Routledge.
40. Gläser, J. (2010), 'Concluding Reflections: From Governance to Authority Relations?', in R. Whitley, J. Gläser and L. Engwall (eds) Reconfiguring Knowledge Production: Changing Authority Relationships in the Sciences and Their Consequences for Intellectual Innovation, pp. 357-69. Oxford: Oxford University Press.
41. Gläser, J. (2013), How Does Governance Change Research Content? On the Possibility of a Sociological Middle-range Theory Linking Science Policy Studies to the Sociology of Scientific Knowledge. Berlin: The Technical University Technology Studies Working Papers.
42. Gläser, J., Lange, S., Laudel, G., and Schimank, U. (2010), 'The Limits of Universality: How Field-specific Epistemic Conditions Affect Authority Relations and their Consequences', in R. Whitley, J. Glä ser and L. Engwall (eds) Reconfiguring Knowledge Production: Changing Authority Relationships in the Sciences and their Consequences for Intellectual Innovation, pp. 291-324. Oxford: Oxford University Press.
43. Gläser, J. and Laudel, G. (2001), 'Integrating Scientometric Indicators into Sociological Studies: Methodical and Methodological Problems', Scientometrics, 52/3: 411-34.
44. Gläser, J., Laudel, G., Hinze, S. and Butler, L. (2002), 'Impact of Evaluationbased Funding on the Production of Scientific Knowledge: What to Worry about, and How to Find Out', Report for the German Ministry for Education and Research. accessed 29 November 2014.
45. Hammarfelt, B. and De Rijcke, S. (2015), 'Accountability in Context: Effects of Research Evaluation Systems on Publication Practices, Disciplinary Norms and Individual Working Routines in the Faculty of Arts at Uppsala University', Research Evaluation, 24/1: 63-77.
46. Hammarfelt, B., Nelhans, G., Eklund, P., and Astrom, F. (n.d.) 'The Heterogeneous Landscape of Bibliometric Indicators: Evaluating Models for Allocating Resources at Swedish Universities', (Manuscript under review) doi:10.1093/reseval/rvv040.
47. Hare, P. G. (2003), 'The United Kingdom's Research Assessment Exercise: Impact on Institutions, Departments, Individuals', Higher Education Management and Policy, 15: 43-62.
48. Hargens, L. L. and Schuman, H. (1990), 'Citation Counts and Social Comparisons: Scientists' Use and Evaluation of Citation Index Data', Social Science Research, 19/3: 205-21.
49. Harley, S. (2002), 'The Impact of Research Selectivity on Academic Work and Identity in UK Universities', Studies in Higher Education, 27/2: 187-205.
50. Harley, D., Acord, S. K., Earl-Novell, S., Lawrence, S., and King, C. J. (2010), Assessing the Future Landscape of Scholarly Communication: An Exploration of Faculty Values and Needs in Seven Disciplines. Center for Studies in Higher Education. accessed 29 November 2014.
51. Hemlin, S. (2006), 'Creative Knowledge Environments for Research Groups in Biotechnology. The Influence of Leadership and Organisational Support in Universities and Business Companies', Scientometrics, 67/1: 121-142.
52. Hessels, L. K., van Lente, H. and Smits, R. E. H. M. (2009), 'In Search of Relevance: The Changing Contract between Science and Society', Science and Public Policy, 36: 387-401.
53. Hicks, D. (2012), 'Performance-based University Research Funding Systems', Research Policy, 41/2: 251-61.
54. Hoecht, A. (2006), 'Quality Assurance in UK Higher Education: Issues of Trust, Control, Professional Autonomy and Accountability', Higher Education, 51/4: 541-63.
55. Huth, E. J. (1986), 'Irresponsible Authorship and Wasteful Publication', Annals of Internal Medicine, 104/2: 257-9.
56. Ingwersen, P. and Larsen, B. (2014), 'Influence of a Performance Indicator on Danish Research Production and Citation Impact 2000-12', Scientometrics, 101/12: 1325-44.
57. Jensen, C. B. (2011), 'Making Lists, Enlisting Scientists', Science Studies, 24/2: 64-84.
58. Jiménez-Contreras, E., de Moya Anegon, F. and Lopez-Cozar, E. D. (2003), 'The Evolution of Research Activity in Spain: The Impact of the National Commission for the Evaluation of Research Activity (CNEAI)', Research Policy, 32/1: 123-42.
59. Keenoy, T. (2005), 'Facing Inwards and Outwards at Once: The liminal Temporalities of Academic Perfomativity', Time and Society, 14/2-3: 303-21.
60. Keevers, L., Treleaven, L., Sykes, C. and Darcy, M. (2012), 'Made to Measure: Taming Practices with Results-based Accountability', Organisation Studies, 33/1: 97-120.
61. Lamont, M. (2012), 'Toward a Comparative Sociology of Valuation and Evaluation', Annual Review of Sociology, 38: 201-21.
62. Laudel, G. and Gläser, J. (2006), 'Tensions between Evaluations and Communication Practices', Journal of Higher Education Policy and Management, 28/3: 289-95.
63. Laudel, G. and Gläser, J. (2014), 'Beyond Breakthrough Research: Epistemic Properties of Research and their Consequences for Research Funding', Research Policy, 43/7: 1204-16.
64. Leisyte, L. and Dee, J. R. (2012), 'Understanding Academic Work in a Changing Institutional Environment', Higher Education: Handbook of Theory and Research, 27: 123-206.
65. Linkova, M. (2014), 'Unable to Resist: Researchers' Responses to Research Assessment in the Czech Republic', Human Affairs, 24/1: 78-88.
66. Luukkonen, T. (2012), 'Conservatism and Risk-taking in Peer Review: Emerging ERC Practices', Research Evaluation 21/1: 48-60.
67. MacRoberts, M. H. and MacRoberts, B. R. (1989), 'Problems of Citation Analysis: A Critical Review', Journal of the American Society for Information Science 40/5: 342-9.
68. Martin, B. and Whitley, R. (2010), 'The UK Research Assessment Exercise: A Case of Regulatory Capture', in R. Whitley, J. Gläser and L. Engwall (eds) Reconfiguring Knowledge Production: Changing Authority Relationships in the Sciences and their Consequences for Intellectual Innovation, pp. 51-80. Oxford: Oxford University Press.
69. Mcnay, I. (1998), 'The Research Assessment Exercise (RAE) and After: "You Never Know How it Will All Turn Out."' Perspectives: Policy and Practice in Higher Education, 2/1: 19-22.
70. Mingers, J. and Willmott, H. (2013), 'Taylorizing Business School Research: On the 'One Best Way' Performative Effects of Journal Ranking Lists', Human Relations, 66/8: 1051-73.
71. Moed, H. F. (2008), 'UK Research Assessment Exercises: Informed Judgments on Research Quality or Quantity?', Scientometrics, 74/1: 153-61.
72. Moed, H. F., Van Leeuwen, T. N. and Reedijk, J. (1996), 'A Critical Analysis of the Journal Impact Factors of Angewandte Chemie and the Journal of the American Chemical Society Inaccuracies in Published Impact Factors Based on Overall Citations Only', Scientometrics, 37/1: 105-16.
73. Moore, W. J., Newman, R. J., Sloane, P. J. and Steely, J. D. (2002), Productivity Effects of Research Assessment Exercises. University of Aberdeen, Department of Economics.
74. Musselin, C. (2013), 'How Peer Review Empowers the Academic Profession and University Managers: Changes in Relationships between the State, Universities and the Professoriate', Research Policy, 42/5: 1165-73.
75. Nedeva, M., Boden, R. and Nugroho, Y. (2012), 'Rank and File: Managing Individual Performance in University Research', Higher Education Policy, 25/3: 335-60.
76. Ossenblok, T. L., Engels, T. C. and Sivertsen, G. (2012), 'The Representation of the Social Sciences and Humanities in the Web of Science-A Comparison of Publication Patterns and Incentive Structures in Flanders and Norway (2005-9)', Research Evaluation, 21/4: 280-90.
77. Osuna, C., Cruz-Castro, L. and Sanz-Menendez, L. (2011), 'Overturning Some Assumptions about the Effects of Evaluation Systems on Publication Performance', Scientometrics, 86: 575-92.
78. Paradeise, C. and Thoenig, J. C. (2013), 'Academic Institutions in Search of Quality: Local Orders and Global Standards', Organisation Studies, 34/2: 189-218.
79. Petersohn, S. (2014), 'Bibliometric Services in Research Evaluation: A New Task Area Strengthening the Jurisdiction of Academic Librarians', 35th IATUL Conference 2014, Espoo, 03.06.2014.
80. Pfeffer, J. and Salancik, G. R. (2003), The External Control of Organisations: A Resource Dependence Perspective. Stanford, CA: Stanford University Press.
81. Pontille, D. and Torny, D. (2010), 'The Controversial Policies of Journal Ratings: Evaluating Social Sciences and Humanities', Research Evaluation, 19/5: 347-60.
82. Porter, T. (1995), Trust in Numbers: The Pursuit of Objectivity in Science and Public Life. Princeton, NY: Princeton University Press.
83. Power, M. (1997), The Audit Society: Rituals of Verification. Oxford: Oxford University Press.
84. Rafols, I., Leydesdorff, L., O'Hare, A., Nightingale, P. and Stirling, A. (2012), 'How Journal Rankings Can Suppress Interdisciplinary Research: A Comparison between Innovation Studies and Business & Management', Research Policy, 41/7: 1262-82.
85. Reale, E. and Seeber, M. (2013), 'Instruments as Empirical Evidence for the Analysis of Higher Education Policies', Higher Education, 65/1: 135-51.
86. Rebora, G. and Turri, M. (2013), 'The UK and Italian Research Assessment Exercises Face to Face', Research Policy, 42/9: 1657-66.
87. Rousseau, S. and Rousseau, R. (2012), 'Interactions between Journal Attributes and Authors' Willingness to Wait for Editorial Decisions', Journal of the American Society for Information Science and Technology, 63/6: 1213-25.
88. Rushforth, A. and De Rijcke, S. (2015), 'Accounting for Impact? The Journal Impact Factor and the Making of Biomedical Research in the Netherlands', Minerva, 53: 117-39.
89. Sa, C. M., Kretz, A., and Sigurdson, K. (2013), 'Accountability, Performance Assessment, and Evaluation: Policy Pressures and Responses from Research Councils', Research Evaluation, 22/2: 105-17.
90. Sauder, M. and Espeland, W. N. (2009), 'The Discipline of Rankings: Tight Coupling and Organisational Change', American Sociological Review, 74/1: 63-82.
91. Schneider, J. W. (2009), 'An Outline of the Bibliometric Indicator Used for Performance-based Funding of Research Institutions in Norway', European Political Science, 8/3: 364-78.
92. Shore, C. (2008), 'Audit Culture and Illiberal Governance Universities and the Politics of Accountability', Anthropological Theory, 8/3: 278-98.
93. Shore, C. (2010), 'Beyond the Multiversity: Neoliberalism and the Rise of the Schizophrenic University', Social Anthropology, 18/1: 15-29.
94. Shore, C., and Wright, S. (2000), The Rise of Audit Culture in Higher Education. In: M., STRATHERN (ed.) Audit Cultures. London: Routledge.
95. Sousa, S. B. and Brennan, J. L. (2014), 'The UK Research Excellence Framework and the Transformation of Research Production', in C. Musselin and P. Teixeira (eds) Reforming Higher Education, pp. 65-80. Dordrecht: Springer.
96. Stephan, P. -E. (2012), How Economics Shapes Science. Cambridge, MA: Harvard University Press.
97. Stöckelova, T. (2014), 'Power at the Interfaces: The Contested Orderings of Academic Presents and Futures in A Social Science Department', Higher Education Policy, 27/4: 435-51.
98. Strathern, M. (ed.) (2000), Audit Cultures: Anthropological Studies in Accountability, Ethics and the Academy. London: Routledge.
99. Taylor, J. (2011), 'The Assessment of Research Quality in UK Universities: Peer Review or Metrics?', British Journal of Management, 22/2: 202-17.
100. Thomas, R. and Davies, A. (2005), 'Theorizing the Micro-politics of Resistance: New Public Management and Managerial Identities in the UK Public Services, ' Organisation Studies 26/5: 683-706.
101. Van Dalen, H. P. and Henkens, K. (2012), 'Intended and Unintended Consequences of a Publish-or-Perish Culture: A Worldwide Survey', Journal of the American Society for Information Science and Technology, 63/7: 1282-93.
102. Van der Meulen, B. (2007), 'Interfering Governance and Emerging Centres of Control: University Research Evaluation in the Netherlands', in R. Whitley and J. Glä ser (eds) The Changing Governance of the Sciences, pp. 191-203. Dordrecht: Springer.
103. Van Noorden, R. (2010), 'Metrics: A Profusion of Measures', Nature, 465/7300: 864-6.
104. Waltman, L. (2015), 'A Review of the Literature on Citation Impact Indicators', Arxiv 1507.02099. [v1] Wed, 8 Jul 2015 11:21:58 GMT.
105. Weingart, P. (2005), 'Impact of Bibliometrics Upon the Science System: Inadvertent Consequences?', Scientometrics, 62/1: 117-31.
106. Whitley, R. (1984), The Intellectual and Social Organisation of the Sciences. Oxford: Oxford University Press.
107. Whitley, R. (2000), The Intellectual and Social Organisation of the Sciences, 2nd edn. Oxford: Oxford University Press.
108. Whitley, R. (2007), 'Changing Governance of the Public Sciences', in R. Whitley and J. Gläser (eds) The Changing Governance of the Sciences. Sociology of the Sciences Yearbook, pp. 3-27. Dordrecht: Springer.
109. Whitley, R. (2011), 'Changing Governance and Authority Relations in the Public Sciences', Minerva 49/4: 359-85.
110. Whitley, R. and Gläser, J. (eds.) (2007), The Changing Governance of the Sciences. Sociology of the Sciences Yearbook. Dordrecht: Springer.
111. Whitley, R., Glä ser, J., and Engwall, L. (eds.) (2010), Reconfiguring Knowledge Production: Changing Authority Relationships in the Sciences and their Consequences for Intellectual Innovation. Oxford: Oxford University Press.
112. Willmott, H. (2011), 'Journal List Fetishism and the Perversion of Scholarship: Reactivity and the ABS List', Organisation, 18/4: 429-42.
113. Wilson, M. and Holligan, C. (2013), 'Performativity, Work-related Emotions and Collective Research Identities in UK University Education Departments: An Exploratory Study', Cambridge Journal of Education, 43/2: 223-41.
114. Woelert, P. (2013), 'The 'Economy of Memory': Publications, Citations, and the Paradox of Effective Research Governance', Minerva, 51/3: 341-62.
115. Wouters, P. (2014), 'The Citation: From Culture to Infrastructure', in B. Cronin and C. R. Sugimoto (eds) Beyond Bibliometrics: Harnessing Multidimensional Indicators of Scholarly Impact, pp. 47-66. Cambridge MA: MIT press.
116. Wouters, P., Thelwall, M., Kousha, K., Waltman, L., de Rijcke, S., Rushforth, A., and Franssen, Th. (2015), The Metric Tide: Literature Review (Supplementary Report I to the Independent Review of the Role of Metrics in Research Assessment and Management). Bristol, UK. HEFCE. DOI:10.13140/RG.2.1.5066.3520.