The assessment of research quality in UK universities: Peer review or metrics?

British Journal of Management

Volumn 22, Issue 2, 2011, Pages 202-217

  Taylor, Jim ^a  

^a LANCASTER UNIVERSITY   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 79955944502     PISSN: 10453172     EISSN: 14678551     Source Type: Journal    
DOI: 10.1111/j.1467-8551.2010.00722.x     Document Type: Article

References (61)

1. Abramo, G., C. A. D'Angelo and A. Caprasecca (2009). 'Allocative efficiency in public research: can bibliometrics help?', Research Policy, 38, pp. 206-215.
2. Adler, N. and A.-W. Harzing (2009). 'When knowledge wins: transcending the sense and nonsense of academic rankings', Academy of Management Learning and Education, 8, pp. 72-95.
3. Auranen, O. and M. Nieminen (2010). 'University research funding and publication performance: an international comparison', Research Policy, 39, pp. 822-834.
4. Barker, K. (2007). 'The UK Research Assessment Exercise: the evolution of a national evaluation system', Research Evaluation, 16, pp. 3-12.
5. Bence, V. and C. Oppenheim (2004). 'The influence of peer review on the research assessment exercise', Journal of Information Science, 30, pp. 347-368.
6. Bessant, J., C. Cooper, S. Dawson, J. Gennard, M. Gardiner, A. Gray, P. Jones, C. Mayer, J. McGee, M. Pidd, G. Rowley, J. Saunders and A. Stark (2003). 'The state of the field in UK management research: reflections of the Research Assessment Exercise Panel', British Journal of Management, 14, pp. 51-68.
7. Cherchye, L. and P. V. Abeele (2005). 'On research efficiency: a micro analysis of Dutch university research in economics and business management', Research Policy, 34, pp. 495-516.
8. Cooper, C. and D. Otley (1998). 'The 1996 Research Assessment Exercise for business and management', British Journal of Management, 9, pp. 73-89.
9. Doyle, J. R., A. J. Arthurs, R. H. Green, L. Mcaulay, M. R. Pitt, P. A. Bottomley and W. Evans (1995). 'The judge, the model of the judge, and the model of the judged as judge: analyses of the UK 1992 Research Assessment Exercise data for business and management studies', Omega Journal of International Management Science, 24, pp. 13-28.
10. Easton, G. and D. M. Easton (2003). 'Marketing journals and the Research Assessment Exercise', Journal of Marketing Management, 19, pp. 5-24.
11. ERA (2010). ERA 2010 Submission Guidelines. Evaluation in Research for Australia, Australian Research Council. Available at
12. van Fleet, D. D., A. McWilliams and D. S. Siegel (2000). 'A theoretical and empirical analysis of journal rankings: the case of formal lists', Journal of Management, 26, pp. 839-861.
13. Geary, J., L. Marriott and M. Rowlinson (2004). 'Journal rankings in business and management and the Research Assessment Exercise in the UK', British Journal of Management, 15, pp. 95-141.
14. Groot, T. and T. Garcia-Valderrama (2006). 'Research quality and efficiency: an analysis of assessments and management issues in Dutch economics and business research programs', Research Policy, 35, pp. 1362-1376.
15. HEFCE (2003). Joint Consultation on the Review of Research Assessment. Consultation by the UK funding bodies on the review by Sir Gareth Roberts, 22 May, Higher Education Funding Council, Bristol. Available at
16. HEFCE (2009a). Interim Report of the REF Bibliometrics Pilot Exercise, June, Higher Education Funding Council, Bristol.
17. HEFCE (2009b). Subject Overview Report. Main Panel I, RAE2008, Higher Education Funding Council, Bristol.
18. HEFCE (2010) Data sources for mainstream QR funding. Available at
19. Hirsch, J. (2005). 'An index to quantify an individual's scientific research output', Proceedings of the National Academy of Sciences of the United States of America, 102, pp. 16569-16572.
20. HM Treasury (2006). Science and Innovation Investment Framework 2004-2014: Next Steps. Budget, March 2006, London. Available at
21. Hobbs, F. D. R. and P. M. Stewart (2006). 'How should we rate research?', British Medical Journal, 332, pp. 983-984.
22. House of Commons (2004a). Research Assessment Exercise: a Re-assessment. Science and Technology Committee, Eleventh Report of Session 2003-04, HC 586, September, London. Available at
23. House of Commons (2004b). Research Assessment Exercise: a Re-assessment: Government Response to the Committee's Eleventh Report of Session 2003-04. First Special Report of Session 2004-05, Science and Technology Committee, HC 34, November, London. Available at
24. Johnes, J., J. Taylor and B. Francis (1993). 'The research performance of UK universities: a statistical analysis of the results of the 1989 Research Selectivity Exercise', Journal of the Royal Statistical Society A, 156, pp. 271-286.
25. Jost, J. T., L. A. Rudman, I. V. Blair, D. R. Carney, N. Dasgupta, J. Glaser and C. D. Hardin (2009). 'The existence of implicit bias is beyond reasonable doubt: a refutation of ideological and methodological objections and executive summary of ten studies that no manager should ignore', Research in Organizational Behavior, 29, pp. 39-69.
26. Kalaitzidakis, P., T. P. Mamuneas and T. Stengos (2003). 'Rankings of academic journals and institutions in economics', Journal of the European Economic Association, 1, pp. 1346-1366.
27. Kelly, A., H. Morris and C. Harvey (2009). Modelling the Outcome of the UK Business and Management Studies RAE 2008 with Reference to the ABS Journal Quality Guide. London: Association of Business Schools, June. Available at
28. Kelly, A., H. Morris, M. Rowlinson and C. Harvey (2009). Academic Journal Quality Guide, Version 3. London: Association of Business Schools, March. Available at
29. Kodrzycki, Y. K. and P. Yu (2006). 'New approaches to ranking economics journals', Contributions to Economic Analysis and Policy, 5, pp. 1-42.
30. Laband, D. N. and M. J. Piette (1994). 'The relative impacts of economics journals: 1970-1990', Journal of Economic Literature, 32, pp. 640-666.
31. Lee, F. S. (2007). 'The Research Assessment Exercise, the state and the dominance of mainstream economics in British universities', Cambridge Journal of Economics, 31, pp. 309-325.
32. Liebowitz, S. L. and J. P. Palmer (1984). 'Assessing the relative impacts of economics journals', Journal of Economic Literature, 22, pp. 77-88.
33. Liner, G. H. (2002). 'Core journals in economics', Economic Inquiry, 40, pp. 138-145.
34. Macdonald, S. and J. Kam (2007). 'Ring a ring o'roses: quality journals and gamesmanship in management studies', Journal of Management Studies, 44, pp. 640-655.
35. Mingers, J. (2009). 'Measuring the contribution of management academics using the Hirsch-index', Operational Research Society, 60, pp. 1143-1153.
36. Mingers, J. and A.-W. Harzing (2007). 'Ranking journals in business and management: a statistical analysis of the Harzing data set', European Journal of Information Systems, 16, pp. 303-316.
37. Mingers, J. and E. A. Lipitakis (2010). 'Counting citations: a comparison of Web of Science and Google Scholar in the field of management'. Working Paper 215, Kent Business School, University of Kent, Canterbury.
38. Mingers, J. and F. Xu (2010). 'The drivers of citations in management science journals', European Journal of Operational Research, 205, pp. 422-430.
39. Mingers, J., K. Watson and P. Scaparra (2009). 'Estimating business and management journal quality from the 2008 Research Assessment Exercise in the UK'. Working Paper 205, Kent Business School, University of Kent, Canterbury.
40. Nederhof, A. J. and A. F. J. van Raan (1993). 'A bibiometric analysis of six economics research groups: a comparison with peer review', Research Policy, 22, pp. 353-368.
41. van Nierop, R. (2009). 'Why do statistics journals have low impact factors?', Statistica Neerlandica, 63, pp. 52-62.
42. Nisbett, R. E. and T. D. Wilson (1977). 'The halo effect: evidence for unconscious alteration of judgments', Journal of Personality and Social Psychology, 35, pp. 250-256.
43. Otley, D. (2002). 'British research in accounting and finance (1996-2000): the 2001 Research Assessment Exercise', British Accounting Review, 34, pp. 387-417.
44. Paul, R. J. (2008). 'Measuring research quality: the United Kingdom Government's Research Assessment Exercise', European Journal of Information Systems, 17, pp. 324-329.
45. Reich, D. A., M. C. Green, T. C. Brock and P. E. Tetlock (2007). 'Biases in research evaluation: inflated assessment, oversight or error-type weighting?', Journal of Experimental Psychology, 43, pp. 633-640.
46. Rinia, E. J., T. N. van Leeuwen, H. G. van Vuren and A. F. J. van Raan (1998). 'Comparative analysis of a set of bibliometric indicators and central peer review criteria', Research Policy, 27, pp. 95-107.
47. Ritzberger, K. (2008). 'A ranking of journals in economics and related fields', German Economics Review, 9, pp. 402-430.
48. Roberts, G. (2003). Review of Research Assessment: A Report to the Funding Bodies. HEFCE Report 2003/22, Bristol. Available at
49. Schneider, F. and H. W. Ursprung (2008). 'The 2008 GEA journal-ranking for the economics profession', German Economics Review, 9, pp. 532-538.
50. Seglen, P. O. (1997). 'Why the impact factor of journals should not be used for evaluating research', British Medical Journal, 314, pp. 498-502.
51. Singh, G., K. M. Haddad and C. W. Chow (2007). 'Are articles in "top" management journals necessarily of higher quality?', Journal of Management Inquiry, 16, pp. 319-331.
52. Starbuck, W. H. (2005). 'How much better are the most-prestigious journals? The statistics of academic publications', Organization Studies, 16, pp. 180-200.
53. Taylor, J. (1994). 'Measuring research performance in business and management studies in the United Kingdom: the 1992 Research Assessment Exercise', British Journal of Management, 5, pp. 275-288.
54. Taylor, J. (1995). 'A statistical analysis of the 1992 Research Assessment Exercise', Journal of the Royal Statistical Society A, 158, pp. 241-261.
55. Taylor, J. and I. Walker (2009). 'Peer assessment of research: how many publications per staff?' Working Paper 2009/035, Lancaster University Management School.
56. Tetlock, P. E. and G. Mitchell (2009). 'Implicit bias and accountability systems: what must organizations do to prevent discrimination?', Research in Organizational Behavior, 29, pp. 3-38.
57. Wall, H. (2009). 'Journal rankings in economics: handle with care'. Working Paper 2009-014A, Federal Bank of St Louis. Available at
58. Walter, G., S. Bloch, G. Hunt and K. Fisher (2003). 'Counting on citations: a flawed way to measure', Medical Journal of Australia, 178, pp. 280-281.
59. Weale, A. R., M. Bailey and P. A. Lear (2004). 'The number of non-citation articles within a journal as a measure of quality: a comparison to the impact factor', BMC Medical Research Methodology, 4, Article 14.
60. Williams, G. (2007). 'Should we ditch impact factors?', British Medical Journal, 334, p. 568.
61. Wilson, T. D., B. M. DePaulo, D. G. Monk and K. J. Klaaren (1993). 'Scientists' evaluation of research: the biasing effects of the importance of the topic', American Psychology Association, 4, pp. 322-325.