Measuring research 'impact' for academic promotion: Issues from the literature

Journal of Higher Education Policy and Management

Volumn 35, Issue 4, 2013, Pages 410-420

  Smith, Kylie M ^a   Crookes, Ellie ^a   Crookes, Patrick A ^a,b,c  

^a UNIVERSITY OF WOLLONGONG   (Australia)

^b UNIVERSITY OF STAVANGER   (Norway)

^c UNIVERSITY OF HUDDERSFIELD   (United Kingdom)

Author keywords

bibliometrics; ERA; Excellence in Research for Australia; impact; research quality

Indexed keywords

EID: 84881633608     PISSN: 1360080X     EISSN: 14699508     Source Type: Journal    
DOI: 10.1080/1360080X.2013.812173     Document Type: Article

References (59)

1. Adam, D. 2002. The counting house. Nature, 415: 726 - 729.
2. Al-Awqati, Q. 2007. Impact factors and prestige. Kidney International, 71: 183 - 185.
3. Beacham, B., Kalucy, L. and McIntyre, E. 2005. Understanding and measuring research impact. Primary Health Care Research and Information Service, 2: 1 - 12.
4. Bennett, D., Genoni, P. and Haddow, G. 2011. FoR codes pendulum: Publishing choices within Australian research assessment. Australian Universities Review, 53: 88 - 98.
5. Bernard Becker Medical Library. (2011). Model for assessing the impact of research. https://becker.wustl.edu/impact/assessment/index.html
6. Bevan, D. 2004. Impact factor 2002 - too much 'impact'?. Clinical Medical Association, 27: 65 - 66.
7. Bloch, S. and Walter, G. 2001. The impact factor: Time for change. Australian and New Zealand Journal of Psychiatry, 35,: 563 - 568.
8. Bordons, M., Fernandez, M.T. and Gomez, I. 2002. Advantages and limitations in the use of impact factor measures for the assessment of research performance in a peripheral country. Scientometrics, 53: 195 - 206.
9. Butler, L. 2003. Modifying publication practices in response to funding formulas. Research Evaluation, 12: 39 - 46.
10. Buxton, M., Hanney, S. and Jones, T. 2004. Estimating the economic value to societies of the impact of health research: A critical review. Bulletin of the World Health Organization, 82: 733 - 739.
11. Coates, H. and Goedegebuure, L. 2010. The real academic revolution: Why we need to reconceptualise Australia's future academic workforce, and eight possible strategies for how to go about this, Carlton: LH Martin Institute.
12. Cooper, S. and Poletti, A. 2011. The new ERA of journal ranking: The consequences of Australia's fraught encounter with 'quality'. Australian Universities Review, 53: 57 - 65.
13. Crookes, P.A., Reis, S. and Jones, S.C. 2010. The development of a ranking tool for refereed journals in which nursing and midwifery researchers publish their work. Nurse Education Today, 30: 420 - 427.
14. Department of Education, Science and Training. 2007. Research Quality Framework: Assessing the quality and impact of research in Australia - RQF Submission Specifications, Canberra: Australian Government.
15. Duryea, M., Hochman, M. and Parfitt, A. 2007. Measuring the impact of research. Research Global, 1,: 8 - 10.
16. Fava, G.A. and Ottolini, F. 2000. Impact factors versus actual citations. Psychotherapy and Psychosomatics, 69: 287 - 286.
17. Favaloro, E.J. 2009. The journal impact factor: Don't expect its demise any time soon. Clinical Chemical Laboratory Medicine, 47: 1319 - 1324.
18. Frank, M. 2003. Impact factors: Arbiter of excellence?. Journal of Medical Library Association, 91: 4 - 6.
19. Garfield, E. 1996. Fortnightly review: How can impact factors be improved?. British Medical Journal, 313, 411-413
20. Garfield, E. 1999. Journal impact factor: A brief review. Canadian Medical Association Journal, 161: 979 - 980.
21. Genoni, P. and Haddow, G. 2009. ERA and the ranking of Australian humanities journals. Australian Humanities Review, 1: 7 - 26.
22. Grant, J. 2006. Measuring the benefits from research, Cambridge, MA: RAND.
23. Grant, J., Brutscher, P.-B., Kirk, S.E., Butler, L. and Wooding, S. 2010. Capturing research impacts: A review of international practice, Cambridge: Higher Education Funding Council for England.
24. Haddow, G. and Genoni, P. 2010. Citation analysis and peer ranking of Australian social science journals. Scientometrics, 85: 471 - 478.
25. Hanney, S. 2004. Personal interaction with researchers or detached synthesis of the evidence: Modelling the health policy paradox. Evaluation and Research in Education, 18: 72 - 82.
26. Hanney, S., Gonzalez, M., Buxton, M. and Kogan, M. 2003. The utilisation of health research in policy-making: Concepts, examples and methods of assessment. Health Research Policy and Systems, 1: 1 - 28.
27. Hanney, S., Grant, J., Wooding, S. and Buxton, M.J. 2004. Proposed methods for reviewing the outcomes of health research: The impact of funding by the UK's Arthritis Research campaign. Health Research Policy and Systems, 2: 1 - 11.
28. Harzing, A.-W. and V.D, Wal, R. 2009. A Google Scholar h-index for journals: An alternate metric to measure journal impact in economics and business. Journal of the American Society for Information Science and Technology, 60: 41 - 46.
29. HEFCE. 2009. Research excellence framework: Second consultation on the assessment and funding of research, London: Author.
30. Hendrix, D. 2010. Tenure metrics: Bibliometric education and services for academic faculty. Medical Reference Services Quarterly, 29: 183 - 189.
31. Hicks, V. 2005. Developing of CIHR framework to measure the imapct of health research, Ontario: CIHR.
32. Jackson, D., Haigh, C. and Watson, R. 2009. Nurses and publications - The impact of the impact factor. Journal of Clinical Nursing, 18: 2537 - 2538.
33. Jarwal, S.D., Brion, A.M. and King, M.L. 2009. Measuring research quality using the journal impact factor, citations and 'Ranked Journals': Blunt instruments or inspired metrics?. Journal of Higher Education Policy and Management, 31: 289 - 300.
34. Jasco, P. 2010. Pragmatic issues in calculating and comparing the quantity and quality of research through rating and ranking of researchers based on peer reviews and bibliometric indicators from Web of Science, Scopus and Google Scholar. Online Information Review, 34: 972 - 982.
35. Kalucy, L., Bowers, E.J. and McIntyre, E. 2007. Primary health care research impact project. Canberra: Australian Government, Department of Health and Ageing
36. Kellogg, W.K. 2004. Logic model development guide, Battle Creek, MI: W.K Kellogg Foundation.
37. Kelly, C.D. and Jennions, M.D. 2006. The h index and career assessment by numbers. TRENDS in Ecology and Evolution, 21: 167 - 170.
38. King, J. 1987. A review of bibliometric and other science indicators and their role in research. Journal of Information Science, 13: 261 - 275.
39. Kostoff, R.N. 1997. Use and misuse of metrics in research evaluation. Science and Engineering Ethics, 3: 109 - 120.
40. Kurmis, A.P. 2003. Understanding the limitations of the journal impact factor. The Journal of Bone and Joint Surgery, 85: 2449 - 2454.
41. Kuruvilla, S., Mays, N. and Walt, G. 2007. Describing the impact of health services and policy research. Journal of Health Services Research Policy, 12: 23 - 31.
42. Lavis, J., Ross, S., McLeod, C. and Gildiner, A. 2003. Measuring the impact of health research. Journal of Health Services Research and Policy, 8: 165 - 170.
43. Lavis, J., Ross, S.E., Hurley, J.E., Hohenadel, J.M., Stoddart, G.L., Woodward, C.A. and Abelson, J. 2002. Examining the role of health services research in public policy making. The Milbank Quarterly, 80: 125 - 154.
44. MacRoberts, M.H. and MacRoberts, B.R. 1989. Problems of citation analysis: A critical review. Journal of the American Society for Information Science, 40: 342 - 349.
45. Martin, B. 2011. ERA: Adverse consequences. Australian Universities Review, 53: 99 - 102.
46. Molas-Gallart, J., Tang, P. and Morrow, S. 2000. Assessing the non-academic impact of grant-funded socio-economic research: Results from a pilot study. Research Evaluation, 9: 171 - 182.
47. Moodie, G. 2006. Vocational education institutions' role in national innovation. Research in Post-Compulsory Education, 11: 131 - 140.
48. Oppenheim, C. 2008. Out with the old and in with the new: The RAE, bibliometrics and the new REF. Journal of Librarianship and Information Science, 40: 147 - 149.
49. Opthof, T. 1997. Sense and nonsense about the impact factor. Cardiovascular Research, 33: 1 - 7.
50. Ortner, H.M. 2010. The impact factor and other performance measures - Much used with little knowledge about. International Journal of Refractory Metals and Hard Materials, 28: 559
51. Sarli, C.C., Dubinsky, E.K. and Holmes, K.L. 2010. Beyong citation analysis: A model for assessment of research impact. Journal of the Medical Library Association, 98: 17 - 23.
52. Seglen, P.O. 1997a. Why the impact factor of journals should be used for evaluating research. British Medical Journal, 314, 498-502
53. Seglen, P.O. 1997b. Citations and journal impact factors: Questionable indicators of research quality. Allergy, 52: 1050 - 1056.
54. Smith, R. 1998. Unscientific practice flourishes in science: Impact factors of journals should not be used in research assessment. British Medical Journal, 316: 1036
55. Smith, R. 2001. Measuring the social impact of research. British Medical Journal, 323: 528
56. Theus, K.T. 1993. Academic reputations: The process of formation and decay. Public Relations Review, 19: 277 - 291.
57. Watson, R. and Thompson, D.R. 2006. Use of factor analysis in Journal of Advanced Nursing: Literature review. Journal of Advanced Nursing, 55: 330 - 341.
58. Wilson, J. 2011. Creative arts research: A long path to acceptance. Australian Universities Review, 53: 68 - 76.
59. Zavos, C., Kountouras, J. and Katsinelos, P. 2006. Impact factors: Looking beyond the absolute figures and journal rankings. Gastrointestinal Endoscopy, 64: 1034