How has healthcare research performance been assessed? A systematic review

Journal of the Royal Society of Medicine

Volumn 105, Issue 6, 2012, Pages 251-261

  Patel, Vanash M ^a   Ashrafian, Hutan ^a   Ahmed, Kamran ^a   Arora, Sonal ^a   Jiwan, Sejal ^a   Nicholson, Jeremy K ^a   Darzi, Ara ^a   Athanasiou, Thanos ^a  

^a ST MARY S HOSPITAL   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; CITATION ANALYSIS; COCHRANE LIBRARY; DATA EXTRACTION; EMBASE; FEASIBILITY STUDY; HEALTH CARE CONCEPTS; HEALTH CARE PERSONNEL; HEALTH CARE QUALITY; HEALTH CARE SYSTEM; HUMAN; JOURNAL IMPACT FACTOR; MEDLINE; PSYCINFO; PUBLICATION; QUALITY CONTROL; RELIABILITY; SYSTEMATIC REVIEW; VALIDITY; WELLBEING; EVALUATION; HEALTH CARE DELIVERY; HEALTH SERVICES RESEARCH; REVIEW; STANDARD;

DELIVERY OF HEALTH CARE; EVALUATION STUDIES AS TOPIC; HEALTH SERVICES RESEARCH; HUMANS;

EID: 79960343124     PISSN: 01410768     EISSN: None     Source Type: Journal    
DOI: 10.1258/jrsm.2011.110005     Document Type: Article

References (79)

1. Ioannidis JP. Academic medicine: the evidence base. BMJ 2004;329:789-92
2. Kanter S. What is academic medicine? Academic Medicine 2008;8:205-6
3. Wilkinson D. ICRAM (the International Campaign to Revitalise Academic Medicine): agenda setting. BMJ 2004;329:787-9
4. Awasthi S, Beardmore J, Clark J, et al. The Future of Academic Medicine: Five Scenarios to 2025. New York, NY: MilbankMemorial Fund, 2005. See http://www.milbank.org/reports/0507FiveFutures/0507FiveFutures.html
5. Ashrafian H, Rao C, Darzi A, Athanasiou T. Benchmarking in surgical research. Lancet 2009;374:1045-7
6. RAE. Research Assessment Excercise. See http://www.rae.ac.uk/
7. IAF. Institutional Assessment Framework. See http://www.deewr.gov.au/HigherEducation/Resources/Pages/InstPerfPortfolios.aspx.
8. REF. Research Excellence Framework. See http://www.hefce.ac.uk/Research/ref/
9. Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ 2009;339: b2700
10. AHMD. The American Heritagew Medical Dictionary. 2010
11. EBM. Oxford Centre for Evidence-based Medicine. See www.cebm.net/levels_of_evidence
12. Ahmed K, Ashrafian H, Hanna GB, Darzi A, Athanasiou T. Assessment of specialists in cardiovascular practice. Nat Rev Cardiol 2009;6:659-67
13. Wass V, Van der Vleuten C, Shatzer J, Jones R. Assessment of clinical competence. Lancet 2001;357:945-9
14. Armstrong PW, Caverson MM, Adams L, Taylor M, Olley PM. Evaluation of the Heart and Stroke Foundation of Canada Research Scholarship Program: research productivity and impact. Can J Cardiol 1997;13:507-16
15. Brown CG, Coward HR, Stowe RC. Assessing the scientific strength of Chile. Arch Biol Med Exp 1991;24:37-47
16. Buss AR. Evaluation of Canadian psychology departments based upon citation and publication counts. Canadian Psychological Review/Psychologie canadienne 1976;17:143-50
17. Byrnes JP, McNamara CC. Evaluating doctoral programs in the developmental sciences. Developmental Review 2001;21:326-54
18. Divic J. Survey of Croatian science from Science Citation Index between 1985 and 1992 scientometrics - one possible approach. Periodicum Biologorum 1994;96:187-96
19. Druss BG, Marcus SC. Tracking publication outcomes of National Institutes of Health grants. Am J Med 2005;118:658-63
20. Ellwein LB, Khachab M, Waldman RH. Assessing research productivity: evaluating journal publication across academic departments. Acad Med 1989;64:319-25
21. Endler NS. Research productivity and scholarly impact of Canadian psychology departments. Canadian Psychological Review/Psychologie canadienne 1977;18:152-68
22. Endler NS. Where the "stars" are: The 25 most cited psychologists in Canada (1972-1976). Canadian Psychological Review/Psychologie canadienne 1979;20:12-21
23. Endler NS, Rushton JP, Roediger HL. Productivity and scholarly impact (citations) of British, Canadian, and U.S. departments of psychology (1975). American Psychologist 1978;33:1064-77
24. Falagas ME, Papastamataki PA, Bliziotis IA. A bibliometric analysis of research productivity in parasitology by different world regions during a 9-year period (1995-2003). BMC Infectious Diseases 2006;2334:56
25. Gordon RA. Research productivity in master's-level psychology programs. Professional Psychology: Research and Practice 1990;21:33-6
26. Gordon RA, Vicari PJ. Eminence in social psychology: A comparison of textbook citation, Social Sciences CitationIndex, and research productivity rankings. Personality and Social Psychology Bulletin 1992;18:26-38
27. Hendrix D. An analysis of bibliometric indicators, National Institutes of Health funding, and faculty size at Association of American Medical Colleges medical schools, 1997-2007. J Med Library Association 2008;96:324-34
28. Housri N, Cheung MC, Koniaris LG, Zimmers TA. Scientific impact of women in academic surgery. J Surg Res 2008;148:13-16
29. Kaplan P, Mysiw WJ, Pease WS. Academic productivity in physical medicine and rehabilitation. Am J Phys Med Rehabil 1992;71:81-5
30. Koren G, Barker G, Mitchell V, Abramowitch L, Strofolino M, Buchwald M. Patient-based research in a tertiary pediatric centre: A pilot study of markers of scientific activity and productivity. Clin Invest Med 1997;20:354-8
31. Lee J, Kraus KL, Couldwell WT. Use of the h index in neurosurgery. Clinical article. J Neurosurg 2009;111: 387-92
32. Lichtman MA, Oakes D. The productivity and impact of The Leukemia & Lymphoma Society scholar program: The apparent positive effect of peer review. Blood Cells Mol Dis 2001;27:1020-7
33. Soteriades ES, Falagas ME. Comparison of amount of biomedical research originating from the European Union and the United States. BMJ 2005;331:192-4
34. Bornmann L, Wallon G, Ledin A. Does the committee peer review select the best applicants for funding? An investigation of the selection process for two European molecular biology organization programmes. PLoS ONE 2008;3:e3480
35. Bovier PA, Guillain H, Perneger TV. Productivity of medical research in Switzerland. J Investig Med 2001;49:77-84
36. Castellano C, Radicchi F. On the fairness of using relative indicators for comparing citation performance in different disciplines. Arch Immunol Ther Exp (Warsz) 2009;57: 85-90
37. Colman AM, Grant SP, Henderson E. Performance of British university psychology departments as measured by number of publications in BPS journals. Win, 1992-1993. Current Psychology 1992;11:362-71
38. Devos P, Lefranc H, Dufresne E, Beuscart R. From bibliometric analysis to research policy: the use of SIGAPS in Lille University Hospital. Stud Health Technol Inform 2006;124:543-8
39. Fava GA, Guidi J, Sonino N. How citation analysis can monitor the progress of research in clinical medicine. Psychother Psychosom 2004;73:331-3
40. Groneberg-Kloft B, Scutaru C, Kreiter C, Kolzow S, Fischer A, Quarcoo D. Institutional operating figures in basic and applied sciences: Scientometric analysis of quantitative output benchmarking. Health Res Policy Syst 2008;6:6
41. Innes JM. The utility of a citation index as a measure of research ability in psychology. Bull Br Psychol Soc 1973;26:227-8
42. Koskinen J, Isohanni M, Paajala H, et al. How to use bibliometric methods in evaluation of scientific research? An example from Finnish schizophrenia research. Nordic J Psychiatry 2008;62:136-43
43. Lewison G, Thornicroft G, Szmukler G, Tansella M. Fair assessment of the merits of psychiatric research. Br J Psychiatry 2007;190:314-18
44. Michalopoulos A, Falagas ME. A bibliometric analysis of global research production in respiratory medicine. Chest 2005;128:3993-8
45. Radicchi F, Fortunato S, Castellano C. Universality of citation distributions: Toward an objective measure of scientific impact. Proc Natl Acad Sci U S A 2008;105:17268-72
46. Rostami-Hodjegan A, Tucker GT. Journal impact factors: a 'bioequivalence' issue? Br J Clin Pharmacol 2001;51:111-17
47. Tutarel O. Geographical distribution of publications in the field of medical education. BMC Med Educ 2002;2:1472-6920
48. Ugolini D, Bogliolo A, Parodi S, Casilli C, Santi L. Assessing research productivity in an oncology research institute: The role of the documentation center. Bull Med Libr Assoc 1997;85:33-8
49. Ugolini D, Casilli C, Mela GS. Assessing oncological productivity: Is one method sufficient? Eur J Cancer 2002;38:1121-5
50. Campos Jimenez E, Campos Ferrer A. Technological analysis of immunology in Spain, 2004-2005. Inmunologia 2007;26:55-61
51. Bornmann L, Daniel H. Selecting scientific excellence through committee peer review - A citation analysis of publications previously published to approval or rejection of post-doctoral research fellowship applicants. Scientometrics 2006;68:427-40
52. Wenneras C, Wold A. Nepotism and sexism in peer-review. Nature 1997;387:341-3
53. Brkic S. The impact of biomedical literature published in the province of Vojvodina on researchers in the world and in Yugoslavia. Medicinski Pregled 2001;54:21-33
54. David D, Moore M, Domuta A. Romanian psychology on the international psychological scene: A preliminary critical and empirical appraisal. European Psychologist 2002;7:153-60
55. Jokic M. Scientometric evaluation of the projects in biology funded by the Ministry of Science and Technology, Republic of Croatia, in the 1991-1996 period. Periodicum Biologorum 2000;102:129-42
56. Puljak L, Vukojevic K, Kojundzic SL, Sapunar D. Assessing clinical and life sciences performance of research institutions in Split, Croatia, 2000-2006. Croat Med J 2008;49:164-74
57. Vuckovic-Dekic L, Ribaric B, Vracar B. Implementation of various criteria for evaluating the scientific output of professional scientists and clinicians-scientists. Arch Oncology 2001;9:103-6
58. Kellner AW, Ponciano LC. H-index in the Brazilian Academy of Sciences: comments and concerns. An Acad Bras Cienc 2008;80:771-81
59. Mugnaini R, Packer AL, Meneghini R. Comparison of scientists of the Brazilian Academy of Sciences and of the National Academy of Sciences of the USA on the basis of the h-index. Braz J Med Biol Res 2008;41: 258-62
60. Torro-Alves N, Herculano RD, Tercariol CAS, Filho OK, Graeff CFO. Hirsch's index: A case study conducted at the Faculdade de Filosofia, Ciencias e Letras de Ribeirao Preto, Universidade de Sao Paulo. Braz J Med Biol Res 2007;40:1529-36
61. Epstein RJ. Journal impact factors do not equitably reflect academic staff performance in different medical subspecialties. J Investig Med 2004;52:531-6
62. Rezaei-Ghaleh N, Azizi F. The impact factor-based quality assessment of biomedical research institutes in Iran: Effect of impact factor normalization by subject. Arch Iran Med 2007;10:182-9
63. Hickie IB, Christensen H, Davenport TA, Luscombe GM. Can we track the impact of Australian mental health research? Aust N Z J Psychiatry 2005;39:591-9
64. Hirsch JE. An index to quantify an individual's scientific research output. Proc Natl Acad Sci U S A 2005;102: 16569-72
65. Seglen PO. Why the impact factor of journals should not be used for evaluating research. BMJ 1997;314:498-502
66. Macilwain C. Science economics: What science is really worth. Nature 2010;465:682-4
67. Kaplan RS, Norton DP. The balanced scorecard - measures that drive performance. Harv Bus Rev 1992;70:71-9
68. Zelman WN, Blazer D, Gower JM, Bumgarner PO, Cancilla LM. Issues for academic health centers to consider before implementing a balanced-scorecard effort. Acad Med 1999;74:1269-77
69. Uzoka FME. A fuzzy-enhanced multicriteria decision analysis model for evaluating university Academics' research output. Information Knowledge Systems Management 2008;7:273-99
70. Smith R. Measuring the social impact of research. BMJ 2001;323:528
71. Weiss AP. Measuring the impact of medical research: moving from outputs to outcomes. Am J Psychiatry 2007;164:206-14
72. Spaapen J, Dijstelbloem H, Wamelink F. Evaluating Research in Context: A method for comprehensive assessment. 2nd edn. The Hague: Consultative Committee of Sector Councils for Research and Development (COS), 2007
73. Buxton M, Hanney S. How can payback from health services research be assessed? J Health Serv Res Policy 1996;1:35-43
74. Falagas ME, Pitsouni EI, Malietzis GA, Pappas G. Comparison of PubMed, Scopus, Web of Science, and Google Scholar: strengths and weaknesses. FASEB J 2008;22:338-42
75. Google. Google Scholar Beta Website. See http://scholar.google.co.uk/intl/en/scholar/about.html.
76. Thomson-Reuters. ISI Web of Knowledge Website. See www.isiknowledge.com/.
77. Elsevier. Scopus Website. See www.scopus.com.
78. Bornmann L, MarxW, Schier H, Rahm E, Thor A, Daniel H. Convergent validity of bibliometric Google Scholar data in the field of chemistry - Citation counts for papers thatwere accepted by Angewandte Chemie International Edition or rejected but published elsewhere, using Google Scholar, Science Citation Index, Scopus, and Chemical Abstracts. Journal of Informetrics 2009;3:27-35
79. Ashrafian H, Patel VM, Skapinakis P, Athanasiou T. Nobel Prizes in Medicine: are clinicians out of fashion? J R Soc Med (in press)