Two h-index benchmarks for evaluating the publication performance of medical informatics researchers

Journal of Medical Internet Research

Volumn 14, Issue 5, 2012, Pages

  Emam, Khaled El ^a,b   Arbuckle, Luk ^a   Jonker, Elizabeth ^a   Anderson, Kevin ^a  

^a CHILDREN S HOSPITAL OF EASTERN ONTARIO   (Canada)

^b UNIVERSITY OF OTTAWA   (Canada)

Author keywords

Bibliometrics; Evaluation; H index; Medical informatics; Research output

Indexed keywords

ARTICLE; MEDICAL INFORMATICS; PUBLISHING; QUALITY CONTROL; RESEARCH; SAMPLE SIZE; VALIDATION STUDY;

BENCHMARKING; MEDICAL INFORMATICS; PUBLISHING; RESEARCH; SAMPLE SIZE;

EID: 84872435794     PISSN: None     EISSN: 14388871     Source Type: Journal    
DOI: 10.2196/jmir.2177     Document Type: Article

References (86)

1. Abbott A, Cyranoski D, Jones N, Maher B, Schiermeier Q, Van Noorden R. Metrics: Do metrics matter? Nature 2010 Jun 17;465(7300):860-862. [doi: 10.1038/465860a] [Medline: 20559361].
2. Académie des Sciences. Annex 2: Evaluation of individual researchers in theoretical and experimental sciences. In: On the Proper Use of Bibliometrics to Evaluate Individual Researchers. Paris: Académie des Sciences, Institut de France; 2011:31-36.
3. Académie des Sciences. Annex 3: Bibliometric practices by discipline. In: On the Proper Use of Bibliometrics to Evaluate Individual Researchers. Paris: Académie des Sciences, Institut de France; 2011:37-47.
4. European Research Council. 2011. Supporting Top Researchers From Anywhere in the World URL: http://erc.europa.eu/[accessed 2012-10-07] [WebCite Cache ID 6BFC6fUjn].
5. European Commission. Assessing Europe's University-Based Research. Luxembourg: European Commission; 2010.
6. Patel V, Ashrafian H, Ahmed K, Arora S, Jiwan S, Nicholson J, et al. How has healthcare research performance been assessed?: a systematic review. J R Soc Med 2011 Jun;104(6):251-261. [doi: 10.1258/jrsm.2011.110005] [Medline: 21659400].
7. Lewison G, Cottrell R, Dixon D. Bibliometric indicators to assist the peer review process in grant decisions. Res Eval 1999;8(1):47-52. [doi: 10.3152/147154499781777621].
8. Fitzpatrick S. Thomson Reuters. 2009 Aug 3. National Institutes of Health Chooses Web Of Science® to Power Electronic Scientific Portfolio Assistant (e-SPA) URL: http://thomsonreuters.com/content/press-room/science/468364 [accessed 2012-10-07] [WebCite Cache ID 6BFD5G6yD].
9. Bertuzzi S. Behind Medicine: Science and Policy. A View From the US National Institutes of Health on How Return on Investment Analysis Can Inform Science Policy. Brussels: European Parliament; 2010.
10. Wadman M. Study says middle sized labs do best. Nature 2010 Nov 18;468(7322):356-357. [doi: 10.1038/468356a] [Medline: 21085145].
11. Hendrix D. An analysis of bibliometric indicators, National Institutes of Health funding, and faculty size at Association of American Medical Colleges medical schools, 1997-2007. J Med Libr Assoc 2008 Oct;96(4):324-334. [Medline: 18979684].
12. National Research Council (US) Committee on an Assessment of Research Doctorate Programs; Ostriker J, Holland P, Kuh C, Voytuk J, editors. In: Ostriker J, Kuh C, Voytuk J, editors. A Data-Based Assessment of Research-Doctorate Programs in the United States (National Research Council). Washington, DC: National Academies Press; 2011.
13. Oppenheim C. The correlation between citation counts and the 1992 research assessment exercise ratings for British research in genetics, anatomy and archaeology. J Doc 1997;53(5):477-487. [doi: 10.1108/EUM0000000007207].
14. Boyack K, Jordan P. Metrics associated with NIH funding: a high-level view. J Am Med Inform Assoc 2011;18(4):423-431 [FREE Full text] [doi: 10.1136/amiajnl-2011-000213] [Medline: 21527408].
15. Lyubarova R, Itagaki B, Itagaki M. The impact of National Institutes of Health funding on U.S. cardiovascular disease research. PLoS ONE 2009;4(7):e6425 [FREE Full text] [doi: 10.1371/journal.pone.0006425] [Medline: 19641617].
16. US National Science Foundation. Research Performance Progress Report (RPPR) URL: http://www.nsf.gov/bfa/dias/policy/rppr/[accessed 2012-09-28] [WebCite Cache ID 6B1O4gYWb].
17. Alonso S, Cabrerizo F, Herrera-Viedma E, Herrera F. h-Index: a review focused in its variants, computation and standardization for different scientific fields. J Informetr 2009;3:273-289. [doi: 10.1016/j.joi.2009.04.001].
18. Iglesias J, Pecharromán C. Scaling the h-index for different scientific ISI fields. Scientometrics 2007;73:303-320. [doi: 10.1007/s11192-007-1805-x].
19. Radicchi F, Fortunato S, Castellano C. Universality of citation distributions: toward an objective measure of scientific impact. Proc Natl Acad Sci USA 2008 Nov 11;105(45):17268-17272 [FREE Full text] [doi: 10.1073/pnas.0806977105] [Medline: 18978030].
20. Podlubny I. Comparison of scientific impact expressed by the number of citations in different fields of science. Scientometrics 2005;64:95-99. [doi: 10.1007/s11192-005-0240-0].
21. Lee J, Kraus K, Couldwell W. Use of the h index in neurosurgery. Clinical article. J Neurosurg 2009 Aug;111(2):387-392. [doi: 10.3171/2008.10.JNS08978] [Medline: 19392590].
22. Vishwanatham R. Citation analysis in journal rankings: medical informatics in the library and information science literature. Bull Med Libr Assoc 1998 Oct;86(4):518-522. [Medline: 9803294].
23. Deshazo J, Lavallie D, Wolf F. Publication trends in the medical informatics literature: 20 years of "Medical Informatics" in MeSH. BMC Med Inform Decis Mak 2009;9:7 [FREE Full text] [doi: 10.1186/1472-6947-9-7] [Medline: 19159472].
24. Morris T, McCain K. The structure of medical informatics journal literature. J Am Med Inform Assoc 1998;5(5):448-466 [FREE Full text] [Medline: 9760393].
25. Morris T. Structural relationships within medical informatics. Proc AMIA Symp 2000:590-594. [Medline: 11079952].
26. Malin B, Carley K. A longitudinal social network analysis of the editorial boards of medical informatics and bioinformatics journals. J Am Med Inform Assoc 2007;14(3):340-348 [FREE Full text] [doi: 10.1197/jamia.M2228] [Medline: 17329730].
27. Hirsch J. An index to quantify an individual's scientific research output. Proc Natl Acad Sci USA 2005 Nov 15;102(46):16569-16572 [FREE Full text] [doi: 10.1073/pnas.0507655102] [Medline: 16275915].
28. Hirsch J. Does the H index have predictive power? Proc Natl Acad Sci USA 2007 Dec 4;104(49):19193-19198 [FREE Full text] [doi: 10.1073/pnas.0707962104] [Medline: 18040045].
29. Norris M, Oppenheim C. The h-index: a broad review of a new bibliometric indicator. J Doc 2010;66(5):681-705. [doi: 10.1108/00220411011066790].
30. Egghe L. An improvement of the h-index: the g-index. ISSI Newsl 2006;2(1):8-9.
31. Antonakis J, Lalive R. Quantifying scholarly impact: IQp Versus the Hirsch h. J Am Soc Inf Sci Technol 2008;59(6):956-969. [doi: 10.1002/asi.v59:6].
32. Bornmann L, Mutz R, Hug S, Daniel H. A multilevel meta-analysis of studies reporting correlations between the h index and 37 different h index variants. J Informetr 2011;5(3):346-359. [doi: 10.1016/j.joi.2011.01.006].
33. Rad A, Brinjikji W, Cloft H, Kallmes D. The H-index in academic radiology. Acad Radiol 2010 Jul;17(7):817-821. [doi: 10.1016/j.acra.2010.03.011] [Medline: 20471868].
34. Hedley-Whyte J, Milamed D, Hoaglin D. Chairpersons of pathology in the United States: limited benchmarks for publications. Am J Clin Pathol 2010 Aug;134(2):185-192 [FREE Full text] [doi: 10.1309/AJCPMVAYD5V8DLZN] [Medline: 20660319].
35. Sypsa V, Hatzakis A. Assessing the impact of biomedical research in academic institutions of disparate sizes. BMC Med Res Methodol 2009;9:33 [FREE Full text] [doi: 10.1186/1471-2288-9-33] [Medline: 19480665].
36. Ponce F, Lozano A. Academic impact and rankings of American and Canadian neurosurgical departments as assessed using the h index. J Neurosurg 2010 Sep;113(3):447-457. [doi: 10.3171/2010.3.JNS1032] [Medline: 20380531].
37. Bould M, Boet S, Sharma B, Shin E, Barrowman N, Grantcharov T. h-Indices in a university department of anaesthesia: an evaluation of their feasibility, reliability, and validity as an assessment of academic performance. Br J Anaesth 2011 Mar;106(3):325-330 [FREE Full text] [doi: 10.1093/bja/aeq403] [Medline: 21258072].
38. Pagel P, Hudetz J. An analysis of scholarly productivity in United States academic anaesthesiologists by citation bibliometrics. Anaesthesia 2011 Oct;66(10):873-878. [doi: 10.1111/j.1365-2044.2011.06860.x] [Medline: 21864299].
39. Pagel P, Hudetz J. H-index is a sensitive indicator of academic activity in highly productive anaesthesiologists: results of a bibliometric analysis. Acta Anaesthesiol Scand 2011 Oct;55(9):1085-1089. [doi: 10.1111/j.1399-6576. 2011.02508.x] [Medline: 22092205].
40. Kashyap R, Pickering B, Assudani J, Singh B, Ahmed A, Chute C, et al. H-index of panel session participants from AMIA 2010 annual meeting. 2011 Presented at: AMIA 2011 Annual Symposium; Oct 22-26, 2011; Washinton, DC. USA.
41. Shahbodaghi A, Sajjadi S. A scientometric investigation of the publication trends of Iranian medical informatics articles based on ISI Citation Databases. J Paramed Sci 2010;1(4):2-11.
42. Bornmann L. Daniel H-D. Does the h-index for ranking of scientists really work? Scientometrics 2005;65(3):391-392. [doi: 10.1007/s11192-005-0281-4].
43. Bornmann L, Wallon G, Ledin A. Is the h index related to (standard) bibliometric measures and to the assessments by peers? An investigation of the h index by using molecular life sciences data. Res Eval 2008;17(2):149-156. [doi: 10.3152/095820208X319166].
44. van Raan A. arxiv. 2006. Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups URL: http://arxiv.org/ftp/physics/papers/0511/0511206.pdf [accessed 2012-10-08] [WebCite Cache ID 6BGH6KOJW].
45. Lovegrove B, Johnson S. Assessment of research performance in biology: how well do peer review and bibliometry correlate? BioScience 2008 Feb 1;58(2):160-164.
46. Bornmann L. Scientific peer review. Annu Rev Inf Sci Technol 2011;45:199-245.
47. Bornmann L, Mutz R, Neuhaus C., Daniel H-D. Citation counts for research evaluation: standards of good practice for analyzing bibliometric data and presenting and interpreting results. Ethics Sci Environ Polit 2008;8:93-102. [doi: 10.3354/esep00084].
48. Benway B, Kalidas P, Cabello J, Bhayani S. Does citation analysis reveal association between h-index and academic rank in urology? Urology 2009 Jul;74(1):30-33. [doi: 10.1016/j.urology.2008.10.045] [Medline: 19567283].
49. Seglen P. The skewness of science. J Am Soc Inf Sci 1992;43(9):628-638. [doi: 10.1002/(SICI)1097-4571(199210)43:9<628].
50. Seglen P. Why the impact factor of journals should not be used for evaluating research. BMJ 1997 Feb 15;314(7079):498-502. [Medline: 9056804].
51. Leydesdorff L, Bornmann L, Mutz R, Opthof T. Turning the tables on citation analysis one more time: principles for comparing sets of documents. J Am Soc Inf Sci Technol 2011;62(7):1370-1381. [doi: 10.1002/Asi.21534].
52. Winkler W. String comparator metrics and enhanced decision rules in the Fellegi-Sunter model of record linkage. In: Proceedings of the Survey Research Methods Section. Alexandria, VA: American Statistical Association; 1990.
53. Otero P, Pedernera F, Montenegro S, Borbolla D, Garcia Marti S, Luna D, et al. Evolution of medical informatics in bibliographic databases. Stud Health Technol Inform 2004;107(Pt 1):301-305. [Medline: 15360823].
54. Sittig D. Identifying a core set of medical informatics serials: an analysis using the MEDLINE database. Bull Med Libr Assoc 1996 Apr;84(2):200-204. [Medline: 8826624].
55. Greenes R, Siegel E. Characterization of an emerging field: approaches to defining the literature and disciplinary boundaries of medical informatics. Proc Annu Symp Comput Appl Med Care 1987 Nov 4:411-415.
56. Raghupathi W, Nerur S. Research themes and trends in health information systems. Methods Inf Med 2008;47(5):435-442. [Medline: 18852917].
57. Sittig D, Kaalaas-Sittig J. A quantitative ranking of the Biomedical Informatics serials. Methods Inf Med 1995 Sep;34(4):397-340. [Medline: 7476471].
58. Andrews J. An author co-citation analysis of medical informatics. J Med Libr Assoc 2003 Jan;91(1):47-56. [Medline: 12568157].
59. Glass R. An assessment of systems and software engineering scholars and institutions. J Syst Softw 1994;27(1):63-67.
60. Glass R. An assessment of systems and software engineering scholars and institutions (1993-1997). J Syst Softw 1998;43(1):59-64. [doi: 10.1016/S0164-1212(98)10037-7].
61. Wong W, Tse T, Glass R, Basili V, Chen T. An assessment of systems and software engineering scholars and institutions (2003-2007 and 2004-2008). J Syst Softw 2011;84(1):162-168. [doi: 10.1016/j.jss.2010.09.036].
62. Ren J, Taylor R. Automatic and versatile publications ranking for research institutions and scholars. Commun ACM 2007;50(6):81-85. [doi: 10.1145/1247001.1247010].
63. Geist R, Chetuparambil M, Hedetniemi S, Turner A. Computing research programs in the U.S. Commun ACM 1996;39(12):96-99. [doi: 10.1145/240483.240505].
64. Glass R, Ramesh V, Vessey I. An analysis of research in computing disciplines. Communications of the ACM 2004;47(6):89-94. [doi: 10.1145/990680.990686].
65. Bornmann L. Daniel H-D. Selecting scientific excellence through committee peer review: a citation analysis of publications previously published to approval or rejection of post-doctoral research fellowship applicants. Scientometrics 2006;68:427-440. [doi: 10.1007/s11192-006-0121-1].
66. Endler N. Where the "stars" are: the 25 most cited psychologists in Canada (1972-1976). Can Psychol Rev 1979;20(1):12-21. [doi: 10.1037/h0081489].
67. Jokic M. Scientometric evalution of the projects in biology funded by the Ministry of Science and Technology, Republic of Croatia, in the 1991-1996 period. Period Biol 2000;102:129-142.
68. Tutarel O. Geographical distribution of publications in the field of medical education. BMC Med Educ 2002 Apr 30;2:3 [FREE Full text] [Medline: 12031092].
69. Opthof T, Wilde A. The Hirsch-index: a simple, new tool for the assessment of scientific output of individual scientists: The case of Dutch professors in clinical cardiology. Neth Heart J 2009 Apr;17(4):145-154. [Medline: 19421360].
70. Garfield E. Citation Indexing: Its Theory and Applications in Science, Technology, and Humanities. New York, NY: Wiley; 1979.
71. Schubert A, Braun T. Cross-field normalizations of scientometric indicators. Scientometrics 1996;36(3):311-324. [doi: 10.1007/BF02129597].
72. National Science Foundation. 2012 Jan. Science and Engineering Indicators 2012 URL: http://www.nsf.gov/statistics/seind12/appendix.htm [accessed 2012-10-08] [WebCite Cache ID 6BGaKExEu].
73. Scopus. 2009 Mar. Frequently Asked Questions: Author Identifier URL: http://www.info.sciverse.com/documents/files/scopus-training/resourcelibrary/ pdf/FAQ-Author-Identifier-09.pdf [accessed 2012-10-08] [WebCite Cache ID 6BGaL0yc6].
74. Jacso P. Testing the Calculation of a Realistic h-index in Google Scholar, Scopus, Web of Science for F. W. Lancaster. Libr Trends 2008;56(4):784-815. [doi: 10.1353/lib.0.0011].
75. Meho L, Yang K. Impact of data sources on citation counts and rankings of LIS faculty: Web of Science versus Scopus and Google Scholar. J Am Soc Inf Sci Technol 2007;58(13):2105-2125. [doi: 10.1002/asi.v58:13].
76. Jacso P. The pros and cons of computing the h-index using Google Scholar. Online Inf Rev 2008;32(3):437-452.
77. Sobel M, Huyett M. Nonparametric definition of the representativeness of a sample: with tables. Bell Syst Tech J 1958;37.
78. Woodruff R. Confidence intervals for medians and other position measures. J Am Stat Assoc 1952;47(260):635-646.
79. Francisco C, Fuller W. Quantile estimation with a complex survey design. Ann Stat 1991;19(1):454-469.
80. Dorfman A, Valliant R. Quantile Variance Estimators in Complex Surveys. Alexandria, VA: American Statistical Association; 1993.
81. Sitter R, Wu C. A note on Woodruff confidence intervals for quantiles. Stat Probab Lett 2001;52(4):353-358.
82. Chatterjee A. Asymptotic properties of sample quantiles from a finite population. Ann Inst Stat Math 2008;63:157-179.
83. Kosmulski M. Calibration against a reference set: a quantitative approach to assessment of the methods of assessment of scientific output. J Informetr 2012;6(3):451-456. [doi: 10.1016/j.joi.2012.03.004].
84. US National Library of Medicine. 2012 Apr 13. NLM's University-Based Biomedical Informatics Research Training Programs URL: http://www.nlm.nih.gov/ ep/GrantTrainInstitute.html [accessed 2012-10-08] [WebCite Cache ID 6BGHoDbwm].
85. Wikipedia. 2012 Aug 31. Open Directory Project URL: http://en.wikipedia. org/wiki/Open-Directory-Project [accessed 2012-10-08] [WebCite Cache ID 6BGHpoStc].
86. Bornmann L, Marx W, Schier H, Rahm E, Thor A, Daniel H. Convergent validity of bibliometric Google Scholar data in the field of chemistry: citation counts for papers that were accepted by Angewandte Chemie International Edition or rejected but published elsewhere, using Google Scholar, Science Citation Index, Scopus, and Chemical Abstracts. J Informetr 2009;3(1):27-35. [doi: 10.1016/j.joi.2008.11.001].