The emperor’s new clothes: the h-index as a guide to resource allocation in higher education

Journal of Higher Education Policy and Management

Volumn 36, Issue 5, 2014, Pages 456-470

  Barnes, Cameron ^a  

^a UNIVERSITY OF NEW ENGLAND   (Australia)

Author keywords

bibliometrics; h factor; h index; Hirsh index; measurement; research impact

Indexed keywords

EID: 84926178317     PISSN: 1360080X     EISSN: 14699508     Source Type: Journal    
DOI: 10.1080/1360080X.2014.936087     Document Type: Article

References (82)

1. Aarhus Universitet. (2013). Criteria for evaluating candidates for positions as full professor, associate professor, senior scientist, or senior advisor. Retrieved from http://scitech.medarbejdere.au.dk/fileadmin/site_files/scitech.medarbejdere.au.dk/files/Evaluation_Criteria.pdf
2. Abatemarco, A., & Dell’Anno, R. (2012). The Italian reform of the academic recruitment system: An appraisal of ANVUR and CUN benchmarks for assessing candidates and commissioners. Rivista Italiana Degli Economisti, 3, 441–480. doi:10.1427/38533
3. Abbott, A., Cyranoski, D., Jones, N., Maher, B., Schiermeier, Q., & Van Noorden, R. (2010). Metrics: Do metrics matter? Nature, 465, 860–862. doi:10.1038/465860a
4. Adler, R., Ewing, J., & Taylor, P. (2008). Citation statistics: A report from the international mathematical union (IMU) in cooperation with the international council of industrial and applied mathematics (ICIAM) and the institute of mathematical statistics (IMS). Berlin: International Mathematical Union.
5. Albion, P.R. (2012). Benchmarking citation measures among the Australian education professoriate. The Australian Educational Researcher, 39, 221–235. doi:10.1007/s13384-012-0060-1
6. Anderson, T.R., Hankin, R.K., & Killworth, P.D. (2008). Beyond the Durfee square: Enhancing the h-index to score total publication output. Scientometrics, 76, 577–588. doi:10.1007/s11192-007-2071-2
7. Batista, P.D., Campiteli, M.G., Kinouchi, O., & Martinez, A.S. (2006). Is it possible to compare researchers with different scientific interests? Scientometrics, 68, 179–189. doi:10.1007/s11192-006-0090-4
8. Bornmann, L., & Daniel, H.D. (2005). Does the h-index for ranking of scientists really work? Scientometrics, 65, 391–392. doi:10.1007/s11192-005-0281-4
9. Bornmann, L., & Daniel, H.D. (2009). The state of h index research. Is the h index the ideal way to measure research performance? EMBO Reports, 10, 2–6. doi:10.1038/embor.2008.233
10. Bornmann, L., & Marx, W. (2011). The h index as a research performance indicator. European Science Editing, 37(3), 77–80. Retrieved from http://www.lutz-bornmann.de/icons/viewpoints.pdf
11. Bornmann, L., Mutz, R., Hug, S.E., & Daniel, H. (2011). A multilevel meta-analysis of studies reporting correlations between the h index and 37 different h index variants. Journal of Informetrics, 5, 346–359. doi:10.1016/j.joi.2011.01.006
12. Bornmann, L., Wallon, G., & Ledin, A. (2008). Is the h index related to (standard) bibliometric measures and to the assessments by peers? An investigation of the h index by using molecular life sciences data. Research Evaluation, 17(2), 149–156. doi:10.3152/095820208X319166
13. Bradley, D., Noonan, P., Nugent, H., & Scales, B. (2008). Review of higher education in Australia, final report. Canberra: Australian Government.
14. Brand, A. (2013, January 22). Re: Faculty appointments and the record of scholarship, Retrieved from http://blogs.lse.ac.uk/impactofsocialsciences/2013/01/22/faculty-appointments-and-the-record-of-scholarship/
15. Burrell, Q. (2007). Should the h-index be discounted? In I.W. Glanzel, A. Schubert, & B. Schlemmer (Eds.), The multidimensional world of Tibor Braun: A multidisciplinary encomium for his 75th birthday (pp. 65–68). Leuven: ISSI.
16. Burrows, R. (2012). Living with the h-index? Metric assemblages in the contemporary academy. The Sociological Review, 60, 355–372. doi:10.1111/j.1467-954X.2012.02077.x
17. fa-index): Levelling the field for small and large institute medical and science scholars. Australasian Physical and Engineering Sciences in Medicine, 33, 299–300. doi:10.1007/s13246-010-0038-0
18. Cacioppo, J., & Cacioppo, S. (2012). Metrics of scholarly impact. Measurement: Interdisciplinary Research and Perspectives, 10, 154–156. doi:10.1080/15366367.2012.720194
19. Castillo, M. (2010). Measuring academic output: The H-index. American Journal of Neuroradiology, 31, 783–784. doi:10.3174/ajnr.A1888
20. Clarke, R. (2009). A citation analysis of Australian information systems researchers: Towards a new era? Australasian Journal of Information Systems, 15(2), 23–44. Retrieved from http://dl.acs.org.au/index.php/ajis/article/view/465/473
21. Corrall, S.K., Kennan, M.A., & Afzal, W. (2013). Bibliometrics and research data management services: Emerging trends in library support for research. Library Trends, 61, 636–674. doi:10.1353/lib.2013.0005
22. Costas, R., & Bordons, M. (2007). The h-index: Advantages, limitations and its relation with other bibliometric indicators at the micro level. Journal of Informetrics, 1, 193–203. doi:10.1016/j.joi.2007.02.001
23. Costas, R., & Bordons, M. (2008). Is g-index better than h-index? An exploratory study at the individual level. Scientometrics, 77, 267–288. doi:10.1007/s11192-007-1997-0
24. Cronin, B., & Meho, L. (2006). Using the h-index to rank influential information scientists. Journal of the American Society for Information Science and Technology, 57, 1275–1278. doi:10.1002/asi.20354
25. Czech Science Foundation. (2012). Manual for GACR standard projects. Retrieved from http://www.gacr.cz/wp-content/uploads/2012/03/Zadavaci-dok-GACR-2012-ST-final.pdf
26. Deem, R., Mok, K.H., & Lucas, L. (2008). Transforming higher education in whose image? Exploring the concept of the ‘worldclass’ university in Europe and Asia. Higher Education Policy, 21, 83–97. doi:10.1057/palgrave.hep.8300179
27. Department of Science and Technology. (2012). Annual Report 2010–2011. Retrieved from http://www.dst.gov.in/about_us/ar10-11/PDF/Chapter%202%20-%2074-91.pdf
28. Derrick, G.E., Haynes, A., Chapman, S., & Hall, W.D. (2011). The association between four citation metrics and peer rankings of research influence of Australian researchers in six fields of public health. PLoS One, 6(4), e18521. doi:10.1371/journal.pone.0018521
29. Drummond, R., & Wartho, R. (2009). RIMS: The research impact measurement service at the University of New South Wales. Australian Academic & Research Libraries, 40, 76–87. doi:10.1080/00048623.2009.10721387
30. Egghe, L. (2010). The Hirsch index and related impact measures. Annual Review of Information Science and Technology, 44, 65–114. doi:10.1002/aris.2010.1440440109
31. Ellison, G. (2010). How does the market use citation data? The Hirsch index in economics (NBER Working Paper 16419). Cambridge, MA: National Bureau of Economic Research.
32. European Commission. (2012). IT4BI-DC application manual V 1.0. Retrieved from http://it4bi-dc.ulb.ac.be/emundus/images/stories/provided_documents/it4bi-dc_application_manual.pdf
33. Evidence Ltd. (2007). The use of bibliometrics to measure research quality in UK higher education institutions. London: Universities UK. Retrieved from http://www.universitiesuk.ac.uk/highereducation/Documents/2007/Bibliometrics.pdf
34. Franceschini, F., & Maisano, D.A. (2010). Analysis of the Hirsch index’s operational properties. European Journal of Operational Research, 203, 494–504. doi:10.1016/j.ejor.2009.08.001
35. Gadd, E. (2011). Citations count: The provision of bibliometrics training by university libraries. SCONUL Focus, 52, 11–13. Retrieved from http://www.sconul.ac.uk/sites/default/files/documents/5_2.pdf
36. Gaster, N., & Gaster, M. (2012). A critical assessment of the h-index. Bioessays: News and Reviews in Molecular, Cellular and Developmental Biology, 34, 830–832. doi:10.1002/bies.201200036
37. Hendrix, D. (2010). Tenure metrics: Bibliometric education and services for academic faculty. Medical Reference Services Quarterly, 29, 183–189. doi:10.1080/02763861003723416
38. Hirsch, J.E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences, 102, 16569–16572. doi:10.1073/pnas.0507655102
39. Hirsch, J.E. (2007). Does the h index have predictive power? Proceedings of the National Academy of Sciences, 104, 19193–19198. doi:10.1073/pnas.0707962104
40. Hirsch, J.E. (2010). An index to quantify an individual’s scientific research output that takes into account the effect of multiple coauthorship. Scientometrics, 85, 741–754. doi:10.1007/s11192-010-0193-9
41. Home, R., Petrie, K., & Wessely, S. (2009). H-index pathology: Implications for medical researchers and practitioners. British Medical Journal, 339, 1447–1448. doi:10.1136/bmj.b5356
42. Hönekopp, J., & Khan, J. (2012). Future publication success in science is better predicted by traditional measures than by the h index. Scientometrics, 90, 843–853. doi:10.1007/s11192-011-0551-2
43. Ireland, T., Macdonald, K., & Sterling, P. (2012). The H-index: What is it, how do we determine it, and how can we keep up with it? In I.A. Tokar, M. Beurskens, S. Keuneke, M. Mahrt, I. Peters, C. Puschmann, … K. Weller (Eds.), Science and the internet (pp. 237–247). Düsseldorf: Düsseldorf University Press.
44. Jacsó, P. (2008). The plausibility of computing the h-index of scholarly productivity and impact using reference-enhanced databases. Online Information Review, 32, 266–283. doi:10.1108/14684520810879872
45. Jensen, P., Rouquier, J.B., & Croissant, Y. (2009). Testing bibliometric indicators by their prediction of scientists promotions. Scientometrics, 78, 467–479. doi:10.1007/s11192-007-2014-3
46. Jin, B., Liang, L., Rousseau, R., & Egghe, L. (2007). The R- and AR-indices: Complementing the h-index. Chinese Science Bulletin, 52(6), 855–863. doi:10.1007/s11434-007-0145-9
47. Jung, I., Wong, T.M., Li, C., Baigaltugs, S., & Belawati, T. (2011). Quality assurance in Asian distance education: Diverse approaches and common culture. International Review of Research in Open and Distance Learning, 12(6), 63–83. Retrieved from http://www.editlib.org/p/54576
48. Kear, R., & Colbert-Lewis, D. (2011). Citation searching and bibliometric measures: Resources for ranking and tracking. College & Research Libraries News, 72, 470–474. Retrieved from http://crln.acrl.org/content/72/8/470.full.pdf
49. Kellner, A., & Ponciano, L. (2008). H-index in the Brazilian academy of sciences: Comments and concerns. Anais da Academia Brasileira de Ciências, 80(4), 771–781. doi:10.1590/S0001-37652008000400016
50. Kelly, C.D., & Jennions, M.D. (2006). The h-index and career assessment by numbers. Trends in Ecology and Evolution, 21(4), 167–170. doi:10.1016/j.tree.2006.01.005
51. Kleis, R. (2012, January 12). Measuring up: Academia adopts the h-index. Resource, 12–15. Retrieved from http://edepot.wur.nl/193112
52. Kotov, N.A. (2010). Fraud, the h-index, and Pasternak. ACS Nano, 4, 585–586. doi:10.1021/nn100182y
53. Lehmann, S., Jackson, A., & Lautrup, B. (2008). A quantitative analysis of indicators of scientific performance. Scientometrics, 76, 369–390. doi:10.1007/s11192-007-1868-8
54. Li, J.A., Sanderson, M., Willett, P., Norris, M., & Oppenheim, C. (2010). Ranking of library and information science researchers: Comparison of data sources for correlating citation data, and expert judgments. Journal of Informetrics, 4, 554–563. doi:10.1016/j.joi.2010.06.005
55. Lovegrove, B., & Johnson, S. (2008). Assessment of research performance in biology: How well do peer review and bibliometry correlate? Bioscience, 58, 160–164. doi:10.1641/B580210
56. Norris, M., & Oppenheim, C. (2010a). The h-index: A broad review of a new bibliometric indicator. Journal of Documentation, 66, 681–705. doi:10.1108/00220411011066790
57. Norris, M., & Oppenheim, C. (2010b). Peer review and the h-index: Two studies. Journal of Informetrics, 4, 221–232. doi:10.1016/j.joi.2009.11.001
58. Ohio State University. (2012). College of Medicine: Appointments, promotion and tenure document. Retrieved from http://oaa.osu.edu/assets/files/documents/MED-APT.pdf
59. Pratelli, L., Baccini, A., Barabesi, L., & Marcheselli, M. (2012). Statistical analysis of the Hirsch index. Scandinavian Journal of Statistics, Theory and Applications, 39, 681–694. doi:10.1111/j.1467-9469.2011.00782.x
60. Rousseau, R. (2008). Reflections on recent developments of the h-index and h-type indices. COLLNET Journal of Scientometrics and Information Management, 2(1), 1–8. doi:10.1080/09737766.2008.10700835
61. Saad, G. (2006). Exploring the h-index at the author and journal levels using bibliometric data of productive consumer scholars and business-related journals respectively. Scientometrics, 69, 117–120. doi:10.1007/s11192-006-0142-9
62. Saad, G. (2010). Applying the h-index in exploring bibliometric properties of elite marketing scholars. Scientometrics, 83, 423–433. doi:10.1007/s11192-009-0069-z
63. m modifies h for multi-authored manuscripts. New Journal of Physics, 10(4), 1–9. doi:10.1088/1367-2630/10/4/040201
64. Schreiber, M. (2013). How relevant is the predictive power of the h-index? A case study of the time-dependent Hirsch index. Retrieved from http://arxiv.org/pdf/1301.2060
65. Schreiber, M., Malesios, C.C., & Psarakis, S. (2012). Exploratory factor analysis for the Hirsch index, 17 h-type variants, and some traditional bibliometric indicators. Journal of Informetrics, 6, 347–358. doi:10.1016/j.joi.2012.02.001
66. Seglen, P.O. (1992). The skewness of science. Journal of the American Society for Information Science, 43, 628–638. doi:10.1002/(SICI)1097-4571(199210)43:9<628::AID-ASI5>3.0.CO;2-0
67. Smith, K.M., Crookes, E., & Crookes, P.A. (2013). Measuring research ‘impact’ for academic promotion: Issues from the literature. Journal of Higher Education Policy and Management, 35, 410–420. doi:10.1080/1360080X.2013.812173
68. Stephen, P. (2012). How economics shapes science. Cambridge, MA: Harvard University Press.
69. Universiteit Utrecht. (2012). Criteria for the position of associate professor (UHD). Retrieved from http://www.uu.nl/SiteCollectionDocuments/Beta/Diensten/PenO/
70. Universiti Malaya. (2013). Academic Icon Unit. Retrieved from http://www.um.edu.my/mainpage.php?module=Maklumat&kategori=71&id=911&papar=1
71. University of Queensland. (2012). Guidelines for evidencing academic achievement. Retrieved from http://www.uq.edu.au/shared/resources/personnel/appraisalAcad/guidelines-evidencing-academic-achievement.pdf
72. University of Surrey. (2012). Promotion procedures for academic staff. Retrieved from http://www.surrey.ac.uk/about/corporate/policies/promotion_procedures_for_academic_staff.pdf
73. Vanclay, J.K. (2007). On the robustness of the h-index. Journal of the American Society for Information Science and Technology, 58, 1547–1550. doi:10.1002/asi.20616
74. 2 index as a possible alternative. Scientometrics, 91(3), 751–771. doi:10.1007/s11192-012-0618-8
75. van Leeuwen, T. (2008). Testing the validity of the Hirsch-index for research assessment purposes. Research Evaluation, 17(2), 157–160. doi:10.3152/095820208X319175
76. van Raan, A.F.J. (2006). Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups. Scientometrics, 67, 491–502. doi:10.1556/Scient.67.2006.3.10
77. Vinkler, P. (2007). Eminence of scientists in the light of the h-index and other scientometric indicators. Journal of Information Science, 33, 481–491. doi:10.1177/0165551506072165
78. Waltman, L., Costas, R., & Van Eck, N.J. (2012). Some limitations of the H index: A commentary on Ruscio and colleagues’ analysis of bibliometric indices. Measurement: Interdisciplinary Research and Perspectives, 10, 172–175. doi:10.1080/15366367.2012.716260
79. Waltman, L., & Van Eck, N.J. (2012). The inconsistency of the h-index. Journal of the American Society for Information Science and Technology, 63, 406–415. doi:10.1002/asi.21678
80. Williamson, J.R. (2009). My h-index turns 40: My midlife crisis of impact. ACS Chemical Biology, 4, 311–313. doi:10.1021/cb9001014
81. Zare, R.N. (2012). Editorial: Assessing academic researchers. Angewandte Chemie International Edition, 51, 7338–7339. doi:10.1002/anie.201201011
82. Zhivotovsky, L.A., & Krutovsky, K.V. (2008). Self-citation can inflate h-index. Scientometrics, 77(2), 373–375. doi:10.1007/s11192-006-1716-2