Metrics to evaluate research performance in academic institutions: A critique of ERA 2010 as applied in forestry and the indirect H 2 index as a possible alternative

Scientometrics

Volumn 91, Issue 3, 2012, Pages 751-771

  Vanclay, Jerome K ^a   Bornmann, Lutz ^b  

^a SOUTHERN CROSS UNIVERSITY   (Australia)

^b MAX PLANCK INSTITUTE   (Germany)

Author keywords

Excellence for Research in Australia (ERA); h index; Percentile; Successive h index

Indexed keywords

EID: 84860572565     PISSN: 01389130     EISSN: None     Source Type: Journal    
DOI: 10.1007/s11192-012-0618-8     Document Type: Article

References (58)

1. Abramo, G., & D'Angelo, C. A. (2011). Evaluating research: from informed peer review to Bibliometrics. Scientometrics, 87, 499-514.
2. Abramo, G., D'Angelo, C. A., & Caprasecca, A. (2009). Allocative efficiency in public research funding: Can bibliometrics help? Research Policy, 38(1), 206-215.
3. Alonso, S., Cabrerizo, F. J., Herrera-Viedma, E., & Herrera, F. (2009). h-Index: A review focused in its variants, computation and standardization for different scientific fields. Journal of Informetrics, 3, 273-289.
4. Anon. (2011). List of forestry universities and colleges. Wikipedia, http://en. wikipedia. org/wiki/List_of_forestry_universities_and_colleges [August 5, 2011].
5. ANZSRC (2008). Australian and New Zealand Standard Research Classification. http://www. abs. gov. au/Ausstats/abs@. nsf/0/6BB427AB9696C225CA2574180004463E [August 5, 2011].
6. ARC (2010a). The Excellence in Research for Australia (ERA) Initiative. http://www. arc. gov. au/era/default. htm [August 5, 2011].
7. ARC (2010b). Ranked Journal List. Australian Research Council. http://www. arc. gov. au/xls/ERA2010_journal_title_list. xls [August 5, 2011].
8. ARC (2011a). What is the change to the journal indicator? Australian Research Council. http://www. arc. gov. au/era/faq. htm#2012 [August 5, 2011].
9. ARC (2011b). ERA 2010 National Report. Australian Research Council, p. 320. http://www. arc. gov. au/pdf/ERA_report. pdf [August 5, 2011].
10. ARC (2011c). ERA 2010 citation benchmark methodology. Australian Research Council, p. 21. http://www. arc. gov. au/pdf/era_2010_citation_benchmark_methods. pdf [August 5, 2011].
11. ARC (2011d). ERA 2010 results by field of research code. Australian Research Council. http://www. arc. gov. au/era/outcomes_2010/FoRindex [August 5, 2011].
12. Arencibia-Jorge, R., Barrios-Almaguer, I., Fernández-Hernández, S., & Carvajal-Espino, R. (2008). Successive H indices and its applying in the institutional evaluation: a case study. Journal of the American Society for Information Science and Technology, 59(1), 155-157.
13. Atkinson, R. & McLoughlin, C. (2011). Dawn of a new ERA? Australasian Journal of Educational Technology, 27(3), iii-viii. http://ascilite. org. au/ajet/ajet27/editorial27-3. html.
14. Bornmann, L. (2010). Mimicry in science? Scientometrics, 86(1), 173-177.
15. Bornmann, L. (2011). Scientific peer review. Annual Review of Information Science and Technology, 45, 199-245.
16. Bornmann, L., de Moya Anegón, F., & Leydesdorff, L. (2010). Do scientific advancements lean on the shoulders of giants? A bibliometric investigation of the Ortega hypothesis. PLoS ONE, 5(10), e13327. doi: 10. 1371/journal. pone. 0013327.
17. Bornmann, L., Mutz, R., & Daniel, H.-D. (2008). Are there better indices for evaluation purposes than the h index? A comparison of nine different variants of the h index using data from Biomedicine. Journal of the American Society for Information Science and Technology, 59(5), 830-837.
18. Bornmann, L., Mutz, R., Hug, S. E., & Daniel, H.-D. (2011a). A multilevel meta-analysis of studies reporting correlations between the h index and 37 different h index variants. Journal of Informetrics, 5, 346-359.
19. Bornmann, L., Schier, H., Marx, W., & Daniel, H.-D. (2011b). Does the h index for assessing single publications really work? A case study on papers published in chemistry. Scientometrics, 89(3), 835-843.
20. Carr, K. (2009). A message from the Minister. Australian Research Council, Discovery Newsletter, Autumn 2009. http://www. arc. gov. au/rtf/discovery_autumn09. rtf.
21. Cooper, S., & Poletti, A. (2011). The new ERA of journal ranking. Australian Universities' Review, 53, 57-65.
22. Donovan, C., & Butler, L. (2007). Testing novel quantitative indicators of research 'quality', esteem and 'user engagement': An economics pilot study. Research Evaluation, 16(4), 231-242.
23. Egghe, L. (2008). Modelling successive h-indices. Scientometrics, 77(3), 377-387.
24. Egghe, L. (2011). The single publication H-index and the indirect H-index of a researcher. Scientometrics, 88(3), 1003-1004.
25. Falagas, M. E., Pitsouni, E. I., Malietzis, G. A., & Pappas, G. (2008). Comparison of PubMed, Scopus, Web of Science, and Google Scholar: strengths and weaknesses. The FASEB Journal, 22, 338-342.
26. Haddow, G., & Genoni, P. (2010). Citation analysis and peer ranking of Australian social science journals. Scientometrics, 85, 471-487.
27. Hicks, D. (2009). Evolving regimes of multi-university research evaluation. Higher Education, 57, 393-404.
28. Hirsch, J. E. (2005). An index to quantify an individual's scientific research output. Proceedings of the National Academy of Sciences, 102, 16569-16572.
29. Jones, D. (2011). Excellence in Research for Australia-a new ERA. Research Trends23, 10-11. http://www. researchtrends. com/wp-content/uploads/2011/06/Research_Trends_Issue23. pdf.
30. Laband, D. N., & Zhang, D. (2006). Citations, publications, and perceptions-based rankings of the research impact of North American forestry programs. Journal of Forestry, 104(5), 162-254.
31. Labbé, C. (2010). Ike Antkare, one of the great stars in the scientific firmament. ISSI Newsletter, 6(2), 48-52.
32. Lamp, J. (2011). 2010 finalised journals in an ANZ Field of Research: 0705 Forestry Sciences. http://lamp. infosys. deakin. edu. au/era/?page=jfordet10& selfor=0705 [5 August 2011].
33. Leydesdorff, L., Bornmann, L., Mutz, R., & Opthof, T. (2011). Turning the tables in citation analysis one more time: principles for comparing sets of documents. Journal of the American Society of Information Science and Technology, 62(7), 1370-1381.
34. Li, J., Burnham, J. F., Lemley, T., & Britton, R. M. (2010). Citation Analysis: Comparison of Web of Science, Scopus, SciFinder, and Google Scholar. Journal of Electronic Resources in Medical Libraries, 7(3), 196-217.
35. Meho, L. I., & Yang, K. (2007). Impact of data sources on citation counts and rankings of LIS faculty: Web of science versus scopus and google scholar. Journal of the American Society for Information Science and Technology, 58, 2105-2125.
36. Moed, H. F. (2009). New developments in the use of citation analysis in research evaluation. Archivum Immunologiae et therapiae Experimentalis, 57, 13-18.
37. Moosa, I. (2011). The demise of the ARC journal ranking scheme: an ex post analysis of the accounting and finance journals. Accounting & Finance, 51(3), 809-836.
38. Moya-Anegón, F. D., Chinchilla-Rodríguez, Z., Vargas-Quesada, B., Corera-Álvarez, E., Muñoz-Fernández, F. J., González-Molina, A., et al. (2007). Coverage analysis of Scopus: A journal metric approach. Scientometrics, 73, 53-78.
39. Norris, M., & Oppenheim, C. (2010). The h-index: a broad review of a new bibliometric indicator. Journal of Documentation, 66(5), 681-705.
40. Northcott, D., & Linacre, S. (2010). Producing spaces for academic discourse: The impact of research assessment exercises and journal quality rankings. Australian Accounting Review, 20, 38-54.
41. Pontille, D., & Torny, D. (2010). The controversial policies of journal ratings: Evaluating social sciences and humanities. Research Evaluation, 15, 347-360.
42. Radicchi, F., & Castellano, C. (2011). Rescaling citations of publications in physics. Physical Review E, 83(4), 6. doi: 10. 1103/PhysRevE. 83. 046116.
43. Rousseau, R., Yang, L., & Yue, T. (2010). A discussion of Prathap's h2-index for institutional evaluation with an application in the field of HIV infection and therapy. Journal of Informetrics, 4(2), 175-184.
44. Ruane, F., & Tol, R. S. J. (2008). Rational (successive) h-indices: An application to economics in the Republic of Ireland. Scientometrics, 75(2), 395-405.
45. Runeson, G. (2011). The demise of the journal ranking: a victory for common sense. Australasian Journal of Construction Economics and Building, 11(2), 99-100.
46. Schreiber, M. (2010). Twenty Hirsch index variants and other indicators giving more or less preference to highly cited papers. Annalen der Physik, 522(8), 536-554.
47. Schubert, A. (2007). Successive h-indices. Scientometrics, 70(1), 201-205.
48. Schubert, A. (2009). Using the h-index for assessing single publications. Scientometrics, 78(3), 559-565.
49. Serenko, A., & Dohan, M. (2011). Comparing the expert survey and citation impact journal ranking methods: Example from the field of Artificial Intelligence. Journal of Informetrics, 5(4), 629-648.
50. Sinclair, R. (2010). RD& E strategy for the forest and wood products sector. Australia: Forest and Wood Products, p. 57. www. daff. gov. au/__data/assets/word_doc/0010/1770751/forest. doc.
51. Singh, G., Haddad, K. M., & Chow, C. W. (2007). Are articles in "top" management journals necessarily of higher quality? Journal of Management Inquiry, 16(4), 319-331.
52. Slyder, J. B., Stein, B. R., Sams, B. S., Walker, D. M., Beale, B. J., Feldhaus, J. J., et al. (2011). Citation pattern and lifespan: A comparison of discipline, institution and individual. Scientometrics, 89(3), 955-966.
53. Svantesson, D. J. B., & White, P. (2009). Entering an era of research ranking-will innovation and diversity survive? Bond Law Review, 21(3), 7.
54. Taylor, J. (2011). The assessment of research quality in UK universities: Peer review or metrics? British Journal of Management, 22, 202-217.
55. Thor, A. & Bornmann, L. (2011). The calculation of the single publication h index and related performance measures: A web application based on Google Scholar data. Online Information Review, 35(2), 291-300. (Utility available here: http://labs. dbs. uni-leipzig. de/gsh/).
56. Vanclay, J. K. (2008). Ranking forestry journals using the h-index. Journal of Informetrics, 2, 326-334.
57. Vanclay, J. K. (2011). An evaluation of the Australian Research Council's journal ranking. Journal of Informetrics, 5, 265-274.
58. Vanclay, J. K. (2012). What was wrong with Australia's journal ranking? Journal of Informetrics, 6, 53-54.