Expert-based versus citation-based ranking of scholarly and scientific publication channels

Journal of Informetrics

Volumn 10, Issue 3, 2016, Pages 693-718

  Saarela, Mirka ^a   Kärkkäinen, Tommi ^a   Lahtonen, Tommi ^a   Rossi, Tuomo ^a  

^a UNIVERSITY OF JYVÄSKYLÄ   (Finland)

Author keywords

Finnish ranking system; Journal citation reports; Journal evaluation; Performance based funding; Scopus

Indexed keywords

DATA MINING; DECISION TREES;

JOURNAL CITATION REPORTS; JOURNAL EVALUATION; PERFORMANCE BASED; RANKING SYSTEM; SCOPUS;

PUBLISHING;

EID: 84971330320     PISSN: 17511577     EISSN: 18755879     Source Type: Journal    
DOI: 10.1016/j.joi.2016.03.004     Document Type: Article

References (47)

1. Abramo G., D'Angelo C.A. Evaluating university research: Same performance indicator, different rankings. Journal of Informetrics 2015, 9(3):514-525. http://www.sciencedirect.com/science/article/pii/S1751157715000462.
2. Agrawal R., Imieliński T., Swami A. Mining association rules between sets of items in large databases. ACM SIGMOD Record. Vol. 22 1993, 207-216. ACM. http://www.almaden.ibm.com/cs/quest/papers/sigmod93.pdf.
3. Ahlgren P., Colliander C., Persson O. Field normalized citation rates, field normalized journal impact and Norwegian weights for allocation of university research funds. Scientometrics 2012, 92(3):767-780. 10.1007/s11192-012-0632-x.
4. Ahlgren P., Waltman L. The correlation between citation-based and expert-based assessments of publication channels: SNIP and SJR vs. Norwegian quality assessments. Journal of Informetrics 2014, 8(4):985-996. http://www.sciencedirect.com/science/article/pii/S1751157714000911.
5. Alpaydin E. Introduction to Machine Learning 2010, The MIT Press, Cambridge, MA, USA. https://mitpress.mit.edu/books/introduction-machine-learning, 2nd ed.
6. Archambault É., Larivière V. History of the journal impact factor: Contingencies and consequences. Scientometrics 2009, 79(3):635-649. 10.1007/s11192-007-2036-x.
7. Auranen O., Nieminen M. University research funding and publication performance - An international comparison. Research Policy 2010, 39(6):822-834. http://www.sciencedirect.com/science/article/pii/S0048733310000764.
8. Bergstrom C.T., West J.D., Wiseman M.A. The Eigenfactor metrics. Journal of Neuroscience 2008, 28(45):11433-11434. http://www.jneurosci.org/content/28/45/11433.short.
9. Bollen J., Van de Sompel H., Hagberg A., Chute R. A principal component analysis of 39 scientific impact measures. PLoS ONE 2009, 4(6):e6022. 10.1371/journal.pone.0006022.
10. Bornmann L., Marx W., Schier H., Rahm E., Thor A., Daniel H.-D. Convergent validity of bibliometric Google Scholar data in the field of chemistry - Citation counts for papers that were accepted by Angewandte Chemie International Edition or rejected but published elsewhere, using Google Scholar, Science Citation Index, Scopus, and Chemical Abstracts. Journal of Informetrics 2009, 3(1):27-35. http://www.sciencedirect.com/science/article/pii/S1751157708000667.
11. Breiman L., Friedman J., Stone C.J., Olshen R.A. Classification and regression trees 1984, CRC press.
12. Bryman A. Triangulation. The SAGE encyclopedia of social science research methods 2004, 1143-1144. Sage Publications, Inc. 10.4135/9781412950589.
13. Butler L. Explaining Australias increased share of {ISI} publications the effects of a funding formula based on publication counts. Research Policy 2003, 32(1):143-155. http://www.sciencedirect.com/science/article/pii/S0048733302000070.
14. Cattaneo M., Meoli M., Signori A. Performance-based funding and university research productivity: The moderating effect of university legitimacy. Journal of Technology Transfer 2014, 41(1):85-104. 10.1007/s10961-014-9379-2.
15. Chang C.-L., McAleer M., Oxley L. Coercive journal self citations, impact factor. Journal Influence and Article Influence, Mathematics and Computers in Simulation 2013, 93:190-197. http://www.sciencedirect.com/science/article/pii/S0378475413000694.
16. Danish Centre for Studies in Research and Research Policy Evaluation of the Norwegian publication indicator - English summary. Tech. rep. 2014, Danish Centre for Studies in Research and Research Policy. http://www.uhr.no/documents/Evaluation_of_the_Norwegian_Publication_Indicator___English_Summary.pdf.
17. Fairclough R., Thelwall M. More precise methods for national research citation impact comparisons. Journal of Informetrics 2015, 9(4):895-906. http://www.sciencedirect.com/science/article/pii/S1751157715300894.
18. Falagas M.E., Kouranos V.D., Arencibia-Jorge R., Karageorgopoulos D.E. Comparison of SCImago journal rank indicator with journal impact factor. FASEB journal 2008, 22(8):2623-2628. http://www.ncbi.nlm.nih.gov/pubmed/18408168.
19. Franceschet M. The difference between popularity and prestige in the sciences and in the social sciences: A bibliometric analysis. Journal of Informetrics 2010, 4(1):55-63. http://www.sciencedirect.com/science/article/pii/S1751157709000698.
20. Garfield E. Citation analysis as a tool in journal evaluation. Science 1972, 471-479. American Association for the Advancement of Science. http://www.elshami.com/Terms/I/impact%20factor-Garfield.pdf.
21. Giovanni A., Tindaro C., D'Angelo C.A. Are the authors of highly cited articles also the most productive ones?. Journal of Informetrics 2014, 8(1):89-97. http://www.sciencedirect.com/science/article/pii/S1751157713000886.
22. González-Pereira B., Guerrero-Bote V.P., Moya-Anegón F. A new approach to the metric of journals' scientific prestige: The SJR indicator. Journal of Informetrics 2010, 4(3):379-391. http://www.sciencedirect.com/science/article/pii/S1751157710000246.
23. Good B., Vermeulen N., Tiefenthaler B., Arnold E. Counting quality? The Czech performance-based research funding system. Research Evaluation 2015, 24(2):91-105.
24. Guerrero-Bote V.P., Moya-Anegón F. A further step forward in measuring journals' scientific prestige: The SJR2 indicator. Journal of Informetrics 2012, 6(4):674-688. http://www.sciencedirect.com/science/article/pii/S1751157712000521.
25. Haustein S., Larivière V. The use of bibliometrics for assessing research: possibilities, limitations and adverse effects. Incentives and performance 2015, 121-139. Springer. http://link.springer.com/chapter/10.1007%2F978-3-319-09785-5_8.
26. He H., Garcia E., et al. Learning from imbalanced data. IEEE Transactions on Knowledge and Data Engineering 2009, 21(9):1263-1284. 10.1109/TKDE.2008.239.
27. Hicks D. Performance-based university research funding systems. Research Policy 2012, 41(2):251-261. http://www.sciencedirect.com/science/article/pii/S0048733311001752.
28. Kärkkäinen T., Saarela M. Robust principal component analysis of data with missing values. Machine learning and data mining in pattern recognition 2015, 140-159. Springer. http://link.springer.com/chapter/10.1007%2F978-3-319-21024-7_10.
29. Kreiman G., Maunsell J. Nine criteria for a measure of scientific output. Frontiers in Computational Neuroscience 2011, 5:48. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3214728/.
30. Kuroda M., Mori Y., Masaya I., Sakakihara M. Alternating least squares in nonlinear principal components. Wiley Interdisciplinary Reviews: Computational Statistics 2013, 5(6):456-464. 10.1002/wics.1279.
31. Moed H.F. Citation analysis in research evaluation 2005, Springer, The Netherlands. http://www.springer.com/us/book/9781402037139.
32. Moed H.F. Measuring contextual citation impact of scientific journals. Journal of Informetrics 2010, 4(3):265-277. http://www.sciencedirect.com/science/article/pii/S1751157710000039.
33. Pruvot E.B., Claeys-Kulik A.-L., Estermann T. Strategies for efficient funding of universities in Europe. The European higher education area: Between critical reflections and future policies 2015, 153-168. Springer International Publishing, Cham. 10.1007/978-3-319-20877-0_11. A. Curaj, L. Matei, R. Pricopie, J. Salmi, P. Scott (Eds.).
34. Puuska H.-M. Scholarly publishing patterns in Finland - A comparison of disciplinary groups, (Ph.D. thesis) 2014, Tampere University Press. https://tampub.uta.fi/bitstream/handle/10024/95381/978-951-44-9480-2.pdf?sequence=1.
35. Saarela M., Kärkkä inen T. Analysing student performance using sparse data of core bachelor courses. JEDM-Journal of Educational Data Mining 2015, 7(1):3-32. http://www.educationaldatamining.org/JEDM/index.php/JEDM/article/view/JEDM056.
36. Schneider J.W. An outline of the bibliometric indicator used for performance-based funding of research institutions in Norway. European Political Science 2009, 8(3):364-378. 10.1057/eps.2009.19.
37. Schneider J.W., Aagaard K., Bloch C.W. What happens when national research funding is linked to differentiated publication counts? A comparison of the Australian and Norwegian publication-based funding models. Research Evaluation 2015, 1-13.
38. Seiler C., Wohlrabe K. How robust are journal rankings based on the impact factor? Evidence from the economic sciences. Journal of Informetrics 2014, 8(4):904-911. http://www.sciencedirect.com/science/article/pii/S1751157714000820.
39. Serenko A., Dohan M. Comparing the expert survey and citation impact journal ranking methods: Example from the field of artificial intelligence. Journal of Informetrics 2011, 5(4):629-648. http://www.sciencedirect.com/science/article/pii/S1751157711000666.
40. Sivertsen G. A performance indicator based on complete data for the scientific publication output at research institutions. ISSI Newsletter 2010, 6(1):22-28.
41. Sivertsen G. Scholarly publication patterns in the social sciences and humanities and their coverage in scopus and web of science. Proceedings of the science and technology indicators conference 2014, 598-604. Universiteit Leiden, Leiden. E. Noyons (Ed.).
42. Sivertsen G., Larsen B. Comprehensive bibliographic coverage of the social sciences and humanities in a citation index: an empirical analysis of the potential. Scientometrics 2012, 91(2):567-575. 10.1007/s11192-011-0615-3.
43. Vanclay J.K. An evaluation of the Australian Research Council's journal ranking. Journal of Informetrics 2011, 5(2):265-274. http://www.sciencedirect.com/science/article/pii/S1751157710000994.
44. Vanclay J.K. Impact factor: outdated artefact or stepping-stone to journal certification?. Scientometrics 2012, 32(2):211-238. 10.1007/s11192-011-0561-0.
45. 2 index as a possible alternative. Scientometrics 2012, 91(3):751-771.
46. Verleysen F.T., Ghesquière P., Engels T. The objectives, design and selection process of the Flemish academic bibliographic database for the social sciences and humanities (vabb-shw). Bibliometrics: Use and abuse in the review of research performance 2014, 117-127.
47. Wilsdon J., Allen L., Belfiore E., Campbell P., Curry S., Hill S., Jones R., Hill J., Kain R., Johnson B., et al. The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management. Tech. rep. 2015, Higher Education Funding Council for England. http://www.hefce.ac.uk/pubs/rereports/Year/2015/metrictide/Title,104463,en.html.