Bibliometric evaluation vs. informed peer review: Evidence from Italy

Research Policy

Volumn 44, Issue 2, 2015, Pages 451-466

  Bertocchi, Graziella ^a   Gambardella, Alfonso ^b   Jappelli, Tullio ^c   Nappi, Carmela A ^d   Peracchi, Franco ^e  

^a UNIVERSITY OF MODENA AND REGGIO EMILIA   (Italy)

^b BOCCONI UNIVERSITY   (Italy)

^c UNIVERSITY OF NAPLES FEDERICO II   (Italy)

^d Responsable for National Scientific Habilitation   (Italy)

^e UNIVERSITY OF ROME TOR VERGATA   (Italy)

Author keywords

Bibliometric evaluation; Informed peer review; Research assessment; VQR

Indexed keywords

BIBLIOMETRIC; BIBLIOMETRIC ANALYSIS; JOURNAL PUBLICATION; LARGE-SCALE RESEARCH; NATIONAL ASSESSMENT; PEER REVIEW; RESEARCH IN ECONOMICS; VQR;

INNOVATION;

EID: 84920653927     PISSN: 00487333     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.respol.2014.08.004     Document Type: Article

References (40)

1. L. Allen, C. Jones, K. Dolby, D. Lynn, and M. Walport Looking for landmarks: the role of expert review and bibliometric analysis in evaluating scientific publication outputs PLoS One 4 2009 e5910
2. F. Bartolucci, V. Dardanoni, and F. Peracchi Ranking scientific journals via latent class models for polytomous item response data EIEF Working Paper No. 13/13 2013 Available at 〈http://www.eief.it/faculty-visitors/faculty-a-z/franco-peracchi/〉
3. L. Bornmann Scientific peer review Annual Review of Information Science and Technology 45 2011 199 245
4. L. Bornmann, and H.-D. Daniel What do citation counts measure? A review of studies on citing behavior Journal of Documentation 64 2008 45 80
5. M. Burger, J.G. Frankfort, and A.F.J. van Raan The use of bibliometric data for the measurement of university research performance Research Policy 14 1985 131 149
6. L. Butler Assessing university research: a plea for a balanced approach Science and Public Policy 34 2007 565 574
7. D. Christelis Imputation of missing data in Waves 1 and 2 of SHARE CSEF Working Paper No. 278 2011 Available at 〈http://www.csef.it/WP/wp278.pdf〉
8. A. Eyre-Walker, and N. Stoletzki The assessment of science: the relative merits of post-publication review, the Impact Factor, and the number of citations PLoS Biology 11 2013 e1001675
9. J. Fagerberg, H. Landström, and B.R. Martin Exploring the emerging knowledge base of "the knowledge society" Research Policy 41 2012 1121 1131
10. J.L. Fleiss Statistical Methods for Rates and Proportions second ed. 1981 John Wiley New York, NY
11. A. Geuna, and B.E. Martin University research evaluation and funding: an international comparison Minerva 41 2003 277 304
12. A.-W. Harzing, and R. van der Wal Comparing the Google Scholar h-index with the ISI Journal Impact Factor 2008 Available at 〈http://www.harzing.com/h-indexjournals.htm〉
13. D. Hicks Performance-based university research funding systems Research Policy 41 2012 251 261
14. D. Hicks The difficulty of achieving full coverage of international social science literature and the bibliometric consequences Scientometrics 44 1999 193 215
15. D. Hicks, and J. Wang Coverage and overlap of the new social sciences and humanities journal lists Journal of the American Society for Information Science and Technology 62 2011 284 294
16. J.A. Jacobs Journal rankings in sociology: using the H index with Google Scholar PSC Working Paper No. 11-05 2011 Available at 〈http://repository.upenn.edu/psc-working-papers/29〉
17. J.R. Landis, and G.G. Koch The measurement of observer agreement for categorical data Biometrics 33 1977 159 174
18. J.M. Lepkowski, T.E. Raghunathan, J. Van Hoewyk, and P. Solenberger A multivariate technique for multiply imputing missing values using a sequence of regression models Survey Methodology 27 2001 85 95
19. L. Linnemer, and P. Combes Inferring missing citations: a quantitative multi-criteria ranking of all journals in economics GREQAM Discussion Paper No. 2010-25 2010 Available at 〈http://www.vcharite.univ-mrs.fr/pp/combes/〉
20. R.E. Little, and D.B. Rubin Statistical Analysis of Missing Data second ed. 2002 John Wiley & Sons New York, NY
21. B.R. Martin The Research Excellence Framework and the "impact agenda": are we creating a Frankenstein monster? Research Evaluation 20 2011 247 254
22. B.R. Martin, and R. Whitley The UK Research Assessment Exercise: a case of regulatory capture? R. Whitley, J. Gläser, L. Engwall, Reconfiguring Knowledge Production: Changing Authority Relationships in the Sciences and their Consequences for Intellectual Innovation 2010 Oxford University Press Oxford 51 80
23. J. Mingers, F. Macri, and D. Petrovici Using the h-index to measure the quality of journals in the field of business and management Information Processing and Management 48 2012 234 241
24. H.F. Moed The future of research evaluation rests with an intelligent combination of advanced metrics and transparent peer review Science and Public Policy 34 2007 575 583
25. H.F. Moed Citation Analysis in Research Evaluation 2005 Springer Dordrecht
26. J. Nicolaisen Citation analysis Annual Review of Information Science and Technology 41 2007 609 641
27. OECD Performance-based Funding for Public Research in Tertiary Education Institutions: Workshop Proceedings 2010 OECD Publishing Paris
28. A.J. Oswald An examination of the reliability of prestigious scholarly journals: evidence and implications for decision-makers Economica 74 2007 21 31
29. I. Rafols, L. Leydesdorff, A. O'Hare, P. Nightingale, and A. Stirling How journal rankings can suppress interdisciplinary research: a comparison between Innovation Studies and Business & Management Research Policy 41 2012 1262 1282
30. G. Rebora, and M. Turri The UK and Italian research assessment exercises face to face Research Policy 42 2013 1657 1666
31. E.J. Rinia, Th.N. van Leeuwen, H.G. van Vuren, and A.F.J. van Raan Comparative analysis of a set of bibliometric indicators and central peer review criteria Research Policy 27 1998 95 107
32. D.B. Rubin Multiple Imputation for Nonresponse in Surveys 1987 John Wiley & Sons New York, NY
33. J.L. Schafer Analysis of Incomplete Multivariate Data 1997 Chapman and Hall Boca Raton, FL
34. P.O. Seglen Why the impact factor of journals should not be used for evaluating research BMJ 314 1997 498 502
35. P.O. Seglen The skewness of science Journal of the American Society for Information Science 43 1992 628 638
36. D. Sgroi, and A.J. Oswald How should peer-review panels behave? Economic Journal 123 2013 F255 F278
37. D.I. Stern Uncertainty measures for economics journal impact factors Journal of Economic Literature 51 2013 173 189
38. M.A. Tanner, and W.H. Wong The calculation of posterior distributions by data augmentation (with discussion) Journal of the American Statistical Association 82 1987 528 550
39. S. van Buuren, J.P.L. Brand, C.G.M. Groothuis-Oudshoorn, and D.B. Rubin Fully conditional specification in multivariate imputation Journal of Statistical Computation and Simulation 76 2006 1049 1064
40. L. Waltman, and R. Costas F1000 Recommendations as a New Data Source for Research Evaluation: A Comparison with Citations 2013 (a∗rXi∗v:1303.3875). Available at 〈http://arxiv.org/pdf/1303.3875v1〉