The VQR, Italy's second national research assessment: Methodological failures and ranking distortions

Journal of the Association for Information Science and Technology

Volumn 66, Issue 11, 2015, Pages 2202-2214

  Abramo, Giovanni ^a   D'Angelo, Ciriaco Andrea ^b  

^a CNR   (Italy)

^b UNIVERSITY OF ROME TOR VERGATA   (Italy)

Author keywords

bibliometrics

Indexed keywords

INFORMATION SCIENCE;

BIBLIOMETRICS; METHODOLOGICAL FAILURES; PERFORMANCE RANKINGS; RESEARCH ASSESSMENT EXERCISE; RESEARCH QUALITY; WEB OF SCIENCE;

QUALITY CONTROL;

EID: 84949836337     PISSN: 23301635     EISSN: 23301643     Source Type: Journal    
DOI: 10.1002/asi.23323     Document Type: Article

References (46)

1. Abramo, G., &, D'Angelo, C.A., (2011). Evaluating research: From informed peer review to bibliometrics. Scientometrics, 87 (3), 499-514.
2. Abramo, G., &, D'Angelo, C.A., (2014). How do you define and measure research productivity? Scientometrics, 101 (2), 1129-1144.
3. Abramo, G., D'Angelo, C.A., &, Caprasecca, A., (2009). Allocative efficiency in public research funding: Can bibliometrics help? Research Policy, 38 (1), 206-215.
4. Abramo, G., D'Angelo, C.A., &, Viel, F., (2010a). Peer review research assessment: A sensitivity analysis of performance rankings to the share of research product evaluated. Scientometrics, 85 (3), 705-720.
5. Abramo, G., D'Angelo, C.A., &, Di Costa, F., (2010b). Citations versus journal impact factor as proxy of quality: Could the latter ever be preferable? Scientometrics, 84 (3), 821-833.
6. Abramo, G., D'Angelo, C.A., &, Di Costa, F., &, Solazzi, M., (2011a). The role of information asymmetry in the market for university-industry research collaboration. The Journal of Technology Transfer, 36 (1), 84-100.
7. Abramo, G., D'Angelo, C.A., &, Di Costa, F., (2011b). National research assessment exercises: A comparison of peer review and bibliometrics rankings. Scientometrics, 89 (3), 929-941.
8. Abramo, G., Cicero, T., &, D'Angelo, C.A., (2012). Revisiting the scaling of citations for research assessment. Journal of Informetrics, 6 (4), 470-479.
9. Abramo, G., Cicero, T., &, D'Angelo, C.A., (2013a). National peer-review research assessment exercises for the hard sciences can be a complete waste of money: The Italian case. Scientometrics, 95 (1), 311-324.
10. Abramo, G., D'Angelo, C.A., &, Rosati, F., (2013b). The importance of accounting for the number of co-authors and their order when assessing research performance at the individual level in the life sciences. Journal of Informetrics, 7 (1), 198-208.
11. Abramo, G., D'Angelo, C.A., &, Di Costa, F., (2014). Inefficiency in selecting products for submission to national research assessment exercises. Scientometric, 98 (3), 2069-2086.
12. Abt, H.A., (1984). Citations to single and multiauthored papers. Publications of the Astronomical Society of the Pacific, 96 (583), 746-749.
13. Adams, J., &, MacLeod, D., (2002). End of the exercise. The RAE's critics are asked for new ideas. The Guardian. Tuesday, July 2, 2002.
14. Aksnes, D.W., &, Taxt, R.E., (2004). Peers reviews and bibliometric indicators: A comparative study at Norwegian University. Research Evaluation, 13 (1), 33-41.
15. Bridgstock, M., (1991). The quality of single and multiple authored papers: An unresolved problem. Scientometrics, 21 (1), 37-48.
16. Butler, L., (2003a). Modifying publication practices in response to funding formulas. Research Evaluation, 12 (1), 39-46.
17. Butler, L., (2003b). Explaining Australia's increased share of ISI publications-The effects of a funding formula based on publication counts. Research Policy, 32, 143-155.
18. D'Angelo, C.A., Giuffrida, C., &, Abramo, G., (2011). A heuristic approach to author name disambiguation in large-scale bibliometric databases. Journal of the American Society for Information Science and Technology, 62 (2), 257-269.
19. Franceschet, M., &, Costantini, A., (2011). The first Italian Research Assessment Exercise: A bibliometric perspective. Journal of Informetrics, 5 (2), 275-291.
20. Geuna, A., &, Martin, B.R., (2003). University research evaluation and funding: An international comparison. Minerva, 41 (4), 277-304.
21. Glanzel, W., &, Moed, H.F., (2002). Journal impact measures in bibliometric research. Scientometrics, 53 (2), 171-193.
22. Harsanyi, M.A., (1993). Multiple authors, multiple problems-bibliometrics and the study of scholarly collaboration: A literature review. Library and Information Science Research, 15 (4), 325-354.
23. Hicks, D., (2012). Performance-based university research funding systems. Research Policy, 41 (2), 251-261.
24. Horrobin, D.F., (1990). The philosophical basis of peer review and the suppression of innovation. Journal of the American Medical Association, 263 (10), 1438-1441.
25. Lipsett, A., (2007). RAE selection gets brutal. Times Higher Education Supplement, 1779, 1.
26. MacRoberts, M.H., &, MacRoberts, B.R., (1996). Problems of citation analysis. Scientometrics, 36 (3), 435-444.
27. Martin, B.R., &, Whitley, R., (2010). The UK Research Assessment Exercise: A case of regulatory capture? In, R. Whitley, J. Gläser, &, L. Engwall, (Eds.), Reconfiguring knowledge production: Changing authority relationships in the sciences and their consequences for intellectual innovation (pp. 51-80). Oxford, UK: Oxford University Press.
28. Meho, L.I., &, Sonnenwald, D.H., (2000). Citation ranking versus peer evaluation of senior faculty research performance: A case study of Kurdish scholarship. Journal of the American Society for Information Science, 51 (2), 123-138.
29. Moed, H.F., (2002). The impact-factors debate: The ISI's uses and limits. Nature, 415 (6873), 731-732.
30. Moed, H.F., &, Van Leeuwen, T.N., (1996). Impact factors can mislead. Nature, 381 (6579), 186.
31. Moxham, H., &, Anderson, J., (1992). Peer review. A view from the inside. Science and Technology Policy, 5 (1), 7-15.
32. Oppenheim, C., (1997). The correlation between citation counts and the 1992 research assessment exercise ratings for British research in genetics, anatomy and archaeology. Journal of Documentation, 53 (5), 477-487.
33. Oppenheim, C., &, Norris, M., (2003). Citation counts and the research assessment exercise V: Archaeology and the 2001 RAE. Journal of Documentation, 56 (6), 709-730.
34. Pendlebury, D.A., (2009). The use and misuse of journal metrics and other citation indicators. Scientometrics, 57 (1), 1-11.
35. Perotti, R., (2008). L'università truccata. Torino, Italy: Einaudi. ISBN 9788806193607.
36. Rinia, E.J., van Leeuwen, T.N., van Vuren, H.G., &, van Raan, A.F.J., (1998). Comparative analysis of a set of bibliometric indicators and central peer-review criteria, evaluation of condensed matter physics in the Netherlands. Research Policy, 27 (1), 95-107.
37. Sastry, T., &, Bekhradnia, B., (2006). Using metrics to allocate research funds: A short evaluation of alternatives to the Research Assessment Exercise. Oxford, UK: Higher Education Policy Institute.
38. Seglen, P.O., (1997). Why the impact factor of journals should not be used for evaluating research. British Medical Journal, 314 (7079), 497-502.
39. Steele, C., Butler, L., &, Kingsley, D., (2006). The publishing imperative: The pervasive influence of publication metrics. Learned Publishing, 19 (4), 277-290.
40. Stoker, G., (2006). Public value management: A new narrative for networked governance? The American Review of Public Administration, 36 (1), 41-57.
41. Thomas, P.R., &, Watkins, D.S., (1998). Institutional research rankings via bibliometric analysis and direct peer-review: A comparative case study with policy implications. Scientometrics, 41 (3), 335-355.
42. Trueba, F.J., &, Guerrero, H., (2004). A robust formula to credit authors for their publications. Scientometrics, 60 (2), 181-204.
43. van Raan, A.F.J., (2005). Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods. Scientometrics, 62 (1), 133-143.
44. van Raan, A.F.J., (2006). Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups. Scientometrics, 67 (3), 491-502.
45. Weingart, P., (2005). Impact of bibliometrics upon the science system: Inadvertent consequences? Scientometrics, 62 (1), 117-131.
46. Zuckerman, H.A., (1968). Patterns of name ordering among authors of scientific papers: A study of social symbolism and its ambiguity. The American Journal of Sociology, 74 (3), 276-291.