How good is a model based on bibliometric indicators in predicting the final decisions made by peers?

Journal of Informetrics

Volumn 8, Issue 2, 2014, Pages 390-405

  Vieira, Elizabeth S ^a   Cabral, José A S ^b   Gomes, José A N F ^a  

^a UNIVERSITY OF PORTO   (Portugal)

^b UNIVERSITY OF PORTO   (Portugal)

Author keywords

Auxiliary instrument; Bibliometric indicators; Peer review; Predictive power

Indexed keywords

BIBLIOMETRIC INDICATORS; DEGREE OF ACCURACY; FINAL DECISION; MODEL-BASED OPC; PEER EVALUATIONS; PEER REVIEW; PREDICTIVE POWER; SCIENTIFIC PERFORMANCE;

COMPUTER APPLICATIONS;

EID: 84894288577     PISSN: 17511577     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.joi.2014.01.012     Document Type: Article

References (27)

1. Abramo G., D'Angelo C.A., Caprasecca A. Allocative efficiency in public research funding: Can bibliometrics help?. Research Policy 2009, 38(1):206-215.
2. Aksnes D.W., Taxt R.E. Peer reviews and bibliometric indicators: A comparative study at a norwegian university. Research Evaluation 2004, 13(1):33-41.
3. Bornmann L. Scientific peer review. Annual Review of Information Science and Technology 2011, 45:199-245.
4. Bornmann L., Daniel H.D. Selection of research fellowship recipients by committee peer review. Reliability, fairness and predictive validity of board of trustees' decisions. Scientometrics 2005, 63(2):297-320.
5. Bornmann L., Daniel H.D. Selecting scientific excellence through committee peer review - A citation analysis of publications previously published to approval or rejection of post-doctoral research fellowship applicants. Scientometrics 2006, 68(3):427-440.
6. Bornmann L., Daniel H.D. Convergent validation of peer review decisions using the h index - Extent of and reasons for type i and type ii errors. Journal of Informetrics 2007, 1(3):204-213.
7. Bornmann L., Mutz R., Daniel H.D. A reliability-generalization study of journal peer reviews: A multilevel meta-analysis of inter-rater reliability and its determinants. PLoS One 2010, 5(12).
8. Bornmann L., Wallon G., Ledin A. Does the committee peer review select the best applicants for funding? An investigation of the selection process for two european molecular biology organization programmes. PLoS One 2008, 3(10).
9. Cicchetti D.V. The reliability of peer-review for manuscript and grant submissions - A cross disciplinary investigation. Behavioral and Brain Sciences 1991, 14(1):119-134.
10. Franceschet M., Costantini A. The first italian research assessment exercise: A bibliometric perspective. Journal of Informetrics 2011, 5(2):275-291.
11. Gonzalez-Pereira B., Guerrero-Bote V.P., Moya-Anegon F. A new approach to the metric of journals' scientific prestige: The SJR indicator. Journal of Informetrics 2010, 4(3):379-391.
12. Hodgson C. How reliable is peer review? An examination of operating grant proposals simultaneously submitted to two similar peer review systems. Journal of Clinical Epidemiology 1997, 50(11):1189-1195.
13. Jayasinghe U.W., Marsh H.W., Bond N. Peer review in the funding of research in higher education: The Australian experience. Educational Evaluation and Policy Analysis 2001, 23(4):343-364.
14. Jayasinghe U.W., Marsh H.W., Bond N. A multilevel cross-classified modelling approach to peer review of grant proposals: The effects of assessor and researcher attributes on assessor ratings. Journal of the Royal Statistical Society Series A-Statistics in Society 2003, 166:279-300.
15. Jayasinghe U.W., Marsh H.W., Bond N. A new reader trial approach to peer review in funding research grants: An Australian experiment. Scientometrics 2006, 69(3):591-606.
16. Marsh H.W., Bonds N.W., Jayasinghe U.W. Peer review process: Assessments by applicant-nominated referees are biased, inflated, unreliable and invalid. Australian Psychologist 2007, 42(1):33-38.
17. Marsh H.W., Jayasinghe U.W., Bond N.W. Improving the peer-review process for grant applications - Reliability, validity, bias, and generalizability. American Psychologist 2008, 63(3):160-168.
18. Moed H.F. Measuring contextual citation impact of scientific journals. Journal of Informetrics 2010, 4(3):265-277.
19. Nederhof A.J., van Raan A.F.J. Peer-review and bibliometric indicators of scientific performance - A comparison of cum laude doctorates with ordinary doctorates in physics. Scientometrics 1987, 11(5-6):333-350.
20. Reale E., Barbara A., Costantini A. Peer review for the evaluation of academic research: Lessons from the italian experience. Research Evaluation 2007, 16(3):216-228.
21. Rinia E.J., van Leeuwen T.N., van Vuren H.G., van Raan A.F.J. Comparative analysis of a set of bibliometric indicators and central peer review criteria - Evaluation of condensed matter physics in the netherlands. Research Policy 1998, 27(1):95-107.
22. Taylor J. The assessment of research quality in UK universities: Peer review or metrics?. British Journal of Management 2011, 22(2):202-217.
23. Train K. Discrete choice methods with simulation 2009, Vol. 2009. Cambridge University Press. 2nd ed.
24. van Raan A.F.J. Comparison of the hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups. Scientometrics 2006, 67(3):491-502.
25. Vieira E.S., Cabral J.A.S., Gomes J.A.N.F. Definition of a model based on bibliometric indicators for assessing applicants to academic positions. Journal of the American Society for Information Science and Technology 2013, (in press). 10.1002/asi.22981/abstract.
26. Vieira E.S., Gomes J.A.N.F. An impact indicator for researchers. Scientometrics 2011, 89(2):607-629.
27. Wood M., Roberts M., Howell B. The reliability of peer reviews of papers on information systems. Journal of Information Science 2004, 30(1):2-11.