A meta-evaluation of scientific research proposals: Different ways of comparing rejected to awarded applications

Journal of Informetrics

Volumn 4, Issue 3, 2010, Pages 211-220

  Bornmann, Lutz ^a   Leydesdorff, Loet ^b   Van den Besselaar, Peter ^c,d  

^a ETH ZURICH   (Switzerland)

^b UNIVERSITY OF AMSTERDAM   (Netherlands)

^c Rathenau Institute KNAW   (Netherlands)

^d VU UNIVERSITY AMSTERDAM   (Netherlands)

Author keywords

Bibliometric quality indicators; Citation rate; Convergent validity and predictive validity; Error; Grant allocation; H Index; Peer review

Indexed keywords

EID: 77957782481     PISSN: 17511577     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.joi.2009.10.004     Document Type: Article

References (35)

1. Allison P.D. Inequality and scientific productivity. Social Studies of Science 1980, 10(2):163-179.
2. Bornmann, L. (in press). Scientific peer review. Annual Review of Information Science and Technology.
3. Bornmann L., Daniel H.-D. Convergent validation of peer review decisions using the h index. Extent of and reasons for type I and type II errors. Journal of Informetrics 2007, 1:204-213.
4. Bornmann L., Daniel H.-D. Multiple publication on a single research study: does it pay? The influence of number of research articles on total citation counts in biomedicine. Journal of the American Society for Information Science and Technology 2007, 58(8):1100-1107.
5. Bornmann L., Daniel H.-D. What do citation counts measure? A review of studies on citing behavior. Journal of Documentation 2008, 64(1):45-80.
6. Bornmann L., Daniel H.-D. The state of h index research. Is the h index the ideal way to measure research performance. EMBO Reports 2009, 10(1):2-6.
7. Bornmann L., Mutz R., Daniel H.-D. Are there better indices for evaluation purposes than the h index? A comparison of nine different variants of the h index using data from biomedicine. Journal of the American Society for Information Science and Technology 2008, 59(5):830-837.
8. Bornmann L., Wallon G., Ledin A. Does the committee peer review select the best applicants for funding? An investigation of the selection process for two European Molecular Biology Organization programmes. PLoS ONE 2008, 3(10).
9. Bornmann L., Wallon G., Ledin A. Is the h index related to (standard) bibliometric measures and to the assessments by peers? An investigation of the h index by using molecular life sciences data. Research Evaluation 2008, 17(2):149-156.
10. De Haan J. Research groups in Dutch sociology 1994, Thesis Publishers, Amsterdam, the Netherlands.
11. Elkana Y., Lederberg J., Merton R.K., Thackray A., Zuckerman H. Toward a metric of science: the advent of science indicators 1978, Wiley, New York, NY.
12. Frey B.S., Osterloh M. Successful management by motivation: Balancing intrinsic and extrinsic incentives 2002, Springer, Berlin, Germany.
13. Garfield E. Citation analysis as a tool in journal evaluation. Science 1972, 178(4060):471-479.
14. Garfield E. Citation indexing-Its theory and application in science, technology, and humanities 1979, Wiley, New York, NY.
15. Hausman J., Hall B.H., Griliches Z. Econometric models for count data with an application to the patents R and D relationship. Econometrica 1984, 52(4):909-938.
16. Hilbe J.M. Negative binomial regression 2007, Cambridge University Press, Cambridge, UK.
17. Hornbostel S., Böhmer S., Klingsporn B., Neufeld J., von Ins M. Funding of young scientist and scientific excellence. Scientometrics 2009, 79(1):171-190.
18. Hosmer D.W., Lemeshow S. Applied logistic regression 2000, Wiley, Chichester, UK. 2 ed.
19. Long J.S., Freese J. Regression models for categorical dependent variables using Stata 2006, Stata Press, Stata Corporation, College Station, TX. 2 ed.
20. Martin B.R., Irvine J. Assessing basic research: Some partial indicators of scientific progress in radio astronomy. Research Policy 1983, 12:61-90.
21. Martin B.R., Irvine J. Evaluating the evaluators. Social Studies of Science 1985, 15:558-585.
22. Melin G., Danell R. The top eight percent: Development of approved and rejected applicants for a prestigious grant in Sweden. Science and Public Policy 2006, 33(10):702-712.
23. Merton R.K. The Matthew effect in science. Science 1968, 159:56-63.
24. Merton R.K. Foreword. Citation indexing: Its theory and application in science, technology and humanities 1979, Wiley, New York, NY, (pp. v-ix). E. Garfield (Ed.).
25. Moed H.F., Burger W.J.M., Frankfort J.G., Van Raan A.F.J. The use of bibliometric data for the measurement of university research performance. Research Policy 1985, 14:131-149.
26. Mulkay M.J. The mediating role of the scientific elite. Social Studies of Science 1976, 6(34):445-470.
27. Narin F. Evaluative bibliometrics: The use of publication and citation analysis in the evaluation of scientific activity 1976, National Science Foundation, Washington, DC.
28. Nederhof A.J., Van Raan A.F.J. Peer review and bibliometric indicators of scientific performance: A comparison of cum laude doctorates with ordinary doctorates in physics. Scientometrics 1987, 11(5):333-350.
29. Straub D.W. Thirty years of service to the IS profession: Time for renewal at MISQ?. MIS Quarterly 2008, 32(1):iii-Viii.
30. Straub D.W. Type II reviewing errors and the search for exciting papers. MIS Quarterly 2008, 32(2):V-X.
31. Thorngate, W., Faregh, N., & Young, M. (2002). Mining the archives: analyses of CIHR research grant applications. Retrieved 28.04.05. http://http-server.carleton.ca/~warrent/reports/mining_the_archives.pdf.
32. Van den Besselaar P., Leydesdorff L. Past performance as predictor of successful grant applications 2007, Rathenau Instituut, Den Haag, the Netherlands.
33. Van den Besselaar P., Leydesdorff L. Past performance, peer review, and project selection: a case study in the social and behavioral sciences. Research Evaluation 2009, 18(4):273-288.
34. Wuchty S., Jones B.F., Uzzi B. The increasing dominance of teams in production of knowledge. Science 2007, 316(5827):1036-1039.
35. Zuckerman H., Merton R.K. Patterns of evaluation in science: Institutionalisation, structure and functions of the referee system. Minerva 1971, 9(1):66-100.