Does the committee peer review select the best applicants for funding? An investigation of the selection process for two European molecular biology organization programmes

PLoS ONE

Volumn 3, Issue 10, 2008, Pages

  Bornmann, Lutz ^a   Wallon, Gerlind ^b   Ledin, Anna ^b,c  

^a ETH ZURICH   (Switzerland)

^b European Molecular Biology Organization   (Germany)

^c ROYAL SWEDISH ACADEMY OF SCIENCES   (Sweden)

Author keywords

[No Author keywords available]

Indexed keywords

ADVISORY COMMITTEE; ANALYTICAL ERROR; AWARDS AND PRIZES; BIOMEDICINE; CONTROLLED STUDY; DECISION MAKING; EUROPEAN UNION; FUNDING; HUMAN; INTERMETHOD COMPARISON; MOLECULAR BIOLOGY; ORGANIZATIONAL EFFICIENCY; PEER REVIEW ORGANIZATION; PERSONNEL MANAGEMENT; POSTDOCTORAL EDUCATION; PROFESSIONAL STANDARDS REVIEW ORGANIZATION; PUBLICATION; QUALITY CIRCLE; RESEARCH ETHICS; REVIEW; TASK PERFORMANCE; ACHIEVEMENT; ARTICLE; EUROPE; EVALUATION; MEDICAL EDUCATION; METHODOLOGY; PEER REVIEW; STANDARD; STATISTICS;

ACHIEVEMENT; EUROPE; FELLOWSHIPS AND SCHOLARSHIPS; MOLECULAR BIOLOGY; PEER REVIEW, RESEARCH; PUBLICATIONS;

EID: 55649086594     PISSN: None     EISSN: 19326203     Source Type: Journal    
DOI: 10.1371/journal.pone.0003480     Document Type: Review

References (53)

1. Ziman J (2000) Real science. What it is, and what it means. Cambridge, UK: Cambridge University Press.
2. Marsh HW, Jayasinghe UW, Bond NW (2008) Improving the peer-review process for grant applications-reliability, validity, bias, and generalizability. American Psychologist 63:160-168.
3. Geisler E (2000) The metrics of science and technology. Westport, CT, USA: Quorum Books.
4. Hemlin S (1996) Research on research evaluations. Social Epistemology 10: 209-250.
5. National Institutes of Health (2000) Recommendations for change at the NIH's Center for Scientific Review: phase 1 report, panel on scientific boundaries for review. Bethesda, MD, USA: Center for Scientific Review (CSR).
6. Ross PF (1980) The sciences' self-management: manuscript refereeing, peer review, and goals in science. Lincoln, MA, USA: The Ross Company.
7. Bornstein RF (1991) The predictive validity of peer-review: a neglected issue. Behavioral and Brain Sciences 14:138-139.
8. Cole S, Cole JR, Simon GA (1981) Chance and consensus in peer-review. Science 214:881-886.
9. Smith R (1988) Problems with peer review and alternatives. British Medical Journal 296:774-777.
10. Harnad S (2007) Open Access scientometrics and the UK Research Assessment Exercise. In: Torres-Salinas D, Moed HF, eds. Proceedings of the 11th Conference of the International Society for Scientometrics and Informetrics. Madrid, Spain: Spanish Research Council (CSIC). pp 27-33.
11. Cole S (1992) Making science. Between nature and society. Cambridge, MA, USA: Harvard University Press.
12. van Raan AFJ (2005) For your citations only? Hot topics in bibliometric analysis. Measurement: Interdisciplinary Research and Perspectives 3:50-62.
13. Garfield E (2002) Highly cited authors. Scientist 16:10.
14. Jennings CG (2006) Quality and value: the true purpose of peer review. What you can't measure, you can't manage: the need for quantitative indicators in peer review.
15. Bornmann L, Daniel H-D (2008) What do citation counts measure? A review of studies on citing behavior. Journal of Documentation 64:45-80.
16. Joint Committee on Quantitative Assessment of Research (2008) Citation statistics. A report from the International Mathematical Union (IMU) in cooperation with the International Council of Industrial and Applied Mathematics (ICIAM) and the Institute of Mathematical Statistics (IMS). Berlin, Germany: International Mathematical Union (IMU).
17. Lokker C, McKibbon KA, McKinlay RJ, Wilczynski NL, Haynes RB (2008) Prediction of citation counts for clinical articles at two years using data available within three weeks of publication: retrospective cohort study. British Medical Journal 336:655-657.
18. Dewitt TW, Nicholson RS, Wilson MK (1980) Science Citation Index and chemistry. Scientometrics 2:265-275.
19. Bornmann L, Daniel H-D (2008) Functional use of frequently and infrequently cited articles in citing publications. A content analysis of citations to articles with low and high citation counts. European Science Editing 34:35-38.
20. Evidence Ltd (2007) The use of bibliometrics to measure research quality in UK higher education institutions. London, UK: Universities UK.
21. Ledin A, Bornmann L, Gannon F, Wallon G (2007) A persistent problem. Traditional gender roles hold back female scientists. EMBO Reports 8:982-987.
22. Bornmann L, Daniel H-D (2008) Selecting manuscripts for a high impact journal through peer review: a citation analysis of Communications that were accepted by Angewandte Chemie International Edition, or rejected but published elsewhere. Journal of the American Society for Information Science and Technology 59:1841-1852.
23. Herbertz H (1995) Does it pay to cooperate? A bibliometric case-study in molecular-biology. Scientometrics 33:117-122.
24. Leimu R, Koricheva J (2005) What determines the citation frequency of ecological papers? Trends in Ecology & Evolution 20:28-32.
25. Aksnes DW (2008) When different persons have an identical author name. How frequent are homonyms? Journal of the American Society for Information Science and Technology 59:838-841.
26. StataCorp (2007) Stata statistical software: release 10. College Station, TX, USA: Stata Corporation.
27. Wuchty S, Jones BF, Uzzi B (2007) The increasing dominance of teams in production of knowledge. Science 316:1036-1039.
28. Bornmann L, Daniel H-D (2007) Multiple publication on a single research study: does it pay? The influence of number of research articles on total citation counts in biomedicine. Journal of the American Society for Information Science and Technology 58:1100-1107.
29. Daniel H-D (1993/2004) Guardians of science. Fairness and reliability of peer review. Weinheim, Germany: Wiley-VCH, Published online 16 July 2004, Wiley Interscience, DOI: 10.1002/3527602208.
30. Long JS, Freese J (2006) Regression models for categorical dependent variables using Stata. College Station, TX, USA: Stata Press, Stata Corporation.
31. Hosmer DW, Lemeshow S (2000) Applied logistic regression. Chichester, UK: John Wiley & Sons, Inc.
32. Chatterjee S, Hadi AS (2006) Regression analysis by example. 4. ed. New York, NY, USA: Wiley-Interscience.
33. Hausman J, Hall BH, Griliches Z (1984) Econometric models for count data with an application to the patents R and D relationship. Econometrica 52: 909-938.
34. Hilbe JM (2007) Negative binomial regression. Cambridge, UK: Cambridge University Press.
35. Cameron AC, Trivedi PK (1998) Regression analysis of count data. Cambridge, UK: Cambridge University Press.
36. Jayasinghe UW (2003) Peer review in the assessment and funding of research by the Australian Research Council. Greater Western Sydney, Australia: University of Western Sydney.
37. Johnson VE (2008) Statistical analysis of the National Institutes of Health peer review system. Proceedings of the National Academy of Sciences 105: 11076-11080.
38. Bornmann L, Daniel H-D (2007) Convergent validation of peer review decisions using the h index: extent of and reasons for type I and type II errors. Journal of Informetrics 1:204-213.
39. Hirsch JE (2005) An index to quantify an individual's scientific research output. Proceedings of the National Academy of Sciences of the United States of America 102: 16569-16572.
40. Bornmann L, Daniel H-D (2007) What do we know about the h index? Journal of the American Society for Information Science and Technology 58: 1381-1385.
41. Bornmann L, Daniel H-D (2005) Does the h-index for ranking of scientists really work? Scientometrics 65:391-392.
42. Bornmann L, Wallon G, Ledin A (2008) Is the h index related to (standard) bibliometric measures and to the assessments by peers? An investigation of the h index by using molecular life sciences data. Research Evaluation 17:149-156.
43. Freeman R, Weinstein E, Marincola E, Rosenbaum J, Solomon F (2001) Careers-competition and careers in biosciences. Science 294:2293-2294.
44. Thorngate W, Faregh N, Young M (2002) Mining the archives: analyses of CIHR research grant applications. Ottawa, Ontario, Canada: Psychology Department, Carlton University.
45. Stehbens WE (1999) Basic philosophy and concepts underlying scientific peer review. Medical Hypotheses 52:31-36.
46. Hornbostel S, Böhmer S, Klingsporn B, Neufeld J, von Ins M (in press) Funding of young scientist and scientific excellence. Scientometrics.
47. Melin G, Danell R (2006) The top eight percent: development of approved and rejected applicants for a prestigious grant in Sweden. Science and Public Policy 33:702-712.
48. van den Besselaar P, Leydesdorff L (2007) Past performance as predictor of successful grant applications. A case study. Den Haag, The Netherlands: Rathenau Instituut SciSA rapport 0704.
49. Bornmann L, Daniel H-D (2006) Selecting scientific excellence through committee peer review - a citation analysis of publications previously published to approval or rejection of post-doctoral research fellowship applicants. Scientometrics 68:427-440.
50. Vinkler P (1997) Relations of relative scientometric impact indicators. The relative publication strategy index. Scientometrics 40:163-169.
51. Vinkler P (1986) Evaluation of some methods for the relative assessment of scientific publications. Scientometrics 10:157-177.
52. van Raan AFJ (1999) Advanced bibliometric methods for the evaluation of universities. Scientometrics 45:417-423.
53. Neuhaus C, Daniel H-D (in press) A new reference standard for citation analysis in chemistry and related fields based on the sections of Chemical Abstracts. Scientometrics.