Early career grants, performance, and careers: A study on predictive validity of grant decisions

Journal of Informetrics

Volumn 9, Issue 4, 2015, Pages 826-838

  van den Besselaar, Peter ^a   Sandström, Ulf ^b  

^a VU UNIVERSITY AMSTERDAM   (Netherlands)

^b ÖREBRO UNIVERSITY   (Sweden)

Author keywords

Academic careers; Career grants; Gender differences; Grant allocation; Predictive validity

Indexed keywords

DECISION MAKING;

ACADEMIC CAREERS; CAREER GRANTS; GENDER DIFFERENCES; GRANT ALLOCATION; PREDICTIVE VALIDITY;

BEHAVIORAL RESEARCH;

EID: 84940823440     PISSN: 17511577     EISSN: 18755879     Source Type: Journal    
DOI: 10.1016/j.joi.2015.07.011     Document Type: Article

References (45)

1. Armstrong P.W., Caverson M.M., Adams L., Taylor M., Olley P.M. Evaluation of the Heart and Stroke Foundation of Canada Research Scholarship Program: Research productivity and impact. Canadian Journal of Cardiology 1997, 13(May):507-516.
2. Bornmann L., Daniel H.-D. Selecting scientific excellence through committee peer review-A citation analysis of publications previously published to approval or rejection of post-doctoral research fellowship applicants. Scientometrics 2006, 68(3):427-440.
3. Bornmann L. Scientific peer review. Annual Review of Information Science and Technology 2011, 45:199-245.
4. Bornmann L., Wallon G., Ledin A. Does the committee peer review select the best applicants for funding? An investigation of the selection process for two European molecular biology organization programmes. PLoS ONE 2008, 3(10):e3480.
5. Bornmann L., Leydesdorff L., van den Besselaar P. A Meta-evaluation of scientific research proposals: Different ways of comparing rejected to awarded applications. Journal of Informetrics 2010, 4(3):211-220.
6. Benda W.G.G., Engels T.C.E. The predictive validity of peer review: A selective review of the judgmental forecasting qualities of peers, and implications for innovation in science. International Journal of Forecasting 2011, 27(1):166-182.
7. Cabezas-Clavijo A., Robinson-García N., Escabias M., Jiménez-Contreras E. Reviewers' ratings and bibliometric indicators: Hand in hand when assessing over research proposals?. PLoS ONE 2013, 8(6):e68258.
8. Campbell D., Picard-Aitken M., Cote G., Caruso J., Valentim R., Edmonds S., et al. Bibliometrics as a performance measurement tool for research evaluation: The case of research funded by the National Cancer Institute of Canada. American Journal of Evaluation 2010, 31:66-83.
9. Cole S. Making science: Between nature and society 1992, Harvard Univ. Press, Cambridge, MA.
10. Danthi N.S., Wu C.O., Shi P., Lauer M. Percentile ranking and citation impact of a large cohort of NIHLBI-funded cardiovascular R01grants. Circulation Research 2014, 114:600-606.
11. Decullier E., Huot L., Chapuis F.R. Fate of protocols submitted to a French national funding scheme: A cohort study. PLoS ONE 2014, 9(6):e99561. 10.1371/journal.pone.0099561.
12. Esser J.K. Alive and well after 25 years: A review of groupthink research. Organizational Behavior and Human Decision Processes 1998, 73(2-3):116-141.
13. Gallo S.A., Carpenter A.S., Irwin D., McPartland C.D., Travis J., Reynders S., et al. The validation of peer review through research impact measures and the implications for funding strategies. PLoS ONE 2014, 9(9):e106474. 10.1371/journal.pone.0106474.
14. Hornbostel S., Böhmer S., Klingsporn B., Neufeld J., von Ins M. Funding of young scientist and scientific excellence. Scientometrics 2009, 79:171-190.
15. Kanter Rosabeth Moss Some effects of proportions on group life: Skewed sex ratios and responses to token women. American Journal of Sociology 1977, 82(5):965-990.
16. Kaltman J.R., Evans F.J., Danthi N.S., Wu C.O., DiMichele D.M., Lauer M.S. Prior publication productivity, grant percentile ranking and topic-normalized citation impact of NIHLBI-funded cardiovascular R01grants. Circulation Research 2014, 115:617-624.
17. Lamont M. How professors think: Inside the curious world of academic judgment 2009, Harvard University Press, Cambridge.
18. Langfeldt L. Decision making constraints and processes of grant peer review, and their effect on review outcome. Social Studies of Science 2001, 31:820-841.
19. Merton R.K. The normative structure of science. The sociology of science: Theoretical and empirical investigations (Ed. Norman Storer) 1973, University of Chicago Press, Chicago, IL. R.K. Merton (Ed.).
20. Mutz R., Bornmann L., Daniel H.-D. Testing for fairness and predictive validity of research funding decisions: A multi-level multiple imputation for missing data approach using ex-ante and ex-post peer evaluation data from the Austrian Science Fund. Journal of the Association for Information Science and Technology 2014, (published online). 10.1002/asi.23315.
21. Neufeld J., Hornbostel S. Funding programmes for young scientists-Do the 'best' apply?. Research Evaluation 2012, 21:270-279.
22. Neufeld J., von Ins M. Informed peer review and uninformed bibliometrics?. Research Evaluation 2011, 20:31-46.
23. Neufeld J., Huber N., Wegner A. Peer review-based selection decisions in individual research funding, applicants publication strategies and performance: The case of the ERC Starting Grants. Research Evaluation 2013, 22(3):1-11.
24. Olbrecht M., Bornmann L. Panel peer review of grant applications: What do we know from research in social psychology on judgment and decision-making in groups?. Research Evaluation 2010, 19(4):293-304.
25. Reinhart M. Peer review of grant applications in biology and medicine. Reliability, fairness, and validity. Scientometrics 2009, 81(3):789-809.
26. Sandström U., Hällsten M. Persistent nepotism in peer review. Scientometrics 2008, 74(2):175-189.
27. Sandström U., Sandström E. The field factor: Towards a metric for academic institutions. Research Evaluation 2009, 18(3):243-250. 10.3152/095820209X466892.
28. Sandström U., Wold A. Centres of excellence: Reward for gender or top-level research", thinking ahead: Research, funding and the future. RJ yearbook 2015/16 2015, Riksbankens Jubileumsfond and Makadam, Stockholm.
29. Saygitov R.T. The impact of funding through the FR president's grants for young scientists on research productivity: A quasi-experimental study and a brief systematic review. PLoS ONE 2014, 9(1):e86969. 10.1371/journal.pone.0086969.
30. Tindale R.S., Meisenhelder Helen M., Dykema-Engblade Amanda A., Hogg Michael A. Shared cognitions in small groups. Blackwell handbook of social psychology: Group processes 2001, 1-30. Blackwell Publishers, Oxford. M.A. Hogg, S. Tindale (Eds.).
31. Van Arensbergen P., van den Besselaar P. The selection of scientific talent in the allocation of research grants. Higher Education Policy 2012, 25:381-405.
32. Van Arensbergen P., van den Besselaar P., van der Weijden I. The selection of talent as a group process; a literature review on the dynamics of decision-making in grant panels. Research Evaluation 2014, 23(4):298-311.
33. Van Arensbergen P., van der Weijden I., van den Besselaar P. The notion of talent: What are the talents we are looking for in science?. Research Evaluation 2014, 23(4):273-284.
34. Van den Besselaar P. A quantitative analysis of the granting decisions in the NWO career grants program-Report commissioned by NWO, Amsterdam 2011, 2011. NWO, Amsterdam.
35. Van den Besselaar P. More competition, better science-On the predictive validity of grant selection. Translational twists and turns: Science as a socio-economic endeavor Proceedings 18th STI 2013, 385-392. S. Hinze (Ed.).
36. Van den Besselaar P., Leydesdorff L. Past performance, peer review, and project selection: A case study in the social and behavioral sciences 2007, Rathenau Institute, Den Haag.
37. Van den Besselaar P., Leydesdorff L. Past performance, peer review, and project selection: A case study in the social and behavioral sciences. Research Evaluation 2009, 18:273-288.
38. Van den Besselaar P., Sandström U., van der Weijden I. The independence indicator. Science & technology indicators 2012 2012, 131-141. OST & Science Metrix, Montreal. Archambault (Ed.).
39. Van den Besselaar, P & U Sandström (under review). Gendered differences in research performance and in careers: A Longitudinal Case Study.
40. Van den Besselaar, P & U Sandström (under review). Measuring independence of mid-career researchers: A proposal for an individual level bibliometric indicator.
41. Van den Brink M., Benschop Y. Gender practices in the construction of academic excellence: Sheep with five legs. Organization 2012, 19(4):p.507-p.524.
42. Van Leeuwen T.N., Moed H.F. Funding decisions, peer review, and scientific excellence in physical sciences, chemistry, and geosciences. Research Evaluation 2012, 21:189-198.
43. Waltman L., van Eck N.-J., Wouters P. Counting publications and citations: Is more always better?. Journal of Informetrics 2013, 7(3):635-641.
44. Waltman L., Van Eck N.J. A systematic empirical comparison of different approaches for normalizing citation impact indicators. Journal of Informetrics 2013, 7(4):833-849.
45. Wennerås C., Wold A. Nepotism and sexism in peer review. Nature 1997, 387(6631):341-343.