What difference does it make? Impact of peer-reviewed scholarships on scientific production

Scientometrics

Volumn 102, Issue 2, 2015, Pages 1167-1188

  Bin, Adriana ^a   Salles Filho, Sergio ^a   Capanema, Luiza Maria ^b   Colugnati, Fernando Antonio Basile ^c  

^a UNIVERSITY OF CAMPINAS   (Brazil)

^b INSTITUTO AGRONÔMICO   (Brazil)

^c FEDERAL UNIVERSITY OF JUIZ DE FORA   (Brazil)

Author keywords

Evaluation; Impact; Peer review; Propensity score; Scholarships; Scientific production

Indexed keywords

EID: 84921851833     PISSN: 01389130     EISSN: 15882861     Source Type: Journal    
DOI: 10.1007/s11192-014-1462-9     Document Type: Article

References (59)

1. Abramo, G., D’Angelo, C. A., & Costa, F. (2010). Citations versus journal impact factor as proxy of quality: Could the latter ever be preferable? Scientometrics,84, 821–833.
2. Asian Development Bank (ADB). (2007). Evaluation study: Japan scholarship program. Philippines: ABD.
3. Amos, L. B., Windham, A. M., Reyes, I. B., Jones, W., & Baran, V. (2009). An impact evaluation of the gates millennium scholars program. Washington, DC: Bill & Melinda Gates Foundation.
4. Arnold, E. (2012). Understanding long-term impacts of R&D funding: The EU framework programme. Research Evaluation,21, 332–343.
5. Auriol, L., Misu, M., & Freeman, R. (2012). Doctoral graduates in times of economic downturn: labour market participation and mobility. Working Party of National Experts on Science and Technology Indicators. Directorate for Science, Technology and Industry-Committee for Scientific and Technological Policy: OECD.
6. Bach, L. (2012). The frontiers of evaluation: Some considerations on the European case. Revista Brasileira de Inovação,11, 67–84.
7. Biotechnology and Biological Sciences Research Council (BBSRC). (2011). David Phillips fellowship scheme. Swindon: BBSRC.
8. Bernal, J. D. (1954). Science in history. Cambridge: MIT Press.
9. Böhmer, S., & von Ins, M. (2009). Different—Not just by label: Research-oriented academic careers in Germany. Research Evaluation,18(3), 177–184.
10. Bornmann, L., Leydesdorff, L., & Besselaar, P. V. (2010). A meta-evaluation of scientific research proposals: Different ways of comparing rejected to awarded applications. Journal of Informetrics,4, 211–220.
11. Brody, S. (2013). Impact factor: Imperfect but not yet replaceable. Scientometrics,96, 255–257.
12. Burgoine, T., Hopkins, P., Rech, M. F., & Zapata, G. P. (2011). ‘These kids can’t write abstracts’: Reflections on a postgraduate writing and publishing workshop. Area,43(4), 463–469.
13. Bush, V. (1945). Science, the endless frontier: A report to the president on a program for postwar scientific research. Washington: United States Government Printing Office.
14. CIDA. (2005). Evaluation of the Canadian Francophonie Scholarship Program (CFSP), 1987–2005. Canada: CIDA.
15. Colugnati, F. A. B., Silva, A. M. A. C., SallesFilho, S. L. M. (2011). Multidimensional evaluation of a program for early-career researcher in Brazil—the young investigator in emerging centers program.In Atlanta Conference on Science and Innovation Policy. Atlanta: IEEE Conference Publications.
16. Edler, J., Georghiou, L., Blind, K., & Uyarra, E. (2012). Evaluating the demand side: New challenges for evaluation. Research Evaluation,21, 33–47.
17. Feller, I. (2013). Peer review and expert panels as techniques for evaluating the quality of academic research. In A. N. Link & N. S. Vonortas (Eds.), Handbook on the Theory and Practice of Program Evaluation (pp. 115–142). Cheltenham: Edward Elgar.
18. Friedman, J. H. (2001). Greedy function approximation: A gradient boosting machine. Annals of Statistics,29(5), 1189–1232.
19. Garfield, E. (1999). Journal impact factor: A brief review. Journal of the Canadian Medical Association,161(8), 979–980.
20. Goldsmith, S. S., Presley, J. B., & Cooley, E. A. (2002). National Science Foundation Graduate Research Fellowship Program, Final Evaluation Report. Virginia: NSF.
21. Halse, C., & Mowbray, S. (2011). The impact of the doctorate. Studies in Higher Education,36(5), 513–525.
22. Heinze, T. (2008). How to sponsor ground-breaking research: a comparison of funding schemes. Science and Public Policy,35(5), 302–318.
23. Hicks, D., & Melkers, J. (2013). Bibliometrics as a tool for research evaluation. In A. N. Link & N. S. Vonortas (Eds.), Handbook on the theory and practice of program evaluation (pp. 323–349). Cheltenham: Edward Elgar.
24. Jacob, B. A., & Lefgren, L. (2011). The impact of NIH postdoctoral training grants on scientific productivity. Research Policy,40, 864–874.
25. Jarvey, P., Usher, A., & Mcelroy, L. (2012). Making research count: Analyzing canadian academic publishing cultures. Toronto: Higher Education Strategy Associates.
26. Kamler, B. (2008). Rethinking doctoral publication practices: Writing from and beyond the thesis. Studies in Higher Education,33(3), 283–294.
27. Kostoff, R., Averch, H., & Chubin, D. (1994). Research impact assessment: Introduction and overview. Evaluation Review,18(1), 3–10.
28. Lee, H., Miozzo, M., & Laredo, P. (2010). Career patterns and competences of PhDs in science and engineering in the knowledge economy: The case of graduates from a UK research-based university. Research Policy,39, 869–881.
29. Leydesdorff, L. (2012). Alternatives to the journal impact factor: I3 and the top-10% (or top-25%?) of the most-highly cited papers. Scientometrics,92, 355–365.
30. Melin, G., & Danell, R. (2006). The top eight percent: Development of approved and rejected applicants for a prestigious grant in Sweden. Science and Public Policy,33(10), 702–712.
31. Moral, F., & Pombo, N. (2011). Informe sociológico sobre la encuesta del valor social de las becas de la Fundación Carolina. Madrid: Fundación Carolina.
32. Mowery, D. C., & Rosenberg, N. (1998). Paths of innovation: Technological change in 20th-century America. Cambridge: Cambridge University Press.
33. Mutz, R., & Daniel, H.-D. (2012). The generalized propensity score methodology for estimating unbiased journal impact factors. Scientometrics,92, 377–390.
34. Navarra-Madsen, J., Bales, R. A., & Hynds, D. L. (2010). Role of scholarships in improving success rates of undergraduate science, technology, engineering and mathematics (STEM) majors. Procedia Social and Behavioral Sciences,8, 458–464.
35. Nelder, J., & Wedderburn, R. (1972). Generalized linear models. Journal of the Royal Statistical Society,135(3), 370–384.
36. Netting, F. E., & Nichols-Casebolt, A. (1997). Authorship and collaboration. Journal of Social Work Education,33(3), 555–564.
37. Neufeld, J., & von Ins, M. (2011). Informed peer review and uninformed bibliometrics? Research Evaluation,20(5), 365–375.
38. Neumann, R., & Tan, K. K. (2011). From PhD to initial employment: The doctorate in a knowledge economy. Studies in Higher Education,36(5), 601–614.
39. Opthof, T., & Leydesdorff, L. (2011). A comment to the paper by Waltman et al. Scientometrics,87, 467–481.
40. Pavitt, K. (1991). What makes basic research economic useful? Research Policy,20, 109–119.
41. Pinheiro, D., Melkers, J., & Youtie, J. (2012). Learning to play the game: Student publishing as an indicator of future scholarly success. Technological Forecasting and Social Change,81, 56–66.
42. Price, D. J. S. (1963). Little science, big science. New York: Columbia University Press.
43. Ridgeway, G. (1999). The state of boosting. Computing Science and Statistics,31, 172–181.
44. Ridgeway, G. (2006). Assessing the effect of race bias in post-traffic stop outcomes using propensity scores. Journal of Quantitative Criminology,22(1), 1–29.
45. Ridgeway, G. (2013). Package GBM. http://cran.r-project.org/web/packages/gbm/gbm.pdf. Acessed 18 November 2013.
46. Rigby, J. (2011). Systematic grant and funding body acknowledgement data for publications: New dimensions and new controversies for research policy and evaluation. Research Evaluation,20(5), 365–375.
47. Rigby, J. (2013). Looking for the impact of peer review: Does count of funding acknowledgements really predict research impact? Scientometrics,94, 57–73.
48. Roach, M., & Sauermann, H. (2010). A taste for science? PhD scientists’ academic orientation and self-selection into research careers in industry. Research Policy,39, 422–434.
49. Rosenbaum, P., & Rubin, D. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika,70(1), 41–55.
50. Salazar, H. J. (2010). Estudio sobre resultados e impactos de los programas de apoyo a la formación de posgrado en Colombia: hacia una agenda de evaluación de calidad. In L. Luchilo (Ed.), Formación de posgrado en América Latina: Políticas de apoyo resultados e impactos (pp. 117–176). Buenos Aires: Eudeba.
51. Salter, A. J., & Martin, B. R. (2001). The economic benefits of publicly funded basic research: A critical review. Research Policy,30, 509–532.
52. Schulz, P. A., & Manganote, E. J. T. (2012). Revisiting country research profiles: Learning about the scientific cultures. Scientometrics,93, 517–531.
53. Statcom Estadísticos Consultores (STATCOM). (2007). Evaluación en profundidad—programas de becas de postgrado. Santiago de Chile: STATCOM.
54. Thompson, D. F., Callen, E. C., & Nahata, M. C. (2009). New indices in scholarship assessment. American Journal of Pharmaceutical Education,73(6), 1–5.
55. Van Raan, A. F. J. (1996). Advanced bibliometric methods as quantitative core of peer review based evaluation and foresight exercises. Scientometrics,36, 397–420.
56. Van Raan, A. F. J. (2005). Measuring science. In H. F. Moed, W. Glänzel, & U. Schmoch (Eds.), Handbook of quantitative science and technology research: The use of publication and patent statistics in studies of S&T systems (pp. 19–50). EUA: Springer Science+Business Media.
57. Vitae, (2010). What do researchers do? Doctoral graduate destinations and impact three years on 2010. RCUK: The Careers Research and Advisory Centre.
58. Waltman, L., Van Eck, N. J., Van Leeuwen, T. N., Visser, M. S., & Van Raan, A. F. J. (2011). On the correlation between bibliometric indicators and peer review: Reply to Opthof and Leydesdorff. Scientometrics,88, 1017–1022.
59. White, H. (1980). A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica,48, 817–830.