Overturning some assumptions about the effects of evaluation systems on publication performance

Scientometrics

Volumn 86, Issue 3, 2011, Pages 575-592

  Osuna, Carmen ^a   Cruz Castro, Laura ^a   Sanz Menéndez, Luis ^a  

^a Institute of Public Goods and Policies   (Spain)

Author keywords

CNEAI; Effects of evaluations; Quasi experimental design; Research evaluation systems; Scientific production; Spain

Indexed keywords

EID: 79251595679     PISSN: 01389130     EISSN: None     Source Type: Journal    
DOI: 10.1007/s11192-010-0312-7     Document Type: Article

References (50)

1. Andersen, L. B., & Pallesen, T. (2008). "Not Just for the Money?" How financial incentives affect the number of publications at Danish Research Institutions. International Public Management Journal, 11(1), 28-47.
2. Braun, T., & Schubert, A. (1988). World flash on basic research: Scientometric versus socio-economic indicators. Scatter plots for 51 countries. 1978-1980. Scientometrics, 13(1-2), 1-9.
3. Braun, T., Glänzel, W., & Schubert, A. (1985). Scientometrics indicators. A 32-country comparative evaluation of publishing performance and citation impact. Singapore, Philadelphia: World Scientific.
4. Butler, L. (2003a). Explaining Australia's increased share of ISI publications. The effects of a funding formula based on publication counts. Research Policy, 32(1), 143-155.
5. Butler, L. (2003b). Modifying publication practices in response to funding formulas. Research Evaluation, 12(1), 39-46.
6. Campbell, D. F. J. (2003). The evaluation of university research in the United Kingdom and the Netherlands, Germany and Austria. In Ph. Shapira & S. Kuhlman (Eds.), Learning from science and technology policy evaluation (pp. 98-131). Cheltenham (UK): Edward Elgar.
7. Campbell, D. T. (1996). Regression artifacts in time-series and longitudinal data. Evaluation and Program Planning, 19(4), 377-389.
8. Campbell, D. T., & Ross, H. L. (1968). The Connecticut crackdown on speeding. Time-series data in quasi-experimental analysis. Law and Society Review, 3(1), 32-54.
9. Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Boston: Houghton Mifflin Co.
10. Crespi, G. A., & Geuna, A. (2008). An empirical study of scientific production: A cross country analysis (1981-2002). Research Policy, 37(4), 565-579.
11. Cruz-Castro, L., & Sanz-Menéndez, L. (2007). Research evaluation in transition: individual versus organisational assessment in Spain. In R. Whitley & J. Glaser (Eds.), The changing governance of the sciences. The advent of the research evaluation systems (pp. 205-224). Dordrecht (NL): Springer.
12. Cruz-Castro, L., Sanz-Menéndez, L. & Martínez, C. (2010). Research centers in transition: patterns of convergence and diversity. The Journal of Technology Transfer, doi: 10. 1007/s10961-010-9168-5 (forthcoming).
13. Georghiou, L., Howells, J., Rigby, J. Glynn, S., Butler, J., Cameron, H., et al. (2000). Impact of the research assessment exercise and the future of quality assurance in the light of changes in the research landscape. A report produced by PREST, University of Manchester for HEFCE Higher Education Funding Council for England. April 2000. http://www. mbs. ac. uk/research/innovation/publications-archive/reports. aspx. Accessed 10 May 2010.
14. Geuna, A., & Martin, B. (2003). University research evaluation and funding: An international comparison. Minerva, 41(4), 277-304.
15. Gläser, J. (2007). The social orders of research evaluation systems. In R. Whitley & J. Glaser (Eds.), The changing governance of the sciences. The advent of the research evaluation systems (pp. 245-266). Dordrecht (NL): Springer.
16. Gómez, I., Fernández, M. T., Zulueta, M. A., & Camí, J. (1995). Analysis of biomedical research in Spain. Research Policy, 24(3), 459-471.
17. Gómez, I., Sancho, R., Bordons, M., & Fernández, M. T. (2006). La I+D en España a través de publicaciones y patentes. In J. Sebastián & E. Muñoz (Eds.), Radiografía de la investigación pública en España (pp. 275-302). Madrid: Biblioteca Nueva.
18. Hicks, D. (2009). Evolving regimes of multi-university research evaluation. Higher Education, 57(4), 393-404.
19. Ingwersen, P., & Jacobs, D. (2004). South African research in selected scientific areas: Status 1981-2000. Scientometrics, 59(3), 405-423.
20. Jiménez-Contreras, E., & Ferreiro-Aláez, L. (1996). Publishing abroad: Fair trade or short sell for non-english-speaking authors: A Spanish Study. Scientometrics, 36(1), 81-95.
21. Jiménez-Contreras, E., Moya-Anegón, F., & López-Cózar, E. (2003). The evolution of research activity in Spain. The impact of National Commission for the Evaluation of Research Activity (CNEAI). Research Policy, 32(1), 123-142.
22. Laudel, G. (2006). The art of getting funding: How scientists adapt to their funding conditions. Science and Public Policy, 33(7), 489-504.
23. Liefner, I. (2003). Funding, resources allocation, and performance in higher education systems. Higher Education, 46(4), 469-489.
24. Méndez, A., & Gómez, I. (1986). The Spanish scientific productivity through eight international databases. Scientometrics, 10(3-4), 207-219.
25. Moed, H. (2008). UK research assessment exercises: Informed judgements on research quality or quantity? Scientometric, 74(1), 153-161.
26. Moed, H. F., Debruin, R. E., & van Leeuwen, T. N. (1995). New bibliometric tools for the assessment of national research performance: Data base description, overview of indicators and first applications. Scientometrics, 33(3), 381-422.
27. Moed, H. F., Leeuwen, T. N., & Visser, M. S. (1999). Trends in publication output and impact of universities in the Netherlands. Research Evaluation, 8(1), 60-67.
28. Mohr, L. B. (2000). Regression artifacts and other customs of dubious desert. Evaluation and Program Planning, 23(4), 397-409.
29. Moya-Anegón, F., & Herrero-Solana, V. (1999). Science in America Latina: A comparison of bibliometric and scientific-technical indicators. Scientometrics, 46(2), 299-320.
30. Moya-Anegón, F., et al. (2007). Indicadores bibliométricos de la actividad científica Española (1990-2004). Madrid: FECYT.
31. Önder, C., Sevkli, M., Altinok, T., & Tavukçuoglu, C. (2008). Institutional change and scientific research: A preliminary bibliometric analysis of institutional influences on Turkey's recent social science publications. Scientometrics, 76(3), 543-560.
32. Persson, O., Glänzel, W., & Danell, R. (2004). Inflationary bibliometric values: The role of scientific collaboration and the need for relative indicators in evaluative studies. Scientometrics, 60(3), 421-432.
33. Rodriguez-Navarro, A. (2009). Sound research, unimportant discoveries: Research, universities, and formal evaluation of research in Spain. Journal of the American Society for Information Science and Technology, 60(9), 1845-1858.
34. Sanz-Menéndez, L. (1995a). Policy choices, institutional constraints and policy learning: The Spanish science and technology policy in the eighties. International Journal of Technology Management, 10(4/5/6), 255-274.
35. Sanz-Menéndez, L. (1995b). Research actors and the state: Research evaluation and evaluation of science and technology policies in Spain. Research Evaluation, 5(1), 79-88.
36. Sanz-Menéndez, L. (1997). Estado, ciencia y tecnología en España (1939-1997). Madrid: Alianza Editorial.
37. Sanz Menéndez, L., & Pfretzchner, J. (1992). Política científica y gestión de la investigación: El CSIC (1986-1990) en el sistema español de ciencia y tecnología. Arbor, 557(Mayo), 9-51.
38. Scimago, G. (2006). Producción ISI y tramos de investigación: Cómo combinarlos en un nuevo indicador. El profesional de la información, 15(3), 227-228.
39. Scimago, G. (2007). Producción ISI y tramos de investigación: Cómo combinarlos en un nuevo indicador (II). El profesional de la información, 16(4), 510-511.
40. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, New York: Houghton Mifflin Co.
41. Shelton, R. D. (2008). Relations between national research investment and publication output: Application to an American paradox. Scientometrics, 74(2), 191-205.
42. SRI International (1988). Research activity in Spain, Portugal and Greece. A 1988 Bibliometric model assessment. (Report prepared by Ailes, Coward and Fresne), Science and Technology Policy Programme, SRI International, Arlington (Vi), mimeo.
43. Talib, A. A. (2001). The continuing behavioural modification of academics since the 1992 research assessment exercise. Higher Education Review, 33(3), 30-46.
44. Taylor, J. (2001). The impact of performance indicators on the work of university academics: Evidence from Australian universities. Higher Education Quarterly, 55(1), 42-61.
45. van Raan, A. F. J. (1997). Science as an international enterprise. Science and Public Policy, 24(5), 290-300.
46. Westerheijden, D. (1997). A solid base for decisions: Use of the VSNU research evaluation in Dutch Universities. Higher Education, 33(4), 397-413.
47. Whitley, R. (2003). Competition and pluralism in public sciences: The impact of institutional frameworks on the organization of academic science. Research Policy, 32(6), 1015-1039.
48. Whitley, R. (2007). The changing governance of the public sciences: The consequences of research evaluation systems for knowledge production in different countries and scientific fields. In R. Whitley & J. Glaser (Eds.), The changing governance of the sciences. The advent of the research evaluation systems (pp. 3-27). Dordrecht (NL): Springer.
49. Whitley, R., & Glaser, J. (Eds.). (2007). The changing governance of the sciences. The advent of the research evaluation systems. Dordrecht (NL): Springer.
50. Zitt, M., Perrot, F., & Barré, R. (1998). The transition from "national" to "transnational" model and related measures of countries' performance. Journal of the American Society of Information Science, 49(1), 30-42.