How should the societal impact of research be generated and measured? A proposal for a simple and practicable approach to allow interdisciplinary comparisons

Scientometrics

Volumn 98, Issue 1, 2014, Pages 211-219

  Bornmann, Lutz ^a   Marx, Werner ^b  

^a MAX PLANCK INSTITUTE   (Germany)

^b MAX PLANCK INSTITUT FÜR FESTKÖRPERFORSCHUNG   (Germany)

Author keywords

Assessment report; IPCC; Research assessment; Social benefits; Societal impact

Indexed keywords

EID: 84891739298     PISSN: 01389130     EISSN: None     Source Type: Journal    
DOI: 10.1007/s11192-013-1020-x     Document Type: Article

References (40)

1. Bell, S., Shaw, B., & Boaz, A. (2011). Real-world approaches to assessing the impact of environmental research on policy. Research Evaluation, 20(3), 227-237.
2. Bornmann, L. (2011). Scientific peer review. Annual Review of Information Science and Technology, 45, 199-245.
3. Bornmann, L. (2012). Measuring the societal impact of research. EMBO Reports, 13(8), 673-676.
4. Bornmann, L. (2013). What is societal impact of research and how can it be assessed? A literature survey. Journal of the American Society of Information Science and Technology, 64(2), 217-233.
5. Bornmann, L., & Daniel, H.-D. (2008). What do citation counts measure? A review of studies on citing behavior. Journal of Documentation, 64(1), 45-80. doi: 10. 1108/00220410810844150.
6. Bornmann, L., Mutz, R., & Daniel, H.-D. (2007). Gender differences in grant peer review: A meta-analysis. Journal of Informetrics, 1(3), 226-238. doi: 10. 1016/j. joi. 2007. 03. 001.
7. Bornmann, L., Mutz, R., Hug, S. E., & Daniel, H. D. (2011). A meta-analysis of studies reporting correlations between the h index and 37 different h index variants. Journal of Informetrics, 5(3), 346-359. doi: 10. 1016/j. joi. 2011. 01. 006.
8. Bowman, N. A. (2012). Effect sizes and statistical methods for meta-analysis in higher education. Research in Higher Education, 53(3), 375-382. doi: 10. 1007/s11162-011-9232-5.
9. Brown, P. (2012). Nothing but the truth. EMBO Reports, 13(11), 964-967.
10. Commission, European. (2010). Assessing Europe's university-based research. Publications Office of the European Union: Expert group on assessment of university-based research. Brussels.
11. Council of Canadian Academies. (2012). Informing research choices: Indicators and judgment: The expert panel on science performance and research funding. Ottawa: Council of Canadian Academies.
12. Donovan, C. (2011). State of the art in assessing research impact: Introduction to a special issue. Research Evaluation, 20(3), 175-179.
13. Erno-Kjolhede, E., & Hansson, F. (2011). Measuring research performance during a changing relationship between science and society. Research Evaluation, 20(2), 131-143. doi: 10. 3152/095820211x12941371876544.
14. Glass, G. V. (1976). Primary, secondary, and meta-analysis. Educational Researcher, 5, 3-8.
15. Godin, B., & Dore, C. (2005). Measuring the impacts of science; beyond the economic dimension, INRS Urbanisation, Culture et Société. Paper presented at the HIST Lecture, Helsinki Institute for Science and Technology Studies, Helsinki, Finland. http://www. csiic. ca/PDF/Godin_Dore_Impacts. pdf. Accessed 6 April 2013.
16. Gregersen, B., Linde, L. T., & Rasmussen, J. G. (2009). Linking between Danish universities and society. Science and Public Policy, 36(2), 151-156. doi: 10. 3152/030234209x406818.
17. Higher Education Funding Council for England. (2009). Research Excellence Framework. Second consultation on the assessment and funding of research. Higher Education Funding Council for England (HEFCE): Bristol.
18. Higher Education Funding Council for England. (2011). Decisions on assessing research impact. Higher Education Funding Council for England (HEFCE): Bristol.
19. Holbrook, J. B. (2010). The use of societal impacts considerations in grant proposal peer review: A comparison of five models. Technology & Innovation, 12(3), 213-224. doi: 10. 3727/194982410x12895770314078.
20. Holbrook, J. B., & Frodeman, R. (2010). Comparative Assessment of Peer Review (CAPR). EU/US workshop on peer review: Assessing "broader impact" in research grant applications. Brussels: European Commission, Directorate-General for Research and Innovation.
21. Holbrook, J. B., & Frodeman, R. (2011). Peer review and the ex ante assessment of societal impacts. Research Evaluation, 20(3), 239-246.
22. Lähteenmäki-Smith, K., Hyytinen, K., Kutinlahti, P., & Konttinen, J. (2006). Research with an impact evaluation practises in public research organisations. Kemistintie: VTT Technical Research Centre of Finland.
23. Marsh, H. W., Bornmann, L., Mutz, R., Daniel, H. D., & O'Mara, A. (2009). Gender effects in the peer reviews of grant proposals: A comprehensive meta-analysis comparing traditional and multilevel approaches. Review of Educational Research, 79(3), 1290-1326. doi: 10. 3102/0034654309334143.
24. Martin, B. R. (2011). The research excellence framework and the 'impact agenda': Are we creating a Frankenstein monster? Research Evaluation, 20(3), 247-254.
25. Matt, G. E., & Navarro, A. M. (1997). What meta-analyses have and have not taught us about psychotherapy effects: A review and future directions. Clinical Psychology Review, 17(1), 1-32.
26. Merton, R. K. (1973). The sociology of science: Theoretical and empirical investigations. Chicago: University of Chicago Press.
27. Mervis, J. (2011). Beyond the data. Science, 334(6053), 169-171. doi: 10. 1126/science. 334. 6053. 169.
28. Meyers, R. A. (Ed.). (2012). Encyclopedia of sustainability science and technology. New York: Springer.
29. Mostert, S., Ellenbroek, S., Meijer, I., van Ark, G., & Klasen, E. (2010). Societal output and use of research performed by health research groups. Health Research Policy and Systems, 8(1), 30.
30. Niederkrotenthaler, T., Dorner, T. E., & Maier, M. (2011). Development of a practical tool to measure the impact of publications on the society based on focus group discussions with scientists. BMC Public Health, 11, 588.
31. Petit, J. C. (2004). Why do we need fundamental research? European Review, 12(2), 191-207.
32. Piwowar, H. (2013). Altmetrics: Value all research products. Nature, 493(7431), 159.
33. Robinson, K. A., & Goodman, S. N. (2011). A systematic examination of the citation of prior research in reports of randomized, controlled trials. Annals of Internal Medicine, 154(1), 50-55.
34. RQF development advisory group. (2006). Research quality framework: Assessing the quality and impact of research in Australia. The recommended RQF (report by the RQF Development Advisory Group). Canberra, Australia: Department of Education, Science and Training.
35. Rymer, L. (2011). Measuring the impact of research-the context for metric development. Turner: The Group of Eight.
36. Salter, A. J., & Martin, B. R. (2001). The economic benefits of publicly funded basic research: A critical review. Research Policy, 30(3), 509-532.
37. Smith, C. H. L. (1997). What's the use of basic science? Retrieved 6 April 2013, from http://wwwnew. jinr. ru/section. asp?sd_id=94.
38. Smith-Spangler, C., Brandeau, M. L., Hunter, G. E., Bavinger, J. C., Pearson, M., Eschbach, P. J., et al. (2012). Are organic foods safer or healthier than conventional alternatives? A systematic review. Annals of Internal Medicine, 157(5), 348-366. doi: 10. 7326/0003-4819-157-5-201209040-00007.
39. van der Meulen, B., & Rip, A. (2000). Evaluation of societal quality of public sector research in the Netherlands. Research Evaluation, 9(1), 11-25.
40. Ziman, J. (2000). Real science. What it is, and what it means. Cambridge: Cambridge University Press.