What is societal impact of research and how can it be assessed? a literature survey

Journal of the American Society for Information Science and Technology

Volumn 64, Issue 2, 2013, Pages 217-233

  Bornmann, Lutz ^a  

^a MAX PLANCK INSTITUTE   (Germany)

Author keywords

social sciences

Indexed keywords

DIAGNOSTIC TOOLS; ECONOMIC RETURNS; KNOWLEDGE TRANSFER; LITERATURE SURVEY; PUBLIC VALUES; RESEARCH EVALUATION; RESEARCH OUTCOME; RESEARCH STUDIES; SOCIETAL BENEFITS; SOCIETAL IMPACTS;

ENVIRONMENTAL IMPACT ASSESSMENTS; KNOWLEDGE MANAGEMENT; SOCIAL SCIENCES; SURVEYS;

RESEARCH;

EID: 84872975676     PISSN: 15322882     EISSN: 15322890     Source Type: Journal    
DOI: 10.1002/asi.22803     Document Type: Review

References (101)

1. Abbott, A., Cyranoski, D., Jones, N., Maher, B., Schiermeier, Q., &; van, Noorden, R., (2010). Metrics: Do metrics matter? Nature, 465, 860-862.
2. Appleseed Inc. (2003). Engines of economic growth: The economic impact of Boston's eight research universities on the metropolitan Boston area. Boston, MA: Author.
3. Australian Research Council. (2011). ERA 2012 submission guidelines, Excellence in Research for Australia. Canberra, Australia: Author.
4. Australian Research Council. (2012). Linkage Projects funding rules for funding commencing in 2012 variation (No. 1). Canberra, Australia: Author.
5. Austrian Science Fund. (2007). Rethinking the impact of basic research on society and the economy. Vienna, Austria: Author.
6. Barré, R., (2005). S&;T Indicators for policy making in a changing science-society relationship. In, H. Moed, W. Glänzel, &;, U. Schmoch, (Eds.), Handbook of quantitative science and technology research (pp. 115-131). Dordrecht, The Netherlands: Springer.
7. Beise, M., &;, Stahl, H., (1999). Public research and industrial innovations in Germany. Research Policy, 28 (4), 397-422. doi: 10.1016/s0048-7333(98)00126-7
8. Bell, S., Shaw, B., &;, Boaz, A., (2011). Real-world approaches to assessing the impact of environmental research on policy. Research Evaluation, 20 (3), 227-237.
9. Bensing, J.M., Caris-Verhallen, W.M.C.M., Dekker, J., Delnoij, D.M.J., &;, Groenewegen, P.P., (2003). Doing the right thing and doing it right: Toward a framework for assessing the policy relevance of health services research. International Journal of Technology Assessment in Health Care, 19 (04), 604-612. doi: 10.1017/S0266462303000564
10. Biotechnology and Biological Sciences Research Council. (2012). BBSRC policy on maximising the impact of research. Swindon, United Kingdom: Author.
11. Bornmann, L., (2010). Mimicry in science? Scientometrics, 86 (1), 173-177.
12. Bornmann, L., (2011). Scientific peer review. Annual Review of Information Science and Technology, 45, 199-245.
13. Bornmann, L., (2012). Measuring the societal impact of research. EMBO Reports, 13 (8), 673-676.
14. Bornmann, L., Mutz, R., Neuhaus, C., &;, Daniel, H.-D., (2008). Use of citation counts for research evaluation: Standards of good practice for analyzing bibliometric data and presenting and interpreting results. Ethics in Science and Environmental Politics, 8, 93-102. doi: 10.3354/esep00084
15. Bozeman, B., &;, Sarewitz, D., (2011). Public value mapping and science policy evaluation. Minerva, 49 (1), 1-23. doi: 10.1007/s11024-011-9161-7
16. Burke, J., Bergman, J., &;, Asimov, I., (1985). The impact of science on society. Washington, DC: National Aeronautics and Space Administration.
17. Bush, V., (1945). Science: The endless frontier. [A report to President Truman outlining his proposal for post-war U.S. science and technology policy.] Washington, DC: United States Government Printing Office.
18. Buxton, M., (2011). The payback of "Payback": Challenges in assessing research impact. Research Evaluation, 20 (3), 259-260.
19. Buxton, M., &;, Hanney, S., (1994). Assessing payback from Department of Health Research and Development: Preliminary report: Vol. 1. The main report (Research Report, No. 19). Uxbridge, United Kingdom: HERG, Brunel University.
20. Buxton, M., &;, Hanney, S., (1996). How can payback from health services research be assessed? Journal of Health Services Research &; Policy, 1 (1), 35-43.
21. Centre for Quality and Change Management. (2011). Final report of Delphi study. E3M project-European indicators and ranking methodology for university third mission. Valencia and Leõn, Spain: Universidad Politécnica de Valencia and the Universidad de Leõn.
22. Committee on Prospering in the Global Economy of the 21st Century. (2007). Rising above the gathering storm: Energizing and employing America for a brighter economic future. National Academy of Sciences, National Academy of Engineering, Institute of Medicine. Washington, DC: National Academies Press.
23. Danish Council. (2006). A tool for assessing research quality and relevance. Copenhagen, Denmark: Danish Council for Research Policy.
24. Department of Education, Science and Training. (2005). Research quality framework: Assessing the quality and impact of research in Australia (Issue paper). Canberra: Commonwealth of Australia.
25. Department of Education, Science and Training. (2006). Research quality framework: Assessing the quality and impact of research in Australia. Research Impact (Report by the RQF Development Advisory Group). Canberra: Commonwealth of Australia.
26. Donovan, C., (2007). The qualitative future of research evaluation. Science and Public Policy, 34 (8), 585-597. doi: 10.3152/030234207X256538
27. Donovan, C., (2008). The Australian Research Quality Framework: A live experiment in capturing the social, economic, environmental, and cultural returns of publicly funded research. New Directions for Evaluation, 2008 (118), 47-60. doi: 10.1002/ev.260
28. Donovan, C., (2011). State of the art in assessing research impact: Introduction to a special issue. Research Evaluation, 20 (3), 175-179.
29. Donovan, C., &;, Hanney, S., (2011). The "Payback Framework" explained. Research Evaluation, 20 (3), 181-183.
30. Evaluating Research in Context (ERiC). (2010). Evaluating the societal relevance of academic research: A guide. Delft, The Netherlands: Delft University of Technology.
31. Erno-Kjolhede, E., &;, Hansson, F., (2011). Measuring research performance during a changing relationship between science and society. Research Evaluation, 20 (2), 131-143. doi: 10.3152/095820211x12941371876544
32. European Commission. (2010). Assessing Europe's university-based research. Expert group on assessment of university-based research. Brussels, Belgium: Publications Office of the European Union.
33. Frank, C., &;, Nason, E., (2009). Health research: Measuring the social, health and economic benefits. Canadian Medical Association Journal, 180 (5), 528-534. doi: 10.1503/Cmaj.090016
34. Frodeman, R., &;, Briggle, A., (2012). The dedisciplining of peer review. Minerva, 50 (1), 3-19. doi: 10.1007/s11024-012-9192-8
35. Gibbons, M., Limoges, C., Nowotny, H., Schwartzman, S., Scott, P., &;, Trow, M., (1994). The new production of knowledge: The dynamics of science and research in contemporary societies. London, United Kingdom: Sage.
36. Godin, B., &;, Doré, C., (2005). Measuring the impacts of science; beyond the economic dimension, INRS Urbanisation, Culture et Société. Paper presented at the HIST Lecture, Helsinki Institute for Science and Technology Studies, Helsinki, Finland. Available at: http://www.csiic.ca/PDF/Godin-Dore-Impacts.pdf
37. Göransson, B., Maharajh, R., &;, Schmoch, U., (2009). New activities of universities in transfer and extension: Multiple requirements and manifold solutions. Science and Public Policy, 36 (2), 157-164. doi: 10.3152/030234209x406863
38. Grant, J., (1999). Evaluating the outcomes of biomedical research on healthcare. Research Evaluation, 8 (1), 33-38.
39. Grant, J., Brutscher, P.-B., Kirk, S., Butler, L., &;, Wooding, S., (2009). Capturing research impacts. Cambridge, United Kingdom: RAND Europe.
40. Gregersen, B., Linde, L.T., &;, Rasmussen, J.G., (2009). Linking between Danish universities and society. Science and Public Policy, 36 (2), 151-156. doi: 10.3152/030234209x406818
41. Hanney, S., Packwood, T., &;, Buxton, M., (2000). Evaluating the benefits from health research and development centres: A categorization, a model and examples of application. Evaluation, 6 (2), 137-160.
42. Health Economics Research Group, Office of Health Economics, RAND Europe. (2008). Medical research: What's it worth? Estimating the economic benefits from medical research in the UK. London, United Kingdom: Evaluation Forum.
43. Higher Education Funding Council for England. (2009). Research Excellence Framework. Second consultation on the assessment and funding of research. September 2009/38. Bristol: Higher Education Funding Council for England.
44. Higher Education Funding Council for England. (2011). Decisions on assessing research impact. Bristol: Higher Education Funding Council for England.
45. Higher Education Funding Council for England. (2012). Panel criteria and working methods. Bristol: Higher Education Funding Council for England.
46. Holbrook, J.B., (2010). The use of societal impacts considerations in grant proposal peer review: A comparison of five models. Technology &; Innovation, 12 (3), 213-224. doi: 10.3727/194982410x12895770314078
47. Holbrook, J., (2012). Re-assessing the science-society relation: The case of the US National Science Foundation's broader impacts merit review criterion (1997-2011). Retrieved from http://www.scienceofsciencepolicy.net/system/files/ attachments/Holbrook-BIC-2.0-final.pdf
48. Holbrook, J.B., &;, Frodeman, R., (2010). Comparative Assessment of Peer Review (CAPR). EU/US Workshop on Peer Review: Assessing "broader impact" in research grant applications. Brussels, Belgium: European Commission, Directorate-General for Research and Innovation.
49. Holbrook, J.B., &;, Frodeman, R., (2011). Peer review and the ex ante assessment of societal impacts. Research Evaluation, 20 (3), 239-246.
50. Kamenetzky, J.R., (in press). Opportunities for impact: Statistical analysis of the National Science Foundation's broader impacts criterion. Science and Public Policy. doi: 10.1093/scipol/scs059
51. Klautzer, L., Hanney, S., Nason, E., Rubin, J., Grant, J., &;, Wooding, S., (2011). Assessing policy and practice impacts of social science research: The application of the Payback Framework to assess the Future of Work programme. Research Evaluation, 20 (3), 201-209.
52. Kuruvilla, S., Mays, N., Pleasant, A., &;, Walt, G., (2006). Describing the impact of health research: A Research Impact Framework. BMC Health Services Research, 6.
53. Lähteenmäki-Smith, K., Hyytinen, K., Kutinlahti, P., &;, Konttinen, J., (2006). Research with an impact evaluation practises in public research organisations, Kemistintie, Finland: VTT Technical Research Centre of Finland.
54. Lamm, G.M., (2006). Innovation works. A case study of an integrated pan-European technology transfer model. B.I.F. Futura, 21 (2), 86-90.
55. Lewison, G., &;, Sullivan, R., (2008). The impact of cancer research: How publications influence UK cancer clinical guidelines. British Journal of Cancer, 98 (12), 1944-1950.
56. Leydesdorff, L., (2012). The Triple Helix of university-industry- government relations. Amsterdam, The Netherlands: Amsterdam School of Communications Research.
57. Link, A.N., &;, Scott, J.T., (2011). The theory and practice of public-sector R&;D economic impact analysis. Gaithersburg, MD: National Institute of Standards and Technology.
58. Macilwain, C., (2010). What science is really worth. Nature, 465 (7299), 682-684.
59. Mansfield, E., (1991). Academic research and industrial innovation. Research Policy, 20 (1), 1-12. doi: 10.1016/0048-7333(91)90080-a
60. Mansfield, E., (1998). Academic research and industrial innovation: An update of empirical findings. Research Policy, 26 (7-8), 773-776. doi: 10.1016/s0048-7333(97)00043-7
61. Mardis, M.A., Hoffman, E.S., &;, McMartin, F.P., (2012). Toward broader impacts: Making sense of NSF's merit review criteria in the context of the National Science Digital Library. Journal of the American Society for Information Science and Technology, 63 (9), 1758-1772. doi: 10.1002/asi.22693
62. Martin, B.R., (2007). Assessing the impact of basic research on society and the economy. Paper presented at the Rethinking the impact of basic research on society and the economy (WF-EST International Conference, 11 May 2007), Vienna, Austria.
63. Martin, B.R., (2011). The Research Excellence Framework and the "impact agenda": Are we creating a Frankenstein monster? Research Evaluation, 20 (3), 247-254.
64. May, R.M., (1998). The scientific investments of nations. Science, 281 (5373), 49-51.
65. Mervis, J., (2011). Beyond the data. Science, 334 (6053), 169-171. doi: 10.1126/science.334.6053.169
66. Miranda, L.C.M., &;, Lima, C.A.S., (2010). On trends and rhythms in scientific and technological knowledge evolution: A quantitative analysis. International Journal of Technology Intelligence and Planning, 6 (1), 76-109.
67. Molas-Gallart, J., Salter, A., Patel, P., Scott, A., &;, Duran, X., (2002). Measuring third stream activities. Final report to the Russell Group of universities. Brighton, United Kingdom: Science and Technology Policy Research Unit, University of Sussex.
68. Molas-Gallart, J., &;, Tang, P., (2011). Tracing "productive interactions" to identify social impacts: An example from the social sciences. Research Evaluation, 20 (3), 219-226.
69. Montada, L., Krampen, G., &;, Burkard, P., (1999). Personal and social orientations of psychology college teachers on evaluative criteria for own job performances: Results of an expert survey in German graduate psychology college teachers. Psychologische Rundschau, 50 (2), 69-89.
70. Mostert, S., Ellenbroek, S., Meijer, I., van, Ark, G., &;, Klasen, E., (2010). Societal output and use of research performed by health research groups. Health Research Policy and Systems, 8 (1), 30.
71. Narin, F., Hamilton, K.S., &;, Olivastro, D., (1997). The increasing linkage between US technology and public science. Research Policy, 26 (3), 317-330. doi: 10.1016/s0048-7333(97)00013-9
72. Nason, E., Curran, B., Hanney, S., Janta, B., Hastings, G., O'Driscoll, M., &;, Wooding, S., (2011). Evaluating health research funding in Ireland: Assessing the impacts of the Health Research Board of Ireland's funding activities. Research Evaluation, 20 (3), 193-200.
73. Niederkrotenthaler, T., Dorner, T.E., &;, Maier, M., (2011). Development of a practical tool to measure the impact of publications on the society based on focus group discussions with scientists. BMC Public Health, 11, 588.
74. Nightingale, P., &;, Scott, A., (2007). Peer review and the relevance gap: Ten suggestions for policy-makers. Science and Public Policy, 34 (8), 543-553. doi: 10.3152/030234207x254396
75. Pålsson, C.M., Göransson, B., &;, Brundenius, C., (2009). Vitalizing the Swedish university system: Implementation of the "third mission." Science and Public Policy, 36 (2), 145-150.
76. Petit, J.C., (2004). Why do we need fundamental research? European Review, 12 (2), 191-207.
77. Roberts, M.R., (2009). Realizing societal benefit from academic research: Analysis of the National Science Foundation's broader impacts criterion. Social Epistemology, 23 (3-4), 199-219. doi: 10.1080/02691720903364035
78. Rowe, G., &;, Wright, G., (2001). Expert opinions in forecasting: The role of the Delphi technique. In, J.S. Armstrong, (Ed.), Principles of forecasting: A handbook for researchers and practitioners (pp. 125-144). Norwell, MA: Kluwer Academic.
79. Royal Netherlands Academy of Arts and Sciences, Association of Universities in the Netherlands. (2010). Standard Evaluation Protocol 2009-2015. Protocol for research assessment in the Netherlands. Amsterdam, The Netherlands: Author.
80. Royal Society. (2011). Knowledge, networks and nations: Global scientific collaboration in the 21st century. London, United Kingdom: Author.
81. Ruegg, R., &;, Feller, I., (2003). A toolkit for evaluating public R&;D investment: Models, methods, and findings from ATP's first decade. Gaithersburg, MD: National Institute of Standards and Technology.
82. Rymer, L., (2011). Measuring the impact of research-The context for metric development. Turner, Australia: The Group of Eight.
83. Salter, A.J., &;, Martin, B.R., (2001). The economic benefits of publicly funded basic research: A critical review. Research Policy, 30 (3), 509-532.
84. Schapper, C.C., Dwyer, T., Tregear, G.W., Aitken, M., &;, Clay, M.A., (2012). Research performance evaluation: The experience of an independent medical research institute. Australian Health Review, 36, 218-223. doi: 10.1071/AH11057
85. Scott, J.E., Blasinsky, M., Dufour, M., Mandal, R.J., &;, Philogene, G.S., (2011). An evaluation of the Mind-Body Interactions and Health Program: Assessing the impact of an NIH program using the Payback Framework. Research Evaluation, 20 (3), 185-192.
86. SISOP. (2011). Conceptualizing the social impact of science. Retrieved from http://sisobproject.wordpress.com/2011/07/20/conceptualizing-the-social- impact-of-science-2/
87. Smith, C.H.L., (1997). What's the use of basic science? Retrieved from http://wwwnew.jinr.ru/section.asp?sd-id=94
88. Smith, R., (2001). Measuring the social impact of research. British Medical Journal, 323 (7312), 528. doi: 10.1136/bmj.323.7312.528
89. Social Impact Assessment Methods for Research and Funding Instruments Through the Study of Productive Interactions (SIAMPI). (2010, December). SIAMPI Workshop, Brussels. Retrieved from http://www.siampi.eu/Content/SIAMPI/ Report%20SIAMPI%20workshop.pdf
90. Social Impact Assessment Methods for Research and Funding Instruments Through the Study of Productive Interactions (SIAMPI). (2011). Final report on social impacts of research. Retrieved from http://www.siampi.edu/
91. Spaapen, J., &;, van Drooge, L., (2011). Introducing "productive interactions" in social impact assessment. Research Evaluation, 20 (3), 211-218.
92. Spaapen, J.B., Dijstelbloem, H., &;, Wamelink, F., (2007). Evaluating research in context: A method for comprehensive assessment. The Hague, The Netherlands: Consultative Committee of Sector Councils for Research and Development.
93. Stephan, P., (2012). How economics shapes science. Cambridge, MA: Harvard University Press.
94. Toole, A.A., (2012). The impact of public basic research on industrial innovation: Evidence from the pharmaceutical industry. Research Policy, 41 (1), 1-12. doi: 10.1016/j.respol.2011.06.004
95. United Nations Development Programme. (2010). The millennium development goals. New York, NY: Author.
96. United States Government Accountability Office. (2012). Designing evaluations. Washington, DC: Author.
97. van der Meulen, B., &;, Rip, A., (2000). Evaluation of societal quality of public sector research in the Netherlands. Research Evaluation, 9 (1), 11-25.
98. van der Weijden, I., Verbree, M., &; van den, Besselaar, P., (2012). From bench to bedside: The societal orientation of research leaders: The case of biomedical and health research in the Netherlands. Science and Public Policy, 39 (3), 285-303. doi: 10.1093/scipol/scr003
99. van Vught, F., &;, Ziegele, F., (Eds.). (2011). Design and testing the feasibility of a multidimensional global university ranking. Final report: Consortium for Higher Education and Research Performance Assessment, CHERPA-Network.
100. Walter, A.I., Helgenberger, S., Wiek, A., &;, Scholz, R.W., (2007). Measuring societal effects of transdisciplinary research projects: Design and application of an evaluation method. Evaluation and Program Planning, 30 (4), 325-338. doi: 10.1016/j.evalprogplan.2007.08.002
101. Ziman, J., (2000). Real science: What it is, and what it means. Cambridge, United Kingdom: Cambridge University Press.