Evaluating research beyond scientific impact: How to include criteria for productive interactions and impact on practice and society

GAIA - Ecological Perspectives for Science and Society

Volumn 22, Issue 2, 2013, Pages 104-114

  Wolf, Birge ^a   Lindenthal, Thomas ^b   Szerencsits, Manfred ^a   Holbrook, J Britt ^c   Heß, Jürgen ^a  

^a UNIVERSITY OF KASSEL   (Germany)

^b UNIVERSITY OF NATURAL RESOURCES AND LIFE SCIENCES   (Austria)

^c UNIVERSITY OF NORTH TEXAS   (United States)

Author keywords

Data assessment; Documentation; Interdisciplinarity; Organic agricul ture; Practice; Productive interactions; Research evaluation; Social societal impact; Sustainability; Transdisciplinarity

Indexed keywords

EID: 84880072749     PISSN: 09405550     EISSN: 26255413     Source Type: Journal    
DOI: 10.14512/gaia.22.2.9     Document Type: Article

References (68)

1. Albrecht, S. 2009. Weltagrarbericht. Synthesebericht. Hamburg: Hamburg University Press.
2. ASIRPA(Socio-economic Analysis of Impacts of Public Agronomic Research). 2012. ASIRPA project in 4 pages. Paris: French National Institute for Agricultu ral Research (INRA). www6.inra.fr/asirpa-eng/ASIRPA-project/International- ASIRPA-workshop-Nov-2012/General-material (accessed February 11, 2013).
3. th edition. Berlin: Walter de Gruyer.
4. Begusch-Pfefferkorn, K. 2006. KLF-Wissensbilanz. Leistungsdarstellung des inter- und transdisziplinären Programms Kulturlandschaftsforschung. In: Fachübergreifende Nachhaltigkeitsforschung. Stand und Visionen am Beispiel nationaler und internationaler Forscherverbünde. Edited by B. Glaeser. Munich: oekom. 119-139.
5. Bell, S., B. Shaw, A. Boaz. 2011. Real-world approaches to assessing the impact of environmental research on policy. Research Evaluation 20/3: 227-237.
6. Bergmann, M. et al. 2005. Qualitätskriterien transdisziplinärer Forschung. Ein Leitfaden für die formative Evaluation von Forschungsprojekten. Frankfurt am Main: Institut für sozial- ökologische Forschung(ISOE).
7. Blackstock, K. L., G. J. Kelly, B. L. Horsey. 2007. Developing and applying a framework to evaluate participatory research for sustainability. Ecological Economics 60/4: 726-742.
8. Boaz, A., S. Fitzpatrick, B. Shaw. 2009. Assessing the impact of research on policy: A literature review. Science and Public Policy 36/4: 255-270.
9. Bornmann, L. 2012. What is societal impact of research and how can it be assessed? A literature survey. Journal of the American Society for Information Science and Technology 64/2: 217-233.
10. Braun, M. et al. 2009. Tools and indicators for community research evaluation and monitoring. Bad Camberg: Proneos. http://ec.europa.eu/research/ evaluations/pdf/archive/fp6-evidence-base/evaluation-studies-and-reports/ evaluation-studies-and-reports-2009/tools-and-indicators-for-community-research- evaluation-and-monitoring.pdf (accessed October 24, 2012).
11. Buxton, M. 2011. The payback of "payback": Challenges in assessing research impact. Research Evaluation 20/3: 259-260.
12. Clark, W. C., P. J. Crutzen, H. J. Schellnhuber. 2005. Science for global sustainability: Toward a new paradigm. CID Working Paper 120. Cambridge, MA: Science, Environment and Development Group, Center for International Development, Harvard University.
13. Daschkeit, A. 2007. Zur Beurteilung transdisziplinärer Forschung. Hinweise auf Bücher zur Transdisziplinarität. GAIA 16/1: 58-65.
14. Daschkeit, A., M.C. Loibl. 2007. Projektspezifische Evaluation human-ökologischer Nachhaltigkeitsforschung. Entwurf eines Bewertungs-schemas. In: Evaluation inter- und transdisziplinärer Forschung. Human-ökologie und Nachhaltigkeitsforschung auf dem Prüfstand. Edited by S. Stoll-Kleemann, C. Pohl. Munich: oekom. 157-171.
15. Davis, J., J.P.D. Gordon, D. Templeton. 2008. Guidelines for assessing the impacts of ACIAR's research activities. ACIAR Impact Assessment Series Report 58. Canberra: Australian Centre for International Agricultural Research (ACIAR).
16. Defila, R., A. Di Giulio. 1999. Evaluationskriterien für inter- und transdisziplinäre Forschung. Panorama 99: 5-11. www.ikaoe.unibe.ch/ forschung/ip/Sondernummer.Pano.1.99.pdf (accessed January 26, 2010).
17. Defila, R., A. Di Giulio, M. Scheuermann. 2008. Management von Forschungs-verbünden. Möglichkeiten der Professionalisierung und Unterstützung. Standpunkte. Weinheim: Wiley-VCH.
18. DFG (German Research Foundation). 2005. Perspektiven der agrarwissenschaftlichen Forschung/Future Perspectives of Agricultural Science and Research. Denkschrift/Memorandum. Weinheim: Wiley-VCH.
19. Donovan, C. 2011. State of the art in assessing research impact: Introduction to a special issue. Research Evaluation 20/3: 175-179.
20. Douthwaite, B., T. Kuby, E. van de Fliert, S. Schulz. 2003. Impact pathway evaluation: An approach for achieving and attributing impact in complex systems. Agricultural Systems 78: 243-265.
21. EC (European Commission). 2010. Assessing Europe's university-based research: Expert group of assessment of university-based research. EUR 24187. Luxembourg: Publications Office of the European Union. http://ec.europa.eu/ research/science-society/document-library/pdf-06/assessing-europe-university- based-research-en.pdf (accessed July 4, 2011).
22. ERiC (Evaluating Research in Context). 2010. Evaluating the societal relevance of academic research: A guide. ERiC publication 1001 EN. www.siampi.eu/Content/SIAMPI/ERiC%20Guide%202010.pdf (accessed January 26, 2010).
23. nd edition. Reinbek: Rowohlt.
24. Formas (Swedish Research Council for Environment, Agricultural Sciences and Spatial Planning). 2007. Evaluation report 2006: Evaluation of research on organic production in Sweden. Stockholm: EDITA Ljunglöfs.
25. Formas. 2009. Evaluation report 2009: Evaluation of SLF-funded research 2002-2007. Stockholm: Forskningsrådet Formas. www.formas.se/PageFiles/ 5324/Formas-SLF.pdf (accessed May 7, 2013).
26. Frodeman, R., J.B. Holbrook. 2011. NSF's struggle to articulate relevance. Science 333/6039: 157-158.
27. Frodeman, R., J.B. Holbrook, C. Mitcham, H. Xiaonan. 2012. Peer review, research integrity, and the governance of science: Practice, theory, and current discussions. Beijing: People's Publishing House.
28. Hennen, L., T. Petermann, C. Scherz. 2004. Partizipative Verfahren der Technik-folgen-Abschätzung und parlamentarische Politikberatung. Neue Formen der Kommunikation zwischen Wissenschaft, Politik und öffentlichkeit. TAB-Arbeitsbericht 96. Berlin: Büro für Technikfolgen- Abschätzung beim Deutschen Bundestag(TAB).
29. Holbrook, J. B. 2012. Re-assessing the science-society relation: The case of the US National Science Foundation's broader impacts merit review criterion (1997-2011). In: Peer review, research integrity, and the governance of science: Practice, theory, and current discussions. Edited by R. Frodeman, J.B. Holbrook, C. Mitcham, H. Xiaonan. Beijing: People's Publishing House. 328-362.
30. Holbrook, J.B., R. Frodeman. 2011. Peer review and the ex ante assessment of societal impacts. Research Evaluation 20/3: 239-246.
31. Huutoniemi, K. 2012. Interdisciplinary accountability in the evaluation of research proposals: Prospects for academic quality control across disciplinary boundaries. Dissertation, University of Helsinki, Finland.
32. Huutoniemi, K., J. T. Klein, H. Bruun, J. Hukkinen. 2010. Analyzing inter-disciplinarity: Typology and indicators. Research Policy 39/1: 79-88.
33. Klautzer, L., S. Hanney, E. Nason, J. Rubin, J. Grant, S. Wooding. 2011. Assessing policy and practice impacts of social science research: The application of the payback framework to assess the Future of Work programme. Research Evaluation 20/3: 201-209.
34. Klein, J. T. 2006. Afterword: The emergent literature on interdisciplinary and transdisciplinary research evaluation. Research Evaluation 15/1: 75-80.
35. Lyall, C., A. Bruce, J. Firn, M. Firn, J. Tait. 2004. Assessing end-use relevance of public sector research organisations. Research Policy 33/1: 73-87.
36. MA(Millennium Ecosystem Assessment). 2005. Ecosystems and human well-being: Synthesis. Washington, D. C.: Island.
37. Manring, S.L. 2007. Creating and managing interorganizational learning networks to achieve sustainable ecosystem management. Organization and Environment 20/3: 325-346.
38. th Biennial Conference of the European Evaluation Society, October 1-5, Helsinki, Finland.
39. Niederkrotenthaler, T., T.E. Dorner, M. Maier. 2011. Development of a practical tool to measure the impact of publications on the society based on focus group discussions with scientists. BMC Public Health 11/1: 588-596.
40. Nowotny, H., P. Scott, M. Gibbons. 2001. Re-thinking science: Knowledge and the public in an age of uncertainty. Cambridge, UK: Polity.
41. Pearson, S., P. Chudleigh, S. Simpson, N. Schofield. 2012. Learning to invest better: Using ex post investment analysis on agri-environmental research and development. Research Evaluation 21/2: 136-151.
42. th International Farm Management Congress: Agriculture: Food, fiber and energy for the future. Edited by H. H. Guither, J.L. Merry, C.E. Merry. Bloomington, IL: International Farm Management Association (IFMA). 99-114.
43. Pohl, C. et al. 2011. Questions to evaluate inter- and transdisciplinary research proposals. Working paper. Bern: td-net for Transdisciplinary Research. www.transdisciplinarity.ch/documents/td-Evaluation2011-workingpaper.pdf (accessed May 6, 2013).
44. Pohl, C., G. Hirsch Hadorn. 2006. Gestaltungsprinzipien für die transdisziplinäre Forschung. Munich: oekom.
45. Rafols, I., L. Leydesdorff, A. O'Hare, P. Nightingale, A. Stirling. 2012. How journal rankings can suppress interdisciplinary research: A comparison between innovation studies and business & management. Research Policy 41/7: 1262-1282.
46. RCUK (Research Councils UK). 2010. Research outcomes project invitation to tender research outcome types. Swindon, UK: RCUK. www.rcuk.ac.uk/documents/ oocp/OutputTypes.pdf (accessed September 21, 2011).
47. RCUK. 2011a. RCUK impact requirements: Frequently asked questions. Swindon, UK: RCUK. www.rcuk.ac.uk/documents/impacts/RCUKImpactFAQ.pdf (accessed September 21, 2011).
48. RCUK. 2011b. RCUK research outcomes system now available. Swindon, UK: RCUK. www.rcuk.ac.uk/media/news/2011news/Pages/111124.aspx (accessed October 24, 2012).
49. Regulation(EC)No1906/2006. 2006. Regulation (EC)No 1906/2006 of the European Parliament and of the Council of 18 December 2006 laying down the rules for the participation of undertakings, research centres and universities in actions under the Seventh Framework Programme and for the dissemination of research results(2007-2013). Official Journal of the European Union L 391: 1-18.
50. Reuber, M., O. Haas. 2009. Evaluierungen in der Deutschen Gesellschaft für Technische Zusammenarbeit(GTZ). Zeitschrift für Evaluation 6/1: 137-150.
51. Rogers, J. D., G. Jordan. 2010. New research evaluation frameworks and methods for systems level learning: Introduction to a special section. Research Evaluation 19/4: 235-237.
52. Schiller, J. et al. 2005. Evaluation interdisziplinärer Programm-Forschung. Ziele, Anlässe und Prozesse. UFZ-Diskussionspapiere 17/2005. Leipzig: Helmholtz-Zentrum für Umweltforschung - UFZ.
53. Schübel, H. R. 2007. Interdisziplinäre Kooperation in transdisziplinärer Forschung besser machen? Prozessbegleitung als formative Evaluation. In: Evaluation inter- und transdisziplinärer Forschung. Humanökologie und Nachhaltigkeitsforschung auf dem Prüfstand. Edited by S. Stoll-Kleemann, C. Pohl. Munich: oekom. 195-210.
54. Shaw, B., S. Bell. 2010. How to evaluate the impact of environmental research on policy: Guidelines and supporting information. London: Policy Studies Institute. www.psi.org.uk/publications/SKEP/Guidelines%20Final%20v1.2. pdf (accessed August 22, 2012).
55. Skolits, G. J., J. A. Morrow, E. M. Burr. 2009. Reconceptualizing evaluator roles. American Journal of Evaluation 30/3: 275-295.
56. Spaapen, J. et al. 2011. SIAMPI final report. www.siampi.eu/Content/ SIAMPI/SIAMPI-Final%20report.pdf (accessed August 22, 2012).
57. Spaapen, J., L. van Drooge. 2011. Introducing "productive interactions" in social impact assessment. Research Evaluation 20/3: 211-218.
58. Stockmann, R. 2004. Was ist eine gute Evaluation? Einführung zu Funktionen und Methoden von Evaluationsverfahren. CEval Arbeitspapiere 9. Saarbrücken: Centrum für Evaluation (CEval). www.ceval.de/typo3/ fileadmin/user-upload/PDFs/workpaper9.pdf (accessed May 7, 2013).
59. Stokols, D. et al. 2003. Evaluating transdisciplinary science. Nicotine and Tobacco Research 5/1: 21-39.
60. Stoll-Kleemann, S. 2007. Potentiale der Evaluation inter- und transdisziplinärer Nachhaltigkeitsforschung und Humanökologie. In: Evaluation inter- und transdisziplinärer Forschung. Humanökologie und Nachhaltigkeitsforschung auf dem Prüfstand. Edited by S. Stoll-Kleemann, C. Pohl. Munich: oekom. 25-40.
61. Tàbara, J. D., J. Jaeger. 2012. Analysis of pathways workshop. www.visionrd4sd.eu/news/71 (accessed September 20, 2012).
62. Uriarte, M., H.A. Ewing, V. T. Eviner, K.C. Weathers. 2007. Constructing a broader and more inclusive value system in science. BioScience 57/1: 71-78.
63. Van Vught, F., F. Ziegele (Eds.). 2011. U-Multirank - Design and testing the feasibility of a multidimensional global university ranking: Final report. CHERPA-Network (Consortium for Higher Education and Research Performance Assessment). http://ec.europa.eu/education/higher-education/doc/multirank-en.pdf (accessed September 12, 2011).
64. VDW (Federation of German Scientists). 2010. Für eine verantwortbare und zukunftsorientierte Forschungspolitik in Deutschland. VDW Positionspapier. Berlin: VDW. www.vdw-ev.de/images/stories/vdwdokumente/aktuelles/vdw-papier-zur- forschungspolitik.pdf (accessed May 16, 2011).
65. Walter, A.I., S. Helgenberger, A. Wiek, R.W. Scholz. 2007. Measuring societal effects of transdisciplinary research projects: Design and application of an evaluation method. Evaluation and Program Planning 30/4: 325-338.
66. WBGU(German Advisory Council on Global Change). 2011. World in transition: A social contract for sustainability. Berlin: WBGU.
67. Wolf, B., T. Lindenthal, M. Szerencsits, J. Heß. Forthcoming. Berücksichtigung der Leistungen für Praxis und Gesellschaft in der Forschungsevaluierung: Praxis-Impact-Synthesebericht. www.orgprints.org.
68. WR (German Council of Science and Humanities). 2013. Empfehlungen zu einem Kerndatensatz Forschung. Berlin: WR. www.wissenschaftsrat.de/download/ archiv/2855-13.pdf (accessed February 5, 2013). Protection