Societal impact evaluation: Exploring evaluator perceptions of the characterization of impact under the REF2014

Research Evaluation

Volumn 24, Issue 3, 2015, Pages 229-241

  Samuel, Gabrielle N ^a   Derrick, Gemma E ^a  

^a BRUNEL UNIVERSITY   (United Kingdom)

Author keywords

Qualitative research; Research evaluation; Research excellence framework; Societal impact

Indexed keywords

EID: 84942250494     PISSN: 09582029     EISSN: 14715449     Source Type: Journal    
DOI: 10.1093/reseval/rvv007     Document Type: Review

References (57)

1. Benda, W.G.G. and Engels, T.C.C. (2011) 'The Predictive Validity of Peer Review: A Selective Review of the Judgmental Forecasting Qualities of Peers, and Implications for Innovation in Science', International Journal of Forecasting, 27: 166-82.
2. Bernstein, A., Hicks, V., Borbey, P. and Campbell, T. 'A Framework to Measure the Impact of Investments in Health Research', OECD Blue Sky II Forum Conference Proceedings, Ottawa, Ontario, Canada, 25 September 2006.
3. Bornmann, L. (2008) 'Scientific Peer Review: An Analysis of the Peer Review Process from the Perspective of Sociology of Science Theories', Human Architecture: Journal of the Sociology of Self-Knowledge, 6: Article 3.
4. -. (2011) 'Scientific Peer Review', Annual Review of Information Science and Technology, 45: 197-245.
5. -. (2013) 'What is Societal Impact of Research and How Can it be Assessed? A Literature Survey', Journal of the American Society for Information Science and Technology, 64: 217-33.
6. Bornmann, L. and Marx, W. (2014) 'How Should the Societal Impact of Research be Generated and Measured? A Proposal for a Simple and Practicable Approach to Allow Interdisciplinary Comparisons', Scientometrics, 98: 211-19.
7. Bornmann, L. and Mungra, P. (2011) 'Improving Peer Review in Scholarly Journals', European Science Editing, 37: 41-3.
8. Bozeman, B. and Boardman, C. (2009) 'Broad Impacts and Narrow Perspectives: Passing the Buck on Science and Social Impacts', Social Epistemology: A Journal of Knowledge, Culture and Policy, 23: 183-98.
9. Brewer, J. D. (2011) 'The Impact of Impact', Research Evaluation, 20/3: 255-56.
10. Brown, N. and Webster, A. (2004) New Medical Technologies and Society. Cambridge: Polity Press.
11. Buxton, M. (2011) 'The Payback of 'Payback': Challenges in Assessing Research Impact', Research Evaluation, 20: 259-60.
12. Buxton, M. and Hanney, S. (1996) 'How Can Payback from Health Services Research be Assessed?', Journal of Health Service Research and Policy, 1: 35-43.
13. Buxton, M. J. and Hanney, S. (2008) 'Developing and Applying the Payback Framework to Assess the Socioeconomic Impact of Health Research', Medicina Clinica, 131: 36-41.
14. Charmaz, K. (2006) Grounded Theory: A Practical Guide Through Qualitative Analysis.
15. de Jong, S., Barker, K., Cox, D., Sveinsdottir, T. and van den Besselaar, P. (2014) 'Understanding Societal Impact Through Productive Interactions: ICT Research as a Case', Research Evaluation, 23: 89-102.
16. de Jong, S., van Arensbergen, P., Daemen, F., van der Meulen, B. and van den Besselaar, P. (2011) 'Evaluation of Research in Context: An Approach and Two Cases', Research Evaluation, 20: 61-72.
17. Derrick, G. and Samuel, G. (2015) 'The Societal Impact-Focused and Quality-Focused Evaluator: Balancing Evaluator Views about the Assessment of the Societal Impact of Health and Medical Research', Social Science and Medicine, In press.
18. Donovan, C. (2008) 'The Australian Research Quality Framework: A Live Experiment in Capturing the Social, Economic, Environmental and Cultural Returns of Publicly Funded Research', New Directions for Evaluation, 118: 47-60.
19. Donovan, C., Butler, L., Butt, A. J., Jones, T. H. and Hanney, S. R. (2014) 'Evaluation of the Impact of National Breast Cancer Foundation-Funded Research', The Medical Journal of Australia, 200: 14-18.
20. Donovan, C. and Hanney, S. (2011) 'The 'Payback Framework' explained', Research Evaluation, 20: 181-83.
21. Frank, C. and Nason, E. (2009) 'Health Research: Measuring the Social, Health and Economic Benefits', CMAJ, 180: 528-34.
22. Glaser, B. G. and Strauss, A. (1967) The Discovery of Grounded Theory: Strategies for Qualitative Research. Chicago: Aldine.
23. Glaser, J. and Laudel, G. (2005) 'Advantages and Dangers of 'Remote' Peer Evaluation', Research Evaluation, 14: 186-98.
24. Grant, J., Brutscher, P., Kirk, S., Butler, L. and Wooding, S. (2009) Capturing Research Impacts: A Review of International Practice. Europe: RAND.
25. Hanney, S. (2005) 'Developing and Applying a Framework for Assessing the Payback from Medical Research', New Technology and National Health Systems. Kiel Institute's International Research Conference, the Institute for World Economics.
26. Hanney, S., Buxton, M., Green, C., Coulson, D. and Raftery, J. (2007) 'An Assessment of the Impact of the NHS Health Technology Assessment Programme', Health Technology Assessment, 11: iii-iv, ix-xi, 1-180.
27. Hanney, S. R., Grant, J., Wooding, S. and Buxton, M. J. (2004) 'Proposed Methods for Reviewing the Outcomes of Health Research: The Impact of Funding by the UK's 'Arthritis Research Campaign'', Health Research Policy and Systems/ BioMed Central, 2: 4.
28. Haynes, A. S., Gillespie, J. A., Derrick, G. E., Hall, W. D., Redman, S., Chapman, S. and Sturk, H. (2011) 'Galvanizers, Guides, Champions, and Shields: The Many Ways That Policymakers Use Public Health Researchers', The Milbank Quarterly, 89: 564-98.
29. Holbrook, J. B. and Frodeman, R. (2011) ''Peer Review and the Ex Ante Assessment of Societal Impacts', Research Evaluation, 20: 239-46.
30. Kalucy, E. C., Jackson-Bowers, E., McIntyre, E. and Reed, R. (2009) 'The Feasibility of Determining the Impact of Primary Health Care Research Projects Using the Payback Framework', Health Research Policy and Systems/BioMed Central, 7: 11-4505-7-11.
31. Kerr, N. L., MacCoun, R. J. and Kramer, G. P. (1996) 'Bias in Judgement: Comparing Individuals and Groups', Psychological Review, 103: 687-719.
32. Klautzer, L., Hanney, S., Nason, E., Rubin, J., Grant, J. and Wooding, S. (2011) 'Assessing Policy and Practice Impacts of Social Science Research: The Application of the Payback Framework to Assess the Future of Work Programme', Research Evaluation, 20: 201-9.
33. Klein, T. and Olbrecht, M. (2011) 'Triangulation of Qualitative and Quantitative Methods in Panel Peer Review Research', International Journal for Cross-Disciplinary Subjects in Education (IJCDSE), 2: 342-48.
34. Kwan, P., Johnston, J., Fung, A.Y.K., Chong, D.S.Y., Collins, R. A. and Lo, S. V. (2007) 'A Systematic Evaluation of Payback of Publicly Funded Health and Health Services Research in Hong Kong', BMC Health Services Research, 7: 121.
35. Lamont, M. and Huutoniemi, K. (2011) 'Comparing Customary Rules of Fairness: Evaluative Practices in Various Types of Peer Review Panels'. In: Camic, C., Gross, N. and Lamont, M. (eds) Social Knowledge in the Making. Chicago: University of Chicago Press.
36. Langfeldt, L. (2001) 'The Decision-Making Constraints and Processes of Grant Peer Review, and Their Effects on the Review Outcome', Social Studies of Science, 31: 820-41.
37. -. (2006) 'The Policy Challenges of Peer Review: Managing Bias, Conflict of Interests and Interdisciplinary Assessments', Research Evaluation, 15: 31-41.
38. Lee, C. J., Sugimoto, C. R., Zhang, G. and Cronin, B. (2013) 'Bias in Peer Review', Journal of the American Society for Information Science and Technology, 64: 2-17.
39. Levi, D. (2007) Group Dynamics for Teams. Los Angeles: Sage Publications.
40. Martin, B. R. (2011) 'The Research Excellence Framework and the 'Impact Agenda': are we Creating a Frankenstein monster', Research Evaluation, 20: 247-54.
41. -. (2012) 'The Evolution of Science Policy and Innovation Studies', Research Policy, 441: 1219-239.
42. Molas-Gallart, J. and Tang, P. (2011a) 'Tracing 'Productive Interactions' to Identify Social Impacts: An Example from the Social Sciences', Research Evaluation, 20: 219-26.
43. Mostert, S. P., Ellenbroek, S.P.H., Meijer, I., van Ark, G. and Klasen, E. C. (2010) 'Societal Output and Use of Research Performed by Health Research Groups', Health Research Policy and System /BioMed Central, 8: 30-4505-8-30.
44. Olbrecht, M., Tibelius, K. and D'Aloisio, G. (2007) 'Examining the Value Added by Committee Discussion in the Review of Applications for Research Awards', Research Evaluation, 16: 79-91.
45. Olbrecht, M. and Bornmann, L. (2010) 'Panel Peer Review of Grant Applications: What do we Know from Research in Social Psychology on Judgment and Decision-Making in Groups?', Research Evaluation, 19: 293-304.
46. Oortwijn, W. J., Hanney, S. R., Ligtvoet, A., Hoorens, S., Wooding, S., Grant, J., Buxton, M. J. and Bouter, L. M. (2008) 'Assessing the Impact of Health Technology Assessment in The Netherlands', International Journal of Technology Assessment in Health Care, 24: 259-69.
47. Penfield, T., Baker, M. J., Scoble, R. and Wykes, M. C. (2014) 'Assessment, Evaluations, and Definitions of Research Impact: A Review', Research Evaluation, 23: 21-32. Research Councils UK. Pathways to Impact. .
48. Research Excellence Framework. (2011) Assessment Framework and Guidance on Submissions. REF 02.2011. .
49. -. (2012a) Panel Criteria and Working Methods. .
50. -. (2012b) Main Panel A Criteria: REF01.2012. .
51. Sandstorm, U. (2009) 'Cognitive Bias in Peer Review: A New Approach', Proceedings of ISSI 2009-12th international conference of the international society for scientometrics and informetrics, vol 2, 2th International Conference of the International-Society-for-Scientometrics-and-Informetrics. Rio de Janeiro, Brazil. 14-17 July. .
52. Scoble, R., Dickson, K., Hanney, S. and Rodgers, G. J. (2010) 'Institutional Strategies for capturing Socio-Economic Impact of Academic Research', Journal of Higher Education Policy and Management, 32: 499-510.
53. Scott, J. E., Blasinsky, M., Dufour, M., Mandai, R. J. and Philogene, G. S. (2011) 'An Evaluation of the Mind-Body Interactions and Health Program: Assessing the Impact of an NIH Program using the Payback Framework', Research Evaluation, 20: 185-92.
54. Spaapen, J. and van Drooge, L. (2011) 'Introducing 'Productive Interactions' in Social Impact Assessment', Research Evaluation, 20: 211-18.
55. Strauss, A. L. (1987) Qualitative Analysis For Social Scientists. Cambridge: Cambridge University Press.
56. van Arensbergen, P., van der Weijden, I. and Van De Besselaar, P. (2014) 'The Selection of Talent as a Group Process. A Literature Review on the Social Dynamics of Decision Making in Grant Panels', Research Evaluation, 23: 298-311.
57. Wooding, S., Hanney, S., Buxton, M. and Grant, J. (2005) 'Payback Arising from Research Funding: Evaluation of the Arthritis Research Campaign', Rheumatology (Oxford, England), 44: 1145-156.