The impact on healthcare, policy and practice from 36 multi-project research programmes: Findings from two reviews

Health Research Policy and Systems

Volumn 15, Issue 1, 2017, Pages

  Hanney, Steve ^a   Greenhalgh, Trisha ^b   Blatch Jones, Amanda ^c   Glover, Matthew ^a   Raftery, James ^c  

^a BRUNEL UNIVERSITY   (United Kingdom)

^b UNIVERSITY OF OXFORD   (United Kingdom)

^c UNIVERSITY OF SOUTHAMPTON   (United Kingdom)

Author keywords

Global Observatory; Health gains; Health technology assessment; Monetisation; Multi project programmes; Payback Framework; Policy impact; Practice impact; Research impact; World Health Report

Indexed keywords

CLINICAL PRACTICE; CONCEPTUAL FRAMEWORK; FUNDING; HEALTH CARE POLICY; HUMAN; INTERVIEW; INVESTMENT; MEDICAL RESEARCH; NARRATIVE; SYSTEMATIC REVIEW; GLOBAL HEALTH; HEALTH CARE DELIVERY; HEALTH IMPACT ASSESSMENT; HEALTH SERVICES RESEARCH; PROFESSIONAL PRACTICE;

BIOMEDICAL RESEARCH; DELIVERY OF HEALTH CARE; GLOBAL HEALTH; HEALTH IMPACT ASSESSMENT; HEALTH POLICY; HEALTH SERVICES RESEARCH; HUMANS; PROFESSIONAL PRACTICE;

EID: 85016419231     PISSN: None     EISSN: 14784505     Source Type: Journal    
DOI: 10.1186/s12961-017-0191-y     Document Type: Article

References (92)

1. World Health Organization. The World Health Report 2013: Research for Universal Health Coverage. Geneva: WHO; 2013.
2. Walugembe DR, Kiwanuka SN, Matovu JKB, Rutebemberwa E, Reichenbach L. Utilization of research findings for health policy making and practice: evidence from three case studies in Bangladesh. Health Res Policy Syst. 2015;13:26. doi: 10.1186/s12961-015-0015-x.
3. Angulo-Tuesta A, Santos LMP. Evaluation of the impact of maternal and neonatal morbidity and mortality research funded by the Ministry of Health in Brazil. Res Eval. 2015;24:355-68. doi: 10.1093/reseval/rvv022.
4. Kok MO, Gyapong JO, Wolffers I, Ofori-Adjei D, Ruitenberg J. Which health research gets used and why? An empirical analysis of 30 cases. Health Res Policy Syst. 2016;14:36. doi: 10.1186/s12961-016-0107-2.
5. Yazdizadeh B, Majdzadeh R, Janani L, Mohtasham F, Nikooee S, Mousavi A, et al. An assessment of health research impact in Iran. Health Res Policy Syst. 2016;14:56. doi: 10.1186/s12961-016-0129-9.
6. Hanney S, Buxton M, Green C, Coulson D, Raftery J. An assessment of the impact of the NHS Health Technology Assessment Programme. Health Technol Assess. 2007;11:53. http://dx.doi.org/10.3310/hta11530.
7. Boaz A, Fitzpatrick S, Shaw B. Assessing the impact of research on policy: a literature review. Sci Public Policy. 2009;36:255-70. http://dx.doi.org/10.3152/030234209X436545.
8. Greenhalgh T, Raftery J, Hanney S, Glover M. Research impact: a narrative review. BMC Med. 2016;14:78. doi: 10.1186/s12916-016-0620-8.
9. Raftery J, Hanney S, Greenhalgh T, Glover M, Blatch-Jones A. Models and applications for measuring the impact of health research: update of a systematic review for the Health Technology Assessment programme. Health Technol Assess. 2016;20:76.
10. Canadian Academy of Health Sciences (CAHS). Making an Impact, A Preferred Framework and Indicators to Measure Returns on Investment in Health Research. Ottawa, ON: CAHS; 2009.
11. Banzi R, Moja L, Pistotti V, Facchini A, Liberati A. Conceptual frameworks and empirical approaches used to assess the impact of health research: an overview of reviews. Health Res Policy Syst. 2011;9:26. http://dx.doi.org/10.1186/1478-4505-9-26.
12. Guthrie S, Wamae W, Diepeveeen S, Wooding S, Grant J. Measuring Research: A Guide to Research Evaluation Frameworks and Tools. Santa Monica, CA: RAND Corporation; 2013.
13. Milat AJ, Bauman AE, Redman S. A narrative review of research impact assessment models and methods. Health Res Policy Syst. 2015;13:18. doi: 10.1186/s12961-015-0003-1.
14. Buxton M, Hanney S. How can payback from health services research be assessed? J Health Serv Res Policy. 1996;1:35-43.
15. Hanney SR, Grant J, Wooding S, Buxton MJ. Proposed methods for reviewing the outcomes of health research: the impact of funding by the UK's 'Arthritis Research Campaign'. Health Res Policy Syst. 2004;2:4. http://dx.doi.org/10.1186/1478-4505-2-4.
16. Kogan M, Henkel M. Government and Research: The Rothschild Experiment in a Government Department. London: Heinemann Educational Books; 1983.
17. Kogan M, Henkel M, Hanney S. Government and Research: 30 Years of Evolution. Dordrecht: Springer; 2006. http://dx.doi.org/10.1007/1-4020-4446-1.
18. Weiss C. The many meanings of research utilization. Public Admin Rev. 1979;39:426-31.
19. Trostle J, Bronfman M, Langer A. How do researchers influence decision-makers? Case studies of Mexican policies. Health Policy Plan. 1999;14:103-14.
20. Lomas J. Using 'linkage and exchange' to move research into policy at a Canadian foundation. Health Aff. 2000;19:236-40. http://dx.doi.org/10.1377/hlthaff.19.3.236.
21. Lavis JN, Ross SE, Hurley JE, Hohenadel JM, Stoddart GL, Woodward CA, et al. Examining the role of health services research in public policymaking. Milbank Q. 2002;80:125-54.
22. Gilson L, Bowa C, Brijlal V, Doherty J, Antezana I, Daura M, et al. The Dynamics of Policy Change: Lessons From Health Financing Reform in South Africa and Zambia. Bethesda: Partnerships for Health Reform Project, Abt Associates Inc.; 2000.
23. Spaapen J, van Drooge L. Introducing 'productive interactions' in social impact assessment. Res Eval. 2011;20:211-8.
24. Kuruvilla S, Mays N, Pleasant A, Walt G. Describing the impact of health research: a research impact framework. BMC Health Serv Res. 2006;6:134. http://dx.doi.org/10.1186/1472-6963-6-134.
25. Walt G. Health Policy: An Introduction to Process and Power. London: Zed Books; 1994.
26. Sauerborn R, Nitayarumphong S, Gerhardus A. Strategies to enhance the use of health systems research for health sector reform. Trop Med Int Health. 1999;4:827-35.
27. Innvær S, Vist G, Trommald M, Oxman A. Health policy-makers' perceptions of their use of evidence: a systematic review. J Health Serv Res Policy. 2002;7:239-44. http://dx.doi.org/10.1258/135581902320432778.
28. Hanney SR, Gonzalez-Block MA, Buxton MJ, Kogan M. The utilisation of health research in policy-making: concepts, examples and methods of assessment. Health Res Policy Syst. 2003;1:2. http://dx.doi.org/10.1186/1478-4505-1-2.
29. Haines A, Kuruvilla S, Borchert M. Bridging the implementation gap between knowledge and action for health. Bull World Health Organ. 2004;82:724-31.
30. Oliver K, Innvar S, Lorenc T, Woodman J, Thomas J. A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Serv Res. 2014;14:2. http://dx.doi.org/10.1186/1472-6963-14-2.
31. Wooding S, Hanney SR, Pollitt A, Grant J, Buxton MJ. Understanding factors associated with the translation of cardiovascular research: a multinational case study approach. Implement Sci. 2014;9:47. http://dx.doi.org/10.1186/1748-5908-9-47.
32. Adam P, Solans-Domènech M, Pons JMV, Aymerich M, Berra S, Guillamon I, et al. Assessment of the impact of a clinical and health services research call in Catalonia. Res Eval. 2012;21:319-28. http://dx.doi.org/10.1093/reseval/rvs024.
33. Alberta Heritage Foundation for Medical Research (AHFMR). Assessment of Health Research Fund Outputs and Outcomes: 1995-2003. Edmonton: AHFMR; 2003.
34. Bodeau-Livinec F, Simon E, Montagnier-Petrissans C, Joël M-E, Féry-Lemonnier E. Impact of CEDIT recommendations: an example of health technology assessment in a hospital network. Int J Technol Assess Health Care. 2006;22:161-8. http://dx.doi.org/10.1017/S0266462306050975.
35. Brambila C, Ottolenghi E, Marin C, Bertrand JT. Getting results used: evidence from reproductive health programmatic research in Guatemala. Health Policy Plan. 2007;22:234-45. http://dx.doi.org/10.1093/heapol/czm013.
36. Buxton M, Hanney S, Packwood T, Roberts S, Youll P. Assessing benefits from Department of Health and National Health Service research and development. Public Money Manag. 2000;20:29-34.
37. Caddell AJ, Hatchette JE, McGrath PJ. Examining the impact of health research facilitated by small peer-reviewed research operating grants in a women's and children's health centre. BMC Res Notes. 2010;3:107. http://dx.doi.org/10.1186/1756-0500-3-107.
38. Donovan C, Butler L, Butt AJ, Jones TH, Hanney SR. Evaluation of the impact of National Breast Cancer Foundation-funded research. Med J Aust. 2014;200:214-8. http://dx.doi.org/10.5694/mja13.10798.
39. Expert Panel for Health Directorate of the European Commission's Research Innovation Directorate General. Review of Public Health Research Projects Financed under the Commission's Framework Programmes for Health Research. Brussels: European Commission; 2013.
40. Ferguson B, Kelly P, Georgiou A, Barnes G, Sutherland B, Woodbridge B. Assessing payback from NHS reactive research programmes. J Manage Med. 2000;14:25-36.
41. Gold M, Taylor EF. Moving research into practice: lessons from the US Agency for Healthcare Research and Quality's IDSRN program. Implement Sci. 2007;2:9. http://dx.doi.org/10.1186/1748-5908-2-9.
42. Gutman MA, Barker DC, Samples-Smart F, Morley C. Evaluation of active living research: progress and lessons in building a new field. Am J Prev Med. 2009;36 Suppl 2:22-33. http://dx.doi.org/10.1016/j.amepre.2008.10.009.
43. Hailey DM, Cowley DE, Dankiw W. The impact of health technology assessment. Commun Health Stud. 1990;14:223-34.
44. Hailey D, Corabian P, Harstall C, Schneider W. The use and impact of rapid health technology assessments. Int J Technol Assess Health Care. 2000;16:651-6.
45. Hanney SR, Watt A, Jones TH, Metcalf L. Conducting retrospective impact analysis to inform a medical research charity's funding strategies: the case of Asthma UK. Allergy Asthma Clin Immunol. 2013;9:17. http://dx.doi.org/10.1186/1710-1492-9-17.
46. Hera. End of Programme Summative Evaluation of the Africa Health Systems Initiative Support to African Research Partnerships (AHSI-RES). Reet: Hera; 2014.
47. Jacob R, Battista R. Assessing technology assessment. Int J Technol Assess Health Care. 1993;9:564-72.
48. Jacob R, McGregor M. Assessing the impact of health technology assessment. Int J Technol Assess Health Care. 1997;13:68-80.
49. Johnston SC, Rootenberg JD, Katrak S, Smith WS, Elkins JS. Effect of a US National Institutes of Health programme of clinical trials on public health and costs. Lancet. 2006;367:1319-27. http://dx.doi.org/10.1016/S0140-6736(06)68578-4.
50. Kingwell BA, Anderson GP, Duckett SJ, Hoole EA, Jackson-Pulver LR, Khachigian LM, et al. Evaluation of NHMRC funded research completed in 1992, 1997 and 2003: gains in knowledge, health and wealth. Med J Aust. 2006;184:282-6.
51. Kwan P, Johnston J, Fung AY, Chong DS, Collins RA, Lo SV. A systematic evaluation of payback of publicly funded health and health services research in Hong Kong. BMC Health Serv Res. 2007;7:121. http://dx.doi.org/10.1186/1472-6963-7-121.
52. McGregor M, Brophy M. End-user involvement in health technology assessment (HTA) development: a way to increase impact. Int J Technol Assess Health Care. 2005;21:263-7.
53. Milat AJ, Laws R, King L, Newson R, Rychetnik L, Rissel C, et al. Policy and practice impacts of applied research: a case study analysis of the New South Wales Health Promotion Demonstration Research Grants Scheme 2000-2006. Health Res Policy Syst. 2013;11:5. http://dx.doi.org/10.1186/1478-4505-11-5.
54. Molas-Gallart J, Tang P, Morrow S. Assessing the non-academic impact of grant-funded socioeconomic research: results from a pilot study. Res Eval. 2000;9:171-82.
55. Oortwijn J, Hanney S, Ligtvoet A, Hoorens S, Wooding S, Grant J, et al. Assessing the impact of health technology assessment in the Netherlands. Int J Technol Assess Health Care. 2008;24:259-69. http://dx.doi.org/10.1017/S0266462308080355.
56. Poortvliet EP, Vijfvinkel D, Vennekens A, van Hoesel P, Daue F. Study into the Impact of the Belgian Health Care Knowledge Centre. Brussels: Belgian Health Care Knowledge Centre; 2010.
57. Reed RL, Kalucy EC, Jackson-Bowers E, McIntyre E. What research impacts do Australian primary health care researchers expect and achieve? Health Res Policy Syst. 2011;9:40. http://dx.doi.org/10.1186/1478-4505-9-40.
58. RSM McClure Watters, Hanney S, Nason E. Evaluation of the impact of HSC R&D Funding in Northern Ireland, Including Benchmarking with other Countries. Belfast: RSM McClure Watters; 2012.
59. Sainty M. Research impact: a United Kingdom Occupational Therapy Research Foundation perspective. Br J Occup Ther. 2013;76:528-37. http://dx.doi.org/10.4276/030802213X13861576675204.
60. Shah S, Ward JE. Outcomes from NHMRC public health research project grants awarded in 1993. Aust NZ J Public Health. 2001;25:556-60.
61. Soper B, Hanney S. Lessons from the evaluation of the UK's NHS R&D Implementation Methods Programme. Imp Sci. 2007;2:7.
62. The Madrillon Group. The Mind-Body Interactions and Health Program Outcome Evaluation. Final Report. Bethesda, MD: Office of Behavioral and Social Sciences Research, National Institutes of Health; 2011.
63. Wisely J. Assessment of the benefits from the National R&D Programme on primary/secondary care interface. London: NHS London Regional Office; 2001.
64. Wisely J. National R&D Programme in the area of mother and child: programme report. London: NHS London Regional Office; 2001.
65. Wooding S, Nason E, Starkey T, Hanney S, Grant J. Mapping the Impact: Exploring the Payback of Arthritis Research. Santa Monica, CA: RAND Corporation; 2009.
66. Zechmeister I, Schumacher I. The impact of health technology assessment reports on decision making in Austria. Int J Technol Assess Health Care. 2012;28:77-84. http://dx.doi.org/10.1017/S0266462311000729.
67. Sarli CC, Dubinsky EK, Holmes KL. Beyond citation analysis: a model for assessment of research impact. J Med Lib Assoc. 2010;98:17-23. http://dx.doi.org/10.3163/1536-5050.98.1.008.
68. Buxton M, Hanney S, Jones T. Estimating the economic value to societies of the impact of health research: a critical review. Bull World Health Organ. 2004;82:733-9.
69. Spaapen J, Dijstelbloem H, Wamelink F. Evaluating Research in Context. A Method for Comprehensive Assessment. 2nd ed. The Hague: Consultative Committee of Sector Councils for Research and Development; 2007.
70. Higher Education Funding Council for England (HEFCE). Research Excellence Framework 2014: Overview Report by Main Panel A and Sub-panels 1 to 6. London: HEFCE; 2015. www.ref.ac.uk/media/ref/content/expanel/member/Main%20Panel%20A%20overview%20report.pdf. Accessed 12 Nov 2016.
71. Higher Education Funding Council for England. Search REF Impact Case Studies. 2014. http://impact.ref.ac.uk/CaseStudies/search1.aspx. Accessed 16 Nov 2016.
72. Higher Education Funding Council for England. Memorandum Submitted by the Higher Education Funding Council for England (FC 50). In: House of Commons Science & Technology Committee (2010). The Impact of Spending Cuts on Science and Scientific Research, vol. 11. HC 335-11. London: The Stationery Office; p. 65-7. http://www.publications.parliament.uk/pa/cm200910/cmselect/cmsctech/335/335ii.pdf. Assessed 10 Feb 2017
73. Mushkin S. Biomedical research: costs and benefits. Cambridge, MA: Ballinger; 1979.
74. Economics A. Exceptional Returns: The Value of Investing in Health R&D in Australia. Sydney, NSW: Australian Society for Medical Research; 2003.
75. Roback K, Dalal K, Carlsson P. Evaluation of health research: measuring costs and socioeconomic effects. Int J Prev Med. 2011;2:203-15.
76. Buxton M, Hanney S, Morris S, Sundmacher L, Mestre-Ferrandiz J, Garau M, et al. Medical Research: What's it Worth? Estimating the Economic Benefits from Medical Research in the UK. London: UK Evaluation Forum; 2008.
77. Glover M, Buxton M, Guthrie S, Hanney S, Pollitt A, Grant J. Estimating the returns to UK publicly funded cancer-related research in terms of the net value of improved health outcomes. BMC Med. 2014;12:99. http://dx.doi.org/10.1186/1741-7015-12-99.
78. de Oliveira C, Nguyen HV, Wijeysundera HC, Wong WW, Woo G, Grootendorst P, et al. Estimating the payoffs from cardiovascular disease research in Canada: an economic analysis. Can Med Assoc Open Access J. 2013;1:E83-90. http://dx.doi.org/10.9778/cmajo.20130003.
79. Guthrie S, Hafner M, Bienkowska-Gibbs T, Wooding S. Returns on research funded under the NIHR Health Technology Assessment (HTA) Programme. Health Technol Assess. 2015;19:67.
80. Alberta Heritage Foundation for Medical Research (AHFMR). A study of the impact of 2001-2002 health technology assessment products: final report. Edmonton: AHFMR; 2003.
81. Aymerich M, Carrion C, Gallo P, Garcia M, Lopez-Bermejo A, Quesada M, et al. Measuring the payback of research activities: a feasible ex-post evaluation methodology in epidemiology and public health. Soc Sci Med. 2012;75:505-10. http://dx.doi.org/10.1016/j.socscimed.2012.03.044.
82. Catalan Agency for Health Technology Assessment and Research. Impact Assessment of the TV3 Telethon on Biomedical Research in Catalonia. Barcelona: Fundació La Marató de TV3, Agència d'Avaluació de Tecnologia i Recerca Mèdiques (AATRM); 2006.
83. Cohen G, Schroeder J, Newson R, King L, Rychetnik L, Milat AJ, et al. Does health intervention research have real world policy and practice impacts: testing a new impact assessment tool. Health Res Policy Syst. 2015;13:3. http://dx.doi.org/10.1186/1478-4505-13-3.
84. NHS Executive Trent. An evaluation of regional R&D funding schemes in Trent. Sheffield: NHS Executive Trent; 1997.
85. Shani S, Siebzehner MI, Luxenburg O, Shemer J. Setting priorities for the adoption of healthtechnologies on a national level: the Israeli experience. Health Policy. 2000;54:169-85.
86. Stryer D, Tunis S, Hubbard H, Clancy C. The outcomes of outcomes and effectiveness research: impacts and lessons from the first decade. Health Serv Res. 2000;35(5 Pt 1):977-93.
87. Graham KER, Chorzempa HL, Valentine PA, Magnan J. Evaluating health research impact: development and implementation of the Alberta Innovates - Health Solutions impact framework. Res Eval. 2012;21:354-67. http://dx.doi.org/10.1093/reseval/rvs027.
88. Searles A, Doran C, Attia J, Knight D, Wiggers J, Deeming S, et al. An approach to measuring and encouraging research translation and research impact. Health Res Policy Syst. 2016;14:60. doi: 10.1186/s12961-016-0131-2.
89. Adam T, Røttingen JA, Kieny MP. Informing the establishment of the WHO Global Observatory on Health Research and Development: a call for papers. Health Res Policy Syst. 2015;13:9.
90. Brennan S, McKenzie JE, Turner T, Redman S, Makkar S, Williamson A, et al. Development and validation of SEER (Seeking, Engaging with and Evaluating Research): a measure of policymakers' capacity to engage with and use research. Health Res Policy Syst. 2017;15:1.
91. Shroff Z, Aulakh B, Gilson L, Agyepong IA, El-Jardali F, Ghaffar A. Incorporating research evidence into decision-making processes: researcher and decision-maker perceptions from five low- and middle-income countries. Health Res Policy Syst. 2015;13:70. doi: 10.1186/s12961-015-0059-y.
92. National Breast Cancer Foundation. World First in Accountability. 2013. http://nbcf.org.au/news/research-news/world-first-in-accountability/. Accessed 16 Nov 2016