Models and applications for measuring the impact of health research: Update of a systematic review for the health technology assessment programme

Health Technology Assessment

Volumn 20, Issue 76, 2016, Pages

  Raftery, James ^a   Hanney, Steve ^b   Greenhalgh, Trish ^c   Glover, Matthew ^b   Blatch Jones, Amanda ^a  

^a UNIVERSITY OF SOUTHAMPTON   (United Kingdom)

^b BRUNEL UNIVERSITY   (United Kingdom)

^c UNIVERSITY OF OXFORD   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

ADOPTION; ATTENTION; CINAHL; COCHRANE LIBRARY; CONTROLLED CLINICAL TRIAL; CONTROLLED STUDY; EMBASE; ENGLAND; FUNDING; HUMAN; LOGIC; MEDICAL RESEARCH; MEDLINE; META ANALYSIS; MODEL; NARRATIVE; OUTCOMES RESEARCH; PRACTICE GUIDELINE; PUBLICATION; RANDOMIZED CONTROLLED TRIAL; SYSTEMATIC REVIEW;

EID: 84992361843     PISSN: 13665278     EISSN: 20464924     Source Type: Journal    
DOI: 10.3310/hta20760     Document Type: Review

References (414)

1. Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet 2009;374:86–9. http://dx.doi.org/10.1016/S0140-6736(09)60329-9.
2. Hanney S, Buxton M, Green C, Coulson D, Raftery J. An assessment of the impact of the NHS Health Technology Assessment Programme. Health Technol Assess 2007;11(53). http://dx.doi.org/10.3310/hta11530.
3. Raftery J, Hanney S, Green C, Buxton M. Assessing the impact of England’s National Health Service R&D Health Technology Assessment program using the ‘payback’ approach. Int J Technol Assess Health Care 2009;25:1–5. http://dx.doi.org/10.1017/S0266462309090011.
4. Banzi R, Moja L, Pistotti V, Facchini A, Liberati A. Conceptual frameworks and empirical approaches used to assess the impact of health research: an overview of reviews. Health Res Policy Syst 2011;9:26. http://dx.doi.org/10.1186/1478-4505-9-26.
5. Boaz A, Fitzpatrick S, Shaw B. Assessing the impact of research on policy: a literature review. Sci Public Policy 2009;36:255–70. http://dx.doi.org/10.3152/030234209X436545.
6. Bornmann L. What is societal impact of research and how can it be assessed? A literature survey. J Am Soc Inf Sci Technol 2013;64:217–33. http://dx.doi.org/10.1002/asi.22803.
7. Canadian Academy of Health Sciences (CAHS). Making an Impact, A Preferred Framework and Indicators to Measure Returns on Investment in Health Research. Ottawa, ON: CAHS; 2009.
8. Department for International Development (DFID). What is the Evidence on the Impact of Research on International Development? London: DFID; 2014.
9. Guthrie S, Wamae W, Diepeveeen S, Wooding S, Grant J. Measuring Research: A Guide to Research Evaluation Frameworks and Tools. Santa Monica, CA: RAND Corporation; 2013.
10. Harris R. Impact of Research on Development Policy and Practice: An Annotated Bibliography. 2013. URL: www.rogharris.org/uploads/5/0/5/4/5054025/impact_of_research_on_development_policy_and_practice:annotated_bibliography.pdf (accessed 16 July 2016).
11. Moore G, Redman S, Haines M, Todd A. What works to increase the use of research in population health policy and programmes: a review. Evid Policy 2011;7:277–305. http://dx.doi.org/10.1332/174426411X579199.
12. Neufeld V, Cole DC, Boyd A, Njelesani D, Bates I, Hanney SR. Perspectives on Evaluating Global Health Research for Development: A Background Paper. Ottawa, ON: Canadian Coalition for Global Health Research; 2014.
13. Oliver K, Innvar S, Lorenc T, Woodman J, Thomas J. A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Serv Res 2014;14. http://dx.doi.org/10.1186/1472-6963-14-2.
14. Penfield T, Baker MJ, Scoble R, Wykes MC. Assessment, evaluations, and definitions of research impact: a review. Res Eval 2014;23:21–32. http://dx.doi.org/10.1093/reseval/rvt021.
15. Yazdizadeh B, Majdzadeh R, Salmasian H. Systematic review of methods for evaluating healthcare research economic impact. Health Res Policy Syst 2010;8:6. http://dx.doi.org/10.1186/1478-4505-8-6.
16. Committee on Measuring Economic and Other Returns on Federal Research Investments. Measuring the Impacts of Federal Investments in Research. A Workshop Summary. Washington, DC: The National Academies; 2011.
17. Orians CE, Abed J, Drew CH, Rose SW, Cohen JH, Phelps J. Scientific and public health impacts of the NIEHS Extramural Asthma Research Program: insights from primary data. Res Eval 2009;18:375–85. http://dx.doi.org/10.3152/095820209X480698.
18. Thornton I, Shaw A. Evaluating the Impact of Research Programmes. London: UK Collaboration on Development Sciences; 2013. URL: www.ukcds.org.uk/resources/evaluating-the-impact-ofresearch-programmes (accessed 16 July 2016).
19. UK Evaluation Forum. Medical Research: Assessing the Benefits to Society. London: UK Evaluation Forum; 2006.
20. Oliver K, Innvar S, Lorenc T, Woodman J, Thomas J. Barriers and facilitators of the use of evidence by policy makers: an updated systematic review. BMC Health Serv Res 2014;14:2.
21. Innvaer S, Vist G, Trommald M, Oxman A. Health policy-makers’ perceptions of their use of evidence: a systematic review. J Health Serv Res Policy 2002;7:239–44. http://dx.doi.org/10.1258/135581902320432778.
22. Buxton M, Hanney S, Morris S, Sundmacher L, Mestre-Ferrandiz J, Garau M, et al. Medical Research: What’s it Worth?: Estimating the Economic Benefits from Medical Research in the UK. London: UK Evaluation Forum; 2008.
23. HM Treasury. Spending Review 2010. Norwich: The Stationery Office; 2010.
24. Access Economics. Exceptional Returns: The Value of Investing in Health Research in Australia 11. Prepared for the Australian Society for Medical Research. Canberra, ACT: ASMR; 2008.
25. Deloitte Access Economics. Returns on NHMRC Funded Research and Development. 2011. URL: www.asmr.org.au/NHMRCReturns.pdf (accessed 14 October 2014).
26. Glover M, Buxton M, Guthrie S, Hanney S, Pollitt A, Grant J. Estimating the returns to UK publicly funded cancer-related research in terms of the net value of improved health outcomes. BMC Med 2014;12. http://dx.doi.org/10.1186/1741-7015-12-99.
27. Guthrie S, Hafner M, Bienkowska-Gibbs T, Wooding S. Returns on research funded under the NIHR Health Technology Assessment (HTA) Programme. Health Technol Assess 2015;19(67).
28. Buxton M, Hanney S, Jones T. Estimating the economic value to societies of the impact of health research: a critical review. Bull World Health Organ 2004;82:733–9.
29. Chalmers I, Haynes B. Systematic reviews: reporting, updating, and correcting systematic reviews of the effects of health care. BMJ 1994;309:862–5. http://dx.doi.org/10.1136/bmj.309.6958.862.
30. Raftery J, Powell J. Health Technology Assessment in the UK. Lancet 2013;382:1278–85. http://dx.doi.org/10.1016/S0140-6736(13)61724-9.
31. Thomas KS, Koller K, Dean T, O’Leary CJ, Sach TH, Frost A, et al. A multicentre randomised controlled trial and economic evaluation of ion-exchange water softeners for the treatment of eczema in children: the Softened Water Eczema Trial (SWET). Health Technol Assess 2011;15(8). http://dx.doi.org/10.3310/hta15080.
32. Dumville JC, Worthy G, Soares M, Bland J, Cullum N, Dowson C, et al. VenUS II: a randomised controlled trial of larval therapy in the management of leg ulcers. Health Technol Assess 2009;13(55). http://dx.doi.org/10.3310/hta13550.
33. Higher Education Funding Council for England (HEFCE). 2014 REF: Assessment Framework and Guidance on Submissions. Panel A Criteria. London: HEFCE; 2012.
34. Weiss C. The many meanings of research utilization. Public Admin Rev 1979;39:426–31.
35. Hanney SR, Gonzalez-Block MA, Buxton MJ, Kogan M. The utilisation of health research in policy-making: concepts, examples and methods of assessment. Health Res Policy Syst 2003;1:2. http://dx.doi.org/10.1186/1478-4505-1-2.
36. Wooding S, Hanney SR, Pollitt A, Grant J, Buxton MJ. Understanding factors associated with the translation of cardiovascular research: a multinational case study approach. Implement Sci 2014;9:47. http://dx.doi.org/10.1186/1748-5908-9-47.
37. Brutscher PB, Wooding S, Grant J. Health Research Evaluation Frameworks; An International Comparison Making an Impact: A Preferred Framework and Indicators to Measure Returns in Investment in Health Research. Appendix A (Commissioned Papers. Ottawa, ON: CAHS; 2009).
38. Grant J, Brutscher BP, Kirk SE, Butler L, Wooding S. Capturing Research Impacts: A Review of International Practice. Santa Monica, CA: RAND Europe; 2009.
39. Buxton M, Hanney S. How can payback from health services research be assessed? J Health Serv Res 1996;1:35–43.
40. Royal Netherlands Academy of Arts and Sciences. The Societal Impact of Applied Research Towards a Quality Assessment System. Amsterdam: Royal Netherlands Academy of Arts and Sciences; 2002.
41. Molas-Gallart J, Tang P, Morrow S. Assessing the non-academic impact of grant-funded socio-economic research: results from a pilot study. Res Eval 2000;9:171–82. http://dx.doi.org/10.3152/147154400781777269.
42. Spaapen J, Sylvain C. Societal Quality of Research: Toward a Method for the Assessment of the Potential Value of Research for Society. London: Science Policy Support Group; 1994.
43. Jacob R, Battista RN. Assessing technology assessment: early results of the Quebec experience. Int J Technol Assess Health Care 1993;9:564–72. http://dx.doi.org/10.1017/S0266462300005481.
44. Jacob R, McGregor M. Assessing the impact of health technology assessment. Int J Technol Assess Health Care 1997;13:68–80. http://dx.doi.org/10.1017/S0266462300010242.
45. Donovan C, Hanney S. The ‘Payback Framework’ explained. Res Eval 2011;20:181–3. http://dx.doi.org/10.3152/095820211X13118583635756.
46. Hanney S, Boaz A, Jones T, Soper B. Engagement in research: an innovative three stage review of the benefits for health-care performance. Health Serv Deliv Res 2013;1.
47. Nason E, Curran B, Hanney S, Janta B, Hastings G, O’Driscoll M, et al. Evaluating health research funding in Ireland: assessing the impacts of the Health Research Board of Ireland’s funding activities. Res Eval 2011;20:193–200. http://dx.doi.org/10.3152/095820211X12941371876823.
48. Hanney SR, Grant J, Wooding S, Buxton MJ. Proposed methods for reviewing the outcomes of health research: the impact of funding by the UK’s ’Arthritis Research Campaign’. Health Res Policy Syst 2004;2:4. http://dx.doi.org/10.1186/1478-4505-2-4.
49. Kogan M, Henkel M. Government and Research: The Rothschild Experiment in a Government Department. London: Heinemann Educational Books; 1983.
50. Henshall C. The impact of Payback research: developing and using evidence in policy. Res Eval 2011;20:257–8. http://dx.doi.org/10.3152/095820211X13118583635873.
51. Hanney SR, Watt A, Jones TH, Metcalf L. Conducting retrospective impact analysis to inform a medical research charity’s funding strategies: the case of Asthma UK. Allergy Asthma Clin Immunol 2013;9:17. http://dx.doi.org/10.1186/1710-1492-9-17.
52. Cohen G, Schroeder J, Newson R, King L, Rychetnik L, Milat AJ, et al. Does health intervention research have real world policy and practice impacts: testing a new impact assessment tool. Health Res Policy Syst 2015;13:3. http://dx.doi.org/10.1186/1478-4505-13-3.
53. Expert Panel for Health Directorate of the European Commission’s Research Innovation Directorate General. Review of Public Health Research Projects Financed under the Commission’s Framework Programmes for Health Research. Brussels: European Commission; 2013.
54. Action Medical Research. Investing in Our Future: Following Up the Impact of Research Training Fellowship Awards from Action Medical Research. Horsham: Action Medical Research; 2009.
55. Anderson R. The policy impact of population health surveys: an illustration of the measurement challenges using the NSW Health Survey. Evid Policy 2006;2:167–83. http://dx.doi.org/10.1332/174426406777068975.
56. Aymerich M, Carrion C, Gallo P, Garcia M, Lopez-Bermejo A, Quesada M, et al. Measuring the payback of research activities: a feasible ex-post evaluation methodology in epidemiology and public health. Soc Sci Med 2012;75:505–10. http://dx.doi.org/10.1016/j.socscimed.2012.03.044.
57. Bennett S, Paina L, Ssengooba F, Waswa D, M’Imunya JM. The impact of Fogarty International Center research training programs on public health policy and program development in Kenya and Uganda. BMC Public Health 2013;13:1–11. http://dx.doi.org/10.1186/1471-2458-13-770.
58. Berra S, Pons JMV, co-ordinators. Impact Assessment of the TV3 Telethon on Biomedical Research in Catalonia. Barcelona: Fundació La Marató de TV3, Agència d’Avaluació de Tecnologia i Recerca Mèdiques (AATRM); 2006.
59. Bunn F. The Impact of Systematic Reviews on Health Care Policy In England. PhD thesis. University of Hertfordshire; 2010.
60. Bunn F, Kendall E. Does nursing research impact on policy? A case study of health visiting research and UK health policy. J Res Nurs 2011;16:169–91. http://dx.doi.org/10.1177/1744987110392627.
61. Bunn F, Trivedi D, Alderson P, Hamilton L, Martin A, Iliffe S. The impact of cochrane systematic reviews: a mixed method evaluation of outputs from Cochrane Review Groups supported by the UK National Institute for Health Research. Syst Rev 2014;3:125. http://dx.doi.org/10.1186/2046-4053-3-125.
62. Donovan C, Butler L, Butt AJ, Jones TH, Hanney SR. Evaluation of the impact of National Breast Cancer Foundation-funded research. Med J Aust 2014;200:214–18. http://dx.doi.org/10.5694/mja13.10798.
63. Engel-Cox JA, Van Houten B, Phelps J, Rose SW. Conceptual model of comprehensive research metrics for improved human health and environment. Environ Health Perspect 2008;116:583–92. http://dx.doi.org/10.1289/ehp.10925.
64. Guinea J, Sela E, Gómez-Núñez A, Mangwende T, Ambali A, Ngum N, et al. Impact oriented monitoring: a new methodology for monitoring and evaluation of international public health research projects. Res Eval 2015;24:131–45. http://dx.doi.org/10.1093/reseval/rvu034.
65. Kalucy EC, Jackson Bowers E, McIntyre E, Reed R. The feasibility of determining the impact of primary health care research projects using the Payback Framework. Health Res Policy Syst 2009;7. http://dx.doi.org/10.1186/1478-4505-7-11.
66. Kwan P, Johnston J, Fung AY, Chong DS, Collins RA, Lo SV. A systematic evaluation of payback of publicly funded health and health services research in Hong Kong. BMC Health Serv Res 2007;7:121. http://dx.doi.org/10.1186/1472-6963-7-121.
67. Longmore J. Evaluation Of The Fellowship Scheme 2007–2012. Prepared for Addenbrooke’s Charitable Trust; 2014.
68. NHS Service Delivery Organisation Research Development Programme. The Impact of the NHS Service Delivery and Organisation Research and Development Programme 2001–2006. London: National Coordinating Centre for Service and Delivery Organisation; 2006.
69. Oortwijn J. Assessing the impact of health technology assessment in the Netherlands. Int J Technol Assess Health Care 2008;24:259–69. http://dx.doi.org/10.1017/S0266462308080355.
70. Reed RL, Kalucy EC, Jackson-Bowers E, McIntyre E. What research impacts do Australian primary health care researchers expect and achieve? Health Res Policy Syst 2011;9. http://dx.doi.org/10.1186/1478-4505-9-40.
71. RSM McClure Watters, Hanney S, Nason E. Evaluation of the impact of HSC R&D Funding in Northern Ireland, Including Benchmarking with other Countries. Belfast: RSM McClure Watters; 2012.
72. Schapper CC, Dwyer T, Tregear GW, Aitken M, Clay MA. Research performance evaluation: the experience of an independent medical research institute. Aust Health Rev 2012;36:218–23. http://dx.doi.org/10.1071/AH11057.
73. Scott JE, Blasinsky M, Dufour M, Mandal RJ, Philogene GS. An evaluation of the Mind Body Interactions and Health Program: assessing the impact of an NIH program using the Payback Framework. Res Eval 2011;20:185–92. http://dx.doi.org/10.3152/095820211X12941371876661.
74. The Madrillon Group. The Mind–Body Interactions and Health Program Outcome Evaluation. Final Report. Bethesda, MD: Office of Behavioral and Social Sciences Research, National Institutes of Health; 2011.
75. Johnston SC, Rootenberg JD, Katrak S, Smith WS, Elkins JS. Effect of a US National Institutes of Health programme of clinical trials on public health and costs. Lancet 2006;367:1319–27. http://dx.doi.org/10.1016/S0140-6736(06)68578-4.
76. Medical Research Council (MRC). Economic Impact Report 2012/13. Swindon: MRC; 2013.
77. Murphy L. The Value of Clinical Trials: A New Zealand Case Study. PhD thesis. Hobart, TAS: University of Tasmania; 2012.
78. Williams I, McIver S, Moore D, Bryan S. The use of economic evaluations in NHS decision-making: a review and empirical investigation. Health Technol Assess 2008;12(7). http://dx.doi.org/10.3310/hta12070.
79. Royal Netherlands Academy of Arts and Sciences. Standard Evaluation Protocol 2009–2015. Protocol for Research Assessment in the Netherlands. Amsterdam: Association of Universities in the Netherlands; 2010.
80. Meijer I. Societal Returns of Scientific Research. How Can we Measure it? Leiden: Centre for Science and Technology Studies, Leiden University; 2012.
81. Spaapen J, van Drooge L, Propp T, van der Meulen B, Shinn T, Marcovich A, et al. Social Impact Assessment Methods for Research and Funding Instruments Through the Study of Productive Interactions (SIAMPI): Final Report on Social Impacts of Research. Amsterdam: Royal Netherlands Academy of Arts and Sciences; 2011. URL: www.siampi.eu/Content/SIAMPI_Final%20report.pdf (accessed 16 July 2016).
82. Bodeau-Livinec F, Simon E, Montagnier-Petrissans C, Joël M-E, Féry-Lemonnier E. Impact of CEDIT recommendations: an example of health technology assessment in a hospital network. Int J Technol Assess Health Care 2006;22:161–8. http://dx.doi.org/10.1017/S0266462306050975.
83. Zechmeister I, Schumacher I. The impact of health technology assessment reports on decision making in Austria. Int J Technol Assess Health Care 2012;28:77–84. http://dx.doi.org/10.1017/S0266462311000729.
84. Adam P, Solans-Domènech M, Pons JMV, Aymerich M, Berra S, Guillamon I, et al. Assessment of the impact of a clinical and health services research call in Catalonia. Res Eval 2012;21:319–28. http://dx.doi.org/10.1093/reseval/rvs024.
85. Graham KER, Chorzempa HL, Valentine PA, Magnan J. Evaluating health research impact: development and implementation of the Alberta Innovates – Health Solutions impact framework. Res Eval 2012;21:354–67. http://dx.doi.org/10.1093/reseval/rvs027.
86. Saskatchewan Health Research Foundation (SHRF). Measuring the Value of Saskatchewan’s Health Research. Saskatoon, SK: SHRF; 2013.
87. Solans-Domènech M, Adam P, Guillamon I, Permanyer-Miralda G, Pons JMV, Escarrabill J. Impact of clinical and health services research projects on decision-making: a qualitative study. Health Res Policy Syst 2013;11. http://dx.doi.org/10.1186/1478-4505-11-15.
88. Laws R, King L, Hardy LL, Milat A, Rissel C, Newson R, et al. Utilization of a population health survey in policy and practice: a case study. Health Res Policy Syst 2013;11:4. http://dx.doi.org/10.1186/1478-4505-11-4.
89. Milat AJ, Laws R, King L, Newson R, Rychetnik L, Rissel C, et al. Policy and practice impacts of applied research: a case study analysis of the New South Wales Health Promotion Demonstration Research Grants Scheme 2000–2006. Health Res Policy Syst 2013;11:5. http://dx.doi.org/10.1186/1478-4505-11-5.
90. Drew CH, Pettibone KG, Ruben E. Greatest ‘HITS’: A new tool for tracking impacts at the National Institute of Environmental Health Sciences. Res Eval 2013;22:307–15. http://dx.doi.org/10.1093/reseval/rvt022.
91. Liebow E, Phelps J, Van Houten B, Rose S, Orians C, Cohen J, et al. Toward the assessment of scientific and public health impacts of the National Institute of Environmental Health Sciences Extramural Asthma Research Program using available data. Environ Health Perspect 2009;117:1147–54. http://dx.doi.org/10.1289/ehp.0800476.
92. Williams V, Eiseman E, Landree E, Adamson D. Demonstrating and Communicating Research Impact. Preparing NIOSH Programs for External Review. Ann Arbor, MI: University of Michigan, RAND; 2009.
93. Wellcome Trust. Assessment Framework Report 2012/13. London: Wellcome Trust; 2014.
94. Eriksen KS, Hervik A. Impacts of Neck Injuries Research at Chalmers University of Technology. Summary. Stockholm: VINNOVA – Swedish Agency for Innovation Systems; 2005.
95. Meagher L, Lyall C, Nutley S. Flows of knowledge, expertise and influence: a method for assessing policy and practice impacts from social science research. Res Eval 2008;17:163–73. http://dx.doi.org/10.3152/095820208X331720.
96. Caddell AJ, Hatchette JE, McGrath PJ. Examining the impact of health research facilitated by small peer-reviewed research operating grants in a women’s and children’s health centre. BMC Res Notes 2010;3:107. http://dx.doi.org/10.1186/1756-0500-3-107.
97. Kuruvilla S, Mays N, Walt G. Describing the impact of health services and policy research. J Health Serv Res Policy 2007;12(Suppl. 1):23–31. http://dx.doi.org/10.1258/135581907780318374.
98. Wilson P, Petticrew M, Calnan M, Nazareth I. Disseminating research findings: what should researchers do? A systematic scoping review of conceptual frameworks. Implement Sci 2010;5:91. http://dx.doi.org/10.1186/1748-5908-5-91.
99. Sainty M. Research impact: a United Kingdom Occupational Therapy Research Foundation perspective. Br J Occup Ther 2013;76:528–37. http://dx.doi.org/10.4276/030802213X13861576675204.
100. Mostert SP, Ellenbroek S, Meijer I, van Ark G, Klasen EC. Societal output and use of research performed by health research groups. Health Res Policy Syst 2010;8:30. http://dx.doi.org/10.1186/1478-4505-8-30.
101. Evans D, Coad J, Cottrell K, Dalrymple J, Davies R, Donald C, et al. Public involvement in research: assessing impact through a realist evaluation. Health Serv Delivery Res 2014;2(36). 1
102. Rycroft-Malone J, Wilkinson J, Burton CR, Harvey G, McCormack B, Graham I, et al. Collaborative action around implementation in Collaborations for Leadership in Applied Health Research and Care: towards a programme theory. J Health Servs Res Policy 2013;18(Suppl. 3):13–26. http://dx.doi.org/10.1177/1355819613498859.
103. Medical Research Council (MRC). Outputs, Outcomes and Impact of MRC Research: 2012 Report. Swindon: MRC; 2013. URL: www.mrc.ac.uk/publications/browse/outputs-report-2012/(accessed 11 July 2016).
104. Wooding S, Nason E, Starkey T, Hanney S, Grant J. Mapping the Impact: Exploring the Payback of Arthritis Research. Santa Monica, CA: RAND Corporation; 2009.
105. Group of Eight. Excellence in Innovation: Research Impacting our Nation’s Future: Assessing the Benefits. Adelaide, SA: Australian Technology Network of Universities; 2012.
106. Higher Education Funding Council for England (HEFCE). Research Excellence Framework 2014: Overview Report by Main Panel A and Sub-panels 1 to 6. London: HEFCE; 2015. URL: www.ref.ac.uk/media/ref/content/expanel/member/Main%20Panel%20A%20overview%20report.pdf (accessed 1 February 2015).
107. Muir R, Arthur E, Berman T, Sedgley S, Herlick Z, Fullgrabe H. Metrics for Research Commercialisation: A Report to the Coordination Committee on Science and Technology. Canberra, ACT: Coordination Committee on Science and Technology; 2005.
108. Department for Innovation, Universities and Science (DIUS). Economic Impacts of Investment in Research & Innovation. London: DIUS; 2007.
109. Spaapen J, Dijstelbloem H, Wamelink F. Evaluating Research in Context. A Method for Comprehensive Assessment. 2nd edn. The Hague: Consultative Committee of Sector Councils for Research and Development; 2007.
110. Spaapen J, van Drooge L. Introducing ‘productive interactions’ in social impact assessment. Res Eval 2011;20:211–18. http://dx.doi.org/10.3152/095820211X12941371876742.
111. Hansen J, Muscat N, Keskimaki L. Measuring and improving the societal impact of health care research. Eurohealth 2013;19:32–5.
112. Hage J. Making an Impact: A Preferred Framework and Indicators to Measure Returns in Investment in Health Research. Appendix A (Commissioned Papers). Ottawa, ON: CAHS; 2009. URL: www.cahs-acss.ca/wp-content/uploads/2011/09/ROI_Appendices.pdf (accessed 11 July 2016).
113. Lewis S MP, Barre L. Pillar II. Estimating the Return on Investment for Health Services Research: A Theoretical and Empirical Analysis. Appendix A (Commissioned Papers). Ottawa, ON: CAHS; 2009. URL: www.cahs-acss.ca/wp-content/uploads/2011/09/ROI_Appendices.pdf (accessed 11 July 2016).
114. Shiel A, Di Ruggiero E. Pillar IV: Population and Public Health Research – ’Assessing the Return on Canada’s Public Investment in Population and Public Health Research: Methods and Metrics’. Ottawa, ON: CAHS; 2009. URL: www.cahs-acss.ca/wp-content/uploads/2011/09/ROI_Appendices.pdf (accessed 11 July 2016).
115. Frank C, Nason E. Health research: measuring the social, health and economic benefits. CMAJ 2009;180:528–34. http://dx.doi.org/10.1503/cmaj.090016.
116. Kok MO, Schuit AJ. Contribution mapping: a method for mapping the contribution of research to enhance its impact. Health Res Policy Syst 2012;10. http://dx.doi.org/10.1186/1478-4505-10-21.
117. Bozeman B. Public Value Mapping of Science Outcomes: Theory and Method. In Bozeman B, Sarewitz D, Feinson S, Foladori G, Gaughan M, Gopta A, et al., editors. Knowledge Flows and Knowledge Collectives: Understanding the Role of Science and Technology Policies in Development. Washington, DC: Center for Science Policy and Outcomes; 2003. pp. 3–48.
118. Sarli CC, Dubinsky EK, Holmes KL. Beyond citation analysis: a model for assessment of research impact. J Med Lib Assoc 2010;98:17–23. http://dx.doi.org/10.3163/1536-5050.98.1.008.
119. United Way. Measuring Program Outcomes: A Practical Approach. Alexandria, VA: United Way of America; 1996.
120. Weiss AP. Measuring the impact of medical research: moving from outputs to outcomes. Am J Psychiatry 2007;164:206–14. http://dx.doi.org/10.1176/ajp.2007.164.2.206.
121. W.K. Kellogg Foundation. Using Logic Models to Being Together Planning, Evaluation and Action: Logic Model Development Guide. 2004. URL: www.smartgivers.org/uploads/logicmodelguidepdf.pdf (accessed 10 June 2016).
122. David PA, Mowery D, Steinmueller WE. Analysing the economic payoffs from basic research. Econ Innov New Technol 1992;2:73–90. http://dx.doi.org/10.1080/10438599200000006.
123. Kuruvilla S, Mays N, Pleasant A, Walt G. Describing the impact of health research: a research impact framework. BMC Health Serv Res 2006;6:134. http://dx.doi.org/10.1186/1472-6963-6-134.
124. Dembe AE, Lynch MS, Gugiu PC, Jackson RD. The translational research impact scale: development, construct validity, and reliability testing. Eval Health Prof 2014;37:50–70. http://dx.doi.org/10.1177/0163278713506112.
125. Van Ark G, Klasen E. Societal impact of health research. J Health Sci 2007;85:259–61.
126. Morgan Jones M, Grant J. Making the Grade: Methodologies for Assessing and Evidencing Research Impact. In Dean A, Wykes M, Stevens H, editors. 7 Essays on Impact. DESCRIBE Project Report for Jisc. Exeter: University of Exeter; 2013. pp. 25–43. URL: www.exeter.ac.uk/media/universityofexeter/research/ourresearchexcellence/describeproject/pdfs/2013_06_04_7_Essays_ on_Impact_FINAL.pdf (accessed 11 July 2016).
127. Medical Research Council. About researchfish. URL: www.mrc.ac.uk/funding/guidance-for-mrcaward- holders/researchfish/about-researchfish/(accessed 6 May 2015).
128. Donovan C. The Australian Research Quality Framework: a live experiment in capturing the social, economic, environmental, and cultural returns of publicly funded research. New Directions Eval 2008;47–60.
129. Milat AJ, Bauman AE, Redman S. A narrative review of research impact assessment models and methods. Health Res Policy Syst 2015;13:18. http://dx.doi.org/10.1186/s12961-015-0003-1.
130. Chang W-C, Tung Y-C, Huang C-H, Yang M-C. Performance improvement after implementing the balanced scorecard: a large hospital’s experience in Taiwan. Total Qual Manag Bus Excellence 2008;19:1143–54. http://dx.doi.org/10.1080/14783360802323560.
131. University Health Network. Achieving Our Vision: Balanced Scorecard. URL: www.uhn.ca/corporate/AboutUHN/AchievingOurVision/Pages/balanced_scorecard.aspx (accessed 7 October 2014).
132. Gutman MA, Barker DC, Samples-Smart F, Morley C. Evaluation of active living research: progress and lessons in building a new field. Am J Prev Med 2009;36(Suppl. 2):22–33. http://dx.doi.org/10.1016/j.amepre.2008.10.009.
133. Arnold E. Understanding long-term impacts of R&D funding: the EU Framework Programme. Res Eval 2012;21:332–43. http://dx.doi.org/10.1093/reseval/rvs025.
134. Schumacher I, Zechmeister I. Assessing the impact of health technology assessment on the Austrian healthcare system. Int J Technol Assess Health Care 2013;29:84–91. http://dx.doi.org/10.1017/S0266462312000748.
135. Rispel LC, Doherty J. Research in support of health systems transformation in South Africa: The experience of the Centre for Health Policy. J Public Health Policy 2011:32(Suppl. 1):10–29. http://dx.doi.org/10.1057/jphp.2011.33.
136. Hera. End of Programme Summative Evaluation of the Africa Health Systems Initiative Support to African Research Partnerships (AHSI-RES). Reet: Hera; 2014.
137. Brambila C, Ottolenghi E, Marin C, Bertrand JT. Getting results used: evidence from reproductive health programmatic research in Guatemala. Health Policy Plan 2007;22:234–45. http://dx.doi.org/10.1093/heapol/czm013.
138. Gold M, Taylor EF. Moving research into practice: lessons from the US Agency for Healthcare Research and Quality’s IDSRN program. Implement Sci 2007;2:9. http://dx.doi.org/10.1186/1748-5908-2-9.
139. Kingwell BA, Anderson GP, Duckett SJ, Hoole EA, Jackson-Pulver LR, Khachigian LM, et al. Evaluation of NHMRC funded research completed in 1992, 1997 and 2003: gains in knowledge, health and wealth. Med J Aust 2006;184:282–6.
140. Poortvliet EP, Vijfvinkel D, Vennekens A, van Hoesel P, Daue F. Study into the Impact of the Belgian Health Care Knowledge Centre. Brussels: Belgian Health Care Knowledge Centre; 2010.
141. Zachariah R, Guillerm N, Berger S, Kumar AMV, Satyanarayana S, Bissell K, et al. Research to policy and practice change: is capacity building in operational research delivering the goods? Trop Med Int Health 2014;19:1068–75. http://dx.doi.org/10.1111/tmi.12343.
142. Murphy E, Dingwall R, Greatbatch D, Parker S, Watson P. Qualitative research methods in health technology assessment: a review of the literature. Health Technol Assess 1998;2(16).
143. Britten N, Campbell R, Pound P, Morgan M, Daker-White G, Pill R, et al. Evaluating meta ethnography: systematic analysis and synthesis of qualitative research. Health Technol Assess 2011;15(43).
144. Nyssen OP, Taylor SJC, Wong G, Higgins A, Steed EM, Bourke L, et al. Does therapeutic writing help people with long-term conditions? Systematic review, realist synthesis and economic considerations. Health Technol Assess 2016:20(27). http://dx.doi.org/10.3310/hta20270.
145. Waterman H, Tillen D, Dickson R, De Koning K. Action research: a systematic review and guidance for assessment. Health Technol Assess 2000;5(23).
146. Nilsen P, Stahl C, Roback K, Cairney P. Never the twain shall meet? – a comparison of implementation science and policy implementation research. Implement Sci 2013;8:63. http://dx.doi.org/10.1186/1748-5908-8-63.
147. Dewey J. The Theory Of Inquiry. New York, NY: Holt, Rinehart & Wiston; 1938.
148. Martin BR. The Research Excellence Framework and the ‘impact agenda’: are we creating a Frankenstein monster? Res Eval 2011;20:247–54. http://dx.doi.org/10.3152/095820211X13118583635693.
149. Upton S, Vallance P, Goddard J. From outcomes to process: evidence for a new approach to research impact assessment. Res Eval 2014;23:352–65. http://dx.doi.org/10.1093/reseval/rvu021.
150. Bozeman B, Rogers JD. A churn model of scientific knowledge value: internet researchers as a knowledge value collective. Res Policy 2002;31:769–94. http://dx.doi.org/10.1016/S0048-7333(01)00146-9.
151. Briggle AR. Opening the black box: the social outcomes of scientific research. Social Epistemol 2014;28:153–66. http://dx.doi.org/10.1080/02691728.2013.782584.
152. Meyer RM. Pillar II: Clinical Research. How to Optimally Measure the Impact of Health Research Funding in Clinical Research. Ottawa, ON: CAHS; 2009. URL: www.cahs-acss.ca/wp-content/uploads/2011/09/ROI_Appendices.pdf (accessed 11 July 2016).
153. Kogan M, Henkel M, Hanney S. Government and Research: 30 Years of Evolution. Dordrecht: Springer; 2006. http://dx.doi.org/10.1007/1-4020-4446-1.
154. Lomas J. The in-between world of knowledge brokering. BMJ 2007;334:129–32. http://dx.doi.org/10.1136/bmj.39038.593380.AE
155. Thomas P. The Aims and Outcomes of Social Policy Research. London: Croom Helm; 1985.
156. Amara N, Ouimet M, Landry R. New evidence on instrumental, conceptual, and symbolic utilization of university research in government agencies. Sci Commun 2004;26:75–106. http://dx.doi.org/10.1177/1075547004267491.
157. Wooding S, Pollitt A, Diepeveen S, Guthrie S, Jones MM, Chonaill SN, et al. Mental Health Retrosight. RAND for the Alliance of Mental Health Research Funders; 2013. URL: http://lbr.rand.org/content/dam/rand/pubs/research_reports/RR400/RR452/RAND_RR452.pdf (accessed 2 December 2014).
158. de Goede J, Putters K, van Oers H. Utilization of epidemiological research for the development of local public health policy in the Netherlands: a case study approach. Soc Sci Med 2012;74:707–14. http://dx.doi.org/10.1016/j.socscimed.2011.11.014.
159. de Jong SP, van Arensbergen P, Daemen F, van der Meulen B, van den Besselaar P. Evaluation of research in context: an approach and two cases. Res Eval 2011;20:61–72. http://dx.doi.org/10.3152/095820211X12941371876346.
160. Nicolini D, Powell J, Conville P, Martinez-Solano L. Managing knowledge in the healthcare sector. A review. Int J Manag Rev 2008;10:245–63. http://dx.doi.org/10.1111/j.1468-2370.2007.00219.x
161. Cherney A, Head B, Boreham P, Povey J, Ferguson M. Research utilization in the social sciences a comparison of five academic disciplines in Australia. Sci Commun 2013;35:780–809. http://dx.doi.org/10.1177/1075547013491398.
162. Lemay MA, Sá C. Complexity sciences: towards an alternative approach to understanding the use of academic research. Evid Policy 2012;8:473–94. http://dx.doi.org/10.1332/174426412X660133.
163. Contandriopoulos D, Lemire M, Denis JL, Tremblay E. Knowledge exchange processes in organizations and policy arenas: a narrative systematic review of the literature. Milbank Q 2010;88:444–83. http://dx.doi.org/10.1111/j.1468-0009.2010.00608.x
164. Westhorp G. Realist Impact Evaluation – An Introduction. London: Overseas Development Institute; 2014.
165. Pawson R, Tilley N. Realistic Evaluation. New York, NY: Sage; 1997.
166. Rycroft-Malone J, Wilkinson JE, Burton CR, Andrews G, Ariss S, Baker R, et al. Implementing health research through academic and clinical partnerships: a realistic evaluation of the Collaborations for Leadership in Applied Health Research and Care (CLAHRC). Implement Sci 2011;6:74. http://dx.doi.org/10.1186/1748-5908-6-74.
167. Brett J, Staniszewska S, Mockford C, Herron-Marx S, Hughes J, Tysall C, et al. A systematic review of the impact of patient and public involvement on service users, researchers and communities. Patient 2014;7:387–95. http://dx.doi.org/10.1007/s40271-014-0065-0.
168. Evans BA, Bedson E, Bell P, Hutchings H, Lowes L, Rea D, et al. Involving service users in trials: developing a standard operating procedure. Trials 2013;14:219. http://dx.doi.org/10.1186/1745-6215-14-219.
169. Staniszewska S, Brett J, Mockford C, Barber R. The GRIPP checklist: strengthening the quality of patient and public involvement reporting in research. Int J Technol Assess Health Care 2011;27:391–99. http://dx.doi.org/10.1017/S0266462311000481.
170. Ward PR, Thompson J, Barber R, Armitage CJ, Boote JD, Cooper CL, et al. Critical perspectives on ‘consumer involvement’in health research Epistemological dissonance and the know-do gap. J Sociol 2010;46:63–82. http://dx.doi.org/10.1177/1440783309351771.
171. Macaulay AC, Commanda LE, Freeman WL, Gibson N, McCabe ML, Robbins CM, et al. Participatory research maximises community and lay involvement. BMJ 1999;319:774. http://dx.doi.org/10.1136/bmj.319.7212.774.
172. White SC. Depoliticising development: the uses and abuses of participation. Dev Pract 1996;6:6–15. http://dx.doi.org/10.1080/0961452961000157564.
173. Jagosh J, Macaulay AC, Pluye P, Salsberg J, Bush PL, Henderson J, et al. Uncovering the benefits of participatory research: implications of a realist review for health research and practice. Milbank Q 2012;90:311–46. http://dx.doi.org/10.1111/j.1468-0009.2012.00665.x
174. Martin S. Co-production of social research: strategies for engaged scholarship. Public Money Manag 2010;30:211–18. http://dx.doi.org/10.1080/09540962.2010.492180.
175. Lasker RD, Weiss ES, Miller R. Partnership synergy: a practical framework for studying and strengthening the collaborative advantage. Milbank Q 2001;79:179–205. http://dx.doi.org/10.1111/1468-0009.00203.
176. Cacari-Stone L, Wallerstein N, Garcia AP, Minkler M. The promise of community-based participatory research for health equity: a conceptual model for bridging evidence with policy. Am J Public Health 2014;104:1615–23. http://dx.doi.org/10.2105/AJPH.2014.301961.
177. Kessler R, Glasgow RE. A proposal to speed translation of healthcare research into practice: dramatic change is needed. Am J Prev Med 2011;40:637–44. http://dx.doi.org/10.1016/j.amepre.2011.02.023.
178. Glasgow RE, Green LW, Taylor MV, Stange KC. An evidence integration triangle for aligning science with policy and practice. Am J Prev Med 2012;42:646–54. http://dx.doi.org/10.1016/j.amepre.2012.02.016.
179. Gibbons M, Limoges C, Nowotny H, Schwartzman S, Scott P, Trow M. The New Production of Knowledge: The Dynamics of Science and Research in Contemporary Societies. London: Sage; 1994.
180. McGough R, Rubenstein S. Academia. Shaping the new science networks. Health Serv J 2013;123:32–3.
181. Currie G, Lockett A, El Enany N. From what we know to what we do: lessons learned from the translational CLAHRC initiative in England. J Health Serv Res Policy 2013;18(Suppl. 3):27–39. http://dx.doi.org/10.1177/1355819613500484.
182. Caldwell SE, Mays N. Studying policy implementation using a macro, meso and micro frame analysis: the case of the Collaboration for Leadership in Applied Health Research & Care (CLAHRC) programme nationally and in North West London. Health Res Policy Syst 2012;10:32. http://dx.doi.org/10.1186/1478-4505-10-32.
183. Evans S, Scarbrough H. Supporting knowledge translation through collaborative translational research initiatives: ‘Bridging’ versus ‘blurring’ boundary-spanning approaches in the UK CLAHRC initiative. Soc Sci Med 2014;106:119–27. http://dx.doi.org/10.1016/j.socscimed.2014.01.025.
184. Kislov R, Waterman H, Harvey G, Boaden R. Rethinking capacity building for knowledge mobilisation: developing multilevel capabilities in healthcare organisations. Implement Sci 2014;9:166. http://dx.doi.org/10.1186/s13012-014-0166-0.
185. Soper B, Yaqub O, Hinrichs S, Marjanovich S, Drabble S, Hanney S, et al. CLAHRCs in practice: combined knowledge transfer and exchange strategies, cultural change, and experimentation. J Health Serv Res Policy 2013;18(Suppl. 3):53–64. http://dx.doi.org/10.1177/1355819613499903.
186. Ling T, Soper B, Marjanovic S, Celia C, Yaqub O, Reding A, et al. Delivering the Aims of the CLAHRCs: Evaluating CLAHRCs’ Strategies and Contributions. Santa Monica, CA: RAND Corporation; 2011. URL: www.rand.org/pubs/working_papers/WR820.html (accessed 11 July 2016).
187. Nowotny H, Scott P, Gibbons M. Mode 2 revisited: the new production of knowledge. Minerva 2003;41:179–94. http://dx.doi.org/10.1023/A:1025505528250.
188. King G, Servais M, Forchuk C, Chalmers H, Currie M, Law M, et al. Features and impacts of five multidisciplinary community-university research partnerships. Health Soc Care Community 2010;18:59–69.
189. Wehrens R, Bekker M, Bal R. Hybrid management configurations in joint research. Sci Technol Human Values 2014;39:6–41. http://dx.doi.org/10.1177/0162243913497807.
190. Hinchcliff R, Greenfield D, Braithwaite J. Is it worth engaging in multi-stakeholder health services research collaborations? Reflections on key benefits, challenges and enabling mechanisms. Int J Qual Health Care 2014;26:124–8. http://dx.doi.org/10.1093/intqhc/mzu009.
191. Hanney S, Kuruvilla S, Soper B, Mays N. Who needs what from a national health research system: lessons from reforms to the English Department of Health’s R&D system. Health Res Policy Syst 2010;8:11. http://dx.doi.org/10.1186/1478-4505-8-11.
192. Swan J, Bresnen M, Robertson M, Newell S, Dopson S. When policy meets practice: colliding logics and the challenges of ‘mode 2’ initiatives in the translation of academic knowledge. Organ Stud 2010;31:1311–40. http://dx.doi.org/10.1177/0170840610374402.
193. Orr K, Bennett M. Public Administration scholarship and the politics of coproducing academic–practitioner research. Public Adm Rev 2012;72:487–95. http://dx.doi.org/10.1111/j.1540-6210.2011.02522.x
194. De Savigny D, Adam T. Systems Thinking For Health Systems Strengthening. Geneva: Alliance for Health Policy and Systems Research, World Health Organization; 2009.
195. Shepherd J. How to Achieve More Effective Services: The Evidence Scosystem. Cardiff: What Works Network/Cardiff University; 2014.
196. Bennet A, Bennet D, Fafard K, Fonda M, Lomond T, Messier L, et al. Knowledge Mobilization in the Social Sciences and Humanities. Frost, WV: MQI Press; 2007.
197. Phipps D. A report detailing the development of a university-based knowledge mobilization unit that enhances research outreach and engagement. Scholar Res Commun 2012;2.
198. Phipps D, Jensen K, Myers JG. Applying Social Sciences Research For Public Benefit Using Knowledge Mobilization and Social Media. In Lopez-Varela A, editor. Theoretical and Methodological Approaches to Social Sciences and Knowledge Management. New York, NY: InTech; 2012. pp. 179–208.
199. Kislov R, Harvey G, Walshe K. Collaborations for Leadership in Applied Health Research and Care: lessons from the theory of communities of practice. Implement Sci 2011;6:64. http://dx.doi.org/10.1186/1748-5908-6-64.
200. Goffman E. Frame Analysis: An Essay on the Organization of Experience. Cambridge, MA: Harvard University Press Frame analysis; 1974.
201. Currie G, Fitzgerald L, Keen J, McBride A, Martin G, Rowley E, et al. An Organizational Behaviour Perspective upon CLAHRCs (Collaboratives for Leadership in Health Research and Care): Mediating Institutional Callenges through Change Agency. 2010. URL: www.download.bham.ac.uk/hsmc/graeme-currie.pdf (accessed 25 March 2015).
202. Ziman J. Is science losing its objectivity? Nature 1996;382:751–4. http://dx.doi.org/10.1038/382751a0.
203. Van de Ven AH, Johnson PE. Knowledge for theory and practice. Acad Manage Rev 2006;31:802–21. http://dx.doi.org/10.5465/AMR.2006.22527385.
204. Kincheloe JL. Describing the bricolage: conceptualizing a new rigor in qualitative research. Qual Inquiry 2001;7:679–92. http://dx.doi.org/10.1177/107780040100700601.
205. Fudenberg HH. Biomedical Institutions, Biomedical Funding, And Public Policy. New York, NY: Plenum Press; 1983. http://dx.doi.org/10.1007/978-1-4684-4448-3.
206. Afseth J. On the value of research in health care. Eur J Oral Sci 1997;105:454–6. http://dx.doi.org/10.1111/j.1600-0722.1997.tb00229.x
207. Weisbrod BA. Costs and benefits of medical research: a case study of poliomyelitis. J Polit Econ 1971:527–44. http://dx.doi.org/10.1086/259766.
208. United Kingdom Council for Scientific Policy. Second Report in Science Policy. London: HSMO; 1967.
209. Mushkin SJ, Landefeld JS. Biomedical Research: Costs and Benefits. Pensacola, FL: Ballinger Publishing; 1979.
210. Drummond MF, Davies LM, Ferris FL. Assessing the costs and benefits of medical research: the diabetic retinopathy study. Soc Sci Med 1992;34:973–81. http://dx.doi.org/10.1016/0277-9536(92)90128-D
211. Salter AJ, Martin BR. The economic benefits of publicly funded basic research: a critical review. Res Policy 2001;30:509–32. http://dx.doi.org/10.1016/S0048-7333(00)00091-3.
212. Mansfield E. Academic research and industrial innovation. Res Policy 1991;20:1–12. http://dx.doi.org/10.1016/0048-7333(91)90080-A
213. Mansfield E. Academic research and industrial innovation: an update of empirical findings. Res Policy 1998;26:773–6. http://dx.doi.org/10.1016/S0048-7333(97)00043-7.
214. Silverstein SC, Garrison HH, Heinig SJ. A few basic economic facts about research in the medical and related life sciences. FASEB J 1995;9:833–40.
215. Funding First. Exceptional Returns: The Economic Value of America’s Investment in Medical Research. New York, NY: Lasker Foundation; 2000.
216. Murphy KM, Topel RH. Measuring The Gains From Medical Research: An Economic Approach. Chicago, IL: University of Chicago Press; 2003. http://dx.doi.org/10.7208/chicago/9780226551791.001.0001.
217. Access Economics. Exceptional Returns: The Value of Investing in Health R&D in Australia. Sydney, NSW: Australian Society for Medical Research; 2003.
218. Roback K, Dalal K, Carlsson P. Evaluation of health research: measuring costs and socioeconomic effects. Int J Prev Med 2011;2:203–15.
219. de Oliveira C, Nguyen HV, Wijeysundera HC, Wong WW, Woo G, Grootendorst P, et al. Estimating the payoffs from cardiovascular disease research in Canada: an economic analysis. Can Med Assoc Open Access J 2013;1:E83–90. http://dx.doi.org/10.9778/cmajo.20130003.
220. Lakdawalla DN, Sun EC, Jena AB, Reyes CM, Goldman DP, Philipson TJ. An economic evaluation of the war on cancer. J Health Econ 2010;29:333–46. http://dx.doi.org/10.1016/j.jhealeco.2010.02.006.
221. McKeown T. The Role Of Medicine. Oxford: Blackwell; 1979.
222. OECD.Stat. PPPs and Exchange Rates. URL: https://stats.oecd.org/Index.aspx?DataSetCode=SNA_ TABLE4 (accessed 11 July 2016).
223. Cutler DM, McClellan M. Is technological change in medicine worth it? Health Aff (Millwood) 2001;20:11–29. http://dx.doi.org/10.1377/hlthaff.20.5.11.
224. Lichtenberg FR. The impact of new drug launches on longevity: evidence from longitudinal, disease-level data from 52 countries, 1982–2001. Int J Health Care Finance Econ 2005;5:47–73. http://dx.doi.org/10.1007/s10754-005-6601-7.
225. Luce BR, Mauskopf J, Sloan FA, Ostermann J, Paramore LC. The return on investment in health care: from 1980 to 2000. Value Health 2006;9:146–56. http://dx.doi.org/10.1111/j.1524-4733.2006.00095.x
226. Hanney SR, Castle-Clarke S, Grant J, Guthrie S, Henshall C, Mestre-Ferrandiz J, et al. How long does biomedical research take? Studying the time taken between biomedical and health research and its translation into products, policy, and practice. Health Res Policy Syst 2015;13:1. http://dx.doi.org/10.1186/1478-4505-13-1.
227. Guthrie S, Pollitt A, Hanney S, Grant J. Investigating Time Lags and Attribution in the Translation of Cancer Research: A Case Study Approach. Santa Monica, CA: RAND Corporation; 2014.
228. Raftery J, Young A, Stanton L, Milne R, Cook A, Turner D, et al. Clinical trial metadata: defining and extracting metadata on the design, conduct, results and costs of 125 randomised clinical trials funded by the National Institute for Health Research Health Technology Assessment programme. Health Technol Assess 2015;19(11). http://dx.doi.org/10.3310/hta19110.
229. Goudie AC, Sutton AJ, Jones DR, Donald A. Empirical assessment suggests that existing evidence could be used more fully in designing randomized controlled trials. J Clin Epidemiol 2010;63:983–91. http://dx.doi.org/10.1016/j.jclinepi.2010.01.022.
230. Jones AP, Conroy E, Williamson PR, Clarke M, Gamble C. The use of systematic reviews in the planning, design and conduct of randomised trials: a retrospective cohort of NIHR HTA funded trials. BMC Med Res Methodol 2013;13:50. http://dx.doi.org/10.1186/1471-2288-13-50.
231. Bhurke S, Cook A, Tallant A, Young A, Williams E, Raftery J. Are Systematic Reviews used in the Planning and Design of NIHR HTA Trials? A Retrospective Cohort. Oxford: Evidence Live Oxford; 2015.
232. Sutton AJ, Cooper NJ, Jones DR, Lambert PC, Thompson JR, Abrams KR. Evidence-based sample size calculations based upon updated meta-analysis. Stat Med 2007;26:2479–500. http://dx.doi.org/10.1002/sim.2704.
233. Roloff V, Higgins J, Sutton AJ. Planning future studies based on the conditional power of a meta-analysis. Stat Med 2013;32:11–24. http://dx.doi.org/10.1002/sim.5524.
234. Claxton K, Griffin S, Koffijberg H, McKenna C. Expected Health Benefits of Additional Evidence Principles, Methods and Applications. York: University of York, Centre for Health Economics; 2013.
235. Griffin S, Welton NJ, Claxton K. Exploring the research decision space: the expected value of information for sequential research designs. Med Decis Making 2010;30:155–62. http://dx.doi.org/10.1177/0272989X09344746.
236. Claxton K, Eggington S, Ginnelly L, Griffin S, McCabe C, Philips Z, et al. A Pilot Study of Value of Information Analysis to Support Research Recommendations for NICE. York: Centre for Health Economics, University of York; 2005.
237. Chilcott J, Brennan A, Booth A, Karnon J, Tappenden P. The role of modelling in prioritising and planning clinical trials. Health Technol Assess 2003;7(23).http://dx.doi.org/10.3310/hta7230.
238. Simpson S, Corney R, Fitzgerald P, Beecham J. A randomised controlled trial to evaluate the effectiveness and cost-effectiveness of counselling patients with chronic depression. Health Technol Assess 2000;4(36).
239. Claxton K, Ginnelly L, Sculpher M, Philips Z, Palmer S. A pilot study on the use of decision theory and value of information analysis as part of the NHS Health Technology Assessment programme. Health Technol Assess 2004;8(31).http://dx.doi.org/10.3310/hta8310.
240. Mohiuddin S, Fenwick E, Payne K. Use of value of information in UK Health Technology Assessments. Int J Technol Assess Health Care 2014;30:553–70. http://dx.doi.org/10.1017/S0266462314000701.
241. United States Senate. The Patient Protection and Affordable Care Act Detailed Summary. URL: www.dpc.senate.gov/healthreformbill/healthbill04.pdf (accessed 15 June 2016).
242. Neumann PJ, Weinstein MC. Legislating against use of cost-effectiveness information. N Engl J Med 2010;363:1495–97. http://dx.doi.org/10.1056/NEJMp1007168.
243. Dijkers M. Introducing GRADE: a systematic approach to rating evidence in systematic reviews and to guideline development. KT Update 2013;1.
244. Gartlehner G, Swinson Evans T, Dobrescu A, Sommer I, Thaler K, Lohr K. The Predictive Validity of GRADE. Oxford: Evidence Live Oxford; 2015.
245. Bower P, Knowles S, Coventry PA, Rowland N. Counselling for mental health and psychosocial problems in primary care. Cochrane Database Syst Rev 2011;9:CD001025. http://dx.doi.org/10.1002/14651858.cd001025.pub3.
246. Suri R, Wallis C, Bush A, Thompson S, Normand C. A comparative study of hypertonic saline, daily and alternate-day rhDNase in children with cystic fibrosis. Health Technol Assess 2003;6(34).
247. Wark P, McDonald VM. Nebulised hypertonic saline for cystic fibrosis. Cochrane Database Syst Rev 2009;2:CD001506. http://dx.doi.org/10.1002/14651858.cd001506.pub3.
248. Garry R, Fountain J, Brown J, Manca A, Mason S. EVALUATE hysterectomy trial: a multicentre randomised trial comparing abdominal, vaginal and laparoscopic methods of hysterectomy. Health Technol Assess 2004;8(26). http://dx.doi.org/10.3310/hta8260.
249. Nieboer TE, Johnson N, Lethaby A, Tavender E, Curr E, Garry R, et al. Surgical approach to hysterectomy for benign gynaecological disease. Cochrane Database Syst Rev 2009;3:CD003677. http://dx.doi.org/10.1002/14651858.cd003677.pub4.
250. Vickers A, Rees R, Zollman C, McCarney R, Smith C. Acupuncture of chronic headache disorders in primary care: randomised controlled trial and economic analysis. Health Technol Assess 2004;8(48). http://dx.doi.org/10.3310/hta8480.
251. Linde K, Allais G, Brinkhaus B, Manheimer E, Vickers A, White AR. Acupuncture for migraine prophylaxis. Cochrane Database Syst Rev 2009;1:CD001218. http://dx.doi.org/10.1002/14651858.cd001218.pub2.
252. Ozolins M, Eady E, Avery A, Cunliffe W, O’Neill C. Randomised controlled multiple treatment comparison to provide a cost-effectiveness rationale for the selection of antimicrobial therapy in acne. Health Technol Assess 2005;9(1). http://dx.doi.org/10.3310/hta9010.
253. Garner SE, Eady A, Bennett C, Newton JN, Thomas K, Popescu CM. Minocycline for acne vulgaris: efficacy and safety. Cochrane Database Syst Rev 2012;8:CD002086. http://dx.doi.org/10.1002/14651858.cd002086.pub2.
254. Cochrane T, Davey R, Matthes Edwards S. Randomised controlled trial of the cost-effectiveness of water-based therapy for lower limb osteoarthritis. Health Technol Assess 2005;9(31). http://dx.doi.org/10.3310/hta9310.
255. Bartels EM, Lund H, Hagen KB, Dagfinrud H, Christensen R, Danneskiold-Samsøe B. Aquatic exercise for the treatment of knee and hip osteoarthritis. Cochrane Database Syst Rev 2007;4:CD005523. http://dx.doi.org/10.1002/14651858.cd005523.pub2.
256. Thomas K, MacPherson H, Ratcliffe J, Thorpe L, Brazier J. Longer term clinical and economic benefits of offering acupuncture care to patients with chronic low back pain. Health Technol Assess 2005;9(32). http://dx.doi.org/10.3310/hta9320.
257. Furlan AD, van Tulder MW, Cherkin D, Tsukayama H, Lao L, Koes BW, et al. Acupuncture and dry-needling for low back pain. Cochrane Database Syst Rev 2005;1:CD001351. http://dx.doi.org/10.1002/14651858.cd001351.pub2.
258. Hobbs FDR, Fitzmaurice DA, Mant J, Murray E, Jowett S, Bryan S, et al. A randomised controlled trial and cost-effectiveness study of systematic screening (targeted and total population screening) versus routine practice for the detection of atrial fibrillation in people aged 65 and over. The SAFE study. Health Technol Assess 2005;9(40). http://dx.doi.org/10.3310/hta9400.
259. Moran PS, Flattery MJ, Teljeur C, Ryan M, Smith SM. Effectiveness of systematic screening for the detection of atrial fibrillation. Cochrane Database Syst Rev 2013;4:CD009586. http://dx.doi.org/10.1002/14651858.cd009586.pub2.
260. Dennis M, Lewis S, Cranswick G, Forbes J. FOOD: a multicentre randomised trial evaluating feeding policies in patients admitted to hospital with a recent stroke. Health Technol Assess 2006;10(2). http://dx.doi.org/10.3310/hta10020.
261. Gomes CAR Jr., Lustosa SAS, Matos D, Andriolo RB, Waisberg DR, Waisberg JW. Percutaneous endoscopic gastrostomy versus nasogastric tube feeding for adults with swallowing disturbances. Cochrane Database Syst Rev 2012;3:CD008096. http://dx.doi.org/10.1002/14651858.cd008096.pub3.
262. Kennedy T, Chalder T, McCrone P, Darnley S, Knapp M. Cognitive behavioural therapy in addition to antispasmodic therapy for irritable bowel syndrome in primary care: a randomised controlled trial. Health Technol Assess 2006;10(9).http://dx.doi.org/10.3310/hta10190.
263. Zijdenbos IL, de Wit NJ, van der Heijden GJ, Rubin G, Quartero AO. Psychological treatments for the management of irritable bowel syndrome. Cochrane Database Syst Rev 2009:1:CD006442. http://dx.doi.org/10.1002/14651858.cd006442.pub2.
264. Nixon J, Nelson A, Cranny G, Iglesias Urrutia C, Hawkins K. Pressure relieving support surfaces: a randomised evaluation. Health Technol Assess 2006;10(22). http://dx.doi.org/10.3310/hta10220.
265. McInnes E, Dumville JC, Jammali-Blasi A, Bell-Syer SEM. Support surfaces for treating pressure ulcers. Cochrane Database Syst Rev 2011;12:CD009490. http://dx.doi.org/10.1002/14651858.cd009490.
266. O’Dowd H, Gladwell P, Rogers C, Hollinghurst S, Gregory A. Cognitive behavioural therapy in chronic fatigue syndrome: a randomised controlled trial of an outpatient group programme. Health Technol Assess 2006;10(37).http://dx.doi.org/10.3310/hta10370.
267. Price JR, Mitchell E, Tidy E, Hunot V. Cognitive behaviour therapy for chronic fatigue syndrome in adults. Cochrane Database Syst Rev 2008;3:CD001027. http://dx.doi.org/10.1002/14651858.cd001027.pub2.
268. Hewison J, Nixon J, Fountain J, Cocks K, Jones C. Amniocentesis results: investigation of anxiety. The ARIA trial. Health Technol Assess 2006;10(50). http://dx.doi.org/10.3310/hta10500.
269. Mujezinovic F, Prosnik A, Alfirevic Z. Different communication strategies for disclosing results of diagnostic prenatal testing. Cochrane Database Syst Rev 2010;11:CD007750. http://dx.doi.org/10.1002/14651858.cd007750.pub2.
270. Jolly K, Taylor R, Lip G, Greenfield S, Raftery J. The Birmingham Rehabilitation Uptake Maximisation Study (BRUM). Home-based compared with hospital-based cardiac rehabilitation in a multi-ethnic population: cost-effectiveness and patient adherence. Health Technol Assess 2007;11(35). http://dx.doi.org/10.3310/hta11350.
271. Taylor RS, Dalal H, Jolly K, Moxham T, Zawada A. Home-based versus centre-based cardiac rehabilitation. Cochrane Database Syst Rev 2010;1:CD007130. http://dx.doi.org/10.1002/14651858.cd007130.pub2.
272. Fader M, Cottenden A, Getliffe K, Gage H, Clarke-O’Neill S. Absorbent products for urinary/faecal incontinence: a comparative evaluation of key product designs. Health Technol Assess 2008;12(29).
273. Fader M, Cottenden AM, Getliffe K. Absorbent products for moderate-heavy urinary and/or faecal incontinence in women and men. Cochrane Database Syst Rev 2008;4:CD007408. http://dx.doi.org/10.1002/14651858.cd007408.
274. Grant A, Wileman S, Ramsay C, Bojke L, Epstein D. The effectiveness and cost-effectiveness of minimal access surgery amongst people with gastro-oesophageal reflux disease – a UK collaborative study. The REFLUX Trial. Health Technol Assess 2008;12(31). http://dx.doi.org/10.3310/hta12310.
275. Wileman SM, McCann S, Grant AM, Krukowski ZH, Bruce J. Medical versus surgical management for gastro-oesophageal reflux disease (GORD) in adults. Cochrane Database Syst Rev 2010;3:CD003243. http://dx.doi.org/10.1002/14651858.cd003243.pub2.
276. Potter J, Mistri A, Brodie F, Chernova J, Wilson E. Controlling Hypertension and Hypotension Immediately Post-Stroke (CHHIPS) – a randomised controlled trial. Health Technol Assess 2009;13(9).http://dx.doi.org/10.3310/hta13090.
277. Bath PMW, Krishnan K. Interventions for deliberately altering blood pressure in acute stroke. Cochrane Database Syst Rev 2014;10;CD000039. http://dx.doi.org/10.1002/14651858. cd000039.pub3.
278. Morrell CJ, Warner R, Slade P, Dixon S, Walters S. Psychological interventions for postnatal depression: cluster randomised trial and economic evaluation. The PONDER trial. Health Technol Assess 2009;13(30).http://dx.doi.org/10.3310/hta13300.
279. Dennis CL, Dowswell T. Psychosocial and psychological interventions for preventing postpartum depression. Cochrane Database Syst Rev 2013;2:CD001134. http://dx.doi.org/10.1002/14651858.cd001134.pub3.
280. Gray AJ, Goodacre S, Newby DE, Masson MA, Sampson F. A multicentre randomised controlled trial of the use of continuous positive airway pressure and non-invasive positive pressure ventilation in the early treatment of patients presenting to the emergency department with severe acute cardiogenic pulmonary oedema: the 3CPO Trial. Health Technol Assess 2009;13(33). http://dx.doi.org/10.3310/hta13330.
281. Vital FMR, Ladeira MT, Atallah ÁN. Non-invasive positive pressure ventilation (CPAP or bilevel NPPV) for cardiogenic pulmonary oedema. Cochrane Database Syst Rev 2013;5:CD005351.http://dx.doi.org/10.1002/14651858.cd005351.pub3
282. Williamson I, Benge S, Barton S, Petrou S, Letley L. A double-blind randomised placebo controlled trial of topical intranasal corticosteroids in 4- to11-year-old children with persistent bilateral otitis media with effusion in primary care. Health Technol Assess 2009;13(37). http://dx.doi.org/10.3310/hta13370.
283. Simpson SA, Lewis R, van der Voort J, Butler CC. Oral or topical nasal steroids for hearing loss associated with otitis media with effusion in children. Cochrane Database Syst Rev 2011;5:CD001935. http://dx.doi.org/10.1002/14651858.cd001935.pub3.
284. Sullivan FM, Swan IR, Donnan PT, Morrison JM, Smith BH, McKinstry B, et al. A randomised controlled trial of the use of aciclovir and/or prednisolone for the early treatment of Bell’s palsy: the BELLS study. Health Technol Assess 2009;13(47). http://dx.doi.org/10.3310/hta13470.
285. Salinas RA, Alvarez G, Daly F, Ferreira J. Corticosteroids for Bell’s palsy (idiopathic facial paralysis). Cochrane Database Syst Rev 2010;3:CD001942. http://dx.doi.org/10.1002/14651858.cd001942.pub4.
286. Jeffcoate WJ, Price PE, Phillips CJ, Game FL, Mudge E. Randomised controlled trial of the use of three dressing preparations in the management of chronic ulceration of the foot in diabetes. Health Technol Assess 2009;13(54).http://dx.doi.org/10.3310/hta13540.
287. Dumville JC, O’Meara S, Deshpande S, Speak K. Hydrogel dressings for healing diabetic foot ulcers. Cochrane Database Syst Rev 2013;7:CD009101. http://dx.doi.org/10.1002/14651858.cd009101.pub3.
288. Michaels JA, Campbell WB, King BM, MacIntyre J, Palfreyman SJ. A prospective randomised controlled trial and economic modelling of antimicrobial silver dressings versus non-adherent control dressings for venous leg ulcers: the VULCAN trial. Health Technol Assess 2009;13(56).http://dx.doi.org/10.3310/hta13560.
289. O’Meara S, Al-Kurdi D, Ologun Y, Ovington LG, Martyn-St James M, Richardson R. Antibiotics and antiseptics for venous leg ulcers. Cochrane Database Syst Rev 2014;1:CD003557. http://dx.doi.org/10.1002/14651858.cd003557.pub5.
290. Lock C, Wilson J, Steen N, Eccles M, Mason H. North of England and Scotland Study of Tonsillectomy and Adeno-tonsillectomy in Children (NESSTAC): a pragmatic randomised controlled trial with a parallel non-randomised preference study. Health Technol Assess 2010;14(13).
291. Burton MJ, Glasziou PP, Chong LY, Venekamp RP. Tonsillectomy or adenotonsillectomy versus non-surgical treatment for chronic/recurrent acute tonsillitis. Cochrane Database Syst Rev 2014;11:CD001802. http://dx.doi.org/10.1002/14651858.cd001802.pub3
292. Peek G, Elbourne D, Mugford M, Tiruvoipati R, Wilson A. Randomised controlled trial and parallel economic evaluation of conventional ventilatory support versus extracorporeal membrane oxygenation for severe adult respiratory failure (CESAR). Health Technol Assess 2010;14(35).http://dx.doi.org/10.3310/hta14350.
293. Tramm R, Ilic D, Davies AR, Pellegrino VA, Romero L, Hodgson C. Extracorporeal membrane oxygenation for critically ill adults. Cochrane Database Syst Rev 2015;1:CD010381. http://dx.doi.org/10.1002/14651858.cd010381.pub2.
294. Lamb S, Lall R, Hansen Z, Castelnuovo E, Withers E. A multicentred randomised controlled trial of a primary-care based cognitive behavioural programme for low back pain. The Back Skills Training (BeST) trial. Health Technol Assess 2010;14(41). http://dx.doi.org/10.3310/hta14410.
295. Williams AC, Eccleston C, Morley S. Psychological therapies for the management of chronic pain (excluding headache) in adults. Cochrane Database Syst Rev 2012;11:CD007407. http://dx.doi.org/10.1002/14651858.cd007407.pub3.
296. Sharp D, Chew-Graham C, Tylee A, Lewis G, Howard L. A pragmatic randomised controlled trial to compare antidepressants with a community-based psychosocial intervention for the treatment of women with postnatal depression: the RESPOND trial. Health Technol Assess 2010;14(43). http://dx.doi.org/10.3310/hta14430.
297. Molyneaux E, Howard LM, McGeown HR, Karia AM, Trevillion K. Antidepressant treatment for postnatal depression. Cochrane Database Syst Rev 2014;9:CD002018. http://dx.doi.org/10.1002/14651858.cd002018.pub2.
298. Cockayne E, Curran M, Denby G, Hashmi F, Hewitt C. EVerT: cryotherapy versus salicylic acid for the treatment of verrucae – a randomised controlled trial. Health Technol Assess 2011;15(32).http://dx.doi.org/10.3310/hta15320.
299. Kwok CS, Gibbs S, Bennett C, Holland R, Abbott R. Topical treatments for cutaneous warts. Cochrane Database Syst Rev 2012;9:CD001781. http://dx.doi.org/10.1002/14651858.cd001781.pub3.
300. Schumi J, Wittes JT. Through the looking glass: understanding non-inferiority. Trials 2011;12:106. http://dx.doi.org/10.1186/1745-6215-12-106.
301. Aberegg SK, Arkes H, Terry PB. Failure to adopt beneficial therapies caused by bias in medical evidence evaluation. Med Decis Making 2006;26:575–82. http://dx.doi.org/10.1177/0272989X06295362.
302. Elshaug AG, Hiller JE, Moss JR. Exploring policy-makers’ perspectives on disinvestment from ineffective healthcare practices. Int J Technol Assess Health Care 2008;24:1–9. http://dx.doi.org/10.1017/S0266462307080014.
303. García-Armesto S, Campillo-Artero C, Bernal Delgado E. Disinvestment in the age of cost-cutting sound and fury. Tools for the Spanish National Health System. Health Policy 2013;110:180–85. http://dx.doi.org/10.1016/j.healthpol.2013.01.007.
304. Haines T, O’Brien L, McDermott F, Markham D, Mitchell D, Watterson D, et al. A novel research design can aid disinvestment from existing health technologies with uncertain effectiveness, cost-effectiveness, and/or safety. J Clin Epidemiol 2014;67:144–51. http://dx.doi.org/10.1016/j.jclinepi.2013.08.014.
305. Henshall C, Schuller T, Mardhani-Bayne L. Using health technology assessment to support optimal use of technologies in current practice: The challenge of ‘disinvestment’. Int J Technol Assess Health Care 2012;28:203–10. http://dx.doi.org/10.1017/S0266462312000372.
306. Hodgetts K, Elshaug AG, Hiller JE. What counts and how to count it: physicians’ constructions of evidence in a disinvestment context. Soc Sci Med 2012;75:2191–9. http://dx.doi.org/10.1016/j.socscimed.2012.08.016.
307. Hodgetts K, Hiller JE, Street JM, Carter D, Braunack-Mayer AJ, Watt AM, et al. Disinvestment policy and the public funding of assisted reproductive technologies: outcomes of deliberative engagements with three key stakeholder groups. BMC Health Serv Res 2014;14:204. http://dx.doi.org/10.1186/1472-6963-14-204.
308. Ibargoyen-Roteta N, Gutiérrez-Ibarluzea I, Asua J. Guiding the process of health technology disinvestment. Health Policy 2010;98:218–26. http://dx.doi.org/10.1016/j.healthpol.2010.06.018.
309. Karnon J, Carlton J, Czoski-Murray C, Smith K. Informing disinvestment through costeffectiveness modelling: is lack of data a surmountable barrier? Appl Health Econ Health Policy 2009;7:1–9. http://dx.doi.org/10.1007/BF03256137.
310. Nuti S, Vainieri M, Bonini A. Disinvestment for re-allocation: a process to identify priorities in healthcare. Health Policy 2010;95:137–43. http://dx.doi.org/10.1016/j.healthpol.2009.11.011.
311. Rooshenas L, Owen-Smith A, Hollingworth W, Badrinath P, Beynon C, Donovan JL. ‘I won’t call it rationing…’: an ethnographic study of healthcare disinvestment in theory and practice. Soc Sci Med 2015;128:273–81. http://dx.doi.org/10.1016/j.socscimed.2015.01.020.
312. Watt AM, Hiller JE, Braunack-Mayer AJ, Moss JR, Buchan H, Wale J, et al. The ASTUTE health study protocol: Deliberative stakeholder engagements to inform implementation approaches to healthcare disinvestment. Implement Sci 2012;7:101. http://dx.doi.org/10.1186/1748-5908-7-101.
313. Watt AM, Willis CD, Hodgetts K, Elshaug AG, Hiller JE. Engaging clinicians in evidence-based disinvestment: role and perceptions of evidence. Int J Technol Assess Health Care 2012;28:211–19. http://dx.doi.org/10.1017/S0266462312000402.
314. Wilson MG, Ellen ME, Lavis JN, Grimshaw JM, Moat KA, Shemer J, et al. Processes, contexts, and rationale for disinvestment: a protocol for a critical interpretive synthesis. Syst Rev 2014;3:143. http://dx.doi.org/10.1186/2046-4053-3-143.
315. de Souza ALC, de Assis Acurcio F, Júnior AAG, do Nascimento RCRM, Godman B, Diniz LM. Insulin glargine in a Brazilian state: should the government disinvest? An assessment based on a systematic review. Appl Health Econ Health Policy 2014;12:19–32. http://dx.doi.org/10.1007/s40258-013-0073-6.
316. Cornelissen E, Mitton C, Davidson A, Reid C, Hole R, Visockas A-M, et al. Determining and broadening the definition of impact from implementing a rational priority setting approach in a healthcare organization. Soc Sci Med 2014;114:1–9. http://dx.doi.org/10.1016/j.socscimed.2014.05.027.
317. Edwards RT, Charles JM, Thomas S, Bishop J, Cohen D, Groves S, et al. A national Programme Budgeting and Marginal Analysis (PBMA) of health improvement spending across Wales: disinvestment and reinvestment across the life course. BMC Public Health 2014;14:837. http://dx.doi.org/10.1186/1471-2458-14-837.
318. Haas M, Hall J, Viney R, Gallego G. Breaking up is hard to do: why disinvestment in medical technology is harder than investment. Aust Health Rev 2012;36:148–52. http://dx.doi.org/10.1071/AH11032.
319. Mitton C, Dionne F, Damji R, Campbell D, Bryan S. Difficult decisions in times of constraint: criteria based resource allocation in the Vancouver Coastal Health Authority. BMC Health Serv Res 2011;11:169. http://dx.doi.org/10.1186/1472-6963-11-169.
320. Mortimer D. Reorienting programme budgeting and marginal analysis (PBMA) towards disinvestment. BMC Health Serv Res 2010;10:288. http://dx.doi.org/10.1186/1472-6963-10-288.
321. Bohmer P, Pain C, Watt A, Abernethy P, Sceats J. Maximising health gain within available resources in the New Zealand public health system. Health Policy 2001;55:37–50. http://dx.doi.org/10.1016/S0168-8510(00)00107-X
322. Daniels T, Williams I, Robinson S, Spence K. Tackling disinvestment in health care services: the views of resource allocators in the English NHS. J Health Organ Manag 2013;27:762–80. http://dx.doi.org/10.1108/JHOM-11-2012-0225.
323. Robinson S, Glasby J, Allen K. ‘It ain’t what you do it’s the way that you do it’: lessons for health care from decommissioning of older people’s services. Health Soc Care Community 2013;21:614–22. http://dx.doi.org/10.1111/hsc.12046.
324. Robinson S, Williams I, Dickinson H, Freeman T, Rumbold B. Priority-setting and rationing in healthcare: evidence from the English experience. Soc Sci Med 2012;75:2386–93. http://dx.doi.org/10.1016/j.socscimed.2012.09.014.
325. Williams IP, Bryan S. Cost-effectiveness analysis and formulary decision making in England: findings from research. Soc Sci Med 2007;65:2116–29. http://dx.doi.org/10.1016/j.socscimed.2007.06.009.
326. Eddama O, Coast J. Use of economic evaluation in local health care decision-making in England: a qualitative investigation. Health Policy 2009;89:261–70. http://dx.doi.org/10.1016/j.healthpol.2008.06.004.
327. Chamberlain CA, Martin RM, Busby J, Gilbert R, Cahill DJ, Hollingworth W. Trends in procedures for infertility and caesarean sections: was NICE disinvestment guidance implemented? NICE recommendation reminders. BMC Public Health 2013;13:112. http://dx.doi.org/10.1186/1471-2458-13-112.
328. Garner S, Docherty M, Somner J, Sharma T, Choudhury M, Clarke M, et al. Reducing ineffective practice: challenges in identifying low-value health care using Cochrane systematic reviews. J Health Serv Res Policy 2013;18:6–12. http://dx.doi.org/10.1258/jhsrp.2012.012044.
329. Garner S, Littlejohns P. Disinvestment from low value clinical interventions: NICEly done? BMJ 2011;343:d4519. http://dx.doi.org/10.1136/bmj.d4519.
330. Pearson S, Littlejohns P. Reallocating resources: how should the National Institute for Health and Clinical Excellence guide disinvestment efforts in the National Health Service? J Health Serv Res Policy 2007;12:160–5. http://dx.doi.org/10.1258/135581907781542987.
331. Walker S, Palmer S, Sculpher M. The role of NICE technology appraisal in NHS rationing. Br Med Bull 2007;81:51–64. http://dx.doi.org/10.1093/bmb/ldm007.
332. Hopper I, Samuel R, Hayward C, Tonkin A, Krum H. Can Medications be Safely Withdrawn in Patients With Stable Chronic Heart Failure? Systematic Review and Meta-analysis. J Card Fail 2014;20:522–32. http://dx.doi.org/10.1016/j.cardfail.2014.04.013.
333. Leggett L, Noseworthy TW, Zarrabi M, Lorenzetti D, Sutherland LR, Clement FM. Health technology reassessment of non-drug technologies: current practices. Int J Technol Assess Health Care 2012;28:220–7. http://dx.doi.org/10.1017/S0266462312000438.
334. Polisena J, Clifford T, Elshaug AG, Mitton C, Russell E, Skidmore B. Case studies that illustrate disinvestment and resource allocation decision-making processes in health care: a systematic review. Int J Technol Assess Health Care 2013;29:174–84. http://dx.doi.org/10.1017/S0266462313000068.
335. Nielsen CP, Funch TM, Kristensen FB. Health technology assessment: research trends and future priorities in Europe. J Health Serv Res Policy 2011;16(Suppl. 2):6–15. http://dx.doi.org/10.1258/jhsrp.2011.011050.
336. Watt AM, Elshaug AG, Willis CD, Hiller JE, ASTUTE Health study group. Assisted reproductive technologies: a systematic review of safety and effectiveness to inform disinvestment policy. Health Policy 2011;102:200–13. http://dx.doi.org/10.1016/j.healthpol.2011.07.007.
337. Charlesworth G, Shepstone L, Wilson E, Thalanany M, Mugford M, Poland F. Does befriending by trained lay workers improve psychological well-being and quality of life for carers of people with dementia, and at what cost? A randomised controlled trial. Health Technology Assessment 2008:12(4).http://dx.doi.org/10.3310/hta12040.
338. Bradbury AW, Adam DJ, Bell J, Forbes JF, Fowkes FGR. Multicentre randomised controlled trial of the clinical and cost-effectiveness of a bypass-surgery-first versus a balloon-angioplasty-first revascularisation strategy for severe limb ischaemia due to infrainguinal disease. The Bypass versus Angioplasty in Severe Ischaemia of the Leg (BASIL) trial. Health Technol Assess 2010;14(14).http://dx.doi.org/10.3310/hta14140.
339. Little P, Turner S, Rumsby K, Warner G, Moore M. Dipsticks and diagnostic algorithms in urinary tract infection: development and validation, randomised trial, economic analysis, observational cohort, and qualitative study. Health Technol Assess 2009;13(19).http://dx.doi.org/10.3310/hta13190.
340. Kerry S, Dundas D, Hilton S, Rink E, Patel S. Routine referral for radiography of patients presenting with low back pain: is patients’ outcome influenced by GPs’ referral for plain radiography? Health Technol Assess 2000;4(20).
341. Kendrick D, Fielding K, Bentley E, Miller P, Kerslake R. The role of radiography in primary care patients with low back pain of at least 6 weeks’ duration: a randomised (unblinded) controlled trial. Health Technol Assess 2001;5(3).1 http://dx.doi.org/10.3310/hta5300.
342. Lomas J. Improving Research Dissemination and Uptake in the Health Sector: Beyond the Sound of One Hand Clapping. Ottawa, ON: Canadian Health Services Research Foundation; 1997.
343. Weiss CH. Have we learned anything new about the use of evaluation? Am J Eval 1998;19:21–33. http://dx.doi.org/10.1016/S1098-2140(99)80178-7.
344. Pittman P, Almeida C. Cross-sector learning among researchers and policy-makers: the search for new strategies to enable use of research results. Cadernos Saude Publica 2006;22(Suppl.):97–108. http://dx.doi.org/10.1590/S0102-311X2006001300019.
345. Hinrichs S, Grant J. A new resource for identifying and assessing the impacts of research. BMC Med 2015;13:148. http://dx.doi.org/10.1186/s12916-015-0364-x
346. Barnhoorn F, McCarthy M, Deville W, Alexanderson K, Voss M, Conceicao C. PHIRE (Public Health Innovation and Research in Europe): methods, structures and evaluation. Eu J Public Health 2013;23(Suppl. 2):6–11. http://dx.doi.org/10.1093/eurpub/ckt148.
347. Battles JB, Farr SL, Weinberg DA. From research to nationwide implementation: the impact of AHRQ’s HAI prevention program. Med Care 2014;52(2 Suppl. 1):91–6. http://dx.doi.org/10.1097/MLR.0000000000000037.
348. Bernstein A, Hicks V, Borbey P, Campbell T.A Framework to Measure the Impact of Investments in Health Research. OECD Blue Sky 11 Forum Conference Proceedings, Ottawa, ON, 25 September 2006.
349. Bumgarner R, Hattaway D, Lamb G, McGann J, Wise H. Center for Global Development: Evaluation of Impact. Washington, DC: Arabella Philanthropic Investment Advisors; 2006.
350. Glennerster R, Kremer M, Williams H. Creating markets for vaccines. Innovations 2006;67. http://dx.doi.org/10.1162/itgg.2006.1.1.67.
351. Bunn F. The Cochrane Injuries Group celebrates the publication of its 100th review: time to reflect on impact. Inj Prev 2010;16:208–9. http://dx.doi.org/10.1136/ip.2010.027136.
352. Buykx P, Humphreys J, Wakerman J, Perkins D, Lyle D, McGrail M, et al. Making evidence count: a framework to monitor the impact of health services research. Aust J Rural Health 2012;20:51–8. http://dx.doi.org/10.1111/j.1440-1584.2012.01256.x
353. Lavis JN, Robertson D, Woodside JM, McLeod CB, Abelson J. How can research organizations more effectively transfer research knowledge to decision makers? Milbank Q 2003;81:221–48. http://dx.doi.org/10.1111/1468-0009.t01-1-00052.
354. Lavis J, Ross S, McLeod C, Gildiner A. Measuring the impact of health research. J Health Serv Res Policy 2003;8:165–70. http://dx.doi.org/10.1258/135581903322029520.
355. Canadian Institutes of Health Research (CIHR). Developing a CIHR Framework to Measure The Impact of Health Research. Ottawa, ON: CIHR; 2005.
356. Carden F. Knowledge to Policy: Making the Most of Development Research. Ottawa, ON: International Centre for Development Research; 2009.
357. Currie M, King G, Rosenbaum P, Law M, Kertoy M, Specht J. A model of impacts of research partnerships in health and social services. Eval Program Plann 2005;28:400–12. http://dx.doi.org/10.1016/j.evalprogplan.2005.07.004.
358. de Goede J, van Bon-Martens MJ, Putters K, van Oers HA. Looking for interaction: quantitative measurement of research utilization by Dutch local health officials. Health Res Policy Syst 2012;10. http://dx.doi.org/10.1186/1478-4505-10-9.
359. Druce N, Gopalan A, Moree M, Nathanson N, Plotkin S, Skolnik R. Evaluation of the International AIDS Vaccine 2003–2007. New York, NY: IAVI; 2009.
360. Bozeman B. Public Value Mapping of Science Outcomes: Theory and Method. URL: http://cspo.org/legacy/library/110215F3AQ_lib_BozemanPVMKnowle.pdf (accessed 14 July 2016).
361. Ensor T, Clapham S, Prasai DP. What drives health policy formulation: insights from the Nepal maternity incentive scheme? Health Policy 2009;90:247–53. http://dx.doi.org/10.1016/j.healthpol.2008.06.009.
362. Garfinkel MS, Sarewitz D, Porter AL. A societal outcomes map for health research and policy. Am J Public Health 2006;96:441–6. http://dx.doi.org/10.2105/AJPH.2005.063495.
363. Gibson B, Ehrlich D, Graff J, Dubois R, Farr M, Chernew M, et al. Real-world impact of comparative effectiveness research findings on clinical practice. Am J Manag Care 2014;20:e208–20.
364. Godin B, Doré C. Measuring the Impacts of Science; Beyond the Economic Dimension. Helsinki: Helsinki Institute for Science and Technology Studies; 2005.
365. Grazier KL, Trochim WM, Dilts DM, Kirk R. Estimating return on investment in translational research: methods and protocols. Eval Health Prof 2013;36:478–91. http://dx.doi.org/10.1177/0163278713499587.
366. Morgan Jones M, Castle-Clarke S, Manville C, Gunashekar S, Grant J. Assessing Research Impact. An International Review of the Excellence in Innovation for Australia Trial. Santa Monica, CA: RAND Corporation; 2013.
367. Largent MA, Lane JI. STAR METRICS and the science of science policy. Rev Policy Res 2012;29:431–8. http://dx.doi.org/10.1111/j.1541-1338.2012.00567.x
368. El Turabi A, Hallsworth M, Ling T, Grant J. A novel performance monitoring framework for health research systems: experiences of the National Institute for Health Research in England. Health Res Policy Syst 2011;9:13. http://dx.doi.org/10.1186/1478-4505-9-13.
369. Ottoson JM, Green LW, Beery WL, Senter SK, Cahill CL, Pearson DC, et al. Policy-contribution assessment and field-building analysis of the Robert Wood Johnson Foundation’s active living research program. Am J Prev Med 2009;36(Suppl. 2):34–43. http://dx.doi.org/10.1016/j.amepre.2008.10.010.
370. Hanney SR, Home PD, Frame I, Grant J, Green P, Buxton MJ. Identifying the impact of diabetes research. Diabet Med 2006;23:176–84. http://dx.doi.org/10.1111/j.1464-5491.2005.01753.x
371. Home PD. Impact of the UKPDS – an overview. Diabet Med 2008;25:2–8. http://dx.doi.org/10.1111/j.1464-5491.2008.02501.x
372. JISC. Embedding Impact Analysis in Research. Bristol: National Coordinating Centre for Public Engagement, University of Bristol, University of the West of England; 2013.
373. Morton S. Exploring and Assessing Research Impact. PhD thesis. Edinburgh: University of Edinburgh; 2012.
374. Kagan JM, Kane M, Quinlan KM, Rosas S, Trochim WM. Developing a conceptual framework for an evaluation system for the NIAID HIV/AIDS clinical trials networks. Health Res Policy Syst 2009;7. http://dx.doi.org/10.1186/1478-4505-7-12.
375. King G, Servais M, Kertoy M, Specht J, Currie M, Rosenbaum P, et al. A measure of community members’ perceptions of the impacts of research partnerships in health and social services. Eval Program Plann 2009;32:289–99. http://dx.doi.org/10.1016/j.evalprogplan.2009.02.002.
376. Caplan N, Morrison A, Stamburgh RJ. The Use of Social Science Knowledge in Policy Decisions at the National Level: A Report to Respondents. Ann Arbor, MI: Center for Research on Utilization of Scientific Knowledge, University of Michigan; 1975.
377. Kryl D, Allen L, Dolby K, Sherbon B, Viney I. Tracking the impact of research on policy and practice: investigating the feasibility of using citations in clinical guidelines for research evaluation. BMJ Open 2012;2. http://dx.doi.org/10.1136/bmjopen-2012-000897.
378. Latour B. Reassembling the Social – An Introduction to Actor-Network-Theory. Oxford: Oxford University Press; 2005.
379. Martin B, Tang P. The Benefits from Publicly Funded Research. Brighton: SPRU, University of Sussex; 2007.
380. McCarthy M. Public health research support through the European structural funds in central and eastern Europe and the Mediterranean. Health Res Policy Syst 2012;10:12. http://dx.doi.org/10.1186/1478-4505-10-12.
381. McLean RK, Graham I, Bosompra K, Choudhry Y, Coen S, MacLeod M, et al. Understanding the performance and impact of public knowledge translation funding interventions: protocol for an evaluation of Canadian Institutes of Health Research knowledge translation funding programs. Implement Sci 2012;7:57. http://dx.doi.org/10.1186/1748-5908-7-57.
382. Moreira T. The Transformation of Contemporary Health Care: The Market, the Laboratory, and the Forum. London: Routledge; 2013.
383. Morlacchi P, Nelson RR. How medical practice evolves: learning to treat failing hearts with an implantable device. Res Policy 2011;40:511–25. http://dx.doi.org/10.1016/j.respol.2011.01.001.
384. Niederkrotenthaler T, Dorner T, Maier M. Development of a practical tool to measure the impact of publications on the society based on focus group discussions with scientists. BMC Public Health 2011;11:588. http://dx.doi.org/10.1186/1471-2458-11-588.
385. Oliver TR, Singer RF. Health services research as a source of legislative analysis and input: the role of the California health benefits review program. Health Serv Res 2006;41:1124–58. http://dx.doi.org/10.1111/j.1475-6773.2006.00523.x
386. Wilson JQ. Political Organizations. New York, NY: Basic Books; 1973.
387. Wilson JQ. The Politics of Regulation. In Wilson JQ, editor. The Politics of Regulation. New York, NY: Basic Books; 1990. pp. 35–94.
388. Arnold RD. The Logic of Congressional Action. New Haven, CT: Yale University Press; 1990.
389. Ovseiko PV, Oancea A, Buchan AM. Assessing research impact in academic clinical medicine: a study using Research Excellence Framework pilot impact indicators. BMC Health Serv Res 2012;12:478. http://dx.doi.org/10.1186/1472-6963-12-478.
390. Percy-Smith J, Speller V, Nutley S. Evidence Informed Policy and Practice: A Review of Approaches used in Health Improvement in Scotland. Edinburgh: NHS Health Scotland; 2006.
391. Walter I, Nutley S, Percy-Smith J, McNeish D, Frost S. Improving the Use of Research in Social Care Practice. London: Social Care Institute for Excellence; 2004.
392. Ritter A, Lancaster K. Measuring research influence on drug policy: a case example of two epidemiological monitoring systems. Int J Drug Policy 2013;24:30–7. http://dx.doi.org/10.1016/j.drugpo.2012.02.005.
393. Hass PM. Introduction: epistemic communities and international policy coordination. International Organization 1992;46:1–35. http://dx.doi.org/10.1017/S0020818300001442.
394. Lomas J. Using ‘linkage and exchange’ to move research into policy at a Canadian foundation. Health Aff 2000;19:236–40. http://dx.doi.org/10.1377/hlthaff.19.3.236.
395. Rosas SR, Schouten JT, Cope MT, Kagan JM. Modeling the dissemination and uptake of clinical trials results. Res Eval 2013;22:179–86. http://dx.doi.org/10.1093/reseval/rvt005.
396. Runyan C, Garrettson M, Yee SL. Development of a set of indicators to evaluate injury control research centers. Eval Rev 2014:38:133–59. http://dx.doi.org/10.1177/0193841X14529287.
397. W.K. Kellogg Foundation. Logic Model Development Guide. Battle Creek, MI: The Foundation; 2004. URL: www.wkkf.org/resource-directory/resource/2006/02/wk-kellogg-foundation-logicmodel- development-guide (accessed 15 July 2016).
398. Sarli CC, Holmes KL. Update to ‘Beyond citation analysis: a model for assessment of research impact’. J Med Libr Assoc 2012;100:82. http://dx.doi.org/10.3163/1536-5050.100.2.002.
399. Scientific Management Review Board. Report on Approaches to Assess the Value of Biomedical Research Supported by the NIH. Bethesda, MD: National Institutes of Health; 2014.
400. Spoth RL, Schainker LM, Hiller-Sturmhöefel S. Translating family-focused prevention science into public health impact: illustrations from partnership-based research. Alcohol Res Health 2011;34:188.
401. Sridharan S, Maplazi J, Vijendran M, Richardson E, Nichols J, Parekh H. Final Report: Evaluation of Teasdale-Corti Initiative. Toronto, ON: The Evaluation Centre for Complex Health Interventions (TECCHI); 2013.
402. Sullivan R, Lewison G, Purushotham AD. An analysis of research activity in major UK cancer centres. Eur J Cancer 2011;47:536–44. http://dx.doi.org/10.1016/j.ejca.2010.10.015.
403. Sutherland WJ, Goulson D, Potts SG, Dicks LV. Quantifying the impact and relevance of scientific research. PLOS ONE 2011;6. http://dx.doi.org/10.1371/journal.pone.0027537.
404. Theobald S, Taegtmeyer M, Squire SB, Crichton J, Simwaka BN, Thomson R, et al. Towards building equitable health systems in sub-Saharan Africa: lessons from case studies on operational research. Health Res Policy Syst 2009;7. http://dx.doi.org/10.1186/1478-4505-7-26.
405. Tremblay G, Zohar S, Bravo J, Potsepp P, Barker M. The Canada Foundation for Innovation’s outcome measurement study: a pioneering approach to research evaluation. Res Eval 2010;19:333–45. http://dx.doi.org/10.3152/095820210X12809191250960.
406. Trochim WM, Marcus SE, Masse LC, Moser RP, Weld PC. The evaluation of large research initiatives – a participatory integrative mixed-methods approach. Am J Eval 2008;29:8–28. http://dx.doi.org/10.1177/1098214007309280.
407. Sherwin CW, Isenson RS. Project Hindsight: a Defense Department study of the utility of research. Science 1967;156:1571–7. http://dx.doi.org/10.1126/science.156.3782.1571.
408. IIT Research Institute. Technology in Retrospect and Critical Events in Science. Chicago, IL: IIT Research Institute; 1968.
409. Wilson PM, Petticrew M, Calnan MW, Nazareth I. Does dissemination extend beyond publication: a survey of a cross section of public funded research in the UK. Implement Sci 2010;5:61. http://dx.doi.org/10.1186/1748-5908-5-61.
410. Xia AD. Comparative effectiveness reviews and their potential impact on formulary access in the United States. Value Health 2011;14:A30. http://dx.doi.org/10.1016/j.jval.2011.02.172.
411. Australian Research Council. ERA 2012 Submission Guidelines, Excellence in Research for Australia. Canberra, ACT: Australian Research Council; 2011.
412. Morgan B. Research impact: income for outcome. Nature 2014;511:S72–75.
413. Manville C, Guthrie S, Henham M-L, Garrod B, Sousa S, Kirtley A, et al. Assessing Impact Submissions for REF 2014: An Evaluation. Cambridge: RAND Europe; 2015. URL: www.hefce.ac.uk/media/HEFCE,2014/Content/Pubs/Independentresearch/2015/REF,impact,submissions/REF_assessing_impact_submissions.pdf (accessed 11 August 2015).
414. Greenhalgh T, Fahy N. Research impact in the community based health sciences: an analysis of 162 case studies from the 2014 UK Research Excellence Framework. BMC Med 2015;13:232 http://dx.doi.org/10.1186/s12916-015-0467-4