The development of SAGE: A tool to evaluate how policymakers’ engage with and use research in health policymaking

Research Evaluation

Volumn 25, Issue 3, 2016, Pages 315-328

  Makkar, Steve R ^a   Brennan, Sue ^b   Turner, Tari ^b   Williamson, Anna ^a   Redman, Sally ^a   Green, Sally ^b  

^a SAX INSTITUTE   (Australia)

^b MONASH UNIVERSITY   (Australia)

Author keywords

Evidence based policy; Evidence informed policy; Health policy; Knowledge translation; Measurement; Public health

Indexed keywords

EID: 85011386017     PISSN: 09582029     EISSN: 14715449     Source Type: Journal    
DOI: 10.1093/reseval/rvv044     Document Type: Article

References (75)

1. Amara, N., Ouimet, M. and Landry, R. (2004)’New Evidence on Instrumental, Conceptual, and Symbolic Utilization of University Research in Government Agencies’, Science Communication, 26/1: 75–106.
2. Andre, F. E. et al. (2008)’Vaccination Greatly Reduces Disease, Disability, Death and Inequity Worldwide’, Bulletin of the World Health Organization, 86/2: 140–46.
3. Armstrong, R. et al. (2014) ‘Understanding Evidence: A Statewide Survey to Explore Evidence-Informed Public Health Decision-Making in a Local Government Setting’, Implementation Science: IS, 9: 188.
4. Beyer, J. M. (1997) ‘Research Utilisation: Bridging a Gap between Communities’, Journal of Management Inquiry, 20: 385.
5. Beyer, J. M. and Trice, H. M. (1982)’The Utilization Process: A Conceptual Framework and Synthesis of Empirical Findings’, Administrative Science Quarterly, 27/4: 591–622.
6. Black, N. (2001)’Evidence Based Policy: Proceed with Care’, BMJ, 323/7307: 275–9.
7. Boaz, A., Fitzpatrick, S. and Shaw, B. (2009) ‘Assessing the Impact of Research on Policy: A Literature Review’, Science and Public Policy, 36/4: 255–70.
8. Bowen, S. et al. (2009) ‘More than “Using Research”: The Real Challenges in Promoting Evidence-Informed Decision-Making’, Health Policy, 4/3: 87–102.
9. Brownson, R. C., Fielding, J. E. and Maylan, C. M. (2009a) ‘Evidence-Based Public Health: A Fundamental Concept for Public Health Practice’, Annual Review of Public Health, 30: 175–201.
10. Brownson, R. C., Chriqui, J. F. and Stamatakis, K. A. (2009b) ‘Understanding Evidence-Based Public Health Policy’, American Journal of Public Health, 99/9: 1576–83.
11. Buchan, H. (2004) ‘Gaps between Best Evidence and Practice: Causes for Concern’, MJA, 180: S48–9.
12. Campbell, D. M. et al. (2009) ‘Increasing the use of Evidence in Health Policy: Practice and Views of Policy makers and Researchers’, Australia and New Zealand Health Policy, 6: 21.
13. Caplan, N., Morrison, A. and Stambaugh, R. (1975) The Use of Social Science Knowledge in Policy Decisions at the National Level. Ann Arbor, MI: University of Michigan, Institute for Social Research.
14. Chagnon, F. et al. (2010) ‘Comparison of Determinants of Research Knowledge Utilization by Practitioners and Administrators in the Field of Child and Family Social Services’, Implementation Science, 5: 41.
15. Cook, T. D. and Campbell, D. T. (1979) Quasi-Experimentation: Design and Analysis Issues for Field Settings. Chicago, IL: Rand McNally College.
16. de Goede, J. et al. (2010) ‘Knowledge in Process? Exploring Barriers between Epidemiological Research and Local Health Policy Development’, Health Research Policy and Systems, 8: 26.
17. de Goede, J. et al. (2012) ‘Looking for Interaction: Quantitative Measurement of Research Utilization by Dutch Local Health Officials’, Health Research Policy and Systems, 10: 9.
18. Dobbins, M. et al. (2001) ‘Factors of the Innovation, Organization, Environment, and Individual that Predict the Influence Five Systematic Reviews had on Public Health Decisions’, International Journal of Technology Assessment in Health Care, 17/4: 467–78.
19. Dobbins, M. et al. (2002) ‘A Framework for the Dissemination and Utilization of Research for Health-Care Policy and Practice’, The Online Journal of Knowledge Synthesis for Nursing, 9: 149–60.
20. Dobbins, M. et al. (2007) ‘Information Transfer: What do Decision Makers Want and Need from Researchers?’, Implementation Science, 2: 20.
21. Dobrow, M. J., Goel, V. and Upshur, R. E. (2004) ‘Evidence-based Health Policy: Context and Utilisation’, Social Science & Medicine, 58/1: 207–17.
22. Ellen, M. E. et al. (2011) ‘Determining Research Knowledge Infrastructure for Healthcare Systems: A Qualitative Study’, Implementation Science, 6: 60.
23. Ellen, M. E. et al. (2013) ‘What Supports do Health System Organizations have in Place to Facilitate Evidence-informed Decision-Making? A Qualitative Study’, Implementation Science, 8: 84.
24. Elliott, H. and Popay, J. (2000) ‘How are Policy Makers Using Evidence? Models of Research Utilisation and Local NHS Policy Making’, Journal of Epidemiology and Community Health, 54: 461–8.
25. Evans, B. A. et al. (2013) ‘How Hard can it be to Include Research Evidence and Evaluation in Local Health Policy Implementation? Results from a Mixed Methods Study’, Implementation Science, 8: 17.
26. Fielding, J. E. and Briss, P. A. (2006) ‘Promoting Evidence-Based Public Health Policy: Can we have Better Evidence and More Action?’, Health Affairs, 4: 969–78.
27. Furlan, A. D. et al. (2009) ‘2009 Updated Method Guidelines for Systematic Reviews in the Cochrane Back Review Group’, Spine, 34/18: 1929–41.
28. Hanna, J. N., Hills, S. L. and Humphreys, J. L. (2004) ‘Impact of Hepatitis A Vaccination of Indigenous Children on Notifications of Hepatitis A in North Queensland’, Medical Journal of Australia, 181/9: 482–5.
29. Hanney, S. R. et al. (2003) ‘The Utilisation of Health Research in Policy-Making: Concepts, Examples and Methods of Assessment’, Health Research Policy and Systems, 1: 2.
30. Haynes, A. et al. (2014) ‘Developing Definitions for a Knowledge Exchange Intervention in Health Policy and Program Agencies: Reflections on Process and Value’, International Journal of Social Research Methodology, 18/2: 145–59.
31. Helmsey-Brown, J. (2004) ‘Facilitating Research Utilisation: A Cross-Sector Review of Research Evidence’, The International Journal of Public Sector Management, 17/6: 534–52.
32. Hennink, M. and Stephenson, R. (2006) ‘Using Research to Inform Health Policy: Barriers and Strategies in Developing Countries’, Journal of Health Communication: International Perspectives, 10: 163–80.
33. Innvaer, S. et al. (2002) ‘Health Policy-makers’ Perceptions of their use of Evidence: A Systematic Review’, Journal of Health Services Research and Policy, 7/4: 239–44.
34. Jadad, A. R. et al. (1996) ‘Assessing the Quality of Reports of Randomized Clinical Trials: Is Blinding Necessary?’, Controlled Clinical Trials, 17/1: 1–12.
35. Kothari, A., Birch, S. and Charles, C. (2005) ‘"Interaction" and Research Utilisation in Health Policies and Programs: Does it Work?’, Health Policy, 71/1: 117–25.
36. Kothari, A. et al. (2009) ‘Is Research Working for You? Validating a Tool to Examine the Capacity of Health Organizations to use Research’, Implementation Science, 4/46.
37. Landry, R., Amara, N. and Lamari, M. (2001a) ‘Climbing the Ladder of Research Utilization’, Science Communication, 22/4: 396–422.
38. Landry, R. (2001b) ‘Utilization of Social Science Research Knowledge in Canada’, Research Policy, 30: 333–49.
39. Landry, R., Lamari, M. and Amara, N. (2003) ‘The Extent and Determinants of the Utilization of University Research in Government Agencies’, Public Administration Review, 63/2: 192–205.
40. LaRocca, R. et al. (2012) ‘The Effectiveness of Knowledge Translation Strategies Used in Public Health: A Systematic Review’, BMC Public Health, 12: 751.
41. Lavis, J. N. et al. (2005) ‘Towards Systematic Reviews that Inform Health Care Management and Policy-Making’, Journal of Health Services Research & Policy, 10/Suppl 1: S1:35–48.
42. Lavis, J. N. et al. (2002) ‘Examining the Role of Health Services Research in Public Policymaking’, Milbank Quarterly, 80/1: 125–54.
43. Lemay, M. A. and Sa, C. (2014) ‘The Use of Academic Research in Public Health Policy and Practice’, Research Evaluation, 23: 79–88.
44. Lewin, S. et al. (2009) ‘SUPPORT Tools for Evidence-informed Health Policymaking (STP) 8: Deciding how much Confidence to Place in a Systematic Review’, Health Research Policy and Systems, 7/Suppl 1: S8.
45. Liverani, M., Hawkins, B. and Parkhurst, J. O. (2013) ‘Political and Institutional Influences on the use of Evidence in Public Health Policy’, PLoS ONE, 8/10: e77404.
46. Long, A. F. et al. (2002), HCPRDU Evaluation Tool for Mixed Methods Studies. Leeds: Nuffield Institute for Health.
47. Makkar, S. R. et al. (in press) ‘The Development of ORACLe: A Measure of an Organisation’s Capacity to Engage in Evidence-Informed Health Policy’, Health Research Policy and Systems.
48. Makkar, S. R. et al. (2015a) ‘Using Conjoint Analysis to Develop a System to Score Research Engagement Actions by Health Decision Makers’, Health Research Policy and Systems, 13: 22.
49. Makkar, S. R. et al. (2015b) ‘Using Conjoint Analysis to Develop a System of Scoring Policymakers’ use of Research in Policy and Program Development’, Health Research Policy and Systems, 13/1: 35.
50. Mays, N., Pope, C. and Popay, J. (2005) ‘Systematically Reviewing Qualitative and Quantitative Evidence to Inform Management and Policy-Making in the Health Field’, Journal of Health Services Research and Policy, 10: 6.
51. Milat, A. et al. (2013) ‘Policy and Practice Impacts of Applied Research: A Case Study Analysis of the New South Wales Health promotion Demonstration Research Grants Scheme 2000-2006’, Health Research Policy and Systems, 11: 5.
52. Mirzoev, T. N. et al. (2012) ‘Research-policy Partnerships - Experiences of the Mental Health and Poverty Project in Ghana, South Africa, Uganda and Zambia’, Health Research Policy and Systems, 10: 30.
53. Mitton, C. et al. (2007) ‘Knowledge Transfer and Exchange: Review and Synthesis of the Literature’, Milbank Quarterly, 85/4: 729–68.
54. Moat, K. A. et al. (2013) ‘Twelve Myths about Systematic Reviews for Health System Policymaking Rebutted’, Journal of Health Services Research and Policy, 18: 44.
55. Morrato, E. H., Elias, M. and Gericke, C. A. (2007) ‘Using Population-based Routine Data for Evidence-Based Health Policy Decisions: Lessons from Three Examples of Setting and Evaluating National Health Policy in Australia, the UK and the USA’, Journal of Public Health, 19/4: 463–71.
56. Oliver, K. et al. (2014) ‘A Systematic Review of Barriers to and Facilitators of the use of Evidence by Policymakers’, BMC Health Services Research, 14: 2.
57. Orton, L. et al. (2011) ‘The use of Research Evidence in Public Health Decision Making Processes: Systematic Review’, PLoS ONE, 6/7: e21704.
58. Oxman, A. D. et al. (2009) ‘SUPPORT Tools for Evidence-Informed Health Policymaking (STP) 2: Improving how your Organisation Supports the use of Research Evidence to Inform Policymaking’, Health Research Policy and Systems, 7/Suppl 1: S2.
59. Redman, S. et al. (2015) ‘The SPIRIT Action Framework: A Structured Approach to Selecting and Testing Strategies to Increase the use of Research in Policy’, Social Science and Medicine, 136–137: 147–55.
60. Ritter, A. (2009) ‘How do Drug Policy Makers Access Research Evidence?’, International Journal of Drug Policy, 20: 70–5.
61. Ross, S. et al. (2003) ‘Partnership Experiences: Involving Decision-Makers in the Research Process’, Journal of Health Services Research and Policy, 8/ Suppl 2: 26–34.
62. Seale, C. and Silverman, D. (1997) ‘Ensuring Rigour in Qualitative Research’, European Journal of Public Health, 7: 379–84.
63. Shea, B. J. et al. (2007) ‘Development of AMSTAR: A Measurement Tool to Assess the Methodological Quality of Systematic Reviews’, BMC Medical Research Methodology, 7: 10.
64. Squires, J. E. et al. (2011) ‘A Systematic Review of the Psychometric Properties of Self-Report Research Utilization Measures Used in Healthcare’, Implementation Science, 6/1: 83.
65. Sumner, A. et al. (2011) ‘What Shapes Research Impact on Policy? Understanding Research Uptake in sexual and reproductive health policy processes in resource poor contexts’, Health Research Policy and Systems, 9/Suppl 1: 53.
66. The CIPHER Investigators. (2014) ‘Supporting Policy in health with Research: An Intervention Trial (SPIRIT)-protocol for a Stepped Wedge Trial’, BMJ Open, 4/7: e005293.
67. Trostle, J., Bronfman, M. and Langer, A. (1999) ‘How do Researchers Influence Decision Makers? Case Studies of Mexican Policies’, Health Policy and Planning, 14/2: 103–14.
68. Walker, I. and Hulme, C. (1999) ‘Concrete Words are Easier to Recall than Abstract Words: Evidence for a Semantic Contribution to Short-term Serial Recall’, Journal of Experimental Psychology: Learning, Memory, and Cognition, 25/5: 1256–71.
69. Wang, S., Moss, J. R. and Hiller, J. E. (2005) ‘Applicability and Transferability of Interventions in Evidence-based Public Health’, Health Promotion International, 21/1: 76–83.
70. Wattenmaker, W. D. and Shoben, E. J. (1987) ‘Context and the Recallability of Concrete and Abstract Sentences’, Journal of Experimental Psychology: Learning, Memory, and Cognition, 13/1: 140–50.
71. Weiss, C. and Bucuvalas, M. J. (1980a) Social Science Research and Decision-Making. New York: Columbia University Press.
72. Weiss, C. H. (1979) ‘The Many Meanings of Research Utilization’, Public Administration Review, 39/5: 426–31.
73. Weiss, C. H. and Bucuvalas, M. J. (1980b) ‘Truth Tests and Utility Tests: Decision-Makers’ Frames of Reference for Social Science Research’, American Sociological Review, 45/2: 302–13.
74. Whitehead, M. et al. (2004) ‘Evidence for Public Health Policy on Inequalities 2: Assembling the Evidence Jigsaw’, Journal of Epidemiology and Community Health, 58: 817–21.
75. Zardo, P. and Collie, A. (2014) ‘Measuring use of Research Evidence in Public Health Policy: a Policy Content Analysis’, Implementation Science, 14: 496.