Policymakers' experience of a capacity-building intervention designed to increase their use of research: A realist process evaluation

Health Research Policy and Systems

Volumn 15, Issue 1, 2017, Pages

  Haynes, Abby ^a,b   Brennan, Sue ^c   Redman, Sally ^a   Williamson, Anna ^a   Makkar, Steve R ^a   Gallego, Gisselle ^d   Butow, Phyllis ^e  

^a SAX INSTITUTE   (Australia)

^b UNIVERSITY OF SYDNEY   (Australia)

^c MONASH UNIVERSITY   (Australia)

^d UNIVERSITY OF NOTRE DAME   (Australia)

^e UNIVERSITY OF SYDNEY   (Australia)

Author keywords

Health policy; Participant perspectives; Process evaluation; Realist evaluation; Research utilisation

Indexed keywords

ARTICLE; ATTITUDE; CAPACITY BUILDING; DATA ANALYSIS; DATA COLLECTION METHOD; EVALUATION STUDY; HEALTH CARE POLICY; HUMAN; INTERPERSONAL COMMUNICATION; OUTCOME ASSESSMENT; PERCEPTION; PROCESS EVALUATION; RANDOMIZED CONTROLLED TRIAL (TOPIC); RESEARCH; WORKSHOP; ADMINISTRATIVE PERSONNEL; FEEDBACK SYSTEM; MANAGEMENT; PROGRAM EVALUATION; QUESTIONNAIRE;

ADMINISTRATIVE PERSONNEL; ATTITUDE; CAPACITY BUILDING; FEEDBACK; HEALTH POLICY; HUMANS; POLICY MAKING; PROGRAM EVALUATION; RESEARCH; SURVEYS AND QUESTIONNAIRES;

EID: 85035214653     PISSN: None     EISSN: 14784505     Source Type: Journal    
DOI: 10.1186/s12961-017-0234-4     Document Type: Article

References (88)

1. Pawson R. The Science of Evaluation: A Realist Manifesto. London: Sage; 2013.
2. Greenhalgh T, Wong G, Jagosh J, Greenhalgh J, Manzano A, Westhorp G, Pawson R. Protocol-the RAMESES II study: developing guidance and reporting standards for realist evaluation. BMJ Open. 2015;5(8):e008567.
3. Pawson R, Tilley N. Realist Evaluation. South Australia: Community Matters; 2004.
4. Astbury B. Some reflections on Pawson's Science of evaluation: a realist manifesto. Evaluation. 2013;19:383-401.
5. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50.
6. Funnell SC, Rogers PJ. Purposeful Program Theory: Effective Use of Theories of Change and Logic Models. San Francisco: Jossey-Bass; 2011.
7. Ritchie J, Lewis J. Qualitative Research Practice: A Guide for Social Science Students and Researchers. London: SAGE; 2003.
8. Rycroft-Malone J, Fontenla M, Bick D, Seers K. A realistic evaluation: the case of protocol-based care. Implement Sci. 2010;5:38.
9. Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, Moore L, O'Cathain A, Tinati T, Wight D, Baird J. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015;350:h1258.
10. Astbury B, Leeuw FL. Unpacking black boxes: mechanisms and theory building in evaluation. Am J Eval. 2010;31:363-81.
11. Nielsen K. How can we make organizational interventions work? Employees and line managers as actively crafting interventions. Hum Relat. 2013;66:1029-50.
12. Nielsen K, Randall R. Opening the black box: Presenting a model for evaluating organizational-level interventions. Eur J Work Organ Psy. 2013;22:601-17.
13. Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004;82:581-629.
14. McLean S, Moss G. They're happy, but did they make a difference? Applying Kirkpatrick's framework to the evaluation of a national leadership program. Can J Prog Eval. 2003;18:1-23.
15. Wells M, Williams B, Treweek S, Coyle J, Taylor J. Intervention description is not enough: evidence from an in-depth multiple case study on the untold role and impact of context in randomised controlled trials of seven complex interventions. Trials. 2012;13:95-111.
16. Datta A, Shaxson L, Pellini A. Capacity, Complexity and Consulting: Lessons from Managing Capacity Development Projects. ODI Working Paper 344. London: Overseas Development Institute; 2012.
17. Moore GF. Developing a Mixed Methods Framework for Process Evaluations of Complex Interventions: The Case of the National Exercise Referral Scheme Policy Trial in Wales. Cardiff: Cardiff University; 2010.
18. Schein EH. Organizational Culture and Leadership. London: John Wiley & Sons; 2010.
19. Devos G, Buelens M, Bouckenooghe D. Contribution of content, context, and process to understanding openness to organizational change: two experimental stimulation studies. J Soc Psychol. 2007;147:607-30.
20. Dixon-Woods M, Leslie M, Tarrant C, Bion J. Explaining Matching Michigan: an ethnographic study of a patient safety program. Implement Sci. 2013;8:70.
21. Andersen H, Røvik KA. Lost in translation: a case-study of the travel of lean thinking in a hospital. BMC Health Serv Res. 2015;15:401.
22. Fixsen DL, Naoom SF, Blase KA, Friedman RM, Wallace F. Implementation Research: A Synthesis of the Literature, Florida Mental Health Institute Publication 231. Tampa: National Implementation Research Network, University of South Florida; 2005.
23. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008;337:a1655.
24. Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, Moore L, O'Cathain A, Tinati T, Wight D, Baird J. Process Evaluation of Complex Interventions: Medical Research Council Guidance. A Report Prepared on Behalf of the MRC Population Health Science Research Network. London: Institute of Education; 2015.
25. Hoddinott P, Britten J, Pill R. Why do interventions work in some places and not others: A breastfeeding support group trial. Soc Sci Med. 2010;70:769-78.
26. Ferlie E, Dopson S, Fitzgerald L, Locock L. Renewing policy to support evidence-based health care. Public Admin. 2009;87:837-52.
27. Oliver K, Lorenc T, Innvær S. New directions in evidence-based policy research: a critical analysis of the literature. Health Res Policy Sys. 2014;12:34.
28. Haynes A, Gillespie JA, Derrick GE, Hall WD, Redman S, Chapman S, Sturk H. Galvanizers, guides, champions, and shields: the many ways that policymakers use public health researchers. Milbank Q. 2011;89:564-98.
29. Liverani M, Hawkins B, Parkhurst JO. Political and institutional influences on the use of evidence in public health policy. A systematic review. PLoS ONE. 2013;8:e77404.
30. Russell J, Greenhalgh T, Byrne E, McDonnell J. Recognizing rhetoric in health care policy analysis. J Health Serv Res Policy. 2008;13:40-6.
31. Majone G. Evidence, argument, and persuasion in the policy process. New Haven, CT: Yale University Press; 1989.
32. Rycroft-Malone J. The PARIHS framework-a framework for guiding the implementation of evidence based practice. J Nurs Care Qual. 2004;19:297-304.
33. Crilly T, Jashapara A, Ferlie E. Research Utilisation & Knowledge Mobilisation: A Scoping Review of the Literature. Report for the National Institute for Health Research Service Delivery and Organization. London: Queen's Printer and Controller of HMSO; 2010.
34. Tomoaia-Cotisel A, Scammon DL, Waitzman NJ, Cronholm PF, Halladay JR, Driscoll DL, Solberg LI, Hsu C, Tai-Seale M, Hiratsuka V, et al. Context matters: the experience of 14 research teams in systematically reporting contextual factors important for practice change. Ann Fam Med. 2013;11:S115-23.
35. Clark AM, MacIntyre PD, Cruickshank J. A critical realist approach to understanding and evaluating heart health programmes. Health. 2007;11:513-39.
36. The CIPHER Investigators. Supporting Policy In health with Research: an Intervention Trial (SPIRIT)-protocol for a stepped wedge trial. BMJ Open. 2014;4(7):e005293.
37. Makkar SR, Turner T, Williamson A, Louviere J, Redman S, Haynes A, Green S, Brennan S. The development of ORACLe: a measure of an organisation's capacity to engage in evidence-informed health policy. Health Res Policy Syst. 2016;14:4.
38. Makkar SR, Brennan S, Turner T, Williamson A, Redman S, Green S. The development of SAGE: a tool to evaluate how policymakers' engage with and use research in health policymaking. Res Evaluat. 2016;25:315-28.
39. Makkar SR, Williamson A, Turner T, Redman S, Louviere J. Using conjoint analysis to develop a system of scoring policymakers' use of research in policy and program development. Health Res Policy Syst. 2015;13:35.
40. Makkar SR, Williamson A, Turner T, Redman S, Louviere J. Using conjoint analysis to develop a system to score research engagement actions by health decision makers. Health Res Policy Syst. 2015;13:22.
41. Brennan SE, McKenzie JE, Turner T, Redman S, Makkar S, Williamson A, Haynes A, Green SE. Development and validation of SEER (Seeking, Engaging with and Evaluating Research): a measure of policymakers' capacity to engage with and use research. Health Res Policy Syst. 2017;15:1.
42. Redman S, Turner T, Davies H, Williamson A, Haynes A, Brennan S, Milat A, O'Connor D, Blyth F, Jorm L, Green S. The SPIRIT Action Framework: A structured approach to selecting and testing strategies to increase the use of research in policy. Soc Sci Med. 2015;136-137:147-55.
43. Fridrich A, Jenny GJ, Bauer GF. The context, process, and outcome evaluation model for organisational health interventions. Biomed Res Int. 2015;2015:Article ID 414832.
44. Westhorp G, Prins E, Kusters C, Hultink M, Guijt I, Brouwers J. Realist Evaluation: An Overview. Report from an Expert Seminar with Dr Gill Westhorp. Wageningen: Centre for Development Innovation, Wageningen University; 2011.
45. Wong G, Westhorp G, Manzano A, Greenhalgh J, Jagosh J, Greenhalgh T. RAMESES II reporting standards for realist evaluations. BMC Med. 2016;14:96.
46. Haynes A, Brennan S, Carter S, O'Connor D, Huckel Schneider C, Turner T, Gallego G. Protocol for the process evaluation of a complex intervention designed to increase the use of research in health policy and program organisations (the SPIRIT study). Implement Sci. 2014;9:113.
47. Martin GP, Ward V, Hendy J, Rowley E, Nancarrow S, Heaton J, Britten N, Fielden S, Ariss S. The challenges of evaluating large-scale, multi-partner programmes: the case of NIHR CLAHRCs. Evid Policy. 2011;7:489-509.
48. Marchal B, Dedzo M, Kegels G. A realist evaluation of the management of a well-performing regional hospital in Ghana. BMC Health Serv Res. 2010;10:24.
49. Salter KL, Kothari A. Using realist evaluation to open the black box of knowledge translation: a state-of-the-art review. Implement Sci. 2014;9:115.
50. Pawson R. Middle range theory and program theory evaluation: from provenance to practice. In: Leeuw F, Vaessen J, editors. Mind the Gap: Perspectives on Policy Evaluation and the Social Sciences, vol. 16. Piscataway, NJ: Transaction Press; 2009. p. 171-203.
51. McEvoy P, Richards D. A critical realist rationale for using a combination of quantitative and qualitative methods. J Res Nurs. 2006;11:66-78.
52. Dalkin SM, Greenhalgh J, Jones D, Cunningham B, Lhussier M. What's in a mechanism? Development of a key concept in realist evaluation. Implement Sci. 2015;10:49.
53. Cartwright N. Knowing what we are talking about: why evidence doesn't always travel. Evid Policy. 2013;9:97-112.
54. Merton RK. On sociological theories of the middle range. In: Social Theory and Social Structure. New York: Simon & Schuster, The Free Press; 1949. pp. 39-53.
55. Punton M, Vogel I, Lloyd R. Reflections from a realist evaluation in progress: scaling ladders and stitching theory. In: CDI Practice Papers. Brighton: Institute of Development Studies; 2016.
56. Abimbola S, Molemodile SK, Okonkwo OA, Negin J, Jan S, Martiniuk AL. 'The government cannot do it all alone': realist analysis of the minutes of community health committee meetings in Nigeria. Health Policy Plan. 2016;31(3):332-45.
57. Adams A, Sedalia S, McNab S, Sarker M. Lessons learned in using realist evaluation to assess maternal and newborn health programming in rural Bangladesh. Health Policy Plan. 2016;31:267-75.
58. Rushmer R, Hunter D, Steven A. Using interactive workshops to prompt knowledge exchange: a realist evaluation of a knowledge to action initiative. Public Health. 2014;128(6):552-60.
59. Haynes A, Brennan S, Redman S, Williamson A, Gallego G, Butow P. Figuring out fidelity: a worked example of the methods used to identify, critique and revise the essential elements of a contextualised intervention in health policy agencies. Implement Sci. 2016;11:23.
60. Danermark B, Ekstrom M, Jakobsen L, Karlsson J. Explaining Society: An Introduction to Critical Realism in the Social Sciences. London: Routledge; 2002.
61. Chen H-T. Practical Program Evaluation: Assessing and Improving Planning, Implementation, and Effectiveness. Thousand Oaks: Sage; 2005.
62. Kitson AL, Rycroft-Malone J, Harvey G, McCormack B, Seers K, Titchen A. Evaluating the successful implementation of evidence into practice using the PARiHS framework: theoretical and practical challenges. Implement Sci. 2008;3:1.
63. Hawe P, Shiell A, Riley T. Theorising interventions as events in systems. Am J Community Psychol. 2009, 43:267-76.
64. Damschroder LJ, Lowery JC. Evaluation of a large-scale weight management program using the consolidated framework for implementation research (CFIR). Implement Sci. 2013;8:51.
65. May C, Finch T. Implementing, embedding, and integrating practices: an outline of normalization process theory. Sociology. 2009;43:535-54.
66. Normalization Process Theory Online Users' Manual, Toolkit and NoMAD Instrument. http://www.normalizationprocess.org. Accessed 3 May 2017.
67. Greenhalgh T, Humphrey C, Hughes J, Macfarlane F, Butler C, Pawson R. How do you modernize a health service? A realist evaluation of whole-scale transformation in London. Milbank Q. 2009;87:391-416.
68. Sayer A. Realism and Social Science. 2nd ed. London: Sage; 2000.
69. Ritchie J, Spencer L, O'Connor W. Carrying out qualitative analysis. In: Ritchie J, Lewis J, editors. Qualitative Research Practice: A Guide for Social Science Students and Researchers. London: SAGE; 2003. p. 219-62.
70. International Pty Ltd QSR. NVivo Qualitative Data Analysis Software: Version 10. 2012.
71. Charlson FJ, Moran AE, Freedman G, Norman RE, Stapelberg NJ, Baxter AJ, Vos T, Whiteford HA. The contribution of major depression to the global burden of ischemic heart disease: a comparative risk assessment. BMC Med. 2013;11:250.
72. Thorne S. Interpretive Description. Walnut Creek: Left Coast Press; 2008.
73. Thomas DR. A general inductive approach for analyzing qualitative evaluation data. Am J Eval. 2006;27:237-46.
74. Silverman D. Doing Qualitative Research: A Practical Handbook. 4th ed. London: SAGE Publications; 2013.
75. Sayer A. Method in Social Science: A Realist Approach. 2nd ed. Abingdon: Routledge; 2010.
76. Sandelowski M. Combining qualitative and quantitative sampling, data collection, and analysis techniques in mixed-method studies. Res Nurs Health. 2000;23:246-55.
77. Caracelli VJ, Greene JC. Data analysis strategies for mixed-method evaluation designs. Educ Eval Policy An. 1993;15:195-207.
78. Tremblay D, Touati N, Roberge D, Denis J-L, Turcotte A, Samson B. Conditions for production of interdisciplinary teamwork outcomes in oncology teams: protocol for a realist evaluation. Implement Sci. 2014;9:76.
79. Reed M. Reflections on the 'realist turn' in organization and management studies. J Manage Stud. 2005;42:1621-44.
80. Jagosh J, Pluye P, Wong G, Cargo M, Salsberg J, Bush PL, Herbert CP, Green LW, Greenhalgh T, Macaulay AC. Critical reflections on realist review: insights from customizing the methodology to the needs of participatory research assessment. Res Synth Methods. 2014;5:131-41.
81. Materially Social. Causality, Method and Imagination. 2016. http://materiallysocial.blogspot.com.au/2016/05/causality-method-and-imagination.html. Accessed 3 May 2017.
82. Dearing JW. Applying diffusion of innovation theory to intervention development. Res Social Work Prac. 2009;19:503-18.
83. Edwards M, Evans M. Getting evidence into policy-making: parliamentary triangle seminar report. In: ANZSIG Insights. Canberra: ANZOG Institute for Governance, University of Canberra; 2011.
84. Aguinis H, Henle CA. Ethics in research. In: Rogelberg S, editor. Handbook of Research Methods in Industrial and Organizational Psychology. Oxford: Blackwell; 2004. p. 34-56.
85. Srivastava UR, Singh M. Psychological empowerment at the work place. Global J Bus Man. 2008;2:53-73.
86. Barnes C, Blake H, Pinder D. Creating and Delivering your Value Proposition: Managing Customer Experience for Profit. London: Kogan Page Publishers; 2009.
87. Bandura A. Self-efficacy in Changing Societies. New York: Cambridge University Press; 1995.
88. Haynes A, Butow P, Brennan S, Williamson A, Redman S, Carter S, Gallego G, Rudge S. The pivotal position of 'liaison people': facilitating a research utilisation intervention in policy agencies. Evid Policy. 2016. https://doi.org/10.1332/174426416X14817284217163.