Mixed methods evaluation in dissemination and implementation science

Dissemination and Implementation Research in Health: Translating Science to Practice, Second Edition

Volumn , Issue , 2017, Pages 335-354

  Palinkas, Lawrence A ^a   Cooper, Brittany Rhoades ^b  

^a UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

^b WASHINGTON STATE UNIVERSITY   (United States)

Author keywords

Best practices; Evaluation; Mixed methods; Qualitative; Quantitative

Indexed keywords

EID: 85044867505     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1093/oso/9780190683214.003.0020     Document Type: Chapter

References (118)

1. Cresswell JW, Plano Clark VL. Designing and conducting mixed method research. 2nd ed. Thousand Oaks, CA: Sage; 2011.
2. Morse JM, Niehaus L. Mixed method design: Principles and procedures. Walnut Creek, CA: Left Coast Press; 2009.
3. Patton MQ. Qualitative research and evaluation methods. 3rd ed. Thousand Oaks, CA: Sage; 2002.
4. Teddlie C, Tashakkori A. Major issues and controversies in the use of mixed methods in the social and behavioral sciences. In: Tashakkori A, Teddlie C, eds. Handbook of mixed methods in the social and behavioral sciences. Thousand Oaks, CA: Sage; 2003:3-50.
5. Kane H, Lewis MA, Williams PA, Kahwati LC. Using qualitative comparative analysis to understand and quantify translation and implementation. Translational Behav Med. 2014;4:201-208.
6. Stange K, Crabtree BF, Miller WL. Publishing multimethod research. Ann Fam Med. 2006;4:292-294.
7. Palinkas LA, Soydan H. Translation and implementation of evidence based practice. New York: Oxford University Press; 2012.
8. Neta G, Sanchez MA, Chambers DA et al. Implementation science in cancer prevention and control: a decade of grant funding by the National Cancer Institute and future directions. Implement Sci. 2015;10:4
9. Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. 2012;50(3):217-226.
10. Proctor EK, Landsverk J, Aarons G, Chambers D, Glisson C, Mittman B. Implementation research in mental health services: an emerging science with conceptual, methodological and training challenges. Adm Policy Ment Health. 2009;36:24-34.
11. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50.
12. Demakis JG, McQueen L, Kizer KW, Fuessner JR. Quality Enhancement Research Initiative (QUERI): a collaboration between research and clinical practice. Med Care. 2000; 38:17-25.
13. Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. 2015;10:53.
14. Morse JM. Approaches to qualitative-quantitative methodological triangulation. Nurs Res. 1991;40:120-123.
15. Sandelowski M. Combining qualitative and quantitative sampling, data collection, and analysis techniques in mixed method studies. Res Nurs Health. 2000;23:246-255.
16. Greene JC, Caracelli VJ, Graham WF. Toward a conceptual framework for mixed method evaluation designs. Educ Eval Policy Analysis. 1989;11:255-274.
17. Morgan DL. Practical strategies for combining qualitative and quantitative methods: applications to health research. Qual Health Res. 1998; 8:262-276.
18. Steckler A, McLeroy KR, Goodman RM, et al Toward integrating qualitative and quantitative methods: an introduction. Health Educ Q. 1992;19:1-8.
19. Cresswell JW, Fetters MD, Ivankova NV. Designing a mixed methods study in primary care. Ann Fam Med. 2004; 2:7-12.
20. Creswell JW, Klassen AC, Plano Clark VL, Smith KC for the Office of Behavioral and Social Sciences Research. Best practices for mixed methods research in the health sciences. August 2011. National Institutes of Health. https://obssr. od.nih.gov/wp-content/uploads/2016/02/Best_ Practices_for_Mixed_Methods_Research.pdf
21. Curry L, Nunez-Smith M. Mixed methods in health services research: A practical primer. Thousand Oaks, CA: Sage Publications; 2015.
22. Cresswell JW. Mixed method research: introduction and application. In: Cizek GJ, ed. Handbook of educational policy. San Diego: Academic Press; 1999:455-472.
23. Waszak C, Sines MC. Mixed methods in psychological research. In: Tashakkori A., Teddlie C., eds. Handbook of mixed methods in the social and behavioral sciences. Thousand Oaks, CA: Sage; 2003:3-50.
24. Palinkas LA, Aarons GA, Horwitz SM, Chamberlain P, Hurlburt, M., Landsverk J. Mixed method designs in implementation research. Adm Policy Ment Health. 2011;38:44-53.
25. Aarons GA, Fettes DL, Sommerfeld DH, Palinkas LA. Mixed methods for implementation research: application to evidence-based practice implementation and staff turnover in community-based organizations providing child welfare services. Child Maltreat. 2012;17:67-79.
26. Albright K, Gechter K, Kempe A. Importance of mixed methods in pragmatic trials and dissemination and implementation research. Acad Pediatr. 2013;13:400-407.
27. Palinkas LA. Qualitative and mixed methods in mental health services and implementation research. J Clin Child Adolesc Psychology. 2014;43:851-861.
28. Whitley R, Gingerich S, Lutz WJ, Mueser KT. Implementing the illness management and recovery program in community mental health settings: facilitators and barriers. Psychiatr Serv. 2009;60:202-209.
29. Marshall T, Rapp CA, Becker DR, Bond GR. Key factors for implementing supported employment. Psychiatr Serv. 2008;59:886-892.
30. Marty D, Rapp C, McHugo G, Whitley R. Factors influencing consumer outcome monitoring in implementation ofevidence-based practices: results from the National EBP Implementation Project. Adm Policy Ment Health. 2008;35:204-211.
31. Rapp CA, Etzel-Wise D, Marty, D, et al. Barriers to evidence-based practice implementation: results of a qualitative study. Community Ment Health J. 2010;46(2):112-118.
32. Swain K, Whitley R, McHugo GJ, Drake RE. The sustainability of evidence-based practices in routine mental health agencies. Community Ment Health J.2010;46(2):119-129.
33. Aarons GA, Ehrhart MG, Farahnak LR, Hurlburt MS. Leadership and organizational change for implementation (LOCI): a randomized mixed method pilot study of a leadership and organization development intervention for evidence-based practice implementation. Implement Sci. 2015;10:11.
34. Trochim WM. An introduction to concept mapping for planning and evaluation. Eval Prog Plan. 1989;12:1-16.
35. Aarons GA, Wells R, Zagursky K, Fettes DL, Palinkas LA. Implementing evidence-based practice in community mental health agencies: multiple stakeholder perspectives. Am J Public Health. 2009;99(11), 2087-2095.
36. Watts BV, Shiner B, Zubkoff L, Carpenter-Song E, Ronconi JM, Coldwell CM. Implementation of evidence-based psychotherapies for post- traumatic stress disorder in VA specialty clinics. Psychiatr Serv. 2014; 65(5):648-653.
37. Kitson A, Harvey G, McCormack B. Enabling the implementation of evidence based practice: a conceptual framework. Qual Health Care. 1998;7:149-158.
38. Damschroder L, Lowery JC. Evaluation of a large- scale weight management program using the consolidated framework for implementation research (CFIR). Implement Sci. 2013;8:51.
39. Ragin CC. Redesigning social inquiry: Fuzzy sets and beyond. Chicago: University of Chicago Press; 2009.
40. Gilburt H, Slade M, Bird V, Oduola S, Craig TK. Promoting recovery-oriented practice in mental health services: a quasi-experimental mixed- methods study. BMC Psychiatry. 2013;13:167.
41. Kramer TF, Burns BJ. Implementing cognitive behavioral therapy in the real world: a case study of two mental health centers. Implement Sci. 2008;3:14.
42. Brunette MF, Asher D, Whitley R, et al. Implementation of integrated dual disorders treatment: a qualitative analysis of facilitators and barriers. Psychiatr Serv. 2008;59:989-995.
43. Duffy SA, Ronis DL, Ewing LA, et al. Implementation of the Tobacco Tactics intervention versus usual care in Trinity Health community hospitals. Implement Sci. 2016;11:147.
44. Zazzali JL, Sherbourne C, Hoagwood KE, Greene D, Bigley MF, Sexton TL. The adoption and implementation of an evidence based practice in child and family mental health services organizations: a pilot study of functional family therapy in New York State. Adm Policy Mental Health. 2008;35:38-49.
45. Blasinsky M, Goldman HH. Unutzer, J. Project IMPACT: a report on barriers and facilitators to sustainability. Adm Policy Mental Health. 2006;33:718-729.
46. Sales AE, Ersek M, Intrator OK, et al. Implementing goals of care conversations with veterans in VA long-term care setting: a mixed methods protocol. Implement Sci. 2016;11:132.
47. Aarons GA, Palinkas LA. Implementation of evidence-based practice in child welfare: service provider perspectives. Adm Policy Ment Health. 2007;34:411-419.
48. Woltmann EM, Whitley R, McHugo GJ., et al. The role of staff turnover in the implementation of evidence-based practices in health care. Psychiatr Serv. 2008;59:732-737.
49. Hargreaves JR, Stangl A, Bond V, et al. HIV-related stigma and universal testing for HIV prevention and care: design of an implementation science evaluation nested in the HPTN 071 (PopART) cluster-randomized trial in Zambia and South Africa. Health Policy Plan. 2016;31:1342-1354.
50. Willging CE, Green AE, Ramos MM. Implementing school nursing strategies to reduce LGBTQ adolescent suicide: a randomized cluster trial study protocol. Implement Sci. 2016;11:145.
51. Chamberlain P, Brown CH, Saldana L. Observational measure of implementation progress in community-based settings: the stages of implementation completion (SIC). Implement Sci. 2011;6:116.
52. Aarons GA, Green AE, Palinkas LA, et al. Dynamic adaptation process to implement an evidence-based child maltreatment intervention. Implement Sci. 2012;7:32
53. Locke J, Beidas RS, Marcus S, et al. A mixed methods study of individual and organizational factors that affect implementation of interventions for children with autism in public schools. Implement Sci. 2016;11:135.
54. Tabak RG, Khoong EC, Chambers D, Brownson RC. Models in dissemination and implementation research: useful tools in public health services and systems research. Am J Prev Med. 2012;43(3):337-350.
55. Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health. 2011;38(1):4-23.
56. Mauricio AM, Tein JY, Gonzales NA, Millsap RE, Dumka LE, Berkel C. Participation patterns among Mexican-American parents enrolled in a universal intervention and their association with child externalizing outcomes. Am J Community Psychol. 2014;54(3-4):370-383.
57. Low S, Smolkowski K, Cook C. What constitutes high-quality implementation of SEL programs? A latent class analysis of Second Step- Implementation. Prev Sci. 2016;17(8):981-991.
58. Mcintosh K, Mercer SH, Nese RNT, Ghemraoui A. Identifying and predicting distinct patterns of implementation in a school-wide behavior support framework speed and patterns of implementation. Prev Sci. 2016:992-1001.
59. Berkel C, Mauricio AM, Schoenfelder E, Sandler IN. Putting the pieces together: an integrated model of program implementation. Prev Sci. 2011;12(1):23-33.
60. Aarons GA, Green AE, Trott E, et al. The roles of system and organizational leadership in system-wide evidence-based intervention sustainment: a mixed-method study. Adm Policy Ment Health. 2016;43(6):991-1008.
61. Welsh JA, Chilenski SM, Johnson L, Greenberg MT, Spoth RL. Pathways to sustainability: 8-Year follow-up from the PROSPER project. J Prim Prev. 2016;37(3):263-286.
62. Green AE, Trott E, Willging CE, Finn NK, Ehrhart MG, Aarons GA. The role of collaborations in sustaining an evidence-based intervention to reduce child neglect. Child Abus Negl. 2016;53:4-16.
63. Hunter SB, Han B, Slaughter ME, Godley SH, Garner BR. Associations between implementation characteristics and evidence-based practice sustainment: a study of the Adolescent Community Reinforcement Approach. Implement Sci. 2015;10:173.
64. Shapiro CJ, Prinz RJ, Sanders MR. Sustaining use of an evidence-based parenting intervention: practitioner perspectives. J Child Fam Stud. 2015;24:1615-1624.
65. Berkel C, Murry VM, Roulston KJ, Brody GH. Understanding the art and science of implementation in the SAAF efficacy trial. Health Educ. 2013;113(4):297-323.
66. Goncy EA, Sutherland KS, Farrell AD, Sullivan TN, Doyle ST. Measuring teacher implementation in delivery of a bullying prevention program: the impact of instructional and procedural adherence and competence on student responsiveness. Prev Sci. 2015;16(3):440-450.
67. Azocar F, Cuffel B, Goldman W, McCarter L. The impact of evidence-based guideline dissemination for the assessment and treatment of major depression in a managed behavioral health care organization. J Behav Health Serv Res. 2003;30:109-118.
68. Beidas RS, Kendall PC. Training therapists in evidence-based practice: a critical review of studies from a systems-contextual perspective. Clin Psychol Sci Pract. 2010;17(1):1-30.
69. Kilbourne AM, Abraham KM, Goodrich DE, et al. Cluster randomized adaptive implementation trial comparing a standard versus enhanced implementation intervention to improve uptake of an effective re-engagement program for patients with serious mental illness. Implement Sci. 2013;8:136.
70. Powell BJ, Proctor EK, Glass JE. A systematic review of strategies for implementing empirically supported mental health interventions. Res Soc Work Pract. 2014;24(2):192-212.
71. Proctor E, Silmere H, Raghavan R, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38(2):65-76.
72. Atkins MS, Frazier SL, Leathers SJ, et al. Teacher key opinion leaders and mental health consultation in low-income urban schools. J Consult Clin Psychol. 2008;76(5):905-908.
73. Lochman JE, Boxmeyer C, Powell N, Qu L, Wells K, Windle M. Dissemination of the Coping Power program: importance of intensity of counselor training. J Consult Clin Psychol. 2009;77(3):397-409.
74. Glisson C, Schoenwald SK, Hemmelgarn A, et al. Randomized trial of MST and ARC in a two-level evidence-based treatment implementation strategy. J Consult Clin Psychol. 2010;78(4):537-550.
75. Kirk MA, Kelley C, Yankey N, Birken SA, Ab adie B, Damschroder L. A systematic review of the use of the Consolidated Framework for Implementation Research. Implement Sci. 2016;11(1):72.
76. Yost J, Ganann R, Thompson D, et al. The effectiveness of knowledge translation interventions for promoting evidence-informed decision-making among nurses in tertiary care: a systematic review and meta-analysis. Implement Sci. 2015;10(1):98.
77. Grudniewicz A, Kealy R, Rodseth RN, Hamid J, Rudoler D, Straus SE. What is the effectiveness of printed educational materials on primary care physician knowledge, behaviour, and patient outcomes: a systematic review and meta-analyses. Implement Sci. 2015;10:164.
78. Davies P, Walker A, Grimshaw J. A systematic review of the use of theory in the design of guideline dissemination and implementation strategies and interpretation of the results of rigorous evaluations. Implement Sci. 2010;5:14.
79. Webb TL, Joseph J, Yardley L, Michie S. Using the internet to promote health behavior change: a systematic review and meta-analysis of the impact of theoretical basis, use of behavior change techniques, and mode of delivery on efficacy. J Med Internet Res. 2010;12:1-18.
80. Brown CH, Curran G, Palinkas LA, et al. An overview of research and evaluation for dissemination and implementation. Annu Rev Public Health. 2017;38(March):1-48.
81. Chaudoir SR, Dugan AG, Barr CH. Measuring factors affecting implementation of health innovations: a systematic review of structural, organizational, provider, patient, and innovation level measures. Implement Sci. 2013:8:22.
82. Lewis CC, Stanick CF, Martinez RG, et al. The Society for Implementation Research Collaboration Instrument Review Project: a methodology to promote rigorous evaluation. Implement Sci. 2015;10:2.
83. Gibbs DA, Krieger KE, Cutbush SL, Clinton- Sherrod AM, Miller S. Implementer-initiated adaptation of evidence-based interventions: kids remember the blue wig. Health Educ Res. 2016;31(3):405-415.
84. Brownson RC, Allen P, Jacob RR, et al. Understanding mis-implementation in public health practice. Am J Prev Med. 2015;48(5):543-551.
85. Cohen DJ, Balasubramanian BA, Gordon L, et al. A national evaluation of a dissemination and implementation initiative to enhance primary care practice capacity and improve cardiovascular disease care: the ESCALATES study protocol. Implement Sci. 2016;11:86.
86. Kozica SL, Lombard CB, Harrison CL, Teede HJ. Evaluation of a large healthy lifestyle program: informing program implementation and scale- up in the prevention of obesity. Implement Sci. 2016;11:151.
87. Colon-Emeric C, Toles M, Cary Jr. MP, et al. Sustaining complex interventions in long-term care: a qualitative study of direct care staff and managers. Implement Sci. 2016;11:94.
88. Fox AB, Hamilton AB, Frayne SN, et al. Effectiveness of an evidence-based quality improvement approach to cultural competence training: The Veterans Affairs "Caring for Women Veterans" program. J Continuing Educ Health Professions. 2016;36:96-103.
89. Lessard S, Bareil C, Lalonde L, et al. External facilitators and interprofessional facilitation teams: a qualitative study of their roles in supporting practice change. Implement Sci. 2016;11:97.
90. Hoagwood KE, Vogel JM, Levitt JM, DAmico PJ, Paisner WI, Kaplan SJ. Implementing an evidence- based trauma treatment in a state system after September 11: the CATS Project. J Am Acad Child Adolesc Psychiatry. 2007;46(6):773-779.
91. McHugh S, Tracey ML, Riordan F, O.Neill K, Mays N, Kearney PM. Evaluating the implementation of a national clinical programme for diabetes to standardize and improve services: a realist evaluation protocol. Implement Sci. 2016;11:07.
92. Palinkas LA, Spear SE, Mendon SJ, et al. Measuring sustainment of prevention programs and initiatives: a study protocol. Implement Sci. 2016;11:95.
93. Saldana L. The states of implementation completion for evidence-based practice: protocol for a mixed methods study. Implement Sci. 2014;9:43.
94. Martinez RG, Lewis CC, Weiner BJ. Instrumentation issues in implementation science. Implement Sci. 2014;9:118.
95. Elsey H, Khanal S, Manandhar S, et al. Understanding implementation and feasibility of tobacco cessation in routine primary care in Nepal: a mixed methods study. Implement Sci. 2016;11:104.
96. Miles MB, Huberman AM. Qualitative data analysis: An expanded sourcebook. 2nd ed. Thousand Oaks, CA: Sage; 1994.
97. Sommerbakk R, Haugen DF, Tjora A, Kaasa S, Hjermstad MJ. Barriers to and facilitators for implementing quality improvements in palliative care-results for a qualitative interview study in Norway. BMC Palliat Care. 2016; 15:61.
98. Grol R, Wensing M. What drives change? Barriers to and incentives for achieving evidence-based practice. Med J Aust. 2004;180(6 Suppl):S57-60.
99. May C, Finch T. Implementation, embedding and integration: an outline of normalization process theory. Sociology. 2009;43:535-554.
100. Padgett DK. Qualitative methods in social work research. 2nd ed. Los Angeles, CA: Sage; 2008.
101. Palinkas LA, Horwitz SM, Green CA, Wisdom JP, Duan N, Hoagwood KE. Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Adm Policy Ment Health. 2015;42:533-544.
102. Tabak RG, Khoong EC, Chambers DA, Brownson RC. Bridging research and practice: models for dissemination and implementation research. Am J Prev Med. 2012;43(3):337-350.
103. Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health. 2011;38:4-23.
104. Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004; 82(4):581-629.
105. Chambers DA, Glasgow RE, Stange KC. The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change. Implement Sci. 2013;8:117.
106. Benz J, Infante A, Oppenheimer C, Scherer H, Wilson W. Developing a conceptual framework to assess the sustainability of community coalitions post-federal funding. Chicago: National Opinion Research Center; 2011.
107. Feinberg ME, Bontempo DE, Greenberg MT. Predictors and level of sustainability of community prevention coalitions. Am J Prev Med. 2008; 34(6):495-501.
108. Gloppen KM, Arthur MW, Hawkins JD, Shapiro VB. Sustainability of the Communities That Care prevention system by coalitions participating in the Community Youth Development Study. J Adolesc Health. 2012; 51(3):259-264.
109. Rhew IC, Brown EC, Hawkins JD, Briney JS. Sustained effects of the Communities That Care system on prevention service system transformation. Am J Public Health. 2013; 103(3):529-535.
110. Glaser BG, Strauss AL. The discovery of grounded theory: Strategies for qualitative research. New York: Aldine de Gruyter, 1967
111. Willms DG, Best JA, Taylor DW, et al. A systematic approach for using qualitative methods in primary prevention research. Med Anthro Q. 1990;4(4), 391-409.
112. Palinkas LA, Spear S, Men don S, Villamar J, Brown CH. Definition and Prediction of Sustainment of Prevention Programs and Initiatives. Presented at the 9th Conference on the Science of Dissemination and Implementation, Washington DC, December 14, 2016.
113. Palinkas LA, Holloway IW, Rice E, Fuentes D, Wu Q, Chamberlain, P. Social networks and implementation of evidence-based practices in youth-serving systems: a mixed methods study. Implement Sci. 2011;6:113.
114. Palinkas LA, Holloway IW, Rice E, Brown CH, Valente T, Chamberlain P. Influence network linkages across treatment conditions in a randomized controlled trial of two implementation strategies for scaling up evidence-based practices in public youth-serving systems. Implement Sci. 2013;8:133.
115. Rice ER, Holloway, IW, Barman-Adhikari A, Fuentes D, Brown CH, Palinkas LA. A mixed methods approach to network data collection. Field Methods. 2014;26(3):252-268.
116. Palinkas LA, Fuentes D, Garcia AR, Finno M, Holloway IW, Chamberlain P. Inter-organizational collaboration in the implementation of evidence-based practices among agencies serving abused and neglected youth. Adm Policy Ment Health. 2014;41:74-85.
117. Palinkas LA, Garcia AR, Aarons GA, et al. Measuring use of research evidence in child- serving systems: the Structured Interview for Evidence Use (SIEU). Res Soc Work Pract. 2016;26(5):550-564.
118. Palinkas LA, Brown CH, Saldana L, Chamberlain P. Association between inter-organizational consensus on use of research evidence and stage of implementation of an evidence-based practice. Implement Sci. 2015;10(Suppl 1):A20.