Innovations in Mixed Methods Evaluations

Annual Review of Public Health

Volumn 40, Issue , 2019, Pages 423-442

  Palinkas, Lawrence A ^a   Mendon, Sapna J ^a   Hamilton, Alison B ^b,c  

^a UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c VA GREATER LOS ANGELES HEALTHCARE SYSTEM   (United States)

Author keywords

effectiveness; evaluation; hybrid designs; implementation; methodology; mixed methods

Indexed keywords

CLINICAL ASSESSMENT; EVALUATION RESEARCH; OUTCOME ASSESSMENT; PUBLIC HEALTH PROBLEM; QUANTITATIVE ANALYSIS; REVIEW; SAMPLING; STRUCTURE ACTIVITY RELATION; HUMAN; METHODOLOGY; PROCEDURES; PROGRAM EVALUATION; PUBLIC HEALTH;

HUMANS; OUTCOME AND PROCESS ASSESSMENT, HEALTH CARE; PROGRAM EVALUATION; PUBLIC HEALTH; RESEARCH DESIGN;

EID: 85063893032     PISSN: 01637525     EISSN: 15452093     Source Type: Book Series    
DOI: 10.1146/annurev-publhealth-040218-044215     Document Type: Review

References (99)

1. Aarons GA, Fettes DL, Sommerfeld DH, Palinkas LA. 2012. Mixed methods for implementation research: Application to evidence-based practice implementation and staff turnover in community-based organizations providing child welfare services. Child Maltreat. 17:67-79
2. Aarons GA,Wells R, Zagursky K, Fettes DL, Palinkas LA. 2009. Implementing evidence-based practice in community mental health agencies: A multiple stakeholder analysis. Am. J. Public Health 99:2087-95
3. Ackerman SL, Sarkar U, Tieu L, Handley MA, Schillinger D, et al. 2017. Meaningful use in the safety net: A rapid ethnography of patient portal implementation at five community health centers in California. J. Am.Med. Inform. Assoc. 24(5):903-12
4. Albright K, Gechter K, Kempe A. 2013. Importance of mixed methods in pragmatic trials and dissemination and implementation research. Acad. Pediatr. 13:400-7
5. Beebe J. 1995. Basic concepts and techniques of rapid appraisal. Hum. Org. 54:42-51
6. Bogart LM, Fu CM, Eyraud J, Cowgill BO,Hawes-Dawson J, et al. 2018. Evaluation of the dissemination of SNaX, a middle school-based obesity prevention intervention, within a large US school district. Transl. Behav. Med. 8:724-32
7. Broder-Fingert S,Walls M, Augustyn M, Beidas R, Mandell D, et al. 2018. A hybrid type I randomized effectiveness-implementation trial of patient navigation to improve access to services for children with autism spectrum disorder. BMC Psychiatry 18:79
8. Brown CH, Curran G, Palinkas LA, Aarons GA, Wells KB, et al. 2017. An overview of research and evaluation designs for dissemination and implementation. Annu. Rev. Public Health 38:1-22
9. Chamberlain P, Brown C, Saldana L. 2011. Observational measure of implementation progress in community-based settings: The stages of implementation completion (SIC). Implement. Sci. 6:116
10. Chamberlain P, Mihalic SF. 1998. Blueprints for Violence Prevention, Book Eight: Multidimensional Treatment Foster Care. Boulder, CO: Cent. Study Prev. Viol.
11. Choy I, Kitto S, Adu-Aryee N, Okrainec A. 2013. Barriers to uptake of laparoscopic surgery in a lowermiddle-income country. Surg. Endosc. 27:4009-15
12. Cohen JA, Mannarino AP, Deblinger E. 2006. Treating Trauma and Traumatic Grief in Children and Adolescents. New York: Guilford Press
13. Cook PA, Hargreaves SC, Burns EJ, de Vocht F, Parrott S, et al. 2018. Communities in charge of alcohol (CICA): A protocol for a stepped-wedge randomized control trial of an alcohol health champions programme. BMC Public Health 18:522
14. Corbin J, Strauss A. 2008. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. Thousand Oaks, CA: Sage
15. Cragun D, Pal T, Vadaparampil ST, Baldwin J, Hampel H, DeBate RD. 2016. Qualitative comparative analysis: A hybrid method for identifying factors associated with program effectiveness. J. Mix. Methods Res. 10(3):251-72
16. Creswell JW, Klassen AC, Plano Clark VL, Clegg Smith K. 2011. Best practices for mixed methods research in the health sciences. Rep., Off. Behav. Soc. Sci. Res., Natl. Inst. Health, Bethesda, MD. https://obssr. od.nih.gov/wp-content/uploads/2016/02/Best-Practices-for-Mixed-Methods-Research.pdf
17. Creswell JW, Plano Clark VL. 2011. Designing and Conducting Mixed Methods Research. Thousand Oaks, CA: Sage. 2nd ed.
18. Creswell JW, Tashakkori A. 2007. Developing publishable mixed methods manuscripts. J. Mix. Methods Res. 1:107-11
19. Cully JA, Armento ME,Mott J,Nadorff MR,Naik AD, et al. 2012. Brief cognitive behavioral therapy in primary care: A hybrid type 2 patient-randomized effectiveness-implementation trial. Implement. Sci. 7:64
20. Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. 2012. Effectiveness-implementation hybrid designs: Combining elements of clinical effectiveness and implementation research to enhance public health impact. Med. Care 50(3):217-26
21. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. 2009. Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implement Sci. 4:50
22. Damschroder LJ, Lowery JC. 2013. Evaluation of a large-scale weight management program using the consolidated framework for implementation research (CFIR). Implement. Sci. 8:51
23. Damschroder LJ, Moin T, Datta SK,Reardon CM, Steinle N, et al. 2015. Implementation and evaluation of the VA DPP clinical demonstration: Protocol for a multi-site non-randomized hybrid effectivenessimplementation type III trial. Implement. Sci. 10:68
24. Damschroder LJ, Reardon CM, Sperber N, Robinson CH, Fickel JJ, Oddone EZ. 2017. Implementation evaluation of the Telephone Lifestyle Coaching (TLC) program: Organizational factors associated with successful implementation. Transl. Behav.Med. 7:233-41
25. Dannifer R, Abrami A, Rapoport R, Sriphanlop P, Sacks R, Johns M. 2015. A mixed-methods evaluation of a SNAP-Ed farmers? market-based nutrition education program. J. Nutr. Educ. Behav. 47:516-25
26. Delbecq AL, Van de Ven AH, Gustafson DH. 1975. Group Techniques for Program Planning: A Guide to Nominal Group and Delphi Processes. Glenview, IL: Foresman
27. Duan N, Bhaumik DK, Palinkas LA, Hoagwood K. 2015. Optimal design and purposeful sampling: Complementary methodologies for implementation research. Admin. Policy Ment. Health Ment. Health Serv. Res. 42:524-32
28. Dy SM,Garg P,NybergD,Dawson PB, Pronovost PJ, et al. 2005. Critical pathway effectiveness: Assessing the impact of patient, hospital care, and pathway characteristics using qualitative comparative analysis. Health Serv. Res. 40(2):499-516
29. Elinder LS, Patterson E, Nyberg G, Norman A. 2018. A Healthy Start Plus for prevention of childhood overweight and obesity in disadvantaged areas through parental support in the school setting-study protocol for a parallel group cluster randomized trial. BMC Public Health 18:459
30. Fetters MD, Molina-Azorin JF, eds. 2018. Description. Journal of Mixed Methods Research. https://au.sagepub.com/en-gb/oce/journal-of-mixed-methods-research/journal201775#description
31. Ford EW, Duncan WJ, Ginter PM. 2005. Health departments? implementation of public health's core functions: An assessment of health impacts. Public Health 119:11-21
32. Forman J, Heisley M, Damschroder LJ, Kaselitz E, Kerr EA. 2017. Development and application of the RE-AIM QuEST mixed methods framework for program evaluation. Prev.Med. Rep. 6:322-28
33. Fox AB, Hamilton AB, Frayne SN, Wiltsey-Stirman S, Bean-Mayberry B, et al. 2016. Effectiveness of an evidence-based quality improvement approach to cultural competence training: The Veterans Affairs? "Caring forWomen Veterans" program. J. Contin. Educ. Health Prof. 36:96-103
34. Gale NK,Heath G,Cameron E, Rashid S,Redwood S. 2013. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med. Res.Methodol. 13:117
35. Glaser BG, Strauss AL. 1967. The Discovery of Grounded Theory: Strategies for Qualitative Research. New York: Aldine de Gruyter
36. Glasgow RE, Vogt SM, Bowles TM. 1999. Evaluating the public health impact of health promotion interventions: The RE-AIM framework. Am. J. Public Health 89(9):1322-27
37. Goepp JG, Meykler S,Mooney NE, Lyon C, Raso R, Julliard K. 2008. Provider insights about palliative care barriers and facilitators: Results of a rapid ethnographic assessment. Am. J.Hosp. Palliat. Care 25:309-14
38. Granholm E, Holden JL, Sommerfeld D, Rufener C, Perivoliotis D, et al. 2015. Enhancing assertive community treatment with cognitive behavioral social skills training for schizophrenia: Study protocol for a randomized controlled trial. Trials 16:438
39. Greene JC, Caracelli VJ, GrahamWF. 1989. Toward a conceptual framework for mixed-method evaluation designs. Educ. Eval. Policy Anal. 11(3):255-74
40. Hamilton AB. 2013. Qualitative methods in rapid turn-Around health services research. Seminar transcript, Spotlight onWomen's Health, VA Health Serv. Res. Dev., Dec. 11. https://www.hsrd.research.va.gov/for-researchers/cyber-seminars/archives/video-Archive.cfm?SessionID=780
41. Hamilton AB, Cohen AN, Glover DL, Whelan F, Chemerinski E, et al. 2013. Implementation of evidence-based employment services in specialty mental health. Health Serv. Res. 48:2224-44
42. Hanson RF, Schoenwald S, Saunders BE, Chapman J, Palinkas LA, et al. 2016. Testing the Community-Based Learning Collaborative (CBLC) implementation model: A study protocol. Int. J. Ment. Health Syst. 10:52
43. Henggler SW,Schoenwald SK,BorduinCM,RowlandMD,Cunningham PB. 2009. Multisystemic Therapy for Antisocial Behavior in Children and Adolescents. New York: Guilford Press. 2nd ed.
44. HsiehHF, Shannon SE. 2005. Three approaches to qualitative content analysis. Qual.Health Res. 15:1277-88
45. Jayawardena A,Wijayasinghe SR,Tennakoon D,Cook T, Morcuendo JA. 2013. Early effects of a ?train the trainer? approach on Ponseti method dissemination: A case study of Sri Lanka. Iowa Orthop. J. 33:153-60
46. Kahwati LC, Lewis MA, Kane H,Williams PA, Nerz P, et al. 2011. Best practices in the Veterans Health Administration's MOVE! weight management program.Am. J. Prev.Med. 41:457-64
47. Kane H, Hinnant L, Day K, Council M, Tzeng J, et al. 2017. Pathways to program success: A qualitative comparative analysis (QCA) of Communities Putting Prevention toWork case study programs. J. Public Health. Manag. Pract. 23:104-11
48. KaneH,Lewis MA,Williams PA, Kahwati LC.2014. Using qualitative comparative analysis to understand and quantify translation and implementation. Transl. Behav.Med. 4:201-8
49. Koenig CJ, Abraham T, Zamora KA,Hill C, Kelly PA, et al. 2016. Pre-implementation strategies to adapt and implement a veteran peer coaching intervention to improve mental health treatment engagement among rural veterans. J. Rural Health 32(4):418-28
50. Kozica SL, Lombard CB, Harrison CL, Teede HJ. 2016. Evaluation of a large healthy lifestyle program: Informing program implementation and scale-up in the prevention of obesity. Implement. Sci. 11:151
51. Lewis CC, Scott K,Marty CN,Marriott BR, Kroenke K, et al. 2015. Implementing measurement-based care (iMBC) for depression in community mental health: A dynamic cluster randomized trial study protocol. Implement. Sci. 10:127
52. Liddle HA. 2002. Multidimensional family therapy treatment for adolescent cannabis users: Vol. 5. Cannabis youth treatment series. Rep., Cent. Subst. Abuse Treat., Subst. Abuse Ment. Health Serv. Adm., Rockville, MD. https://files.eric.ed.gov/fulltext/ED478685.pdf
53. Lincoln YS, Guba EG. 1985. Naturalistic Inquiry. Beverly Hills, CA: Sage
54. Martinez JL, Duncan LR, Rivers SE, BertoliMC, Latimer-Cheung AE, Salovey P. 2017. Healthy Eating for Life English as a second language curriculum: Applying the RE-AIM framework to evaluate a nutrition education intervention targeting cancer risk reduction. Transl. Behav.Med. 7:657-66
55. Marx A. 2010. Crisp-set qualitative comparative analysis (csQCA) and model specification: Benchmarks for future csQCA applications. Int. J. Mult. Res. Approaches 4(2):138-58
56. Miles MB, Huberman AM. 1994. Qualitative Data Analysis: An Expanded Sourcebook. Thousand Oaks, CA: Sage. 2nd ed.
57. Moore GF, Audrey S, Barker M, Bond L, Bonell C, et al. 2015. Process evaluation of complex interventions: Medical Research Council guidance. BMJ 350:h1258
58. Morse JM. 2005. Evolving trends in qualitative research: Advances in mixed-method design. Qual. Health Res. 15(5):583-85
59. Morse JM, Niehaus L. 2009. MixedMethodDesign: Principles and Procedures.Walnut Creek,CA: Left Coast Press
60. O?Cathain A,Murphy E, Nicholl J. 2008.The quality ofmixedmethods studies in health services research. J. Health Serv. Res. Policy 13(2):92-98
61. Onwuegbuzie AJ, Poth C. 2016. Editors? afterword: Toward evidence-based guidelines for reviewing mixed methods research manuscripts submitted to journals. Int. J. Qual. Methods 15(1). https://doi. org/10.1177/1609406916628986
62. Palinkas LA. 2014. Qualitative and mixedmethods in mental health services and implementation research. J. Clin. Child Adolesc. Psychol. 43:851-61
63. Palinkas LA, Aarons GA, Horwitz SM, Chamberlain P, Hurlburt M, Landsverk J. 2011. Mixed method designs in implementation research. Adm. Policy Ment. Health 38:44-53
64. Palinkas LA, Campbell M, Saldana L. 2018. Agency leaders? assessments of feasibility and desirability of implementation of evidence-based practices in youth-serving organizations using the stages of implementation completion. Front. Public Health 6:161
65. Palinkas LA, Cooper BR. 2018. Mixed methods evaluation in dissemination and implementation science. In Dissemination and Implementation Research in Health: Translating Science to Practice, ed. RC Brownson, GA Colditz, EK Proctor, pp. 335-53. New York: Oxford Univ. Press. 2nd ed.
66. Palinkas LA, Darnell D, Zatzick D. 2017. Developing clinical ethnographic implementation methods for rapid assessments in acute care clinical trials. Paper presented at the 10th Conference on the Science of Dissemination and Implementation,Washington, DC, Dec. 5
67. Palinkas LA, Holloway IW, Rice E, Fuentes D,Wu Q, Chamberlain P. 2011. Social networks and implementation of evidence-based practices in public youth-serving systems: Amixedmethods study. Implement. Sci. 6:113
68. Palinkas LA, Horwitz SM, Green CA,Wisdom JP, Duan N, Hoagwood KE. 2015. Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Admin. Policy Ment. Health 42:533-44
69. Palinkas LA, Spear SE, Mendon SJ,Villamar J, Brown CH. 2018. Development of a system for measuring sustainment of prevention programs and initiatives. Implement. Sci. 13(Suppl. 3):A16 (Abstr.)
70. Palinkas LA, Spear SE, Mendon SJ, Villamar J, Valente T, et al. 2016. Measuring sustainment of prevention programs and initiatives: A study protocol. Implement. Sci. 11:95
71. Patton MQ. 2002. Qualitative Research and Evaluation Methods. Thousand Oaks, CA: Sage. 3rd ed.
72. Pelto PJ. 2015. What is so new about mixed methods? Qual. Health Res. 25(6):734-45
73. Pluye P, Hong QN. 2014. Combining the power of stories and the power of numbers: Mixed methods research and mixed studies reviews. Annu. Rev. Public Health 35:29-45
74. Powell BJ, Stanick CF,HalkoHM,Dorsey CN,Weiner BJ, et al. 2017.Toward criteria for pragmaticmeasurement in implementation research and practice: A stakeholder-driven approach using concept mapping. Implement. Sci. 12:118
75. Ragin CC. 1987. The Comparative Method:Moving Beyond Qualitative and Quantitative Strategies. Berkeley: Univ. Calif. Press
76. Ragin CC. 2009. Redesigning Social Inquiry: Fuzzy Sets and Beyond. Chicago: Univ. Chicago Press
77. Rihoux B, Ragin CC. 2008. Configurational Comparative Methods: Qualitative Comparative Analysis (QCA) and Related Techniques. Vol. 51. Thousand Oaks, CA: Sage
78. Rogers E, Fernandez S, Gillespie C, Smelson D, Hagedorn HJ, et al. 2013. Telephone care coordination for smokers in VA mental health clinics: Protocol for a hybrid type-2 effectiveness-implementation trial. Addict. Sci. Clin. Pract. 8:7
79. Rosas LG, Lv N, Xiao L, Lewis MA, Zavella P, et al. 2016. Evaluation of a culturally-Adapted lifestyle intervention to treat elevated cardiometabolic risk of Latino adults in primary care (Vida Sana): A randomized controlled trial. Contemp. Clin. Trials. 48:30-40
80. SandelowskiM. 2000. Combining qualitative and quantitative sampling, data collection, and analysis techniques in mixed-method studies. Res. Nurs. Health 23(3):246-55
81. Schneider CQ,Wagemann C. 2010. Standards of good practice in qualitative comparative analysis (QCA) and fuzzy-sets. Comp. Sociol. 9(3):397-418
82. Schwingel A, Gálvez P, Linares D, Sebastião E. 2017. Using a mixed-methods RE-AIM framework to evaluate community health programs for older Latinas. J. Aging Health 29(4):551-93
83. Schwitters A, Lederer P, Zilversmit L, Gudo PS, Ramiro I, et al. 2015. Barriers to health care in rural Mozambique: A rapid assessment of planned mobile health clinics for ART. Glob. Health Sci. Pract. 3:109-16
84. Scrimshaw SCM, Hurtado E. 1987. Rapid Assessment Procedures for Nutrition and Primary Health Care: Anthropological Approaches to Improving Programme Effectiveness. Los Angeles, CA: UCLA Lat. Am. Cent.
85. Shanks CB, Harden S. 2016. A reach, effectiveness, adoption, implementation,maintenance evaluation of weekend backpack food assistance programs. Am. J. Health Promot. 30(7):511-20
86. Stange KC, Crabtree BF, Miller WL. 2006. Publishing multimethod research. Ann. Fam. Med. 4:292-94
87. Swindle T, Johnson SL,Whiteside-Mansell L, Curran GM. 2017. A mixed methods protocol for developing and testing implementation strategies for evidence-based obesity prevention in childcare: A cluster randomized hybrid type III trial. Implement. Sci. 12:90
88. Teddlie C, Tashakkori A. 2003. Major issues and controversies in the use of mixed methods in the social and behavioral sciences. In Handbook of MixedMethods in the Social and Behavioral Sciences, ed.ATashakkori, C Teddlie, pp. 3-50. Thousand Oaks, CA: Sage
89. Thorpe KE, Zwarenstein M, Oxman AD, Treweek S, Furberg CD, et al. 2009. A pragmatic-explanatory continuum indicator summary (PRECIS): A tool to help trial designers. J. Clin. Epidemiol. 62:464-75
90. Thygeson NM, Solberg LL, Asche SE, Fontaine P, Pawlson LG, Scholle SH. 2012. Using fuzzy set qualitative comparative analysis (fs/QCA) to explore the relationship between medical "homeness" and quality. Health Serv. Res. 47(1 Pt. 1):22-45
91. TrochimWMK.1989.An introduction to conceptmapping for planning and evaluation.Eval. Prog. Plann. 12:1-16
92. Van der Kleij RM, Crone MR, Paulussen TG, Van de Gar VM, Reis R. 2015. A stitch in time saves nine? A repeated cross-sectional case study on the implementation of the intersectoral community approach Youth At a HealthyWeight. BMC Public Health 15:1032
93. Vindrola-Padros C, Vindrola-Padros B. 2018. Quick and dirty? A systematic review of the use of rapid ethnographies in healthcare organization and delivery. BMJ Qual. Saf. 27:321-30
94. Waltz TJ, Powell BJ,Matthieu MM, Damschroder LJ, ChinmanMJ, et al. 2015. Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: Results from the Expert Recommendations for ImplementingChange (ERIC) study. Implement.Sci. 10:109
95. Watts BV, Shiner B, Zubkoff L, Carpenter-Song E, Ronconi JM, Coldwell CM. 2014. Implementation of evidence-based psychotherapies for posttraumatic stress disorder in VA specialty clinics. Psychiatr. Serv. 65(5):648-53
96. WHO (World Health Organ WHO (World Health Organ.). 2012. Changing mindsets: Strategy on health policy and systems research. Rep., WHO, Geneva. http://www.who.int/alliance-hpsr/alliancehpsr-changingmindsets-strategyhpsr.pdf
97. Wright A, Sittig DF, Ash JS, Erikson JL, Hickman TT, et al. 2015. Lessons learned from implementing service-oriented clinical decision support at four sites: A qualitative study. Int. J. Med. Inform. 84:901-11
98. Zatzick D, Rivera F, Jurkovich G, Russo J, Trusz SG, et al. 2011. Enhancing the population impact of collaborative care interventions: Mixed method development and implementation of stepped care targeting posttraumatic stress disorder and related comorbidities after acute trauma. Gen. Hosp. Psychiatry 33:123-34
99. Zatzick DF,Russo J, DarnellD,ChambersDA, Palinkas LA, et al. 2016. An effectiveness-implementation hybrid trial study protocol targeting posttraumatic stress disorder and comorbidity. Implement. Sci. 11:58