Critical Appraisal of Mixed Methods Studies

Journal of Mixed Methods Research

Volumn 7, Issue 4, 2013, Pages 302-327

  Heyvaert, Mieke ^a   Hannes, Karin ^a   Maes, Bea ^a   Onghena, Patrick ^a  

^a UNIVERSITY OF LEUVEN   (Belgium)

Author keywords

critical appraisal; mixed methods research; quality assessment; quality evaluation; research synthesis; systematic review

Indexed keywords

EID: 84884361077     PISSN: 15586898     EISSN: 15586901     Source Type: Journal    
DOI: 10.1177/1558689813479449     Document Type: Article

References (90)

1. *Alborz A.,McNally R.Developing methods for systematic reviewing in health services delivery and organization: An example from a review of access to health care for people with learning disabilities. Part 2. Evaluation of the literature-A practical guide.Health Information and Libraries Journal. 2004;21:227-236.
2. Alise M. A.,Teddlie C.A continuation of the paradigm wars? Prevalence rates of methodological approaches across the social/behavioral sciences.Journal of Mixed Methods Research. 2010;4 (2): 103-126.
3. Biesta G. J. J.Sage handbook of mixed methods in social & behavioral research. 2010). Pragmatism and the philosophical foundations of mixed methods research. In TashakkoriA.TeddlieC. (Eds.), Sage handbook of mixed methods in social & behavioral research (2nd ed., pp. 95-118). Thousand Oaks, CA: Sage.Tashakkori A.Teddlie C., ed. Thousand Oaks, CA: Sage; 2010:95-118.
4. Bryman A.Paradigm peace and the implications for quality.International Journal of Social Research Methodology. 2006a;9:111-126.
5. Bryman A.Integrating quantitative and qualitative research: How is it done?.Qualitative Research. 2006b;6:97-113.
6. *Bryman A.,Becker S.,Sempik J.Quality criteria for quantitative, qualitative and mixed methods research: A view from social policy.International Journal of Social Research Methodology. 2008;11:261-276.
7. *Caracelli V.,Riggin L.Mixed-method evaluation: Developing quality criteria through concept mapping.Evaluation Practice. 1994;15:139-152.
8. Collins K. M. T.,Onwuegbuzie A. J.,Sutton I. L.A model incorporating the rationale and purpose for conducting mixed-methods research in special education and beyond.Learning Disabilities: A Contemporary Journal. 2006;4:67-100.
9. Cooper H. M.Research synthesis and meta-analysis: A step-by-step approach. 2010). Research synthesis and meta-analysis: A step-by-step approach (4th ed.). London, England: Sage.London, England: Sage; 2010:.
10. Cooper H. M.,Hedges L. V.The handbook of research synthesis. New York, NY: Russell Sage Foundation; 1994:.
11. The handbook of research synthesis and meta-analysis. 2009). The handbook of research synthesis and meta-analysis (2nd ed.). New York, NY: Russell Sage Foundation.Cooper H. M.Hedges L. V.Valentine J. C., ed. New York, NY: Russell Sage Foundation; 2009:.
12. Creswell J. W.Research design: Qualitative, quantitative, and mixed methods approaches. 2003). Research design: Qualitative, quantitative, and mixed methods approaches (2nd ed.). Thousand Oaks, CA: Sage.Thousand Oaks, CA: Sage; 2003:.
13. Creswell J. W.How mixed methods has developed. Keynote address for the Fourth Annual Mixed Methods Conference; 2008Fitzwilliam College, Cambridge University, England; 2008.
14. Creswell J. W.Mapping the field of mixed methods research.Journal of Mixed Methods Research. 2009;3 (2): 95-108.
15. Creswell J. W.Sage handbook of mixed methods in social & behavioral research. 2010). Mapping the developing landscape of mixed methods research. In TashakkoriA.TeddlieC. (Eds.), Sage handbook of mixed methods in social & behavioral research (2nd ed., pp. 45-68). Thousand Oaks, CA: Sage.Tashakkori A.Teddlie C., ed. Thousand Oaks, CA: Sage; 2010:45-68.
16. *Creswell J. W.,Plano Clark V. L.Designing and conducting mixed methods research. Thousand Oaks, CA: Sage; 2007:.
17. Creswell J. W.,Plano Clark V. L.Designing and conducting mixed methods research. 2011). Designing and conducting mixed methods research (2nd ed.). Thousand Oaks, CA: Sage.Thousand Oaks, CA: Sage; 2011:.
18. Day C.,Sammons P.,Gu Q.Combining qualitative and quantitative methodologies in research on teachers' lives, work, and effectiveness: From integration to synergy.Educational Researcher. 2008;37:330-342.
19. Dellinger A. B.Validity and the review of literature.Research in the Schools. 2005;12 (2): 41-54.
20. *Dellinger A. B.,Leech N. L.Toward a unified validation framework in mixed methods research.Journal of Mixed Methods Research. 2007;1 (4): 309-332.
21. *Dybå T.,Dingsøyr T.,Hanssen G. K.Applying systematic reviews to diverse study types: An experience report. Proceedings of the First International Symposium on Empirical Software Engineering and Measurement; 2007Washington, DC: IEEE Computer Society; 2007. 225.
22. Feilzer M. Y.Doing mixed methods research pragmatically: Implications for the rediscovery of pragmatism as a research paradigm.Journal of Mixed Methods Research. 2010;4 (1): 6-16.
23. Fidel R.Are we there yet? Mixed methods research in library and information science.Library & Information Science Research. 2008;30:265-272.
24. Gaber J.Meta-needs assessment.Evaluation and Program Planning. 2000;23:139-147.
25. Gliner J. A.,Morgan G. A.Research methods in applied settings: An integrated approach to design and analysis. Mahwah, NJ: Lawrence Erlbaum; 2000:.
26. *Greene J. C.Mixed methods in social inquiry. San Francisco, CA: Jossey-Bass; 2007:.
27. Greene J. C.Is mixed methods social inquiry a distinctive methodology?.Journal of Mixed Methods Research. 2008;2 (1): 7-22.
28. Greene J. C.,Benjamin L.,Goodyear L.The merits of mixing methods in evaluation.Evaluation. 2001;7:25-44.
29. Greene J. C.,Caracelli V. J.Handbook of mixed methods in social and behavioral research. Tashakkori A.Teddlie C., ed. Thousand Oaks, CA: Sage; 2003:91-110.
30. Greene J. C.,Caracelli V. J.,Graham W. F.Toward a conceptual framework for mixed-method evaluation designs.Educational Evaluation and Policy Analysis. 1989;11:255-274.
31. Greene J. C.,Hall J. N.Sage handbook of mixed methods in social & behavioral research. 2010). Dialectics and pragmatism: Being of consequence. In TashakkoriA.TeddlieC. (Eds.), Sage handbook of mixed methods in social & behavioral research (2nd ed., pp. 119-143). Thousand Oaks, CA: Sage.Tashakkori A.Teddlie C., ed. Thousand Oaks, CA: Sage; 2010:119-143.
32. Hall B.,Howard K.A synergistic approach: Conducting mixed methods research with typological and systemic design consideration.Journal of Mixed Methods Research. 2008;2 (3): 248-269.
33. Hannes K.,Lockwood C.Pragmatism as the philosophical underpinning of the Joanna Briggs meta-aggregative approach to qualitative evidence synthesis.Journal of Advanced Nursing. 2011;67:1632-1642.
34. Harden A.,Thomas J.Methodological issues in combining diverse study types in systematic reviews.International Journal of Social Research Methodology. 2005;8:257-271.
35. Harden A.,Thomas J.Sage handbook of mixed methods in social & behavioral research. 2010). Mixed methods and systematic reviews: Examples and emerging issues. In TashakkoriA.TeddlieC. (Eds.), Sage handbook of mixed methods in social & behavioral research (2nd ed., pp. 749-774). Thousand Oaks, CA: Sage.Tashakkori A.Teddlie C., ed. Thousand Oaks, CA: Sage; 2010:749-774.
36. Hart L. C.,Smith S. Z.,Swars S. L.,Smith M. E.An examination of research methods in mathematics education (1995-2005).Journal of Mixed Methods Research. 2009;3 (1): 26-41.
37. Heyvaert M.,Maes B.,Onghena P.Applying mixed methods research at the synthesis level: An overview.Research in the Schools. 2011;18 (1): 12-24.
38. Heyvaert M.,Maes B.,Onghena P.Mixed methods research synthesis: Definition, framework, and potential.Quality & Quantity. 2013;47:659-676.
39. HigginsJ. P. T.AltmanD. G. (2008). Assessing risk of bias in included studies. In HigginsJ. P. T.GreenS. (Eds.), Cochrane handbook for systematic reviews of interventions (Version 5.0.1, pp. 187-241). Retrieved from http://www.cochrane-handbook.org.
40. HigginsJ. P. T.GreenS. (Eds.). (2009). Cochrane handbook for systematic reviews of interventions (Version 5.0.2). Retrieved from http://www.cochrane-handbook.org.
41. Hurmerinta-Peltomaki L.,Nummela N.Mixed methods in international business research: A value-added perspective.Management International Review. 2006;46:439-459.
42. Hutchinson S. R.,Lovell C. D.A review of methodological characteristics of research published in key journals in higher education: Implications for graduate research teaching.Research in Higher Education. 2004;45:383-403.
43. International Centre for Allied Health Evidence, University of South Australia. (n.d.). Critical appraisal tools. Retrieved from http://www.unisa.edu.au/Research/Sansom-Institute-for-Health-Research/Research-at-the-Sansom/Research-Concentrations/Allied-Health-Evidence/Resources/CAT/.
44. Johnson R. B.,Onwuegbuzie A. J.Mixed methods research: A research paradigm whose time has come.Educational Researcher. 2004;33:14-26.
45. Johnson R. B.,Onwuegbuzie A. J.,Turner L. A.Toward a definition of mixed methods research.Journal of Mixed Methods Research. 2007;1 (2): 112-133.
46. Katrak P.,Bialocerkowski A. E.,Massy-Westropp N.,Kumar V. S. S.,Grimmer K. A.A systematic review of the content of critical appraisal tools.BMC Medical Research Methodology. 2004;4:22-33.
47. Khan K. S.,Riet G.,Popay J.,Nixon J.,Kleijnen J.Undertaking systematic reviews of research on effectiveness. 2001). Undertaking systematic reviews of research on effectiveness (Centre for Reviews and Dissemination Report No. 4, 2nd ed., pp. 1-20). York, England: University of York.York, England: University of York; 2001:1-20.
48. **Leech N. L.,Dellinger A. B.,Brannagan K. B.,Tanaka H.Evaluating mixed research studies: A mixed methods approach.Journal of Mixed Methods Research. 2010;4 (1): 17-31.
49. LettsL.WilkinsS.LawM.StewartD.BoschJ.WestmorlandM. (2007). Critical Review Form-Qualitative Studies (Version 2.0). Retrieved from http://www.srs-mcmaster.ca/Portals/20/pdf/ebp/qualreview_version2.0.pdf.
50. Lincoln Y. S.,Guba E. G.Naturalistic inquiry. Newbury Park, CA: Sage; 1985:.
51. Lincoln Y. S.,Guba E. G.Naturalistic evaluation. New directions for program evaluation, No. 90. Williams D. D., ed. San Francisco, CA: Jossey-Bass; 1986:78-84.
52. Major C. H.,Savin-Baden M.An introduction to qualitative research synthesis: Managing the information explosion in social science research. London, England: Routledge; 2010:.
53. Maxwell J. A.Understanding and validity in qualitative research.Harvard Educational Review. 1992;62:279-300.
54. Mertens D. M.Transformative mixed methods research.Qualitative Inquiry. 2010;16:469-474.
55. Mertens D. M.Transformative mixed methods: Addressing inequities.American Behavioral Scientist. 2012;56:802-813.
56. Mertens D. M.,Bledsoe K.,Sullivan M.,Wilson A.Sage handbook of mixed methods in social & behavioral research. 2010). Utilization of mixed methods for transformative purposes. In TashakkoriA.TeddlieC. (Eds.), Sage handbook of mixed methods in social & behavioral research (2nd ed., pp. 193-214). Thousand Oaks, CA: Sage.Tashakkori A.Teddlie C., ed. Thousand Oaks, CA: Sage; 2010:193-214.
57. Messick S.Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning.American Psychologist. 1995;50:741-749.
58. Moffatt S.,White M.,Mackintosh J.,Howel D.Using quantitative and qualitative data in health services research: What happens when mixed method findings conflict?.BMC Health Services Research. 2006;6:28-38.
59. Morgan D. L.Paradigms lost and pragmatism regained: Methodological implications of combining qualitative and quantitative methods.Journal of Mixed Methods Research. 2007;1 (1): 48-76.
60. Newman I.,Benz C. R.Qualitative-quantitative research methodology: Exploring the interactive continuum. Carbondale: Southern Illinois University Press; 1998:.
61. O'Cathain A.Handbook of mixed methods in social & behavioral research. 2010). Assessing the quality of mixed methods research: Towards a comprehensive framework. In TashakkoriA.TeddlieC. (Eds.), Handbook of mixed methods in social & behavioral research (2nd ed., pp. 531-555). Thousand Oaks, CA: Sage.Tashakkori A.Teddlie C., ed. Thousand Oaks, CA: Sage; 2010:531-555.
62. *O'Cathain A.,Murphy E.,Nicholl J.The quality of mixed methods studies in health services research.Journal of Health Services Research & Policy. 2008;13:92-98.
63. Onwuegbuzie A. J.,Collins K. M. T.,Leech N. L.,Dellinger A. B.,Jiao Q. G.The Research on Stress and Coping in Education Series: Vol. 5. Toward a broader understanding of stress and coping: Mixed methods approaches. Gates G. S.Gmelch W. H.Wolverton M.Collins K. M. T.Onwuegbuzie A. J.Jiao Q. G., ed. Charlotte, NC: Information Age; 2010:169-212.
64. *Onwuegbuzie A. J.,Johnson R. B.The validity issue in mixed research.Research in the Schools. 2006;13 (1): 48-63.
65. **Onwuegbuzie A. J.,Johnson R. B.,Collins K. M. T.Assessing legitimation in mixed research: A new framework.Quality & Quantity. 2011;45:1253-1271.
66. Onwuegbuzie A. J.,Leech N. L.Taking the "Q" out of research: Teaching research methodology courses without the divide between quantitative and qualitative paradigms.Quality & Quantity. 2005a;39:267-296.
67. Onwuegbuzie A. J.,Leech N. L.On becoming a pragmatic researcher: The importance of combining quantitative and qualitative research methodologies.International Journal of Social Research Methodology. 2005b;8:375-387.
68. Pawson R.,Greenhalgh T.,Harvey G.,Walshe K.Realist review: A new method of systematic review designed for complex policy interventions.Journal of Health Services Research & Policy. 2005; (10): 21-34.
69. Petticrew M.,Roberts H.Systematic reviews in the social sciences: A practical guide. Malden, MA: Blackwell; 2006:.
70. *Pluye P.,Gagnon M. P.,Griffiths F.,Johnson-Lafleur J.A proposal for concomitantly appraising qualitative, quantitative and mixed methods primary studies included in systematic mixed studies reviews. Paper presented at the 37th Annual Meeting of the North American Primary Care Research Group; 2009aMontreal, Quebec, Canada; 2009a.
71. **Pluye P.,Gagnon M. P.,Griffiths F.,Johnson-Lafleur J.A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative, and mixed methods primary studies in mixed studies reviews.International Journal of Nursing Studies. 2009b;46:529-546.
72. *Pluye P.,Grad R. M.,Dunikowski L.,Stephenson R.Impact of clinical information-retrieval technology on physicians: A literature review of quantitative, qualitative and mixed methods studies.International Journal of Medical Informatics. 2005;74:745-768.
73. Public Health Resource Unit. (2006a). Critical Appraisal Skills Programme (CASP). 10 questions to help you make sense of qualitative research. Retrieved from http://www.sph.nhs.uk/sph-files/casp-appraisal-tools/Qualitative%20Appraisal%20Tool.pdf.
74. Public Health Resource Unit. (2006b). Critical Appraisal Skills Programme (CASP). 11 questions to help you make sense of a case control study. Retrieved from http://www.sph.nhs.uk/sph-files/casp-appraisal-tools/Case%20Control%2011%20Questions.pdf.
75. Saini M.,Shlonsky A.Systematic synthesis of qualitative research. Oxford, England: Oxford University Press; 2012:.
76. *Sale J. E. M.,Brazil K.A strategy to identify critical appraisal criteria for primary mixed-method studies.Quality & Quantity. 2004;38:351-365.
77. Sandelowski M.,Voils C. I.,Barroso J.Defining and designing mixed research synthesis studies.Research in the Schools. 2006;13 (1): 29-40.
78. Sweetman D.,Badiee M.,Creswell J. W.Use of the transformative framework in mixed methods studies.Qualitative Inquiry. 2010;16:441-454.
79. Tashakkori A.,Creswell J. W.The new era of mixed methods.Journal of Mixed Methods Research. 2007;1 (1): 3-7.
80. Tashakkori A.,Teddlie C.Mixed methodology: Combining qualitative and quantitative approaches. Thousand Oaks, CA: Sage; 1998:.
81. Handbook of mixed methods in social & behavioral research. Tashakkori A.Teddlie C., ed. Thousand Oaks, CA: Sage; 2003a:.
82. Tashakkori A.,Teddlie C.Handbook of mixed methods in social & behavioral research. Tashakkori A.Teddlie C., ed. Thousand Oaks, CA: Sage; 2003b:671-701.
83. Tashakkori A.,Teddlie C.Validity issues in mixed methods research: Calling for an integrative framework. Paper presented at the annual meeting of the American Educational Research Association; 2006San Francisco, CA; 2006.
84. **Tashakkori A.,Teddlie C.Advances in mixed methods research: Theories and applications. Bergman M. M., ed. London, England: Sage; 2008:101-119.
85. **Teddlie C.,Tashakkori A.Handbook of mixed methods in social & behavioral research. Tashakkori A.Teddlie C., ed. Thousand Oaks, CA: Sage; 2003:3-50.
86. *Teddlie C.,Tashakkori A.Foundations of mixed methods research: Integrating quantitative and qualitative approaches in the social and behavioral sciences. Thousand Oaks, CA: Sage; 2009:.
87. Truscott D.,Swars S.,Smith S.,Thornton-Reid F.,Zhao Y.,Dooley C.A cross-disciplinary examination of the prevalence of mixed methods in educational research: 1995-2005.International Journal of Social Research Methodology. 2010;13:317-328.
88. Whittemore R.,Chase S. K.,Mandle C. L.Validity in qualitative research.Qualitative Health Research. 2001;11:522-537.
89. Whittemore R.,Knafl K.The integrative review: Updated methodology.Journal of Advanced Nursing. 2005;52:546-553.
90. Young J. M.,Solomon M. J.How to critically appraise an article.Nature Clinical Practice Gastroenterology & Hepatology. 2009;6:82-91.