Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT)

Journal of Evaluation in Clinical Practice

Volumn 24, Issue 3, 2018, Pages 459-467

  Hong, Quan Nha ^a   Gonzalez Reyes, Araceli ^a   Pluye, Pierre ^a  

^a MCGILL UNIVERSITY   (Canada)

Author keywords

mixed studies reviews; qualitative research; quality appraisal; systematic review; usability; usefulness; utility

Indexed keywords

ARTICLE; CONTROLLED STUDY; DESCRIPTIVE RESEARCH; FEMALE; HUMAN; INFORMATION PROCESSING; INTERVENTION STUDY; MALE; MIXED METHOD STUDY; MIXED METHODS APPRAISAL TOOL; PRIORITY JOURNAL; QUALITATIVE ANALYSIS; QUANTITATIVE ANALYSIS; QUESTIONNAIRE; SEMI STRUCTURED INTERVIEW; STATISTICAL ANALYSIS; THEMATIC ANALYSIS; INTERNATIONAL COOPERATION; INTERVIEW; METHODOLOGY; QUALITATIVE RESEARCH; STANDARDS; TOTAL QUALITY MANAGEMENT;

FEMALE; HUMANS; INTERNATIONALITY; INTERVIEWS AS TOPIC; MALE; QUALITATIVE RESEARCH; QUALITY IMPROVEMENT; RESEARCH DESIGN;

EID: 85042174152     PISSN: 13561294     EISSN: 13652753     Source Type: Journal    
DOI: 10.1111/jep.12884     Document Type: Article

References (42)

1. Heyvaert M, Maes B, Onghena P. Mixed methods research synthesis: definition, framework, and potential. Qual Quant. 2013;47(2):659-676. https://doi.org/10.1007/s11135-011-9538-6
2. Pluye P, Hong QN. Combining the power of stories and the power of numbers: mixed methods research and mixed studies reviews. Annu Rev Public Health. 2014;35(1):29-45. Available at: https://doi.org/10.1146/annurev-publhealth-032013-182440
3. Dixon-Woods M, Agarwal S, Jones D, Young B, Sutton A. Integrative approaches to qualitative and quantitative evidence. London, UK: Health Development Agency; 2004.
4. Pluye P, Gagnon MP, Griffiths F, Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in mixed studies reviews. Int J Nurs Stud. 2009;46(4):529-546. Available at: https://doi.org/10.1016/j.ijnurstu.2009.01.009
5. Heyvaert M, Hannes K, Onghena P. Using Mixed Methods Research Synthesis for Literature Reviews: The Mixed Methods Research Synthesis Approach. Thousand Oaks, CA: Sage Publications; 2016.
6. Whittemore R, Knafl K. The integrative review: updated methodology. J Adv Nurs. 2005;52(5):546-553. https://doi.org/10.1111/j.1365-2648.2005.03621.x
7. Hong QN, Pluye P, Bujold M, Wassef M. Convergent and sequential synthesis designs: implications for conducting and reporting systematic reviews of qualitative and quantitative evidence. Syst Rev. 2017;6(1):61. https://doi.org/10.1186/s13643-017-0454-2
8. Harden A, Gough D. Quality and relevance appraisal. In: Gough D, Oliver S, Thomas J, eds. An Introduction to Systematic Reviews. London, UK: Sage Publications; 2012:153-178.
9. Burls A. What is Critical Appraisal? Newmarket, UK: Hayward Medical Communications; 2009.
10. Petticrew M, Roberts H. How to appraise the studies: an introduction to assessing study quality. In: Petticrew M, Roberts H, eds. Systematic Reviews in the Social Sciences: A Practical Guide Padstow. UK: Wiley-Blackwell; 2006:125-163.
11. West SL, King V, Carey TS, et al. Systems to Rate the Strength of Scientific Evidence. Rockville, MD: Agency for Healthcare Research and Quality; 2002.
12. Bai A, Shukla VK, Bak G, Wells G. Quality Assessment Tools Project Report. Ottawa: Canadian Agency for Drugs and Technologies in Health; 2012.
13. Deeks JJ, Dinnes J, D'Amico R, et al. Evaluating non-randomised intervention studies. Health Technol Assess. 2003;7(27):iii-iix. https://doi.org/10.3310/hta7270
14. Heyvaert M, Hannes K, Maes B, Onghena P. Critical appraisal of mixed methods studies. J Mix Methods Res. 2013;7(4):302-327. https://doi.org/10.1177/1558689813479449
15. Santiago-Delefosse M, Gavin A, Bruchez C, Roux P, Stephen S. Quality of qualitative research in the health sciences: analysis of the common criteria present in 58 assessment guidelines by expert users. Soc Sci Med. 2016;148:142-151. https://doi.org/10.1016/j.socscimed.2015.11.007
16. Katrak P, Bialocerkowski AE, Massy-Westropp N, Kumar S, Grimmer KA. A systematic review of the content of critical appraisal tools. BMC Med Res Methodol. 2004;4(22). https://doi.org/10.1186/1471-2288-4-22
17. Crowe M, Sheppard L. A review of critical appraisal tools show they lack rigor: alternative tool structure is proposed. J Clin Epidemiol. 2011;64(1):79-89. https://doi.org/10.1016/j.jclinepi.2010.02.008
18. Sanderson S, Tatt ID, Higgins JP, Sanderson S, Tatt ID, Higgins JPT. Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int J Epidemiol. 2007;36(3):666-676. https://doi.org/10.1093/ije/dym018
19. Whiting P, Wolff R, Mallett S, Simera I, Savović J. A proposed framework for developing quality assessment tools. Syst Rev. 2017;6(1):204. https://doi.org/10.1186/s13643-017-0604-6
20. Pace R, Pluye P, Bartlett G, et al. Testing the reliability and efficiency of the pilot Mixed Methods Appraisal Tool (MMAT) for systematic mixed studies review. Int J Nurs Stud. 2012;49(1):47-53. https://doi.org/10.1016/j.ijnurstu.2011.07.002
21. Souto RQ, Khanassov V, Hong QN, Bush PL, Vedel I, Pluye P. Systematic mixed studies reviews: updating results on the reliability and efficiency of the Mixed Methods Appraisal Tool. Int J Nurs Stud. 2015;52(1):500-501. https://doi.org/10.1016/j.ijnurstu.2014.08.010
22. Pluye P, Robert E, Cargo M, et al. Proposal: A mixed methods appraisal tool for systematic mixed studies reviews. Montreal, CA: Department of Family Medicine, McGill University. http://mixedmethodsappraisaltoolpublic.pbworks.com. Archived by WebCite® at http://www.webcitation.org/5tTRTc9yJ. Accessed month date, 2011.
23. Sandelowski M. Focus on research methods-whatever happened to qualitative description? Res Nurs Health. 2000;23(4):334-340. https://doi.org/10.1002/1098-240X(200008)23:4<334::AID-NUR9>3.0.CO;2-G
24. Sandelowski M. What's in a name? Qualitative description revisited. Res Nurs Health. 2010;33(1):77-84. https://doi.org/10.1002/nur.20362
25. Neergaard MA, Olesen F, Andersen RS, Sondergaard J. Qualitative description—the poor cousin of health research? BMC Med Res Methodol. 2009;9(1):52
26. Porta MS, Greenland S, Hernán M, dos Santos SI, Last JM. A Dictionary of Epidemiology. New York: Oxford University Press; 2014.
27. Fereday J, Muir-Cochrane E. Demonstrating rigor using thematic analysis: a hybrid approach of inductive and deductive coding and theme development. Int J Qual Methods. 2006;5(1):80-92.
28. Nielsen J. Usability Engineering. San Francisco, CA: Morgan Kaufmann; 1994.
29. Khorsan R, Crawford C. How to assess the external validity and model validity of therapeutic trials: a conceptual approach to systematic review methodology. Evid Based Complement Alternat Med. 2014;2014:694804-694812. https://doi.org/10.1155/2014/694804
30. Schmuckler MA. What is ecological validity? A dimensional analysis. Inf Dent. 2001;2(4):419-436. https://doi.org/10.1207/S15327078IN0204_02
31. Hartling L, Bond K, Santaguida PL, Viswanathan M, Dryden DM. Testing a tool for the classification of study designs in systematic reviews of interventions and exposures showed moderate reliability and low accuracy. J Clin Epidemiol. 2011;64(8):861-871. https://doi.org/10.1016/j.jclinepi.2011.01.010
32. Seo H-J, Kim SY, Lee YJ, et al. A newly developed tool for classifying study designs in systematic reviews of interventions and exposures showed substantial reliability and validity. J Clin Epidemiol. 2016;70:200-205. https://doi.org/10.1016/j.jclinepi.2015.09.013
33. Simera I, Moher D, Hirst A, Hoey J, Schulz KF, Altman DG. Transparent and accurate reporting increases reliability, utility, and impact of your research: reporting guidelines and the EQUATOR Network. BMC Med. 2010;8(1). https://doi.org/10.1186/1741-7015-8-24
34. Mhaskar R, Djulbegovic B, Magazin A, Soares HP, Kumar A. Published methodological quality of randomized controlled trials does not reflect the actual quality assessed in protocols. J Clin Epidemiol. 2012;65(6):602-609. https://doi.org/10.1016/j.jclinepi.2011.10.016
35. Sandelowski M, Barroso J. Reading qualitative studies. Int J Qual Methods. 2002;1(1):74-108. https://doi.org/10.1177/160940690200100107
36. Higgins JP, Green S. Cochrane Handbook for Systematic Reviews of Interventions. Wiley Online Library: Chichester, UK; 2008.
37. Faggion CM Jr. Risk of bias assessment should not go beyond reporting assessment. J Clin Epidemiol. 2016;72:126-127. https://doi.org/10.1016/j.jclinepi.2015.11.014
38. Carroll C, Booth A, Lloyd-Jones M. Should we exclude inadequately reported studies from qualitative systematic reviews? An evaluation of sensitivity analyses in two case study reviews. Qual Health Res. 2012;C22(10):1425-1434. https://doi.org/10.1177/1049732312452937
39. Verhage A, Boels D. Critical appraisal of mixed methods research studies in a systematic scoping review on plural policing: assessing the impact of excluding inadequately reported studies by means of a sensitivity analysis. Qual Quant. 2016;51(4):1449-1468. https://doi.org/10.1007/s11135-016-0345-y
40. Haynes SN, Richard D, Kubany ES. Content validity in psychological assessment: a functional approach to concepts and methods. Psychol Assess. 1995;7(3):238-247.
41. Francis JJ, Johnston M, Robertson C, et al. What is an adequate sample size? Operationalising data saturation for theory-based interview studies. Psychol Health. 2010;25(10):1229-1245. https://doi.org/10.1080/08870440903194015
42. Olsson C, Ringner A, Borglin G. Including systematic reviews in PhD programmes and candidatures in nursing—‘Hobson's choice’? Nurse Educ Pract. 2014;14(2):102-105. https://doi.org/10.1016/j.nepr.2014.01.005