Panning for the gold in health research: Incorporating studies’ methodological quality in meta-analysis

Psychology and Health

Volumn 30, Issue 1, 2015, Pages 135-152

  Johnson, Blair T ^a   Low, Robert E ^a   MacDonald, Hayley V ^a  

^a UNIVERSITY OF CONNECTICUT   (United States)

Author keywords

Cochrane Collaboration; data quality; health promotion; meta analysis; meta regression; methodological studies; scoring methods

Indexed keywords

HEALTH PROMOTION; HUMAN; MEDICAL RESEARCH; META ANALYSIS (TOPIC); METHODOLOGY; PROCEDURES; THEORETICAL MODEL;

BIOMEDICAL RESEARCH; HEALTH PROMOTION; HUMANS; META-ANALYSIS AS TOPIC; MODELS, THEORETICAL; RESEARCH DESIGN;

EID: 84918535527     PISSN: 08870446     EISSN: 14768321     Source Type: Journal    
DOI: 10.1080/08870446.2014.953533     Document Type: Article

References (66)

1. Abraham, C., Johnson, B. T., de Bruin, M., & Luszczynska, A. (2014). Enhancing reporting of behavior change intervention evaluations. Journal of Acquired Immune Deficiency Syndromes, 66, S293–S299.10.1097/QAI.0000000000000231
2. Ahn, S., & Becker, B. J. (2011). Incorporating quality scores in meta-analysis. Journal of Educational and Behavioral Statistics, 36, 555–585.10.3102/1076998610393968
3. Allen, I. E., & Olkin, I. (1999). Estimating time to conduct a meta-analysis from number of citations retrieved. The Journal of the American Medical Association, 282, 634–635.10.1001/jama.282.7.634
4. Ayling, K., Brierley, S., Johnson, B., Heller, S., & Eiser, C. (2015). How standard is standard care? Exploring control group outcomes in behaviour change interventions for young people with type 1 diabetes. Psychology & Health, 30(1), 85–103.
5. Bishop, F. L., Fenge-Davies, A. L., Kirby, S., & Geraghty, A. W. A. (2015). Context effects and behaviour change techniques in randomized trials: A systematic review using the example of trials to increase adherence to physical activity in musculoskeletal pain. Psychology & Health, 30(1), 104–121.
6. Bland, J. M., & Altman, D. G. (1995). Multiple significance tests: The Bonferroni method. British Medical Journal, 310, 170.10.1136/bmj.310.6973.170
7. Borenstein, M. (2013). Comprehensive meta-analysis version 3. Englewood Cliffs, NJ: Biostat.
8. Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2011). Introduction to meta-analysis. New York, NY: Wiley.
9. Brown, J. C., Huedo-Medina, T. B., Pescatello, L. S., Pescatello, S. M., Ferrer, R. A., & Johnson, B. T. (2011). Efficacy of exercise interventions in modulating cancer-related fatigue among adult cancer survivors: A meta-analysis. Cancer Epidemiology, Biomarkers and Prevention, 20, 123–133.10.1158/1055-9965.EPI-10-0988
10. Brown, J. C., Huedo-Medina, T. B., Pescatello, L. S., Ryan, S. M., Pescatello, S. M., Moker, E., … Johnson, B. T. (2012). The efficacy of exercise in reducing depressive symptoms among cancer survivors: A meta-analysis. PLoS ONE, 7, e30955.
11. Card, N. A. (2012). Applied meta-analysis for social science research. New York, NY: Guilford.
12. Cochrane Collaboration. (2008). Review manager (revman), version 5.0. Copenhagen: Author.
13. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
14. Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155–159.10.1037/0033-2909.112.1.155
15. Cook, T. D., & Campbell, D. T. (1976). The design and conduct of quasi-experiments and true experiments in field settings. In M. D. Dunnette (Ed.), Handbook of industrial and organizational psychology (pp. 223–326). Chicago, IL: Rand McNally.
16. Conn, V. S., Hafdahl, A. R., Moore, S. M., Nielsen, P. J., & Brown, L. M. (2009). Meta-analysis of interventions to increase physical activity among cardiac subjects. International Journal of Cardiology, 133, 307–320.10.1016/j.ijcard.2008.03.052
17. Conn, V. S., & Rantz, M. J. (2003). Research methods: Managing primary study quality in meta-analyses. Research in Nursing & Health, 26, 322–333.
18. Coyne, J. C., Thombs, B. D., & Hagedoorn, M. (2010). Ain’t necessarily so: Review and critique of recent meta-analyses of behavioral medicine interventions in health psychology. Health Psychology, 29, 107–116.10.1037/a0017633
19. Crutzen, R., Viechtbauer, W., Spigt, M., & Kotz, D. (2015). Differential attrition in health behaviour change trials: A systematic review and meta-analysis. Psychology & Health, 30(1), 122–134.
20. Deeks, J. J., Dinnes, J., D’amico, R., Sowden, A., Sakarovitch, C., Song, F., … Altman, D. (2003). Evaluating non-randomised intervention studies. Health Technology Assessment, 7(27), 1–179.
21. Detsky, A. S., Naylor, C. D., O’Rourke, K., McGeer, A. J., & L’Abbé, K. A. (1992). Incorporating variations in the quality of individual randomized trials into meta-analysis. Journal of Clinical Epidemiology, 45, 255–265.10.1016/0895-4356(92)90085-2
22. de Bruin, M., McCambridge, J., & Prins, J. M. (2015). Reducing the risk of bias in health behaviour change trials: Improving trial design, reporting, or bias assessment criteria? A review and case study. Psychology & Health, 30(1), 8–34.
23. Downs, S. H., & Black, N. (1998). The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. Journal of Epidemiology and Community Health, 52, 377–384.10.1136/jech.52.6.377
24. Evers, S. M. A. A., Hiligsmann, M., & Adarkwah, C. C. (2015). Risk of bias in trial-based economic evaluations: Identification of sources and bias-reducing strategies. Psychology & Health, 30(1), 52–71.
25. Ferrer, R. A., Huedo-Medina, T. B., Johnson, B. T., Ryan, S. M., & Pescatello, L. S. (2011). Exercise interventions for cancer survivors: A meta-analysis of quality of life outcomes. Annals of Behavioral Medicine, 41, 32–47.10.1007/s12160-010-9225-1
26. Fisher, J. D., & Fisher, W. A. (1992). Changing AIDS-risk behavior. Psychological Bulletin, 111, 455–474.10.1037/0033-2909.111.3.455
27. Harbord, R. M., & Higgins, J. P. T. (2008). Meta-regression. Stata Journal, 8, 493–519.
28. Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.
29. Hedges, L. V., & Pigott, T. D. (2004). The power of statistical tests for moderators in meta-analysis. Psychological Methods, 9, 426.10.1037/1082-989X.9.4.426
30. Hedges, L. V., Pustejovsky, J. E., & Shadish, W. R. (2012). A standardized mean difference effect size for single case designs. Research Synthesis Methods, 3, 224–239.10.1002/jrsm.v3.3
31. Heiwe, S., & Jacobson, S. H. (2011). Exercise training for adults with chronic kidney disease. Cochrane Database of Systematic Reviews (10), CD003236.
32. Higgins, J. P. T., & Green, S. (Eds.). (2008). Cochrane handbook for systematic reviews of interventions (Vol. 5). West Sussex: Wiley Online Library.
33. Higgins, J. P. T., & Thompson, S. G. (2004). Controlling the risk of spurious findings from meta-regression. Statistics in Medicine, 23, 1663–1682.10.1002/(ISSN)1097-0258
34. Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J. M., Gavaghan, D. J., & McQuay, H. J. (1996). Assessing the quality of reports of randomized clinical trials: Is blinding necessary? Controlled Clinical Trials, 17, 1–12.10.1016/0197-2456(95)00134-4
35. Janssen, V., Gucht, V. D., Dusseldorp, E., & Maes, S. (2013). Lifestyle modification programmes for patients with coronary heart disease: A systematic review and meta-analysis of randomized controlled trials. European Journal of Preventive Cardiology, 20, 620–640.10.1177/2047487312462824
36. Johnson, B. T., Dove, N. L., & Boynton, M. H. (2011). On being basic and applied at the same time: Intersections between social and health psychology. In M. Mark, S. Donaldson, & B. A. Campbell (Eds.), The intersection of social psychology and program evaluation (pp. 321–341). New York, NY: Guilford.
37. Johnson, B. T., Low, R. E., & LaCroix, J. M. (2012). Systematic reviews to support evidence-based medicine (2nd edition) by Khalid Khan, Regina Kunz, Jos Kleijnen and Gerd Antes: A review. Research Synthesis Methods, 4, 102–108.
38. Johnson, B. T., MacDonald, H. V., Bruneau, M. L., Jr, Goldsby, T. U., Brown, J. C., Huedo-Medina, T. B., & Pescatello, L. S. (2014). Methodological quality of meta-analyses on the blood pressure response to exercise: A review. Journal of Hypertension, 32, 706–723.10.1097/HJH.0000000000000097
39. Johnson, B. T., Michie, S., & Snyder, L. B. (2014). Effects of behavioral intervention content on HIV prevention outcomes: A meta-review of meta-analyses. Journal of Acquired Immune Deficiency Syndromes, 66, S259–S270.10.1097/QAI.0000000000000235
40. Johnson, B. T., Redding, C. A., DiClemente, R. J., Mustanski, B. S., Dodge, B., Sheeran, P., … Fishbein, M. (2010). A network-individual-resource model for HIV prevention. AIDS and Behavior, 14, 204–221.10.1007/s10461-010-9803-z
41. Johnson, B. T., Scott-Sheldon, L. A. J., Huedo-Medina, T. B., & Carey, M. P. (2011). Interventions to reduce sexual risk for human immunodeficiency virus in adolescents: A meta-analysis of trials, 1985–2008. Archives of Pediatric and Adolescent Medicine, 165, 77–84.
42. Johnson, B. T., & Turco, R. M. (1992). The value of goodness-of-fit indices in meta-analysis: A comment on Hall and Rosenthal. Communications Monographs, 59, 388–396.10.1080/03637759209376279
43. Kangas, M., Bovbjerg, D. H., & Montgomery, G. H. (2008). Cancer-related fatigue: A systematic and meta-analytic review of non-pharmacological therapies for cancer patients. Psychological Bulletin, 134, 700–741.10.1037/a0012825
44. Khan, K., Kunz, R., Kleijnen, J., & Antes, G. (2011). Systematic reviews to support evidence-based medicine (2nd ed.). London: CRC Press.
45. Lau, J., Ioannidis, J. P. A., Terrin, N., Schmid, C. H., & Olkin, I. (2006). The case of the misleading funnel plot. British Medial Journal, 333, 597–600.10.1136/bmj.333.7568.597
46. Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis (Vol. 49). Thousand Oaks, CA: Sage.
47. Luepker, R., Johnson, S., Breslow, L., Chobanian, A., Davis, C., Duling, B., … McBride, P. (1996). Physical activity and cardiovascular health. Journal of the American Medical Association, 276, 241–246.
48. McCambridge, J. (2015). From question-behaviour effects in trials to the social psychology of research participation. Psychology & Health, 30(1), 72–84.
49. Michie, S., Johnson, B. T., & Johnston, L. (in press). Advancing cumulative evidence on behaviour change techniques and interventions: A comment on Peters, de Bruin, and Crutzen. Health Psychology Review.
50. Michie, S., Richardson, M., Johnston, M., Abraham, C., Francis, J., Hardeman, W., … Wood, C. E. (2013). The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: Building an international consensus for the reporting of behavior change interventions. Annals of Behavioral Medicine, 46, 81–95.10.1007/s12160-013-9486-6
51. Miller, W. R., Brown, J. M., Simpson, T. L., Handmaker, N. S., Bien, T. H., Luckie, L. F., … Tonigan, J. S. (1995). A methodological analysis of the alcohol treatment outcome literature. In R. K. Hester & W. R. Miller (Eds.), Handbook of alcoholism treatment approaches: Effective alternatives (2nd ed., pp. 12–24). Boston, MA: Allyn & Bacon.
52. O’Connor, D., Green, S., & Higgins, J. P. T. (2008). Defining the review question and developing criteria for including studies. In J. P. T. Higgins & S. Green (Eds.), Cochrane handbook for systematic reviews of interventions: Cochrane book series (pp. 83–94). Chichester: Wiley-Blackwell.
53. Pescatello, L. S., Arena, R., Riebe, D., Thompson, P. D. & Williams, L. (2014). ACSM’s guidelines for exercise testing and prescription. Baltimore, MD: Wolters Kluwer Health/Lippincott Williams & Wilkins.
54. Rothman, A. J., Hertel, A. W., Baldwin, A. S., & Bartels, R. (2008). Integrating theory and practice: Understanding the determinants of health behavior change. In J. Shah & W. Gardner (Eds.), Handbook of motivation science (pp. 494–507). New York, NY: Guilford.
55. Rubin, D. B. (1992). Meta-analysis: Literature synthesis or effect-size surface estimation? Journal of Educational and Behavioral Statistics, 17, 363–374.
56. Schunemann, H. J., Oxman, A. D., Vist, G. E., Higgins, J. P. T., Deeks, J. J., & Glasziou, P. (2011). Interpreting results and drawing conclusions. In Cochrane Handbook for systematic reviews of interventions (pp. 359–388). West Sussex: Wiley-Blackwell.
57. Shadish, W. R. (1989). The perception and evaluation of quality in science. In B. Gholson, W. R. Shadish, R. A. Neimeyer, & A. C. Houts (Eds.), Psychology of science: Contributions to metascience (pp. 383–426). New York, NY: Cambridge University Press.10.1017/CBO9781139173667
58. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton, Mifflin and Company.
59. Shea, B. J., Grimshaw, J. M., Wells, G. A., Boers, M., Andersson, N., Hamel, C., … Bouter, L. M. (2007). Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews. BMC Medical Research Methodology, 7, 10.10.1186/1471-2288-7-10
60. Slavin, R. E. (1986). Best-evidence synthesis: An alternative to meta-analytic and traditional reviews. Educational Researcher, 15, 5–11.10.3102/0013189X015009005
61. Tarquinio, C., Kivits, J., Minary, L., Coste, J., & Alla, F. (2015). Evaluating complex interventions: Perspectives and issues for health behaviour change interventions. Psychology & Health, 30(1), 35–51.
62. Thompson, S. G., & Higgins, J. (2002). How should meta-regression analyses be undertaken and interpreted? Statistics in Medicine, 21, 1559–1573.10.1002/(ISSN)1097-0258
63. Valentine, J. C. (2009). Judging the quality of primary research. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), Handbook of research synthesis and meta-analysis (2nd ed., pp. 129–146). New York, NY: Russell Sage Foundation.
64. Valentine, J. C., & Cooper, H. (2008). A systematic and transparent approach for assessing the methodological quality of intervention effectiveness research: The Study Design and Implementation Assessment Device (Study DIAD). Psychological Methods, 13, 130–149.10.1037/1082-989X.13.2.130
65. Verhagen, A. P., de Vet, H. C., de Bie, R. A., Kessels, A. G., Boers, M., Bouter, L. M., & Knipschild, P. G. (1998). The Delphi list: A criteria list for quality assessment of randomized clinical trials for conducting systematic reviews developed by Delphi consensus. Journal of Clinical Epidemiology, 51, 1235–1241.10.1016/S0895-4356(98)00131-0
66. Wells, G. A., Shea, B., O’Connell, D., Peterson, J., Welch, V., Losos, M., & Tugwell, P. (2000). The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. Retrieved from http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp