Charting the landscape of graphical displays for meta-analysis and systematic reviews: A comprehensive review, taxonomy, and feature analysis

BMC Medical Research Methodology

Volumn 20, Issue 1, 2020, Pages

  Kossmeier, Michael ^a   Tran, Ulrich S ^a   Voracek, Martin ^a  

^a UNIVERSITY OF VIENNA   (Austria)

Author keywords

Data visualization; Forest plot; Funnel plot; Galbraith plot; Graphical display; L'Abb plot; Meta analysis; Network meta analysis; Research synthesis; Systematic reviews

Indexed keywords

COMPUTER GRAPHICS; HUMAN; INFORMATION RETRIEVAL; META ANALYSIS (TOPIC); METHODOLOGY; PROCEDURES; PUBLICATION; REPRODUCIBILITY;

COMPUTER GRAPHICS; HUMANS; INFORMATION STORAGE AND RETRIEVAL; META-ANALYSIS AS TOPIC; PUBLICATIONS; REPRODUCIBILITY OF RESULTS; RESEARCH DESIGN; SYSTEMATIC REVIEWS AS TOPIC;

EID: 85079082773     PISSN: None     EISSN: 14712288     Source Type: Journal    
DOI: 10.1186/s12874-020-0911-9     Document Type: Review

References (135)

1. Tukey JW. Exploratory data analysis. Reading: Addison-Wesley; 1977.
2. Tufte ER. The visual display of quantitative information. 2nd ed. Cheshire: Graphics Press; 2001.
3. Krause A, O'Connell M. A picture is worth a thousand tables: graphics in life sciences. New York: Springer; 2012.
4. Chen C-H, Härdle WK, Unwin A. Handbook of data visualization. Berlin: Springer; 2008.
5. Glass GV. Primary, secondary, and meta-analysis of research. Educ Res. 1976;5:3-8.
6. Glass GV. Integrating findings: the meta-analysis of research. Rev Educ Res. 1977;5:351-79.
7. Light RJ, Singer JD, Willet JB. The visual presentation and interpretation of meta-analyses. In: Cooper H, Hedges LV, editors. The handbook of research synthesis. New York: Russell Sage; 1994. p. 439-53.
8. DuMouchel W, Normand S-L. Computer-modeling and graphical strategies for meta-analysis. In: Stangl DK, Berry DA, editors. Meta-analysis in medicine and health policy. New York: Marcel Dekker; 2000. p. 127-78.
9. Borman GD, Grigg JA. Visual and narrative interpretation. In: Cooper H, Hedges LV, Valentine JC, editors. The handbook of research synthesis and meta-analysis. 2nd ed. New York: Russell Sage; 2009. p. 497-519.
10. Bax L, Ikeda N, Fukui N, Yaju Y, Tsuruta H, Moons KG. More than numbers: the power of graphs in meta-analysis. Am J Epidemiol. 2009;169:249-55.
11. Anzures-Cabrera J, Higgins JP. Graphical displays for meta-analysis: an overview with suggestions for practice. Res Synth Methods. 2010;1:66-80.
12. Kiran A, Crespillo AP, Rahimi K. Graphics and statistics for cardiology: data visualisation for meta-analysis. Heart. 2017;103:19-23.
13. Salanti G, Ades AE, Ioannidis JP. Graphical methods and numerical summaries for presenting results from multiple-treatment meta-analysis: an overview and tutorial. J Clin Epidemiol. 2011;64:163-71.
14. Chaimani A, Higgins JP, Mavridis D, Spyridonos P, Salanti G. Graphical tools for network meta-analysis in STATA. PLoS One. 2013;8:e76654.
15. Borenstein M, Hedges LV, Higgins JP, Rothstein HR. Introduction to meta-analysis. Chichester: Wiley; 2009.
16. The Cochrane Collaboration. Review Manager (RevMan) (Version 5.3) [Computer program]. Copenhagen: The Nordic Cochrane Centre; 2014.
17. Bax L, Yu LM, Ikeda N, Tsuruta H, Moons KG. Development and validation of MIX: comprehensive free software for meta-analysis of causal research data. BMC Med Res Methodol. 2006;6:50.
18. Palmer TM, Sterne JAC, editors. Meta-analysis in Stata: an updated collection from the Stata journal. 2nd ed. College Station: Stata Press; 2015.
19. NCSS 12 statistical software. NCSS, LLC. Kaysville, Utah, USA, ncss.com/software/ncss; 2018.
20. R Core Team. R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing; 2018. https://www.R-project.org/
21. Dewey M. CRAN task view: meta-analysis. 2018. https://cran.r-project.org/web/views/MetaAnalysis.html. Accessed 14 May 2018.
22. Schild AH, Voracek M. Less is less: a systematic review of graph use in meta-analyses. Res Synth Methods. 2013;4:209-19.
23. Schild AH, Voracek M. Finding your way out of the forest without a trail of bread crumbs: development and evaluation of two novel displays of forest plots. Res Synth Methods. 2015;6:74-86.
24. Ogilvie D, Fayter D, Petticrew M, Sowden A, Thomas S, Whitehead M, Worthy G. The harvest plot: a method for synthesising evidence about the differential effects of interventions. BMC Med Res Methodol. 2008;8:1.
25. Krahn U, Binder H, König J. A graphical tool for locating inconsistency in network meta-analyses. BMC Med Res Methodol. 2013;13:1.
26. Krahn U, Binder H, König J. Visualizing inconsistency in network meta-analysis by independent path decomposition. BMC Med Res Methodol. 2014;1:131.
27. Smith ML, Glass GV, Miller TI. The benefits of psychotherapy. Baltimore: Johns Hopkins University Press; 1980.
28. Dias S, Ades AE, Welton NJ, Jansen JP, Sutton AJ. Network meta-analysis for decision-making. Chichester: Wiley-Blackwell; 2018.
29. Barrowman NJ, Myers RA. Raindrop plots: a new way to display collections of likelihoods and distributions. Am Stat. 2003;57:268-74.
30. Sutton AJ, Cooper NJ, Jones DR, Lambert PC, Thompson JR, Abrams KR. Evidence-based sample size calculations based upon updated meta-analysis. Stat Med. 2007;26:2479-500.
31. Yang G, Xie MG. gmeta [R software package]. 2010. https://CRAN.R-project.org/package=gmeta
32. Nakagawa S, Noble DW, Senior AM, Lagisz M. Meta-evaluation of meta-analysis: ten appraisal questions for biologists. BMC Biol. 2017;15:18.
33. Walker AM, Martin-Moreno JM, Artalejo FR. Odd man out: a graphical approach to meta-analysis. Am J Public Health. 1988;78:961-6.
34. Thompson CG. Graphing effects as fuzzy numbers in meta-analysis. J Mod Appl Stat Methods. 2016;15:49.
35. Light RJ, Pillemer DB. Summing up: the science of reviewing research. Cambridge: Harvard University Press; 1984.
36. Egger M, Smith GD, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. Br Med J. 1997;315:629-34.
37. Elvik R. Evaluating the statistical conclusion validity of weighted mean results in meta-analysis by analysing funnel graph diagrams. Accid Anal Prev. 1998;30:255-66.
38. Duval S, Tweedie R. Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics. 2000;56:455-63.
39. Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L. Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry. J Clin Epidemiol. 2008;61:991-6.
40. Langan D, Higgins JP, Gregory W, Sutton AJ. Graphical augmentations to the funnel plot to assess the impact of additional evidence on a meta-analysis. J Clin Epidemiol. 2012;65:511-9.
41. Crowther MJ, Langan D, Sutton AJ. Graphical augmentations to the funnel plot to assess the impact of additional evidence on a meta-analysis. Stata J. 2012;12:605-22.
42. Chevance A, Schuster T, Steele R, Ternès N, Platt RW. Contour plot assessment of existing meta-analyses confirms robust association of statin use and acute kidney injury risk. J Clin Epidemiol. 2015;68:1138-43.
43. Radua J. metansue [R package software]. 2015. https://CRAN.R-project.org/package=metansue
44. Bowden J, Jackson C. Weighing evidence "steampunk" style via the Meta-Analyser. Am Stat. 2016;70:385-94.
45. Galbraith RF. A note on graphical presentation of estimated odds ratios from several clinical trials. Stat Med. 1988;7:889-94.
46. Gee T. Capturing study influence: the concept of 'gravity' in meta-analysis. Aust Couns Res J. 2005;1:52-75.
47. Barendregt JJ, Doi S. MetaXL User Guide. Sunrise Beach: EpiGear International; 2016.
48. Lau J, Ioannidis JP, Schmid CH. Summing up evidence: one answer is not always enough. Lancet. 1998;351:123-7.
49. Weinhandl ED, Duval S. Generalization of trim and fill for application in meta-regression. Res Synth Methods. 2012;3:51-67.
50. Dusseldorp E, Van Genugten L, van Buuren S, Verheijden MW, van Empelen P. Combinations of techniques that effectively change health behavior: evidence from Meta-CART analysis. Health Psychol. 2014;33:1530-40.
51. van Lissa CJ. metaforest [R package software]. 2017. S
52. Wang XV, Cole B, Bonetti M, Gelber RD. Meta-STEPP: subpopulation treatment effect pattern plot for individual patient data meta-analysis. Stat Med. 2016;35:3704-16.
53. Thompson SG. Controversies in meta-analysis: the case of the trials of serum cholesterol reduction. Stat Methods Med Res. 1993;2:173-92.
54. Sutton AJ, Abrams KR, Jones DR, Sheldon TA, Song F. Methods for meta-analysis in medical research. Chichester: Wiley; 2000.
55. Baujat B, Mahé C, Pignon JP, Hill C. A graphical method for exploring heterogeneity in meta-analyses: application to a meta-analysis of 65 trials. Stat Med. 2002;21:2641-52.
56. Barrowman NJ, Fang M, Sampson M, Moher D. Identifying null meta-analyses that are ripe for updating. BMC Med Res Methodol. 2003;3:13.
57. Viechtbauer W, Cheung MWL. Outlier and influence diagnostics for meta-analysis. Res Synth Methods. 2010;1:112-25.
58. Poorolajal J, Mahmoodi M, Majdzadeh R, Fotouhi A. Metaplot: a novel stata graph for assessing heterogeneity at a glance. Iran J Public Health. 2010;39:102-4.
59. Olkin I, Dahabreh IJ, Trikalinos TA. GOSH: a graphical display of study heterogeneity. Res Synth Methods. 2012;3:214-23.
60. Beath KJ. A finite mixture method for outlier detection and robustness in meta-analysis. Res Synth Methods. 2014;5:285-93.
61. Mavridis D, Moustaki I, Wall M, Salanti G. Detecting outlying studies in meta-regression models using a forward search algorithm. Res Synth Methods. 2017;8:199-211.
62. Mathur MB, VanderWeele TJ. Sensitivity analysis for unmeasured confounding in meta-analyses. 2017. https://arxiv.org/pdf/1707.09076.pdf. Accessed 31 July 2017.
63. Kulinskaya E, Koricheva J. Use of quality control charts for detection of outliers and temporal trends in cumulative meta-analysis. Res Synth Methods. 2010;1:297-307.
64. Lau J, Antman EM, Jimenez-Silva J, Kupelnick B, Mosteller F, Chalmers TC. Cumulative meta-analysis of therapeutic trials for myocardial infarction. N Engl J Med. 1992;327:248-54.
65. Pogue JM, Yusuf S. Cumulating evidence from randomized trials: utilizing sequential monitoring boundaries for cumulative meta-analysis. Control Clin Trials. 1997;18:580-93.
66. Ioannidis JP, Contopoulos-Ioannidis DG, Lau J. Recursive cumulative meta-analysis: a diagnostic for the evolution of total randomized evidence from group and individual patient data. J Clin Epidemiol. 1999;52:281-91.
67. Villanueva EV, Zavarsek S. Evaluating heterogeneity in cumulative meta-analyses. BMC Med Res Methodol. 2004;4:18.
68. Scheibehenne B, Jamil T, Wagenmakers EJ. Bayesian evidence synthesis can reconcile seemingly inconsistent results: the case of hotel towel reuse. Psychol Sci. 2016;27:1043-6.
69. Heck DW, Gronau QF, Wagenmakers E-J. metaBMA [R software package]. 2017. https://CRAN.R-project.org/package=metaBMA
70. Viechtbauer W. Conducting meta-analyses in R with the metafor package. J Stat Softw. 2010;36:1-48.
71. van Houwelingen HC, Zwinderman KH, Stijnen T. A bivariate approach to meta-analysis. Stat Med. 1993;12:2273-84.
72. Wang MC, Bushman BJ. Using the normal quantile plot to explore meta-analytic data sets. Psychol Methods. 1998;3:46-54.
73. Higgins JP, Green S, editors. Cochrane handbook for systematic reviews of interventions. Chichester: Wiley; 2008.
74. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;151:264-9.
75. Panesar SS, Rao C, Vecht JA, Mirza SB, Netuveli G, Morris R, et al. Development of the Veritas plot and its application in cardiac surgery: an evidence-synthesis graphic tool for the clinician to assess multiple meta-analyses reporting on a common outcome. Can J Surg. 2009;52:E137-45.
76. Keus F, Wetterslev J, Gluud C, van Laarhoven CJ. Evidence at a glance: error matrix approach for overviewing available evidence. BMC Med Res Methodol. 2010;10:1.
77. Thomson HJ, Thomas S. The effect direction plot: visual display of non-standardised effects across multiple outcome domains. Res Synth Methods. 2013;4:95-101.
78. Wang DD, Shams-White M, Bright OJM, Parrott JS, Chung M. Creating a literature database of low-calorie sweeteners and health studies: evidence mapping. BMC Med Res Methodol. 2016;16:1.
79. Hanji MB. Meta-analysis in psychiatry research: fundamental and advanced methods. Toronto: Apple Academic Press; 2017.
80. L'Abbé KA, Detsky AS, O'Rourke K. Meta-analysis in clinical research. Ann Intern Med. 1987;107:224-33.
81. Jimenez FJ, Guallar E, Martín-Moreno JM. A graphical display useful for meta-analysis. Eur J Pub Health. 1997;7:101-5.
82. Voest EE, van Houwelingen JC, Neijt JP. A meta-analysis of prognostic factors in advanced ovarian cancer with median survival and overall survival (measured with the log (relative risk)) as main objectives. Eur J Cancer. 1989;25:711-20.
83. Brannick MT, Gültaş M. Cross-hairs: a scatterplot for meta-analysis in R. Res Synth Methods. 2017;8:53-63.
84. Moses LE, Shapiro D, Littenberg B. Combining independent studies of a diagnostic test into a summary ROC curve: data-analytic approaches and some additional considerations. Stat Med. 1993;12:1293-316.
85. Phillips B, Stewart LA, Sutton AJ. 'Cross hairs' plots for diagnostic meta-analysis. Res Synth Methods. 2010;1:308-15.
86. Deeks JJ, Altman DG. Effect measures for meta-analysis of trials with binary outcomes. In: Egger M, Smith GD, Altman DG, editors. Systematic reviews in health care: meta-analysis in context. 2nd ed. London: BMJ Books; 2001. p. 313-35.
87. Olliaro P, Vaillant MT. Alternative visual displays of metaanalysis of malaria treatment trials to facilitate translation of research into policy. Diagn Microbiol Infect Dis. 2010;68:422-31.
88. Steinhauser S, Schumacher M, Rücker G. Modelling multiple thresholds in meta-analysis of diagnostic test accuracy studies. BMC Med Res Methodol. 2016;16:97.
89. Shadish WR, Sweeney RB. Mediators and moderators in meta-analysis: there's a reason we don't let dodo birds tell us which psychotherapies should have prizes. J Consult Clin Psychol. 1991;59:883-93.
90. Rhodes DR, Barrette TR, Rubin MA, Ghosh D, Chinnaiyan AM. Meta-analysis of microarrays: interstudy validation of gene expression profiles reveals pathway dysregulation in prostate cancer. Cancer Res. 2002;62:4427-33.
91. Choi JK, Yu U, Kim S, Yoo OJ. Combining multiple microarray studies and modeling interstudy variation. Bioinformatics. 2003;19:i84-90.
92. Hattie J. Visible learning: a synthesis of over 800 meta-analyses relating to achievement. London: Routledge; 2008.
93. Bowden J, Tierney JF, Copas AJ, Burdett S. Quantifying, displaying and accounting for heterogeneity in the meta-analysis of RCTs using standard and generalised Q statistics. BMC Med Res Methodol. 2011;11:41.
94. Feichtinger J, McFarlane RJ, Larcombe LD. CancerMA: a web-based tool for automatic meta-analysis of public cancer microarray data. Database. 2012;2012:bas055.
95. Gartlehner G, Schultes MT, Titscher V, Morgan LC, Bobashev GV, Williams P, West SL. User testing of an adaptation of fishbone diagrams to depict results of systematic reviews. BMC Med Res Methodol. 2017;17:169.
96. Babatunde OO, Tan V, Jordan JL, Dziedzic K, Chew-Graham CA, Jinks C, et al. Evidence flowers: an innovative, visual method of presenting "best evidence" summaries to health professional and lay audiences. Res Synth Methods. 2018;9:273-84.
97. Schweder T, Spjøtvoll E. Plots of p-values to evaluate many tests simultaneously. Biometrika. 1982;69:493-502.
98. Dear KB, Begg CB. An approach for assessing publication bias prior to performing a meta-analysis. Stat Sci. 1992;7:237-45.
99. Copas J, Shi JQ. Meta-analysis, funnel plots and sensitivity analysis. Biostatistics. 2000;1:247-62.
100. Ioannidis JP, Trikalinos TA. An exploratory test for an excess of significant findings. Clin Trials. 2007;4:245-53.
101. Gerber AS, Malhotra N. Publication bias in empirical sociological research: do arbitrary significance levels distort published results? Sociol Methods Res. 2008;37:3-30.
102. Formann AK. Estimating the proportion of studies missing for meta-analysis due to publication bias. Contemp Clin trials. 2008;29:732-9.
103. Han B, Eskin E. Interpreting meta-analyses of genome-wide association studies. PLoS Genet. 2012;8:e1002555.
104. Simonsohn U, Nelson LD, Simmons JP. P-curve: a key to the file-drawer. J Exp Psychol Gen. 2014;143:534-47.
105. Taylor AE, Munafò MR. Triangulating meta-analyses: the example of the serotonin transporter gene, stressful life events and major depression. BMC Psychol. 2016;4:23.
106. Schwarzer G, Carpenter JR, Rücker G. Meta-analysis with R. Cham: Springer; 2015.
107. Harrison S, Jones HE, Martin RM, Lewis SJ, Higgins JP. The albatross plot: a novel graphical tool for presenting results of diversely reported studies in a systematic review. Res Synth Methods. 2017;8:281-9.
108. Lumley T. Network meta-analysis for indirect treatment comparisons. Stat Med. 2002;21:2313-24.
109. König J, Krahn U, Binder H. Visualizing the flow of evidence in network meta-analysis and characterizing mixed treatment comparisons. Stat Med. 2013;32:5414-29.
110. Batson S, Score R, Sutton AJ. Three-dimensional evidence network plot system: covariate imbalances and effects in network meta-analysis explored using a new software tool. J Clin Epidemiol. 2017;86:182-95.
111. Song F, Harvey I, Lilford R. Adjusted indirect comparison may be less biased than direct comparison for evaluating new pharmaceutical interventions. J Clin Epidemiol. 2008;61:455-63.
112. Tan SH, Cooper NJ, Bujkiewicz S, Welton NJ, Caldwell DM, Sutton AJ. Novel presentational approaches were developed for reporting network meta-analysis. J Clin Epidemiol. 2014;67:672-80.
113. Phillippo DM, Dias S, Ades AE, Didelez V, Welton NJ. Sensitivity of treatment recommendations to bias in network meta-analysis. J R Stat Soc Ser A Stat Soc. 2018;181:843-67.
114. Salanti G, Kavvoura FK, Ioannidis JP. Exploring the geometry of treatment networks. Ann Intern Med. 2008;148:544-53.
115. Chung H, Lumley T. Graphical exploration of network meta-analysis data: the use of multidimensional scaling. Clin Trials. 2008;5:301-7.
116. Hawkins N, Scott DA, Woods BS, Thatcher N. No study left behind: a network meta-analysis in non-small-cell lung cancer demonstrating the importance of considering all relevant data. Value Health. 2009;12:996-1003.
117. Naci H. Communication of treatment rankings obtained from network meta-analysis using data visualization. Circ Cardiovasc Qual Outcomes. 2016;9:605-8.
118. Salanti G, Marinho V, Higgins JP. A case study of multiple-treatments meta-analysis demonstrates that covariates should be considered. J Clin Epidemiol. 2009;62:857-64.
119. Cipriani A, Furukawa TA, Salanti G, Geddes JR, Higgins JP, Churchill R, et al. Comparative efficacy and acceptability of 12 new-generation antidepressants: a multiple-treatments meta-analysis. Lancet. 2009;373:746-58.
120. Cooper NJ, Sutton AJ, Morris D, Ades AE, Welton NJ. Addressing between-study heterogeneity and inconsistency in mixed treatment comparisons: application to stroke prevention treatments in individuals with non-rheumatic atrial fibrillation. Stat Med. 2009;28:1861-81.
121. Donegan S, Dias S, Tudur-Smith C, Marinho V, Welton NJ. Graphs of study contributions and covariate distributions for network meta-regression. Res Synth Methods. 2018;9:243-60.
122. Senn S, Gavini F, Magrez D, Scheen A. Issues in performing a network meta-analysis. Stat Methods Med Res. 2013;22:169-89.
123. Salanti G, Del Giovane C, Chaimani A, Caldwell DM, Higgins JP. Evaluating the quality of evidence from a network meta-analysis. PLoS One. 2014;9:e99682.
124. Veroniki AA, Straus SE, Fyraridis A, Tricco AC. The rank-heat plot is a novel way to present the results from a network meta-analysis including multiple outcomes. J Clin Epidemiol. 2016;76:193-9.
125. Rücker G, Schwarzer G. Resolve conflicting rankings of outcomes in network meta-analysis: partial ordering of treatments. Res Synth Methods. 2017;8:526-36.
126. Rücker G, Schwarzer G, Krahn U, König J. netmeta [R software package]. 2017. https://CRAN.R-project.org/package=netmeta
127. Cipriani A, Furukawa TA, Salanti G, Chaimani A, Atkinson LZ, Ogawa Y, et al. Comparative efficacy and acceptability of 21 antidepressant drugs for the acute treatment of adults with major depressive disorder: a systematic review and network meta-analysis. Lancet. 2018;391:1357-66.
128. Freiman JA, Chalmers TC, Smith H Jr, Kuebler RR. The importance of beta, the type II error and sample size in the design and interpretation of the randomized control trial: survey of 71 negative trials. N Engl J Med. 1978;299:690-4.
129. Lewis S, Clarke M. Forest plots: trying to see the wood and the trees. Br Med J. 2001;322:1479-80.
130. Lakens D, Scheel AM, Isager PM. Equivalence testing for psychological research: a tutorial. Adv Methods Pract Psychol Sci. 2018;1:259-69.
131. Voracek M, Kossmeier M, Tran US. Which data to meta-analyze, and how? A specification-curve and multiverse-analysis approach to meta-analysis. Z Psychol. 2019;227:64-82.
132. APA Publications and Communications Board Working Group on Journal Article Reporting Standards. Reporting standards for research in psychology: why do we need them? What might they be? Am Psychol. 2008;63:839-51.
133. Cooper H, Hedges LV, Valentine JC, editors. The handbook of research synthesis and meta-analysis. 3rd ed. New York: Russell Sage; 2019.
134. Bax L, Yu LM, Ikeda N, Moons KG. A systematic comparison of software dedicated to meta-analysis of causal studies. BMC Med Res Methodol. 2007;7:40.
135. Tay L, Parrigon S, Huang Q, LeBreton JM. Graphical descriptives: a way to improve data transparency and methodological rigor in psychology. Perspect Psychol Sci. 2016;11:692-701.