Combining statistical evidence

International Statistical Review

Volumn 82, Issue 2, 2014, Pages 214-242

  Kulinskaya, Elena ^a   Morgenthaler, Stephan ^b   Staudte, Robert G ^c  

^a UNIVERSITY OF EAST ANGLIA   (United Kingdom)

^b EPFL   (Switzerland)

^c LA TROBE UNIVERSITY   (Australia)

Author keywords

Effect size; Meta analysis; Meta regression; Random effects; Review; Software

Indexed keywords

EID: 84905442437     PISSN: 03067734     EISSN: 17515823     Source Type: Journal    
DOI: 10.1111/insr.12037     Document Type: Article

References (140)

1. Agresti A. & Coull B. (1998). Approximate is better than 'exact' for interval estimation of binomial proportions. Am. Stat., 52, 119-126.
2. Akritas M. & Papadatos N. (2004). Heteroscedastic one-way ANOVA and lack-of-fit tests. J. Amer. Statist. Assoc., 99, 368-382.
3. Anscombe F. (1948). The transformation of Poisson, binomial and negative binomial data. Biometrika, 35, 246-254.
4. Arends L. (2006). Multivariate Meta-Analysis: Modelling the Heterogeneity; Mixing Apples and Oranges: Dangerous or Delicious? Alblasserdam: Haveka B V.
5. Arends L., Hamsa T., van Houwelingen H., Heijenbrok-Kai M., Hunink M. & Stijnen T. (2008a). Bivariate random effects meta-analysis of ROC curves. Med. Decis. Making, 28, 621-638.
6. Arends L., Hoes A., Lubsen J., Grobbee D. & Stijnen T. (2000). Baseline risk as predictor of treatment benefit: three clinical meta-re-analyses. Stat. Med., 19, 3497-3518.
7. Arends L., Hunink M. & Stijnen T. (2008b). Meta-analysis of summary survival curve data. Stat. Med., 27, 4381-4396.
8. Arends L., Voko Z. & Stijnen T. (2003). Combining multiple outcome measures in a meta-analysis: an application. Stat. Med., 22, 1335-1353.
9. Bagos P. & Nikolopoulos G. (2009). Generalized least squares for assessing trends in cumulative meta-analysis with applications in genetic epidemiology. J. Clin. Epidemiol., 62, 1037-1044.
10. Baker R. & Jackson D. (2010). Inference for meta-analysis with a suspected temporal trend. Biometrical J., 52, 538-551.
11. Bax L., Yu L.-M., Ikeda N. & Moons K. (2007). Systematic comparison of software dedicated to meta-analysis of causal studies. BMC Med. Res. Methodol., 7, 40.
12. Berkey C.S., Hoaglin D.C., Mosteller F. & Colditz G.A. (1995). A random-effects regression model for meta-analysis. Stat. Med., 14, 395-411.
13. Biggerstaff B. & Jackson D. (2008). The exact distribution of Cochran's heterogeneity statistic in one-way random effects meta-analysis. Stat. Med., 27, 6093-6110.
14. Biggerstaff B. & Tweedie R. (1997). Incorporating variability in estimates of heterogeneity in the random effects model in meta-analysis. Stat. Med., 16, 753-768.
15. Böhning D. & Viwatwongkasen C. (2005). Revisiting proportion estimators. Stat. Methods. Med. Res., 14, 147-169.
16. Borenstein M., Hedges L.V., Higgins J.P. & Rothstein H.R. (2009). Introduction to Meta-Analysis. New York: Wiley.
17. Brok J., Thorlund K., Gluud C. & Wetterslev J. (2008). Trial sequential analysis reveals insufficient information size and potentially false positive results in many meta-analyses. J. Clin. Epidemiol., 61, 763-769.
18. Brown L., Cai T. & DasGupta A. (2001). Interval estimation for a binomial proportion. Stat. Sci., 16, 101-112.
19. Buonaccorsi J. (2006). Estimation in two-stage models with heteroscedasticity. Int. Stat. Rev., 74, 403-418.
20. Chang C.-H. & Pal N. (2008). Testing on the common mean of several normal distributions. Comput. Stat. Data Anal., 53, 321-333.
21. Chen H., Manning A.K. & Dupuis J. (2012). A method of moments estimator for random effect multivariate meta-analysis. Biometrics, 68, 1278-1284.
22. Cochran W. (1937). Problems arising in the analysis of a series of similar experiments. J. R. Stat. Soc., 4, 102-118.
23. Colditz G., Brewer T., Berkey C.S., Wilson M.E., Burdick E., Fineberg H.V. & Mosteller F. (1994). Efficacy of bcg vaccine in the prevention of tuberculosis. Meta-analysis of the published literature. JAMA, 271, 698-702.
24. Cooper H., Hedges L.V. & Valentine J.C. (2009). Handbook of Research Synthesis and Meta-Analysis. New York: Russell Sage Foundation.
25. Deeks J., Dinnes J., D'Amico R., Sowden A., Sakarovitch C., Song F., Petticrew M. & Altman D. (2003). Evaluating non-randomised intervention studies. Health Technol. Asses., 7, 1-173.
26. Deeks J. & Higgins J. (2010). Statistical Algorithms in Review Manager 5. Technical Report, Cochrane Collaboration. On Behalf of the Statistical Methods Group of the Cochrane Collaboration.
27. Demidenko E. (2004). Mixed Models. Theory and Applications. New York: John Wiley & Sons.
28. Demidenko E., Sargent J. & Onega T. (2012). Random effects coefficient of determination for mixed and meta-analysis models. Commun. Statist.-Theory Meth., 41, 953-969.
29. DerSimonian R. & Kacker R. (2007). Random-effects model for meta-analysis of clinical trials: an update. Contemp. Clin. Trials, 28, 105-114.
30. DerSimonian R. & Laird N. (1986). Meta-analysis in clinical trials. Control. Clin. Trials, 7, 177-188.
31. Duval S. & Tweedie R. (2000a). A nonparametric "trim and fill" method of accounting for publication bias in meta-analysis. J. Amer. Statist. Assoc., 95, 89-98.
32. Duval S. & Tweedie R. (2000b). Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56, 455-463.
33. Egger M., Smith G., Schneider M. & Minder C. (1997). Bias in meta-analysis detected by a simple, graphical test. Brit. Med. J., 315, 629-634.
34. Fisher S. (1932). Statistical Methods for Research Workers. Genesis Publishing Pvt Ltd. Reisssued within Statistical Methods, Experimental Design and Scientific Inference, with a Foreword by F. Yates (edited by J.H. Bennett). Oxford: Oxford University Press, 1990.
35. Fleiss J. (1993). Review papers: the statistical basis of meta-analysis. Stat. Methods Med. Res., 2, 121-145.
36. Gasparrini A., Armstrong B. & Kenward M.G. (2012). Multivariate meta-analysis for non-linear and other multi-parameter associations. Stat. Med., 31, 3821-3839.
37. Gehr B., Weiss C. & Porzsolt F. (2006). The fading of reported effectiveness. A meta-analysis of randomised controlled trials. BMC Med. Res. Methodol., 6, 25.
38. Givens G., Smith D. & Tweedie R. (1997). Publication in meta-analysis: a Bayesian data-augmentation approach to account for issues exemplified in the passive smoking debate. Stat. Sci., 12, 221-250.
39. Goudie A., Sutton A., Jones D.R. & Donald A. (2010). Empirical assessment suggests that existing evidence could be used more fully in designing randomized controlled trials. J. Clin. Epidemiol., 63, 983-991.
40. Harbord R., Deeks J., Egger M., Whiting P. & Sterne J. (2007). A unification of models for meta-analysis of diagnostic accuracy studies. Biostatistics, 8, 239-251.
41. Hardy R. & Thompson S. (1996). A likelihood approach to meta-analysis. Stat. Med., 15, 619-629.
42. Hartung J. (1999). An alternative method for meta-analysis. Biometrical J., 41, 901-916.
43. Hartung J. & Knapp G. (2003). An alternative test procedure for meta-analysis. In Meta-Analysis, Eds H. Schulze, H. Holling & D. Bøhning, pp. 53-69. Gøttingen: Hogrefe & Huber.
44. Hartung J. & Makambi K. (2003). Reducing the number of unjustified significant results in meta-analysis. Commun. Stat.-Simul. Comput., 32, 1179-1190. DOI: 10.1081/SAC-120023884.
45. Hedges L. & Olkin I. (1985). Statistical Methods for Meta-Analysis. Orlando: Academic Press.
46. Hedges L. & Olkin I. (1993). Regression models in research synthesis. Am. Stat., 37, 137-140.
47. Hedges L., Tipton E. & Johnson M. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Res. Synth. Method, 1, 39-65.
48. Henmi M. & Copas J. (2010). Confidence intervals for random effects meta-analysis and robustness to publication bias. Stat. Med., 29, 2969-2983.
49. Higgins J., Green S. (eds). (2011). Cochrane Handbook for Systematic Reviews of Interventions: Version 5.1.0 [updated March 2011]. Cochrane Collaboration.
50. Higgins J. & Thompson S. (2002) Quantifying heterogeneity in meta-analysis. Stat. Med., 21, 1531-1558.
51. Higgins J., Thompson S. & Spiegelhalter D. (2009). A re-evaluation of random-effects meta-analysis. J. R. Stat. Soc. Ser. A, 172, 137-159.
52. Higgins J., Whitehead A. & Simmonds M. (2011). Sequential methods for random-effects meta-analysis. Stat. Med., 30, 903-921.
53. Hoaglin D., Hawkins N., Jansen J., Scott D., Itzler R., Cappelleri J., Boersma C., Thompson D., Larholt K., Diaz M. & Barrett A. (2011). Conducting indirect-treatment-comparison and network-meta-analysis studies: report of the ISPOR task force on indirect treatment comparisons good research practices: part 2. Value Health, 14, 429-437.
54. Huizenga H., Visser I. & Dolan C. (2011). Testing overall and moderator effects in random effects meta-regression. Br. J. Math. Stat. Psychol., 64, 1-19.
55. Jackson D. (2006). The power of the standard test for the presence of heterogeneity in meta-analysis. Stat. Med., 25, 2688-2699.
56. Jackson D., Riley R. & White I. (2011). Multivariate meta-analysis: potential and promise. Stat. Med., 30, 2481-2498.
57. Jackson D., White I. & Thompson S. (2010). Extending DerSimonian and Laird's methodology to perform multivariate random effects meta-analyses. Stat. Med., 29, 1282-1297.
58. James G. (1951). The comparison of several groups of observations when the ratios of the population variances are unknown. Biometrika, 38, 324-329.
59. Jansen J., Fleurence R., Devine B., Itzler R., Barrett A., Hawkins N., Lee K., Boersma C., Annemans L. & Cappelleri J. (2011). Interpreting indirect treatment comparisons and network meta-analysis for health-care decision making: report of the ISPOR task force on indirect treatment comparisons good research practices: part 1. Value Health, 14, 417-428.
60. Jones A.P., Riley R.D., Williamson P.R. & Whitehead A. (2009). Meta-analysis of individual patient data versus aggregate data from longitudinal clinical trials. Clin. Trials, 6, 16-27.
61. Kampichler C. & Bruckner A. (2009). The role of microarthropods in terrestrial decomposition: a meta-analysis of 40 years of litterbag studies. Biol. Rev., 84, 375-389.
62. Konnerup M. & Kongsted H. (2009). There is more to seeing than meets the eye: observational studies, research synthesis, and the social sciences, Technical Report Technical Report, London School of Economics and Political Science, Centre for the Philosophy of Natural and Social Science: Contingency and Dissent in Science.
63. Koricheva J., Gurevitch J. & Mengeresen K. (2013).Handbook of Meta-Analysis in Ecology and Evolution. Princeton: Princeton University Press.
64. Koricheva J, Jennions M & Lau J. (2011). Temporal changes in effect sizes: causes, detection and implications. In The Handbook of Meta-Analysis for Ecology and Evolution, Eds. J. Koricheva, J. Gurevitch, K. Mengersen Princeton: Princeton University Press.
65. Kulinskaya E., Dollinger M. & Bjørkestøl K. (2011a). On the moments of Cochran's Q statistic; with application to the meta-analysis of risk difference. Res. Synth. Method, 2, 254-270.
66. Kulinskaya E., Dollinger M. & Bjørkestøl K. (2011b). Testing for homogeneity in meta-analysis I. The one parameter case: standardized mean difference. Biometrics, 67, 203-212.
67. Kulinskaya E. & Koricheva J. (2010). Use of quality control charts for detection of outliers and temporal trends in cumulative meta-analysis. Res. Synth. Method, 1, 297-307.
68. Kulinskaya E., Morgenthaler S. & Staudte R. (2008). Meta Analysis: A Guide to Calibrating and Combining Statistical Evidence, Wiley Series in Probability and Statistics. Chichester: John Wiley & Sons, Ltd.
69. Kulinskaya E. & Olkin I. (2014). An overdispersion model in meta analysis. Stat. Model., 14(1).
70. Kulinskaya E., Staudte R. & Gao H. (2003). Power approximations in testing for unequal means in a one-way ANOVA weighted for unequal variances. Commun. Statist.-Theory Meth., 32, 2353-2371.
71. Lan K. & DeMets D. (1983). Discrete sequential boundaries for clinical trials. Biometrika, 70, 659-663.
72. Lan K.K.G., Hu M. & Cappelleri J. (2003). Applying the law of iterated logarithm to cumulative meta-analysis of a continuous endpoint. Stat. Sinica, 13, 1135-1145.
73. Lau J., Antman E., Jimenez-Silva J., Kupelnick B.M.F. & Chalmers T. (1992). Cumulative meta-analysis of therapeutic trials for myocardial infarction. N. Engl. J. Med., 327, 248-254.
74. Lehrer J. (2010). The Truth Wears Off. New York: The New Yorker.
75. Li Y., Shi L. & Roth H. (1994). The bias of the commonly-used estimate of variance in meta-analysis. Commun. Statist.-Theory Meth., 23, 1063-1085.
76. Light R. & Pillemer D. (1984). Summing Up: The Science of Reviewing Research. Cambridge: Harvard University Press.
77. Lin D. & Zeng D. (2010). On the relative efficiency of using summary statistics versus individual-level data in meta-analysis. Biometrika, 97, 321-332.
78. Malloy M., Prendergast L. & Staudte R. (2011). Comparison of methods for fixed effect meta-regression of standardized differences of means. EJS, 5, 83-101.
79. Malloy M., Prendergast L. & Staudte R. (2013). Transforming the model-T: random effects meta-analysis with stable weights. Stat. Med., 32, 1842-1864. DOI: 10.1002/sim.5666.
80. Mandel J. & Paule R. (1970). Interlaboratory evaluation of a material with unequal number of replicates. Anal. Chem., 42, 1194-1197.
81. Mathew T., Nahtman T., von Rosen D. & Sinha B. (2010). Nonnegative estimation of variance components in heteroscedastic one-way random effects ANOVA models. Statistics, 44, 557-569.
82. Mathew T. & Nordström K. (1999). On the equivalence of meta-analysis using literature and using individual patient data. Biometrics, 55, 1221-1223.
83. Mathew T. & Nordström K. (2010). Comparison of one-step and two-step meta-analysis models using individual patient data. Biometrical J., 52, 271-287.
84. Normand S.-L. (1999). Meta-analysis: formulating, evaluating, combining, and reporting. Stat. Med., 18, 321-359.
85. Paul M., Riebler A., Bachmann L., Rue H. & Held L. (2010). Bayesian bivariate meta-analysis of diagnostic test studies using integrated nested Laplace approximations. Stat. Med., 29, 1325-1339.
86. Pigott T.D. (2012). Advances in Meta-Analysis. Berlin: Springer Science+ Business Media.
87. Pogue J. & Yusuf S. (1997). Cumulating evidence from randomized trials: utilizing sequential monitoring boundaries for cumulative meta-analysis. Control. Clin. Trials, 18, 580-593.
88. Putter H., Fiocco M. & Stijnen T. (2010). Meta-analysis of diagnostic test accuracy studies with multiple thresholds using survival methods. Biometrical J., 52, 95-110.
89. Reeves B.C., Wells G.A. (eds). (2013). Special Issue of Research Synthesis: Inclusion of Non-Randomized Studies in Systematic Reviews, Vol. 4 New York: John Wiley & Sons.
90. Riley R., Thompson J. & Abrams K. (2008). An alternative model for bivariate random-effects meta-analysis when the within-study correlations are unknown. Biostatistics, 9, 172-186.
91. Ritz J., Demidenko E. & Spiegelman D. (2008). Multivariate meta-analysis for data consortia, individual patient meta-analysis, and pooling projects. J. Stat. Plan. Infer., 138, 1919-1933.
92. Rosenthal R. (1979). The file drawer problem and tolerance for null results. Psychol. Bull., 86, 638-641.
93. Rothstein H.R., Sutton A.J. & Borenstein M. (2006). Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments. New York: Wiley.
94. Rukhin A. (2003). Two procedures of meta-analysis in clinical trials and interlaboratory studies. Tatra Mt. Math. Pub., 26, 155-168.
95. Rukhin A. (2007). Estimating common vector mean in interlaboratory studies. J. Multivariate Anal., 98, 435-454.
96. Rukhin A. (2009). Weighted means statistics in interlaboratory studies. Metrologia, 46, 323-331.
97. Rukhin A. (2013). Estimating heterogeneity variance in meta-analysis. J. Roy. Stat. Soc. Ser. B, 75, 451-469.
98. Rukhin A., Biggerstaff B. & Vangel M. (2000). Restricted maximum likelihood estimation of a common mean and the Mandel-Paule algorithm. J. Stat. Plan. Infer., 83, 319-330.
99. Rukhin A. & Vangel M. (1998). Estimation of a common mean and weighted means statistics. J. Amer. Statist. Assoc., 93, 303-308.
100. Salanti G. & Ioannidis J. (2009). Synthesis of observational studies should consider credibility ceilings. J. Clin. Epidemiol., 62, 115-122.
101. SAS Inc. (1989). SAS/STAT User's Guide, Version 6. Cary, North-Carolina: SAS Institute Inc.
102. Schulze R., Holling H. & Böhning D. (2003). Meta-Analysis. Cambridge: Hogrefe & Huber.
103. Sharma G. & Mathew T. (2011). Higher order inference for the consensus mean in inter-laboratory studies. Biometrical J., 53, 128-136.
104. Shuster J. (2010). Empirical vs natural weighting in random effects meta-analysis. Stat. Med., 29, 1259-1265.
105. Sidik K. & Jonkman J. (2002). A simple confidence interval for meta-analysis. Stat. Med., 21, 3153-3159.
106. Sidik K. & Jonkman J.N. (2003). On constructing confidence intervals for a standardized mean difference in meta-analysis. Commun. Stat.-Simul. Comput., 32, 1191-1203. DOI: 10.1081/SAC-120023885.
107. Sidik K. & Jonkman J.N. (2006). Robust variance estimation for random effects meta-analysis. Comput. Stat. Data Anal., 50, 3681-3701. DOI: 10.1016/j.csda.2005.07.019.
108. Sidik K. & Jonkman J.N. (2007). A comparison of heterogeneity variance estimators in combining results of studies. Stat. Med., 50, 1964-1981. DOI: 10.1002/sim.2688.
109. Simmonds M.C., Higgins J.P.T, Stewart L.A., Tierney J.F., Clarke M.J. & Thompson S.G. (2005). Meta-analysis of individual patient data from randomized trials: a review of methods used in practice. Clin. Trials, 2, 209-217.
110. Simpson R. & Pearson K. (1904). Report on certain enteric fever inoculation statistics. Brit. Med. J., 2, 1243-1246.
111. Sinha B. (1985). Unbiased estimation of the variance of the Graybill-Deal estimator of the common mean of several normal populations. Can. J. Stat., 13, 243-247.
112. Sørensen H. (2008). Small sample distribution of the likelihood ratio test in the random effects model. J. Stat. Plan. Infer., 138, 1605-1614.
113. Stanley T.D. & Doucouliagos H. (2012). Meta-Regression Analysis in Economics and Business, Vol. 5. Routledge.
114. Staudte R. (2010). Book review of 'Multivariate meta-analysis: modeling the heterogeneity; mixing apples and oranges; dangerous or delicious?', by Lidia R. Arends, Alblasserdam, 2006. Res. Synth. Method, 1, 316-318.
115. Sterne J.A. (ed). (2009). Meta-Analysis in Stata: An Updated Collection from the Stata Journal. College Station, TX: Stata Press.
116. Stroup D., Berlin J., Morton S., Olkin I., Williamson G., Rennie D., Moher D., Becker B., Sipe A. & Thacker S.B. (2000). Meta-analysis of observational studies in epidemiology. A proposal for reporting. JAMA, 283, 2008-2012.
117. Sutton A., Abrams K., Jones D., Sheldon T. & Song F. (2000). Methods for Meta-Analysis in Medical Research. London: John Wiley & Sons.
118. Sutton A.J. & Higgins J. (2008). Recent developments in meta-analysis. Stat. Med., 27, 625-650.
119. Takkouche B., Cadarso-Suarez C. & Spiegelman D. (1999). Evaluation of old and new tests of heterogeneity in epidemiologic meta-analysis. Am. J. Epidemiol., 150, 206-215.
120. Thompson S., Ekelund U., Jebb S., Lindroos A., Mander A., Sharp S., Turner R. & Wilks D. (2011). A proposed method of bias adjustment for meta-analyses of published observational studies. Int. J. Epidemiol., 40, 765-777.
121. Tippett L. (1931). The Methods of Statistics. London: Williams & Norgate Ltd.
122. Trikalinos T. & Olkin I. (2008). A method for the meta-analysis of mutually exclusive binary outcomes. Stat. Med., 27, 4279-4300.
123. vander Tweel I. & Bollen C. (2010). Sequential meta-analysis: an efficient decision-making tool. Clin. Trials, 7, 136-146.
124. van Houwelingen H., Arends L. & Stijnen T. (2002). Advanced methods in meta-analysis: multivariate approach and meta-regression. Stat. Med., 21, 589-624.
125. van Houwelingen H., van Zwinderman K. & Stijnen T. (1993). A bivariate approach to meta-analysis. Stat. Med., 12, 2273-2284.
126. Vardi Y. (1985). Empirical distribution in selection bias models. Ann. Stat., 13, 178-203.
127. Viechtbauer W. (2007). Hypothesis tests for population heterogeneity in meta-analysis. Br. J. Math. Stat. Psychol., 60, 29-60.
128. Viechtbauer W. (2009). Conducting meta-analyses in r with the metafor package. J. Stat. Softw., 36, 1-48.
129. Wallace B.C., Schmid C.H., Lau J. & Trikalinos T.A. (2009). Meta-analyst: software for meta-analysis of binary, continuous and diagnostic data. BMC Med. Res. Methodol., 9, 80.
130. Welch B. (1951). On the comparison of several mean values: an alternative approach. Biometrika, 38, 330-336.
131. Wetterslev J., Thorlund K., Brok J. & Gluud C. (2008). Trial sequential analysis may establish when firm evidence is reached in cumulative meta-analysis. J. Clin. Epidemiol., 61, 64-75.
132. White I.R. (2009). Multivariate random-effects meta-analysis. Stata J., 9, 40-56.
133. Whitehead A. (1997). A prospectively planned cumulative meta-analysis applied to a series of concurrent clinical trials. Stat. Med., 16, 2901-2913.
134. Whitehead A. (2002). Meta-Analysis of Controlled Clinical Trials, Applied Statistics., Chichester: Wiley.
135. Whitehead J. (1999). A unified theory for sequential clinical trials. Stat. Med., 18, 2271-2286.
136. Wilson E. (1927). Probable inference, the law of succession, and statistical inference. J. Amer. Statist. Assoc., 22, 209-212.
137. Wolf F.M. (1986). Meta-Analysis: Quantitative Methods for Research Synthesis, Vol. 59., London: SAGE Publications, Incorporated.
138. Wolpert R. & Mengersen K. (2004). Adjusted likelihoods for synthesizing empirical evidence from studies that differ in quality and design: effects of environmental tobacco smoke. Stat. Sci., 19, 450-471.
139. Xie M., Singh K. & Strawderman W.E. (2011). Confidence distributions and a unifying framework for meta-analysis. J. Amer. Statist. Assoc., 106(493): 320-333.
140. Yates F. & Cochran W. (1938). The analysis of groups of experiments. J. Agric. Sci, 28, 410-269.