Systematic reviews and meta-analyses in the health sciences: Best practice methods for research syntheses

Social Science and Medicine

Volumn 233, Issue , 2019, Pages 237-251

  Johnson, Blair T ^a   Hennessy, Emily A ^a  

^a UNIVERSITY OF CONNECTICUT   (United States)

Author keywords

Evidence synthesis; Meta analysis; Methodological quality; Research synthesis; Risk of bias; Systematic reviews

Indexed keywords

ACADEMIC RESEARCH; BEST MANAGEMENT PRACTICE; FUTURE PROSPECT; HEALTH GEOGRAPHY; LITERATURE REVIEW; META-ANALYSIS; RESEARCH WORK; RISK ASSESSMENT;

CLINICAL OUTCOME; CLINICAL RESEARCH; GOOD CLINICAL PRACTICE; HEALTH CARE POLICY; HEALTH SCIENCE; HUMAN; METHODOLOGY; POPULATION RESEARCH; PREFERRED REPORTING ITEMS FOR SYSTEMATIC REVIEWS AND META-ANALYSES; QUALITY CONTROL; RESEARCH; REVIEW; SAMPLING; STANDARD; STUDY DESIGN; SYSTEMATIC REVIEW; LITERATURE; MEDICAL RESEARCH; META ANALYSIS; PRACTICE GUIDELINE; PUBLISHING;

BIOMEDICAL RESEARCH; EDITORIAL POLICIES; GUIDELINES AS TOPIC; HUMANS; PUBLISHING; RESEARCH DESIGN; REVIEW LITERATURE AS TOPIC;

EID: 85067571539     PISSN: 02779536     EISSN: 18735347     Source Type: Journal    
DOI: 10.1016/j.socscimed.2019.05.035     Document Type: Review

References (112)

1. Balshem, H., Helfand, M., Schünemann, H.J., Oxman, A.D., Kunz, R., Brozek, J., Vist, G.E., Falck-Ytter, Y., Meerpohl, J., Norris, S., Grade guidelines: 3. Rating the quality of evidence. J. Clin. Epidemiol. 64:4 (2011), 401–406.
2. Bayes, T., Price, R., Canton, J., An Essay towards Solving a Problem in the Doctrine of Chances. 1763.
3. Becker, B.J., Failsafe N or file-drawer number. Rothstein, H.R., Sutton, A.J., Borenstein, M., (eds.) Publication Bias in Meta-Analysis Prevention, Assessment and Adjustments, 2005, Wiley Chichester, England, 111–125.
4. Begg, C.B., Mazumdar, M., Operating characteristics of a rank correlation test for publication bias. Biometrics 50:4 (1994), 1088–1101.
5. Booth, A., Clear and present questions: formulating questions for evidence based practice. Libr. Hi Tech 24:3 (2006), 355–368.
6. Borenstein, M., Hedges, L.V., Higgins, J.P., Rothstein, H.R., Introduction to Meta-Analysis. 2011, John Wiley & Sons.
7. Borenstein, M., Higgins, J.P.T., Hedges, L.V., Rothstein, H.R., Basics of meta‐analysis: I2 is not an absolute measure of heterogeneity. Res. Synth. Methods 8:1 (2017), 5–18.
8. Canfield, S.E., Dahm, P., Rating the quality of evidence and the strength of recommendations using GRADE. World J. Urol. 29:3 (2011), 311–317.
9. Card, N.A., Applied Meta-Analysis for Social Science Research. 2015, Guilford Publications.
10. Carter, B.U., Single screen of citations with excluded terms: an approach to citation screening in systematic reviews. Syst. Rev., 7(1), 2018 111-018-0782-x.
11. Chalmers, I., Hedges, L.V., Cooper, H., A brief history of research synthesis. Eval. Health Prof. 25:1 (2002), 12–37.
12. Cheung, M.W., Modeling dependent effect sizes with three-level meta-analyses: a structural equation modeling approach. Psychol. Methods 19:2 (2014), 211–229.
13. Cohen, B.H., Calculating a factorial ANOVA from means and standard deviations. Understand. Stat.: Statistical Issues in Psychology, Education, and the Social Sciences 1:3 (2002), 191–203.
14. Cooke, A., Smith, D., Booth, A., Beyond PICO: the SPIDER tool for qualitative evidence synthesis. Qual. Health Res. 22:10 (2012), 1435–1443.
15. Cotterill, S., Knowles, S., Martindale, A., Elvey, R., Howard, S., Coupe, N., Wilson, P., Spence, M., Getting Messier with TIDieR: Embracing Context and Complexity in Intervention Reporting. 2018.
16. De Barra, M., Reporting bias inflates the reputation of medical treatments: a comparison of outcomes in clinical trials and online product reviews. Soc. Sci. Med. 177 (2017), 248–255.
17. De Vibe, M., Bjørndal, A., Tipton, E., Hammerstrøm, K.T., Kowalski, K., Mindfulness based stress reduction (MBSR) for improving health, quality of life, and social functioning in adults. Campbell Syst. Rev., 11, 2017, 10.4073/csr.2017.11.
18. Deeks, J., Dinnes, J., D'amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., Altman, D., Evaluating non-randomised intervention studies. Health Technol. Assess., 7(27), 2003 iii-x, 1-173.
19. Delaney, A., Tamás, P.A., Searching for evidence or approval? A commentary on database search in systematic reviews and alternative information retrieval methodologies. Res. Synth. Methods 9:1 (2018), 124–131.
20. Denyer, D., Tranfield, D., Van Aken, J.E., Developing design propositions through research synthesis. Organ. Stud. 29:3 (2008), 393–413.
21. Dwan, K., Altman, D.G., Arnaiz, J.A., Bloom, J., Chan, A.W., Cronin, E., Decullier, E., Easterbrook, P.J., Von Elm, E., Gamble, C., Ghersi, D., Ioannidis, J.P., Simes, J., Williamson, P.R., Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One, 3(8), 2008, e3081.
22. Easterbrook, P.J., Berlin, J.A., Gopalan, R., Matthews, D.R., Publication bias in clinical research. Lancet 337:8746 (1991), 867–872.
23. Ebrahim, S., Bance, S., Athale, A., Malachowski, C., Ioannidis, J.P., Meta-analyses with industry involvement are massively published and report no caveats for antidepressants. J. Clin. Epidemiol. 70 (2016), 155–163.
24. Egger, M., Smith, G., Schneider, M., Minder, C., Bias in meta-analysis detected by a simple, graphical test. Br. Med. J. 315:7109 (1997), 629–634.
25. Elliott, J.H., Synnot, A., Turner, T., Simmonds, M., Akl, E.A., Mcdonald, S., Salanti, G., Meerpohl, J., Maclehose, H., Hilton, J., Tovey, D., Shemilt, I., Thomas, J., Living Systematic Review Network. Living systematic review: 1. Introduction-the why, what, when, and how. J. Clin. Epidemiol. 91 (2017), 23–30.
26. European Food, S.A., Application of systematic review methodology to food and feed safety assessments to support decision making. Efsa guidance for those carrying out systematic reviews. Eur. Food Saf. Auth. J. 8:6 (2010), 1–89.
27. Fanelli, D., Negative results are disappearing from most disciplines and countries. Scientometrics 90:3 (2012), 891–904.
28. Fanelli, D., Ioannidis, J.P., US studies may overestimate effect sizes in softer research. Proc. Natl. Acad. Sci. U.S.A. 110:37 (2013), 15031–15036.
29. Franco, A., Malhotra, N., Simonovits, G., Social science. Publication bias in the social sciences: unlocking the file drawer. Science 345:6203 (2014), 1502–1505.
30. Garner, P., Hopewell, S., Chandler, J., Maclehose, H., Schünemann, H.,J., Akl, E.A., Beyene, J., Chang, S., Churchill, R., Dearness, K., Guyatt, G., Lefebvre, C., Liles, B., Marshall, R., Martínez García, L., Mavergames, C., Nasser, M., Qaseem, A., Sampson, M., Soares-Weiser, K., Takwoingi, Y., Thabane, L., Trivella, M., Tugwell, P., Welsh, E., Wilson, E.C., When and How to Update Systematic Reviews: Consensus and Checklist. 2016, British Medical Journal Publishing Group.
31. Glass, G.V., Primary, secondary, and meta-analysis of research. Educ. Res. 5:10 (1976), 3–8.
32. Grant, M.J., Booth, A., A typology of reviews: an analysis of 14 review types and associated methodologies. Health Inf. Libr. J. 26:2 (2009), 91–108.
33. Hanna, J.M., The Quality of Education Leadership Doctoral Dissertations in the United States: an Empirical Review., 2015, Doctoral Dissertation edn. Marshall University.
34. Haynes, R.B., Forming research questions. J. Clin. Epidemiol. 59:9 (2006), 881–886.
35. Hedges, L.V., Effect sizes in cluster-randomized designs. J. Educ. Behav. Stat. 32:4 (2007), 341–370.
36. Hedges, L.V., Olkin, I., Statistical Methods for Meta-Analysis. 1985, Academic Press Inc, Orlando FL, USA.
37. Hedges, L.V., Tipton, E., Johnson, M.C., Robust variance estimation in meta-regression with dependent effect size estimates. Res. Synth. Methods 1:1 (2010), 39–65.
38. Hennessy, E.A., Johnson, B.T., Keenan, C., Best practice guidelines and essential steps to conduct rigorous and systematic meta-reviews. Appl. Psychol. Health Well-Being, 2019 (in press).
39. Hennessy, E.A., Tanner-Smith, E.E., Effectiveness of brief school-based interventions for adolescents: a meta-analysis of alcohol use prevention programs. Prev. Sci. Official J. Soc. Prev. Res. 16:3 (2015), 463–474.
40. Higgins, J.P., (eds.) Cochrane Handbook for Systematic Reviews of Interventions, 5 edn., 2008, Wiley-Blackwell, Chichester.
41. Higgins, J., Thompson, S., Deeks, J., Altman, D., Statistical heterogeneity in systematic reviews of clinical trials: a critical appraisal of guidelines and practice. J. Health Serv. Res. Policy 7:1 (2002), 51–61.
42. Higgins, J.P., Thompson, S.G., Deeks, J.J., Altman, D.G., Measuring inconsistency in meta-analyses. BMJ Br. Med. J. (Clin. Res. Ed.) 327:7414 (2003), 557–560.
43. Higgins, J.P., Altman, D.G., Gotzsche, P.C., Juni, P., Moher, D., Oxman, A.D., Savovic, J., Schulz, K.F., Weeks, L., Sterne, J.A., Cochrane Bias Methods Group and Cochrane Statistical Methods Group. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ (Clin. Res. Ed.), 343, 2011, d5928.
44. Hoaglin, D.C., Practical challenges of I(2) as a measure of heterogeneity. Res. Synth. Methods, 8(3), 2017, 254.
45. Hoffmann, T., Glasziou, P., Boutron, I., Milne, R., Perera, R., Moher, D., Altman, D., Barbour, V., Macdonald, H., Johnston, M., Lamb, S., DIXON - Woods, M., Mcculloch, P., Wyatt, J., Chan, A., Michie, S., Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. Br. Med. J., 2014.
46. Hozo, S.P., Djulbegovic, B., Hozo, I., Estimating the mean and variance from the median, range, and the size of a sample. BMC Med. Res. Methodol., 5, 2005 13-2288-5-13.
47. Ioannidis, J.P., Meta-research: the art of getting it wrong. Res. Synth. Methods 1:3–4 (2010), 169–184.
48. Ioannidis, J.P., The mass production of redundant, misleading, and conflicted systematic reviews and meta‐analyses. Milbank Q. 94:3 (2016), 485–514.
49. Ioannidis, J.P., Does evidence-based hearsay determine the use of medical treatments?. Soc. Sci. Med. 177 (2017), 256–258.
50. Joanna Briggs Institute, Joanna Briggs Institute Reviewers' Manual: 2011 Edition, 2011, University of Adelaide, Adelaide, South Australia.
51. Johnson, B.T., Eagly, A.H., Meta-analysis of social-personality psychological research. Reis, H.T., (eds.) Handbook of Research Methods in Social and Personality Psychology, second ed., 2014, Cambridge University Press, London, 675–707.
52. Johnson, B.T., Huedo-Medina, T.B., Depicting estimates using the intercept in meta-regression models: the moving constant technique. Res. Synth. Methods 2:3 (2011), 204–220.
53. Johnson, B.T., Low, R.E., Macdonald, H.V., Panning for the gold in health research: incorporating studies' methodological quality in meta-analysis. Psychol. Health 30:1 (2015), 135–152.
54. Johnson, B.T., Cromley, E.K., Marrouch, N., Spatiotemporal meta-analysis: reviewing health psychology phenomena over space and time. Health Psychol. Rev. 11:3 (2017), 280–291.
55. Kaufman, M.R., Cornish, F., Zimmerman, R.S., Johnson, B.T., Health behavior change models for HIV prevention and AIDS care: practical recommendations for a multi-level approach. J. Acquir. Immune Defic. Syndr. 66:Suppl. 3 (2014), S250–S258.
56. Kenny, D.A., Correlation and Causality. 1979, Wiley, New York 1979.
57. Kirkham, J.J., Altman, D.G., Williamson, P.R., Bias due to changes in specified outcomes during the systematic review process. PLoS One, 5(3), 2010, e9810.
58. Kirkham, J.J., Dwan, K.M., Altman, D.G., Gamble, C., Dodd, S., Smyth, R., Williamson, P.R., The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ, 340, 2010, c365.
59. Knottnerus, J.A., Tugwell, P., The way in which effects are analyzed and communicated can make a difference for decision making. J. Clin. Epidemiol. 72 (2016), 1–3.
60. Kugley, S., Wade, A., Thomas, J., Mahood, Q., Jorgensen, A.M.K., Hammerstrom, K., Sathe, N., Searching for Studies: A Guide to Information Retrieval for Campbell. 2016, The Campbell Collaboration, Oslo.
61. Lai, N.M., Teng, C.L., Lee, M.L., Interpreting systematic reviews: are we ready to make our own conclusions? A cross-sectional study. BMC Med., 9(1), 2011 30-30.
62. Lasserson, T., Churchill, R., Chandler, J., Tovey, D., Higgins, J.P.T., Standards for the reporting of protocols of new Cochrane Intervention reviews. Higgins, J.P.T., Lasserson, T., Chandler, J., Tovey, D., Churchill, R., (eds.) Methodological Expectations of Cochrane Intervention Reviews, 2016, Cochrane, London.
63. Lau, J., Ioannidis, J.P., Terrin, N., Schmid, C.H., Olkin, I., The case of the misleading funnel plot. BMJ 333:7568 (2006), 597–600.
64. Lennon, C.A., Huedo-Medina, T.B., Gerwien, D.P., Johnson, B.T., A role for depression in sexual risk reduction for women? A meta-analysis of HIV prevention trials with depression outcomes. Soc. Sci. Med. 75:4 (2012), 688–698.
65. Lewin, S., Glenton, C., Munthe-Kaas, H., Carlsen, B., Colvin, C.J., Gulmezoglu, M., Noyes, J., Booth, A., Garside, R., Rashidian, A., Using qualitative evidence in decision making for health and social interventions: an approach to assess confidence in findings from qualitative evidence syntheses (GRADE-CERQual). PLoS Med., 12(10), 2015, e1001895.
66. Lipsey, M., Wilson, D.B., Practical Meta-Analysis. 2001, Sage Publications, Thousand Oaks Ca, USA.
67. Macdonald, H.V., Johnson, B.T., Huedo-Medina, T.B., Livingston, J., Forsyth, K.C., Kraemer, W.J., Farinatti, P.T., Pescatello, L.S., Dynamic resistance training as stand-alone antihypertensive lifestyle therapy: a meta-analysis. Journal of the American Heart Association, 5(10), 2016, 10.1161/JAHA.116.003231.
68. Marrouch, N., Johnson, B.T., Geocoding: using space to enhance social psychological research. Blanton, H., Lacroix, L.M., Webster, G.D., (eds.) Measurement in Social Psychology, 2019, Psychology Press, New York, 201–227.
69. Marshall, I.J., Noel‐Storr, A., Kuiper, J., Thomas, J., Wallace, B.C., Machine learning for identifying randomized controlled trials: an evaluation and practitioner's guide. Res. Synth. Methods 9:4 (2018), 602–614.
70. Mckenzie, Methodological Issues in Meta-Analysis of Randomized Controlled Trials with Continuous Outcomes. PhD edn, 2011, Monash University, Melbourne, Australia.
71. Michie, S., Richardson, M., Johnston, M., Abraham, C., Francis, J., Hardeman, W., Eccles, M.P., Cane, J., Wood, C.E., The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: building an international consensus for the reporting of behavior change interventions. Ann. Behav. Med. 46:1 (2013), 81–95.
72. Moher, D., Liberati, A., Tetzlaff, J., Altman, D.G., PRISMA Group, Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. J. Clin. Epidemiol. 62:10 (2009), 1006–1012.
73. Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L.A., Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst. Rev., 4(1), 2015, 1.
74. Mortensen, M.L., Adam, G.P., Trikalinos, T.A., Kraska, T., Wallace, B.C., An exploration of crowdsourcing citation screening for systematic reviews. Res. Synth. Methods 8:3 (2017), 366–386.
75. Munafo, M.R., Stothart, G., Flint, J., Bias in genetic association studies and impact factor. Mol. Psychiatry 14:2 (2009), 119–120.
76. Nakagawa, S., Noble, D.W., Senior, A.M., Lagisz, M., Meta-evaluation of meta-analysis: ten appraisal questions for biologists. BMC Biol., 15(1), 2017, 18.
77. O'Neill, J., Tabish, H., Welch, V., Petticrew, M., Pottie, K., Clarke, M., Evans, T., PARDO Pardo, J., Waters, E., White, H., Tugwell, P., Applying an equity lens to interventions: using PROGRESS ensures consideration of socially stratifying factors to illuminate inequities in health. J. Clin. Epidemiol. 67:1 (2014), 56–64.
78. Page, M.J., Mckenzie, J.E., Forbes, A., Many scenarios exist for selective inclusion and reporting of results in randomized trials and systematic reviews. J. Clin. Epidemiol. 66:5 (2013), 524–537.
79. Pan, Z., Trikalinos, T.A., Kavvoura, F.K., Lau, J., Ioannidis, J.P., Local literature bias in genetic epidemiology: an empirical evaluation of the Chinese literature. PLoS Med., 2(12), 2005, e334.
80. Paynter, R., Banez, L.L., Erinoff, E., Lege-Matsuura, J., Potter, S., Commentary on EPC methods: an exploration of the use of text-mining software in systematic reviews. J. Clin. Epidemiol. 84 (2017), 33–36.
81. Polanin, J.R., Hennessy, E.A., Tanner-Smith, E.E., A review of meta-analysis packages in R. J. Educ. Behav. Stat. 42:2 (2017), 206–242.
82. Przybyla, P., Brockmeier, A.J., Kontonatsios, G., LE Pogam, M.A., Mcnaught, J., Von Elm, E., Nolan, K., Ananiadou, S., Prioritising references for systematic reviews with RobotAnalyst: a user study. Res. Synth. Methods 9:3 (2018), 470–488.
83. Rathbone, J., Hoffmann, T., Glasziou, P., Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers. Syst. Rev., 4, 2015 80-8015-0067-6.
84. Reid, A.E., Dovidio, J.F., Ballester, E., Johnson, B.T., HIV prevention interventions to reduce sexual risk for African Americans: the influence of community-level stigma and psychological processes. Soc. Sci. Med.(103), 2014, 118–125 1982.
85. Rubin, M., When does HARKing hurt? Identifying when different types of undisclosed post hoc hypothesizing harm scientific progress. Rev. Gen. Psychol., 21(4), 2017, 308.
86. Rucker, G., Schwarzer, G., Carpenter, J.R., Schumacher, M., Undue reliance on I(2) in assessing heterogeneity may mislead. BMC Med. Res. Methodol., 8, 2008 79-2288-8-79.
87. Schmid, C.H., Heterogeneity: multiplicative, additive or both?. Res. Synth. Methods 8:1 (2017), 119–120.
88. Schmidt, F.L., Hunter, J.E., Methods of Meta-Analysis: Correcting Error and Bias in Research Findings. third ed., 2015, Sage publications, Thousand Oaks, CA.
89. Schwarzer, G., Schumacher, M., Rücker, G., Sole Reliance on I 2 May Mislead. 2017, BMJ Publishing Group Ltd and British Cardiovascular Society.
90. Shadish, W.R., Lecy, J.D., The meta‐analytic big bang. Res. Synth. Methods 6:3 (2015), 246–264.
91. Shea, B.J., Reeves, B.C., Wells, G., Thuku, M., Hamel, C., Moran, J., Moher, D., Tugwell, P., Welch, V., Kristjansson, E., AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ, 358, 2017, j4008.
92. Shojania, K.G., Sampson, M., Ansari, M.T., Ji, J., Doucette, S., Moher, D., How quickly do systematic reviews go out of date? A survival analysis. Ann. Intern. Med. 147:4 (2007), 224–233.
93. Siddaway, A.P., Wood, A.M., Hedges, L.V., How to do a systematic review: a best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses. Annu. Rev. Psychol. 70 (2019), 747–770.
94. Simpson, R., Pearson, K., Report on certain enteric fever inoculation statistics. Br. Med. J., 1904, 1243–1246.
95. Stanley, T.D., Doucouliagos, H., Meta‐regression approximations to reduce publication selection bias. Res. Synth. Methods 5:1 (2014), 60–78.
96. Stanley, T.D., Doucouliagos, H., Neither fixed nor random: weighted least squares meta-analysis. Stat. Med. 34:13 (2015), 2116–2127.
97. Sterne, J.A.C., Higgins, J.P.T., Reeves, B.C., ON BEHALF OF THE DEVELOPMENT GROUP FOR ROBINS-I. A Tool for Assessing Risk of Bias in Non-randomized Studies of Interventions. 7 edn, 2016.
98. Stigler, S.M., The History of Statistics: the Measurement of Uncertainty before 1900. 1986, Harvard University Press.
99. Strutt, J.W., Presidential Address at the 54th Meeting of the British Association for the Advancement of Science. 1884, John Murray, London 1884.
100. Sumner, J.A., Carey, R.N., Michie, S., Johnston, M., Edmondson, D., Davidson, K.W., Using rigorous methods to advance behaviour change science (vol 2, pg 797, 2018). Nature Human Behaviour, 3(1), 2019 101-101.
101. Tipton, E., Pustejovsky, J.E., Ahmadi, H., Current Practices in Meta‐regression in Psychology, Education, and Medicine. 2019, Research synthesis methods.
102. Tricco, A.C., Lillie, E., Zarin, W., O'brien, K., Colquhoun, H., Kastner, M., Levac, D., Ng, C., Sharpe, J.P., Wilson, K., Kenny, M., Warren, R., Wilson, C., Stelfox, H.T., Straus, S.E., A scoping review on the conduct and reporting of scoping reviews. BMC Med. Res. Methodol., 16, 2016 15-016-0116-4.
103. Valentine, J.C., Judging the Quality of Primary Research. The Handbook of Research Synthesis and Meta-Analysis. 2009, 129–146.
104. Van Bavel, J.J., Mende-Siedlecki, P., Brady, W.J., Reinero, D.A., Contextual sensitivity in scientific reproducibility. Proc. Natl. Acad. Sci. Unit. States Am. 113:23 (2016), 6454–6459.
105. Viechtbauer, W., Conducting meta-analyses in R with the metafor package. J. Stat. Softw., 36(3), 2010.
106. Viechtbauer, W., Cheung, M.W., Outlier and influence diagnostics for meta‐analysis. Res. Synth. Methods 1:2 (2010), 112–125.
107. Von Hippel, P.,T., The heterogeneity statistic I 2 can be biased in small meta-analyses. BMC Med. Res. Methodol., 15(1), 2015.
108. Whiting, P., Savovic, J., Higgins, J.P., Caldwell, D.M., Reeves, B.C., Shea, B., Davies, P., Kleijnen, J., Churchill, R., ROBIS Group, ROBIS: a new tool to assess risk of bias in systematic reviews was developed. J. Clin. Epidemiol. 69 (2016), 225–234.
109. Wildridge, V., Bell, L., How CLIP became ECLIPSE: a mnemonic to assist in searching for health policy/management information. Health Inf. Libr. J. 19:2 (2002), 113–115.
110. Williamson, P.R., Gamble, C., Altman, D.G., Hutton, J.L., Outcome selection bias in meta-analysis. Stat. Methods Med. Res. 14:5 (2005), 515–524.
111. Wilson, D.B., Practical meta-analysis effect size calculator [online calculator]. n.d.-last update, Available: https://www.campbellcollaboration.org/effect-size-calculato.html. (Accessed 4 August 2018)
112. Yavchitz, A., Ravaud, P., Altman, D.G., Moher, D., Hrobjartsson, A., Lasserson, T., Boutron, I., A new classification of spin in systematic reviews and meta-analyses was developed and ranked according to the severity. J. Clin. Epidemiol. 75 (2016), 56–65.