On the reproducibility of meta-analyses: Six practical recommendations

BMC Psychology

Volumn 4, Issue 1, 2016, Pages

  Lakens, Daniël ^a   Hilgard, Joe ^b   Staaks, Janneke ^c  

^a EINDHOVEN UNIVERSITY OF TECHNOLOGY   (Netherlands)

^b UNIVERSITY OF MISSOURI   (United States)

^c UNIVERSITY OF AMSTERDAM   (Netherlands)

Author keywords

Meta analysis; Open science; Reporting guidelines; Reproducibility

Indexed keywords

HUMAN; META ANALYSIS (TOPIC); PUBLISHING; REPRODUCIBILITY; STATISTICAL ANALYSIS;

DATA INTERPRETATION, STATISTICAL; HUMANS; META-ANALYSIS AS TOPIC; PUBLICATION BIAS; REPRODUCIBILITY OF RESULTS;

EID: 85008452833     PISSN: None     EISSN: 20507283     Source Type: Journal    
DOI: 10.1186/s40359-016-0126-3     Document Type: Article

References (63)

1. American Psychological Association. Publication manual of the American psychological association. Washington DC: American Psychological Association; 2009.
2. Anderson CA, Shibuya A, Ihori N, Swing EL, Bushman BJ, Sakamoto A, Rothstein HR, Saleem M. Violent video game effects on aggression, empathy, and prosocial behavior in Eastern and Western countries: A meta-analytic review. Psychol Bull. 2010;136:151-73. doi:10.1037/a0018251.
3. Bem D, Tressoldi PE, Rabeyron T, Duggan M. Feeling the future: A meta-analysis of 90 experiments on the anomalous anticipation of random future events. 2014. Available at SSRN: http://ssrn.com/ abstract=2423692 Accessed 11 Apr 2014.
4. Booth A, Papaioannou D, Sutton A. Systematic approaches to a successful literature review. London: Sage; 2011.
5. Borenstein M, Hedges LV, Higgins JP, Rothstein HR. Introduction to meta-analysis. Chichester, UK: Wiley; 2009.
6. Brady SS, Matthews KA. Effects of media violence on health-related outcomes among young men. Arch Pediatr Adolesc Med. 2006;160(4):341-7. doi:10.1001/archpedi.160.4.341.
7. Braver SL, Thoemmes FJ, Rosenthal R. Continuously cumulating metaanalysis and replicability. Perspect Psychol Sci. 2014;9(3):333-42. doi:10.1177/ 1745691614529796.
8. Brooks MC. Press start: Exploring the effects of violent video games on boys. Diss Abstr Int. 2000;60:6419B.
9. Buscemi N, Hartling L, Vandermeer B, Tjosvold L, Klassen TP. Single data extraction generated more errors than double data extraction in systematic reviews. J Clin Epidemiol. 2006;59(7):697-703.
10. Bushman BJ, Rothstein HR, Anderson CA. Much ado about something: Violent video game effects and a school of red herring: Reply to Ferguson and Kilburn (2010). Psychol Bull. 2010;136(2):182-7. http://dx.doi.org/10. 1037/a0018718.
11. Carter EC, McCullough ME. Publication bias and the limited strength model of self-control: has the evidence for ego depletion been overestimated?. Front Psychol. 2014;5. doi:10.3389/fpsyg.2014.00823.
12. The Campbell Collaboration. Campbell Systematic Reviews: Policies and Guidelines. Campbell Syst Rev 2014. 2014a;Supplement 1. http://dx.doi.org/ 10.4073/csrs.2014.1.
13. The Campbell collaboration. Methodological Expectations of Campbell Collaboration Intervention Reviews (MEC2IR) DRAFT Version 0.1, June 23, 2014 (Updated September 2014). 2014. Retrieved from: http://www. campbellcollaboration.org/Methods_Resources/MEC2IR.php.
14. Campbell L, Loving TJ, LeBel EP, Enhancing Transparency of the Research Process to Increase Accuracy of Findings: A Guide for Relationship Researchers. 2014. Available at SSRN: http://ssrn.com/abstract=2435185 or http://dx.doi.org/10.2139/ssrn.2435185.
15. Chambers CD, Feredoes E, Muthukumaraswamy SD, Etchells P. Instead of "playing the game" it is time to change the rules: Registered Reports at AIMS Neuroscience and beyond. AIMS Neurosci. 2014;1(1):4-17.
16. Churches T. The benefits of reproducible research: a public health example. 2013. Retrieved from https://github.com/timchurches/meta-analyses/ blob/master/benefits-of-reproducible-research/benefits-of-reproducibleresearch. md.
17. The Cochrane Collaboration. Cochrane Handbook for Systematic Reviews of Interventions (Version 5.1.0), Higgins J, Green S, editors. 2011. Retrieved from http://www.cochrane-handbook.org. Accessed 11 Apr 2014.
18. Cohn LB. Violent video games: Aggression, arousal, and desensitization in young adolescent boys. University of Southern California: Doctoral dissertation; 1995.
19. Coyne JC, Kok RN. Salvaging psychotherapy research: a manifesto. J Evid-Based Psychotherapies. 2014;14(2):105-24.
20. Derry CJ, Derry S, McQuay HJ, Moore RA. Systematic review of systematic reviews of acupuncture published 1996-2005. Clin Med. 2006;6(4):381-6. http://dx.doi.org/10.7861/clinmedicine.6-4-381.
21. Dudden RF, Protzko SL. The systematic review team: contributions of the health sciences librarian. Med Ref Serv Q. 2011;30(3):301-15. http://dx.doi. org/10.1080/02763869.2011.590425.
22. EQUATOR Network. Library for health research reporting. 2014. Retrieved from: http://www.equator-network.org/library/. Accessed, 21 Oct 2014.
23. Ferguson CJ. Comment: Why meta-analyses rarely resolve ideological debates. Emot Rev. 2014;6:251-2.
24. Ferguson CJ. Violent video games don't exist. The Huffington Post. 2014. Retrieved from http://www.huffingtonpost.com/christopher-j-ferguson/ violent-video-games-dont-_b_5051733.html. Accessed 11 Apr 2014.
25. Ferguson CJ. Do Angry Birds make for angry children? A meta-analysis of video game influences on children's and adolescents' aggression, mental health, prosocial behavior and academic performance. Perspect Psychol Sci. (in press).
26. Ford AC, Guyatt GH, Talley NJ, Moayyedi P. Errors in the conduct of systematic reviews of pharmacological interventions for irritable bowel syndrome. Am J Gastroenterol. 2009;105(2):280-8.
27. Gandrud C. Reproducible research with R and RStudio. CRC Press. 2013.
28. Goodyear-Smith FA, van Driel ML, Arroll B, Del Mar C. Analysis of decisions made in meta-analyses of depression screening and the risk of confirmation bias: a case study. BMC Med Res Methodol. 2012;12(1):76. doi:10.1186/1471-2288-12-76.
29. Gorin SS, Krebs P, Badr H, Janke EA, Jim HS, Spring B, Jacobsen PB. Metaanalysis of psychosocial interventions to reduce pain in patients with cancer. J Clin Oncol. 2012;30(5);539-47. JCO-2011.
30. Gøtzsche PC, Hróbjartsson A, Marić K, Tendal B. Data extraction errors in meta-analyses that use standardized mean differences. JAMA. 2007;298(4):430-7.
31. Harris MR. The librarian's roles in the systematic review process: a case study. J Med Libr Assoc. 2005;93(1):81-7.
32. Hoffman, K. Effects of playing versus witnessing video game violence on attitudestoward aggression and acceptance of violence as a means of conflict resolution. Dissertation Abstracts International. 1995;56(03):747. (University Microfilms No. 9522426)
33. Horton J, Vandermeer B, Hartling L, Tjosvold L, Klassen TP, Buscemi N. Systematic review data extraction: cross-sectional study showed that experience did not increase accuracy. J Clin Epidemiol. 2010;63(3):289-98.
34. Iyengar S, Greenhouse JB. Selection models and the file drawer problem. Stat Sci. 1988;3:109-17.
35. Jauhar S, McKenna PJ, Radua J, Fung E, Salvador R, Laws KR. Cognitivebehavioural therapy for the symptoms of schizophrenia: systematic review and meta-analysis with examination of potential bias. Br J Psychiatry. 2014;204(1):20-9.
36. Jones AP, Remmington T, Williamson PR, Ashby D, Smyth RL. High prevalence but low impact of data extraction and reporting errors were found in Cochrane systematic reviews. J Clin Epidemiol. 2005;58:741-2.
37. Journal Article Reporting Standards Working Group. Reporting standards for research in psychology: Why do we need them? What might they be? Am Psychol. 2008;63:839-51.
38. Klein RA, Ratliff KA, Vianello M, Adams RB, Bahnik S, Bernstein MJ, Nosek BA. Investigating variation in replicability: A "Many Labs" replication project. Soc Psychol. 2014;45(3):142-52. doi:10.1027/1864-9335/a000178.
39. Lakens D, Evers ER. Sailing From the Seas of Chaos Into the Corridor of Stability Practical Recommendations to Increase the Informational Value of Studies. Perspect Psychol Sci. 2014;9(3):278-92.
40. Maggio LA, Tannery NH, Kanter SL. Reproducibility of literature search reporting in medical education reviews. Acad Med. 2011;86(8):1049-54.
41. Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med. 2009;6(6):e1000097. doi:10.1371/ journal. pmed1000097.
42. Nosek BA, Lakens D. Registered Reports. Soc Psychol. 2014;45(3):137-41. doi:10.1027/1864-9335/a000192.
43. Open Science Collaboration. An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspect Psychol Sci. 2012;7:657-60. doi:10.1177/1745691612462588.
44. Oyserman D, Lee SW. Does culture influence what and how we think? Effects of priming individualism and collectivism. Psychol Bull. 2008;134(2):311. doi: 10.1037/0033-2909.134.2.311.
45. Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L. Performance of the trim and fill method in the presence of publication bias and between-study heterogeneity. Stat Med. 2007;26(25):4544-62.
46. Pinfield S, Cox AM, Smith J. Research data management and libraries: relationships, activities, drivers and influences. PLoS One. 2014;9:e114734. doi: 10.1371/journal. pone.0114734.
47. R Core Team. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0. 2012. URL http://www.R-project.org/. Accessed 11 Apr 2014
48. Rothstein HR, Sutton AJ, Borenstein M, editors. Publication bias in metaanalysis: Prevention, assessment and adjustments. Wiley; 2006.
49. Rouder JN, Morey RD. A Bayes factor meta-analysis of Bem's ESP claim. Psychon Bull Rev. 2011;18(4):682-9. doi:10.3758/s13423-011-0088-7.
50. Scargle JD. Publication bias: the "file-drawer" problem in scientific inference. J Scientific Explor. 2000;14(1):91-106.
51. Sacks HS, Reitman D, Pagano D, Kupelnick B. Meta-analysis: an update. The Mount Sinai J Med, New York. 1995;63(3-4):216-24.
52. Simons DJ, Holcombe AO, Spellman BA. An Introduction to Registered Replication Reports at Perspectives on Psychological Science. Perspect Psychol Sci. 2014;9(5):552-5.
53. Schimmack U. The ironic effect of significant results on the credibility of multiple-study articles. Psychol Methods. 2012;17(4):551-66.
54. Sharpe D. Why the resistance to statistical innovations? Bridging the communication gap. Psychol Methods. 2013;18(4):572-82.
55. Simonsohn U, Nelson LD, Simmons JP. P-curve: A key to the file-drawer. J Exp Psychol Gen. 2014;143(2):534-47. doi:10.1037/a0033242.
56. Stanley TD, Doucouliagos H. Meta-regression approximations to reduce publication selection bias. Res Synth Methods. 2014;5:60-78. doi:10.1002/ jrsm.1095.
57. Sutton AJ, Abrams KR. Bayesian methods in meta-analysis and evidence synthesis. Stat Methods Med Res. 2001;10(4):277-303. doi:10.1177/ 096228020101000404.
58. Swinkels A, Briddon J, Hall J. Two physiotherapists, one librarian and a systematic literature review: collaboration in action. Health Inf Libr J. 2006; 23(4):248-56. http://dx.doi.org/10.1111/j.1471-1842.2006.00689.x.
59. Terrin N, Schmid CH, Lau J, Olkin I. Adjusting for publication bias in the presence of heterogeneity. Stat Med. 2003;22(13):2113-26.
60. University of York, Centre for Reviews and Dissemination. PROSPERO. International prospective register of systematic reviews. 2014. http://www. crd.york.ac.uk/PROSPERO/. Accessed 11 Apr 2014.
61. De Wit R, van Dam F, Zandbelt L, van Buuren A, van der Heijden K, Leenhouts G, Loonstra S. A pain education program for chronic cancer pain patients: follow-up results from a randomized controlled trial. Pain. 1997;73(1):55-69.
62. De Wit R, Van Dam F. From hospital to home care: a randomized controlled trial of a Pain Education Programme for cancer patients with chronic pain. J Adv Nurs. 2001;36(6):742-54.
63. De Wit R, van Dam F, Loonstra S, Zandbelt L, Buuren A, Heijden K, Abu-Saad HH. Improving the quality of pain treatment by a tailored pain education programme for cancer patients in chronic pain. Eur J Pain. 2001;5(3):241-56.