Ongoing developments in meta-analytic and quantitative synthesis methods: Broadening the types of research questions that can be addressed

Review of Education

Volumn 4, Issue 1, 2016, Pages 5-27

  O'Mara Eves, Alison ^a   Thomas, James ^a  

^a UNIVERSITY OF LONDON   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 85044602979     PISSN: None     EISSN: 20496613     Source Type: Journal    
DOI: 10.1002/rev3.3062     Document Type: Article

References (81)

1. Ahn, S., Ames, A.J. & Myers, N.D. (2012) A review of meta-analyses in education: Methodological strengths and weaknesses, Review of Educational Research, 82(4), 436–476.
2. Begg, C.B. & Mazumdar, M. (1994) Operating characteristics of a rank correlation test for publication bias, Biometrics, 50(4), 1088–1101.
3. Borenstein, M., Hedges, L., Higgins, J. & Rothstein, H. (2009) Introduction to meta analysis (Chichester, Wiley).
4. Bornmann, L., Lutz, R. & Daniel, H.D. (2007) Gender differences in grant peer review: A meta-analysis, Journal of Informetrics, 1(3), 226–238.
5. Brunton, G., O'Mara-Eves, A. & Thomas, J. (2014) The ‘active ingredients’ for successful community engagement with disadvantaged expectant and new mothers: A qualitative comparative analysis, Journal of Advanced Nursing, 70(12), 2847–2860.
6. Caldwell, D.M. (2014) An overview of conducting systematic reviews with network meta-analysis, Systematic Reviews, 3, 109. doi:10.1186/2046-4052-3-109.
7. Citrome, L. (2007) Show me the evidence: Using number needed to treat, Southern Medical Journal, 100(9), 881–884.
8. Cohen, J. (1977) Statistical power analysis for the behavioral sciences (New York, Academic Press).
9. Cohen, J. (1992) A power primer, Psychological Bulletin, 112(1), 155–159.
10. Cohn, L.D. & Becker, B.J. (2003) How meta-analysis increases statistical power, Psychological Methods, 8, 243–253.
11. Cooper, H.M., Hedges, L.V. & Valentine, J.C. (Eds) (2009) The handbook of research synthesis and meta-analysis, 2nd edn (New York, Russell Sage Foundation).
12. Cortina, J.M. (2003) Apples and oranges (and pears, oh my!): The search for moderators in meta-analysis, Organizational Research Methods, 6(4), 415–439.
13. Debray, T.P., Moons, K.G., Abo-Zaid, G.M., Koffijberg, H. & Riley, R.D. (2013) Individual participant data meta-analysis for a binary outcome: One-stage or two-stage?, PLoS One, 8(4), e60650. doi:10.1371/journal.pone.0060650.
14. Deeks, J.J., Altman, D.G. & Bradburn, M.J. (2001) Statistical methods for examining heterogeneity and combining results from several studies in meta-analysis, in: M. Egger, G.D. Smith & D.G. Altman (Eds) Systematic reviews in health care: Meta-analysis in context, 2nd edn (London, BMJ Publishing Group).
15. Duval, S. & Tweedie, R. (2000a) A nonparametric ‘trim and fill’ method of accounting for publication bias in meta-analysis, Journal of the American Statistical Association, 95(449), 89–98.
16. Duval, S. & Tweedie, R. (2000b) Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis, Biometrics, 56(2), 455–463.
17. Education Endowment Foundation (2014) Sutton Trust–EEF Teaching and Learning Toolkit (version September 2014) (London, Education Endowment Foundation).
18. Egger, M., Ebrahim, S. & Smith, G.D. (2002) Editorial: Where now for meta-analysis?, International Journal of Epidemiology, 31(1), 1–5.
19. Egger, M., Smith, G.D. & Phillips, A.N. (1997a) Meta-analysis: Principles and procedures, British Medical Journal, 315(7121), 1533–1537.
20. Egger, M., Smith, G.D., Schneider, M. & Minder, C. (1997b) Bias in meta-analysis detected by a simple, graphical test, British Medical Journal, 315(7109), 629–634.
21. Erez, A., Bloom, M.C. & Wells, M.T. (1996) Using random rather than fixed effects models in meta-analysis: Implications for situational specificity and validity generalization, Personnel Psychology, 49(2), 275–306.
22. Evans, D. (2012) ‘He's not the messiah …’: Interview with John Hattie, Times Educational Supplement Magazine, 14 September 2012. Available online at: https://www.tes.co.uk/article.aspx?storycode=6290240 (accessed 17 August 2015).
23. Eysenck, H.J. (1994) Systematic reviews: Meta-analysis and its problems, British Medical Journal, 309(6957), 789–792.
24. Glass, G.V. (1976) Primary, secondary and meta-analysis of research, Educational Researcher, 5(10), 3–8.
25. Glass, G.V. (2000) Meta-analysis at 25. http://www.gvglass.info/papers/meta25.html (Accessed on 25 February 2015).
26. Glass, G.V. (2015) Meta-analysis at middle age: A personal history, Research Synthesis Methods, 6(3), doi:10.1002/jrsm.1133.
27. Goldstein, H. (2003) Multilevel statistical models (3rd, rev edn (London, Hodder Arnold).
28. Goldstein, H., Yang, M., Omar, R., Turner, R. & Thompson, S. (2000) Meta-analysis using multilevel models with an application to the study of class size effects, Journal of the Royal Statistical Society: Series C (Applied Statistics), 49(3), 399–412. doi:10.1111/1467-9876.00200.
29. Gough, D., Thomas, J. & Oliver, S. (2012) Clarifying differences between review designs and methods, Systematic Reviews, 1(28), doi:10.1186/2046-4053-1-28.
30. Hafdahl, A. (2007) Combining correlation matrices: Simulation analysis of improved fixed-effects methods, Journal of Educational and Behavioral Statistics, 32(2), 180–205.
31. Haney, P. & Durlak, J.A. (1998) Changing self-esteem in children and adolescents: A meta-analytic review, Journal of Clinical Child Psychology, 27(4), 423–433.
32. Harden, A. (2010) Mixed-methods systematic reviews: Integrating quantitative and qualitative findings [monograph on the internet]. London, UK, FOCUS: Technical Brief No. 25; 2010. Available online at: www.ncddr.org/kt/products/focus/focus25/ (accessed 10 July 2015).
33. Harden, A. & Thomas, J. (2005) Methodological issues in combining diverse study types in systematic reviews, Internal Journal of Social Research Methodology, 8(3), 257–271.
34. Harden, A. & Thomas, J. (2010) Mixed methods and systematic reviews: Examples and emerging issues, in: A. Tashakkori, C. Teddlie (Eds) Sage handbook of mixed methods in social & behavioral research (vol. 2) (Thousand Oaks, CA, Sage), 749–774.
35. Hattie, J. (2008) Visible learning: A synthesis of over 800 meta-analyses relating to achievement (Abingdon, Oxon, Routledge).
36. Hattie, J. & Timperley, H. (2007) The power of feedback, Review of Educational Research, 77(1), 81–112.
37. Hedges, L.V. (2013) Recommendations for practice: Justifying claims of generalizability, Educational Psychology Review, 25, 331–337.
38. Hedges, L.V. & Vevea, J.L. (1998) Fixed- and random-effects models in meta-analysis, Psychological Methods, 3(4), 486–504.
39. Hedges, L. V., Tipton, E. & Johnson, M. C. (2010) Robust variance estimation in meta-regression with dependent effect size estimates, Research Synthesis Methods, 1(1), 39–65.[Erratum in 1(2): 164–165. doi:10.1002/jrsm.5.]
40. Higgins, J. & Green, S. (Eds) (2011) Cochrane handbook for systematic reviews of interventions version 5.1.0 (updated March 2011). Available online at: www.cochrane-handbook.org (accessed 15 June 2015).
41. Higgins, S. (2014) Meta-analysis, education research & the Sutton Trust–EEF Teaching and Learning Toolkit, paper presented at CEM's 30 Years of Evidence in Education conference, 23 September 2014, London, UK.
42. Hox, J.J. (2010) Multilevel analysis: Techniques and applications (2nd edn) (Quantitative Methodology Series) (Hove, Routledge).
43. Hox, J.J. & De Leeuw, E.D. (2003) Multilevel models for meta-analysis, in: S.P. Reise & N. Duan (Eds) Multilevel modelling: Methodological advances, issues, and applications (Mahwah, NJ, Lawrence Erlbaum Associates).
44. Hunter, J.E. & Schmidt, F.L. (2004) Methods of meta-analysis: Correcting error and bias in research findings (Thousand Oaks, CA, Sage).
45. Hutton, J.L. (2000) Number needed to treat: Properties and problem, Journal of the Royal Statistical Society Series A, 163(3), 403–419.
46. Ioannidis, J.P. & Trikalinos, T.A. (2007) The appropriateness of asymmetry tests for publication bias in meta-analyses: A large survey, Canadian Medical Association Journal, 176(8), 1091–1096.
47. Kalaian, S.A. & Kasim, R.M. (2008a) Applications of multilevel models for meta-analysis, in: A.A. O'Connell & D.B. McCoach (Eds) Multilevel analysis of educational data (Greenwich, CT, Information Age Publishing).
48. Kalaian, S.A. & Kasim, R.M. (2008b) Why multivariate meta-analysis methods for studies with multivariate outcomes?, paper presented at the Mid Western Educational Research Association Annual Meeting, Westin Great Southern Hotel, Columbus, Ohio, 15–18 October 2008.
49. Kalaian, S.A. & Raudenbush, S.W. (1996) A multivariate mixed linear model for meta-analysis, Psychological Methods, 1(3), 227–235.
50. Lau, J., Ioannidis, J.P.A., Terrin, N., Schmid, C.H. & Olkin, I. (2006) The case of the misleading funnel plot, British Medical Journal, 333(7568), 597–600.
51. Lipsey, M. & Wilson, D.B. (1993) The efficacy of psychological, educational, and behavioral treatment, American Psychologist, 48(12), 1181–1209.
52. Lipsey, M.W. & Wilson, D.B. (2001) Practical meta-analysis (Thousand Oaks, CA, Sage).
53. Lipsey, M. W., Puzio, K., Yun, C., Hebert, M. A., Steinka-Fry, K., Cole, M. W., Roberts, M., Anthony, K. S. & Busick, M. D. (2012) Translating the statistical representation of the effects of education interventions into more readily interpretable forms (NCSER 2013-3000) (Washington, DC, National Center for Special Education Research, Institute of Education Sciences, US Department of Education).
54. Marsh, H.W. & Bornmann, L. (2009) Do women have less success in peer review?, Nature, 459(7246), 602.
55. Marsh, H.W., Bornmann, L., Mutz, R., Daniel, H.-D. & O'Mara, A. (2009) Gender effects in the peer reviews of grant proposals: A comprehensive meta-analysis comparing traditional and multilevel approaches, Review of Educational Research, 79(3), 1290–1326.
56. Marsh, H. & O'Mara, A. (2008) Self-concept is as multidisciplinary as it is multidimensional: A review of theory, measurement, and practice in self-concept research, in: H. Marsh, R. Craven, D. McInerney (Eds) Self-processes, learning and enabling human potential. International advances in self research (Charlotte, NC, Information Age), 87–115.
57. Mills, E.J., Thorlund, K. & Ioannidis, J.P.A. (2013) Demystifying trial networks and network meta-analysis, British Medical Journal, 346, f2914. doi: 10.1136/bmj.f2914.
58. O'Mara, A. J. (2009) Methodological and substantive applications of meta-analysis: Multilevel modelling, simulation, and the construct validation of self-concept. Doctoral dissertation, University of Oxford, UK.
59. O'Mara, A.J., Marsh, H.W., Craven, R.G. & Debus, R. (2006) Do self-concept interventions make a difference?, A synergistic blend of construct validation and meta-analysis, Educational Psychologist, 41(3), 181–206.
60. O'Mara-Eves, A., Brunton, G., McDaid, D., Oliver, S., Kavanagh, J., Jamal, F., Matosevic, T., Harden, A. & Thomas, J. (2013) Community engagement to reduce inequalities in health: a systematic review, meta-analysis and economic analysis, Public Health Research, 1(4), 1–548.
61. O'Muircheartaigh, C. & Hedges, L.V. (2014) Generalizing from unrepresentative experiments: A stratified propensity score approach, Journal of the Royal Statistical Society: Series C, 63(2), 195–210.
62. Petticrew, M. (2015) Time to rethink the systematic review catechism? Moving from ‘what works’ to ‘what happens’, Systematic Reviews, 4(36), doi:10.1186/s13643-015-0027-1.
63. Ragin, C. (2008a) Redesigning social inquiry: Fuzzy sets and beyond (Chicago, IL, University of Chicago Press).
64. Ragin, C. (2008b) User's guide to fuzzy-set/qualitative comparative analysis (Tucson, AZ, University of Arizona. Available online at: http://www.u.arizona.edu/~cragin/fsQCA/download/fsQCAManual.pdf (accessed 17 August 2015).
65. Raudenbush, S.W. (1994) Random effects model, in: H. Cooper & L.V. Hedges (Eds) The handbook of research synthesis (New York, Russell Sage Foundation), 301–321.
66. Raudenbush, S.W. & Bryk, A.S. (1985) Empirical Bayes meta-analysis, Journal of Educational Statistics, 10, 75–98.
67. Raudenbush, S.W. & Bryk, A.S. (2002) Hierarchical linear models: Applications and data analysis methods, 2nd edn (Newbury Park, CA, Sage).
68. Riley, R., Lambert, P.C. & Abo-Zaid, G. (2010) Meta-analysis of individual participant data: rationale, conduct, and reporting, British Medical Journal, 340, c221. doi: 10.1136/bmj.c221.
69. Riley, R.D., Thompson, J.R. & Abrams, K.R. (2008) An alternative model for bivariate random-effects meta-analysis when the within-study correlations are unknown, Biostatistics, 9(1), 172–186.
70. Rosenthal, R. (1979) The file drawer problem and tolerance for null results, Psychological Bulletin, 86(3), 638–641.
71. Sandelowski, M., Voils, C.I. & Barroso, J. (2006) Defining and designing mixed research synthesis studies, Research in the Schools, 13(1), 29–44.
72. Stanley, T.D. (2008) Meta-regression methods for detecting and estimating empirical effects in the presence of publication selection, Oxford Bulletin of Economics and Statistics, 70(1), 103–127.
73. Sutton, A. (2009) Publication bias, in: H. Cooper, L.V. Hedges & J.C. Valentine (Eds) The handbook of research synthesis and meta-analysis, 2nd edn (New York, Russell Sage Foundation), 435–452.
74. Sutton, A.J., Song, F., Gilbody, S.M. & Abrams, K.R. (2000) Modelling publication bias in meta-analysis: A review, Statistical Methods in Medical Research, 9(5), 421–445.
75. Tabachnick, B.G. & Fidell, L.S. (2012) Using multivariate statistics, 6th edn (New York, Pearson).
76. Thomas, J., Harden, A., Oakley, A., Oliver, S., Sutcliffe, K., Rees, R., et al. (2004) Integrating qualitative research with trials in systematic reviews: An example from public health, British Medical Journal, 328(7446), 1010–1012.
77. Thomas, J., O'Mara-Eves, A. & Brunton, G. (2014) Using qualitative comparative analysis (QCA) in systematic reviews of complex interventions: A worked example, Systematic Reviews, 3, 67. doi:10.1186/2046-4053-3-67.
78. Thompson, S.G. & Higgins, J.P.T. (2002) How should meta-regression analyses be undertaken and interpreted?, Statistics in Medicine, 21(11), 1559–1574.
79. Thorlund, K. & Mills, E.J. (2012) Sample size and power considerations in network meta-analysis, Systematic Reviews, 1, 41. doi:10.1186/2046-4053-1-41.
80. Torgerson, C.J. (2006) Publication bias: The Achilles’ heel of systematic reviews?, British Journal of Educational Studies, 54(1).
81. Wolf, F.M. (1986) Meta-analysis: Quantitative methods for research synthesis (Beverly Hills, CA, Sage).