Systematic reviews of research in education: aims, myths and multiple methods

Review of Education

Volumn 4, Issue 1, 2016, Pages 84-102

  Gough, David ^a   Thomas, James ^a  

^a UNIVERSITY COLLEGE LONDON   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 85179277408     PISSN: None     EISSN: 20496613     Source Type: Journal    
DOI: 10.1002/rev3.3068     Document Type: Article

References (60)

1. Akobeng, A.K. (2005) Understanding systematic reviews and meta-analysis, Archives of Disease in Childhood, 90(8), 845–848.
2. Ball, S.J. (2001) ‘You've been NERFed!’ Dumbing down the academy: National Educational Research Forum: ‘A national strategy? consultation paper’: A brief and bilious response, Journal of Education Policy, 16(3), 265–268.
3. Barnett-Page, E. & Thomas, J. (2009) Methods for the synthesis of qualitative research: A critical review, BMC Medical Research Methodology, 9(59), doi:10.1186/1471-2288-9-59.
4. Bastian, H., Glasziou, P. & Chalmers, I. (2010) Seventy-five trials and eleven systematic reviews a day: How will we ever keep up?, PLoS Med, 7(9), e1000326. doi:10.1371/journal.pmed.1000326.
5. Britten, N., Campbell, R., Pope, C., Donovan, J., Morgan, M. & Pil, R. (2002) Using meta ethnography to synthesise qualitative research: A worked example, Journal of Health Services Research & Policy, 7(4), 209–215.
6. Chalmers, I. (2003) Trying to do more good than harm in policy and practice: The role of rigorous, transparent, up-to-date evaluations, Annals of the American Academy of Political and Social Science, 589(1), 22–40.
7. Chandler, J., Churchil, R., Higgins, J., Lasseron, T. & Tovey, D. (2013) Methodological standards for the conduct of new Cochrane Intervention Reviews. Available online at: http://editorial-unit.cochrane.org/sites/editorial-unit.cochrane.org/files/uploads/MECIR_conduct_standards%202.3%2002122013.pdf (accessed 14 January 2016).
8. Commons Select Committee (2015) ‘Evidence check’ web forum, Available online at: http://www.parliament.uk/business/committees/committees-a-z/commons-select/education-committee/dfe-evidence-check-forum/ (accessed 14 February 2016).
9. Cooper, H., Hedges, L.V. & Valentine, J.C. (2009) The handbook of research synthesis and meta-analysis (2nd edn) (New York, Russell Sage Foundation).
10. Creswell, J. & Plano Clark, V. (2011) Designing and conducting mixed methods research (Thousand Oaks, CA, Sage).
11. Hargreaves, D.H. (1996) Teaching as a research-based profession: possibilities and prospects. Teacher Training Agency.
12. Davies, P. (1999) What is evidence-based education?, British Journal of Educational Studies, 47(2), 108–121.
13. Education Endowment Foundation (2015) Teaching and learning toolkit. Available online at: https://educationendowmentfoundation.org.uk/toolkit/toolkit-a-z/ (accessed 14 February 2016).
14. Elliott, J., Sim, I., Thomas, J., Owens, N., Dooley, G., Riis, J., Wallace, B., Thomas, J., Noel-Storr, A., Rada, G., Struthers, C., Howe, T., MacLehose, H., Brandt, L., Kunnamo, I. & Mavergames, C. (2014a) #CochraneTech: technology and the future of systematic reviews (editorial), Cochrane Library, doi:10.1002/14651858.ED000091.
15. Elliott, J.H., Turner, T., Clavisi, O., Thomas, J., Higgins, J.P.T., Mavergames, C. & Gruen, R.L. (2014b) Living systematic reviews: An emerging opportunity to narrow the evidence–practice gap, PLoS Med, 11(2), e1001603. doi:10.1371/journal.pmed.1001603.
16. Gough, D. (2013a) Meta-narrative and realist reviews: Guidance, rules, publication standards and quality appraisal, BMC Medicine, 11(22), doi:10.1186/1741-7015-11-22.
17. Gough, D. (2013b) Knowledge mobilisation in education in England, in: B. Levin, J. Qi, H. Edelstein & J. Sohn (Eds) The impact of research in education: An international perspective (Chicago, IL, University of Chicago Press).
18. Gough, D. & Elbourne, D. (2002) Systematic research synthesis to inform policy, practice and democratic debate, Social Policy and Society, 1(3), 225–236.
19. Gough, D. & Kenny, C. (2014) Social values underlying priority setting, paper presented at the 22nd Cochrane Colloquium, Hyderabad, 21–26 September 2014).
20. Gough, D., Oliver, S. & Thomas, J. (2012a) An introduction to systematic reviews (London, Sage).
21. Gough, D., Thomas, J. & Oliver, S. (2012b) Clarifying differences between review designs and methods, Systematic Reviews, 1(28), doi:10.1186/2046-4053-1-28.
22. Gough, D., Tripney, J., Kenny, C. & Buk-Berge, E. (2011) Evidence informed policy in education in Europe: EIPEE final project report (London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London).
23. Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., Kyriakidou, O. & Peacock, R. (2005) Storylines of research in diffusion of innovation: A meta-narrative approach to systematic review, Social Science & Medicine, 61(2), 417–430.
24. Guyatt, G.H., Oxman, A.D., Vist, G.E., Kunz, R., Falck-Ytter, Y., Alonso-Coello, P., & Schünemann, H.J. (2008) GRADE: An emerging consensus on rating quality of evidence and strength of recommendations, British Medical Journal, 336(7650), 924–926.
25. Hammersley, M. (2001) On ‘systematic’ reviews of research literatures: A ‘narrative’ response to Evans & Benefield, British Educational Research Journal, 27(5), 543–554.
26. Harden, A. & Thomas, J. (2010) Mixed methods and systematic reviews: Examples and emerging issues, in: A. Tashakkori & C. Teddlie (Eds) SAGE Handbook of Mixed Methods in Social & Behavioral Research (Thousand Oaks, CA, Sage), 749–774.
27. Heyvaert, M., Maes, B. & Onghena, P. (2011) Applying mixed methods research at the synthesis level: An overview, Research in the Schools, 18(1), 12–24.
28. James Lind Library (2015) Preparing and maintaining systematic reviews. Unbiased, up-to-date systematic reviews of all the relevant, reliable evidence are needed to provide trustworthy evidence to inform practice and policy. Available online at: http://www.jameslindlibrary.org/research-topics/bringing-it-all-together-for-the-benefit-of-patients-and-the-public/preparing-and-maintaining-systematic-reviews (accessed 14 February 2016).
29. Kelly, M.P., Heath, I., Howick, J. & Greenhalgh, T. (2015) The importance of values in evidence-based medicine, BMC Medical Ethics, 16(69), doi:10.1186/s12910-015-0063-3.
30. Marshall, I., Kuiper, J. & Wallace, B. (2014) Automating risk of bias assessment for clinical trials, in: Proceedings of the 5th ACM Conference on Bioinformatics, Computational Biology, and Health Informatics (New York, ACM), 88–95.
31. Moher, D., Liberati, A., Tetzlaff, J. & Altman, D.G. (2009) Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement, British Medical Journal, 339(b2535), doi:10.1136/bmj.b2535.
32. National Institute for Health and Care Excellence (2010) Measuring effectiveness and cost effectiveness: The QALY. Available online at: https://www.nice.org.uk/article/pmg9/chapter/glossary#quality-adjusted-life-year-qaly (accessed 14 February 2016).
33. Noblit, G. W. & Hare, R. D. (1988) Meta-ethnography: Synthesizing qualitative studies (Newbury Park, CA, Sage).
34. Noyes, J., Gough, D., Lewin, S., Mayhew, A., Michie, S., Pantoja, T., Petticrew, M., Pottie, K., Rehfuess, E., Shemilt, I., Shepperd, S., Sowden, A., Tugwell, P. & Welch, V. (2013) A research and development agenda for systematic reviews that ask complex questions about complex interventions, Journal of Clinical Epidemiology, 65(11), 1262–1270.
35. Nutley, S.M., Walter, I. & Davies, H.T.O. (2007) Using evidence: How research can inform public services (Bristol: Policy Press).
36. Nye E, Melendez-Torres G.J., Bonell C (2016) Origins, Methods, and advances in qualitative meta-synthesis. Review of Education.
37. O'Mara-Eves, A. & Thomas, J. (2016) Ongoing developments in meta-analytic and quantitative synthesis methods: Broadening the types of research questions that can be addressed, Review of Education.
38. O'Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M. & Ananiadou, S. (2015) Using text mining for study identification in systematic reviews: A systematic review of current approaches, Systematic Reviews, 4(5), doi:10.1186/2046-4053-4-5.
39. Oakley, A. (2003) Research evidence, knowledge management and educational practice: Early lessons from a systematic approach, London Review of Education, 1(1), 21–33.
40. Oakley, A., Gough, D., Oliver, S. & Thomas, J. (2005) The politics of evidence and methodology: Lessons from the EPPI-Centre, Evidence & Policy: A Journal of Research, Debate and Practice, 1(1), 5–32.
41. Oliver, S., Clarke-Jones, L., Rees, R., Milne, R., Buchanan, P., Gabbay, J., Gyte, G., Oakley, A. & Stein, K. (2004) Involving consumers in research and development agenda setting for the NHS: Developing an evidence-based approach, Health Technology Assessment, 8(15), 1–148.
42. Oliver, S., Rees, R., Clarke-Jones, L., Milne, R., Oakley, A., Gabbay, J., Stein, K., Buchanan, P. & Gyte, G. (2008) A multidimensional conceptual framework for analysing public involvement in health services research, Health Expectations, 11(1), 72–84.
43. Pawson, R. (2002) Evidence-based policy: The promise of ‘realist synthesis’, Evaluation, 8(3), 340–358.
44. Peersman, G. (1996) A descriptive mapping of health promotion studies in young people (London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London).
45. Pluye, P. & Hong, Q.N. (2014) Combining the power of stories and the power of numbers: Mixed methods research and mixed studies reviews, Annual Review of Public Health, 35(1), 29–45.
46. Riley, R.D., Lambert, P.C. & Abo-Zaid, G. (2010) Meta-analysis of individual participant data: rationale, conduct, and reporting, British Medical Journal, 340(c221), doi:10.1136/bmj.c221.
47. Sandelowski, M., Voils, C.I., Leeman, J. & Crandell, J.L. (2012) Mapping the mixed methods–mixed research synthesis terrain, Journal of Mixed Methods Research, 6(4), 317–331. doi:10.1177/1558689811427913.
48. Shemilt, I., McDaid, D., Marsh, K., Henderson, C., Bertranou, E., Mallander, J., Drummond, M., Mugford, M. & Vale, L. (2013) Issues in the incorporation of economic perspectives and evidence into Cochrane reviews, Systematic Reviews, 2(83), doi:10.1186/2046-4053-2-83.
49. Stansfield, C., Thomas, J. & Kavanagh, J. (2013) ‘Clustering’ documents automatically to support scoping reviews of research: A case study, Research Synthesis Methods, 4(3), 230–241.
50. Thomas, J. & Harden, A. (2008) Methods for the thematic synthesis of qualitative research in systematic reviews, BMC Medical Research Methodology, 8(45), doi:10.1186/1471-2288-8-45.
51. Thomas, J., Harden, A., Oakley, A., Oliver, S., Sutcliffe, K., Rees, R., Brunton, G. & Kavanagh, J. (2004) Integrating qualitative research with trials in systematic reviews: An example from public health, British Medical Journal, 328, 1010–1012.
52. Thomas, J., McNaught, J. & Ananiadou, S. (2011) Applications of text mining within systematic reviews, Research Synthesis Methods, 2(1), 1–14.
53. Thomas, J., Noel-Storr, A. & Elliott, J. (2015) Human and machine effort in Project Transform: How intersecting technologies will help us to identify studies reliably, efficiently and at scale, Cochrane Methods, Suppl. (1), 37–41. Available online at: http://www.cochranelibrary.com/dotAsset/f29304b6-4776-4e3f-a884-dc73df8933c2.pdf
54. Thomas, J., O'Mara-Eves, A. & Brunton, G. (2014) Using qualitative comparative analysis (QCA) in systematic reviews of complex interventions: a worked example, Systematic Reviews, 3(67), doi:10.1186/2046-4053-3-67.
55. Tsafnat, G., Dunn, A.G., Glasziou, P. & Coiera, E. (2013) The automation of systematic reviews, British Medical Journal, 345(7891), doi.org/10.1136/bmj.f139.
56. UK Government (1999) Modernising government. White Paper (London, HMSO).
57. Voils, C.I., Sandelowski, M., Barroso, J. & Hasselblad, V. (2008) Making sense of qualitative and quantitative findings in mixed research synthesis studies, Field Methods, 20(1), 3–25.
58. Wallace, B. C., Small, K., Brodley, C. E., Lau, J., Schmid, C. H., Bertram, L., Lill, C. M., Cohen, J. T. & Trikalinos, T. A. (2012) Toward modernizing the systematic review pipeline in genetics: Efficient updating via data mining, Genetics in Medicine, 14, 663–669.
59. Weiss, C.H. (1979) The many meanings of research utilization, Public Administration Review, 39(5), 426–431.
60. Whitty, G. (2006) Education(al) research and education policy making: Is conflict inevitable?, British Educational Research Journal, 32(2), 159–176.