Few studies exist examining methods for selecting studies, abstracting data, and appraising quality in a systematic review

Journal of Clinical Epidemiology

Volumn 106, Issue , 2019, Pages 121-135

  Robson, Reid C ^a   Pham, Ba’ ^a   Hwee, Jeremiah ^b   Thomas, Sonia M ^a   Rios, Patricia ^a   Page, Matthew J ^c   Tricco, Andrea C ^a,b  

^a ST MICHAEL'S HOSPITAL   (Canada)

^b UNIVERSITY OF TORONTO   (Canada)

^c MONASH UNIVERSITY   (Australia)

Author keywords

Data abstraction; Knowledge synthesis; Methodology; Quality appraisal; Study selection; Systematic reviews

Indexed keywords

COCHRANE LIBRARY; DATA EXTRACTION; DIAGNOSTIC ACCURACY; DIAGNOSTIC TEST ACCURACY STUDY; EMBASE; HUMAN; MEDLINE; PRACTICE GUIDELINE; QUALITY CONTROL; RELIABILITY; REVIEW; SOFTWARE; SYNTHESIS; SYSTEMATIC REVIEW; CROSS-SECTIONAL STUDY; DOCUMENTATION; RANDOMIZED CONTROLLED TRIAL (TOPIC);

ABSTRACTING AND INDEXING; CROSS-SECTIONAL STUDIES; HUMANS; RANDOMIZED CONTROLLED TRIALS AS TOPIC; SYSTEMATIC REVIEWS AS TOPIC;

EID: 85057106568     PISSN: 08954356     EISSN: 18785921     Source Type: Journal    
DOI: 10.1016/j.jclinepi.2018.10.003     Document Type: Review

References (61)

1. Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev, 4, 2015, 1.
2. Higgins, J., Green, S., Cochrane handbook for systematic reviews of interventions version 5.1.0. 2011, The Cochrane Collaboration Available at http://handbook-5-1.cochrane.org/.
3. Systematic reviews: CRD's guidance for undertaking reviews in health care, 2009, University of York, Centre for Reviews & Dissemination Available at https://www.york.ac.uk/media/crd/Systematic_Reviews.pdf.
4. Owens, D., Lohr, K.N., Atkins, D., Treadwell, J., Reston, J., Bass, E., et al. Methods guide for effectiveness and comparative effectiveness reviews. 2014, Agency for Healthcare Research and Quality, Rockville, MD.
5. Institute of medicine committee on standards for systematic reviews of comparative effectiveness R. Eden, J., Levit, L., Berg, A., Morton, S., (eds.) Finding what works in health care: standards for systematic reviews, 2011, National Academies Press (US) Copyright 2011 by the National Academy of Sciences. All rights reserved., Washington (DC).
6. Moher, D., Liberati, A., Tetzlaff, J., Altman, D.G., Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med 4 (2009), 264–269.
7. Joanna Briggs Institute Reviewers’ Manual: 2015 edition/supplement, 2015, Methodology for JBI Scoping Reviews, Australia.
8. Methodological expectations of Campbell Collaboration intervention reviews: conduct standards, 2017 Available at https://campbellcollaboration.org/library/campbell-methods-conduct-standards.html.
9. McGowan, J., Sampson, M., Salzwedel, D.M., Cogo, E., Foerster, V., Lefebvre, C., PRESS peer review of electronic search strategies: 2015 guideline statement. J Clin Epidemiol 75 (2016), 40–46.
10. Higgins, J.P., Altman, D.G., Gotzsche, P.C., Juni, P., Moher, D., Oxman, A.D., et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ, 343, 2011, d5928.
11. Wells G, Shea B, O'Connell D, Peterson J, Welch V, Losos M, et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. The Ottawa Hospital Research Institute: Ottawa, ON.
12. Whiting, P., Rutjes, A.W., Reitsma, J.B., Bossuyt, P.M., Kleijnen, J., The development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews. BMC Med Res Methodol, 3, 2003, 25.
13. Lucas, N.P., Macaskill, P., Irwig, L., Bogduk, N., The development of a quality appraisal tool for studies of diagnostic reliability (QAREL). J Clin Epidemiol 63 (2010), 854–861.
14. Moher, D., Hopewell, S., Schulz, K.F., Montori, V., Gotzsche, P.C., Devereaux, P.J., et al. CONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trials. Int J Surg 10 (2012), 28–55.
15. Vandenbroucke, J.P., Elm, E., Altman, D.G., Gøtzsche, P.C., Mulrow, C.D., Pocock, S.J., et al. Strengthening the reporting of observational studies in epidemiology (strobe): explanation and elaboration. Ann Intern Med 147 (2007), W163–W194.
16. Mathes, T., Klassen, P., Pieper, D., Frequency of data extraction errors and methods to increase data extraction quality: a methodological review. BMC Med Res Methodol, 17, 2017, 152.
17. Tricco, A.C., Antony, J., Zarin, W., Strifler, L., Ghassemi, M., Ivory, J., et al. A scoping review of rapid review methods. BMC Med, 13, 2015, 224.
18. Tricco, A., Robson, R., Thomas, S., Pham, B., Page, M., Accuracy, reliability, impact, and efficiency of different methods for selecting studies, abstracting data, and appraising quality in a systematic review: a systematic review protocol. 2016, PROSPERO Available at http://www.crd.york.ac.uk/PROSPERO/display_record.php?ID=CRD42016047877.
19. Moher, D., Stewart, L., Shekelle, P., All in the family: systematic reviews, rapid reviews, scoping reviews, realist reviews, and more. Syst Rev, 4, 2015, 183.
20. Wallace, B.C., Dahabreh, I.J., Schmid, C.H., Lau, J., Trikalinos, T.A., Modernizing the systematic review process to inform comparative effectiveness: tools and methods. J Comp Eff Res 2 (2013), 273–282.
21. O'Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., Ananiadou, S., Erratum to: using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst Rev, 4, 2015, 59.
22. Tsafnat, G., Dunn, A., Glasziou, P., Coiera, E., The automation of systematic reviews. BMJ, 346, 2013, f139.
23. Newton D. Synthesi.SR. Toronto, Ontario: Knowledge Translation Program, 2012, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, ON.
24. Edwards, P., Clarke, M., DiGuiseppi, C., Pratap, S., Roberts, I., Wentz, R., Identification of randomized controlled trials in systematic reviews: accuracy and reliability of screening records. Stat Med 21 (2002), 1635–1640.
25. Doust, J.A., Pietrzak, E., Sanders, S., Glasziou, P.P., Identifying studies for systematic reviews of diagnostic tests was difficult due to the poor sensitivity and precision of methodologic filters and the lack of information in the abstract. J Clin Epidemiol 58 (2005), 444–449.
26. Yip, R., Islami, F., Zhao, S., Tao, M., Yankelevitz, D.F., Boffetta, P., Errors in systematic reviews: an example of computed tomography screening for lung cancer. Eur J Cancer Prev 23 (2014), 43–48.
27. Shemilt, I., Khan, N., Park, S., Thomas, J., Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews. Syst Rev, 5, 2016, 140.
28. Ng, L., Pitt, V., Huckvale, K., Clavisi, O., Turner, T., Gruen, R., et al. Title and Abstract Screening and Evaluation in Systematic Reviews (TASER): a pilot randomised controlled trial of title and abstract screening by medical students. Syst Rev, 3, 2014, 121.
29. Cooper, M., Ungar, W., Zlotkin, S., An assessment of inter-rater agreement of the literature filtering process in the development of evidence-based dietary guidelines. Public Health Nutr 9 (2006), 494–500.
30. Freitas de Souza, R., Sequeira, P., Nasser, M., Is Google Translate useful for the selection of studies to be included in Cochrane reviews. 17th Cochrane Colloquium, Singapore, 2009, 11–14.
31. Busse, J.W., Bruno, P., Malik, K., Connell, G., Torrance, D., Ngo, T., et al. An efficient strategy allowed English-speaking reviewers to identify foreign-language articles eligible for a systematic review. J Clin Epidemiol 67 (2014), 547–553.
32. Berlin, J.A., On behalf of University of Pennsylvania Meta-analysis Blinding Study Group. Does blinding of readers affect the results of meta-analyses?. Lancet 350 (1997), 185–186.
33. Mateen, F.J., Oh, J., Tergas, A.I., Bhayani, N.H., Kamdar, B.B., Titles versus titles and abstracts for initial screening of articles for systematic reviews. J Clin Epidemiol 5 (2013), 89–95.
34. Wang, Z., Asi, N., Elraiyah, T.A., Abu Dabrh, A.M., Undavalli, C., Glasziou, P., et al. Dual computer monitors to increase efficiency of conducting systematic reviews. J Clin Epidemiol 67 (2014), 1353–1357.
35. Buscemi, N., Hartling, L., Vandermeer, B., Tjosvold, L., Klassen, T.P., Single data extraction generated more errors than double data extraction in systematic reviews. J Clin Epidemiol 59 (2006), 697–703.
36. Horton, J., Vandermeer, B., Hartling, L., Tjosvold, L., Klassen, T.P., Buscemi, N., Systematic review data extraction: cross-sectional study showed that experience did not increase accuracy. J Clin Epidemiol 63 (2010), 289–298.
37. Tendal, B., Higgins, J.P., Juni, P., Hrobjartsson, A., Trelle, S., Nuesch, E., et al. Disagreements in meta-analyses using outcomes measured on continuous or rating scales: observer agreement study. BMJ, 339, 2009, b3128.
38. Jayaram, M.K., Mansi, K., Haynes, E., Adams, C.E., Furtado, V.A., Catch 22: is the future of systematic reviewing only for the experienced? [unknown]. 2014, Cochrane Schizophrenia Group.
39. Florence Z, Schulz T, Pearson A. Inter-reviewer agreement: an analysis of the degree to which agreement occurs when using tools for the appraisal, extraction and meta-synthesis of qualitative research findings. Abstracts of the 13th Cochrane Colloquium. Melbourne, Australia 2005. p. 69.
40. Gresham, G., Matsumura, S., Li, T., Faster may not be better: data abstraction for systematic reviews. 2014, Cochrane Eyes and Vision Group. US Cochrane Center, U. S.
41. Balk, E.M., Chung, M., Chen, M.L., Trikalinos, T.A., Kong Win Chang, L., Assessing the accuracy of Google Translate to allow data extraction from trials published in non-English languages. Methods research report. 2013, Agency for Healthcare Research and Quality (US), Rockville (MD).
42. Selph, S.S., Ginsburg, A.D., Chou, R., Impact of contacting study authors to obtain additional data for systematic reviews: diagnostic accuracy studies for hepatic fibrosis. Syst Rev, 3, 2014, 107.
43. Gibson, C.A., Bailey, B.W., Carper, M.J., LeCheminant, J.D., Kirk, E.P., Huang, G., et al. Author contacts for retrieval of data for a meta-analysis on exercise and diet restriction. Int J Technol Assess Health Care 22 (2006), 267–270.
44. Cahill K, Perera R, Selwood M. Electronic extraction of graphical data [abstract]. XV Cochrane Colloquium, Sao Paulo, Brazil 2007, 153-154p 2007 Oct 23-27.
45. de Oliveira, I.R., Santos-Jesus, R., Po, A.L., Poolsup, N., Extracting numerical data from published reports of pharmacokinetics investigations: method description and validation. Fundam Clin Pharmacol 17 (2003), 471–472.
46. Higgins, J.P.T.G.S., (eds.) Cochrane Handbook for Systematic Reviews of Interventions 4.2.6, 2006, John Wiley & Sons, Ltd., Chichester, UK.
47. da Costa, B.R., Beckett, B., Diaz, A., Resta, N.M., Johnston, B.C., Egger, M., et al. Effect of standardized training on the reliability of the Cochrane risk of bias assessment tool: a prospective study. Syst Rev, 6, 2017, 44.
48. Sands, M.L., Murphy, J.R., Use of kappa statistic in determining validity of quality filtering for meta-analysis: a case study of the health effects of electromagnetic radiation. J Clin Epidemiol 49 (1996), 1045–1051.
49. Fourcade, L., Boutron, I., Moher, D., Ronceray, L., Baron, G., Ravaud, P., Development and evaluation of a pedagogical tool to improve understanding of a quality checklist: a randomised controlled trial. PLoS Clin Trials, 2, 2007.
50. Oremus, M., Oremus, C., Hall, G.B.C., McKinnon, M.C., Graham, A., Gregory, C., et al. Inter-rater and test-retest reliability of quality assessments by novice student raters using the Jadad and Newcastle-Ottawa Scales. BMJ Open, 2, 2012 https://www.ncbi.nlm.nih.gov/pubmed/22855629.
51. Jadad, A.R., Moore, R.A., Carroll, D., Jenkinson, C., Reynolds, D.J., Gavaghan, D.J., et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary?. Control Clin Trials 17 (1996), 1–12.
52. Berard, A., Andreu, N., Tetrault, J., Niyonsenga, T., Myhal, D., Reliability of Chalmers’ scale to assess quality in meta-analyses on pharmacological treatments for osteoporosis. Ann Epidemiol 10 (2000), 498–503.
53. Clark, H.D., Wells, G.A., Huet, C., McAlister, F.A., Salmi, L.R., Fergusson, D., et al. Assessing the quality of randomized trials: reliability of the Jadad scale. Control Clin Trials 20 (1999), 448–452.
54. Verhagen, A.P., de Vet, H.C., de Bie, R.A., Kessels, A.G., Boers, M., Knipschild, P.G., Balneotherapy and quality assessment: interobserver reliability of the Maastricht criteria list and the need for blinded quality assessment. J Clin Epidemiol 51 (1998), 335–341.
55. Armijo-Olivo, S., Ospina, M., da Costa, B.R., Egger, M., Saltaji, H., Fuentes, J., et al. Poor reliability between Cochrane reviewers and blinded external reviewers when applying the Cochrane risk of bias tool in physical therapy trials. PLoS One, 9, 2014, e96920.
56. Hartling, L., Hamm, M.P., Milne, A., Vandermeer, B., Santaguida, P.L., Ansari, M., et al. Testing the risk of bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs. J Clin Epidemiol 66 (2013), 973–981.
57. Robertson, C., Ramsay, C., Gurung, T., Mowatt, G., Pickard, R., Sharma, P., et al. Practicalities of using a modified version of the Cochrane Collaboration risk of bias tool for randomised and non-randomised study designs applied in a health technology assessment setting. Res Synth Methods 5 (2014), 200–211.
58. Vale, C.L., Tierney, J.F., Burdett, S., Can trial quality be reliably assessed from published reports of cancer trials: evaluation of risk of bias assessments in systematic reviews. BMJ, 346, 2013, f1798.
59. Littlewood, C., Ashton, J., Chance-Larsen, K., May, M., Sturrock, B., The quality of reporting might not reflect the quality of the study: implications for undertaking and appraising a systematic review. J Man Manip Ther 20 (2012), 130–134.
60. Crowe, M., Sheppard, L., Campbell, A., Comparison of the effects of using the Crowe Critical Appraisal Tool versus informal appraisal in assessing health research: a randomised trial. Int J Evid Based Healthc 9 (2011), 444–449.
61. Dixon-Woods, M., Sutton, A., Shaw, R., Miller, T., Smith, J., Young, B., et al. Appraising qualitative research for inclusion in systematic reviews: a quantitative and qualitative comparison of three methods. J Health Serv Res Policy 12 (2007), 42–47.