Not all systematic reviews are created equal: Considerations for appraisal

Evidence-Based Communication Assessment and Intervention

Volumn 1, Issue 3, 2007, Pages 138-150

  Schlosser, Ralf W ^a   Wendt, Oliver ^b   Sigafoos, Jeff ^c  

^a NORTHEASTERN UNIVERSITY   (United States)

^b PURDUE UNIVERSITY   (United States)

^c UNIVERSITY OF TASMANIA   (Australia)

Author keywords

Critical appraisal; Evidence based practice; Systematic reviews

Indexed keywords

EID: 49549109808     PISSN: 17489539     EISSN: 17489547     Source Type: Journal    
DOI: 10.1080/17489530701560831     Document Type: Article

References (86)

1. Agency for Healthcare Research and Quality. 2002. Criteria for determining disability in speech–language disorders. Available at:http://www.ahrq.gov/clinic/epcsums/spdissum.htm (accessed 13 September, 2007)
2. Auperin, A, Pignon, J-P, and Poynard, T. 1997. Review article:Critical review of meta-analyses of randomized clinical trials in hepatogastroenterology. Aliment Pharmacol Ther, 11:215–225.
3. Bochner, S, Center, Y, Chapparo, C, and Donelly, M. 1999. How effective are programs based on conductive education? A report of two studies. Journal of Intellectual & Development Disability, 24:227–242.
4. Bopp, KD, Brown, KE, and Mirenda, P. 2004. Speech–language pathologists’ role in the delivery of positive behavior support for individuals with developmental disabilities. American Journal of Speech-Language Pathology, 13:5–19.
5. Brady, N. 2007. Data seem to support manual sign interventions for children with autism, but more high-quality research is needed. Evidence-Based Communication Assessment and Intervention, 1:16–17.
6. Campbell, PH, Milbourne, S, Dugan, LM, and Wilcox, MJ. 2006. A review of evidence on practices for teaching young children to use assistive technology devices. Topics in Early Childhood Special Education, 26:3–13.
7. Choi, PT-L, Halpern, SH, Malik, N, Jadad, AR, Tramèr, MR, and Walder, B. 2001. Examining the evidence in anesthesia literature:A critical appraisal of systematic reviews. Anesthesia and Analgesia, 92:700–709.
8. Clarke, M. 2000. “ The QUORUM statement ”. In Lancet Vol. 26, 756–757.
9. Cohen, J. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measureement, 20:37–46.
10. Cook, DJ, Sackett, DL, and Spitzer, WO. 1995. Methodologic guidelines for systematic reviews of randomized control trials in health care from the Potsdam Consultation on meta-analysis. Journal of Clinical Epidemiology, 48:167–171.
11. Cook, DJ, Mulrow, CD, and Haynes, RB. 1997. Synthesis of best evidence for clinical decisions. Annals of Internal Medicine, 126:376–380.
12. Cook, TD, Cooper, H, Cordray, DS, Hartmann, H, Hedges, LV, Light, RJ. 1992. Meta-analysis for explanation:A casebook, New York:Russell Sage Foundation.
13. Cooper, H, and Hedges, LV. 1994. The handbook of research synthesis, New York:Russell Sage Foundation.
14. Darrah, J, Watkins, B, Chen, L, and Bonin, C. 2003. Effects of conductive education intervention for children with a diagnosis of cerebral palsy:An AACPDM evidence report., Rosemont, IL, USA:American Academy for Cerebral Palsy and Developmental Medicine (AACPDM), AACPDM Evidence Report:34.
15. DeCoster, J. 2005. “ Meta-analysis. ”. In Encyclopedia of social measurement, Edited by:Kempf-Leonard, K. Vol. 2, 683–688. San Diego, CA:Academic Press.
16. Dollaghan, CA. 2004. Evidence-based practice in communication disorders:what do we know, and when do we know it?. Journal of Communication Disorders, 37:391–400.
17. Eagly, A, H and Wood, W. 1994. “ Using research to plan future research. ”. In The handbook of research synthesis, Edited by:Cooper, H, and Hedges, LV. 485–500. New York:Russell Sage Foundation.
18. Egger, M, and Davey Smith, G. 1998. Meta-analysis:Bias in location and selection of studies. British Medical Journal, 316:61–66.
19. Egger, M, and Smith, GD. 2001. “ Principles of and procedures for systematic reviews. ”. In Systematic reviews in health care., 2nd ed, Edited by:Egger, M, Smith, GM, and Altman, DG. 23–42. London:British Medical Journal.
20. Egger, M, Zellweger-Zahner, T, Schneider, M, Junker, C, Lengeler, C, and Antes, G. 1997. Language bias in randomised controlled trials published in English and German. Lancet, 350:326–329.
21. Eysenck, HJ. 1978. An exercise in mega-silliness. American Psychology, 33:517
22. Fleiss, JL. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76:378–382.
23. Granlund, M. 2007. “ Presents characteristics of AAC interventions for students with severe disabilities, but judgements about effectiveness do not follow from methodology [Abstract]. ”. In Evidence-Based Communication Assessment and Intervention, Vol. 1, 67–68.
24. Abstract of Snell, M, Chen, L-H, and Hoover, K. 2007. “ Teaching augmentative communication to students with severe disabilities:A review of intervention research 1997–2003. ”. In Research and Practice for Persons with Severe Handicaps Vol. 31, 203–214.
25. Grégoire, G, Derderian, F, and LeLorier, J. 1995. Selecting the language of the publications included in a meta-analysis:Is there a Tower of Babel bias?. Journal of Clinical Epidemiology, 48:159–163.
26. Guyatt, G, and Rennie, D. 2002. “ Users’ guide to the medical literature ”. In Essentials of evidence-based clinical practice, Chicago:AMA Press.
27. Herder, C, Howard, C, Nye, C, and Vanryckeghem, M. 2006. Effectiveness of behavioral stuttering treatment:A systematic review and meta-analysis. Contemp Iss Comm Sci Dis, 33:61–73.
28. Higginbotham, DJ. 2003. “ Formulating research questions ”. In The efficacy of augmentative and alternative communication:Toward evidence-based practice, Edited by:Schlosser, RW. 43–57. San Diego:Academic Press.
29. Higgins, JPT, and Green, S. 2006. Cochrane handbook for systematic reviews of interventions 4.2.6 [updated September, 2006]. Available at:http://www.cochrane.org/resources/handbook/hbook.htm (accessed 6th October, 2006)
30. Humphris, D, Littlejohns, P, Victor, C, O’Halloran, P, and Peacock, J. 2000. Implementing evidence-based practice:Factors that influence the use of research evidence by occupational therapists. Br J Occup Ther, 63:516–522.
31. Hyung Bok Yoo, H, and Quebuz, TT. 2004. Locating and selecting appraisal studies for reviews. Chest, 125:798
32. Jackson, GB. 1980. Methods for integrative reviews. Review of Educational Research, 50:438–460.
33. Jüni, P, Holenstein, F, Sterne, J, Bartlett, C, and Egger, M. 2002. Direction and impact of language bias in meta-analyses of controlled trials:empirical study. International Journal Of Epidemiology, 31:115–123.
34. Landis, JR, and Koch, GG. 1977. The measurement of observer agreement for categorical data. Biometrics, 33:159–174.
35. Lauer, JM, and Asher, JW. 1988. Composition research:Empirical designs, New York:Oxford University Press.
36. Law, M. 2002. Evidence-based rehabilitation:A guide to practice, Thoroughfare, NJ:Slack Inc.
37. Law, J. 2007. Behavioral stuttering treatments are effective but no one treatment approach is more effective over other treatment approaches. Evidence-based Communication Assessment and Intervention, 1:14–15.
38. Law, M, and Plunkett, C. 2006. Grading study quality in systematic reviews. Contemp Iss Comm Sci Dis, 33:28–36.
39. Light, R, and Pillemer, D. 1984. Summing up:The science of reviewing research, Cambridge, MA:Harvard University Press.
40. Lipsey, MW, and Wilson, DB. 2001. Practical meta-analysis, Thousand Oaks, CA:Sage Publications.
41. Maher, CG, Sherrington, C, Herbert, RD, Moseley, AM, and Elkins, M. 2003. Reliability of the PEDro scale for rating quality of randomized controlled trials. Physical Theraphy, 83:713–721.
42. Mancil, GR. 2006. Functional communication training:A review of the literature related to children with autism. Education and Training in Development Disabilities, 41:213–224.
43. McAuley, L, Pham, B, Tugwell, P, and Moher, D. 2000. Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses?. The Lancet, 356:1228–1231.
44. Meline, T. 2006. Selecting studies for systematic review:Inclusion and exclusion criteria. Contemp Iss Comm Sci Dis, 33:21–27.
45. Millar, D, Light, JC, and Schlosser, RW. 2006. The impact of augmentative and alternative communication intervention on the speech production of individuals with developmental disabilities:A research review. Journal of Speech Language and Hearing Research, 49:248–264.
46. Moher, D, Cook, DJ, Eastwood, S, Olkin, I, Rennie, D, and Stroup, DF. 1999. Improving the quality of reporting of meta-analysis of randomized controlled trials:The QUOROM statement. Lancet, 354:1896–1900.
47. Moher, D, Pham, B, Klassen, TP, Schulz, KF, Berlin, JA, Jadad, AR, and Liberati, A. 2000. What contributions do languages other than English make on the results of meta-analyses?. Journal of Clinical Epidemiology, 53:964–972.
48. Moher, D, Schulz, KF, and Altman, DG. 2001. The CONSORT statement:Revised recommendations for improving the quality of reports of parallel-group randomized trials. Lancet, 357:1191–1194.
49. Moher, D, Tetzlaff, J, Tricco, AC, Sampson, M, and Altman, DG. 2007. Epidemiology and reporting characteristics of systematic reviews. PLoS Medicine, 4:e78
50. Montori, VM, Swiontkowski, MF, and Cook, DJ. 2003. Methodologic issues in systematic reviews and meta-analyses. Clinical Orthopaedics and Related Research, 413:43–54.
51. Mortaz Hejri, S. 2003. Critical Appraisal tools for systematic reviews. Available at http://ssrc.tums.ac.ir/SystematicReview/Appraisal-Tools.asp (accessed 13 September, 2007)
52. Oxman, AD, Cook, DJ, and Guyatt, GH. 1994. Users’ guide to the medical literature. VI. How to use an overview. Evidence-Based Medicine Working Group. Journal ofAmerican Medical Association, 272:1367–1371.
53. Perepletchikova, F, and Kazdin, AE. 2005. Treatment integrity and therapeutic change:Issues and research recommendations. Clinical Psychology, 12:365–383.
54. Peterson, L, Homer, AL, and Wonderlich, SA. 1982. The integrity of independent variables in behavior analysis. Journal Of Applied Behavior Analysis, 15:477–492.
55. Petticrew, M, and Roberts, H. 2006. Systematic reviews in the social sciences:A practical guide, Malden, MA:Blackwell Publishing Co.
56. Reilly, S, and Oates, J. 2003. Evidence-based practice in speech pathology, London:Whurr.
57. Rosenthal, R. 1978. The "file drawer problem" and tolerance for null results. Psychological Bulletin, 86:638–641.
58. Rothstein, H, Sutton, AJ, and Borenstein, M. 2005. Publication bias in meta-analysis:Prevention, assessment and adjustments, UK:John Wiley & Sons.
59. Sackett, DL, and Gent, M. 1979. Controversy in counting and attributing events in clinical trials. New England Journal of Medicine, 301:1410–1412.
60. Sackett, DL, Haynes, RB, and Tugwell, P. 1985. Clinical epidemiology:A basic science of clinical medicine, Boston:Little, Brown.
61. Sackett, DL, Straus, SE, Richardson, WS, Rosenberg, W, and Haynes, RB. 2000. Evidence-based medicine:How to practice and teach EBM, 2nd, Edinburgh:Churchill Livingstone.
62. Sacks, HS, Reitman, D, Pagano, D, and Kupelnick, B. 1996. Meta-analysis:An update. Mount Sinai Journal Of Medicine, 63:216–224.
63. Schlesselman, JJ, and Collins, JA. 2003. Evaluating systematic reviews and meta-analyses. Seminars in Reproductive Medicine, 21:95–105.
64. Schlosser, RW. 1999. Social validation of interventions in augmentative and alternative communication. Augment Altern Commun, 15:234–247.
65. Schlosser, RW. 2002. On the importance of being earnest about treatment integrity. Augment Altern Comm, 18:36–44.
66. Schlosser, RW. 2003a. The efficacy of augmentative and alternative communication:Toward evidence-based practice, San Diego, CA:Academic Press.
67. Schlosser, RW. 2003b. “ Synthesizing efficacy research in AAC ”. In The efficacy of augmentative and alternative communication:Towards evidence-based practice, Edited by:Schlosser, RW. 230–258. San Diego, CA:Academic Press.
68. Schlosser, RW. 2007. Training of conversation partners of children with cerebral palsy appears effective for common communication targets but methodological concerns render these findings suggestive. Evidence Based Communication Assessement and Intervention, 1:12–13.
69. Schlosser, RW, and Goetze, H. 1992. “ Effectiveness and treatment validity of interventions addressing self-injurious behavior:From narrative reviews to meta-analysis ”. In Advances in learning and behavioral disabilities, Edited by:Scruggs, TE, and Mastropieri, MA. Vol. 7, 135–175. Greenwich, CT:JAI Press, Inc.
70. Schlosser, RW, and Lee, D. 2000. Promoting generalization and maintenance in augmentative and alternative communication:A meta-analysis of 20 years of effectiveness research. Augment Altern Comm, 16:208–227.
71. Schlosser, RW, and O’Neil-Pirozzi, TM. 2006. Problem formulation in evidence-based practice and systematic reviews. Contemp Iss Comm Sci Dis, 33:5–10.
72. Schlosser, RW, and Raghavendra, P. 2004. Evidence-based practice in augmentative and alternative communication. Augment Altern Comm, 20:1–21.
73. Schlosser, RW, Wendt, O, Angermeier, K, and Shetty, M. 2005. Searching for and finding evidence in augmentative and alternative communication:Navigating a scattered literature. Augment Altern Comm, 21:233–255.
74. Schwartz, JB, and Nye, C. 2006. A systematic review, synthesis, and evaluation of the evidence for teaching sign language to children with autism. EBP Briefs, 1:1–17.
75. Scruggs, TE. 1992. “ Single-subject research methodology in the study of learning and behavioral disorders:Design, analysis, and synthesis ”. In Advances in learning and behavioral disabilities, Edited by:Scruggs, TE, and Mastropieri, M. A., Vol. 7, 223–248. Greenwich, CT:TE.
76. Scruggs, TE, and Mastropieri, MA. 1998. Summarizing single-subject research:Issues and applications. Behaviour Modification, 22:221–242.
77. Slavin, RE. 1987. “ Best-evidence synthesis:An alternative to meta-analytic and traditional reviews ”. In Evaluation studies:Review annual 12, Edited by:Shadish, W. R., and Reichardt, C. S., 667–673. Thousand Oaks, CA:Sage Publications.
78. Snell, ME, Chen, L-H, and Hoover, K. 2006. Teaching augmentative and alternative communication to students with severe disabilities:A review of intervention research, 1997–2003. Res Pract Pers Sev Disabil, 31:203–214.
79. Straus, SE, Richardson, WS, Glasziou, P, and Haynes, RB. 2005. Evidence-based medicine:How to practice and teach EBM, 3rd, Edinburgh:Elsevier Science.
80. Turner, HM, and Bernard, RM. 2006. Calculating and synthesizing effect sizes. Contemp Iss Comm Sci Dis, 33:42–55.
81. Verhagen, AP, de Vet, HC, de Bie, RA, Kessels, AG, Boers, M, Bouter, LM, and Knipschild, PG. 1998. The Delphi list:A criteria list for quality assessment of randomized clinical trials for conducting systematic reviews developed by Delphi consensus. Journal of Clinical Epidemiology, 51:1235–1241.
82. Wendt, O. 2006. The effectiveness of augmentative and alternative communication for children with autism:A meta-analysis of intervention outcomes [dissertation]., West Lafayette, IN:Purdue University.
83. Wendt, O. 2007. Recommended practices for teaching assistive technology use to infants and young children with low incidence disabilities seem to have little empirical support but methodological concerns limit the validity of this review. Evidence Based Communication Assessment and Intervention, 1:60–62.
84. West, S, King, V, Carey, T, S, Lohr, K, N, McKoy, N, Sutton, S, F and Lux, L. 2002. “ Systems to Rate the Strength of Scientific Evidence. ”. In Evidence Report/Technology Assessment No. 47 (Prepared by the Research Triangle Institute–University of North Carolina Evidence-based Practice Center under Contract No. 290-97-0011). AHRQ Publication No. 02-E016., Rockville, MD:Agency for Healthcare Research and Quality.
85. White, HD. 1994. “ Scientific communication and literature retrieval ”. In The handbook of research synthesis, Edited by:Cooper, H., and Hedges, L. V., 41–55. New York:Russell Sage Foundation.
86. Zipoli, RP, and Kennedy, M. 2005. Evidence-based practice among speech–language pathologists:Attitudes, utilization, and barriers. American Journal of Speech-Language Pathology, 14:208–220.