Rating methodological quality: Toward improved assessment and investigation

Accountability in Research

Volumn 12, Issue 4, 2005, Pages 299-313

  Moyer, Anne ^a   Finney, John W ^b  

^a STONY BROOK UNIVERSITY   (United States)

^b VA PALO ALTO HEALTH CARE SYSTEM   (United States)

Author keywords

Bias; Evidence based medicine; Meta analysis; Methodology; Quality; Systematic reviews

Indexed keywords

ARTICLE; BIOMEDICAL AND BEHAVIORAL RESEARCH; CLINICAL TRIAL; EMPIRICAL RESEARCH; EPIDEMIOLOGY; META ANALYSIS; METHODOLOGY; PUBLISHING; REPRODUCIBILITY; STANDARD;

BIOMEDICAL AND BEHAVIORAL RESEARCH;

BIAS (EPIDEMIOLOGY); CLINICAL TRIALS; EMPIRICAL RESEARCH; JOURNALISM; META-ANALYSIS; REPRODUCIBILITY OF RESULTS; RESEARCH DESIGN;

EID: 33748060986     PISSN: 08989621     EISSN: 15455815     Source Type: Journal    
DOI: 10.1080/08989620500440287     Document Type: Article

References (65)

1. Altman, D. G. (1996). Better reporting of randomized controlled trials: The CONSORT statement: Authors must provide enough information for readers to know how the trial was performed, British Medical Journal, 313: 570-571.
2. Altman, D. G., Schulz, K. F., Moher, D., Egger, M., Davidoff S, Elbourne, D., Gøtzsche, P. C., Lang, T., and for the CONSORT Group. (2001). The revised CONSORT statement for reporting randomized trials: Explanation and elaboration, Annals of Internal Medicine, 134: 663-694.
3. Alderson, P., Green, S., and Higgins J. P. T., (Eds.). Cochrane Reviewers' Handbook 4.2.2 [updated December 2003]. Available at http://www.cochrane.org/resources/handbook/hbook.htm (last accessed March 21, 2004).
4. Benson, K., and Hartz, A. J. (2000). A comparison of observational studies and randomized controlled trials, New England Journal of Medicine, 342: 1878-1886.
5. Berlin, J. A., and Rennie, D. (1999). Measuring the quality of trials: The quality of quality scales, JAMA, 282: 1083-1085.
6. Bero, L., and Rennie, D. (1995). The Cochrane Collaboration: Preparing, maintaining, and disseminating systematic reviews of the effects of health care, JAMA, 274: 1935-1938.
7. Chalmers, I, Adams, M., Dickersin, K., Hetherington, J., Tarnow-Mordi, W., Meinert, C., Tonascia, S., and Chalmers, T. C. (1990). A cohort study of summary reports of controlled trials, JAMA, 272: 1401-1405.
8. Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., and Altman, D. G. (2004). Emipical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols to published articles, JAMA, 291: 2457-2465.
9. Chalmers, T. C., Smith, H., Jr., Blackburn, B., Silverman, B., Schroeder, B., Reitmean, D., and Ambroz, A. (1981). A method for assessing the quality of a randomized control trial, Controlled Clinical Trials, 2: 31-49.
10. Cho, M. K., and Bero, L. A. (1996). The quality of drug studies published in symposium proceedings, American Journal of Internal Medicine, 124: 485-489.
11. Clarke, M., and Oxman, A. D. (eds.). (2000). Cochrane reviewers handbook 4.1.1. [Updated December 2000], In, The Cochrane Library, No. 4, 2000. Oxford: Update Software: Updated quarterly.
12. Clarke, H. D., Wells, G. A., Huët, C., McAlister, F. A., Salmi, L. R., Fergusson, D., and Laupacis, A. (1999). Assessing the quality of randomized trials: Reliability of the Jadad scale, Controlled Clinical Trials, 20: 448-452.
13. Colditz, G. A., Miller, J. N., and Mosteller, F. (1989). How study design affects outcomes in comparisons of therapy. I, Medical. Statistics in Medicine, 8: 441-454.
14. Concato, J., Shah, N., and Horwitz, R. I. (2000). Randomized, controlled trials, observational studies, and the hierarchy of research designs, New England Journal of Medicine, 342: 1887-1892.
15. Cook, D. J., Sackett, D. L., and Spitzer, W. O. (1995). Methodologic guidelines for systematic reviews of randomized control trials in health care from the Potsdam consultation on meta-analysis, Journal of Clinical Epidemiology, 48: 167-171.
16. De Angelis, C. D., Drazen, J. M., Frizelle, F. A., Huag, C., Hoey, J., Horton, R., Kotzin, S., Laine, C., Marusic, A., Overbeke, A. J. P. M., Shroeder, T. V., Sox, H. C., and Van Der Weyeden, M. B. (2005). Is this clinical trial fully regis-tered?-A statement from the International Committee of Medical Journal Editors, JAMA, 352: 2436-2438.
17. Detsky, A. S., Naylor, C. D., O'Rourke, K., McGeer, A. J., and L'Abbé, K. A. (1992). Incorporating variations in the quality of individual randomized trials into meta-analysis, Journal of Clinical Epidemiology, 45: 255-265.
18. Dickersin, K., Min, Y. I., and Meinert, C. L. (1991). Factors influencing publication of research results: Follow-up of applications submitted to two institutional review boards, JAMA, 267: 374-378.
19. Easterbrook, P. J., Berlin, J. A., Gopalan, R., and Matthews, D. R. (1991). Publication bias in clinical trials, Lancet, 337: 867-872.
20. Egger, M., Jüni, P., Bartlett, C., and for the CONSORT Group. (2001). Value of flow diagrams in reports of randomized trials, JAMA, 285: 1996-1999.
21. Egger, M., Jüni, P., Bartlett, C., Holstein, F., and Sterne, J. (2003). How important are comprehensive literature searches and the assessment of trial quality in systematic reviews?, Health Technology Assessment, 7: 1-76.
22. Emerson, J. D., Burdick, E., Hoaglin, D. C., Mosteller, F., and Chalmers, T. C. (1990). An empirical study of the possible relation of treatment differences to quality scores in controlled randomized clinical trials, Controlled Clinical Trials, 11: 339-352.
23. Greenland, S. (1994a). A critical look at some popular meta-analytic methods, American Journal of Epidemiology, 140: 290-296.
24. Greenland, S. (1994b). Quality scores are useless and potentially misleading. Reply to "Re: A critical look at some popular analytic methods", American Journal of Epidemiology, 140: 300-302.
25. Greer, N., Mosser, G., Logan, G., and Halaas, G. W. (2000). A practical approach to evidence grading, Joint Commission Journal on Quality Improvement, 26: 700-712.
26. Heinsman, D. T. and Shadish, W. R. (1996). Assignment methods in experimentation: When do nonrandomized experiments approximate answers from randomized experiments?, Psychological Methods, 1: 154-169.
27. Ioannidis, J. P. A. and Lau, J. (1998). Can quality of clinical trials and meta-anal-ysis be quantified?, Lancet, 352: 590-591.
28. Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J. M., Gavaghan, D. J., and McQuay, H. J. (1996). Assessing the quality of reports of randomized clinical trials: Is blinding necessary, Controlled Clinical Trials, 17: 1-12.
29. Jadad, A. R. and Rennie, D. (1998). The randomized trial gets a middle-aged checkup, JAMA, 279: 319-320.
30. Jüni, P., Witschi, A., Bloch, R., and Egger, M. (1999). The hazards of scoring the quality of clinical trials for meta-analysis, JAMA, 282: 1054-1060.
31. Khalid, K., Daya, S., and Jadad, A. (1996). The importance of quality of primary studies in producing unbiased systematic reviews, Archives of Internal Medicine, 156: 661-666.
32. Kraemer, H. C., Gardner, C., Brooks, C., and Yesavage, J. A. (1998). Advantages of excluding underpowered studies in meta-analysis: Inclusionist versus exclusionist viewpoints, Psychological Methods, 3: 23-31.
33. Kunz, R. and Oxman, A. D. (1998). The unpredictability paradox: Review of empirical comparisons of randomised and non-randomised clinical trials, British Medical Journal, 317: 1185-1190.
34. Levin, A. (2001). Reporting standards and the transparency of trials, Annals of Internal Medicine, 134: 169-172.
35. Lipsey, M. W. and Wilson, D. B. (1993). The efficacy of psychological, educational, and behavioral treatment: Confirmation from meta-analysis, American Psychologist, 48: 1181-1209.
36. Lohr, K. N. and Carey, T. S. (1999). Assessing "best evidence": Issues in grading the quality of studies for systematic reviews, Joint Commission Journal on Quality Improvement, 25: 470-479.
37. MacLehose, R. R., Reeves, B. C., Harvey, I. M., Sheldon, T. A., Russell, I. T., and Black, A. M. S. (2000). A systematic review of comparisons of effect sizes derived from randomised and non-randomised studies, Health Technology Assessment, 4: 1-154.
38. Miller, J. N., Colditz, G. A., and Mosteller, F. (1989). How study design affects outcomes in comparisons of therapy. II: Surgical, Statistics in Medicine, 8: 455-466.
39. Miller, W. R., Brown, J. M., Simpson, T. L., Handmaker, N. S., Bien, T. H., Luckie, L. F., Montgomery, H. A., Hester, R. K., and Tonigan, J. S. (1995). What works? A methodological analysis of the alcohol treatment outcome literature. In R. K., Hester, and W. R., Miller (eds.), Handbook of Alcoholism Treatment Approaches: Effective Alternatives (2nd ed.) 12-44. Boston: Allyn and Bacon.
40. th anniversary of Archie Cochrane's effectiveness and efficiency. (pp. 250-271). London: BMJ Publishing Group.
41. Moher, D., Cook, D. J., Jadad, A. R., Tugwell, P., Moher, M., Jones, A., Pham, B., and Klassen, T. P. (1999). Assessing the quality of reports of randomized trials: Implications for the conduct of meta-analyses, Health Technology Assessment, 3: 1-99.
42. Moher, D., Jadad, A. R., Nichol, G., Penman, M., Tugwell, P., and Walsh, S. (1995). Assessing the quality of randomized controlled trials: An annotated bibliography of scales and checklists, Controlled Clinical Trials, 16: 62-73.
43. Moher, D., Jadad, A. R., and Tugwell, P. (1996). Assessing the quality of randomized controlled trials: Current issues and future directions, International Journal of Technology Assessment in Health Care, 12: 195-208.
44. Moher, D., Jones, A., Lepage, L., and for the CONSORT Group. (2001). Use of the CONSORT statement and quality of reports of randomized trials: A comparative before-and-after evaluation, JAMA, 285: 1992-1995.
45. Moher, D. and Olkin, I. (1995). Meta-analysis of randomized control trials: A concern for standards, JAMA, 274: 1962-1964.
46. Moher, D., Pham, B., Jones, A., Cook, D. J., Jadad, A. R., Moher, M., Tugwell, P., and Klassen, T. P. (1998). Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses?, Lancet, 352: 609-613.
47. Moher, D., Schulz, K. F., Altman, D., and for the CONSORT group. (2001). The CONSORT statement: Revised recommendations for improving the quality of reports of parallel- group randomized trials, JAMA, 285: 1987-1991.
48. Moscati, R., Jehle, D., Ellis, D., Fiorello, A., and Landi, M. (1994). Positive-outcome bias: Comparison of emergency medicine and general medicine literatures, Academic Emergency Medicine, 1: 267-271.
49. Moyer, A., Finney, J. W., and Swearingen, C. (2002). Methodological characteristics and quality of alcohol treatment outcome studies, 1970-1998: An expanded evaluation, Addiction, 97: 253-263.
50. Moyer, A., Finney, J. W., Swearingen, C. E., and Vergun, P. (2002). Brief interventions for alcohol problems: A meta-analytic review of controlled investigations in treatment-seeking and non-treatment-seeking populations, Addiction, 97: 279-292.
51. National Registry of Evidence-based Programs and Practices. (2005). Available at http://modelprograms.samsha.gov/template.cfm?page=default. Accessed Nov. 18, 2005.
52. Ottenbacher, K. (1992). Impact of random assignment on study outcome: An empirical examination, Controlled Clinical Trials, 12: 50-61.
53. Rosenthal, R. (1979). The "file drawer" problem and tolerance for null results, Psychological Bulletin, 86: 638-641.
54. Ruiz-Canela, M., de Irala-Estevez, J., Martínez-González, M. A., Gómez-Gracia, E., and Fernández-Crehuet, J. (2001). Methodological quality and reporting of ethical requirements in clinical trials, Journal of Medical Ethics, 27: 172-176.
55. Sackett, D. L., Richardson, W. S., Rosenberg, W., and Haynes, R. B. (1997). Evidence-based medicine: How to practice and teach EBM. New York: Churchill Livingstone.
56. Schulz, K. F., Chalmers, I., Grimes, D. A., and Altman, D. G. (1994). Assessing the quality of randomization from reports of controlled trials published in obstetrics and gynecology journals, JAMA, 272: 125-128.
57. Schulz, K. F., Chalmers, I., Hayes, R. J., and Altman, D. G. (1995). Empirical evidence of bias: Dimensions of methodological quality associated with estimates of treatment effects in controlled trials, JAMA, 273: 408-412.
58. Shadish, W. R., and Ragsdale, K. (1996). Random versus nonrandom assignment in controlled experiments: Do you get the same answer?, Journal of Consulting and Clinical Psychology, 64: 1290-1305.
59. Tritchler, D. (1999). Modelling study quality in meta-analysis, Statistics in Medicine, 18: 2135-2145.
60. Verhagen, A. P., de Vet, H. C. W., de Bie, R. A., Kessels, A. G. H., Boers, M., Bouter, L. M., and Knipschild, P. G. (1998). The Delphi List: A criteria list for quality assessment of randomized clinical trials for conducting systematic reviews developed by Delphi consensus, Journal of Clinical Epidemiology, 51: 1235-1241.
61. Weiss, B., and Weisz, J. R. (1990). The impact of methodological factors on child psychotherapy outcome research: A meta-analysis for researchers, Journal of Abnormal Child Psychology, 18: 639-670.
62. West, S., King, V., Carey, T. S., et al. (2002) Systems to Rate the Strength of Scientific Evidence. Evidence Report/Technology Assessment No. 47 (Prepared by the Research Triangle Institute-University of North Carolina Evidence-based Practice Center under Contract No. 290-97-0011). AHRQ Publication No. 02-E016. Rockville, MD: Agency for Healthcare Research and Quality.
63. What Works Clearinghouse. (2005). Available at www.w-w-c.org. Accessed Nov. 18, 2005.
64. Wilson, D. B., and Lipsey, M. W. (2001). The role of method in treatment effectiveness research: Evidence from meta-analysis, Psychological Methods, 6: 413-429.
65. Wortman, P. (1994). Judging research quality. In Cooper, H., and Hedges, L. V. (eds.), Handbook of research synthesis (pp. 97-109). New York: Russell Sage Foundation.