Methodological quality assessment tools of non-experimental studies: A systematic review;Instrumentos de evaluación de la calidad metodológica de estudios no experimentales: Una revisión sistemática

Anales de Psicologia

Volumn 28, Issue 2, 2012, Pages 617-628

  Jarde, Alexander ^a   Losilla, Josep Maria ^a   Vives, Jaume ^a  

^a UNIVERSITAT AUTÒNOMA DE BARCELONA   (Spain)

Author keywords

Methodological quality; Non experimental studies; Quality as sessment tools; Systematic review

Indexed keywords

EID: 84860896206     PISSN: 02129728     EISSN: 16952294     Source Type: Journal    
DOI: None     Document Type: Review

References (93)

1. American Psychological Association. (2010). Publication Manual of the American Psychological Association, Sixth Edition (6th ed.). Washing-ton, DC: Author.
2. Angelillo, I. F., & Villari, P. (1999). Residential exposure to electromagnetic fields and childhood leukaemia: A meta-analysis. Bulletin of the World Health Organization, 77(11), 906-915.
3. Ariëns, G. A. M., Van Mechelen, W., Bongers, P. M., Bouter, L. M., & Van der Wal, G. (2000). Physical risk factors for neck pain. Scandinavian Jour-nal of Work, Environment & Health, 26(1), 7-19.
4. Atluri, S., Datta, S., Falco, F. J., & Lee, M. (2008). Systematic review of diag-nostic utility and therapeutic effectiveness of thoracic facet joint inter-ventions. Pain Physician, 11(5), 611-629.
5. Avis, M. (1994). Reading research critically. II. An introduction to appraisal: Assessing the evidence. Journal of Clinical Nursing, 3(5), 271-277.
6. Berra, S., Elorza-Ricart, J. M., Estrada, M. D., & Sánchez, E. (2008). A tool for the critical appraisal of epidemiological cross-sectional studies. [Ins-trumento para la lectura crítica y la evaluación de estudios epidemiológi-cos transversales] Gaceta sanitaria/S.E.S.P.A. S, 22(5), 492-497.
7. Bhutta, A. T., Cleves, M. A., Casey, P. H., Cradock, M. M., & Anand, K. J. S. (2002). Cognitive and behavioral outcomes of school-aged children who were born preterm: A meta-analysis. JAMA: The Journal of the Amer-ican Medical Association, 288(6), 728-737.
8. Bishop, F. L., Prescott, P., Chan, Y. K., Saville, J., von Elm, E., & Lewith, G. T. (2010). Prevalence of Complementary Medicine Use in Pediatric Cancer: A Systematic Review. Pediatrics, 125(4), 768-776.
9. Blagojevic, M., Jinks, C., Jeffery, A., & Jordan, K. (2010). Risk factors for onset of osteoarthritis of the knee in older adults: a systematic review and meta-analysis. Osteoarthritis and Cartilage, 18(1), 24-33.
10. Borghouts, J. A., Koes, B. W., & Bouter, L. M. (1998). The clinical course and prognostic factors of non-specific neck pain: A systematic review. Pain, 77(1), 1-13.
11. Buckingham, J., Fisher, B., & Saunders, D. (2003a). Worksheet for using an article about prognosis. Evidence Based Medicine Toolkit. Retrieved May 5, 2009, from http://www.ebm.med.ualberta.ca/
12. Buckingham, J., Fisher, B., & Saunders, D. (2003b). Worksheet for Using an Article About Causation of Harm. Evidence Based Medicine Toolkit. Re-trieved May 5, 2009, from http://www.ebm.med.ualberta.ca/
13. Cameron, I., Crotty, M., Currie, C., Finnegan, T., Gillespie, L., Gillespie, W., et al. (2000). Geriatric rehabilitation following fractures in older people: A systematic review. Health Technology Assessment, 4(2).
14. Campbell, H., & Rudan, I. (2002). Interpretation of genetic association stud-ies in complex disease. The Pharmacogenomics Journal, 2, 349-360.
15. Campos-Outcalt, D., Senf, J., Watkins, A. J., & Bastacky, S. (1995). The ef-fects of medical school curricula, faculty role models, and biomedical research support on choice of generalist physician careers: A review and quality assessment of the literature. Academic Medicine: Journal of the Asso-ciation of American Medical Colleges, 70(7), 611-619.
16. Carretero-Dios, H., & Pérez, C. (2007). Standards for the development and review of instrumental studies: Considerations about test selection in psychological research. International Journal of Clinical and Health Psychology, 7(3), 863-882.
17. Carruthers, S. G., Larochelle, P., Haynes, R. B., Petrasovits, A., & Schiffrin, E. L. (1993). Report of the canadian hypertension society consensus conference: 1. introduction. Canadian Medical Association Journal, 149(3), 289-293.
18. Centre for Evidence-Based Medicine (CEBM). Critical appraisal worksheets. Unpublished document. Retrieved April 4, 2009, from http://www.cebm.net/index.aspx?o=1913.
19. Cho, M. K., & Bero, L. A. (1994). Instruments for assessing the quality of drug studies published in the medical literature. The Journal of the American Medical Association, 272(2), 101-104.
20. Cole, D. C., & Hudak, P. L. (1996). Prognosis of nonspecific work-related musculoskeletal disorders of the neck and upper extremity. American Journal of Industrial Medicine, 292, 657-668.
21. Corrao, G., Bagnardi, V., Zambon, A., & Arico, S. (1999). Exploring the dose-response relationship between alcohol consumption and the risk of several alcohol-related conditions: A meta-analysis. Addiction, 94(10), 1551-1573.
22. Cowley, D. E. (1995). Prostheses for primary total hip replacement: A criti-cal appraisal of the literature. International Journal of Technology Assessment in Health Care, 11(4), 770-778.
23. Critical Appraisal Skills Programme Español (CASPe). (2008a). Critical ap-praisal tools [herramientas para lectura crítica]-Case-Control studies. CASPe. Retrieved May 5, 2009, from http://www.redcaspe.org/herramientas/index.htm
24. Critical Appraisal Skills Programme Español (CASPe). (2008b). Critical ap-praisal tools [herramientas para lectura crítica]-Cohort studies. CASPe. Retrieved May 5, 2009, from http://www.redcaspe.org/herramientas/index.htm
25. Deeks, J. J., Dinnes, J., D'Amico, R., Sowden, A. J., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D.G. (2003). Evaluating non-randomized intervention studies. Health Technology Assessment, 7(27), 1-173.
26. Downs, S. H., & Black, N. (1998). The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. Journal of Epidemi-ology and Community Health, 52(6), 377-384.
27. DuRant, R. H. (1994). Checklist for the evaluation of research articles. Jour-nal of Adolescent Health, 15(1), 4-8.
28. Effective Practice, Informatics and Quality Improvement (EPIQ). (2008a). Critically appraised topics (CATs) checklists for quantative studies: Prognostic and Risk Factor Studies. Retrieved January 9, 2010 from http://www.epiq.co.nz
29. Effective Practice, Informatics and Quality Improvement (EPIQ). (2008b). Critically appraised topics (CATs) checklists for quantative studies: Case-control studies. Retrieved January 9, 2010 from http://www.epiq.co.nz
30. Effective Public Health Practice Project (EPHPP). (2009). Quality assessment tool for quantitative studies. Retrieved January 9, 2010, from http://www.ephpp.ca/PDF/QATool.pdf
31. Esdaile, J. M., & Horwitz, R. I. (1986). Observational studies of cause-effect relationships: An analysis of methodologic problems as illustrated by the conflicting data for the role of oral contraceptives in the etiology of rheumatoid arthritis. Journal of Chronic Diseases, 39(10), 841-852.
32. Federal Focus (1996). Principles for evaluating epidemiologic data in regula-tory risk assessment. Unpublished document. Retrieved January 9, 2010, from http://www.fedfocus.org/science/london-principles.html.
33. Fowkes, F. G., & Fulton, P. M. (1991). Critical appraisal of published re-search: Introductory guidelines. British Medical Journal, 302, 1136-1140.
34. Gardner, M. J., Machin, D., & Campbell, M. J. (1986). Use of check lists in assessing the statistical content of medical studies. British Medical Journal, 292, 810-812.
35. Genaidy, A. M., Lemasters, G. K., Lockey, J., Succop, P., Deddens, J., Sobeih, T., & Dunning, K. (2007). An epidemiological appraisal instru-ment-a tool for evaluation of epidemiological studies. Ergonomics, 50(6), 920-960.
36. Glasgow University. Critical appraisal checklist for an article on harm or causation. Unpublished document. Retrieved January 9, 2010, from http://www.gla.ac.uk/media/media_64043_en.pdf
37. Glasziou, P., Vandenbroucke, J. P., & Chalmers, I. (2004). Assessing the quality of research. British Medical Journal, 328, 39-42.
38. Greer, N., Mosser, G., Logan, G., & Halaas, G. W. (2000). A practical ap-proach to evidence grading. Journal on Quality Improvement, 26(12), 700-712.
39. Gyorkos, T. W., Tannenbaum, T. N., Abrahamowicz, M., Oxman, A. D., Scott, E. A., Millson, M. E., et al. (1994). An approach to the develop-ment of practice guidelines for community health interventions. Canadi-an Journal of Public Health, 85(Supplement I), S8-S13.
40. Hadorn, D. C., Baker, D., Hodges, J. S., & Hicks, N. (1996). Rating the qual-ity of evidence for clinical practice guidelines. Journal of Clinical Epidemi-ology, 49(7), 749-754.
41. Harden, A., Garcia, J., Oliver, S., Rees, R., Shepherd, J., Brunton, G., & Oakley, A. (2004). Applying systematic review methods to studies of people's views: An example from public health research. Journal of Epi-demiology and Community Health, 58(9), 794-800.
42. Hernández-Avila, M., Garrido, F. y Salazar-Martínez, E. (2000). Sesgos en estudios epidemiológicos. Salud Pública de México, 42(5), 438-446.
43. Higgins, J. P. T., & Green, S. (2008). Cochrane handbook for systematic reviews of interventions. New York: John Wiley & Sons Inc.
44. Jüni, P., Altman, D. G., & Egger, M. (2001). Systematic reviews in health care: Assessing the quality of controlled clinical trials. British Medical Journal 323, 42-46.
45. Jüni, P., Witschi, A., Bloch, R., & Egger, M. (1999). The hazards of scoring the quality of clinical trials for meta-analysis. Journal of the American Medi-cal Association, 282(11), 1054-1060.
46. Khan, K.S., ter Riet, G., Glanville, J., Sowden, A.J., & Kleijnen, J. (2001). Undertaking systematic reviews of research on effectiveness. CRD's guidance for those carrying out or commissioning reviews. CRD Report Number 4 (2nd Edi-tion). Publications Office, NHS Centre for Reviews and Dissemination, University of York. Retrieved April 4, 2009, from http://www.york.ac.uk/inst/crd/pdf/crdreport4_complete.pdf
47. Kopec, J.A., & Esdaile, J.M. (1990). Bias in case-control studies. A review. Journal of Epidemiology and Community Health, 44(3), 179-186.
48. Kreulen, C. M., Creugers, N. H., & Meijering, A. C. (1998). Meta-analysis of anterior veneer restorations in clinical studies. Journal of Dentistry, 26(4), 345-353.
49. Krogh, C. L. (1985). A checklist system for critical review of medical litera-ture. Medical Education, 19, 392-395.
50. Kwakkel, G., Wagenaar, R. C., Kollen, B. J., & Lankhorst, G. J. (1996). Pre-dicting disability in stroke-a critical review of the literature. Age and Age-ing, 25(6), 479-489.
51. Laupacis, A., Wells, G., Richardson, W. S., & Tugwell, P. (1994). Users' guides to the medical literature. V. How to use an article about progno-sis. Journal of the American Medical Association, 272(3), 234-237.
52. Levine, M., Walter, S., Lee, H., Haines, T., Holbrook, A., & Moyer, V. (1994). Users' guides to the medical literature. IV. How to use an article about harm. Journal of the American Medical Association, 271(20), 1615-1619.
53. Lichtenstein, M. J., Mulrow, C. D., & Elwood, P. C. (1987). Guidelines for reading case-control studies. Journal of Chronic Diseases, 40(9), 893-903.
54. Liddle, J., Williamson, M., & Irwig, L. (1996). Method for evaluating research guideline evidence (MERGE). Sydney: NSW Health Department.
55. Littenberg, B., Weinstein, L. P., McCarren, M., Mead, T., Swiontkowski, M. F., Rudicel, S. A., et al. (1998). Closed fractures of the tibial shaft. The Journal of Bone and Joint Surgery, 80(2), 174-183.
56. Loney, P. L., & Stratford, P. W.(1999). The prevalence of low back pain in adults: A methodological review of the literature. Physical Therapy, 79(4), 384-396.
57. López de Argumedo, M., Reviriego, E., Andrío, E., Rico, R., Sobradillo, N., & Hurtado de Saracho, I. (2006). Revisión externa y validación de instrumen-tos metodológicos para la lectura crítica y la síntesis de la evidencia científica. Vito-ria-Gasteiz: Osteba-Servicio de Evaluación de Tecnologías Sanitarias. Departamento de Sanidad. Gobierno Vasco.
58. Mann, C. J. (2003). Observational research methods. Research design II: Cohort, cross sectional, and case-control studies. Emergency Medicine Journal, 20(1), 54-60.
59. Martin, J. L. R., Pérez, V., Sacristán, M., & Álvarez, E. (2005). Is grey litera-ture essential for a better control of publication bias in psychiatry? An example from three meta-analyses of schizophrenia. European Psychiatry, 20(8), 550-553.
60. Margetts, B. M., Thompson, R. L., Key, T., Durry, S., Nelson, M., Bingham, S., et al. (1995). Development of a scoring system to judge the scientific quality of information from case-control and cohort studies of nutrition and disease. Nutrition and Cancer, 24(3), 231-239.
61. Margetts, B. M., Vorster, H. H., & Venter, C. S. (2002). Evidence-based nu-trition-review of nutritional epidemiological studies. South African Jour-nal of Clinical Nutrition, 15, 68-73.
62. New Zealand Guidelines Group (2001). Handbook for the preparation of explicit evidence based clinical practice guidelines. Generic appraisal tool for epidemiology (GATE) methodology checklist. Unpublished document. Retrieved January 9, 2010, from http://www.nzgg.org.nz/index.cfm?fuseaction=down load&fusesubaction=template&libraryID=102
63. Nguyen, Q. V., Bezemer, P. D., Habets, L., & Prahl-Andersen, B. (1999). A systematic review of the relationship between overjet size and traumatic dental injuries. European Journal of Orthodontics, 21(5), 503-515.
64. Parker, S. (2006). Guidelines for the critical appraisal of a paper. Unpublished doc-ument. Retrieved January 9, 2010, from http://www.surgical-tutor.org.uk/default-home.htm?papers/appraisal.htm~right.
65. Pérez-Ri{dotless}os, M., Ruano-Ravina, A., Etminan, M., & Takkouche, B. (2009). A meta-analysis on wood dust exposure and risk of asthma. Allergy, 65, 467-473.
66. Rangel, S. J., Kelsey, J., Colby, C. E., Anderson, J. D., & Moss, R. L. (2003). Development of a quality assessment scale for retrospective clinical studies in pediatric surgery. Journal of Pediatric Surgery, 38(3), 390-396.
67. Reed, D. A., Cook, D. A., Beckman, T. J., Levine, R. B., Kern, D. E., & Wright, S. M. (2007). Association between funding and quality of published medical education research. Journal of the American Medical Associa-tion, 298(9), 1002-1009.
68. Reisch, J. S., Tyson, J. E., & Mize, S. G. (1989). Aid to the evaluation of therapeutic studies. Pediatrics, 84(5), 815-827.
69. Sanderson, S., Tatt, I. D., & Higgins, J. P. (2007). Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: A systematic review and annotated bibliography. International Journal of Epi-demiology, 36(3), 666-676.
70. o 50. Edinburgh: SIGN.
71. Shadish, W.R., Cook, T.D., & Campbell, D.T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.
72. Scottish Intercollegiate Guidelines Network. (2008). SIGN 50: A guideline de-veloper's handbook. Edinburgh: SIGN.
73. Shaughnessy, J.J., Zechmeister, E.B., & Zechmeister, J.S. (2007). Métodos de investigación en psicología, 7a ed. Madrid: McGraw-Hill.
74. Sica, G.T. (2006). Bias in Research Studies. Radiology, 238 (3), 780-789.
75. Solomon, D. H., Bates, D. W., Panush, R. S., & Katz, J. N. (1997). Costs, outcomes, and patient satisfaction by provider type for patients with rheumatic and musculoskeletal conditions: A critical review of the lit-erature and proposed methodologic standards. Annals of Internal Medi-cine, 127(1), 52-60.
76. Spitzer, W. O., Lawrence, V., Dales, R., Hill, G., Archer, M. C., Clark, P., Morgan, P.P. (1990). Links between passive smoking and disease: A best-evidence synthesis. Clinical and Investigative Medicine, 13(1), 17-42.
77. Steinberg, E. P., Eknoyan, G., Levin, N. W., Eschbach, J. W., Golper, T. A., Owen, W. F., et al. (2000). Methods used to evaluate the quality of evi-dence underlying the national kidney foundation-Dialysis outcomes quality initiative clinical practice guidelines: Description, findings, and implications. American Journal of Kidney Diseases, 36(1), 1-9.
78. Stock, S. R. (1991). Workplace ergonomic factors and the development of musculoskeletal disorders of the neck and upper limbs: A meta-analysis. American Journal of Industrial Medicine, 19(1), 87-107.
79. Streiner, D. L., Norman, G. R., & Fulton, C. (1991). Health measurement scales: a practical guide to their development and use. International Jour-nal of Rehabilitation Research, 14(4), 364.
80. The Joanna Briggs Institute (2008). Joanna briggs institute reviewers' manual. Un-published document. Retrieved January 9, 2010, from http://www.joann riggs.edu.au/pdf/JBIReviewManual_CiP11449.pdf.
81. Tseng, T. Y., Breau, R. H., Fesperman, S. F., Vieweg, J., & Dahm, P. (2008). Evaluating the evidence: the methodological and reporting quality of comparative observational studies of surgical interventions in urological publications. British Journal of Urology International,103, 1026-1031.
82. Valentine, J.C., & Cooper, H. (2008) A systematic and transparent approach for assessing the methodological quality of intervention effectiveness research: the Study Design and Implementation Assessment Device (Study DIAD). Psychological Methods, 13(2), 130-149.
83. van der Windt, D. A., Thomas, E., Pope, D. P., de Winter, A. F., Macfar-lane, G. J., Bouter, L. M., et al. (2000). Occupational risk factors for shoulder pain: A systematic review. Occupational and Environmental Medi-cine, 57(7), 433-442.
84. Vandenbroucke, J. P., von Elm, E., Altman, D. G., Gøtzsche, P. C., Mulrow, C. D., Pocock, S. J., Poole, C., Schlesselman, J.J., & Egger, M. (2007). Strengthening the reporting of observational studies in epidemi-ology (STROBE): Explanation and elaboration. Epidemiology, 18(6), 805-835.
85. Vitali, S. H., & Randolph, A. G. (2005). Assessing the quality of case-control association studies on the genetic basis of sepsis. Pediatric Critical Care Medicine, 6(3), S74-S77.
86. Weightman, A. L., Mann, M. K., Sander, L., & Turley, R. L. (2004). Ques-tions to assist with the critical appraisal of an observational study e.g. cohort, case-control, cross-sectional. Health Evidence Bulletins-Wales, 2009.
87. Wells, G. A., et al. The newcastle-ottawa scale (NOS) for assessing the quality of non-randomized studies in meta-analyses., Unpublished document. Retrieved January 9, 2010, from http://www.ohri.ca/programs/clinical_epidemiology/oxford.htm.
88. Wells, K., & Littell, J. H. (2009). Study quality assessment in systematic re-views of research on intervention effects. Research on Social Work Practice, 19(1), 52-62.
89. Welsh Child Protection Systematic Review Group (2006). Evidence sheet: Child prOtection neurological injuries CONI) critical appraisal forms. university of wales. Unpublished document. Retrieved April 4, 2009, from http://www.core-info.cardiff.ac.uk.
90. West, S., King, V., Carey, T. S., Lohr, K. N., McKoy, N., Sutton, S. F., & Lux, L. (2002). Systems to rate the strength of scientific evidence (Evidence Re-port/Technology Assessment No. 47. AHRQ Publication No. 02-E016). Rockville, MD: Agency for Healthcare Research and Quality.
91. Wong, W. C., Cheung, C. S., & Hart, G. J. (2008). Development of a quality assessment tool for systematic reviews of observational studies QATSO) of HIV prevalence in men having sex with men and associat-ed risk behaviours. Emerging Themes in Epidemiology, 5, 23-26.
92. Zaza, S., Wright-De Agüero, L. K., Briss, P. A., Truman, B. I., Hopkins, D. P., Hennessy, M. H., Pappaioanou, M. (2000). Data collection instrument and procedure for systematic reviews in the Guide to Community Preventive Services. Task force on community preventive services. American Journal of Preventive Medicine, 18(1S), 44-74.
93. Zola, P., Volpe, T., Castelli, G., Sismondi, P., Nicolucci, A., Parazzini, F., et al. (1989). Is the published literature a reliable guide for deciding be-tween alternative treatments for patients with early cervical cancer? In-ternational Journal of Radiation Oncology, Biology, Physics, 16(3), 785-797.