How credible are the study results? Evaluating and applying internal validity tools to literature-based assessments of environmental health hazards

Environment International

Volumn 92-93, Issue , 2016, Pages 617-629

  Rooney, Andrew A ^a   Cooper, Glinda S ^b   Jahnke, Gloria D ^a   Lam, Juleen ^c   Morgan, Rebecca L ^d   Boyles, Abee L ^a   Ratcliffe, Jennifer M ^e   Kraft, Andrew D ^b   Schünemann, Holger J ^d   Schwingl, Pamela ^e   Walker, Teneille D ^b   Thayer, Kristina A ^a   Lunn, Ruth M ^a  

^a NATIONAL INSTITUTE OF ENVIRONMENTAL HEALTH SCIENCES   (United States)

^b U S ENVIRONMENTAL PROTECTION AGENCY   (United States)

^c UNIVERSITY OF CALIFORNIA   (United States)

^d MCMASTER UNIVERSITY   (Canada)

^e INTEGRATED LABORATORY SYSTEMS   (United States)

Author keywords

Environmental health; Hazard assessment; Internal validity; Risk of bias; Systematic review

Indexed keywords

ANIMALS; BIOHAZARDS; DECISION MAKING; ENVIRONMENTAL PROTECTION AGENCY; GRADING; HEALTH; HEALTH RISKS; PROGRAM TRANSLATORS; RISK ASSESSMENT;

ENVIRONMENTAL HEALTH; HAZARD ASSESSMENT; HETEROGENEOUS COLLECTIONS; INTERNAL VALIDITY; NATIONAL TOXICOLOGY PROGRAMS; SOURCES OF INFORMATIONS; SYSTEMATIC REVIEW; U.S. ENVIRONMENTAL PROTECTION AGENCY;

HEALTH HAZARDS;

ENVIRONMENTAL ASSESSMENT; ENVIRONMENTAL HAZARD; HAZARD ASSESSMENT; RISK ASSESSMENT;

ARTICLE; CONFLICT OF INTEREST; CONFOUNDING VARIABLE; DECISION MAKING; ECOLOGICAL FALLACY; ENVIRONMENTAL EXPOSURE; ENVIRONMENTAL HEALTH; HEALTH HAZARD; HUMAN; INTERNAL VALIDITY; NONHUMAN; OUTCOME ASSESSMENT; PRIORITY JOURNAL; RISK ASSESSMENT; SENSITIVITY ANALYSIS; LITERATURE; PROCEDURES; PUBLIC HEALTH;

ANIMALIA;

DECISION MAKING; ENVIRONMENTAL HEALTH; HUMANS; PUBLIC HEALTH; REVIEW LITERATURE AS TOPIC;

EID: 84956855539     PISSN: 01604120     EISSN: 18736750     Source Type: Journal    
DOI: 10.1016/j.envint.2016.01.005     Document Type: Article

References (55)

1. Agerstrand, M., Kuster, A., Bachmann, J., Breitholtz, M., Ebert, I., Rechenberg, B., et al. Reporting and evaluation criteria as means towards a transparent use of ecotoxicity data for environmental risk assessment of pharmaceuticals. Environ. Pollut. 159 (2011), 2487–2492.
2. AHRQ (Agency for Healthcare Research and Quality), AHRQ training modules for the systematic reviews methods guide., 2013 Available: http://www.effectivehealthcare.ahrq.gov/index.cfm/tools-and-resources/slide-library/ (accessed 11 October 2013).
3. Atkins, D., Eccles, M., Flottorp, S., Guyatt, G.H., Henry, D., Hill, S., et al. Systems for grading the quality of evidence and the strength of recommendations I: critical appraisal of existing approaches The GRADE Working Group. BMC Health Serv. Res., 4, 2004, 38.
4. Bal-Price, A., Coecke, S., Guidance on Good Cell Culture Practice (GCCP). Aschner, M., Suñol, C., Bal-Price, A., (eds.) Cell Culture Techniques Neuromethods, vol. 56, 2011, Humana Press, New York, NY, 1–25.
5. Balshem, H., Helfand, M., Schunemann, H.J., Oxman, A.D., Kunz, R., Brozek, J., et al. GRADE guidelines: 3. Rating the quality of evidence. J. Clin. Epidemiol. 64 (2011), 401–406.
6. Barnes, D.E., Bero, L.A., Scientific quality of original research articles on environmental tobacco smoke. Tob. Control. 6 (1997), 19–26.
7. Barnes, D.E., Bero, L.A., Why review articles on the health effects of passive smoking reach different conclusions. J. Am. Med. Assoc. 279 (1998), 1566–1570.
8. Bero, L.A., Why the Cochrane risk of bias tool should include funding source as a standard item. Cochrane Database Syst. Rev., 12, 2013, ED000075.
9. Bero, L., Oostvogel, F., Bacchetti, P., Lee, K., Factors associated with findings of published trials of drug–drug comparisons: why some statins appear more efficacious than others. PLoS Med., 4, 2007, e184.
10. Beronius, A., Molander, L., Ruden, C., Hanberg, A., Facilitating the use of non-standard in vivo studies in health risk assessment of chemicals: a proposal to improve evaluation criteria and reporting. J. Appl. Toxicol. 34 (2014), 607–617.
11. Birnbaum, L.S., Thayer, K.A., Bucher, J.R., Wolfe, M.S., Implementing systematic review at the National Toxicology Program. Environ. Health Perspect. 121 (2013), A108–A109.
12. Blair, A., Stewart, P., Lubin, J.H., Forastiere, F., Methodological issues regarding confounding and exposure misclassification in epidemiological studies of occupational exposures. Am. J. Ind. Med. 50 (2007), 199–207.
13. Christensen, K., Christensen, C.H., Wright, J.M., Galizia, A., Glenn, B.S., Scott, C.S., et al. The use of epidemiology in risk assessment: challenges and opportunities. Hum. Ecol. Risk. Assess. Int. J. 21 (2014), 1644–1663.
14. Cooper, G.S., Lunn, R.M., Ågerstrand, M., Glenn, B.S., Kraft, A.D., Luke, A.M., et al. Study sensitivity: Evaluating the ability to detect effects in systematic reviews of chemical exposures. Environ. Int. 92-93 (2016), 605–610 (in this issue).
15. Craig, P., Cooper, C., Gunnell, D., Haw, S., Lawson, K., Macintyre, S., et al. Using natural experiments to evaluate population health interventions: new Medical Research Council guidance. J. Epidemiol. Community Health 66 (2012), 1182–1186.
16. Dunn, A.G., Arachi, D., Hudgins, J., Tsafnat, G., Coiera, E., Bourgeois, F.T., Financial conflicts of interest and conclusions about neuraminidase inhibitors for influenza: an analysis of systematic reviews. Ann. Intern. Med. 161 (2014), 513–518.
17. ECVAM (European Centre for the Validation of Alternative Methods), ToxRTool — Toxicological data Reliability Assessment Tool., 2009 Available: http://ecvam.jrc.it/page_pdf.cfm?voce=s&idvoce=110.
18. EFSA (European Food Safety Authority), Application of systematic review methodology to food and feed safety assessments to support decision making. EFSA J., 2010, 1–90 (Available: http://www.efsa.europa.eu/en/efsajournal/pub/1637.htm accessed 15 October 2013).
19. Ghio, A.J., Sobus, J.R., Pleil, J.D., Madden, M.C., Controlled human exposures to diesel exhaust. Swiss Med. Wkly., 142, 2012, w13597.
20. Guyatt, G.H., Oxman, A.D., Kunz, R., Woodcock, J., Brozek, J., Helfand, M., et al. GRADE guidelines: 7. Rating the quality of evidence–inconsistency. J. Clin. Epidemiol. 64 (2011), 1294–1302.
21. Guyatt, G.H., Oxman, A.D., Montori, V., Vist, G., Kunz, R., Brozek, J., et al. GRADE guidelines: 5. Rating the quality of evidence–publication bias. J. Clin. Epidemiol. 64 (2011), 1277–1282.
22. Guyatt, G.H., Oxman, A.D., Vist, G., Kunz, R., Brozek, J., Alonso-Coello, P., et al. GRADE guidelines: 4. Rating the quality of evidence-study limitations (risk of bias). J. Clin. Epidemiol. 64 (2011), 407–415.
23. Higgins, J., Green, S., Cochrane Handbook for Systematic Reviews of Interventions., 2011 Available: www.cochrane-handbook.org (accessed 3 February 2013).
24. Hill, A.B., The environment and disease: association or causation?. Proc. R. Soc. Med. 58 (1965), 295–300.
25. Hirst, J.A., Howick, J., Aronson, J.K., Roberts, N., Perera, R., Koshiaris, C., et al. The need for randomization in animal trials: an overview of systematic reviews. PLoS One, 9, 2014, e98856.
26. IARC (International Agency for Reserach on Cancer), Cancer: Causes, Occruances and Control. No. 100, 1990, IARC Scientific Publications, Lyon, France, 352.
27. IOM (Institute of Medicine), Conflict of Interest in Medical Research, Education, and Practice. 2009, The National Academies Press, Washington, DC, 414 (Available: http://www.nap.edu/download.php?record_id=12598# accessed 23 May 2015).
28. IOM (Institute of Medicine), Finding What Works in Health Care: Standards for Systematic Reviews. 2011, The National Academies Press, Washington, DC, 318 (Available: http://www.nap.edu/openbook.php?record_id=13059 accessed 3 May 2013).
29. Krauth, D., Anglemyer, A., Philipps, R., Bero, L., Nonindustry-sponsored preclinical studies on statins yield greater efficacy estimates than industry-sponsored studies: a meta-analysis. PLoS Biol., 12, 2014, e1001770.
30. Krauth, D., Woodruff, T., Bero, L., Instruments for assessing risk of bias and other methodological criteria of published animal studies: a systematic review. Environ. Health Perspect. 121 (2013), 985–992.
31. Lexchin, J., Bero, L.A., Djulbegovic, B., Clark, O., Pharmaceutical industry sponsorship and research outcome and quality: systematic review. Br. Med. J. 326 (2003), 1167–1170.
32. Liberati, A., Altman, D.G., Tetzlaff, J., Mulrow, C., Gotzsche, P.C., Ioannidis, J.P., et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med., 6, 2009, e1000100.
33. Lundh, A., Sismondo, S., Lexchin, J., Busuioc, O.A., Bero, L., Industry sponsorship and research outcome. Cochrane Database Syst. Rev., 12, 2012, MR000033.
34. McPartland, J., Lam, J., Lanier-Christensen, C., A valuable contribution toward adopting systematic review in environmental health. Environ. Health Perspect., 122, 2014, A10.
35. Mollenhauer, M.A., Bradshaw, S.G., Fair, P.A., McGuinn, W.D., Peden-Adams, M.M., Effects of perfluorooctane sulfonate (PFOS) exposure on markers of inflammation in female B6C3F1 mice. J. Environ. Sci. Health, Part A: Tox. Hazard. Subst. Environ. Eng. 46 (2011), 97–108.
36. Morgan, R.L., Thayer, K.A., et al. GRADE: Assessing the quality of evidence in environmental and occupational health. Environ. Int. 92-93 (2016), 611–616 (in this issue).
37. NRC (National Research Council), Review of EPA's Integrated Risk Information System (IRIS) Process. 2014, Press TNA, Washington, DC, 1–154 (Available: http://www.nap.edu/openbook.php?record_id=18764 accessed 5 January 2015).
38. NTP (National Toxicology Program), Webinar on OHAT approach for systematic review., 2013 (September 26, 2013. Available: http://ntp.niehs.nih.gov/go/40490 accessed 28 January 2014).
39. NTP (National Toxicology Program), Informational meeting on the draft OHAT Approach for Systematic Review and Evidence Integration for Literature-based Health Assessments. 2013 (April 23, 2013. Available: http://ntp.niehs.nih.gov/go/38751 accessed 28 January 2014).
40. NTP (National Toxicology Program), Handbook for Preparing Report on Carcinogens Monographs - July 2015. 2015, Carcinogens OotRo, RTP, NC (Available: http://ntp.niehs.nih.gov/go/rochandbook accessed 20 July 2015).
41. NTP (National Toxicology Program), OHAT Risk of Bias Rating Tool for Human and Animal Studies - January 2015. 2015, Office of Health Assessment and Translation, RTP, NC (Available: http://ntp.niehs.nih.gov/go/38673 accessed 25 Jan 2015).
42. NTP (National Toxicology Program), Handbook for Conducting a Literature-Based Health Assessment Using OHAT Approach for Systematic Review and Evidence — January 9 2015. 2015, Office of Health Assessment and Translation, RTP, NC (Available: http://ntp.niehs.nih.gov/go/38673 accessed 25 Jan 2015).
43. Perlis, R.H., Perlis, C.S., Wu, Y., Hwang, C., Joseph, M., Nierenberg, A.A., Industry sponsorship and financial conflict of interest in the reporting of clinical trials in psychiatry. Am. J. Psychiatry 162 (2005), 1957–1960.
44. Popelut, A., Valet, F., Fromentin, O., Thomas, A., Bouchard, P., Relationship between sponsorship and failure rate of dental implants: a systematic approach. PLoS One, 5, 2010, e10274.
45. Rich, D.Q., Liu, K., Zhang, J., Thurston, S.W., Stevens, T.P., Pan, Y., et al. Differences in birth weight associated with the 2008 Beijing Olympic air pollution reduction: results from a natural experiment. Environ. Health Perspect., 2015 (advanced publication).
46. Rooney, A.A., Boyles, A.L., Wolfe, M.S., Bucher, J.R., Thayer, K.A., Systematic review and evidence integration for literature-based environmental health science assessments. Environ. Health Perspect. 122 (2014), 711–718.
47. Rothman, K.J., Greenland, S., Causation and causal inference in epidemiology. Am. J. Public Health 95:Suppl. 1 (2005), S144–S150.
48. Schneider, K., Schwarz, M., Burkholder, I., Kopp-Schneider, A., Edler, L., Kinsner-Ovaskainen, A., et al. "ToxRTool", a new tool to assess the reliability of toxicological data. Toxicol. Lett. 189 (2009), 138–144.
49. Schünemann, H.J., Oxman, A.D., Higgins, J.P.T., Vist, G.E., Glasziou, P., Guyatt, G.H., et al. Chapter 11: presenting results and ‘summary of findings’ tables. JPT, H., Green, S., (eds.) Cochrane Handbook for Systematic Reviews of Interventions Version 510 [updated March 2011], 2012, The Cochrane Collaboration.
50. Sterne, J.A., Why the Cochrane risk of bias tool should not include funding source as a standard item. Cochrane Database Syst. Rev., 12, 2013, ED000076.
51. Sterne, J., Higgins, J., Reeves, B., on behalf of the development group for ACROBAT-NRSI, A Cochrane Risk Of Bias Assessment Tool: for Non-Randomized Studies of Interventions (ACROBAT-NRSI), Version 1.0.0. 2014 (Available: http://www.riskofbias.info accessed 28 September 2014).
52. Thayer, K.A., Wolfe, M.S., Rooney, A.A., Boyles, A.L., Bucher, J.R., Birnbaum, L.S., Intersection of systematic review methodology with the NIH reproducibility initiative. Environ. Health Perspect. 122 (2014), A176–A177.
53. Viswanathan, M., Ansari, M., Berkman, N.D., Chang, S., Hartling, L., McPheeters, L.M., et al. Assessing the Risk of Bias of Individual Studies When Comparing Medical Interventions. 2012, Agency for Healthcare Research and Quality (AHRQ), Rockville, MD (Publication No. 12-EHC047-EF. Available: http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productid=998 accessed 3 January 2013).
54. WHO (International Programme on Chemical Safety, World Health Organization), Principles for the Assessment of Risks to Human Health From Exposure to Chemicals. 1999, IPCS Environmental Health Criteria 210, Geneva, 312 (Available: http://www.inchem.org/documents/ehc/ehc/ehc210.htm).
55. Woodruff, T.J., Sutton, P., The Navigation Guide Systematic Review Methodology: a rigorous and transparent method for translating environmental health science into better health outcomes. Environ. Health Perspect. 122 (2014), 1007–1014.