Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study

PLoS Medicine

Volumn 15, Issue 11, 2018, Pages

  Zech, John R ^a   Badgeley, Marcus A ^b   Liu, Manway ^b   Costa, Anthony B ^c   Titano, Joseph J ^c   Oermann, Eric Karl ^c  

^a CALIFORNIA PACIFIC MEDICAL CENTER   (United States)

^b VERILY LIFE SCIENCES   (United States)

^c MOUNT SINAI SCHOOL OF MEDICINE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ADULT; ALGORITHM; ARTICLE; ATELECTASIS; CARDIOMEGALY; CROSS-SECTIONAL STUDY; DISEASE BURDEN; EDEMA; EMPHYSEMA; FEMALE; HOSPITAL PLANNING; HUMAN; JOINT EFFUSION; LEARNING; MAJOR CLINICAL STUDY; MALE; MIDDLE AGED; NATIONAL HEALTH ORGANIZATION; NERVE CELL NETWORK; PATIENT CARE; PNEUMONIA; PREDICTIVE VALUE; PREVALENCE; RECEIVER OPERATING CHARACTERISTIC; TASK PERFORMANCE; THORAX RADIOGRAPHY; VALIDATION PROCESS; AGED; CLINICAL TRIAL; COMPUTER ASSISTED DIAGNOSIS; DIAGNOSTIC IMAGING; MULTICENTER STUDY; PROCEDURES; RADIOLOGY INFORMATION SYSTEM; REPRODUCIBILITY; RETROSPECTIVE STUDY; UNITED STATES;

ADULT; AGED; CROSS-SECTIONAL STUDIES; DEEP LEARNING; DIAGNOSIS, COMPUTER-ASSISTED; FEMALE; HUMANS; MALE; MIDDLE AGED; PNEUMONIA; PREDICTIVE VALUE OF TESTS; RADIOGRAPHIC IMAGE INTERPRETATION, COMPUTER-ASSISTED; RADIOGRAPHY, THORACIC; RADIOLOGY INFORMATION SYSTEMS; REPRODUCIBILITY OF RESULTS; RETROSPECTIVE STUDIES; UNITED STATES;

EID: 85056277365     PISSN: 15491277     EISSN: 15491676     Source Type: Journal    
DOI: 10.1371/journal.pmed.1002683     Document Type: Article

References (29)

1. Wang X, Peng Y, Lu L, Lu Z, Bagheri M, Summers RM, ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases; 2017 [cited 1 July 2018]. Preprint. Available from: https://arxiv.org/abs/1705.02315
2. Rajpurkar P, Irvin J, Zhu K, Yang B, Mehta H, Duan T, et al. CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning; 2017 [cited 1 July 2018]. Preprint. Available from: https://arxiv.org/abs/1711.05225
3. Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, et al. Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs. JAMA. 2016;316: 2402–2410. doi: 10.1001/jama.2016.17216 pmid: 27898976
4. Ting DSW, Cheung CY-L, Lim G, Tan GSW, Quang ND, Gan A, et al. Development and Validation of a Deep Learning System for Diabetic Retinopathy and Related Eye Diseases Using Retinal Images From Multiethnic Populations With Diabetes. JAMA. 2017;318: 2211–2223. doi: 10.1001/jama.2017.18152 pmid: 29234807
5. Kermany DS, Goldbaum M, Cai W, Valentim CCS, Liang H, Baxter SL, et al. Identifying Medical Diagnoses and Treatable Diseases by Image-Based Deep Learning. Cell. Elsevier; 2018;172: 1122–1131. doi: 10.1016/j.cell.2018.02.010 pmid: 29474911
6. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, et al. ImageNet Large Scale Visual Recognition Challenge; 2014 [cited 1 July 2018]. Preprint. Available from: https://arxiv.org/abs/1409.0575
7. Lecun Y, Bottou L, Bengio Y, Haffner P, Gradient-based learning applied to document recognition. In Intelligent signal processing. IEEE Press. 2001. p. 306–351
8. Recht B, Roelofs R, Schmidt L, Shankar V, Do CIFAR-10 Classifiers Generalize to CIFAR-10?; 2018 [cited 1 July 2018]. Preprint. Available from: https://arxiv.org/abs/1806.00451
9. Rothwell PM, External validity of randomised controlled trials: “To whom do the results of this trial apply?”Lancet. 2005;365: 82–93. doi: 10.1016/S0140-6736(04)17670-8 pmid: 15639683
10. Pandis N, Chung B, Scherer RW, Elbourne D, Altman DG, CONSORT 2010 statement: extension checklist for reporting within person randomised trials. BMJ. 2017;357: j2835. doi: 10.1136/bmj.j2835 pmid: 28667088
11. Cabitza F, Rasoini R, Gensini GF, Unintended Consequences of Machine Learning in Medicine. JAMA. 2017;318: 517–518. doi: 10.1001/jama.2017.7797 pmid: 28727867
12. Demner-Fushman D, Kohli MD, Rosenman MB, Shooshan SE, Rodriguez L, Antani S, et al. Preparing a collection of radiology examinations for distribution and retrieval. J Am Med Inform Assoc. 2016;23: 304–310. doi: 10.1093/jamia/ocv080 pmid: 26133894
13. LeCun Y, Bengio Y, Hinton G, Deep learning. Nature. 2015;521: 436–444. doi: 10.1038/nature14539 pmid: 26017442
14. He K, Zhang X, Ren S, Sun J, Deep Residual Learning for Image Recognition; 2015 [cited 1 July 2018]. Preprint. Available from: https://arxiv.org/abs/1512.03385
15. Huang G, Liu Z, Weinberger KQ, van der Maaten L, Densely Connected Convolutional Networks; 2016 [cited 1 July 2018]. Preprint. Available from: arXiv: https://arxiv.org/abs/1608.06993
16. Zhang C, Bengio S, Hardt M, Recht B, Vinyals O, Understanding deep learning requires rethinking generalization; 2016 [cited 1 July 2018]. Preprint. Available from: https://arxiv.org/abs/1611.03530
17. Zech J, Pain M, Titano J, Badgeley M, Schefflein J, Su A, et al. Natural Language-based Machine Learning Models for the Annotation of Clinical Radiology Reports. Radiology. 2018;287(2): 570–580. doi: 10.1148/radiol.2018171093 pmid: 29381109
18. Paszke A, Gross S, Chintala S, Chanan G, Yang E, DeVito Z, et al. Automatic differentiation in PyTorch; 2017 [cited 1 July 2018]. Preprint. Available from: https://openreview.net/forum?id=BJJsrmfCZ
19. Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A, Learning Deep Features for Discriminative Localization; 2015. Preprint. Available from: https://arxiv.org/abs/1512.04150
20. DeLong ER, DeLong DM, Clarke-Pearson DL, Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44: 837–845. pmid: 3203132
21. Robin X, Turck N, Hainard A, Tiberti N, Lisacek F, Sanchez J-C, et al. pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics. 2011;12: 77. doi: 10.1186/1471-2105-12-77 pmid: 21414208
22. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, et al. Scikit-learn: Machine Learning in Python. J Mach Learn Res. 2011;12: 2825–2830
23. Collins GS, Reitsma JB, Altman DG, Moons KGM, Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): The TRIPOD Statement. Ann Intern Med2015; 162:55–63. doi: 10.7326/M14-0697 pmid: 25560714
24. Steyerberg EW, Vergouwe Y, Towards better clinical prediction models: seven steps for development and an ABCD for validation. European Heart Journal2014; 35:1925–1931. doi: 10.1093/eurheartj/ehu207 pmid: 24898551
25. Gale W, Oakden-Rayner L, Carneiro G, Bradley AP, Palmer LJ, Detecting hip fractures with radiologist-level performance using deep neural networks; 2017 [cited 1 July 2018]. Preprint. Available from: https://arxiv.org/abs/1711.06504
26. Mutasa S, Chang PD, Ruzal-Shapiro C, Ayyala R, MABAL: a Novel Deep-Learning Architecture for Machine-Assisted Bone Age Labeling. J Digit Imaging. 2018;31(4): 513–519. doi: 10.1007/s10278-018-0053-3 pmid: 29404850
27. Geras KJ, Wolfson S, Gene Kim S, Moy L, Cho K, High-Resolution Breast Cancer Screening with Multi-View Deep Convolutional Neural Networks; 2017 [cited 1 July 2018]. Preprint. Available from: https://arxiv.org/abs/1703.07047
28. Oakden-Rayner L, Exploring the ChestXray14 dataset: problems. 18Dec2017 [cited 26 Jan 2018]. In: Luke Oakden-Rayner [Internet]. Available from: https://lukeoakdenrayner.wordpress.com/2017/12/18/the-chestxray14-dataset-problems/
29. Mandell LA, Wunderink RG, Anzueto A, Bartlett JG, Campbell GD, Dean NC, et al. Infectious Diseases Society of America/American Thoracic Society consensus guidelines on the management of community-acquired pneumonia in adults. Clin Infect Dis. 2007;44Suppl 2: S27–72