Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer

JAMA - Journal of the American Medical Association

Volumn 318, Issue 22, 2017, Pages 2199-2210

  Bejnordi, Babak Ehteshami ^a   Veta, Mitko ^b   Van Diest, Paul Johannes ^c   Van Ginneken, Bram ^a   Karssemeijer, Nico ^a   Litjens, Geert ^a   Van Der Laak, Jeroen A W M ^a   Hermsen, Meyke ^c,e   Manson, Quirine F ^a   Balkenhol, Maschenka ^e   Geessink, Oscar ^c,e,f   Stathonikos, Nikolaos ^a   Van Dijk, Marcory C R F ^g   Bult, Peter ^e   Beca, Francisco ^h   Beck, Andrew H ^h,i   Wang, Dayong ^h,i   Khosla, Aditya ^i,j   Gargeya, Rishab ^k   Irshad, Humayun ^h   Zhong, Aoxiao ^l   Dou, Qi ^l,m   Li, Quanzheng ^l   Chen, Hao ^m   Lin, Huang Jing ^m   Heng, Pheng Ann ^m   Haß, Christian ⁿ   Bruni, Elia ⁿ   Wong, Quincy ^o   Halici, Ugur ^p   Öner, Mustafa Ümit ^p   Cetin Atalay, Rengul ^p   Berseth, Matt ^q   Khvatkov, Vitali ^r   Vylegzhanin, Alexei ^r   Kraus, Oren ^s   Shaban, Muhammad ^t   Rajpoot, Nasir ^t,u   Awan, Ruqayya ^v   Sirinukunwattana, Korsuk ^t   Qaiser, Talha ^t   Tsang, Yee Wah ^u   Tellez, David ^a   Annuscheit, Jonas ^w   Hufnagl, Peter ^w   Valkonen, Mira ^x   Kartasalo, Kimmo ^w,x   Latonen, Leena ^x   Ruusuvuori, Pekka ^w,x   Liimatainen, Kaisa ^w   Albarqouni, Shadi ^y   Mungal, Bharti ^y   George, Ami ^y   Demirci, Stefanie ^y   Navab, Nassir ^y   Watanabe, Seiryo ^z   Seno, Shigeto ^z   Takenaka, Yoichi ^z   Matsuda, Hideo ^z   Phoulady, Hady Ahmady ^aa   Kovalev, Vassili ^ab   Kalinovsky, Alexander ^ab   Liauchuk, Vitali ^ab   Bueno, Gloria ^ac   Fernandez Carrobles, M Milagro ^ac   Serrano, Ismael ^ac   Deniz, Oscar ^ac   Racoceanu, Daniel ^d,ad   Venâncio, Rui ^e   more..

^a RADBOUD UNIVERSITY MEDICAL CENTER   (Netherlands)

^b EINDHOVEN UNIVERSITY OF TECHNOLOGY   (Netherlands)

^c UNIVERSITY MEDICAL CENTER UTRECHT   (Netherlands)

^d PONTIFICIA UNIVERSIDAD CATÓLICA DEL PERÚ   (Peru)

^e SORBONNE UNIVERSITÉ   (France)

^f Laboratorium Pathologie Oost Nederland   (Netherlands)

^g RIJNSTATE HOSPITAL   (Netherlands)

^h HARVARD MEDICAL SCHOOL   (United States)

ⁱ PathAI Inc Inc   (United States)

^j MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^k The Harker School   (United States)

^l MASSACHUSETTS GENERAL HOSPITAL   (United States)

^m CHINESE UNIVERSITY OF HONG KONG   (Hong Kong)

ⁿ ExB Communication Systems GmbH   (Germany)

^o Munich Business School   (Germany)

^p MIDDLE EAST TECHNICAL UNIVERSITY   (Turkey)

^q NLP Logix LLC   (United States)

^r Smart Imaging Technologies   (United States)

^s UNIVERSITY OF TORONTO   (Canada)

^t UNIVERSITY OF WARWICK   (United Kingdom)

^u UNIVERSITY HOSPITALS COVENTRY AND WARWICKSHIRE NHS TRUST   (United Kingdom)

^v Hochschule für Technik und Wirtschaft ^*  (Qatar)

^w Institut für Klimaschutz Energie und Mobilität IKEM Hochschule für Wirtschaft und Verwaltung HWR   (Germany)

^x TAMPERE UNIVERSITY OF TECHNOLOGY   (Finland)

^y TECHNICAL UNIVERSITY OF MUNICH   (Germany)

^z OSAKA UNIVERSITY   (Japan)

^aa UNIVERSITY OF SOUTH FLORIDA   (United States)

^ab UNITED INSTITUTE OF INFORMATICS PROBLEMS   (Belarus)

^ac UNIVERSITY OF CASTILLA LA MANCHA   (Spain)

^ad INSERM   (France)

more..

Author keywords

[No Author keywords available]

Indexed keywords

EOSIN; HEMATOXYLIN;

ARTICLE; AUTOMATION; BREAST CANCER; CANCER CLASSIFICATION; CANCER DIAGNOSIS; DIAGNOSTIC IMAGING; DIAGNOSTIC REASONING; DIAGNOSTIC TEST ACCURACY STUDY; HUMAN; IMMUNOHISTOCHEMISTRY; LEARNING ALGORITHM; LYMPH NODE METASTASIS; MACHINE LEARNING; NETHERLANDS; PATHOLOGIST; PRIORITY JOURNAL; SENSITIVITY AND SPECIFICITY; SIMULATION; ALGORITHM; BREAST TUMOR; COMPARATIVE STUDY; FEMALE; PATHOLOGY; RECEIVER OPERATING CHARACTERISTIC;

ALGORITHMS; BREAST NEOPLASMS; FEMALE; HUMANS; LYMPHATIC METASTASIS; MACHINE LEARNING; PATHOLOGISTS; PATHOLOGY, CLINICAL; ROC CURVE;

EID: 85038431889     PISSN: 00987484     EISSN: 15383598     Source Type: Journal    
DOI: 10.1001/jama.2017.14585     Document Type: Article

References (34)

1. Griffin J, Treanor D. Digital pathology in clinical use: where are we now and whatisholding usback? Histopathology. 2017; 70(1):134-145.
2. Madabhushi A, Lee G. Image analysis and machine learning in digital pathology: challenges and opportunities. Med Image Anal. 2016; 33:170-175.
3. Gulshan V, Peng L, Coram M, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA. 2016; 316(22):2402-2410.
4. Esteva A, Kuprel B, Novoa RA, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017; 542 (7639):115-118.
5. Vestjens JHMJ, Pepels MJ, de Boer M, et al. Relevant impact of central pathology review on nodal classification in individual breast cancer patients. Ann Oncol. 2012; 23(10):2561-2566.
6. Litjens G, Sánchez CI, Timofeeva N, et al. Deep learning as a tool for increased accuracy and efficiency of histopathological diagnosis. Sci Rep. 2016; 6:26286.
7. Reed J, Rosman M, Verbanac KM, Mannie A, Cheng Z, Tafra L. Prognostic implications of isolated tumor cells and micrometastases in sentinel nodes of patients with invasive breast cancer: 10-year analysis of patients enrolled in the Prospective East Carolina University/Anne Arundel Medical Center Sentinel Node Multicenter Study. J Am Coll Surg. 2009; 208(3):333-340.
8. Chagpar A, Middleton LP, Sahin AA, et al. Clinical outcome ofpatients with lymph node-negative breast carcinoma who have sentinel lymph node micrometastases detectedbyimmunohistochemistry. Cancer. 2005; 103(8):1581-1586.
9. Pendas S, Dauway E, Cox CE, et al. Sentinel node biopsy and cytokeratin staining for the accurate staging of 478 breast cancer patients. Am Surg. 1999; 65(6):500-505.
10. Chakraborty DP. Recent developments in imaging system assessment methodology, FROC analysis and the search model. Nucl Instrum Methods Phys Res A. 2011; 648 supplement 1: S297-S301.
11. Efron B. Bootstrap methods: another look at the jackknife. Ann Stat. 1979; 7(1):1-26.
12. Gallas BD, Chan H-P, D'Orsi CJ, et al. Evaluating imaging and computer-aided detection and diagnosis devices at the FDA. Acad Radiol. 2012; 19 (4):463-477.
13. Obuchowski NA, Beiden SV, Berbaum KS, et al. Multireader, multicase receiver operating characteristic analysis: an empirical comparison of five methods. Acad Radiol. 2004; 11(9):980-995.
14. Hillis SL, Obuchowski NA, Berbaum KS. Power estimation for multireader ROC methods. Acad Radiol. 2011; 18(2):129-142. doi:10.1016/j.acra.2010.09.007
15. Upton G, Cook I. A Dictionary of Statistics 3e. Oxford, UK: Oxford University Press; 2014.
16. Mason SJ, Graham NE. Areas beneath the relative operating characteristics (ROC) and relative operating levels (ROL) curves: statistical significance and interpretation. Q J R Meteorol Soc. 2002; 128(584):2145-2166. doi:10.1256/003590002320603584
17. GitHub. DIDSR/iMRMC. https://github.com/DIDSR/iMRMC. Accessed November 14, 2017.
18. GitHub. CAMELYON16. https://github.com/computationalpathologygroup/CAMELYON16. Accessed November 14, 2017.
19. Lowe DG. Distinctive image features from scale-invariant keypoints. Int J Comput Vis. 2004; 60(2):91-110. https://people.eecs.berkeley.edu/~malik/cs294/lowe-ijcv04.pdf. Accessed November 13, 2017.
20. Ojala T, Pietikainen M, Maenpaa T. Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans Pattern Anal Mach Intell. 2002; 24(7): 971-987. doi:10.1109/TPAMI.2002.1017623
21. Haralick RM, Shanmugam K, Dinstein I. Textural features for image classification. IEEE Trans Syst Man Cybern. 1973; SMC-3(6):610-621. http://haralick.org/journals/TexturalFeatures.pdf. Accessed November 13, 2017.
22. Cortes C, Vapnik V. Support-vector networks. Mach Learn. 1995; 20(3):273-297. http://image.diku.dk/imagecanon/material/cortes-vapnik95.pdf. Accessed November 13, 2017.
23. Breiman L. Random forests. Mach Learn. 2001; 45(1):5-32. http://www.math.univ-toulouse.fr/~agarivie/Telecom/apprentissage/articles/randomforest2001.pdf. Accessed November 13, 2017.
24. Szegedy C, Wei L, Yangqing J, et al. Going deeper with convolutions. Paper presented at: IEEE Conference on Computer Vision and Pattern Recognition; June 7-12, 2015; Boston, MA. http://ieeexplore.ieee.org/document/7298594/. Accessed November 13, 2017.
25. He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. Paper presented at: IEEE Conference on Computer Vision and Pattern Recognition; June 27-30, 2016; Las Vegas, NV. https://www.cv-foundation.org/openaccess/content-cvpr-2016/papers/He-Deep-Residual-Learning-CVPR-2016-paper.pdf. Accessed November 13, 2017.
26. Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. https://arxiv.org/pdf/1409.1556.pdf. Accessed November 13, 2017.
27. Kendall A, Badrinarayanan V, Cipolla R. Bayesian segnet: model uncertainty in deep convolutional encoder-decoder architectures for scene understanding. http://mi.eng.cam.ac.uk/~cipolla/publications/inproceedings/2017-BMVC-bayesian-SegNet.pdf. Accessed November 13, 2017.
28. Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolutional neural networks. Paper presented at: Advances in Neural Information Processing Systems 25; December 3-8, 2012; Lake Tahoe, NV. https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf. Accessed November 13, 2017.
29. Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation. Paper presented at: International Conference on Medical Image Computing and Computer-Assisted Intervention; October 5-9, 2015; Munich, Germany. https://pdfs.semanticscholar.org/0704/5f87709d0b7b998794e9fa912c0aba912281.pdf. Accessed November 13, 2017.
30. Zheng S, Jayasumana S, Romera-Paredes B, et al. Conditional random fields as recurrent neural networks. Paper presented at: IEEE Conference on Computer Vision and Pattern Recognition; June 7-12, 2015; Boston, MA. http://www.robots.ox.ac.uk/~szheng/papers/CRFasRNN.pdf. Accessed November 13, 2017.
31. Viola P, Jones M. Fast and robust classification using asymmetric adaboost and a detector cascade. Paper presented at: Advances in Neural Information Processing Systems 15; December 9-14, 2002; Vancouver, British Columbia, Canada. https://pdfs.semanticscholar.org/90f6/e2c454909f819f20d9eb6c731ba709bbe8b6.pdf. Accessed November 13, 2017.
32. Albarqouni S, Baur C, Achilles F, Belagiannis V, Demirci S, Navab N. AggNet: deep learning from crowds for mitosis detection in breast cancer histology images. IEEE Trans Med Imaging. 2016; 35 (5):1313-1321.
33. Dorfman DD, Berbaum KS, Metz CE. Receiver operating characteristic rating analysis: generalization to the population of readers and patients with the jackknife method. Invest Radiol. 1992; 27(9):723-731.
34. Bejnordi BE, Litjens G, Timofeeva N, et al. Stain specific standardization of whole-slide histopathological images. IEEE Trans Med Imaging. 2016; 35(2):404-415.