Deep Convolutional Neural Networks Enable Discrimination of Heterogeneous Digital Pathology Images

EBioMedicine

Volumn 27, Issue , 2018, Pages 317-328

  Khosravi, Pegah ^a   Kazemi, Ehsan ^b   Imielinski, Marcin ^a,c   Elemento, Olivier ^a   Hajirasouliha, Iman ^a  

^a WEILL CORNELL MEDICAL COLLEGE   (United States)

^b YALE UNIVERSITY   (United States)

^c NEW YORK GENOME CENTER   (United States)

Author keywords

Biomarkers; Classification; Convolutional Neural Network; Deep learning; Digital pathology imaging; Tumor heterogeneity

Indexed keywords

BIOLOGICAL MARKER;

ACCURACY; ALGORITHM; AREA UNDER THE CURVE; ARTICLE; BLADDER CANCER; BREAST CANCER; CELL HETEROGENEITY; CONVOLUTIONAL NEURAL NETWORK; HISTOPATHOLOGY; HUMAN; HUMAN TISSUE; IMMUNOHISTOCHEMISTRY; LEARNING; LUNG CANCER; NERVE CELL NETWORK; PATHOLOGY; PHYSICAL PERFORMANCE; PRIORITY JOURNAL; RECEIVER OPERATING CHARACTERISTIC; SENSITIVITY AND SPECIFICITY; TUMOR GROWTH; ARTIFICIAL NEURAL NETWORK; CLASSIFICATION; IMAGE PROCESSING; NEOPLASM;

ALGORITHMS; HUMANS; IMAGE PROCESSING, COMPUTER-ASSISTED; NEOPLASMS; NEURAL NETWORKS (COMPUTER); ROC CURVE;

EID: 85039560062     PISSN: None     EISSN: 23523964     Source Type: Journal    
DOI: 10.1016/j.ebiom.2017.12.026     Document Type: Article

References (90)

1. Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. Tensorflow: A system for large-scale machine learning. OSDI, 2016, 265–283.
2. Akram, S., Javed, M.Y., Qamar, U., Khanum, A., Hassan, A., Artificial neural network based classification of lungs nodule using hybrid features from computerized tomographic images. Appl. Math. Inf. Sci., 9, 2015, 183.
3. Allouche, O., Tsoar, A., Kadmon, R., Assessing the accuracy of species distribution models: prevalence, kappa and the true skill statistic (tss). J. Appl. Ecol. 43 (2006), 1223–1232.
4. Antony, J., McGuinness, K., O'Connor, N.E., Moran, K., Quantifying radiographic knee osteoarthritis severity using deep convolutional neural networks. Pattern Recognition (ICPR), 2016 23rd International Conference on, 2016, IEEE, 1195–1200.
5. Beck, A.H., Sangoi, A.R., Leung, S., Marinelli, R.J., Nielsen, T.O., Van De Vijver, M.J., West, R.B., Van De Rijn, M., Koller, D., Systematic analysis of breast cancer morphology uncovers stromal features associated with survival. Sci. Transl. Med., 3, 2011 108ra113–108ra113.
6. Bengio, Y., Lamblin, P., Popovici, D., Larochelle, H., Greedy layer-wise training of deep networks. Advances in neural information processing systems, 2007, 153–160.
7. Bhargava, R., Sinha, S., Kwak, J.T., Multimodal Microscopy for Automated Histologic Analysis of Prostate Cancer. 2011 US Patent App. 13/090,384.
8. Bloom, K., Harrington, D., Enhanced accuracy and reliability of her-2/neu immunohistochemical scoring using digital microscopy. Am. J. Clin. Pathol. 121 (2004), 620–630.
9. Bottou, Y.L.L., GO, M., K.: Efficient backprop. Neural Networks: Tricks of the Trade, 1998, Springer.
10. Bouzin, C., Saini, M.L., Khaing, K.-K., Ambroise, J., Marbaix, E., Grégoire, V., Bol, V., Digital pathology: elementary, rapid and reliable automated image analysis. Histopathology 68 (2016), 888–896.
11. Bueno-de Mesquita, J.M., Nuyten, D., Wesseling, J., van Tinteren, H., Linn, S., van De Vijver, M., The impact of inter-observer variation in pathological assessment of node-negative breast cancer on clinical risk assessment and patient selection for adjuvant systemic treatment. Ann. Oncol. 21 (2009), 40–47.
12. Carneiro, G., Zheng, Y., Xing, F., Yang, L., Review of deep learning methods in mammography, cardiovascular, and microscopy image analysis. Deep Learning and Convolutional Neural Networks for Medical Image Computing, 2017, Springer, 11–32.
13. Chen, X., Xu, Y., Wong, D.W.K., Wong, T.Y., Liu, J., Glaucoma detection based on deep convolutional neural network. Engineering in Medicine and Biology Society (EMBC), 2015 37th Annual International Conference of the IEEE, 2015, IEEE, 715–718.
14. Chen, Y.-J., Fan, C.-Y., Chang, K.-H., Manufacturing intelligence for reducing false alarm of defect classification by integrating similarity matching approach in cmos image sensor manufacturing. Comput. Ind. Eng. 99 (2016), 465–473.
15. Cohen, J., A coefficient of agreement for nominal scales. Educ. Psychol. Meas. 20 (1960), 37–46.
16. Conde, E., Angulo, B., Redondo, P., Toldos, O., García-García, E., Suárez-Gauthier, A., Rubio-Viqueira, B., Marrón, C., García-Luján, R., Sánchez-Céspedes, M., et al. The use of p63 immunohistochemistry for the identification of squamous cell carcinoma of the lung. PLoS One, 5, 2010, e12209.
17. Dong, F., Irshad, H., Oh, E.-Y., Lerwill, M.F., Brachtel, E.F., Jones, N.C., Knoblauch, N.W., Montaser-Kouhsari, L., Johnson, N.B., Rao, L.K., et al. Computational pathology to discriminate benign from malignant intraductal proliferations of the breast. PloS One, 9, 2014, e114885.
18. Doyle, S., Feldman, M., Tomaszewski, J., Madabhushi, A., A boosted bayesian multiresolution classifier for prostate cancer detection from digitized needle biopsies. IEEE Trans. Biomed. Eng. 59 (2012), 1205–1218.
19. Drish, J., Obtaining calibrated probability estimates from support vector machines. Technique Report, Department of Computer Science and Engineering, 2001, University of California, San Diego, CA.
20. Esteva, A., Kuprel, B., Novoa, R.A., Ko, J., Swetter, S.M., Blau, H.M., Thrun, S., Dermatologist-level classification of skin cancer with deep neural networks. Nature 542 (2017), 115–118.
21. Fatima, N., Cohen, C., Lawson, D., Siddiqui, M.T., Ttf-1 and napsin a double stain. Cancer Cytopathol. 119 (2011), 127–133.
22. Felipe De Sousa, E.M., Wang, X., Jansen, M., Fessler, E., Trinh, A., De Rooij, L.P., De Jong, J.H., De Boer, O.J., Van Leersum, R., Bijlsma, M.F., et al. Poor-prognosis colon cancer is defined by a molecularly distinct subtype and develops from serrated precursor lesions. Nat. Med. 19 (2013), 614–618.
23. Fogel, Y., Feder, M., On the problem of on-line learning with log-loss. Information Theory (ISIT), 2017 IEEE International Symposium on, 2017, IEEE, 2995–2999.
24. Gavrielides, M.A., Gallas, B.D., Lenz, P., Badano, A., Hewitt, S.M., Observer variability in the interpretation of her2/neu immunohistochemical expression with unaided and computer-aided digital microscopy. Arch. Pathol. Lab. Med. 135 (2011), 233–242.
25. Greenspan, H., van Ginneken, B., Summers, R.M., Guest editorial deep learning in medical imaging: overview and future promise of an exciting new technique. IEEE Trans. Med. Imaging 35 (2016), 1153–1159.
26. Gulshan, V., Peng, L., Coram, M., Stumpe, M.C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. Jama 316 (2016), 2402–2410.
27. Gurcan, M.N., Histopathological image analysis: Path to acceptance through evaluation. Microsc. Microanal. 22 (2016), 1004–1005.
28. Gutman, D.A., Cobb, J., Somanna, D., Park, Y., Wang, F., Kurc, T., Saltz, J.H., Brat, D.J., Cooper, L.A., Kong, J., Cancer digital slide archive: an informatics resource to support integrated in silico analysis of tcga pathology data. J. Am. Med. Inform. Assoc. 20 (2013), 1091–1098.
29. Hamilton, P.W., Wang, Y., Boyd, C., James, J.A., Loughrey, M.B., Hougton, J.P., Boyle, D.P., Kelly, P., Maxwell, P., McCleary, D., et al. Automated tumor analysis for molecular profiling in lung cancer. Oncotarget, 6, 2015, 27938.
30. Hanley, J.A., McNeil, B.J., The meaning and use of the area under a receiver operating characteristic (roc) curve. Radiology 143 (1982), 29–36.
31. Hardiman, K.M., Ulintz, P.J., Kuick, R., Hovelson, D.H., Gates, C.M., Bhasi, A., Grant, A.R., Liu, J., Cani, A.K., Greenson, J., et al. Intra-tumor genetic heterogeneity in rectal cancer. Lab. Investig. J. Tech. Methods Pathol., 96, 2016, 4.
32. He, K., Zhang, X., Ren, S., Sun, J., Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, 770–778.
33. Higgins, J.P., Kaygusuz, G., Wang, L., Montgomery, K., Mason, V., Zhu, S.X., Marinelli, R.J., Presti, J.C. Jr, van de Rijn, M., Brooks, J.D., Placental s100 (s100p) and gata3: markers for transitional epithelium and urothelial carcinoma discovered by complementary dna microarray. Am. J. Surg. Pathol. 31 (2007), 673–680.
34. Hinton, G.E., Osindero, S., Teh, Y.-W., A fast learning algorithm for deep belief nets. Neural Comput. 18 (2006), 1527–1554.
35. Hinton, G.E., Salakhutdinov, R.R., Reducing the dimensionality of data with neural networks. Science 313 (2006), 504–507.
36. Ioffe, S., Szegedy, C., Batch normalization: Accelerating deep network training by reducing internal covariate shift. International Conference on Machine Learning, 2015, 448–456.
37. Janowczyk, A., Chandran, S., Singh, R., Sasaroli, D., Coukos, G., Feldman, M.D., Madabhushi, A., High-throughput biomarker segmentation on ovarian cancer tissue microarrays via hierarchical normalized cuts. IEEE Trans. Biomed. Eng. 59 (2012), 1240–1252.
38. Jiang, M., Zhang, S., Huang, J., Yang, L., Metaxas, D.N., Scalable histopathological image analysis via supervised hashing with multiple features. Med. Image Anal. 34 (2016), 3–12.
39. Kaufman, P.A., Bloom, K.J., Burris, H., Gralow, J.R., Mayer, M., Pegram, M., Rugo, H.S., Swain, S.M., Yardley, D.A., Chau, M., et al. Assessing the discordance rate between local and central her2 testing in women with locally determined her2-negative breast cancer. Cancer 120 (2014), 2657–2664.
40. Khayyata, S., Yun, S., Pasha, T., Jian, B., McGrath, C., Yu, G., Gupta, P., Baloch, Z., Value of p63 and ck5/6 in distinguishing squamous cell carcinoma from adenocarcinoma in lung fine-needle aspiration specimens. Diagn. Cytopathol. 37 (2009), 178–183.
41. Khosravi, P., Gazestani, V.H., Pirhaji, L., Law, B., Sadeghi, M., Goliaei, B., Bader, G.D., Inferring interaction type in gene regulatory networks using co-expression data. Algorithms Mol. Biol., 10, 2015, 23.
42. Kim, E., Corte-Real, M., Baloch, Z., A deep semantic mobile application for thyroid cytopathology. Proc. SPIE, 2016, 97890A.
43. Korbar, B., Olofson, A.M., Miraflor, A.P., Nicka, K.M., Suriawinata, M.A., Torresani, L., Suriawinata, A.A., Hassanpour, S., Deep-learning for Classification of Colorectal Polyps on Whole-slide Images. 2017 arXiv preprint arXiv:1703.01550.
44. Kothari, S., Phan, J.H., Stokes, T.H., Wang, M.D., Pathology imaging informatics for quantitative analysis of whole-slide images. J. Am. Med. Inform. Assoc. 20 (2013), 1099–1108.
45. Krizhevsky, A., Sutskever, I., Hinton, G.E., Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 2012, 1097–1105.
46. Lakhani, S.R., Reis-Filho, J.S., Fulford, L., Penault-Llorca, F., van der Vijver, M., Parry, S., Bishop, T., Benitez, J., Rivas, C., Bignon, Y.-J., et al. Prediction of brca1 status in patients with breast cancer using estrogen receptor and basal phenotype. Clin. Cancer Res. 11 (2005), 5175–5180.
47. Lam, K.-M., He, X.-J., Choi, K.-S., Using artificial neural network to predict mortality of radical cystectomy for bladder cancer. Smart Computing (SMARTCOMP), 2014 International Conference on, 2014, IEEE, 201–207.
48. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., Gradient-based learning applied to document recognition. Proc. IEEE 86 (1998), 2278–2324.
49. Lemaître, G., Martí, R., Freixenet, J., Vilanova, J.C., Walker, P.M., Meriaudeau, F., Computer-aided detection and diagnosis for prostate cancer based on mono and multi-parametric mri: a review. Comput. Biol. Med. 60 (2015), 8–31.
50. Lin, M., Chen, Q., Yan, S., Network in Network. 2013 arXiv preprint arXiv:1312.4400.
51. Litjens, G., Kooi, T., Bejnordi, B.E., Setio, A.A.A., Ciompi, F., Ghafoorian, M., van der Laak, J.A., van Ginneken, B., Sánchez, C.I., A Survey on Deep Learning in Medical Image Analysis. 2017 arXiv preprint arXiv:1702.05747.
52. Liu, Y., Gadepalli, K., Norouzi, M., Dahl, G.E., Kohlberger, T., Boyko, A., Venugopalan, S., Timofeev, A., Nelson, P.Q., Corrado, G.S., et al. Detecting Cancer Metastases on Gigapixel Pathology Images. 2017 arXiv preprint arXiv:1703.02442.
53. Lo, S.-C., Lou, S.-L., Lin, J.-S., Freedman, M.T., Chien, M.V., Mun, S.K., Artificial convolution neural network techniques and applications for lung nodule detection. IEEE Trans. Med. Imaging 14 (1995), 711–718.
54. Louis, D.N., Gerber, G.K., Baron, J.M., Bry, L., Dighe, A.S., Getz, G., Higgins, J.M., Kuo, F.C., Lane, W.J., Michaelson, J.S., et al. Computational pathology: an emerging definition. Arch. Pathol. Lab. Med. 138 (2014), 1133–1138.
55. Marinelli, R.J., Montgomery, K., Liu, C.L., Shah, N.H., Prapong, W., Nitzberg, M., Zachariah, Z.K., Sherlock, G.J., Natkunam, Y., West, R.B., et al. The Stanford tissue microarray database. Nucleic Acids Res. 36 (2007), D871–D877.
56. Marino, F.Z., Liguori, G., Aquino, G., La Mantia, E., Bosari, S., Ferrero, S., Rosso, L., Gaudioso, G., De Rosa, N., Scrima, M., et al. Intratumor heterogeneity of alk-rearrangements and homogeneity of egfr-mutations in mixed lung adenocarcinoma. PloS One, 10, 2015, e0139264.
57. Menegola, A., Fornaciali, M., Pires, R., Avila, S., Valle, E., Towards Automated Melanoma Screening: Exploring Transfer Learning Schemes. 2016 arXiv preprint arXiv:1609.01228.
58. Mohanty, S.P., Hughes, D.P., Salathé, M., Using deep learning for image-based plant disease detection. Front. Plant Sci., 7, 2016.
59. Mosquera-Lopez, C., Agaian, S., Velez-Hoyos, A., Thompson, I., Computer-aided prostate cancer diagnosis from digitized histopathology: a review on texture-based systems. IEEE Rev. Biomed. Eng. 8 (2015), 98–113.
60. Movshovitz-Attias, Y., Yu, Q., Stumpe, M.C., Shet, V., Arnoud, S., Yatziv, L., Ontological supervision for fine grained classification of street view storefronts. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, 1693–1702.
61. Nassar, A., Radhakrishnan, A., Cabrero, I.A., Cotsonis, G.A., Cohen, C., Intratumoral heterogeneity of immunohistochemical marker expression in breast carcinoma: a tissue microarray-based study. Appl. Immunohistochem. Mol. Morphol. 18 (2010), 433–441.
62. Network, C.G.A.R., et al. Comprehensive genomic characterization of squamous cell lung cancers. Nature, 489, 2012, 519.
63. Network, C.G.A.R., et al. Comprehensive molecular profiling of lung adenocarcinoma. Nature, 511, 2014, 543.
64. Perez, E.A., Suman, V.J., Davidson, N.E., Martino, S., Kaufman, P.A., Lingle, W.L., Flynn, P.J., Ingle, J.N., Visscher, D., Jenkins, R.B., Her2 testing by local, central, and reference laboratories in specimens from the north central cancer treatment group n9831 intergroup adjuvant trial. J. Clin. Oncol. 24 (2006), 3032–3038.
65. Polyak, K., Heterogeneity in breast cancer. J. Clin. Invest., 121, 2011, 3786.
66. Prashanth, R., Deepak, K., Meher, A.K., High accuracy predictive modelling for customer churn prediction in telecom industry. International Conference on Machine Learning and Data Mining in Pattern Recognition, 2017, 391–402.
67. Razzak, M.I., Naz, S., Zaib, A., Deep Learning for Medical Image Processing: Overview, Challenges and Future. 2017 arXiv preprint arXiv:1704.06825.
68. Roche, P.C., Suman, V.J., Jenkins, R.B., Davidson, N.E., Martino, S., Kaufman, P.A., Addo, F.K., Murphy, B., Ingle, J.N., Perez, E.A., Concordance between local and central laboratory her2 testing in the breast intergroup trial n9831. J. Natl. Cancer Inst. 94 (2002), 855–857.
69. Roth, K.A., Almeida, J.S., Coming Into Focus: Computational Pathology as the New Big Data Microscope. 2015.
70. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. Imagenet Large Scale Visual Recognition Challenge. 2014 arXiv preprint arXiv:1409.0575.
71. Sarnecki, J.S., Burns, K.H., Wood, L.D., Waters, K.M., Hruban, R.H., Wirtz, D., Wu, P.-H., A robust nonlinear tissue-component discrimination method for computational pathology. Lab. Investig. 96 (2016), 450–458.
72. Scagliotti, G.V., Parikh, P., Von Pawel, J., Biesma, B., Vansteenkiste, J., Manegold, C., Serwatowski, P., Gatzemeier, U., Digumarti, R., Zukin, M., et al. Phase iii study comparing cisplatin plus gemcitabine with cisplatin plus pemetrexed in chemotherapy-naive patients with advanced-stage non-small-cell lung cancer. J. Clin. Oncol. 26 (2008), 3543–3551.
73. Schroff, F., Kalenichenko, D., Philbin, J., Facenet: A unified embedding for face recognition and clustering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, 815–823.
74. Simard, P.Y., Steinkraus, D., Platt, J.C., et al. Best practices for convolutional neural networks applied to visual document analysis. ICDAR, 2003, 958–962.
75. Simonyan, K., Zisserman, A., Very Deep Convolutional Networks for Large-Scale Image Recognition. 2014 arXiv preprint arXiv:1409.1556.
76. Szegedy, C., Ioffe, S., Vanhoucke, V., Alemi, A.A., Inception-v4, inception-resnet and the impact of residual connections on learning. AAAI, 2017, 4278–4284.
77. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A., Going deeper with convolutions. Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, 1–9.
78. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z., Rethinking the inception architecture for computer vision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, 2818–2826.
79. Tang, P., Wang, J., Bourne, P., Molecular classifications of breast carcinoma with similar terminology and different definitions: are they the same?. Hum. Pathol. 39 (2008), 506–513.
80. van der Laak, J.A., Pahlplatz, M.M., Hanselaar, A.G., de Wilde, P., Hue-saturation-density (hsd) model for stain recognition in digital images from transmitted light microscopy. Cytometry A 39 (2000), 275–284.
81. Vandenberghe, M.E., Scott, M.L., Scorer, P.W., Söderberg, M., Balcerzak, D., Barker, C., Relevance of deep learning to facilitate the diagnosis of her2 status in breast cancer. Sci. Rep., 7, 2017.
82. Vincent, L., Soille, P., Watersheds in digital spaces: an efficient algorithm based on immersion simulations. IEEE Trans. Pattern Anal. Mach. Intell. 13 (1991), 583–598.
83. Vogel, C., Bloom, K., Burris, H., Gralow, J., Mayer, M., Pegram, M., Rugo, H., Swain, S., Yardley, D., Chau, M., et al. P1-07-02: Discordance Between Central and Local Laboratory her2 Testing From a Large her2-negative Population in Virgo, A Metastatic Breast Cancer Registry. 2011.
84. Wang, L.-W., Qu, A.-P., Yuan, J.-P., Chen, C., Sun, S.-R., Hu, M.-B., Liu, J., Li, Y., Computer-based image studies on tumor nests mathematical features of breast cancer and their clinical prognostic value. PLoS One, 8, 2013, e82314.
85. Webster, J., Dunstan, R., Whole-slide imaging and automated image analysis: considerations and opportunities in the practice of pathology. Vet. Pathol. 51 (2014), 211–223.
86. Xu, Y., Jia, Z., Wang, L.-B., Ai, Y., Zhang, F., Lai, M., Eric, I., Chang, C., Large scale tissue histopathology image classification, segmentation, and visualization via deep convolutional activation features. BMC Bioinf., 18, 2017, 281.
87. Youden, W.J., Index for rating diagnostic tests. Cancer 3 (1950), 32–35.
88. Yu, K.-H., Zhang, C., Berry, G.J., Altman, R.B., Ré, C., Rubin, D.L., Snyder, M., Predicting non-small cell lung cancer prognosis by fully automated microscopic pathology image features. Nat. Commun., 7, 2016.
89. Zadrozny, B., Elkan, C., Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. ICML, 2001, 609–616.
90. Zawistowski, M., Sussman, J.B., Hofer, T.P., Bentley, D., Hayward, R.A., Wiitala, W.L., Corrected roc analysis for misclassified binary outcomes. Stat. Med. 36 (2017), 2148–2160.