Spatial Organization and Molecular Correlation of Tumor-Infiltrating Lymphocytes Using Deep Learning on Pathology Images

Cell Reports

Volumn 23, Issue 1, 2018, Pages 181-193.e7

  Saltz, Joel ^a   Gupta, Rajarsi ^a,d   Hou, Le ^b   Kurc, Tahsin ^a   Singh, Pankaj ^c   Nguyen, Vu ^b   Samaras, Dimitris ^b   Shroyer, Kenneth R ^d   Zhao, Tianhao ^d   Batiste, Rebecca ^d   Van Arnam, John ^e   Caesar Johnson, Samantha J ^h   Demchok, John A ^h   Felau, Ina ^h   Kasapi, Melpomeni ^h   Ferguson, Martin L ^h   Hutter, Carolyn M ^h   Sofia, Heidi J ^h   Tarnuzzer, Roy ^h   Wang, Zhining ^h   Yang, Liming ^h   Zenklusen, Jean C ^h   Shmulevich, Ilya ^f   Lazar, Alexander J ^c   Rao, Arvind U K ^c   Sharma, Ashish ^g   Thorsson, Vésteinn ^f   more..

^a Stony Brook University   (United States)

^b STONY BROOK UNIVERSITY   (United States)

^c UNIVERSITY OF TEXAS MD ANDERSON CANCER CENTER   (United States)

^d Stony Brook School of Dental Medicine   (United States)

^e UNIVERSITY OF PENNSYLVANIA   (United States)

^f INSTITUTE FOR SYSTEMS BIOLOGY   (United States)

^g EMORY UNIVERSITY SCHOOL OF MEDICINE   (United States)

^h Julia)

Author keywords

artificial intelligence; bioinformatics; computer vision; deep learning; digital pathology; immuno oncology; lymphocytes; machine learning; tumor microenvironment; tumor infiltrating lymphocytes

Indexed keywords

EOSIN; HEMATOXYLIN;

ADOLESCENT; ADULT; ARTICLE; ARTIFICIAL NEURAL NETWORK; CANCER GENETICS; CANCER SURVIVAL; CELL INFILTRATION; CELL STRUCTURE; CONTROLLED STUDY; CORRELATION COEFFICIENT; DEEP LEARNING METHOD; FEMALE; GENE MAPPING; HISTOPATHOLOGY; HUMAN; HUMAN CELL; HUMAN GENOME; HUMAN TISSUE; IMAGE ANALYSIS; MACHINE LEARNING; MAJOR CLINICAL STUDY; MALE; MOLECULAR PATHOLOGY; OVERALL SURVIVAL; PRIORITY JOURNAL; SPATIAL ANALYSIS; STAINING; T LYMPHOCYTE SUBPOPULATION; TUMOR ASSOCIATED LEUKOCYTE; COMPUTER ASSISTED DIAGNOSIS; METABOLISM; NEOPLASM; PATHOLOGY; PROCEDURES;

DEEP LEARNING; HUMANS; IMAGE INTERPRETATION, COMPUTER-ASSISTED; LYMPHOCYTES, TUMOR-INFILTRATING; NEOPLASMS;

EID: 85044586207     PISSN: None     EISSN: 22111247     Source Type: Journal    
DOI: 10.1016/j.celrep.2018.03.086     Document Type: Article

References (62)

1. Angell, H., Galon, J., From the immune contexture to the Immunoscore: the role of prognostic and predictive immune markers in cancer. Curr. Opin. Immunol. 25 (2013), 261–267.
2. Bailey, P., Chang, D.K., Nones, K., Johns, A.L., Patch, A.-M., Gingras, M.-C., Miller, D.K., Christ, A.N., Bruxner, T.J.C., Quinn, M.C., et al., Australian Pancreatic Cancer Genome Initiative. Genomic analyses identify molecular subtypes of pancreatic cancer. Nature 531 (2016), 47–52.
3. Ball, G.H., and Hall, D.J. (1965). ISODATA, a novel method of data analysis and pattern classification. In: Technical Report April 1965 prepared for the Information Sciences Branch of the Office of Naval Research. Stanford Research Institute - Clearinghouse for Federal Scientific and Technical Information, pp. 2–50.
4. Banfield, J.D., Raftery, A.E., Model-Based Gaussian and Non-Gaussian Clustering. Biometrics 49 (1993), 803–821.
5. Bayramoglu, N., and Heikkila, J. (2016). Transfer learning for cell nuclei classification in histopathology images. In Computer Vision – ECCV 2016 Workshops, G. Hua and H. Jégou, eds., Lecture Notes in Computer Science (Springer), pp. 532–539.
6. Bodenhofer, U., Kothmeier, A., Hochreiter, S., APCluster: an R package for affinity propagation clustering. Bioinformatics 27 (2011), 2463–2464.
7. Broussard, E.K., Disis, M.L., TNM staging in colorectal cancer: T is for T cell and M is for memory. J. Clin. Oncol. 29 (2011), 601–603.
8. Cancer Genome Atlas Network. Genomic Classification of Cutaneous Melanoma. Cell 161 (2015), 1681–1696.
9. Cancer Genome Atlas Research Network. Integrated genomic analyses of ovarian carcinoma. Nature 474 (2011), 609–615.
10. Cancer Genome Atlas Research Network. Comprehensive and Integrated Genomic Characterization of Adult Soft Tissue Sarcomas. Cell 171 (2017), 950–965.e28.
11. Charoentong, P., Finotello, F., Angelova, M., Mayer, C., Efremova, M., Rieder, D., Hackl, H., Trajanoski, Z., Pan-cancer Immunogenomic Analyses Reveal Genotype-Immunophenotype Relationships and Predictors of Response to Checkpoint Blockade. Cell Rep. 18 (2017), 248–262.
12. Chen, H., Qi, X., Yu, L., Dou, Q., Qin, J., Heng, P.A., DCAN: Deep contour-aware networks for object instance segmentation from histology images. Med. Image Anal. 36 (2017), 135–146.
13. Cireşan, D.C., Giusti, A., Gambardella, L.M., and Schmidhuber, J. (2013). Mitosis Detection in Breast Cancer Histology Images with Deep Neural Networks. In Medical Image Computing and Computer-Assisted Intervention – MICCAI 2013, Volume 8150, K. Mori, I. Sakuma, Y. Sato, C. Barillot, and N. Navab, eds., Lecture Notes in Computer Science (Springer), pp. 411–418.
14. Cooper, L.A., Demicco, E.G., Saltz, J.H., Powell, R.T., Rao, A., Lazar, A.J., PanCancer insights from The Cancer Genome Atlas: the pathologist's perspective. J. Pathol., 2017, 10.1002/path.5028 Published online December 30, 2017.
15. Crowson, A.N., Magro, C.M., Mihm, M.C., Prognosticators of melanoma, the melanoma report, and the sentinel lymph node. Mod. Pathol. 19:Suppl 2 (2006), S71–S87.
16. Drasgow, F., Polychoric and Polyserial Correlations. Balakrishnan, N., Colton, T., Everitt, B., Piegorsch, W., Ruggeri, F., Teugels, J.L., (eds.) Wiley StatsRef: Statistics Reference, 2014, John Wiley & Sons, 68–74.
17. FDA News Release (2017). FDA allows marketing of first whole slide imaging system for digital pathology. https://www.fda.gov/NewsEvents/Newsroom/PressAnnouncements/ucm552742.htm.
18. Frey, B.J., Dueck, D., Clustering by passing messages between data points. Science 315 (2007), 972–976.
19. Fridman, W.H., Pagès, F., Sautès-Fridman, C., Galon, J., The immune contexture in human tumours: impact on clinical outcome. Nat. Rev. Cancer 12 (2012), 298–306.
20. Galon, J., Costes, A., Sanchez-Cabo, F., Kirilovsky, A., Mlecnik, B., Lagorce-Pagès, C., Tosolini, M., Camus, M., Berger, A., Wind, P., et al. Type, density, and location of immune cells within human colorectal tumors predict clinical outcome. Science 313 (2006), 1960–1964.
21. Galon, J., Angell, H.K., Bedognetti, D., Marincola, F.M., The continuum of cancer immunosurveillance: prognostic, predictive, and mechanistic signatures. Immunity 39 (2013), 11–26.
22. Graves, A., and Jaitly, N. (2014). Towards end-to-end speech recognition with recurrent neural networks. In: Proceedings of the 31st International Conference on Machine Learning, Volume 32. JMLR, pp. 1764–1772.
23. Hendry, S., Salgado, R., Gevaert, T., Russell, P.A., John, T., Thapa, B., Christie, M., van de Vijver, K., Estrada, M.V., Gonzalez-Ericsson, P.I., et al. Assessing Tumor-infiltrating Lymphocytes in Solid Tumors: A Practical Review for Pathologists and Proposal for a Standardized Method From the International Immunooncology Biomarkers Working Group: Part 1: Assessing the Host Immune Response, TILs in Invasive Breast Carcinoma and Ductal Carcinoma In Situ, Metastatic Tumor Deposits and Areas for Further Research. Adv. Anat. Pathol. 24 (2017), 235–251.
24. Hendry, S., Salgado, R., Gevaert, T., Russell, P.A., John, T., Thapa, B., Christie, M., van de Vijver, K., Estrada, M.V., Gonzalez-Ericsson, P.I., et al. Assessing Tumor-Infiltrating Lymphocytes in Solid Tumors: A Practical Review for Pathologists and Proposal for a Standardized Method from the International Immuno-Oncology Biomarkers Working Group: Part 2: TILs in Melanoma, Gastrointestinal Tract Carcinomas, Non-Small Cell Lung Carcinoma and Mesothelioma, Endometrial and Ovarian Carcinomas, Squamous Cell Carcinoma of the Head and Neck, Genitourinary Carcinomas, and Primary Brain Tumors. Adv. Anat. Pathol. 24 (2017), 311–335.
25. Hou, L., Samaras, D., Kurc, T., Gao, Y., Davis, J.E., and Saltz, J.H. (2016a). Patch-based convolutional neural network for whole slide tissue image classification. In: Computer Vision and Pattern Recognition. arXiv:1504.07947v5.
26. Hou, L., Singh, K., Samaras, D., Kurc, T.M., Gao, Y., Seidman, R.J., Saltz, J.H., Automatic histopathology image analysis with CNNs. In: 2016 New York Scientific Data Summit (NYSDS). 2016, IEEE, 1–6.
27. Hou, L., Nguyen, V., Samaras, D., Kurc, T.M., Gao, Y., Zhao, T., and Saltz, J.H. (2017). Sparse Autoencoder for Unsupervised Nucleus Detection and Representation in Histopathology Images. Computer Vision and Pattern Recognition. arXiv:1704.00406v2.
28. Huang, G., Liu, Z., Weinberger, K.Q., and van der Maaten, L. (2016). Densely connected convolutional networks. Computer Vision and Pattern Recognition. arXiv:1608.06993v5.
29. Hubert, L., Schultz, J., Quadratic Assignment as a General Data-analysis Strategy. British Journal of Mathematical and Statistical Psychology 29 (1976), 190–241.
30. Iglesia, M.D., Vincent, B.G., Parker, J.S., Hoadley, K.A., Carey, L.A., Perou, C.M., Serody, J.S., Prognostic B-cell signatures using mRNA-seq in patients with subtype-specific breast and ovarian cancer. Clin. Cancer Res. 20 (2014), 3818–3829.
31. Iglesia, M.D., Parker, J.S., Hoadley, K.A., Serody, J.S., Perou, C.M., Vincent, B.G., Genomic Analysis of Immune Cell Infiltrates Across 11 Tumor Types. J. Natl. Cancer Inst., 108, 2016, djw144.
32. Kardos, J., Chai, S., Mose, L.E., Selitsky, S.R., Krishnan, B., Saito, R., Iglesia, M.D., Milowsky, M.I., Parker, J.S., Kim, W.Y., Vincent, B.G., Claudin-low bladder tumors are immune infiltrated and actively immune suppressed. JCI Insight, 1, 2016, e85902.
33. Kokkinos, I. (2017). Ubernet: Training a universal convolutional neural network for low-,mid-, and high-level vision using diverse datasets and limited memory. Computer Vision and Pattern Recognition. arXiv:1609.02132v1.
34. Li, B., Severson, E., Pignon, J.C., Zhao, H., Li, T., Novak, J., Jiang, P., Shen, H., Aster, J.C., Rodig, S., et al. Comprehensive analyses of tumor immunity: implications for cancer immunotherapy. Genome Biol., 17, 2016, 174.
35. Masci, J., Meier, U., Cireşan, D., Schmidhuber, J., Stacked Convolutional Auto-Encoders for Hierarchical Feature Extraction. Honkela, T., Duch, W., Girolami, M., Kaski, S., (eds.) Artificial Neural Networks and Machine Learning – ICANN 2011, 2011, Springer, 52–59.
36. Mlecnik, B., Bindea, G., Pagès, F., Galon, J., Tumor immunosurveillance in human cancers. Cancer Metastasis Rev. 30 (2011), 5–12.
37. Mlecnik, B., Tosolini, M., Kirilovsky, A., Berger, A., Bindea, G., Meatchi, T., Bruneval, P., Trajanoski, Z., Fridman, W.H., Pagès, F., Galon, J., Histopathologic-based prognostic factors of colorectal cancers are associated with the state of the local immune reaction. J. Clin. Oncol. 29 (2011), 610–618.
38. Murthy, V., Hou, L., Samaras, D., Kurc, T.M., and Saltz, J.H. (2017). Center-focusing multitask CNN with injected features for classification of glioma nuclear images. Winter Conference on Applications of Computer Vision (WACV) - Computer Vision and Pattern Recognition. arXiv:1612.06825v2.
39. Newman, A.M., Liu, C.L., Green, M.R., Gentles, A.J., Feng, W., Xu, Y., Hoang, C.D., Diehn, M., Alizadeh, A.A., Robust enumeration of cell subsets from tissue expression profiles. Nat. Methods 12 (2015), 453–457.
40. Noh, H., Hong, S., and Han, B. (2015). Learning Deconvolution Network for Semantic Segmentation. 2015 IEEE International Conference on Computer Vision (ICCV) - Computer Vision and Pattern Recognition. arXiv:1505.04366v1, pp. 1520–1528.
41. Ranzato, M., Poultney, C., Chopra, S., and LeCun, Y. (2006). Efficient learning of sparse representations with an energy-based model. Proceedings of the 19th International Conference on Neural Information Processing Systems, pp. 1137-1144.
42. Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016). You only look once: Unified, real-time object detection. Computer Vision and Pattern Recognition. arXiv:1506.02640v5.
43. Ren, S., He, K., Girshick, R., and Sun, J. (2015). Faster R-CNN: Towards real-time object detection with region proposal networks. In: Computer Vision and Pattern Recognition. arXiv:1506.01497v3.
44. Rooney, M.S., Shukla, S.A., Wu, C.J., Getz, G., Hacohen, N., Molecular and genetic properties of tumors associated with local immune cytolytic activity. Cell 160 (2015), 48–61.
45. Rutledge, W.C., Kong, J., Gao, J., Gutman, D.A., Cooper, L.A.D., Appin, C., Park, Y., Scarpace, L., Mikkelsen, T., Cohen, M.L., et al. Tumor-infiltrating lymphocytes in glioblastoma are associated with specific genomic alterations and related to transcriptional class. Clin. Cancer Res. 19 (2013), 4951–4960.
46. Salgado, R., Denkert, C., Demaria, S., Sirtaine, N., Klauschen, F., Pruneri, G., Wienert, S., Van den Eynden, G., Baehner, F.L., Penault-Llorca, F., et al., International TILs Working Group 2014. The evaluation of tumor-infiltrating lymphocytes (TILs) in breast cancer: recommendations by an International TILs Working Group 2014. Ann. Oncol. 26 (2015), 259–271.
47. Saltz, J., Sharma, A., Iyer, G., Bremer, E., Wang, F., Jasniewski, A., DiPrima, T., Almeida, J.S., Gao, Y., Zhao, T., et al. A Containerized Software System for Generation, Management, and Exploration of Features from Whole Slide Tissue Images. Cancer Res. 77 (2017), e79–e82.
48. Scott, A.J., and Symons, M. J. (1971). Clustering Methods Based on Likelihood Ratio Criteria. Biometrics, 27, 387–397.
49. Sharma, A., Kazerouni, A., Saghar, N., Commean, P., Tarbox, L., Prior, F., Framework for Data Management and Visualization of The National Lung Screening Trial Pathology Images. Pathology Informatics Summit 2014, 2014, J. Pathol. Inform., S30–S31.
50. Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. Computer Vision and Pattern Recognition. arXiv:1409.1556v6.
51. Sirinukunwattana, K., Ahmed Raza, S.E., Yee-Wah Tsang, Snead, D.R., Cree, I.A., Rajpoot, N.M., Locality sensitive deep learning for detection and classification of nuclei in routine colon cancer histology images. IEEE Trans. Med. Imaging 35 (2016), 1196–1206.
52. Su, H., Xing, F., Kong, X., Xie, Y., Zhang, S., and Yang, L. (2015). Robust cell detection and segmentation in histopathological images using sparse reconstruction and stacked denoising autoencoders. In Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, Volume 9351, N. Navab, J. Hornegger, W. Wells, and A. Frangi, eds., Lecture Notes in Computer Science (Springer), pp. 383–390.
53. Theano Development Team (2016). Theano: A Python framework for fast computation of mathematical expressions. Symbolic Computation; Learning; Mathematical Software. arXiv:1605.02688v1.
54. Thorsson, V., Gibbs, D.L., Brown, S.D., Wolf, D., Bortone, D.S., Yang, T.-H.O., Porta-Pardo, E., Gao, G., Plaisier, C.L., Eddy, J.A., et al. The Immune Landscape of Cancer. Immunity, 48, 2018, 10.1016/j.immuni.2018.03.023.
55. Trinchieri, G., Cancer and inflammation: an old intuition with rapidly evolving new concepts. Annu. Rev. Immunol. 30 (2012), 677–706.
56. Wang, S., Yao, J., Xu, Z., and Huang, J. (2016). Subtype cell detection with an accelerated deep convolution neural network. In Medical Image Computing and Computer-Assisted Intervention – MICCAI 2016, Volume 9901, S. Ourselin, L. Joskowicz, M. Sabuncu, G. Unal, and W. Wells, eds., Lecture Notes in Computer Science (Springer), pp. 640–648.
57. Wilkerson, M.D., Yin, X., Hoadley, K.A., Liu, Y., Hayward, M.C., Cabanski, C.R., Muldrew, K., Miller, C.R., Randell, S.H., Socinski, M.A., et al. Lung squamous cell carcinoma mRNA expression subtypes are reproducible, clinically important, and correspond to normal cell types. Clin. Cancer Res. 16 (2010), 4864–4875.
58. Xie, Y., Kong, X., Xing, F., Liu, F., Su, H., and Yang, L. (2015a). Deep voting: A robust approach toward nucleus localization in microscopy images. In Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, Volume 9351, N. Navab, J. Hornegger, W. Wells, and A. Frangi, eds., Lecture Notes in Computer Science (Springer), pp. 374–382.
59. Xie, Y., Xing, F., Kong, X., Su, H., and Yang, L. (2015b). Beyond classification: structured regression for robust cell detection using convolutional neural network. In Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, Volume 9351, N. Navab, J. Hornegger, W. Wells, and A. Frangi, eds., Lecture Notes in Computer Science (Springer), pp. 358–365.
60. Xu, Z., and Huang, J. (2016). Detecting 10,000 cells in one second. In Medical Image Computing and Computer-Assisted Intervention – MICCAI 2016, Volume 9901, S. Ourselin, L. Joskowicz, M. Sabuncu, G. Unal, and W. Wells, eds., Lecture Notes in Computer Science (Springer), pp. 676–684.
61. Xu, Y., Jia, Z., Ai, Y., Zhang, F., Lai, M., and Chang, E.I.C. (2015) Deep convolutional activation features for large scale brain tumor histopathology image classification and segmentation. 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 947–951.
62. Zhao, T., Hou, L., Nguyen, V., Gao, Y., Samaras, D., Kurc, T.M., and Saltz, J.H. (2017). Using machine methods to score tumor-infiltrating lymphocytes in lung cancer. USCAP 2017 Annual Meeting. Proffered papers, Section A.